This is me again! In previous pages, we have learned how LLMs are trained, how they work, and how they become so powerful.
Despite the high capabilities of LLMs, they are prone to hallucinations, which are plausible yet nonfactual content.
Let's understand this better.
LLMs aren’t designed to pump out facts. Rather, as we have seen, LLMs compose responses that are statistically likely, based on patterns in their training data and on subsequent fine-tuning. Moreover, systems based on LLMs, during fine-tuning, are developed to generate highly convincing and human-like responses, which makes detecting these hallucinations particularly challenging.
Since LLMs are being increasingly adopted and influencing decision-making directly, possible misleading information provided by AI systems has the potential to spread misinformation and false beliefs, or even cause harm.
A famous example was in 2023 from a US lawyer, Steven Schwartz, who cited non-existent legal cases in a court filing after using ChatGPT.
Armstrong, K. (2023, May 27). ChatGPT: US lawyer admits using AI for case research. BBC News. https://www.bbc.com/news/world-us-canada-65735769
Hallucinations are defined as generated content that is either nonsensical or unfaithful to the provided source content.
Huang, Lei, Weijiang Yu, Weitao Ma, et al. "A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions." ACM Trans. Inf. Syst. 43, no. 2 (2025): 42:1-42:55. https://doi.org/10.1145/3703155
We have seen that for the pre-training phase, a massive corpora of text from the internet, mainly, is used for training purposes. However, this data can contain misinformation such as fake news and unfounded rumors and biases, which could be memorised by the LLMs, and misinformation and biases can be amplified.
A famous example was the Google AI tool that suggested adding glue to pizza sauce to stop the cheese from sliding off and said geologists recommend humans eat one rock per day. These outputs were traced back to a post on the social network Reddit, or articles written by satirical site, The Onion, which were probably included in the training process of the LLM.
McMahon, L., & Kleinman, Z. (2024, May 24). Google AI search tells users to glue pizza and eat rocks. BBC News. https://www.bbc.com/news/articles/cd11gzejgz4o
To reduce the presence of misinformation and biases, an intuitive approach involves the careful selection of high-quality pre-training data from reliable sources. For this, AI developers are investing in data filtering for quality and safety. These filters are normally classifiers that assign a score to each document, and often value high-quality corpora like books, and academic texts, and avoid websites with personal information, adult content, or with toxic content.
Although LLMs are pre-trained in a huge amount of corpora text, the pre-trained data do not include evolving world knowledge or content restricted by copyright rules. Additionally, LLMs also may fail to memorize all factual knowledge encountered during pre-training, especially the less frequent long-tail knowledge. Consequently, when LLMs encounter information that falls outside their limited knowledge boundaries, they are more susceptible to generating hallucinations.
To reduce the knowledge boundary problem, AI developers are constantly using the RAG method, which simply consists of retrieving information from external sources. Normally, relevant knowledge is firstly retrieved from external sources, and then the final response is generated by a generator conditioning on both user query and retrieved documents. By decoupling external knowledge from LLM, RAG can effectively alleviate the hallucination caused by the knowledge gap without affecting the performance of LLM.
For example, in the first versions of ChatGPT, this mitigation strategy was not implemented. However, this is now implemented, and when asked for recent information, ChatGPT searches for external sources before generating an answer. In the example below, ChatGPT answer correctly the factual query even it is a recent event, which is out of its training data.
Screenshot of a ChatGPT-generated summary about Venezuela’s president in early January 2026. Note. Generated by ChatGPT (OpenAI, 2026).
During the fine tuning and preference alignment training phases, LLMs are trained to perform tasks similarly to humans and be aligned with humans preferences. In this process, confident answers are rewarded, even if they are guesses, and answers with uncertainty are penalized, even if they are honest.
This could cause a behavior called sycophancy, that is when a model sometimes produces outputs that diverge from their internal beliefs in order to appease the user (i.e., the human evaluators).
Huang, Lei, Weijiang Yu, Weitao Ma, et al. "A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions." ACM Trans. Inf. Syst. 43, no. 2 (2025): 42:1-42:55. https://doi.org/10.1145/3703155.
Jones, Nicola. "AI Hallucinations Can’t Be Stopped — but These Techniques Can Limit Their Damage." Nature 637, no. 8047 (2025): 778–80. https://doi.org/10.1038/d41586-025-00068-5.
According to researchers, most chatbots based on LLMs have some kind of internal measure of confidence. For example, Google’s Gemini system has a user option called double-check response, which will highlight parts of its answer in green (to show it has been verified by an Internet search) or brown (for disputed or uncertain content).
Jones, Nicola. "AI Hallucinations Can’t Be Stopped — but These Techniques Can Limit Their Damage." Nature 637, no. 8047 (2025): 778–80. https://doi.org/10.1038/d41586-025-00068-5.
Other researchers defend the idea that AI companies should reveal the confidence scores alongside each response, and when the score is low, chatbots should be encouraged to refuse to answer. However, if chatbots refuse to provide uncertainty reasons too often, then users would simply seek answers elsewhere, and it could bring damage to the company.
Jones, Nicola. "AI Hallucinations Can’t Be Stopped — but These Techniques Can Limit Their Damage." Nature 637, no. 8047 (2025): 778–80. https://doi.org/10.1038/d41586-025-00068-5.
Another possible way to mitigate, is through prompt engineering. In the next pages, we are going to explore some prompt strategies that users can use to decrease the probability of getting hallucinated outputs from LLMs.
The presented previously are only some of the origins and mitigations strategies of hallucination in LLMs. These mitigations strategies have been shown very effective in the latest versions of LLMs, as can be seen in the following image:
Jones, Nicola. "AI Hallucinations Can’t Be Stopped — but These Techniques Can Limit Their Damage." Nature 637, no. 8047 (2025): 778–80. https://doi.org/10.1038/d41586-025-00068-5.
Although the improvement of LLMs regarding hallucinations, researchers say that eliminating them completely is impossible since hallucinations stem from the fundamental mathematical and logical structure of LLMs.
Moreover, as pointed out recently by a report from OpenAI, "accuracy of LLMs will never reach 100% because, regardless of model size, search and reasoning capabilities, some real-world questions are inherently unanswerable."
Banerjee, Sourav, Ayushi Agarwal, and Saloni Singla. "LLMs Will Always Hallucinate, and We Need to Live with This." In Intelligent Systems and Applications, edited by Kohei Arai. Springer Nature Switzerland, 2025. https://doi.org/10.1007/978-3-031-99965-9_39.
"Why Language Models Hallucinate." December 18, 2025. https://openai.com/index/why-language-models-hallucinate/.
Therefore, it is still very important to be critical and cautious when using LLMs responses (specially for factual queries), mainly in professional contexts, or in safety-critical situations such as health advice.