AI models like ChatGPT found to generate convincing falsehoods while aiding fact-checking

The disruptive power of generative artificial intelligence in shaping the global information ecosystem has come under new scrutiny. A comprehensive scoping review reveals that large language models (LLMs) can both fuel and fight misinformation across critical domains such as health, politics, and public discourse.

The research, titled “Generative AI and Misinformation: A Scoping Review of the Role of Generative AI in the Generation, Detection, Mitigation, and Impact of Misinformation,” was published in AI & Society. Collecting evidence from 24 empirical studies published between 2023 and 2024, the review highlights a growing paradox: while LLMs can generate convincing falsehoods at scale, they also demonstrate significant potential as tools for fact-checking, misinformation mitigation, and safeguarding information integrity.

How generative AI fuels the spread of falsehoods

The review warns that generative AI has become a powerful driver of misinformation by exploiting human cognitive biases and ideological leanings. Ten of the studies analyzed revealed that LLMs such as ChatGPT-4.0, Bard, and Bing Chat are capable of producing content that can be both highly persuasive and misleading.

Research led by Gabriel and colleagues (2024) found that ChatGPT-4.0 could generate personalized false headlines tailored to user demographics, making them harder for audiences to recognize as false. Kim and colleagues (2024) demonstrated that AI models can manipulate numbers, textual data, and even visual evidence to fabricate convincing narratives, including altered images such as inserting liquid water into Mars rover photos.

Other studies cited in the review reveal how generative AI systems have amplified political and health-related falsehoods. Makhortykh et al. (2024) showed that more than a quarter of responses by mainstream chatbots reproduced unverified Russian disinformation about the war in Ukraine. In health, Menz et al. (2024) reported that LLMs such as ChatGPT and Bard generated large-scale cancer misinformation blogs with fabricated citations and testimonials.

Alarmingly, safeguards against misuse appear fragile. While Claude 2, a model noted for its strong refusal mechanisms, initially resisted generating false information, later evaluations showed weakened defenses. This trend suggests that mitigation mechanisms built into LLMs may degrade over time, raising long-term concerns for information safety.

AI as a misinformation detector and corrective tool

The review also underscores the promise of LLMs in detecting and correcting misinformation when properly deployed. Six studies assessed detection capabilities, finding that the performance of LLMs is strongly influenced by the phrasing of prompts, the domain of information, and the cultural and linguistic context.

Deiana et al. (2023) reported that GPT-4.0 outperformed GPT-3.5 in debunking widely circulated vaccine myths, producing more accurate and comprehensive responses. Garbarino and Bragazzi (2024) compared ChatGPT-4.0 and Bard in assessing sleep-related misinformation, finding both moderately reliable, with Bard showing slight advantages in factual assessment.

Yet, the limitations were stark. Kumar et al. (2024) demonstrated that zero-shot prompting, in which models classify misinformation without examples, often worked better than few-shot prompting, contradicting conventional expectations. The study also found that adding sentiment cues to training data reduced detection accuracy.

Moreover, Węcel et al. (2023) revealed significant variability in ChatGPT’s ability to identify falsehoods, with accuracy often only marginally better than chance and heavily dependent on how prompts were worded. This inconsistency signals that without standardized evaluation protocols and improved safeguards, LLMs’ detection potential remains uneven.

Impact on audience perceptions and path forward

Perhaps the most concerning insight from the review is the tangible effect of AI-generated misinformation on human decision-making and trust. Seven studies explored this dimension. In a series of experiments, Kim et al. (2023) found that incorrect travel advice provided by ChatGPT reduced people’s willingness to visit destinations, even when explicit corrections were presented later. The impact was most pronounced when users encountered false information early in their decision-making process.

Another study by Spitale et al. (2023) demonstrated that AI-generated disinformation on health and environmental issues was more compelling and harder for users to detect than human-written falsehoods. Participants often trusted AI-generated accurate information more than equivalent human-written content, reflecting the influence of the so-called “machine heuristic,” where users perceive algorithmic outputs as objective or authoritative.

This dual dynamic, the erosion of trust through exposure to falsehoods and the reliance on AI for credible information, underlines what the authors term “epistemic ambivalence.” The review argues that this paradox must be urgently addressed through coordinated regulatory, technological, and educational responses.

The authors recommend standardized evaluation protocols, multilingual and culturally inclusive benchmarks, continuous monitoring of safety mechanisms, user-facing safeguards such as warning labels, and digital watermarking for AI-generated content. They stress that only through robust regulation and cross-sector collaboration can the benefits of generative AI be leveraged without undermining democratic discourse and public trust.