GenAI’s double-edged impact on education: Misconceptions, bias and opportunities

A new study examines how generative artificial intelligence (GenAI) intersects with misinformation in education, revealing both peril and promise for learners, teachers, and policymakers. Published in the journal AI & Society,  the systematic scoping review presents a multidimensional understanding of GenAI’s impact at the individual, community, and systemic levels within educational environments.

The study, titled “GenAI and Misinformation in Education: A Systematic Scoping Review of Opportunities and Challenges”, analyzes empirical evidence from 20 peer-reviewed articles, consolidating insights into GenAI's capacity to identify, prevent, and inadvertently reproduce misinformation in academic settings. The research aims to inform best practices for AI literacy, responsible tool adoption, and pedagogical innovation.

How is misinformation manifesting across educational levels?

The study divides misinformation into three interconnected tiers: the individual level, where students internalize misconceptions or naive beliefs; the community level, involving shared misbeliefs among educators or peer groups; and the system level, where flawed policies or curricula perpetuate inaccuracies.

At the individual level, which accounted for 80% of the reviewed research, GenAI exhibited a paradoxical performance. On one hand, it struggled significantly in areas like mathematics, where identifying and correcting student misconceptions often requires a conceptual, not just statistical, understanding. Tools such as ChatGPT-3.5 and Llama-2 models failed to reliably detect math misconceptions or generate corrective feedback. Even advanced prompt engineering and model fine-tuning did not yield consistent improvements. However, in subjects like English and science, the same tools demonstrated more accurate correction capabilities, indicating domain-dependent performance variability.

On the other hand, GenAI showed more consistent efficacy in prebunking, a strategy involving early exposure to weakened misinformation and counterarguments to build cognitive resistance. In multiple studies, custom GenAI applications like SNIFFER and role-playing educational games powered by GPT-3 effectively engaged students in identifying misinformation tactics. Despite some shortcomings, such as persistent errors in simulating mental health conditions, these tools managed to inspire policy interest, critical thinking, and even career aspirations in fields like media and technology.

However, the most alarming trend at the individual level was GenAI’s tendency to reproduce misinformation. Students often accepted AI-generated outputs uncritically, leading to essays, projects, and classroom discussions riddled with fabricated facts and sources. Misconceptions about GenAI itself also emerged, with younger students believing the tools possessed human-like reasoning capabilities or infallibility.

Where are the gaps at the community and system levels?

The review exposes a critical research gap at the community level, where only two studies were identified. One study highlighted how pre-service teachers misused GenAI in geometry instruction due to their own misconceptions about AI capabilities. This lack of foundational understanding limited their effectiveness in using AI tools for classroom planning or instruction. Another study documented media engineering students who built their own AI chatbots connected to verified databases, only to discover the bots still hallucinated information despite accurate source access. These findings underscore the challenge of community-level AI integration without comprehensive digital literacy training.

At the system level, the review revealed a similar scarcity of research. One included study evaluated bias in GenAI-generated images of chemists, revealing stark gender imbalances across major tools. Models like DreamStudio produced male-only representations of chemists, while others, including DALL·E2 and Adobe Firefly, displayed skewed occupational roles by gender. Such systemic bias in AI outputs can reinforce stereotypes and misinform students about real-world diversity in STEM fields.

Another promising case at the system level came from an AI literacy initiative where undergraduate students evaluated GenAI chatbot outputs for reliability, bias, and misinformation. Their findings were synthesized into a comprehensive framework aimed at improving AI integration into curriculum design and educational policy. However, this framework has yet to be institutionalized, highlighting the nascent stage of systemic AI oversight in education.

What path forward balances risk with innovation?

The authors stress the importance of a multi-level and risk-benefit-informed approach to GenAI implementation in educational contexts. While GenAI offers scalable and personalized tools for combating misinformation, particularly in digital literacy and prebunking activities, it also introduces risks of overreliance, unchecked errors, and reinforcement of systemic biases. The paper urges stakeholders to avoid unsupervised GenAI use for factual learning or curriculum generation in subjects where conceptual understanding is paramount, such as mathematics and science.

Future research directions proposed by the authors include conducting comparative benchmarking of GenAI models on standardized misinformation detection tasks, evaluating prompt engineering sensitivity, and developing longitudinal studies to assess how successive model updates affect educational performance. Researchers are also encouraged to expand inquiry into the interactions between educational levels, exploring how GenAI use at one level (e.g., systemic policy changes) ripples through teacher communities and student experiences.

The review recommends prioritizing low-risk, high-benefit applications, such as GenAI-enhanced media literacy instruction with clearly communicated model limitations. For moderate-risk scenarios, like student-led GenAI exploration in structured environments, careful scaffolding and peer collaboration are advised. High-risk, low-benefit applications, including blind reliance on GenAI-generated learning content, should be strictly avoided.