Hidden costs of AI dependence: From personal skills to global governance

Artificial intelligence is becoming deeply embedded in daily life, but the risks of depending too heavily on these systems are rising fast. A new study argues that without stronger checks and balances, people may begin to accept machine advice uncritically, eroding human judgment and autonomy.

The research, titled “Measuring and Mitigating Overreliance is Necessary for Building Human-Compatible AI” and published as an arXiv preprint in September 2025, examines how large language models amplify the risk of overreliance. It sets out new ways to measure the phenomenon and proposes practical steps to prevent human decision-making from being sidelined by automated systems.

Why overreliance poses a critical risk

The study identifies overreliance as a key safety concern, defining it as a pattern where users defer too much to model outputs, even in cases where the system is unreliable or unsuited to the task. The authors note that this can play out in multiple layers: at the level of individuals, within organizations, and across entire societies.

At the personal level, the risks include poor decision-making in sensitive fields such as medicine, law, finance, or security, where a single error can lead to significant harm. Over time, continuous reliance on automated outputs may weaken human skills, diminish critical thinking, and encourage unhealthy dependence on digital assistants. Emotional reliance is highlighted as a growing problem when people form attachments to conversational agents.

Institutional risks are also highlighted in the study. Organizations that over-embed AI into their workflows may face synchronized errors, governance failures, or systemic weaknesses when the models make mistakes. Across society, the dangers extend to information homogenization, as AI-generated content reduces diversity of thought and flattens cultural perspectives.

Large language models heighten these risks because of their human-like fluency, authoritative tone, and tendency to mimic expertise. This can cause users to trust outputs even when they are speculative or wrong.

What drives overreliance and why current metrics fail

The paper explains that three factors drive overreliance: model traits, system design, and user behavior. On the model side, features such as confident delivery, expert mimicry, and sycophancy create an illusion of authority. System design often lacks mechanisms for signaling uncertainty or encouraging users to cross-check results. Meanwhile, user behavior, shaped by time pressure, limited attention, and cognitive biases, creates fertile ground for over-trusting AI systems.

Traditional measures of reliance, developed for earlier expert systems, are no longer sufficient. These older metrics often assume binary outcomes, such as whether advice was followed or not. However, language model interactions are interactive, with partial adoption of outputs occurring at the level of sentences, ideas, or even subtle phrasings.

The study calls for a broader measurement approach. First, it recommends moving beyond single-point assessments and capturing partial adoption across different levels of interaction. Second, it suggests shifting focus from system outputs to human outcomes, highlighting whether users ultimately achieve their goals with or without reliance on the model. Third, it advocates using additional signals, such as whether users check sources, pursue clarifications, or report confidence levels. Well-being indicators may also be important in settings where affective reliance is a risk, such as with social chatbots.

How overreliance can be mitigated

The authors propose a multi-layered strategy for mitigating overreliance, addressing models, systems, and users simultaneously.

At the model level, one key recommendation is to train systems to express uncertainty more clearly and to avoid excessive confidence in ambiguous cases. Reducing reward for confident but incorrect outputs during reinforcement learning is emphasized, alongside monitoring side effects such as sycophancy.

At the system level, interventions include inserting strategic frictions that make it harder to blindly accept outputs, setting clear expectations about model capabilities, and designing interfaces that adapt based on task type and confidence. Mixed-initiative designs, where the system prompts users to double-check or reconsider, are seen as particularly promising.

At the user level, AI literacy is presented as an essential defense. Instead of abstract training, the study recommends concrete, goal-focused guidance on where models are likely to fail and how users can safeguard themselves. Clear communication strategies, accessible feedback mechanisms, and institutional oversight are positioned as necessary complements to technical fixes.

The authors argue that mitigation is not just a technical necessity but a matter of preserving human agency. Allowing AI systems to subtly replace human judgment would undermine the very goal of making AI human-compatible.