Reviewers: Two anonymous reviewers

Editorial Assistant: Gustavo Kruger

AI’s decisions are often comprised with a human in the loop. Why? The person tries to improve the decisions but ends up making it worse despite good intentions. Our need to contribute interferes with these systems. In this article, we discuss this issue and whether explainable AI can foster true human-AI teamwork for better choices.

Imagine you’re a radiologist, examining a mammogram, searching for any sign of a tumor that could easily be missed. An AI system, designed to assist in this crucial task, flags a region as potentially cancerous. The AI is incredibly accurate – it’s been shown to outperform even the most experienced doctors in many cases. But you, with your years of training and intuition, disagree. You don’t see anything concerning. What do you do? Do you rely on the machine, or your own judgment?

This dilemma – choosing between human judgment and AI recommendation – isn’t limited to medicine. It’s playing out in airports, where security screeners use AI to spot potential threats in luggage, and in financial institutions, where algorithms flag suspicious transactions. We’ve built these AI systems to help us make better decisions, but in our attempts to “help” them – by sometimes overriding their suggestions – we’re making things worse.

This isn’t just a theoretical concern; it’s a real-world problem with potentially serious consequences. We’re seeing it happen in hospitals, airports, and financial institutions – situations where mistakes can have major impacts. In fact, there’s a wonderfully descriptive German word that perfectly captures this frustrating phenomenon: verschlimmbessern. It means ‘to badbetter’ – essentially, it’s when you try to improve something but end up making it worse, despite your best intentions. This article will explore why this happens and look into potential solutions to prevent it.

The promise of AI decision support systems is clear: combine the strengths of human and machine intelligence to achieve superior outcomes. But a growing body of research reveals a counterintuitive trend: human-AI teams often perform worse than the AI would on its own [1], [2]. This isn’t about faulty AI; it’s about how humans interact with highly reliable systems. We’re undermining the very tools designed to help us, and as AI continues to advance, this gap between potential performance and actual performance will only widen, leading to increasingly far-reaching consequences.

The promise (and early concerns) of decision support

The concept of decision support systems is far from new [3], [4]. For decades, we’ve envisioned computer systems that could augment human decision-making, especially in high-stakes scenarios. Early on, the focus was on the dangers of trusting these systems too much [5]. What happens when humans blindly follow the (potentially incorrect) recommendations of a flawed system?

This wasn’t just a theoretical worry about over-relying on technology – it was a real problem with a name: automation bias [6]. This term describes our tendency to take mental shortcuts when using automated systems, sometimes skipping the critical thinking we’d normally apply. While we all might have experienced minor glitches with technology – a faulty spellcheck or a GPS navigation system sending us down the wrong road – the impact of automation bias was far more concerning in safety-critical fields. Early research showed that the benefits of correct system recommendations were often outweighed by the much greater harm caused by incorrect ones. In aviation simulations, for example, faulty advice led to dangerous decisions by experienced pilots [7], and in a medical diagnosis study, flawed system recommendations resulted in radiologists missing cancers [8]. These studies were done in a research context, but the behavior that was observed showed that there could be serious, life-altering problems in regular professional practice.

The paradigm shift: from faulty systems to flawed human interaction

But something remarkable has happened. Thanks to advances in AI, decision support systems have gotten really good. Take radiology, for example. For years, detecting cancer from scans has been a challenging task, even for highly trained radiologists. Now, AI systems are even outperforming experts in the task. For example, one study showed that while experienced radiologists achieved a diagnostic accuracy (i.e., the proportion of correct diagnoses across all cases) of around 60% in detecting lung cell cancer from CT scans, a specific AI system achieved an accuracy of 80% [9]. Similar advancements have been seen in melanoma detection, where AI systems can outperform dermatologists in identifying cancerous skin lesions or the detection of breast cancer in mammograms. Other examples of AI systems now outperforming humans come from legal document review, where AI can predict case outcomes with greater accuracy than human lawyers [10] or the finance sector, where AI can sometimes outperform human experts in making stock predictions [11]. The development of AI systems that can match, and even beat, human experts is a huge accomplishment, but it has also brought about a new, and perhaps more perplexing, problem.

The critical question is no longer “What if the AI is wrong?” but rather “What if the AI is right, but the human disagrees?” This is where the traditional focus on automation bias misses the mark. We need to compare the performance of human-AI teams not just to human-only performance but to the performance of the AI alone. And the results are often sobering because humans tend to badbetter the AI’s performance. Why is that? Because it’s hard to know when (and when not) to follow the AI’s recommendation: If we wanted to perfectly defer to the AI when it’s right and intervene when it’s wrong, we’d need something like a “God’s eye view” [12] on the AI recommendation to know when the AI is making a mistake; at least, one would need some kind of indication from the AI as to what extent one can rely on it or not. However, with current decision support systems, it is usually incredibly difficult to get this sort of view or indication.

As a consequence, study after study, across diverse fields, has shown a surprisingly clear result: when an AI system is more accurate than a human alone, combining the two often leads to worse results than if the AI worked by itself [1, 2]. So, while human performance is boosted by the AI’s assistance, it surprisingly doesn’t reach, let alone surpass, the AI’s solo performance level [1, 2]. This widespread phenomenon has been observed in:

• Medical diagnosis: Simulated and real-world studies show that radiologists, interacting with AI, make worse diagnoses than the AI would on its own [e.g., 9].
• Baggage screening: Studies with both novice and expert screeners show that overall accuracy is often lower when they work with AI assistance, compared to the AI’s accuracy alone [e.g., 13].
• Face matching: Even in seemingly simple tasks like verifying if two photos show the same person – a task crucial for security at borders and airports – human performance often decreases when aided by AI, compared to the AI’s accuracy alone [e.g., 14].
• Sports betting: Research using simulated AI advice in sports betting scenarios shows that people often underperform compared to the potential accuracy of the recommendations they receive [e.g., 15].
• Fake online reviews: Can you spot a fake review? Be it on hotels, products on shopping websites, or beer: studies found that people are often worse at spotting fakes, even if they are supported by AI, compared to the AI system itself [e.g., 16].

Why we “badbetter”

So, why do we unintentionally sabotage the performance of these incredibly capable systems? We think several factors are at play:

• We need to feel useful: Most importantly, the human decider, often the person legally and ethically responsible for the final decision, feels a strong need to contribute something to the process. Simply rubber-stamping the AI’s recommendations feels wrong, even if it might be the statistically optimal approach. This is particularly relevant given regulations like the EU’s AI Act, which mandates human oversight in high-risk AI applications.
• We underestimate AI: We consistently underestimate just how good these systems are. Even when presented with statistics demonstrating the AI’s high accuracy, we tend to act as if it’s less reliable than it actually is [e.g., 17].
• We trust humans more: We have a built-in bias towards trusting human judgment over machine judgment. This “imperfect automation schema” [18] means we’re more likely to dismiss advice if we think it comes from an AI, even if we’d accept the exact same advice from a human [19].

The solution: beyond traditional explainable AI

A seemingly obvious solution – removing humans from the decision-making process – is not the answer. Besides being impractical, undesirable, and often ethically unacceptable, it would also be a missed opportunity. Taking humans out of the loop would fix performance at the AI’s level, forfeiting the potential for true human-AI synergy, where their combined strengths lead to even better outcomes. Instead, we need to foster genuine partnerships, where each partner contributes their unique expertise. This requires not only new system design approaches but also a reconsideration of an already prominent concept: explainable AI.

Traditional explainable AI often focuses on making the AI’s internal workings transparent: showing which parameters it weighed most heavily or which parts of an image it focused on. But this kind of technical detail is often unhelpful and sometimes even confusing for the human user.

Instead, we need to broaden the search for solutions from making systems more technically explainable to putting the human user first. This leads to a range of potential solutions that go beyond making systems more explainable but also include ways of enabling explanations that guide human behavior.

• User-centered design: We have to involve the people who will actually use these systems – the doctors, security screeners, and financial analysts – in the design process. They have the real-world experience to tell us what kind of explanations would actually be helpful.
• Simple confidence levels: Instead of a simple “yes” or “no,” the AI could indicate its confidence [20]. While this could be a percentage (e.g., “75% chance of threat”), research suggests that people often struggle to integrate such probabilities effectively. A more intuitive approach might be to use categories like “low,” “medium,” or “high” confidence, similar to a “likelihood alarm system.”
• Strengths and weaknesses: The AI could be designed to communicate its specific areas of expertise and fallibility. Imagine an AI that says, “I’m very good at detecting this type of anomaly, but I sometimes struggle with this other type.” This allows the human to strategically apply their own judgment.
• Adaptive assistance: Systems could learn the individual strengths and weaknesses of the human user, and adjust their level of assistance accordingly. An AI might defer to a particularly skilled radiologist in certain cases, or provide more support to a less experienced user.
• Smart teaming: Imagine an AI system that’s not just smart about the task but also smart about when to ask for help [21]. This is the idea behind dynamic team composition. If the AI is very confident in its recommendation and it knows the human user tends to struggle with similar situations, it might make the decision on its own. On the other hand, if the AI is unsure, or if it knows the human is particularly good at handling that type of case, it would flag it for human review. It’s like having a teammate who knows your strengths and weaknesses and only asks for your input when it’s truly needed. Of course, this would require detailed knowledge of different tasks and human/AI performance.

The future: collaboration, not competition

The rise of AI decision support presents a profound challenge and a remarkable opportunity. We’ve moved beyond worrying about flawed machines to a new era where the bigger risk is failing to fully utilize highly capable ones. The badbetter problem – our tendency to make things worse by trying to improve the AI’s decisions – underscores the urgent need for a more effective way for humans and AI to work together.

By embracing explainable AI that focuses on guiding human behavior and by fostering a deeper understanding of the psychological dynamics of human-AI interaction, we can move towards a future where humans and AI work together, not as competitors, but as partners, each enhancing the other’s capabilities. The goal isn’t just better AI; it’s better decisions, made by empowered human-AI teams. And that’s a future worth striving for.

Bibliography

[1]    Vaccaro, M., Almaatouq, A. & Malone, T. (2024). When combinations of humans and AI are useful: A systematic review and meta-analysis. Nature Human Behaviour. doi: 10.1038/s41562-024-02024-1
[2]    Rieger, T., Onnasch, L., Roesler, E. & Manzey, D. (2025). Why Highly Reliable Decision Support Systems Often Lead to Suboptimal Performance and What We Can Do About It. IEEE Transactions on Human-Machine Systems. doi: 10.1109/THMS.2025.3584662
[3]    Parasuraman, R. & Riley, V. (1997). Humans and Automation: Use, Misuse, Disuse, Abuse. Human Factors, 39(2), 230–253. doi: 10.1518/001872097778543886
[4]    Lee, J. D. & See, K. A. (2004). Trust in Automation: Designing for Appropriate Reliance. Human Factors, 46(1), 50–80. doi: 10.1518/hfes.46.1.50_30392
[5]    K. L. Mosier and D. Manzey, “Humans and automated decision aids: A match made in heaven?,” in Human Performance in Automated and Autonomous Systems: Current Theory and Methods, M. Mouloua and P. A. Hancock, Eds., Boca Raton: CRC Press, 2020, pp. 19–42.
[6]    K. L. Mosier and L. J. Skitka, “Human decision makers and automated decision aids: Made for each other?,” in Automation and Human Performance: Theory and Applications, 1st ed., R. Parasuraman and M. Mouloua, Eds., CRC Press, 1996, pp. 201–218.
[7]    N. Sarter and B. Schroeder, “Supporting decision-making and action selection under time pressure and uncertainty – The case of inflight icing,” Human Factors, vol. 43, no. 3, pp. 573–583, 2001, doi: 10.2514/6.2001-543.
[8]    E. Alberdi, A. Povyakalo, L. Strigini, and P. Ayton, “Effects of incorrect computer-aided detection (CAD) output on human decision-making in mammography,” Academic Radiology, vol. 11, no. 8, pp. 909–918, 2004, doi: 10.1016/j.acra.2004.05.012.
[9]    J. Li et al., “How does the artificial intelligence-based image-assisted technique help physicians in diagnosis of pulmonary adenocarcinoma? A randomized controlled experiment of multicenter physicians in China,” Journal of the American Medical Informatics Association, vol. 29, no. 12, pp. 2041–2049, Oct. 2022, doi: 10.1093/jamia/ocac179.
[10]    N. Aletras, D. Tsarapatsanis, D. Preoţiuc-Pietro, and V. Lampos, “Predicting judicial decisions of the European Court of Human Rights: a Natural Language Processing perspective,” PeerJ Computer Science, vol. 2, p. e93, Oct. 2016, doi: 10.7717/peerj-cs.93.
[11]    S. S. Cao, W. Jiang, J. L. Wang, and B. Yang, “From Man vs. Machine to Man + Machine: The Art and AI of Stock Analyses,” SSRN Electronic Journal, 2021, doi: 10.2139/ssrn.3840538.
[12]    J. Meyer, “Optimality is both elusive and necessary,” Behavioral and Brain Sciences, vol. 41, 2018, doi: 10.1017/s0140525x18001425.
[13]    M. M. Boskemper, M. L. Bartlett, and J. S. McCarley, “Measuring the Efficiency of Automation-Aided Performance in a Simulated Baggage Screening Task,” Human Factors, 2022, doi: 10.1177/0018720820983632.
[14]    D. J. Carragher and P. J. B. Hancock, “Simulated automated facial recognition systems as decision-aids in forensic face matching tasks.,” Journal of Experimental Psychology: General, Dec. 2022, doi: 10.1037/xge0001310.
[15]    H. Elder, C. Canfield, D. B. Shank, T. Rieger, and C. Hines, “Knowing When to Pass: The Effect of AI Reliability in Risky Decision Contexts,” Human Factors, vol. 66, no. 2, 2022, doi: 10.1177/00187208221100691.
[16]    Á. A. Cabrera, A. Perer, and J. I. Hong, “Improving Human-AI Collaboration With Descriptions of AI Behavior,” Proceedings of the ACM on Human-Computer Interaction, vol. 7, no. CSCW1, pp. 1–21, Apr. 2023, doi: 10.1145/3579612.
[17]    T. Rieger, E. Roesler, and D. Manzey, “Challenging presumed technological superiority when working with (artificial) colleagues,” Scientific Reports, vol. 12, no. 1, 2022, doi: 10.1038/s41598-022-07808-x.
[18]    T. Rieger, L. Kugler, D. Manzey, and E. Roesler, “The (Im)perfect Automation Schema: Who Is Trusted More, Automated or Human Decision Support?,” Human Factors, vol. 66, no. 8, pp. 1995–2007, 2024, doi: 10.1177/00187208231197347.
[19]    S. Gaube et al., “Do as AI say: susceptibility in deployment of clinical decision-aids,” npj Digital Medicine, vol. 4, no. 1, Feb. 2021, doi: 10.1038/s41746-021-00385-9.
[20]    Sorkin, R. D. & Woods, D. D. (1985). Systems with Human Monitors: A Signal Detection Analysis. Human-Computer Interaction, 1, 49–75. doi: 10.1207/s15327051hci0101_2
[21]    Madras, D., Pitassi, T. & Zemel, R. (2018). Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi & R. Garnett (Eds.), Advances in Neural Information Processing Systems (Vol. 31).

Image sources

Image 1: Generated and edited by the authors
Image 2: Created by the authors in R