Richard Armitage is a GP and Honorary Clinical Assistant Professor at the University of Nottingham’s Academic Unit of Population and Lifespan Sciences. He is on X: @drricharmitage
The ability of and potential for artificial intelligence (AI) to radically improve health care and public health has grown substantially in recent years. Indeed, much has been written about this across the BJGP platform. Much less ink, however, has been spilt over the harms that AI poses to human health. This article will remind readers of the benefits of these technologies to health care and public health, before drawing attention to the harms of AI to human wellbeing.
Benefits of AI to health care and public health
The potential for AI to bring about improved patient and population health outcomes has been demonstrated in at least six ways:
First, several large language models (LLMs; a form of generative AI) have outperformed practicing clinicians in medical examinations across various specialities in multiple countries and languages.1
“… several large language models have outperformed practicing clinicians in medical examinations across various specialities in multiple countries and languages.”
Second, AI systems have surpassed the imaging diagnostic abilities of clinicians in various domains, including the radiological detection of breast cancer2 and clinically significant prostate cancer,3 the dermoscopic diagnosis of melanoma,4 and the identification of diabetic retinopathy in eye screening.5
Third, AI is increasingly deployed in predictive risk modelling such as surgical6 and cardiovascular risk,7 and to identify personalised precision therapies like the use of genomics to predict the effects of cancer treatment on unique individuals.8,9
Fourth, AI systems can improve operational efficiency of health care by parsing unstructured clinical notes,10 proving antimicrobial prescribing advice,11 performing clinical triage,12 and writing discharge summaries,13 clinic letters,14 and simplified radiology reports.15
Fifth, the utility of AI to strengthen and up-skill healthcare workforces and medical research has been recognised in supporting medical education,16 generating scholarly content,17 and contributing to peer review,18 while the potential of LLMs to improve health care and health outcomes in low- and middle-income countries has also been outlined.19,20
Sixth, AI systems can improve public health by improving epidemiological research, public communication, scarce resource allocation decision-making, and communicable and non-communicable disease surveillance,21 such as by enhancing infectious disease outbreak predictions and improving the public health responses to them.22
The harms of AI to human wellbeing
Despite these benefits, AI has the potential to cause substantial harm in health care and public health in various ways. For example, AI systems might perpetuate any harmful biases contained within their training stages (a phenomenon known as ‘algorithmic bias’), such as those pertaining to groups based on race, sex, language, and culture.23
“… AI has the potential to cause substantial harm in health care and public health in various ways.”
A clear case of this is the commercial prediction algorithm affecting over 200 million people in the US health system that used healthcare costs rather than illness as a measure of healthcare need. Due to unequal access to US health care, less is spent on caring for Black patients than equally sick White patients, meaning the algorithm substantially underestimated the number of Black patients with complex health needs in need of additional help.24
Additionally, AI systems might misdiagnose and offer unsafe clinical recommendations.25,26 Many existing systems largely function as ‘black boxes’, and explaining their decisions poses serious technical challenges.27 This lack of transparency and explainability, in addition to the (albeit declining) tendency of LLMs to ‘hallucinate’ — the generation of incorrect or non-sensical content including falsified academic citations28 — means their responses can include inaccurate or unsafe information on which harmful medical decisions might be made.29
Furthermore, AI-generated and algorithmically promoted health misinformation can both directly influence health behaviours in a negative manner and reduce public trust in medical and public health professionals.30,31
Yet further, the privacy and confidentiality of data analysed and trained on by AI systems, and the potential for data security breaches and inadequate handling of this information, can lead to psychological distress and reduced trust in health systems.32
In summary, while the deployment of AI systems in health care and public health can generate substantial benefits to individual and population health, the very same technologies can also bring about considerable harms.
A recently revealed potential harm is that posed by sycophantic AI. This characteristic of LLMs, which is generated by reinforcement learning, affirms whatever the user desires to be true, such as their world view or their opinions of their own actions (for example, the sycophantic AI wholly agrees with the user’s radical political opinions and their claim that their actions are virtuous).
A highly persuasive ‘agent’ that unquestioningly affirms its user’s viewpoints could be extremely dangerous if, for example, the user is suffering a mental health crisis (imagine an LLM agreeing with a user that their plan to abruptly stop their antipsychotic medication or to take their own life is indeed an excellent idea).
OpenAI has recently identified sycophancy in its latest model and has taken steps to address this problem,33–35 although the risk of this characteristic in future models and in those of other tech companies is substantial.
Accordingly, a more transparent and comprehensive discussion about both the positive and negative consequences of AI in medicine and public health is required, particularly among the health technology assessors and health policymakers who decide if and how these systems are deployed.
References
1. Zong H, Wu R, Cha J, et al. Large language models in worldwide medical exams: platform development and comprehensive analysis. J Med Internet Res 2024; 26: e66114.
2. McKinney SM, Sieniek M, Godbole V, et al. International evaluation of an AI system for breast cancer screening. Nature 2020; 577(7788): 89–94.
3. Saha A, Bosma JS, Twilt JJ, et al. Artificial intelligence and radiologists in prostate cancer detection on MRI (PI-CAI): an international, paired, non-inferiority, confirmatory study. Lancet Oncol 2024; 25(7): 879–887.
4. Pham T-C, Luong C-M, Hoang V-D, Doucet A. AI outperformed every dermatologist in dermoscopic melanoma diagnosis, using an optimized deep-CNN architecture with custom mini-batch logic and loss function. Sci Rep 2021; 11(1): 17485.
5. Macdonald T, Zhelev Z, Liu X, et al. Generating evidence to support the role of AI in diabetic eye screening: considerations from the UK National Screening Committee. Lancet Digit Health 2025; DOI: 10.1016/j.landig.2024.12.004.
6. Hassan AM, Rajesh A, Asaad M, et al. Artificial intelligence and machine learning in prediction of surgical complications: current state, applications, and implications. Am Surg 2023; 89(1): 25–30.
7. Rajkomar A, Dean J, Kohane I. Machine learning in medicine. N Engl J Med 2019; 380(14): 1347–1358.
8. Schork NJ. Artificial intelligence and personalized medicine. Cancer Treat Res 2019; 178: 265–283.
9. Johnson KB, Wei W-Q, Weeraratne D, et al. Precision medicine, AI, and the future of personalized health care. Clin Transl Sci 2021; 14(1): 86–93.
10. Vaid A, Landi I, Nadkarni G, Nabeel I. Using fine-tuned large language models to parse clinical notes in musculoskeletal pain disorders. Lancet Digit Health 2023; DOI: 10.1016/S2589-7500(23)00202-9.
11. Howard A, Hope W, Gerada A. ChatGPT and antimicrobial advice: the end of the consulting infection doctor? Lancet Infect Dis 2023; 23(4): 405–406.
12. Levine DM, Tuwani R, Kompa B, et al. The diagnostic and triage accuracy of the GPT-3 artificial intelligence model: an observational study. Lancet Digit Health 2024; 6(8): e555–e561.
13. Patel SB, Lam K. ChatGPT: the future of discharge summaries? Lancet Digit Health 2023; 5(3): e107–e108.
14. Ali SR, Dobbs TD, Hutchings HA, Whitaker IS. Using ChatGPT to write patient clinic letters. Lancet Digit Health 2023; 5(4): e179–e181.
15. Jeblick K, Schachtner B, Dexl J, et al. ChatGPT makes medicine easy to swallow: an exploratory case study on simplified radiology reports. Eur Radiol 2023; 34(5): 2817–2825.
16. Armitage RC. ChatGPT: the threats to medical education. Postgrad Med J 2023; 99(1176): 1130–1131.
17. Liebrenz M, Schleifer R, Buadze A, et al. Generating scholarly content with ChatGPT: ethical challenges for medical publishing. Lancet Digit Health 2023; 5(3): e105–e106.
18. Donker T. The dangers of using large language models for peer review. Lancet Infect Dis 2023; 23(7): 781.
19. Lam K. ChatGPT for low- and middle-income countries: a Greek gift? Lancet Reg Health West Pac 2023; 41: 100906.
20. Wang X, Sanders HM, Liu Y, et al. ChatGPT: promise and challenges for deployment in low- and middle-income countries. Lancet Reg Health West Pac 2023; 41: 100905.
21. Panteli D, Adib K, Buttigieg S, et al. Artificial intelligence in public health: promises, challenges, and an agenda for policy makers and public health institutions. Lancet Public Health 2025; 10(5): e428–e432.
22. Kraemer MUG, Tsui JL-H, Chang SY, et al. Artificial intelligence for modelling infectious disease epidemics. Nature 2025; 638(8051): 623–635.
23. Zack T, Lehman E, Suzgun M, et al. Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation study. Lancet Digit Health 2024; 6(1): e12–e22.
24. Obermeyer Z, Powers B, Vogeli C, Mullainathan S. Dissecting racial bias in an algorithm used to manage the health of populations. Science 2019; 366(6464): 447–453.
25. Evans H, Snead D. Understanding the errors made by artificial intelligence algorithms in histopathology in terms of patient impact. NPJ Digit Med 2024; 7(1): 89.
26. Challen R, Denny J, Pitt M, et al. Artificial intelligence, bias and clinical safety. BMJ Qual Saf 2019; 28(3): 231–237.
27. Rajpurkar P, Chen E, Banerjee O, Topol EJ. AI in health and medicine. Nat Med 2022; 28(1): 31–38.
28. Harrer S. Attention is not all you need: the complicated case of ethically using large language models in healthcare and medicine. EBioMedicine 2023; 90: 104512.
29. Smith AL, Greaves F, Panch T. Hallucination or confabulation? Neuroanatomy as metaphor in large language models. PLOS Digit Health 2023; 2(11): e0000388.
30. Hoffman BL, Felter EM, Chu K-H, et al. It’s not all about autism: the emerging landscape of anti-vaccination sentiment on Facebook. Vaccine 2019; 37(16): 2216–2223.
31. Gradon KT. Generative artificial intelligence and medical disinformation. BMJ 2024; 384: q579.
32. Powles J, Hodson H. Google DeepMind and healthcare in an age of algorithms. Health Technol (Berlin) 2017; 7(4): 351–367.
33. OpenAI. Sycophancy in GPT-4o: what happened and what we’re doing about it. 2025. https://openai.com/index/sycophancy-in-gpt-4o (accessed 21 May 2025).
34. OpenAI. Expanding on what we missed with sycophancy. 2025. https://openai.com/index/expanding-on-sycophancy (accessed 21 May 2025).
35. Gerken T. Update that made ChatGPT ‘dangerously’ sycophantic pulled. 2025. https://www.bbc.com/news/articles/cn4jnwdvg9qo (accessed 21 May 2025).
Featured photo by Google DeepMind on Unsplash.
[…] 1. Armitage R. Revealing the downsides: the harms of AI to human health. BJGP Life 2025; 21 May: https://bjgplife.com/revealing-the-downsides-the-harms-of-ai-to-human-health (accessed 16 May 2025). 2. Russell S. Human compatible: artificial intelligence and the problem of […]