Prompt injection attacks: an inherent vulnerability of healthcare AI agents

Richard Armitage is a GP and Honorary Clinical Assistant Professor at the University of Nottingham’s Academic Unit of Population and Lifespan Sciences. He is on X: @drricharmitage

Artificial intelligence (AI) agents are rapidly emerging across various sectors, with healthcare representing one of the most promising frontiers for deployment.¹  These sophisticated systems, built on large language model foundations, possess remarkable capabilities that extend far beyond simple text generation.  They can process complex information, maintain memory through recall and reflection mechanisms, facilitate interaction between different systems, leverage specialised tools, and operate with significant autonomy.²

In healthcare settings, AI agents are transforming clinical operations by automating traditionally labour-intensive workflows.  These systems can efficiently process clinical correspondence, analyse investigation results, and conduct routine medication reviews.  When coupled with speech recognition technology, AI agents are being deployed for both clinical and non-clinical patient-facing tasks including triaging, diagnosis and management suggestions, clinical record writing, discharge summary production, patient registration, and automated appointment scheduling.³  A major feature of AI agents is their ability to act autonomously to achieve specific goals.  For example, an agent with access to a patient’s electronic health record could process incoming clinical correspondence, automatically arrange appropriate investigations, and schedule relevant consultations with a GP or practice nurse.

While these tools promise to improve patient care and reduce administrative burden on GPs and wider practice staff, they come with substantial inherent vulnerabilities.  One such risk recently came to light through Microsoft 365 Copilot, an AI agent built into Microsoft Office workplace applications.  The flaw represents the first documented “zero-click” attack on an AI agent – an attack that requires no user interaction, such as clicking a malicious link – capable of accessing sensitive information from apps and data sources connected to the agent.  The vulnerability, known as “EchoLeak,” would allow hackers to trigger an attack simply by sending an email containing a prompt that deceives the AI agent into leaking the user’s personal information back to the hacker.⁴  Unlike phishing and malware attacks, which rely on users mistakenly clicking malicious links, the AI agent would autonomously execute the prompt embedded in the email.  These security flaws are known as ‘LLM scope violation vulnerabilities’ – where the model is tricked into accessing or exposing data beyond its permitted boundaries – or ‘prompt injection attacks.’  Microsoft has stated that the issue has been fixed in Microsoft 365 Copilot and that no customers were affected.⁵

Given the highly sensitive nature of information stored within healthcare settings, the vulnerability of AI agents to this kind of attack is deeply concerning.  GP practices handle the comprehensive records of thousands of patients, including their medical histories, mental health data, prescription details, and safeguarding information.  A successful attack on an AI agent in general practice could expose entire practice databases of highly sensitive information.  Such breaches would not only violate patient confidentiality and data protection regulations but could also compromise NHS patient safety protocols and undermine public trust in digital healthcare technologies.

As the NHS, and primary care in particular, increasingly invests in AI-powered tools to address workforce pressures, improve patient care, and bolster efficiency, ensuring robust security measures to safeguard against these tools’ inherent vulnerabilities becomes paramount.

References

1. M Moritz, E Topol, P Rajpurkar. Coordinated AI agents for advancing healthcare. Nature Biomedical Engineering 01 April 2025; 9: 432–438. DOI: 10.1038/s41551-025-01363-2
2. J Qiu, K Lam, G Li, et al. LLM-based agentic systems in medicine and healthcare. Nature Machine Intelligence 05 December 2024; 6, 1418–1420. DOI: 10.1038/s42256-024-00944-1
3. SA Gebreab, K Salah, R Jayaraman, et al. LLM-Based Framework for Administrative Task Automation in Healthcare. 2024 12th International Symposium on Digital Forensics and Security (ISDFS), San Antonio, TX, USA, 2024: 1-7, DOI: 10.1109/ISDFS60797.2024.10527275
4. Aim Labs Team. Breaking down ‘EchoLeak’, the First Zero-Click AI Vulnerability Enabling Data Exfiltration from Microsoft 365 Copilot. 11 June 2025. https://www.aim.security/lp/aim-labs-echoleak-blogpost [accessed 17 June 2025]
5. S Goldman. Exclusive: New Microsoft Copilot flaw signals broader risk of AI agents being hacked—‘I would be terrified’. Fortune 11 June 2025. https://fortune.com/2025/06/11/microsoft-copilot-vulnerability-ai-agents-echoleak-hacking/ [accessed 17 June 2025]

Featured Photo by sebastiaan stam on Unsplash