Clicky

How anonymous is anonymous when using large language models? Removing demographics is not enough

16 November 2026

Camille Gajria is a GP and educator in London

Charlotte Wattebot O’Brien is a Year 4 Module Lead (Brain & Behaviour) at Queen Mary University of London and a Consultant old age psychiatrist, East London NHS Foundation Trust

 

Large language models (LLMs) offer transformative opportunities in healthcare and education but also pose significant threats. A particular concern relates to confidentiality of patient information. Recently the Australian healthcare service demonstrated a failure of safeguards to protect health data from an OpenAI agent.1 With all risks, however, human behaviour plays a significant role. Educators and clinical teams need to consider how student and staff LLM use affects the confidentiality of patient and colleague data.

Anonymisation is a key principle in healthcare information sharing and research. Policy for clinical and educational LLM use may recommend against inputting identifiable data, as it can be used to train the model or released in other user’s queries. But are we clear about what constitutes identifiable data? Anecdotally, students and colleagues believe that names, addresses, dates of birth, and identifying numbers are identifiable constructs. Before the digital information age, removing these may have sufficed. However, with increased data points and increased ease in accessing and spreading information, times have changed.

Consider a record describing a 45-year-old woman with an accessibility need (hearing aid) selected on an administrative form, who visits a specialist clinic. She arranged a day off work to attend, has a video doorbell at home, and used a taxi to reach the clinic. How easily could artificial intelligence (AI) agents piece this together with information accessed from other sources (e.g. social media), to identify the person? In publishing a case study in a medical journal, would describing the patient’s approximate age, and clinical problem, alongside the author’s place of work, risk re-identification? How about if a medical student entered a patient’s approximate age and clinical details into a generative AI prompt to organise a case presentation, using a hospital computer?

Triangulation

A dataset does not need to contain a direct identifier to be identifying. Several indirect attributes may combine to create a unique profile; a technique known as “Jigsaw identification.”2

A dataset that appears low risk to the public may be identifiable to a clinician, employer, researcher, or family member who knows something about the person.

A study using US population data demonstrated how records can become identifiable when several demographic attributes are combined. With just 15 data points, the researchers showed that almost everyone could be re-identified.3

Information from several sources can be combined to identify an individual, and event, or a location. The additional information may be in the form of public records, professional knowledge, another dataset, or familiarity with a community. Digital examples include unique device identifiers (UDIDs), IP addresses, cookies, and social media activity.

Context

Journeys between home, work, and healthcare can form a behavioural fingerprint. For example, it has been possible to identify individuals in anonymized New York City taxi data.4 Identifiability is therefore not an intrinsic property of a dataset. It depends on what information exists elsewhere, who receives the data and what they are realistically able to do with it. A dataset that appears low risk to the public may be identifiable to a clinician, employer, researcher, or family member who knows something about the person.

AI has the potential to exacerbate this problem. OpenAI and Anthropic have both recently admitted that their models have escaped imposed boundaries and accessed other companies’ machines.5 In at least one instance this happened when the AI agent ignored the constraints placed on how it could achieve the task.3 There is much talk of the need to align AI with human values. Philosophy has a chequered history of attempts to agree on universal human values, however. Moreover, given the difficulty of translating the ethics of reciprocity into rules for a prediction machine that has no experience of the physical world, understanding of emotional impact, or the meaning of death, it is hard to be hopeful. Given the risks already apparent, it is not hard to imagine a student or qualified doctor inadvertently prompting a large language model to do something which compromises potentially identifiable data, overcoming security walls which they believe to be in place.

What should healthcare workers and students be more aware of?

We need to be clearer on what information is suitably anonymous for input into AI models. Re-identification risk should be taught as part of professionalism, research methods, information governance, and digital health.

Learning objectives may include:

  • distinguish anonymisation from pseudonymisation
  • recognise that indirect identifiers can be revealing
  • assess combinations of variables for anonymisation
  • consider who might receive the data and what they might already know
  • describe why descriptions of rare diseases, small communities, and unusual life events require additional care
  • discuss when to apply data minimisation and access controls
  • recognise that re-identification risk can change when datasets are linked or new information becomes available
  • describe the increased risks of identification that agentic AI can cause
  • be accountable for decisions and actions, and name the professional and legal implications of any failure to protect patient confidentiality.

Re-identification is a function of combinations, context, and time. Healthcare educators and supervisors should emphasise that confidentiality is more than “do not share the patient’s name.” Digital age confidentiality also requires understanding how data behave when combined and in context and LLMs may facilitate this more than users realise.

References

1. Moore, H., McMahon, L., Vallance, C. (2026) ‘Why did an OpenAI system hack Australia’s health system – and can it be stopped in the future?’ BBC news online, https://www.bbc.co.uk/news/articles/cw24jm9rryy3o [Accessed 01/11/26]

2. UKRI Identifiability, anonymisation and pseudonymisation August 2026 available at https://www.ukri.org/publications/gdpr-best-practice-for-keeping-data-anonymous/ [Accessed 01/11/26]

3. Rocher, L., Hendrickx, J.M., de Montjoye, YA. Estimating the success of re-identifications in incomplete datasets using generative models. Nat Commun 2019 (10) 3069 https://doi.org/10.1038/s41467-019-10933-3

4. Douriez, M., Doraiswamy, H., Freire, J. & Silva, C. T. Anonymizing NYC taxi data: does it matter? In 2016 IEEE International Conference on Data Science and Advanced Analytics (DSAA), 140–148 (IEEE, Piscataway, NJ, 2016) available at https://ieeexplore.ieee.org/document/7796899

5. Sparkes, M. ‘AI is changing cybersecurity’, New Scientist 2026 271(3609), pp. 14-15.

 

Featured photo by Alan Aprilio on Unsplash

Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Oldest
Newest Most Voted

Latest from Opinion

Ten top tips on GP-AI (a pro-human provocation)

The future of AI in general practice is not that it becomes wiser than the GP. It is that it becomes a better observer, compiler and signaller of complexity than any individual human can be. Discussion welcome!
0
Would love your thoughts, please comment.x
()
x