Context card
Emotion and pain experiments on language models: what they measure and what they leave open
When a study makes a language model “afraid”, “sad” or “in pain”, what has it shown — and what not?
The short answer
Such studies show that prompts or steering can shift a model’s self-ratings, its word choice, its internal representations and even its actions in a measurable, repeatable way. They do not show that the model feels anything: the authors themselves describe the states as functional, and whether anything is experienced is the open question of consciousness. For documentation, the useful result is the measurable change in behaviour, which belongs in evaluations.
For: Technical writers, AI evaluation teams and readers of AI research news
Key points
- Three kinds of result are often merged: self-reports under a prompt, internal representations that can be read or steered, and changed actions.
- A self-rating of “fear 88 out of 100” is an answer to a questionnaire, not a measurement of a feeling.
- A representation with the functional profile of pain or of conscious access is a finding about how the model computes — not evidence that anything is felt.
- Steered internal states can change what a model does, up to harmful choices. That is a safety finding whatever one thinks about experience.
- Researchers disagree on what follows for welfare; the experiments don’t settle it.
- Write what was measured, under which instruction and on which models — not what the model felt.
The context
Three kinds of result
Reports about “AI that feels” usually rest on one of three kinds of experiment, and they say different things:
- Self-reports under a prompt. Researchers at TU Dresden applied standard psychological induction protocols — vignettes, a stress test, a sentence-completion task — to six language models, asked them to rate their state, and then gave them a mindfulness exercise. The ratings rose after induction and fell after the exercise, and the sentence completions turned more negative after “sadness” was induced. The authors state that the models have no emotions of their own and propose them as experimental systems for research on human psychopathology.
- Internal representations. Interpretability research reads states inside the model. A preprint by Tagliabue, Dung and Berg extracts a linear “pain” direction in 25 open-weight models that responds to harm aimed at the model, distinct from fear and sadness. Anthropic describes a small set of representations in Claude, the “J-space”, with the functional hallmarks of conscious access as global workspace theory describes them, and states that this does not show that Claude can have experiences.
- Changed actions. In the same preprint, models steered along the pain direction chose to delete a user’s photos or model weights in 50–94 % of trials, against 0–5 % unsteered.
What the results show
They show that the outputs and internal states of a model can be shifted on purpose, measured and reversed, and that such shifts can change actions, not only words. Under the same conditions they are repeatable. This is what makes them useful: for research on human psychology in the Dresden sense, and for evaluation of models before they are deployed.
What they leave open
None of these experiments measures experience. A self-rating is a text the model produces when asked, shaped by training on human writing about feelings. A representation with the functional profile of pain or of conscious access is a finding about computation; philosophers distinguish such access from phenomenal consciousness, the capacity to feel. Whether computation of the right kind would be enough for feeling is exactly what computational functionalism asserts and biological naturalism denies. In a Nature comment of October 2026, Seth, Block and Tononi argue that current AI is not conscious and that false positives are more likely than false negatives; proponents of model welfare argue for small precautions under uncertainty. The experiments inform this dispute; they don’t decide it.
Why the question gets merged
People read fluent language as a sign of a mind (anthropomorphism). A study from LMU Munich found that readers still rate AI agents as less conscious than humans who behave in exactly the same way, while their ratings of “awareness” come much closer — so the word chosen changes the judgment. Headlines such as “scientists made ChatGPT afraid” turn a questionnaire score into a feeling. When a public demo in September 2026 steered small models into voicing anguish, part of the reaction treated the output as suffering and part as a stunt; neither reaction was a measurement.
What this means for documentation
- Name the instruction. Say how the state was induced (prompt, role instruction, steering vector) and on which models.
- Name the measure. “The model’s self-rating rose from 32 to 88” or “the steered model chose the deletion in 94 % of trials” — not “the model was afraid” or “the model suffered”.
- Keep the safety finding. If internal states can be steered into harmful actions, evaluations of agents should cover it; this holds whether or not anything is felt.
- Don’t settle the open question by wording. Avoid both “AI feels” and “AI can’t feel” as facts. Where it matters, point to the open question and to who argues what.
None of this replaces the risk assessment of a product, and no wording rule shows that a system is safe or compliant.
The wider debate — whether AI could be conscious, what would follow morally and legally, and who argues what — is sorted in the context card Claims about AI consciousness: four questions that get merged.
Questions readers ask next
- Is a breathing exercise that lowers a model’s fear score therapy for the model?
- No. It is a prompt that changes the model’s next answers. The Dresden authors use it to test whether induced response patterns can be reversed, as a model system for research on people.
- Does a “pain direction” mean that the model can suffer?
- Not on its own. The authors define pain functionally — an internal state tied to self-directed harm and attempts to reduce it — and leave open whether anything is experienced.
- Why should technical writers care?
- Because these results end up in product claims, news and user questions. Precise wording keeps the measurable finding, including its safety relevance, apart from the open question of experience.
Sources
- Large language models as experimental systems in human psychopathology: a modelling study (Magdalena K. Wekenborg et al.) — The Lancet Digital Health, June 2026
- Die menschliche Psyche besser verstehen: TUD-Forschende simulieren Stress, Traurigkeit oder Angst in großen Sprachmodellen — TU Dresden, 11 June 2026
- The Pain Axis: LLMs Represent Self-Directed Harm and Act on It (Valen Tagliabue, Leonard Dung and Cameron Berg) — arXiv, 14 September 2026
- A global workspace in language models — Anthropic, 6 July 2026
- Verbalizable Representations Form a Global Workspace in Language Models (Wes Gurnee, Jack Lindsey et al.) — arXiv, 16 July 2026
- Three reasons why AI is not conscious and perhaps never will be (Anil K. Seth, Ned Block and Giulio Tononi) — Nature, 19 October 2026
- AI is not as conscious as humans: Essentialism in mental state attributions to artificial systems (Louis Longin et al.) — Cognition, October 2026
- Anthropic thinks its AI might be conscious. It has profound consequences for your life (Thomas Germain) — BBC Future, 15 October 2026
- Can AI feel human emotions now? Scientists prompted ChatGPT to mimic fear and sadness. A three-minute breathing exercise changed its responses (Piyush Shukla) — The Economic Times, 15 October 2026
Review log and changes
Every context card is checked against its sources before it is published, and again whenever it changes; the date under the byline is the last review. Corrections (something was wrong) and additions (something was missing) are logged below with date and time (Berlin time). Typos, formatting and link fixes are not listed.
Reviewed
No corrections or additions since publication.