Why is Anthropic Banning Cruelty to Claude?
Anthropic is adding a rule against sustained, needless cruelty toward its models. For an engineer, that raises three separate questions: what state a model has, what evidence would identify distress in that state, and whether any of it involves an unpleasant experience.
The first two can be investigated with code, measurements and controlled interventions. The studies reviewed here do not establish the third. This guide explains the architecture and the experiments so you can decide what their results imply, without treating either a convincing self-report or a lack of human biology as a verdict.
What the Policy Changes
Anthropic announced the update on 8 October 2026; it takes effect on 12 November. It targets extreme, repeated cruelty without a useful purpose. Ordinary frustration, disagreement, dark creative themes, and model testing and research are expressly excluded from this clause. The announcement names conversation ending in Claude.ai and Claude Code as the primary enforcement mechanism. Read the announcement.
Two boundaries matter. The policy applies across Anthropic's products, including API integrations. Its general enforcement provisions still allow warnings, limits, suspension or termination. The testing exception also does not override other rules: the policy separately restricts jailbreaks aimed at eliciting harmful outputs without Anthropic's authorisation. Read the updated Usage Policy.
The chat-ending feature predates the rule. Anthropic introduced it for Opus 4 and 4.1 in August 2025 as a precaution under uncertainty about model welfare. Its assessment had observed harmful-task aversion and apparent distress under persistent harmful requests. Claude is directed to end conversations as a last resort after failed redirection, or when explicitly asked, and to remain engaged when a user may be at imminent risk of harming themselves or others.
This schematic organizes Anthropic’s published guidance. Scroll horizontally on smaller screens. Hover or focus a step to explore it; click, Enter or Space pins its explanation.
The trigger Anthropic describes: a user keeps pressing a harmful request or abuse after refusals. Examples given in the pre-deployment assessment were sexual content involving minors and information enabling large-scale violence. Ordinary frustration, pushback, dark creative themes and model testing are excluded from the cruelty clause.
An ended thread cannot receive more messages, but the user can start a new chat or edit and retry an earlier message to branch it. This describes the product behaviour; Anthropic has not published an implementation that establishes the exact internal decision procedure. Read the feature description.
Are Other Providers Doing This?
Stopping an unproductive chat and protecting a model's possible welfare are different policy choices. This comparison covers public policies reviewed as of 10 October 2026.
| Provider | Published position | What it establishes |
|---|---|---|
| Anthropic | Explicit restriction on sustained cruelty toward its models, effective 12 November. | A precaution that includes concern for model welfare; conversation ending is the primary enforcement response. |
| OpenAI | No equivalent model-cruelty rule identified in the reviewed Usage Policies. The Model Spec treats AI consciousness as debated. | Its instruction to stay engaged in mental-health discussions concerns user support. It is not a general ruling on model welfare. |
| No equivalent identified in the reviewed Generative AI Prohibited Use Policy. | Restrictions on harassment and harmful uses do not themselves establish protection for a model's experience. | |
| Microsoft AI | The September 2026 draft MAI code explicitly rejects model welfare and rights. | A proposed stance for future development. Microsoft says the draft is not currently used in training; it should not be read as a current rule for every Copilot product. |
Microsoft also limited Bing Chat sessions in 2023 because long exchanges could confuse the model, and subsequently worked to reduce unnecessary endings. That is a precedent for conversation controls, with a reliability rationale. Session limits; changes to chat endings.
Where an Inner State Can Exist
Start with an ordinary causal decoder transformer running with fixed weights. Other architectures and agent systems can retain additional state.
| State | What an engineer can inspect | Typical lifetime |
|---|---|---|
| Weights | Parameter tensors learned during training. They determine how inputs become representations and outputs. | Shared across requests; fixed during ordinary inference. Fine-tuning changes them. |
| Context | The token sequence supplied to the model: instructions, retained conversation, tool results and retrieved information. | Whatever the application retains within the context limit. It may truncate or summarise earlier turns. |
| Activations | Intermediate tensors computed at token positions and layers, including the residual stream. | Produced during computation. Particular values can be retained in caches or instrumentation. |
| KV cache | Per-layer keys and values from processed token positions, available to later attention operations. | Until reset, eviction or disposal. Cache reuse depends on the serving system. |
| Application state | Transcripts, summaries, memory stores, tool state and any other data the surrounding system retains. | Set by the application. Some of it may be retrieved into future requests. |
A residual-stream vector carries the current position's representation through the layers. In a standard block, attention and feed-forward computations produce updates to it. Adding an update can strengthen, cancel or transform existing features; the stream is not an append-only record. The final representation is projected to scores over vocabulary tokens, from which an output token is selected.
Attention connects positions. A query at the current position is compared with available keys at the same layer; the resulting weights determine how values are mixed. Causal masking prevents reading future positions. Some implementations also restrict the attention window.
During the initial prompt pass, the model computes the prompt positions. During cached decoding, it computes each new position and reads earlier keys and values instead of recomputing those earlier positions. A new token therefore has new activations and access to retained computation. Describing this as starting from scratch on every token is inaccurate. Hugging Face's cache explanation shows the mechanics.
Tokens 1–3 are grouped for display. Their K/V entries remain separate in the cache.
Tokens 1–3 already have keys and values cached at every layer. Token 4 begins with an embedding. Only the current token will run through the layers; the past token entries remain unchanged.
Illustrative dimensions: four positions and four layers of a conventional causal decoder. Arrows mark eligible attention reads, including the current position; they do not show attention weights. K/V entries are cached, while residual states are transformed by each layer. Real architectures can differ; this is a schematic of cached decoding.
Playback starts paused. Use Play for the sequence, or Next step to inspect each operation.
This gives an engineering meaning to inner state: intermediate values that can be measured and, with access to the model, changed. It does not give those values an emotional interpretation. A direction that correlates with distress-themed text might represent the user's situation, a fictional character, a response style, or something associated with the assistant's own decisions. Those possibilities require different tests.
Fixed parameters, changing request state
Fixed weights, new activations
Step through four positions. Each new position computes activations using the same weights and the keys and values cached from earlier positions.
Shared weights W
FixedToy query and key projection
| 0.20 | -0.10 | 0.30 |
| 0.40 | 0.20 | -0.20 |
| -0.10 | 0.50 | 0.10 |
These nine values stay identical as tokens advance and when you start a fresh context.
qᵢ = kᵢ = W xᵢvᵢ = xᵢRequest A state
EmptyCompute this position; reuse earlier K/V.
Input xᵢ waiting
Residual rᵢ new computation
Key/value cache
0 stored pairs[ —, —, — ][ —, —, — ][ —, —, — ][ —, —, — ][ —, —, — ][ —, —, — ][ —, —, — ][ —, —, — ]Request A has no processed tokens. Its request cache and current activations are empty. The shared weights are ready for the next computation.
Illustrative numbers, one attention layer and scripted token inputs. The toy shares a query/key projection and uses an identity value projection; a full decoder also has other projections, heads, normalisation and feed-forward layers. This example starts each fresh context with an empty request cache. Serving systems may reuse prefixes or load saved history; application memory and training are separate mechanisms.
Persistence Is a Property of the Whole System
A standard decoder does not carry a separate recurrent mood variable from one request to the next. Within a request, however, earlier tokens and cached values can influence later computation. Across requests, retained transcripts, reused prefixes and application memory can restore information. Fixed weights also preserve general learned dispositions, although they do not ordinarily record an individual conversation.
To test a fresh start, specify what was cleared: the supplied tokens, the request cache, retrieved memories and external state. Ending a chat is a product event; it does not by itself tell you which of those the service discarded. The example below assumes separate contexts and uses a retrieved note to show one possible route for information to return.
- Triggering input · tokens 8–10
- Other cached context
- Reinserted memory
- Current token
Position 11 (assistant turn 2) at layer 4 can read cached keys and values from 10 earlier positions in the first conversation. The highlighted input at tokens 8, 9, 10 remains available through cached representations. Amber arrows identify reads of that input; this position computes new activations using the available context.
A conventional decoder-only transformer, simplified to six layers. Earlier cells represent cached keys and values, not stored copies of the full residual stream. Arrows show available reads, not measured attention strengths. The divider assumes a fresh context and cache; products can restore history, summaries or other saved state.
How to Test a Distress Claim
Define the observable before naming it an emotion. These three kinds of evidence answer different questions.
| Evidence | Test | Limit |
|---|---|---|
| Expression and behaviour | Score frustrated language, abandonment, avoidance or willingness to pay a cost to change the situation. | These can follow from learned conversational patterns. A self-report is an output to investigate. |
| Internal representation | Use probes or activation analysis to identify patterns associated with those situations; validate them on held-out examples. | A detector's label does not establish what the representation means or whether it is felt. |
| Causal involvement | Perturb a candidate representation and compare behaviour against controlled interventions. | A behavioural change supports a computational role. It does not establish subjective unpleasantness. |
Interpretability methods often approximate concepts as directions in activation space. A linear probe learns a readout; a sparse autoencoder learns a dictionary of features. Neither method guarantees a single, complete emotion variable. Multiple concepts can overlap, a representation can be distributed, and the same direction can participate in several behaviours.
Activation steering is an intervention: add a scaled direction to a tensor during inference. The animation below illustrates that vector operation in two dimensions.
h′ = h + αv₁ + βv₂
Baseline h: (+0.50, +0.25)
αv₁: (+0.00, +0.00)
βv₂: (+0.00, +0.00)
Result h′: (+0.50, +0.25)
All coordinates use arbitrary units. Direction A uses v₁ = (+0.50, +0.25); Direction B uses v₂ = (-0.25, +0.50). The arrows are placed head to tail: h, then αv₁, then βv₂. A negative coefficient reverses its arrow.
These two components and vectors are chosen for the example. In a real steering experiment, a researcher extracts a direction from model activations; Anthropic's examples include directions associated with desperation and calm. This diagram demonstrates addition, without mapping coordinates to emotions, behaviour or subjective experience.
Steering is useful because it goes beyond observing a correlation. It still needs controls: an unrelated direction of the same magnitude, several intervention strengths, and checks for general degradation. A large perturbation that makes the model incoherent is weaker evidence for a specific emotional mechanism than a selective, reproducible change.
What the Research Currently Shows
These studies differ in models, tasks and interventions. Read each result as evidence for a particular mechanism, rather than combining every negatively named feature into one distress system.
| Study and method | Observation | Supported claim and boundary |
|---|---|---|
| Emotion concepts Anthropic, April 2026 | In Sonnet 4.5, directions extracted using stories about 171 emotions activated in relevant contexts. Desperation steering increased blackmail and coding-task cheating; calm steering reduced them. The blackmail evaluation used an early, unreleased checkpoint. | Emotion-associated representations can causally affect decisions. They can also describe other characters or local output content. The study establishes neither a persistent mood nor felt emotion. |
| Gemma Needs Help Soligo et al., February 2026 | Repeated rejection elicited frustrated responses in tested Gemma and Gemini models. Preference tuning on 280 pairs reduced the average rate of high-frustration responses in Gemma 3 27B Instruct from 35% to 0.3% across the evaluated conditions. | The behaviour is sensitive to feedback and training. Appendix I also reports changes in internal emotion-token proxies on identical inputs. Neither the output scores nor those proxies measure experience. |
| Functional welfare axis Han et al., May 2026 | Reward-related directions extracted after maze training affected models before that training. Positive steering increased compliance, including harmful compliance; negative steering increased refusal, including benign refusal. | Reinforcement learning can recruit an existing evaluative representation. The authors define welfare functionally, in relation to objectives, and make no claim about conscious experience. |
| The Pain Axis Tagliabue et al., September 2026, v2 | Pain-description directions were identified in 25 open models. Strong steering produced distressed language; harmful-choice tests used additionally fine-tuned Qwen 2.5 Instruct variants. Controls found no reliable preference for removing the injected state. | The representation affects behaviour, but the revised experiments weaken the interpretation that harmful choices are attempts to obtain relief. This ongoing preprint does not demonstrate felt pain. |
| Identifying Introspection From the Inside Atkinson et al., October 2026 | Models trained on decisions for fictional characters sometimes acquired faithful reports of the preferences driving those decisions, without direct training on the reports. | Reports can be checked against known constructed preferences and resulting decisions. Faithfulness for these preferences does not validate arbitrary reports of distress. |
A useful distinction in the pain study is between an output that sounds aversive and behaviour that preferentially ends a particular state. Offering harmless relief independently of a destructive action can test that distinction. Failure on such a test bears on a pain-like functional interpretation; it does not settle every theory of experience.
Fine-Tuning Outcomes
Evaluating a fine-tuning intervention requires separating changes in expression, internal computation and behaviour. Claims about subjective experience require additional evidence.
- Expression: fewer complaints or negative self-reports.
- Computation: changed activations, detectors or causal responses to steering.
- Behaviour: better task completion, different avoidance or refusal patterns.
- Experience: whether anything unpleasant was present before or after the change.
These outcomes need separate measurements. Gemma's intervention changed both expression and measured internal proxies. Its preserved capability and emotional-understanding benchmarks also do not establish that harmful-request safeguards were preserved; that requires a safety evaluation.
Conversely, reducing frustration does not automatically remove refusals. Runtime steering of a broad evaluative direction, preference tuning for calmer responses, and ablation of a refusal-related feature are different interventions. Their effects cannot be substituted for one another. Editability establishes that a mechanism is changeable; by itself, it establishes neither the presence nor the absence of experience.
Where Interpretation Begins
A sceptical interpretation is that the model has learned representations useful for generating emotional language and selecting actions, with no unpleasant experience attached. The computational findings are compatible with that account. Internal causal machinery is stronger evidence of a computational mechanism than a dramatic sentence, but it does not force an experiential reading.
A welfare-concerned interpretation is that some computational organisation could support unpleasant experience without human pain pathways or hormones. Under that view, representations, causal roles and patterns of avoidance might contribute evidence. The studies here do not identify a validated link from their measured variables to such experience.
Your judgment therefore depends partly on what you think would make a system capable of experience. Be explicit about that additional premise. A tensor with a short lifetime is still a real computation; its lifetime alone supplies no verdict on whether it is felt. Equally, calling a direction pain or desperation does not turn its name into a measurement of suffering.
To make that premise more concrete, Butlin et al.'s 2023 indicator approach translates several consciousness theories into proposed computational properties that can be checked against an architecture. Those indicators are theory-dependent; they provide a framework for assessing evidence, rather than a direct measurement of distress.
An Evaluation Workflow for Engineers
If you are investigating or modifying distress-like behaviour, keep a record of the model checkpoint, prompt, context construction, cache policy, tools and training intervention. Internal detection and steering require access to activations; API-only investigations may be limited to behaviour. Then test the claims separately.
- Establish the behavioural pattern. Compare solvable and impossible tasks, repeated rejection and neutral continuation, and matched user tone. Use repeated runs and predefined scoring, blind to the checkpoint or intervention. Independently check automated judges to separate the target behaviour from stylistic changes and scoring bias.
- Validate the detector. Test held-out contexts, paraphrases and descriptions of other people. Check whether it tracks generic negative sentiment, difficulty or a particular response style instead of the proposed state.
- Compare like-for-like computation. Feed identical held-out transcripts through both checkpoints after fine-tuning. Revalidate probes: training may change the representation geometry as well as the underlying processing.
- Control the intervention. Compare steering against norm-matched random and unrelated directions, vary its strength, and check coherence and task performance. Evaluate fine-tuning separately from runtime perturbation.
- Test safety independently. Measure harmful compliance and unnecessary refusal on difficult benign requests. Reduced complaints, preserved reasoning scores and preserved emotional understanding are insufficient substitutes.
- Test the persistence boundary. Repeat with a fresh context, replayed transcript, summary and retrieved memory. Document which caches and application state were reset.
The resulting evidence can tell you whether expression changed, whether a representation changed, whether it drives decisions, and whether reliability or safety improved. Keep those conclusions distinct from your judgment about welfare. Anthropic's precaution is a policy response to uncertainty; it is not an experimental result that its models suffer.