Prompt sensitivity

Tweak the wording of a working prompt, maybe to make it read more naturally, and the outputs can shift in ways you did not expect. The task and intent are unchanged, but the model produces something noticeably different.

This is prompt sensitivity, one of the more disorienting properties of language models. The model is not inconsistent for no reason. There is a precise explanation, and it points directly to how to defend against it.

A language model does not interpret meaning. It predicts likely continuations of text. Different phrasings produce different probability distributions over outputs, even when they mean the same thing to a human reader.

Why small changes matter

Changing a word or reordering a sentence changes the input the model uses to predict its next token. The model has no semantic understanding of what you meant; it has learned statistical patterns from text and uses them to predict what follows. A subtly different input activates different patterns and shifts the distribution over outputs.

This is why “Summarise this article” and “Write a summary of this article” can produce outputs of different structure and length, even though they mean the same thing. The model saw both phrasings in training, but the texts that followed each were not identically distributed.

The most common sources of sensitivity

Some changes produce larger output shifts than others. Be deliberate about these.

Verb choice. As covered in Chapter 2, different action verbs produce structurally different outputs. Even synonymous verbs shift tone and detail: “explain” and “describe” are close in meaning but activate different response patterns.

Phrasing order. Position matters, as covered in Chapter 1. Moving a constraint from the beginning of a prompt to the end can change how strongly it is applied, even with identical words.

Framing. How you frame the task shapes the model’s interpretation. “What are the weaknesses of this argument?” and “Critically evaluate this argument” target the same analysis, but the second tends to produce more structured, thorough output, because “critically evaluate” is strongly associated with academic analysis in training data.

Negation. Negative instructions are processed less reliably than positive ones, as covered in Chapter 2. And even among positive instructions, hedged phrasing (“try to keep it concise”) is less effective than direct phrasing (“keep it under 100 words”).

Specificity is the main defense

The relationship is direct: the more precisely a prompt constrains the output, the less room wording variation has to shift it. A prompt that specifies the action verb, audience, scope, format, and length has few degrees of freedom left, so rewording it produces similar outputs.

A vague prompt is sensitive by construction. If it leaves most output characteristics unspecified, small changes in wording are all the model has to go on, and it will use them.

Temperature and sensitivity

A related but distinct source of variation is temperature, a parameter that controls how much randomness is introduced during generation. At higher temperatures, the model samples more freely from its probability distribution, producing more varied outputs. At lower temperatures, it follows the highest-probability path.

Temperature is not a prompt property; it is a generation parameter set at the API level. But it interacts with sensitivity: a sensitive prompt at high temperature produces highly variable outputs, while a specific prompt at low temperature produces consistent ones. If consistency matters, controlling both beats controlling either alone.

What sensitivity tells you about your prompt

Prompt sensitivity is a diagnostic as much as a problem. If small wording changes produce large output shifts, the prompt has underspecified something. The model is filling a gap, and different phrasings fill it differently. The right response is to find the gap and close it with a more specific instruction, an explicit format, or a clearer audience. Hunting for the exact magic wording is not: it is brittle, it will not generalize, and it will fail when the input changes.

In the next lesson, we’ll look at how to test prompts systematically so you are not discovering these gaps in production.