Temperature vs Top-p in LLMs, Measured

Every chat completions API exposes temperature and top_p, every provider's docs recommend changing one and not the other, and almost nobody says why in a way you can check. The usual explanation is "they both control randomness", which is true and useless for picking a number.

They are not two dials on the same axis. Temperature reshapes the probability distribution; top_p then truncates whatever distribution it is handed. Measured on a fixed set of logits, the same top_p=0.9 lets the model choose from 1 token at temperature 0.2 and 5 tokens at temperature 1.8, a 5x swing from a parameter you did not touch.

What Each One Does

A model's final layer emits a logit per vocabulary entry, an unnormalized score. Softmax turns those into probabilities.

Temperature divides every logit before the softmax. Below 1.0 the gaps between logits widen, so the leading token takes more of the probability mass and output gets more deterministic. Above 1.0 the gaps compress and the tail gets more mass. At exactly 0 the distribution collapses to the single highest logit and sampling becomes deterministic.

Top-p, also called nucleus sampling, takes the resulting distribution, sorts it, and keeps the smallest set of tokens whose probabilities sum to at least p. The rest are discarded and cannot be chosen at all. Top-k does the same with a fixed count instead of a probability mass.

The order is what matters: temperature runs first, top_p runs on its output. So the size of the nucleus depends on the temperature.

Measuring the Interaction

Fixed logits, a real softmax, a real nucleus calculation. Node 18 or newer, no dependencies.

js
Copy
// sampling.mjs - Node 18+, no dependencies. Run: node sampling.mjs
// A realistic next-token distribution: one confident favorite, a few plausible
// alternatives, and a long tail of near-zero junk.
const VOCAB = ["the", "a", "this", "our", "your", "some", "each", "every", "any", "no"]
const LOGITS = [8.2, 6.9, 5.4, 4.8, 4.1, 2.6, 1.9, 1.2, 0.4, -0.8]

const softmax = (logits, temperature) => {
  if (temperature === 0) return logits.map((_, i) => (i === logits.indexOf(Math.max(...logits)) ? 1 : 0))
  const scaled = logits.map((l) => l / temperature)
  const max = Math.max(...scaled)
  const exp = scaled.map((l) => Math.exp(l - max))
  const sum = exp.reduce((a, b) => a + b, 0)
  return exp.map((e) => e / sum)
}

// top_p (nucleus): keep the smallest set of tokens whose probabilities sum to >= p.
const nucleusSize = (probs, p) => {
  const sorted = [...probs].sort((a, b) => b - a)
  let acc = 0
  for (let i = 0; i < sorted.length; i++) {
    acc += sorted[i]
    if (acc >= p) return i + 1
  }
  return sorted.length
}

const entropy = (probs) => -probs.filter((p) => p > 0).reduce((a, p) => a + p * Math.log2(p), 0)

for (const t of [0.0, 0.2, 0.5, 0.7, 1.0, 1.3, 1.8]) {
  const probs = softmax(LOGITS, t)
  const order = probs.map((p, i) => [p, i]).sort((a, b) => b[0] - a[0])
  const n = nucleusSize(probs, 0.9)
  console.log(
    `temp ${t.toFixed(1)}  p(top) ${(order[0][0] * 100).toFixed(1).padStart(5)}%  entropy ${entropy(probs).toFixed(2)}` +
      `  top_p=0.9 keeps ${n}  ->  ${order.slice(0, n).map(([, i]) => VOCAB[i]).join(", ")}`,
  )
}

Run on September 14, 2026 with Node 25.5.0:

text
Copy
vocabulary of 10, fixed logits: 8.2, 6.9, 5.4, 4.8, 4.1, 2.6, 1.9, 1.2, 0.4, -0.8

 temp   p(top)  entropy   top_p=0.9  top_p=0.95   tokens reachable at top_p=0.9
--------------------------------------------------------------------------------------------
  0.0   100.0%     0.00           1           1   the
  0.2    99.8%     0.02           1           1   the
  0.5    92.6%     0.41           1           2   the
  0.7    84.3%     0.76           2           2   the, a
  1.0    71.9%     1.25           2           3   the, a
  1.3    61.4%     1.67           3           4   the, a, this
  1.8    48.4%     2.17           5           6   the, a, this, our, your

Same top_p=0.9 setting, different temperature:
  at temperature 0.2 the model may choose from 1 token
  at temperature 1.8 the model may choose from 5 tokens
  that is a 5x change in candidates from a knob you did not touch

What the Table Shows

The nucleus is a function of temperature. top_p=0.9 is not "the top 90 percent of tokens", it is however many tokens happen to reach 90 percent cumulative mass after temperature has reshaped things. At 0.2 that is one token, so top_p does nothing at all. At 1.8 it is five. Setting both and expecting independent effects is why tuning sessions produce contradictory results.

Low temperature makes top_p inert. Anywhere below about 0.5 on this distribution the leading token already holds more than 90 percent of the mass, so the nucleus is a single token and top_p cannot constrain anything. If you are running at temperature 0.2 and adjusting top_p to control creativity, you are adjusting nothing.

Temperature 0 is not "low randomness", it is a different mode. The distribution collapses to a single token with entropy 0.00. Note that this makes sampling deterministic, not the API: batching, floating-point non-associativity on GPUs and mixture-of-experts routing can still produce different outputs for identical requests.

Entropy is the honest measure. It rises from 0.00 to 2.17 bits across the range, which is what "more creative" actually means: the model has more genuinely available choices at each step.

How to Set Them

1. Change one. Leave the other at its default. This is what every provider's documentation says and the table above is the reason. Tune temperature and leave top_p at 1.0, or pin temperature at 1.0 and tune top_p.
2. Use temperature 0 for extraction, classification and structured output. Anywhere there is a correct answer, sampling from the tail can only hurt.
3. Prefer top_p when you want a floor on quality. Nucleus sampling removes the junk tail outright, so the model cannot pick a token it considers nearly impossible. Temperature alone keeps the tail reachable, just less likely.
4. Do not read a temperature as a scale across providers. The value divides that model's logits, and logit magnitudes differ per model, so 0.7 on one provider is not 0.7 on another. Re-tune when you switch.
5. If a prompt only works at a specific temperature, the prompt is fragile. Sampling parameters are a last adjustment, not a fix for an under-specified prompt.

Conclusion

Temperature and top_p are sequential, not parallel: one reshapes the distribution and the other truncates the result, which is why the same top_p means different things at different temperatures. The practical advice everyone repeats, change one at a time, turns out to be arithmetic rather than folklore. Paste your own logit distribution into the script above and the right values for your use case stop being guesswork.

Related DevToolLab Tools

• JSON Schema Validator - the alternative to tuning sampling for reliability, since a constrained output format removes the failure mode rather than making it rarer.
• Random Number Generator - generate and pin the seeds you use when trying to reproduce a sampled generation.
• LLM Token Cost Calculator - price the extra output that higher temperatures tend to produce before it shows up on an invoice.
• Readability Checker - score generations at different settings to compare output style with a number instead of an impression.

Related Guides

• How LLM Tokenization Actually Works - the layer below sampling, where the candidate tokens come from in the first place.
• LLM Structured Outputs Guide - constraining generation with a schema instead of a temperature.
• LLM Evals Guide - how to tell whether a settings change actually improved anything.
• Best Prompt Management Tools - versioning the prompts and parameters together, so a temperature change is traceable.