Temperature, Top-p, and Sampling Settings Explained

When the model predicts the next token, it assigns a probability to every possible token in its vocabulary. Temperature scales these probabilities before sampling.

• Temperature 0: The model always picks the highest-probability token. Every run produces the same output for the same input. Use this for factual tasks where consistency matters.
• Temperature 0.3 to 0.5: The model mostly picks high-probability tokens but occasionally selects a less likely one. Good for tasks that need some variety without going off track.
• Temperature 0.7 to 1.0: The model samples more broadly. Outputs become more creative, diverse, and occasionally surprising. Good for brainstorming, creative writing, or generating multiple options.
• Temperature above 1.0: The probability distribution flattens further. Rare tokens become more likely. Outputs can become incoherent. Rarely useful in production.

Think of temperature as a confidence dial. At 0, the model is locked into its single best guess. As you turn it up, the model becomes willing to take more risks with word choice.

Prompt: "Describe Nairobi in one sentence."

Temperature 0 (five runs, identical):

"Nairobi is the capital and largest city of Kenya, known for its
vibrant culture, growing tech scene, and proximity to Nairobi
National Park."

Temperature 0.5 (three different runs):

Run 1: "Nairobi is a bustling East African capital where tech
startups and wildlife parks coexist within city limits."

Run 2: "Nairobi is the capital of Kenya, a city where the
hum of matatus blends with the energy of Silicon Savannah."

Run 3: "Nairobi is Kenya's capital city, a growing hub for
technology and commerce in East Africa."

Temperature 1.0 (three different runs):

Run 1: "Nairobi is a city of contrasts, where glass towers
cast shadows over open-air markets selling fresh mangoes
at dawn."

Run 2: "Concrete and acacia trees share the skyline of
Nairobi, a capital that refuses to sit still."

Run 3: "A sprawling city perched on the edge of a national
park, Nairobi pulses with ambition and gridlock in
equal measure."

Notice how the outputs become more varied and stylistically distinct as temperature rises. The factual core stays roughly the same, but the language choices diverge.

Top-p takes a different approach to controlling randomness. Instead of scaling probabilities, it limits which tokens are considered at all.

With top-p = 0.9, the model only samples from the smallest set of tokens whose combined probability is at least 90%. All other tokens are excluded, no matter how the probabilities are scaled.

• Top-p = 0.1: Very few tokens are considered. Output is highly predictable.
• Top-p = 0.9: Most of the probability mass is included. Output is varied but usually sensible.
• Top-p = 1.0: All tokens are considered. Equivalent to no top-p filtering.

The advantage of top-p over temperature is that it adapts to the distribution. If the model is very confident about the next token (one token has 95% probability), top-p = 0.9 effectively picks that one token. If the model is uncertain (many tokens share the probability), top-p = 0.9 allows sampling from all of them.

You rarely need to tune both temperature and top-p. Most developers set one and leave the other at its default (1.0). Here are sensible starting points:

• Factual Q&A, data extraction, classification: Temperature 0. You want the same correct answer every time.
• Customer support chatbot: Temperature 0.3. Consistent tone with slight natural variation so it does not sound robotic.
• Code generation: Temperature 0 to 0.2. Correctness matters more than creativity. Wrong code is worse than boring code.
• Creative writing, brainstorming: Temperature 0.7 to 1.0. You want variety and surprise.
• Generating multiple options: Temperature 0.8 to 1.0 with n > 1 (request multiple completions). Higher temperature ensures the options are distinct.

When in doubt, start at temperature 0.3 and adjust based on output quality. If the outputs feel repetitive or robotic, increase. If they feel chaotic or make factual errors, decrease.

Both OpenAI and Anthropic support temperature and top-p parameters:

// OpenAI example
const response = await openai.chat.completions.create({
  model: 'gpt-4o',
  messages: [
    { role: 'user', content: 'Describe Nairobi in one sentence.' },
  ],
  temperature: 0.3, // low randomness
  // top_p: 0.9,    // alternative: use top-p instead
});

// Anthropic example
const message = await anthropic.messages.create({
  model: 'claude-sonnet-4-20250514',
  max_tokens: 256,
  messages: [
    { role: 'user', content: 'Describe Nairobi in one sentence.' },
  ],
  temperature: 0.3,
});

Most API documentation recommends setting either temperature or top-p, not both at the same time. Setting both can produce unpredictable interactions between the two sampling strategies.