Temperature, Top-P and Other Generation Parameters

Before a model picks the next token, it produces raw scores called logits for every token in its vocabulary, which are converted into probabilities by a softmax function. Temperature is applied before that conversion: each logit is divided by the temperature value, and softmax then converts the result into probabilities. A temperature of 1.0 leaves the distribution exactly as the model produced it. Below 1.0 the distribution sharpens, so the highest-probability tokens become relatively more likely and lower-probability tokens are further suppressed. At the extreme, temperature 0 is effectively greedy decoding: the model always picks the single highest-probability token, producing highly consistent output for the same input. Above 1.0 the distribution flattens, so tokens the model considered unlikely become more competitive, producing more varied and potentially surprising output. Pushed too high, this tips into incoherence as the model starts selecting tokens it would ordinarily dismiss. Think of it as a confidence dial: at low temperature you get a cautious expert who gives one clear, considered answer, while at high temperature you get a freewheeling brainstorm where unusual connections emerge but nonsense occasionally slips in too. A common misconception is that temperature changes what the model knows. It does not. It changes how the model samples from what it knows. A temperature of 0 does not make the model more accurate, it makes it more consistent, and if the most probable token happens to be wrong you will get the same wrong answer every time. As starting points that vary by model and task, extraction and classification suit a temperature around 0 to 0.2, summarisation and question-answering around 0.3 to 0.5, and creative or generative tasks around 0.7 to 1.0.