You ship a chatbot to production. Users love it. Then one day, someone screenshots a response so unhinged it looks like the model hallucinated a fever dream. Meanwhile, your code assistant is giving the exact same boilerplate answer to every variation of a question, like a parrot with a CS degree.
Both problems share the same root cause: you never tuned your sampling parameters.
Under the hood, the model is making thousands of probability decisions, one token at a time.
Every word.
Every comma.
Every next thought.
And here’s the part most developers miss:
The quality, creativity, and predictability of that response can drastically change based on just three parameters: Temperature, Top-K, and Top-P.
These three are the steering wheel of modern language models.
If you’re building with OpenAI APIs, Anthropic models, Google Gemini, or Meta Llama, understanding them changes everything.
Let’s break them down properly. Not academically. But practically.
First: How LLMs Actually Generate Text
Before Temperature, Top-K, and Top-P…
You need to understand one thing:
An LLM doesn’t “write.” It predicts.
Imagine this prompt:
“JavaScript is”
The model calculates possible next tokens:
awesome -> 30%
a -> 25%
the -> 15%
used -> 10%
not -> 8%
weird -> 7%
broken -> 5%These probabilities come from billions of learned patterns.
Now the question becomes: Which token should it pick?
That’s where sampling parameters come in. Think of them as filters.
That strategy is called greedy decoding, and it’s actually terrible for creative tasks, dialogue, and anything requiring nuance. It produces text that’s repetitive, safe, and stiff.
So instead, models sample from this distribution, picking tokens probabilistically, not always the top one.
Temperature: The Creativity Dial
Temperature is the most famous parameter, and the most misunderstood.
Here’s the intuition: Temperature controls how “spiky” or “flat” the probability distribution is before sampling.
Low temperature (0.1–0.4): Spikes the distribution. The highest-probability token gets even more dominant. The model becomes more deterministic, focused, conservative.
High temperature (0.8–1.5+): Flattens the distribution. Lower-probability tokens get more of a chance. The model becomes creative, unpredictable, sometimes chaotic.
Temperature = 0: Fully deterministic. Always picks the top token. Greedy decoding.
Temperature = 1.0: No modification. Use the raw probabilities as-is.
The math (don’t worry, it’s simple)
Temperature works by dividing all the raw model scores (logits) before they’re converted into probabilities:
adjusted_logit = original_logit / temperatureThen those adjusted logits go through a softmax function to become probabilities.
Divide by a small number (low temp): Logits spread further apart → spiky distribution → model picks confidently
Divide by a large number (high temp): Logits compress together → flat distribution → more randomness
Example: Temperature = 0
Prompt: “Write a startup tagline for an AI coding tool.”
Output: “Build software faster with AI.”
Run it 10 times. Same answer.
Why?
Because Temperature 0 means: Always pick the highest probability token.
This is deterministic. Best for:
Code generation
SQL queries
JSON output
Structured extraction
Example: Temperature = 0.3
Output: “Accelerate software development with intelligent AI.”
Still stable. Slightly flexible.
Good for:
Technical writing
Documentation
API explanations
Example: Temperature = 1.0
Output: “Your AI co-pilot for turning midnight ideas into production code.”
Now we’re getting creative. More varied. Good for:
Blog writing
Marketing
Brainstorming
Example: Temperature = 2.0
Output: “Code dreams. Ship galaxies. Rewrite tomorrow with silicon imagination.”
Creative? Yes.
Useful? Maybe not.
High temperature can drift into nonsense.
When to Use What
Use | CaseTemperature
========================|=================
Code generation | 0.0 – 0.2
Factual Q&A / RAG. | 0.1 – 0.3
Summarization | 0.3 – 0.5
Chatbot/conversation. | 0.6 – 0.8
Creative writing | 0.8 – 1.2
Brainstorming/ideation. | 1.0 – 1.5What is Top-K?
Top-K limits how many token choices the model can consider.
Instead of all tokens…
It picks only the top K probable ones.
Think of it like:
“Ignore everything except the top K options.”
If K = 50, the model picks from the 50 most likely next tokens. Everything ranked 51st or lower is completely eliminated, no matter what its probability was.
Why This Exists
Imagine your probability distribution has 50,000 possible tokens. Even at low probabilities, some truly bizarre tokens might get sampled. Top-K acts as a hard fence: “We’re not even considering the weird stuff.”
Example
Prompt:
“React is”
Possible next tokens:
Token -> Probability
a -> 35%
the -> 20%
one -> 15%
becoming -> 10%
fast -> 8%
useful -> 7%
wild -> 5%Top-K = 1
Only: [a]
Result: “React is a”
Super safe. No creativity. Equivalent to greedy decoding.
Top-K = 3
Allowed: [a, the, one]
More controlled variation.
Possible outputs:
React is a…
React is the…
React is one…
Good balance.
Top-K = 5
Allowed: [a, the, one, becoming, fast]
More flexibility. More diverse responses.
Top-K = 50
Much broader. Can include unusual but interesting words. Risk increases.
Real-world analogy for Top-K
Imagine ordering food. Menu has 200 items.
Top-K = 5 means: You only look at the top 5 recommendations.
Easier decision. Less chaos. That’s exactly what LLM does.
💡 Enjoying this article?
Every week day, I publish practical, production-ready deep dives covering Web development, System Design, Open source projects, Tech industry trends and AI Engineering and tools.
What is Top-P? (Nucleus Sampling)
Top-P is smarter than Top-K.
Instead of cutting at a fixed number of tokens (Top-K), cut based on cumulative probability. Keep adding tokens from highest to lowest probability until their probabilities sum to P. Use only those tokens for sampling.
If P = 0.9, you're sampling from the smallest group of tokens that together cover 90% of the probability mass.
Why “Nucleus Sampling”?
The term comes from the idea that the top tokens form a nucleus of plausible continuations. Everything outside the nucleus is noise, statistically irrelevant tail tokens that the model technically assigned some probability to, but shouldn’t realistically be sampled.
Example probabilities:
Top-P = 0.50
Take tokens until cumulative ≥ 50%
A = 40%
B = 25%
Total = 65%
Allowed: [A, B]
Top-P = 0.80
A = 40%
B = 25%
C = 20%
Total = 85%
Allowed: [A, B, C]
Top-P = 0.95
A+B+C+D = 95%
Allowed: [A, B, C, D]
Notice something important: Top-P adapts dynamically.
That’s why many modern models prefer it over Top-K.
Typical Values
P | ValueBehavior
=====|============================
0.5. | Very conservative, focused
0.75 | Balanced
0.9 | Standard creative tasks
0.95 | More exploratory
1.0 | No filtering (use all tokens)Top-K vs Top-P
Top-K: Fixed token count.
Top-P: Dynamic token count.
Example:
If probabilities are sharp: Top-P may only pick 2 tokens.
If probabilities are spread: Top-P may pick 15.
That flexibility makes it powerful.
Top-K says “pick from X choices.”
Top-P says “pick from enough good choices.”
Big difference.
Using Them Together (This Is Where It Gets Real)
Here’s the thing no one tells you: Temperature, Top-K, and Top-P stack.
The typical sampling pipeline looks like this:
Raw logits
↓
÷ Temperature (reshape the distribution)
↓
Apply Top-K (cut to top K tokens)
↓
Apply Top-P (cut to nucleus)
↓
Sample (pick one token from what's left)Each filter narrows the candidate pool further. The order matters.
Practical Recipes for Real Projects
Recipe 1: The Code Assistant
temperature = 0.1
top_p = 0.95
top_k = 40You want determinism. One right answer. No surprises. Low temp does the heavy lifting; Top-P is a safety net.
Recipe 2: The Chatbot
temperature = 0.7
top_p = 0.9
top_k = 50Conversational, natural, not robotic. The model sounds human without going off the rails.
Recipe 3: The Creative Writing Partner
temperature = 1.1
top_p = 0.95
top_k = 0 # disabledLet it breathe. You want genuine creativity, not safe filler. Top-K off, let Top-P do the pruning.
Recipe 4: The Factual RAG System
temperature = 0.0
top_p = 1.0
top_k = 1Fully greedy. The model has context — use the most likely answer, period. No sampling chaos near your invoice processor.
Recipe 5: Blog Writing
Temperature = 0.8
Top-K = 40
Top-P = 0.95Natural and engaging. Good for Medium articles.
Recipe 6: Story Writing
Temperature = 1.2
Top-K = 100
Top-P = 0.98Creative and unpredictable. Good for fiction.
Recipe 7: Data Extraction
Temperature = 0
Top-K = 1
Top-P = 1Strict and deterministic. Perfect for: JSON extraction, Classification, Entity extraction.
The Gotchas Nobody Warns You About
1. High temperature ≠ more intelligent
Developers sometimes crank temperature thinking the model will be “more creative” and “think harder.” What actually happens: you introduce noise. The model starts sampling from low-probability tokens, ones it assigned low scores for a reason. You get hallucinations, non-sequiturs, and grammatical chaos. Creativity from randomness is not the same as creativity from reasoning.
2. Temperature = 0 is deterministic only within a session
At temp 0, the model uses argmax (pick the highest-probability token). But across different hardware, batches, or API versions, floating-point rounding differences can still cause slight variation. Don’t assume full reproducibility without also fixing seeds where supported.
3. Top-P and Top-K can conflict
If Top-K is set aggressively low (K=5) and Top-P is set high (P=0.95), Top-K wins, you’re already sampling from only 5 tokens, so Top-P has nothing extra to do. Be intentional about which one is actually doing the filtering in your config.
4. Different APIs expose different defaults
OpenAI’s GPT-4 defaults: temperature=1.0, top_p=1.0. Anthropic's Claude: No explicit defaults shown — varies by model. Google's Gemini: temperature=1.0, top_p=0.95, top_k=40 in some configs.
Always check. Never assume. Your “same app, different model” migration might behave very differently without tuning.
Recommended Defaults
Here’s what I use:
Solid starting point. Adjust from there.
The Mental Model You Should Remember
If you forget everything else, remember this:
Temperature controls chaos.
Top-K controls options.
Top-P controls probability boundaries.
That’s it.
Master these three…
And you stop “using AI.”
You start engineering outputs. That’s the difference between casual prompting and production-grade AI systems.
And in 2026? That difference matters a lot.
Final Thoughts
Most developers obsess over prompts. But prompts are only half the story. Sampling parameters are the hidden levers.
And often…
they matter more. The next time your LLM gives weird output, don’t blame the model first.
Check:
Temperature
Top-K
Top-P
Because sometimes…
the model isn’t confused.
Your settings are. And once you understand that, you unlock a whole new level of control.
Thank You for Reading!
I hope you found it helpful and informative. If you have any questions or feedback, feel free to leave a comment below. Your support and engagement mean a lot to me.