How LLMs Work: A Beginner’s Playbook on Tokens, Context Window and Temperature

Meera is a developer in Pune. Last week her manager said, “Can we add an AI assistant to our app?” She nodded, opened a chat window, typed a question, and got a perfect answer. Then she asked the same question again and got a slightly different one. Then she pasted a long document and the assistant forgot the first half.

Meera did not need more magic. She needed to understand how LLMs work. Once you know three small ideas, tokens, the context window and temperature, most of the strange behaviour makes sense. This chapter teaches those ideas the way Meera learned them: one simple play at a time.

Last verified: 27 October 2026. AI tools change fast; check the linked sources for anything newer.

Who this is for: complete beginners, with or without coding experience. Time needed: about 30 to 45 minutes. You only need a free chat assistant open in another tab.

How LLMs work, in simple words

LLM stands for large language model. It is a program trained on a huge amount of text, such as books, articles, websites and code. Its training goal is very small: given the text so far, guess what comes next.

Think of the autocomplete on your phone, but trained on a library instead of your messages. When you type “Good”, it suggests “morning”. An LLM does the same thing, one small piece at a time, but it has seen so much text that its guesses can form whole answers, emails and programs.

The precise version: an LLM is a neural network, made of billions of learned numbers called weights, that predicts the next token in a sequence. Microsoft’s learning guide on LLM fundamentals describes it the same way (see Sources).

Two things follow from this. First, an LLM does not look up facts in a database; it produces likely text, which is why it can sound confident and still be wrong. Second, it only produces text. It cannot send an email or open a file by itself. Other code has to do that, which is the idea behind our AI agents playbook.

The playbook at a glance

1. Play 1: See your text as tokens.
2. Play 2: Watch the model guess one token at a time.
3. Play 3: Respect the context window.
4. Play 4: Turn the temperature dial.
5. Play 5: Know what the model cannot do.

Play 1: See your text as tokens

Goal: understand the unit an LLM actually reads.

A model does not read letters or whole words. It reads tokens: small chunks of text. Imagine a box of Lego pieces. Common words are one piece, like “hello”. Rare words are built from several pieces, so “unbelievable” can become “un”, “believ” and “able”. Each piece also has a number, and the model really works on those numbers.

A rough rule for English is that one token is about three quarters of a word. Tokens matter because they are the unit of almost everything: price, speed and memory limits are all counted in tokens, not words.

1. Open any online tokenizer tool, such as the one linked in the Microsoft guide.
2. Paste one short sentence and look at how it is split.
3. Now paste a rare word or a sentence in Hindi and compare the number of tokens.

Meera tried her own name and was surprised that it took three tokens. That is when “pay per token” stopped sounding abstract. If you want to see tokens turn into money, read how one price change hit developers in our Gemini price playbook.

You are done when: you can explain why a long prompt costs more than a short one.

Play 2: Watch the model guess one token at a time

Goal: understand how an answer is built.

An LLM writes its answer like a person finishing a sentence out loud, one word after another, never going back. The steps are simple:

1. Your prompt is turned into tokens and given to the model.
2. The model scores every token it knows by how likely it is to come next.
3. One token is picked from those scores.
4. That token is added to the text, and the whole thing is fed in again.
5. This repeats until the model decides to stop or hits a length limit.

This is why chat tools show text appearing word by word, and why a long answer takes longer than a short one. It is also why the first words you write in a prompt shape everything that follows.

Meera asked her assistant to finish “The capital of Maharashtra is” and then watched it stream “Mumbai”. She realised there was no hidden brain looking up a map. It was a very good guesser.

You are done when: you can say in one sentence that an LLM predicts the next token, over and over.

Play 3: Respect the context window

Goal: understand why the assistant forgets.

The context window is the model’s desk. Everything it can use must fit on that desk at once: your instructions, the chat so far, any document you pasted, and the answer it is writing. When the desk is full, something has to be left off.

Windows come in different sizes. The Microsoft guide says modern models range from about 4 thousand to over 1 million tokens, and the window is always finite. Also, the model has no memory between separate requests. A chat app feels like it remembers because it sends the earlier messages again every time.

1. Start a new chat and give the assistant three facts about yourself.
2. Ask it to repeat them. It can, because they are on the desk.
3. Open another new chat and ask again. It cannot, because that desk is empty.

Meera’s pasted document was simply too big for the desk, so the start fell off. Her fix was to paste only the pages that mattered. To learn how to bring in just the right pages automatically, see our RAG guide.

You are done when: you can explain why a fresh chat does not remember an old one.

Play 4: Turn the temperature dial

Goal: control how safe or creative the answers are.

In Play 2 the model picked one token from a list of likely ones. Temperature decides how adventurous that pick is. Think of a cook following a recipe. At low temperature the cook always follows the recipe exactly. At high temperature the cook tries a pinch of something new.

• Low (around 0 to 0.3): steady and repeatable. Good for facts, code and data tasks.
• High (around 0.7 to 1.0): varied and creative. Good for names, ideas and stories.

These ranges come from the Microsoft guide and differ a little between providers. Even at 0, answers are not guaranteed to be identical every time, so never build something that depends on word-for-word repeats.

Meera asked for ten names for her team’s project. At a low setting the names were safe and similar. At a high setting they were fun but sometimes silly. She now picks the dial for the job.

You are done when: you can choose low or high temperature for a given task and say why.

Play 5: Know what the model cannot do

Goal: trust the model the right amount.

1. It has a cutoff date. It does not know events after its training ended unless you give them to it.
2. It can make things up. This is called a hallucination: confident text that is not true.
3. It can slip on exact maths. Use a calculator tool for precise numbers.
4. It cannot act alone. It writes a request; your code runs it. See function calling.

Meera asked about a news event from yesterday and got a vague answer. She understood why: the model’s knowledge stopped at its cutoff. She now pastes in fresh facts whenever they matter.

You are done when: you check any important answer against a trusted source before using it.

How an LLM learns, in three short stages

You do not need this to use an LLM, but it explains why models behave differently. The Microsoft guide describes three stages.

• Pretraining: the model reads huge amounts of text and learns to guess the next token. Result: a base model that completes text but does not follow instructions well.
• Post-training: people show it good example conversations and rate its answers. Result: a helpful assistant.
• Reasoning training: newer models practise thinking step by step. Result: better at maths and code, but slower and more token-hungry.

Which setting for which job

Mistakes to avoid

1. Treating the model like a search engine. Fix: ask for sources and check them.
2. Pasting everything. More text is not always better; it costs more and can bury the useful part. Fix: paste only what matters.
3. Using high temperature for facts. Fix: lower it for anything that must be correct.
4. Expecting memory across chats. Fix: send the needed background again, or store it in your app.
5. Vague prompts. Fix: say the task, the format and who the answer is for. Our prompt guide shows how.

How to know it is working

• You can predict roughly how many tokens a prompt uses.
• You can say why an answer changed between two tries.
• You stop being surprised when a long chat loses its early details.
• You verify important claims before using them.

Your checklist

• ☐ I tried a tokenizer and saw how text is split.
• ☐ I can explain next-token prediction in one sentence.
• ☐ I know what a context window is and that a new chat starts empty.
• ☐ I chose a temperature on purpose for one task.
• ☐ I can list four things an LLM cannot do alone.

Frequently asked questions

Does an LLM really understand what I write?

It handles language very well, but its core job is predicting likely next tokens. Treat it as a strong helper that can be wrong, not as a person who knows what is true.

How many words is a token?

For English, a token is about three quarters of a word, so 100 tokens is roughly 75 words. Other languages and code often use more tokens for the same meaning.

Why do I get different answers to the same question?

The model samples from likely tokens, so answers can vary. Lower temperature reduces this but does not remove it completely.

Is a bigger context window always better?

No. A bigger window lets you fit more, but it does not make the model focus better, and more tokens cost more. Send what is useful.

Does the model learn from my chat?

Not while you talk to it. The model’s learning happened during training. Whether a provider uses chats for future training depends on its own policy, so read it.

Meera’s next step

A week later Meera went back to her manager. “The assistant is not magic,” she said. “It guesses the next token, it can only see what fits on its desk, and I choose how adventurous it is.” Her manager smiled and said that was the first clear explanation anyone had given the team.

Your next action: spend ten minutes with a tokenizer and one chat assistant, then continue to the AI agents playbook to see what happens when you let a model use tools. The next foundation chapter on this roadmap is embeddings and vector search.

Sources: LLM Fundamentals, Microsoft Learn; Deep Dive into LLMs like ChatGPT, Andrej Karpathy. The characters and scenes in this story are made up to explain the ideas.