By the end of this station, you can:
- Describe the steps between typing a message and getting a response
- Explain what a context window and a knowledge cutoff are
- Predict when an assistant's answer might be outdated or inconsistent
It feels like a conversation. It's actually a prediction loop.
When you chat with an AI assistant, it can feel like texting a very fast friend. Behind the scenes, something more mechanical is happening: your message is converted into tokens, and a language model predicts a response one token at a time.
Understanding that loop helps you use assistants better. It explains why they are great at some things, why they sometimes get facts wrong, and why rephrasing a question can change the answer.
What happens when you press Send
Your message is packaged
The app combines your new message with earlier messages in the chat and with hidden instructions written by the company that runs the assistant.
Text becomes tokens
The combined text is split into tokens — small chunks of words — that the model can process as numbers.
The model predicts, one token at a time
The model estimates which token is likely to come next, picks one, adds it to the text, and repeats until the response is finished.
Tokens become text again
The predicted tokens are turned back into words and shown to you — often streamed word by word as they are generated.
Key terms
- Prompt
- The input you give an AI tool — your question, instructions, and any text or examples you include.
- System instructions
- Hidden directions set by the company or developer that shape how an assistant behaves, such as its tone and what it should refuse to do.
- Context window
- The maximum amount of text (measured in tokens) a model can consider at once, including the conversation so far and its own reply.
- Knowledge cutoff
- The point in time after which a model's training data stops, so it may not know about later events unless it can look them up.
Knowledge check 1
+10 XP on first tryMemory has limits
Within one chat, an assistant can refer back to earlier messages because they are sent along with each new message. But the context window has a size limit. In a very long conversation, older parts may be trimmed or summarized, and the assistant can lose track of details you gave it early on.
Some apps offer features that save notes between chats, and some can use tools like web search or a calculator. Not every assistant has these features, so check what the tool you're using can actually do.
Myth vs. fact
MythIf I ask the same question twice, I'll get the same answer.
FactMany assistants add some randomness when choosing tokens, so wording, details and even conclusions can change between runs. That's one more reason to verify important answers.
MythThe assistant is searching the internet every time I ask something.
FactMany answers come only from patterns learned during training. Some assistants can search the web, but only when that feature exists and is turned on.
Learn moreWhy do responses appear word by word?
Because the model really does generate its answer one token at a time. Many apps show each token as soon as it's produced, a technique called streaming, so you can start reading before the full answer is finished.
It also means the model doesn't plan the whole answer in advance the way you might outline an essay. Each new token depends on everything that came before it — which is why an early mistake can snowball.
Knowledge check 2
+10 XP on first trySources for this lesson
- University of Helsinki and MinnaLearn. Elements of AI. Full citation
- National Institute of Standards and Technology (NIST). Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1, 2023. Full citation
Finish this station
Answer every knowledge check (right or wrong) to unlock completion.
- Knowledge check 1 — not answered yet
- Knowledge check 2 — not answered yet
0/2 checks answered