Key Takeaways
- A chatbot is a prediction engine: it reads text in pieces called tokens and generates each answer one token at a time, choosing the most likely continuation .
- Hallucinations are built into the design: OpenAI's own research says models are rewarded for guessing rather than saying "I don't know," so they produce fluent, confident, wrong an
- Your chats are not automatically private training data- ChatGPT lets you opt out of training in Data Controls, and Claude has a similar setting - but the defaults differ by product
It Predicts. It Does Not Look Things Up.
When you ask a chatbot a question, nothing searches a database of answers. A large language model is, in OpenAI's own description, a vast set of numbers — weights — tuned during training to capture patterns in how language fits together. It does not store copies of its training text and does not copy-and-paste from it. Ask it something, and it generates a reply the way it was trained to: predicting the most likely next piece of text, one piece at a time, until the answer is complete.
Those pieces are called tokens — chunks of characters that can be part of a word, a whole word, or punctuation. For English, one token is roughly four characters, and 100 tokens work out to about 75 words. Everything — price, speed, memory limits — is counted in tokens.
How It Learned: Guess, Check, Repeat
Training runs in stages. In pre-training, the model reads enormous amounts of text — from the public internet, licensed sources, and data provided by users and trainers — playing one game billions of times: hide the next word, guess it, adjust. (OpenAI's own example starts with "Instead of turning left, she turned ___": early guesses are near-random; after millions of sentences, they aren't.) Then post-training makes the raw predictor behave like an assistant — fine-tuning plus RLHF, reinforcement learning from human feedback, where people rank pairs of answers and the model learns to prefer the higher-ranked style.
Because prediction is probabilistic, there is genuine randomness in the machinery — a dial developers call temperature. Turn it up and outputs get more varied and creative; turn it down and they get more conservative. It's also why the same question can earn different answers on different days.
Why It Sounds Confident When It's Wrong
Here is the part the makers admit themselves. In a 2025 research paper, OpenAI scientists wrote that language models "sometimes guess when uncertain, producing plausible yet incorrect statements instead of admitting uncertainty" — and named the deeper cause: training and testing reward guessing. Most benchmarks give a point for a right answer and zero for "I don't know," so a model that bluffs outperforms an honest one — an "epidemic" of penalized uncertainty, in the paper's words. Some facts can't be learned as patterns at all: if a person's birthday appears exactly once in training data, there is nothing to learn it from — the model will still produce a date, fluent and false. That is a hallucination: not a glitch, but the prediction engine doing its job with no pattern to stand on.
Memory, and Your Privacy
A chatbot's context window is its working memory — everything in the conversation, system instructions included, and on current models it can reach hundreds of thousands of tokens, up to about a million on some. Anthropic's documentation adds a warning worth knowing: as the window fills, recall degrades — a phenomenon it calls context rot. Long chat ≠ perfectly remembered chat.
And your words? The defaults matter. On consumer ChatGPT, your content may be used to train models unless you turn off "Improve the model for everyone" in Settings → Data Controls; temporary chats are not used, and business/API data is not used by default. Claude has an equivalent training setting, with retention of 30 days if it's off and up to five years if it's on — and Anthropic states it does not sell user data. These settings change; check them rather than assuming.
The Twikup Take
A chatbot is the world's best autocomplete wearing a person's voice. Trust it like you'd trust a brilliant, eager intern who has read everything and verified nothing: wonderful for drafts and explanations, dangerous for birthdays, quotes and citations — anything where "sounds right" isn't the same as "is right."
Image: AI-generated illustration.