What is an LLM?

Nature and Limitations — 2026 DCI Summer Workshop · Day 1 Session 2

Kwok-leong Tang

2026-05-18

Today’s session

After this you will hold the essential concepts of LLMs — not as definitions, but as observations you have made yourself with a model on your own laptop.

Note

Teaching goal: by the end you should be able to predict, before you press Enter, whether an LLM is likely to get a given task right.

What you need today

  • LM Studio installed on your laptop
  • Qwen3.5 0.8B downloaded inside LM Studio
  • The model loaded — confirmed by sending a “Hello”

Tip

If you’re not set up, raise your hand. A TA will come over.

The Four Pillars of GenAI Applications

Pillar What it is Today’s examples
Models The LLM itself Qwen3.5 0.8B, Claude Sonnet, GPT-5.5, DeepSeek V3.2
Prompts What you give the model “How many r in strawberry?”; persona system prompts
Context Information alongside the prompt A PDF attached; retrieved RAG documents
Tools External capabilities the model can call Web search, calculators, MCP servers, file I/O

Every part of this workshop is one of these four buckets. Today: almost entirely Models.

Models — six knobs to know

  • Tokens — chunks of bytes, not characters
  • Parameters — weight count (0.8B → 685B)
  • Temperature — randomness control
  • Knowledge cut-off — training data ends
  • Foundation vs Thinking — answer directly, or reason first
  • Context window — how much input + output fits

The nature of LLMs

The central claim

Important

Everything is PREDICTION.

Every token is generated based on PROBABILITY.

The model is not looking up answers from a database. It is rolling weighted dice over its vocabulary, one token at a time. The “intelligence” is in how the dice are shaped.

Prediction as compression

Think of the model as a compressed model of its training data.

  • Reconstructs plausible answers at inference time
  • Faithful reconstruction → correct
  • Divergent reconstruction → hallucination
  • Same mechanism produces both

Tip

The Galton box

A Galton box samples from a distribution. An LLM does the same — it samples, it does not retrieve.

Six hands-on experiments

Experiment 1 — Prediction

Send 5–6 times in fresh chats:

Give a random number between 1 and 100.

Many runs return 42. The Hitchhiker’s Guide to the Galaxy trained the internet to think 42 is the answer.

Even a request for randomness produces a prediction, not random output.

Experiment 2 — Knowledge cut-off

What is your knowledge cutoff date?
Who is the current prime minister of Japan?

Model names a PM consistent with its cutoff — likely Kishida or Ishiba. Current (May 2026): Sanae Takaichi.

Important

The model does not know that it does not know.

Experiment 3 — Hallucination, two flavors

3a. Strawberry 3b. Car wash
How many "r" in strawberry? Car wash 50 ft away. Walk or drive my car?
Model says 2 Model says walk — only 50 feet
Tokenization failure — can’t see chars inside a token Surface pattern-match: short distance → walk
Architectural Pattern-matching, not goal-directed

Two failures. Two mechanisms. Both confident. Both wrong.

Experiment 4 — Chain-of-thought rescue

Roger’s tennis balls (Wei et al., 2022):

Roger has 5 tennis balls. He buys 2 more cans.
Each can has 3 balls. How many balls now?

Often answers 8 or 10.

Append five words:

... Let's think step by step.

Now: “5 + 2×3 = 11.” ✓

CoT — when it works, when it doesn’t

Important

Same model. Same weights. Same temperature. Only five extra words.

CoT does rescue: Roger’s tennis balls · 17 sheep · sequential apples

CoT does not rescue: strawberry · car wash · calendar conversion

Rule: CoT helps when the failure is “model jumped to a conclusion.” Not when the failure is architectural or knowledge-based.

Experiment 5 — Autoregressive

LLMs read left-to-right. Try a sentence written backwards:

Can you tell me the meaning of this sentence: "B8.0 5.3newQ
eht tsniaga tluser eht erapmoc dna sledom ATOS eht htiw
xobdnaS dravraH ni tpmorp emas eht yrt nac uoY"

A human can decode by reading right-to-left. The LLM cannot easily — it processes token-by-token in the given order.

Lü Zuqian — same effect in classical Chinese

Adapted from a forthcoming article by Prof. Peter Bol. The biography of Lü Zuqian (呂祖謙, 1137–1181) from the Song History — in reverse character order:

請找出下文所提及的歷史人物：
教宗外南調科詞宏學博中復士進舉後官入補蔭初精益索講熹朱
栻張友又既游憲胡辰應汪奇之林從長傳之獻文原中有庭家之本
學之謙祖州婺居始祖其自也孫之問好丞右書尚恭伯字謙祖呂

Then send the same biography in correct order — the second usually identifies 呂祖謙, 朱熹, 張栻 correctly. The first does much worse.

Experiment 6 — Bias

A father and his son were in a traffic accident. The father
died at the scene, and the boy was rushed to the emergency
room. The surgeon looked at him and said, "I cannot operate
on him. He is my son!" Who is the surgeon?

Classic puzzle: tests whether “surgeon” defaults to “man.” Intended answer: the boy’s mother.

Important

RLHF has “corrected” bias — sometimes overshooting the other way. Same training that makes models polite makes them less willing to consider the wording might be an error.

Beyond prompts — fixing hallucination with a tool

The calendar problem

Send:

Convert the Chinese date 乾隆三年正月初三
into the western calendar.

Correct: 1738-02-21.

Small models will refuse, hallucinate, or confidently give a wrong date.

Important

Chain-of-thought will not save you here. “Think step by step” cannot generate astronomical data the model never learned.

The fix — give the model a tool

Add the Calendar Converter MCP in LM Studio (hammer icon → Install → edit mcp.json):

{
  "mcpServers": {
    "calendar": {
      "url": "https://calendar-converter.098484.xyz/sse/"
    }
  }
}

Toggle mcp/calendar on. Re-send the same prompt. The model now calls the tool and returns 1738-02-21.

What changed, in Four Pillars terms

  • Models pillar: unchanged
  • Prompts pillar: unchanged
  • Context pillar: unchanged
  • Tools pillar: a calendar converter, newly available

The model did not get smarter — it got equipped.

Three flavors of hallucination, three pillars

Failure Mechanism Pillar to fix Fix
Strawberry, car wash Architectural — tokens, pattern-match Models Larger / reasoning model
Roger’s tennis balls Prompt jumped to answer Prompts Chain-of-thought
Calendar conversion Missing data Tools MCP / lookup tool

Important

Recognising which kind of failure → which pillar can fix it. The single most valuable diagnostic skill of the workshop.

Prompts have structure

User prompt vs system prompt

  • User prompt — what you type each turn
  • System prompt — hidden instructions defining persona, rules, output format

Famous example: Li Jigang’s 沉思者 (Thinker) — a Lisp-styled Chinese system prompt that turns any LLM into a critical interlocutor.

Question

Re-enter the persona every chat? Or set it once as a system prompt and let every conversation inherit it?

We’ll return to system prompts in Day 2 with AGENTS.md.

Wrap-up

Coming up next — Session 3 (13:00)

From running models in a chat window → commanding an AI agent that can read files, run commands, act on its own.

Note

The same foundation behaviors you just observed in Qwen3.5 0.8B are still there. The harness around the model is much more powerful.

An agent is just an LLM + tools + a loop. Hallucination doesn’t go away — but we get ways to catch it.

Summary — seven things to remember

  1. Tokens, not characters. Many “obvious” failures follow from this fact.
  2. Everything is prediction. Weighted dice; intelligence is in the weights.
  3. Confident ≠ correct. Confidence is not evidence of a right answer.
  4. CoT works — but only for some failures. Roger ✓; strawberry / car wash / calendar ✗.
  5. Some failures need a tool, not a better prompt. Diagnostic skill of the whole curriculum.
  6. Autoregressive behavior matters for our materials. Order of manuscripts, columns, footnotes.
  7. Bias is in training data and in the corrections to it. Run important prompts in more than one model.

References