Sign up and we'll remember how far you've come.
Now, let's build your very own AI.
A toy word-bigram model: it looks only at the previous word. It does not learn embeddings, attention, or facts. The word splitter is a teaching example, not an LLM tokenizer.
A Plan That Leads to Real Development
Start small like a toy, understand the principles, then grow it bigger over time.
Gather words
Collect sentences about the topic you want to learn. Just 10 sentences is enough to start.
Count next words
Make a table of how many times each word follows another word.
Compute probability scores
Count each transition, including sentence endings. A candidate's probability is its count divided by the total count after the selected word. Neural language models learn weights instead of just storing this small table.
Compare two selection methods
Choose the most likely candidate (greedy), or draw according to probabilities (sampling). Sentence-end stops generation. Sampling can produce the same result again. Generated text is not checked for truth.
Improve it your way
Add better sentences, more data, and different selection rules to grow it into your own AI.
A Tiny Probability Calculator
The table shows observed frequencies at temperature 1. Temperature changes sampling probabilities, not the learned counts or factual accuracy.
15 observed word tokens
| Previous word Next word | can | fly | sing | swim | End of sentence |
|---|---|---|---|---|---|
| Birds | 3 | 0 | 0 | 0 | 0 |
| can | 0 | 1 | 2 | 2 | 0 |
| fly | 0 | 0 | 0 | 0 | 1 |
| sing | 0 | 0 | 0 | 0 | 2 |
| Fish | 2 | 0 | 0 | 0 | 0 |
| swim | 0 | 0 | 0 | 0 | 2 |
Key Concept
Count each transition, including sentence endings. A candidate's probability is its count divided by the total count after the selected word. Neural language models learn weights instead of just storing this small table.
A neural language model learns numerical weights from training examples. At generation time, those weights and the available context determine scores for candidate tokens. The model does not simply look up a stored internet sentence or count only the word immediately before it.
This lab vs a real language model
| This lab (bigram) | Neural language model | |
|---|---|---|
| What it looks at | Only the previous word | The long context it is given |
| What it learns | A table of counts | Numeric weights |
| Embeddings and attention | No | Yes |
| Checks facts by itself | No | No |
A Development Request to Hand to Laria
Send this request to Laria to discuss how to implement this toy bigram in code. It is a small teaching model, not a complete modern language model.
Help me understand this toy bigram model. Training text: Birds can fly. Birds can sing. Birds can sing. Fish can swim. Fish can swim.. Start: . Ask me to predict a next-word probability before showing the calculation. Explain why this differs from a neural language model.