Where Do a Model's Weights Come From? I Trained One From Nothing, in Spanish and English
In this series I've explained tensors, tokenizers and how alphabets become numbers. And when I taught a small model a made-up language in the Kite.NET series, I used LoRA: two thin strips of numbers added on top of a model that already existed. That skips the biggest question: where did the model's numbers come from in the first place?
The answer surprises most people: a model starts as random numbers. Nobody writes grammar or facts into it. Every weight starts as a small random value, and training slowly turns that noise into something that writes.
I didn't want to just say that, so I did it.
The experiment: a model from nothing, in two languages
I trained a tiny language model from scratch on Don Quijote: the Spanish original, and separately its English translation (both public domain, from Project Gutenberg). Same book, same model, two languages.
| Model | a small GPT-style transformer: 4 layers, 192 numbers per position, reads 128 characters at a time |
| Weights | 1,836,764 (Spanish) / 1,838,689 (English) |
| Reads | one character at a time: 92 different characters in the Spanish book, 97 in the English one |
| Training | 5,000 steps × 64 snippets × 128 characters = 40,960,000 characters read |
| Hardware | my Mac Studio (M1 Max) GPU, with Apple's MLX |
| Time | 6.4 minutes (Spanish), 5.9 minutes (English) |
The book is about 2.1 million characters, so the model read it roughly 20 times. I kept the last 10% of the book apart and never trained on it: that's the "unseen text" the model is scored on.
Every few hundred steps I saved what the model writes after the prompt "Sancho dijo" (or "Sancho said"), always with the same random seed so the samples are comparable, plus a snapshot of one of its weight matrices.
Step 0: random on purpose
Before training, every weight matrix is filled with random numbers from a bell curve centred on 0, with a spread (standard deviation) of 0.02. Biases start at 0. That's the recipe GPT-2 used.
Here's what that model writes:
Sancho dijoÁï7B4tfqÑH,gJÁOiMQ]fL eóxp,0Dtlfá.;0¡VFrHBdIJOüGB](fXPÁJIí
Pure noise. Every one of the 92 characters is about equally likely, and that has a number: the model's loss (how surprised it is by the real next character) starts at 4.51, which is exactly what random guessing among 92 characters gives (the natural log of 92 is 4.52).
Training: guess the next character, millions of times
Training is the loop from the tensor article:
- Show the model a snippet of the book.
- At every position, it guesses the next character.
- The loss measures how wrong the guesses were.
- Compute, for every one of the 1.8 million weights, which direction would have made the guess a bit better, and nudge each weight a tiny bit that way.
That's all. There's no grammar lesson, no dictionary and no list of characters in the book. The only task is "guess the next character", and everything else has to emerge from it.
Watch it emerge, in Spanish:
| Step | Loss ↓ | "Sancho dijo…" |
|---|---|---|
| 0 | 4.51 | Sancho dijoÁï7B4tfqÑH,gJÁOiMQ]fL eóxp,0Dtlfá. |
| 25 | 2.91 | Sancho dijoi ostiq a,gon,ice a eóe atotebea etosre |
| 100 | 2.45 | Sancho dijora doln s egon, co hare e de tebe. étos, |
| 800 | 1.69 | Sancho dijo que le me gon y no había de tengo don Quijote, |
| 1,500 | 1.38 | Sancho dijo: — Si señor dicho que el trabajo de su señora, y el del arrado de Rocinante |
| 5,000 | 1.22 | Sancho dijo, alzó un palo, con la historia de los remedios, y el del artificio poco le llevarían alabó, y la notaba el cura y el de la Mancha |
And in English:
| Step | Loss ↓ | "Sancho said…" |
|---|---|---|
| 0 | 4.59 | Sancho saidNe0I&“dS’EC ekàkqt Mlqm3W7‘beDfX’_Dm_Úrñn |
| 100 | 2.67 | Sancho said e I sed thee sist al he t be f sil tar tofg |
| 800 | 2.03 | Sancho saidse I bedite end othat he herealight tareng |
| 1,500 | 1.63 | Sancho said eight offere said Mouscel beauty Don Quixote; and who have will to by to the |
| 5,000 | 1.34 | Sancho said the begged ending all the beauty Dulcinea; aftering the knights-errant of it with a fell of the world |
The order things appear in is the fascinating part:
- First, which characters are common. By step 25 the noise is gone: lots of spaces, vowels, commas. No uppercase Ñ or ¡ in the middle of words any more.
- Then the rhythm of words. By step 100, groups of letters with spaces between them: "co hare e de tebe". Not words yet, but word-shaped.
- Then real words. By step 800: "que", "había", "don Quijote".
- Then names, punctuation and dialogue. By step 1,500 it opens a line of dialogue with "—", as the book does, and Rocinante shows up; by the end the English model names Dulcinea.
It never becomes good: "llevarían alabó" isn't Spanish, and the English is grammatical nonsense. A 1.8-million-weight model that read one book can learn the surface of a language, not meaning. But every bit of that came from random numbers and one rule: guess the next character.
Play with it here: pick a language, drag through the training steps, and watch the text, the loss and the weights change together.
What happens to the weights
The heat map in the explainer is a 32 × 32 corner of one weight matrix (the first layer's attention "query" matrix). At step 0 it's fine-grained random noise, and its numbers have a spread of 0.020, exactly as initialised. During training they spread out: by step 5,000 the spread is 0.073 in the Spanish model, and the largest number grew from 0.089 to 0.53. The weights that matter got pushed far from zero, in specific directions.
That's what "learning" physically is: 1.8 million numbers drifting, one tiny nudge at a time, from random to useful.
Why random, and not zero?
If random noise is just going to be overwritten, why not start every weight at 0? I tried that too, with the same model and the same Spanish book, with every weight set to zero:
| Step | Loss ↓ | "Sancho dijo…" |
|---|---|---|
| 0 | 4.52 | Sancho dijoVï7B¿tfqÑH,gJUOiMQChL eóxp,0Ñ-lb |
| 1,500 | 3.13 | Sancho dijoir ostnq ,gln,icn eóep,t tlblorntosre duq,s,e |
It learns a little, then stalls. The loss goes from 4.52 to 3.13 and stays just above a floor of 3.03. That floor is what a model scores when it knows only how common each character is, and nothing about order.
The reason: when every weight is zero, every neuron computes the same thing and receives the same correction, and most of them receive no correction at all, because a signal multiplied by zero is zero. Nothing breaks the tie. Only the biases (the per-character baseline) can move, so the model learns that "e" and space are common, and nothing more. The model starts from the same place every time and can never tell its neurons apart.
Random numbers break the symmetry: every neuron starts slightly different, so each one can learn something different. The scale matters too. Too big and the signals explode as they pass through the layers; too small and they fade to nothing. That's why recipes like "a bell curve with spread 0.02" exist.
My toy vs a real model
The same recipe (random start, guess the next token, nudge the weights) makes the models you use every day. The difference is scale. Meta's Llama 3 paper describes its largest model:
| My toy | Llama 3, 405B | |
|---|---|---|
| Weights | 1.8 million | 405 billion (~220,000×) |
| Text read | 41 million characters | 15.6 trillion tokens |
| Compute | about 4.5×10¹⁴ operations* | 3.8×10²⁵ operations (~84 billion×) |
| Hardware | one Mac Studio, 6 minutes | up to 16,000 H100 GPUs |
* Rule of thumb: training costs about 6 × weights × text read, so 6 × 1.84 million × 41 million.
Sources: The Llama 3 Herd of Models (Meta, 2024). For scale, GPT-3 in 2020 had 175 billion parameters.
That's why almost nobody trains a model from nothing. It's done once, at enormous cost, and everyone else builds on top.
The ladder: from nothing to your own model
This closes the loop on everything in the series:
- Pretraining (this article): random numbers + trillions of tokens of "guess the next token" = a base model that knows language and a lot of facts. Months, thousands of GPUs.
- Fine-tuning: start from those weights, not from random, and keep training on your own examples. Every weight can move. Hours to days.
- LoRA: start from those weights, freeze them all, and train only two thin strips per matrix, about 1–2% of the numbers. That's what I did in Kite.NET: an afternoon on a Mac.
Each step down the ladder is cheaper because it reuses the weights someone else grew from random numbers.
How much to trust this
- It's a toy. 1.8 million weights, one book, character-level, 6 minutes. It shows the mechanism, not what a real model can do. Real models read tokens, not characters, and are more than a hundred thousand times bigger.
- It's starting to memorize. At the end, the Spanish model's loss on the training text (1.08) is lower than on the unseen 10% (1.22): it has read the same book about 20 times. A real model reads far more text than it could memorize.
- One run per language, one seed. Different random starts give different weights and slightly different numbers; the progression is the same.
- The scale comparison is approximate: characters vs tokens, and a rule-of-thumb compute estimate for my side.
The one-line mental model
A model's weights are born as random noise and grown into knowledge by one rule, repeated billions of times: guess the next piece of text, then nudge every number a little toward a better guess.