What Is a Perceptron? The Smallest Piece of Every AI Model
This series took a model apart: tensors, tokenizers, where weights come from, and finally a model picking its next word through 24 layers. Now let's zoom all the way in, to the smallest piece of all that machinery: the perceptron.
The unit
Frank Rosenblatt described it in 1958, in a paper with a wonderful title: "The perceptron: A probabilistic model for information storage and organization in the brain" (Psychological Review, vol. 65). The idea fits on one line:
output = activation( w·x + b )
- Take the inputs
x(some numbers). - Multiply each one by its weight
wand add them up. - Add a bias
b. - Pass the result through an activation. Rosenblatt's was a step: if the sum is above zero, say 1; otherwise, say 0.
That's a perceptron: a tiny machine that looks at some numbers and says yes or no. Every "neuron" in a modern network is this same thing, with a smoother activation.
Here's one that does AND (it only says 1 when both inputs are 1), with weights my experiment learned:
output = step(0.2·x₁ + 0.1·x₂ − 0.2)
x₁=1, x₂=1: 0.2 + 0.1 − 0.2 = 0.1 > 0 → 1
x₁=1, x₂=0: 0.2 − 0.2 = 0.0 ≤ 0 → 0
x₁=0, x₂=1: 0.1 − 0.2 = −0.1 ≤ 0 → 0
x₁=0, x₂=0: − 0.2 = −0.2 ≤ 0 → 0
Geometrically, w·x + b = 0 is a straight line. Everything on one side gets a 1, everything on the other side a 0. A perceptron is a line drawn across its inputs.
How it learns
Rosenblatt also gave it a learning rule, and it's beautifully simple: show it an example; if it's right, do nothing; if it's wrong, nudge every weight a little toward the right answer.
pred = 1 if xi @ w + b > 0 else 0 # step activation
if pred != yi: # Rosenblatt's rule: nudge only when wrong
w += lr * (yi - pred) * xi; b += lr * (yi - pred); mistakes += 1
Repeat over the examples until it stops making mistakes. It's the ancestor of the "nudge every weight a tiny bit" loop from where weights come from, just without the calculus.
Try it. Pick AND, OR or XOR, drag the weights yourself, or press Train and watch the rule move the line:
The experiment: AND, OR, and the XOR wall
I ran Rosenblatt's rule in plain NumPy on my Mac: start with every weight at zero, learning rate 0.1, up to 100 passes over the four examples.
| Problem | When is the answer 1? | One perceptron |
|---|---|---|
| AND | both inputs are 1 | learned perfectly in 6 epochs |
| OR | at least one input is 1 | learned perfectly in 4 epochs |
| XOR | exactly one input is 1 | never. After 100 epochs it's right on 2 of 4, no better than guessing |
(The playground runs the same algorithm in your browser and gets the same numbers.)
XOR isn't hard. A child can do it: "one or the other, but not both". But look at the four points: the two 1s sit on opposite corners, and so do the two 0s. No single straight line can put both 1s on one side and both 0s on the other. And a perceptron is a straight line. It's not a matter of training longer; the answer is outside what one perceptron can represent.
That limit has a famous place in history. In 1969 Marvin Minsky and Seymour Papert published Perceptrons, a book that analysed rigorously what single-layer perceptrons can and can't compute, XOR included. Enthusiasm and funding for neural networks dropped for years afterwards; the book is often cited as one of the causes of the first "AI winter".
The fix: stack them
What a single line can't do, two lines can. Put two perceptrons in a hidden layer, each drawing its own line, and a third perceptron on top that combines their answers: "1 if you're between the two lines". That's XOR.
The catch was training it. Rosenblatt's rule only tells the output how to change; it says nothing about the hidden perceptrons in the middle. The answer is backpropagation: send the error backwards through the network and compute how much each weight, in every layer, contributed to it. It was made famous in 1986 by David Rumelhart, Geoffrey Hinton and Ronald Williams, in "Learning representations by back-propagating errors" (Nature, vol. 323).
Backpropagation needs one more change: the step activation has to go. A step jumps from 0 to 1, so its slope is zero almost everywhere, and the error has no slope to flow back along. Replace it with a smooth S-curve (the sigmoid) and the gradients flow.
My experiment: 2 inputs → 2 hidden perceptrons → 1 output, sigmoid activations, backpropagation, 10 different random starts:
| XOR solved? | |
|---|---|
| 8 of 10 random starts | yes: outputs 0.008, 0.992, 0.991, 0.007 for the four cases |
| 2 of 10 random starts | no: stuck in a solution that's only partly right |
Two of ten got stuck. That's honest, and it's real: with so few neurons, some random starts lead to a dead end. Bigger networks have many more paths to a good solution, which is one reason they train more reliably. Part 2 of the playground lets you try your own random starts and see both outcomes.
From one perceptron to a layer to a model
Here's where it connects to everything else in the series. One perceptron computes w·x + b. Put many perceptrons side by side, all reading the same inputs, and you have a layer. Write their weights as the columns of a matrix and the whole layer is:
Y = activation( X @ W + b )
One matrix multiplication. That's why the tensor article said multiplication is the verb: a layer of perceptrons is exactly one. A perceptron is the scalar case of a tensor operation.
And the model from my last article, Qwen3.5-2B? Each of its 24 layers has a feed-forward block with 6,144 hidden units, each reading the 2,048 numbers that represent a token. That's about 147,000 perceptron-like units, each with 2,048 weights, plus the attention machinery around them. Modern units are refined versions: smooth activations (Qwen uses SiLU instead of a step), and a "gated" design in which two of these weighted sums are multiplied together. But the core of each one is still Rosenblatt's line: multiply, add, decide.
How much to trust this
- The experiment is tiny on purpose: four data points, one learning rate, 100 epochs for the single perceptron and 10 random starts for the two-layer network. The single-perceptron numbers (6 and 4 epochs) depend on the learning rate and the order of the examples; the XOR failure doesn't, because no setting can fix it.
- The history is compressed. Backpropagation had earlier roots before 1986, and the AI winter had more than one cause.
The one-line mental model
A perceptron is a straight line: weigh the inputs, add a bias, decide which side you're on. One line can't learn XOR; layers of them, trained by backpropagation, can learn far more, and a 2-billion-parameter model has about 147,000 of them in its feed-forward blocks alone.