Study guide · Neural networks · 1943

A Logical Calculus of the Ideas Immanent in Nervous Activity

Warren McCulloch, Walter Pitts · Bulletin of Mathematical Biophysics

A neurophysiologist and a teenage logician ask what a brain could compute if each nerve cell were a simple on/off switch. Their answer is the first artificial neuron, and a proof that networks of them can carry out any logical rule you can write down.

The paper in four sentences

A nerve cell either fires or it doesn't. McCulloch and Pitts treat that as a yes/no statement, like a light that is on or off.

A model neuron fires when enough of its "go" inputs are on at once and none of its "stop" inputs are. The number it needs is its threshold.

By wiring such neurons together you can build AND, OR and NOT, and from those any logical rule, including rules about time ("this happened, then that").

Loops of neurons can hold a signal, which gives the network memory. The weak spot: the wiring is fixed, so this neuron cannot learn. That came 15 years later with the perceptron.

Idea · Introduction

A nerve cell is a yes-or-no switch

In 1943 scientists already knew a key fact about nerve cells (neurons): a neuron does not fire "a little". Either the electrical pulse travels down it at full strength, or nothing happens. Biologists call this all-or-none.

Warren McCulloch, a doctor studying the brain, and Walter Pitts, a self-taught logician who was still a teenager, noticed that "all-or-none" sounds a lot like "true or false". If a firing neuron means "true", then a brain is a machine that combines true/false statements. That is exactly what logic studies.

So they asked a precise question: if neurons are switches, what can a network of them work out?

Everyday pictureThink of a neuron as a doorbell that only rings if enough people press their buttons at the same moment, and nobody is holding the "do not disturb" switch.

TakeawayBecause a neuron fires fully or not at all, its state can be read as a true/false statement, and the brain can be studied with logic.

Check yourself
What does "all-or-none" mean for a neuron?
It either sends its full pulse or none at all. There is no half-strength firing.
Why did that make logic a good tool?
Logic deals with statements that are either true or false, which matches a neuron that is either firing or not.

Idea · Assumptions

The model neuron: count the votes

Their neuron has two kinds of incoming wires. Excitatory wires vote "fire". Inhibitory wires are a veto. Each moment, the neuron counts how many excitatory wires are active. If the count reaches its threshold and no inhibitory wire is active, it fires in the next moment.

That is the whole model. There are no shades of grey, no strengths: every "go" wire counts the same, and one veto beats any number of votes. Try it below.

Click the inputs to switch them on. The neuron fires when the number of active "go" inputs reaches the threshold and the veto is off.

TakeawayA McCulloch–Pitts neuron fires when at least threshold excitatory inputs are on and no inhibitory input is on.

Check yourself
With threshold 2, inputs A and B on, and the veto on, does it fire?
No. In this model a single active inhibitory input blocks firing completely.
Do some excitatory inputs count more than others?
No. In the 1943 model every excitatory input counts the same. Weighted inputs came with later models.

Idea · Nets without circles

Three settings give you AND, OR and NOT

Change only the threshold and the same neuron behaves like different logic words. With two inputs and threshold 2, both must be on: that is AND. With threshold 1, either one is enough: that is OR. And a neuron that is kept firing by a steady input, unless the veto stops it, gives NOT.

AND, OR and NOT are enough to build any logical rule at all. This is the same fact that makes computer chips possible: every chip is, at bottom, a huge pile of these three operations. McCulloch and Pitts showed neurons could do it in 1943, before the first general-purpose electronic computers were built.

Pick a gate. The table shows every combination of inputs and whether the neuron fires. Fire = true, silent = false.

TakeawayOne kind of neuron, with different thresholds and vetoes, can act as AND, OR or NOT, which is enough to build any logic.

Check yourself
A neuron with two inputs fires when either is on. What is its threshold?
1. That makes it an OR gate.
Why is NOT the odd one out?
It needs the veto. The neuron is kept firing by a steady input, and the input you want to negate is wired in as the inhibitory one.

Idea · Expository figures

Time matters: why a cold touch can feel hot

Each neuron takes one tick of time to pass its signal on. So a network doesn't only combine signals; it also combines signals from different moments. That lets it detect patterns in time, like "cold for one tick, then nothing".

The paper's favourite example is a real illusion: if something very cold touches your skin for just a moment and is removed, you can briefly feel heat. Their small network explains how. A "cold" feeling needs the cold signal to last two ticks in a row. A brief cold touch fails that test, and the same wiring routes it to the "hot" feeling instead.

Click any square in the top two rows to switch a receptor on at that tick, or pick a preset. The bottom rows show what the network reports. Rules, from the paper: feel hot at tick t if the heat receptor fired at t−1, or the cold receptor fired at t−3 but not at t−2. Feel cold at t if the cold receptor fired at both t−2 and t−1.

TakeawayBecause signals take time to travel, networks of simple neurons can recognise patterns over time, and this can explain real illusions.

Check yourself
Why does a brief cold touch produce "hot" here?
"Cold" needs two ticks of cold in a row. A single tick fails that, and the rule for "hot" includes "cold at t−3 but not at t−2", which a single tick of cold satisfies.
Why is the hot feeling delayed compared with the heat receptor?
Every neuron in the chain adds one tick of delay.

Idea · Nets with circles

Loops give a network memory

If you wire a neuron so that its output comes back to its own input, one pulse is enough to keep it firing tick after tick. The network now remembers that something happened, even after the input is gone.

McCulloch and Pitts called these "nets with circles". They argued that with loops, plus something like an outside tape to read and write, such a network could compute anything Alan Turing's imaginary machine could compute. That is a big claim: in principle, a brain built from simple switches is as powerful as any computer.

Send one pulse, then advance time. The loop keeps the neuron firing with no further input until a veto stops it. This is one bit of memory.

TakeawayLoops let a signal circulate, so the network can remember the past. With loops and outside storage, the authors argued, neurons could compute whatever a Turing machine can.

Check yourself
What does a self-loop remember?
That a pulse arrived at some point in the past. It keeps firing until something inhibits it.
Can this 1943 network learn from experience?
No. Its wiring and thresholds are fixed. Learning from examples arrived with Rosenblatt's perceptron in 1958.

Idea · Why it lasted

What this paper started, and what it got wrong

Real neurons turned out to be messier than on/off switches: their inputs have different strengths, their timing is not in neat ticks, and inhibition usually weakens firing instead of cancelling it. So this is not an accurate picture of the brain.

But as an idea it was enormously productive. John von Neumann used these neurons to describe the logic of early computers. The mathematician Stephen Kleene studied what these networks can recognise, which led to "regular expressions", still used in every text search today. And the basic unit, "add up inputs, compare with a threshold, fire or not", is the ancestor of every artificial neuron in modern AI.

Later refinementThe missing piece was learning. Rosenblatt's perceptron (1958) gave each input an adjustable strength (a weight) and a rule for changing it from examples.

TakeawayThe 1943 neuron was too simple for biology but just right for computing: it is the starting point of artificial neural networks.

Follow the idea

Led to: Perceptron (1958)