Neural network training: guide

How a network learns to tell cats from dogs · Week 3: Structured Data and Machine Learning

A very small neural network learns to sort animals into cats and dogs. You can watch every weight change while it learns, see which weights the error pushes hardest, and test the result on animals it has never seen. The procedure is backpropagation, the same one used by the large models later in the course, at a size where every number is still visible.

Open the network Open with fixed starting weights (seed 1)

1. The screen, part by part

After 100 epochs on the overlapping data (seed 1): 11 of 12 training animals right, all 4 unseen animals right, and the loss levelling off at about 0.2.

Links that open in a given state: ?seed=1 (the same starting weights every time, for a class demonstration), ?data=clear, ?steps=20; combine them, for example ?seed=1&steps=10. Other keys: P presentation mode, ? help, G this guide.

2. The data: overlapping, clear-cut and unseen animals

Each animal is four made-up numbers between 0 and 1: ear pointiness, whisker length, nose wetness and fur pattern.

3. What happens in one epoch

The training animals are shuffled. For each one in turn:

1. Forward pass. Each hidden neuron adds up the four inputs, each multiplied by its weight, plus a bias. If the sum is negative the neuron stays at 0 (ReLU). The two outputs do the same with the six hidden values, and softmax turns them into two probabilities.
2. Loss. How surprised the network is by the right answer: −log(probability it gave to the right label). 50/50 gives 0.69; 99% sure and right gives 0.01; 99% sure and wrong gives 4.6.
3. Backward pass. Working back from the output, the network works out for every weight whether nudging it up or down would have made the loss smaller, and by how much (the gradient).
4. Update. Every weight moves a little against its gradient: new weight = old weight − learning rate × gradient.

The halos and the Biggest weight changes list add up these small updates over the whole epoch.

4. Walkthrough 1: the first epochs

Open with seed 1  about 5 minutes. With seed 1 the numbers below are exactly what you will see.

1. Look before you press. The lines are thin and mixed: the weights are random. Ask: if it guessed now, how often would it be right?
2. Step once (Space). Loss 0.70: the loss of a coin toss. Training accuracy 83.3%, unseen 75%. With random weights the network happens to lean one way.
3. Step to epoch 10. Training accuracy 91.7% (11 of 12), unseen 100%. The loss curve drops steeply, then flattens.
4. Press Train (Enter). The training loss levels off at about 0.2 and never reaches 0. The next two walkthroughs explain why.

5. Walkthrough 2: backpropagation, weight by weight

1. Reset, then Step a few times, watching the pink halos. Early on, the halos cluster around one or two hidden neurons: the error is pushed through the neurons that are active, and neurons that output 0 pass no error back.
2. Read the Biggest weight changes list aloud. With seed 1 the top entry after the first Step is Whisker Length → H4 +0.061: after this epoch, long whiskers push H4 a little more.
3. Later in training you will often see pairs like H4 → Dog +0.017 and H4 → Cat −0.017: the two outputs compete, so a gain for one is a loss for the other.
4. Watch the size of the changes. With seed 1 the top change grows to about 0.14 by epoch 20, then falls to about 0.02 by epoch 100. It never reaches 0, because Cat 6 and Dog 6 pull the same weights in opposite directions in every epoch (walkthrough 3). With other seeds the tug-of-war stays larger, around 0.05.

6. Walkthrough 3: the animals it cannot get right

1. Train to 100 epochs and scroll to Every animal, right now.
2. Cat 5 (folded ears) and Dog 5 (pointy ears) end up right, usually at 95% or more: the network has learned to use whiskers and nose as well as ears.
3. Cat 6 and Dog 6 have the same bar, because they have the same numbers. One of them always gets a ✗; which one depends on the starting weights. Training accuracy stops at 11 of 12 (91.7%), and the loss cannot fall below 0.12: this pair alone costs 2 × 0.69 / 12.
4. Look at the unseen animals. In simulated runs the network gets all four right in almost every run, including the folded-ear cat and the pointy-ear dog. It has learned something that carries over.

7. Walkthrough 4: easy data, and the learning rate

Open with the clear-cut data

With only the eight clear-cut animals, training accuracy reaches 100% within a few epochs and the loss goes almost to 0. The task looks solved, but the network has never met a cat with folded ears. Compare the unseen animals with those in the overlapping run.

Then run both datasets at three learning rates, pressing Reset in between. Typical results from 200 simulated runs of each, after 100 epochs:

A larger learning rate learns faster here. With 0.5 the loss curve on the overlapping data jitters, because each step overshoots a little. Ask the room why nobody sets the learning rate to 100.

8. Questions for class

9. What the demonstration leaves out

The data are illustrative: invented animals, four hand-picked features, and a test set of only four. Real image classifiers start from pixels, not features, have millions of weights, train on batches of thousands of examples, and are tested on thousands of held-out images. The mechanics of forward pass, loss, backward pass and update are exactly the same.

Used in the Week 3 lecture. Source: tools/neural-net.