How neural networks learn sounds mysterious, but the core loop is short enough to describe in a paragraph. The network makes a guess, a score says how wrong the guess was, and then millions of adjustable numbers get nudged a tiny amount in whatever direction would have made the guess slightly better. Repeat that a few billion times and a pile of arithmetic starts recognizing faces, translating sentences or predicting the next word. Two ideas do nearly all the work: gradient descent, which decides how to move, and backpropagation, which works out which way each number should move.

What is a neural network actually made of?

Strip away the brain metaphors and a neural network is a long chain of simple calculations. Each artificial neuron takes some numbers in, multiplies each by a weight, adds the results together and passes the total through a small function that bends it. The output goes on to the next layer of neurons. Stack enough layers and you have a deep network.

The weights are the important part. A weight sets how strongly one neuron’s signal affects the next, so the weights together are the network’s entire memory of what it has learned. A modern language model has billions of them. A small image classifier might have a few million. Before training, they are set to small random values, which means an untrained network produces noise, not even useful mistakes.

That is the first thing to hold on to. Learning, in this context, does not mean storing facts the way a database does. It means adjusting weights until the network’s outputs match what we want.

What does it mean for a network to “learn”?

Take the supervised case, which is the easiest to picture. You have a pile of examples, each pairing an input with a correct answer: a photo and the label “cat”, or an email and the label “spam”. The network sees an input, produces its own answer, and gets compared against the correct one. If the two disagree, the weights change.

One more point, and it surprises people: this learning happens during a training phase, before the product reaches you. When you chat with a deployed model, its weights are normally frozen. It is applying what it learned, not rewriting itself as you type. Models are usually retrained or fine-tuned in separate, deliberate runs.

How does the network know how wrong it is?

Through a loss function, which boils the gap between the network’s answer and the right answer down to a single number. Big number, bad guess. Zero, perfect guess.

Which loss you pick depends on the task. For predicting a quantity such as a house price, a common choice is squared error: take the difference between prediction and truth, then square it, so that large misses are punished much harder than small ones. For sorting things into categories, cross-entropy is the usual choice, which penalizes the network heavily when it is confident and wrong.

The whole of training is an attempt to push this one number down, averaged across the training examples. Everything else is method.

Here is a deliberately tiny example. Imagine a network with a single weight, w, whose job is to scale its input. Its prediction is w times the input. We feed in 2 and the correct answer is 6, since this toy world happens to triple things. With w set to 1, the prediction is 2, the miss is 4, and the squared error is 16. We need w to get bigger, and we need to know by how much.

What is gradient descent?

Picture standing on a hillside in thick fog, wanting to reach the valley floor. You cannot see the valley. You can feel the slope under your feet, though. So you step downhill, feel the slope again, step again, and eventually you stop going down.

That is gradient descent. The hillside is the loss, plotted against every weight at once. The slope is the gradient, a list of numbers saying how much the loss would rise or fall if each weight were nudged. The method moves each weight a little in the direction that lowers the loss, which is opposite the gradient. Walk the other way, uphill, and you would be doing gradient ascent, which maximizes the function instead.

Back to the toy. The loss is (w times 2, minus 6) squared. Its slope with respect to w works out to 2 times (prediction minus target) times the input, which at w equal to 1 is 2 times negative 4 times 2, or negative 16. A negative slope means increasing w lowers the loss. If we move w by the slope times a step size of 0.05, w becomes 1.8. The prediction is now 3.6 and the squared error has dropped from 16 to about 5.8. One more step takes w to about 2.28, and the error to roughly 2.1. After a handful of steps w settles near 3 and the error is essentially gone.

Real networks do exactly this, only with billions of weights instead of one, and a loss surface with more dimensions than anyone can picture.

What does the learning rate do?

The step size in that example, 0.05, is the learning rate, and it matters more than almost any other setting. If it is too small, training crawls and may never get anywhere useful in a reasonable time. If it is too large, each step overshoots the valley, lands on the opposite slope, overshoots again, and the loss can bounce around or blow up entirely.

Another practical wrinkle is the word “stochastic.” Computing the true gradient over millions of training examples for every single step would be painfully slow. Stochastic gradient descent estimates the slope from a small random batch of examples instead. The estimate is noisy, but it is cheap, and the noise turns out to be tolerable and may even help. The method is described in standard references as the basic algorithm used to train most deep networks today.

What is backpropagation, and why is it needed?

Gradient descent needs the gradient. For our one-weight toy, working it out took one line of algebra. For a network with many layers and billions of weights, you need the slope of the loss with respect to every single weight, and a naive approach, wiggling each weight to see what happens, would require an absurd number of passes through the network.

Backpropagation is the efficient answer. Strictly speaking, the term refers only to the algorithm for computing the gradient; people often use it loosely for the whole training procedure, and it helps to keep the two apart. It rests on the chain rule from calculus, which says how to compute the effect of a change that passes through a chain of steps.

The procedure has two halves. In the forward pass, an input travels through the network and the intermediate values at each layer are stored. The loss is then computed at the output. In the backward pass, the error signal travels back one layer at a time. Each layer takes the signal from the layer after it, works out how much each of its own weights contributed, and hands a modified signal to the layer before it. Because each layer’s result is reused, nothing gets computed twice, and the backward pass costs roughly as much as the forward one.

That is the whole trick, and it is why training networks with many layers became practical. Backpropagation answers “which way should each weight move?” Gradient descent answers “so how far should I actually move it?”

Where did these ideas come from?

The history is messier than the textbook version, and priority is genuinely disputed. Here is the sequence as recorded in the standard reference accounts.

The chain rule itself was published by Leibniz in 1676. In 1960 Henry Kelley published gradient methods for optimal flight paths, and in 1967 Shun’ichi Amari described what is usually cited as the first multilayer perceptron trained by stochastic gradient descent. Seppo Linnainmaa published the reverse mode of automatic differentiation, the mathematical core of backpropagation, in 1970. Paul Werbos applied the method to multilayer networks in the way that became standard in 1982.

The version most people have heard of is the 1986 paper in Nature by David Rumelhart, Geoffrey Hinton and Ronald Williams, which showed that networks trained this way could learn useful internal representations. It did a great deal to popularize the approach. It did not invent it from nothing, and several people had arrived at parts of it earlier.

Interest faded through the 2000s and returned in the 2010s, helped by graphics processors that could do the huge matrix arithmetic quickly. In 2012 AlexNet, trained on two consumer graphics cards over five to six days, won the ImageNet image-recognition contest with a top-5 error rate of 15.3 percent, more than 10 points ahead of the runner-up. The field took notice.

In 2024, Geoffrey Hinton received the Nobel Prize in Physics, with the recognition tied to foundational work on artificial neural networks.

What can go wrong during training?

Quite a lot, and much of the craft of machine learning lives here.

The loss surface is not a clean bowl. Gradient descent with backpropagation is only guaranteed to find a local minimum, a valley that is not necessarily the lowest one, and it can slow to a crawl on flat plateaus and near saddle points. In practice, large networks often reach solutions that work well anyway, and exactly why remains an active research topic rather than a closed question.

The most familiar failure is overfitting. A network with far more capacity than the task needs can memorize its training examples, noise included, and then do badly on new inputs. The standard defence is to hold back some data the network never trains on and check performance there, along with regularization methods that discourage over-complicated solutions.

Why can’t we just read what the network learned?

Because the knowledge is smeared across all the weights. There is no neuron labeled “cat ears” waiting to be found. Some units do turn out to respond to recognizable features, and researchers have built techniques to probe them, but a complete, human-readable account of why a large network gave a particular answer is generally not available. That is why neural networks are often called black boxes, and it is the motivation for the research field known as explainable AI.

Common questions

Is backpropagation the same thing as gradient descent?

No. Backpropagation computes the gradient, meaning the slope of the loss for every weight. Gradient descent, or a variant such as Adam, uses that gradient to update the weights. They are partners, and people often say “backpropagation” for the pair.

Does a chatbot learn from me while we talk?

Usually not in the sense of changing its weights. Training happens beforehand in separate runs. A conversation can still shape the replies within that chat, because earlier messages are fed back in as context, and companies may use conversations for later training depending on their policies, which vary.

Why does training take so long?

Each step requires a forward and backward pass over a batch of examples, and a large model needs a vast number of steps across a huge dataset. Even AlexNet, small by today’s standards, ran for 90 passes over its training set, called epochs, taking five to six days on two graphics cards.

Does the human brain use backpropagation?

That is unsettled. The mechanism as written requires information to flow backward through exactly the same connections used going forward, which real neurons are not known to do in a simple way. Researchers have proposed brain-compatible approximations, but I have not seen a settled answer, so treat any confident claim either way with caution.

Can a network get stuck and stop learning?

Yes. A learning rate that is too high can make the loss bounce, a bad starting point or a plateau can stall progress, and gradients that vanish in deep stacks can freeze early layers. Training runs are monitored and restarted or adjusted for exactly these reasons.

What to remember

A neural network is a big function with adjustable weights. Training measures its error with a loss, uses backpropagation to compute how each weight affected that error, and uses gradient descent to move the weights a little in the helpful direction. Repeat at scale, with plenty of data and fast hardware, and useful behavior emerges.

The method is old in its parts and recent in its success. The mathematics goes back centuries, the training recipe took shape over the second half of the twentieth century, and the hardware to make it dramatic arrived around 2012. Its limits are real: local minima, overfitting, fragility and opacity. None of that is magic, and none of it is solved.

Further reading

The standard textbook for readers who want the math behind training. See it on Amazon (affiliate link)

As an Amazon Associate I earn from qualifying purchases. A commission may be earned if you buy through this link, at no extra cost to you.

Sources