← All explainers
An explainer
Neural
Networks
Cover & TOC
The learning game
  • 01 The prediction game
  • 02 Kinds of learning
  • 03 Turning the world into numbers
  • 04 The simplest model
  • 05 Measuring wrongness
  • 06 Drawing a boundary
How learning happens
  • 07 Which way is downhill?
  • 08 Gradient descent
  • 09 Tuning the descent
  • 10 Batches, epochs & noise
From a line to a network
  • 11 Neural networks
  • 12 Why depth wins
  • 13 Backpropagation
  • 14 The training loop
  • 15 Where training starts
  • 16 When gradients vanish
  • 17 Shapes of networks
Learning vs. memorizing
  • 18 Learning vs. memorizing
  • 19 Data quality & the three splits
  • 20 Judging a model
The bigger picture
  • 21 A short history
  • 22 Embeddings
  • 23 From this loop to an LLM
§ Glossary
Section 16

When gradients vanish

The problem that stalled deep learning, and the fixes

Premium section

Unlock “How Neural Networks Learn, from the ground up”

This section is part of the paid explainer. Buy it once for lifetime access to all 23 sections. Sections 01, 02, 03, 04 are free to read.

Sign in to buy · $8

New here? Create an account

← Previous · §15
Where training starts
Next · §17 →
Shapes of networks
© 2026 Knowlific Inc. All rights reserved.
Terms of Service Privacy Policy