Deep Learning / Neural Networks

Create a free account to open this chapter.

• What a prediction is ~12 min
• Features: turning the world into numbers ~12 min
• A linear model is a weighted sum ~16 min
• Weights: what turning one knob does ~14 min
• The bias: shifting the whole answer ~12 min
• How wrong was the guess? Error and squared error ~14 min
• Fitting a line by eye ~16 min

Create a free account to open this chapter.

• A neuron is a weighted sum with a switch ~14 min
• The sigmoid: squashing any number into 0 to 1 ~16 min
• Why a neuron needs a bend ~13 min
• The perceptron: drawing a line between two groups ~16 min
• A problem one line cannot solve: XOR ~15 min
• A single neuron, start to finish ~18 min
• The brain story, and what a neuron really computes ~10 min

Create a free account to open this chapter.

• A vector is a list of numbers ~12 min
• Adding vectors and stretching them ~13 min
• The dot product: a weighted sum in one symbol ~15 min
• A matrix is a table, one row per neuron ~13 min
• Matrix times vector: every row does its own weighted sum ~16 min
• Matrix times matrix, and why shapes must match ~14 min
• Shapes and the batch dimension ~15 min

Create a free account to open this chapter.

• Several neurons side by side, a layer ~15 min
• Stacking layers, hidden layers ~14 min
• Counting the parameters ~12 min
• The forward pass, one layer at a time ~18 min
• A network is a function built from functions ~13 min
• Architecture, how many layers, how wide ~14 min
• A full forward pass, by hand ~20 min
• What a hidden layer sees ~16 min

Create a free account to open this chapter.

• Sigmoid, up close ~14 min
• Tanh: squashing to between -1 and 1 ~15 min
• ReLU and leaky ReLU: the simplest bend ~15 min
• GELU, SiLU, and ELU: the smooth ReLUs ~14 min
• Flat spots, and choosing an activation ~16 min

Create a free account to open this chapter.

• Probability is a number between 0 and 1 ~12 min
• A distribution is a list of probabilities adding to 1 ~12 min
• Expected value: the long-run average ~14 min
• Given that: conditional probability ~13 min
• Surprise and entropy ~18 min
• Cross-entropy: comparing your guess to the truth ~20 min
• KL divergence: going deeper (optional) ~14 min

Create a free account to open this chapter.

• Why training needs a single number ~13 min
• Mean squared error ~16 min
• From scores to probabilities: the softmax ~17 min
• Binary cross-entropy: yes or no ~16 min
• Cross-entropy loss for many classes ~18 min
• Matching the loss to the task ~12 min
• Why squared error is a probability statement: going deeper (optional) ~16 min

Create a free account to open this chapter.

• Steepness: the slope of a straight line ~13 min
• The slope of a curve at one point ~17 min
• The derivative: a formula for the slope ~16 min
• The slope of each activation function ~16 min
• Two knobs: partial derivatives ~16 min
• The gradient: all the slopes in one list ~14 min
• The chain rule: slopes multiply through a chain ~17 min

Create a free account to open this chapter.

• Training is a search for the lowest loss ~12 min
• One step downhill: the update rule ~16 min
• The learning rate: how big a step ~14 min
• Two weights at once: walking down a bowl ~13 min
• Fitting the line with gradient descent ~20 min
• A few examples at a time: stochastic and minibatch descent ~16 min
• Momentum and Adam: smarter steps ~15 min

Create a free account to open this chapter.

• The problem — a slope for every weight ~14 min
• Computational graphs — drawing the forward pass ~16 min
• The chain rule on a graph — passing blame backward ~18 min
• Backpropagation through one neuron ~20 min
• Backpropagation through the 2-2-1 network, by hand ~28 min
• Backpropagation in matrix form ~20 min
• Autograd — the computer does the chain rule ~18 min

Create a free account to open this chapter.

• Putting it together: the training loop ~18 min
• Epochs, iterations, and batches ~13 min
• Watching training: the loss curve ~12 min
• Validation: checking on data the network never saw ~16 min
• Early stopping and checkpoints ~16 min
• Learning-rate schedules and warmup ~15 min
• Gradient clipping: keeping steps under control ~12 min
• Randomness, seeds, and why two runs differ ~13 min
• Training a network in PyTorch: what each line is ~20 min

Create a free account to open this chapter.

• Memorizing versus learning: overfitting and underfitting ~17 min
• The bias-variance tradeoff, in plain terms ~15 min
• Weight decay: keeping weights small ~17 min
• Dropout: training with neurons switched off ~18 min
• Data augmentation and more data ~14 min
• Cross-validation and data leakage ~18 min
• The confusion matrix ~16 min
• Precision, recall, and F1 ~18 min
• ROC curves and AUC (optional) ~15 min
• When the test set lies: calibration, imbalance, and shift (optional) ~16 min

Create a free account to open this chapter.

• Vanishing and exploding gradients ~17 min
• Why all weights cannot start equal ~15 min
• Choosing the starting weights: Xavier and He ~19 min
• Normalizing the inputs ~15 min
• Batch normalization ~19 min
• Layer normalization, RMSNorm, and choosing one ~19 min

Create a free account to open this chapter.

• A fixed diagnostic order ~18 min
• Overfitting a tiny batch first ~16 min
• Loss that won't decrease ~16 min
• Loss that explodes ~15 min
• Accuracy that's stuck ~14 min
• Checking labels and data leakage ~15 min
• Checking tensor shapes and gradients ~16 min
• Numerical instability ~15 min

Create a free account to open this chapter.

• Why fully connected layers don't suit images ~16 min
• Invariance and locality ~17 min
• Cross-correlation and convolution ~16 min
• A hand-computed convolution ~18 min
• Edge detection with a hand-built kernel ~17 min
• From hand-built to learned kernels ~18 min
• Padding and stride ~16 min
• Multiple channels and 1×1 convolution ~18 min
• Pooling ~15 min
• LeNet, end to end ~20 min

Create a free account to open this chapter.

• AlexNet and the missing ingredients ~14 min
• VGG blocks ~15 min
• Network in Network ~14 min
• Inception and GoogLeNet ~15 min
• ResNet and residual blocks ~17 min
• DenseNet ~15 min
• Architecture design spaces ~15 min
• CNNs and Vision Transformers today ~15 min
• Transfer learning: freeze, replace, and fine-tune ~18 min
• An introduction to object detection ~13 min
• An introduction to image segmentation ~13 min

Create a free account to open this chapter.

• Tokenization ~16 min
• Vocabularies and out-of-vocabulary words ~14 min
• Why one-hot vectors fail as a representation ~14 min
• word2vec: skip-gram and CBOW ~18 min
• Negative sampling ~14 min
• GloVe and co-occurrence statistics ~15 min
• Word similarity and analogy ~16 min
• Toward contextual embeddings ~12 min

Create a free account to open this chapter.

• Sequential data and the autoregressive idea ~16 min
• n-gram language models and perplexity ~18 min
• From shared weights to a hidden state ~20 min
• Unrolling a recurrent network ~15 min
• A character-level language model ~22 min
• Gradient clipping in recurrent networks ~14 min
• Backpropagation through time ~24 min
• Truncated and randomized truncated BPTT ~18 min
• What recurrent networks struggle with ~16 min

Create a free account to open this chapter.

• The long-term dependency problem ~14 min
• The LSTM cell, gate by gate ~24 min
• The GRU as a simplification ~20 min
• Deep and bidirectional recurrent networks ~16 min
• The encoder-decoder architecture ~18 min
• Teacher forcing ~16 min
• Greedy, exhaustive, and beam search ~18 min
• Sequence applications: two architectures, one task ~12 min
• Where sequence tasks go next ~10 min

Create a free account to open this chapter.

• Why a fixed-size representation falls short ~18 min
• An analogy for attention ~16 min
• Queries, keys, and values ~22 min
• Attention pooling and the Nadaraya-Watson connection ~20 min
• Scoring functions: dot-product to additive ~20 min
• Scaled dot-product attention ~22 min
• Why the scaling factor? ~20 min
• Masking ~18 min
• Multi-head attention ~22 min
• Self-attention versus CNNs and RNNs ~20 min

Create a free account to open this chapter.

• Positional encoding ~18 min
• The position-wise feed-forward network ~14 min
• Residual connections and layer normalization ~16 min
• The Transformer encoder ~16 min
• The Transformer decoder ~18 min
• Training a small Transformer ~20 min
• The Vision Transformer ~16 min
• Encoder-only, decoder-only, and encoder-decoder families ~16 min
• Context windows and KV caching ~16 min
• Efficient attention and a look at state-space models ~16 min
• Loading and inspecting a pretrained model ~14 min

Create a free account to open this chapter.

• Autoencoders ~22 min
• Variational autoencoders: a preview ~12 min
• Generative adversarial networks: a preview ~10 min
• Diffusion models: a preview ~10 min
• Language models at scale: a preview ~10 min
• Fine-tuning and LoRA: a preview ~10 min
• Hyperparameter search ~14 min
• Production training and deployment: a preview ~14 min
• The end-to-end decision chain ~20 min

Create a free account to open this chapter.

• Numbers and variables ~10 min
• Lists: a feature vector, typed out ~10 min
• Dictionaries, storage looked up by name ~9 min
• Functions: def and return ~12 min
• Loops: the same weighted sum, five times ~11 min
• if / else: the step activation, in code ~12 min
• Imports: borrowing other people's code ~8 min
• A first plot ~10 min

Create a free account to open this chapter.

• An array is a vector ~9 min
• Shape ~8 min
• Indexing and slicing ~10 min
• Elementwise operations ~9 min
• np.dot and @: the neuron in one line ~12 min
• Broadcasting: a library convention, not a math rule ~12 min
• Random numbers and seeds ~9 min
• Reductions: .sum(), .mean(), .argmax() ~10 min

Create a free account to open this chapter.

• The line plot: plt.plot(x, y) ~10 min
• Scatter and a fitted line: plt.scatter, then plt.plot ~12 min
• Plotting a loss curve ~12 min
• Subplots, titles, and legends ~12 min
• A function and its slope, on the same picture ~12 min

Create a free account to open this chapter.

• What a notebook is ~8 min
• Cells run in the order you run them ~9 min
• Markdown cells hold prose and math ~8 min
• Restart and run all ~8 min

Create a free account to open this chapter.

• What Colab is ~6 min
• Opening a notebook from this course ~6 min
• Running cells ~8 min
• CPU vs. GPU runtime ~8 min
• Installing packages ~6 min
• Saving and sharing ~6 min
• Loading notebooks from a repository ~4 min

Create a free account to open this chapter.

• Tensors are arrays ~10 min
• Shapes, devices, and dtypes ~9 min
• nn.Linear is Wx + b ~16 min
• Activations are just functions ~12 min
• Losses are Chapter 7 ~18 min
• Autograd is Chapter 10 ~14 min
• Optimizers are Chapter 9 ~14 min
• DataLoader is Chapter 11's batches ~13 min
• Writing your own nn.Module ~16 min
• The training loop, and seeds ~18 min