Quest 3 of 15
Neural Networks & PyTorch Fundamentals
Move from classical ML to deep learning: perceptrons, multi-layer networks, activation functions, backpropagation intuition, and PyTorch tensors, autograd, and nn.Module. Following the 'build then use' pattern, you will implement a tiny network in NumPy, then rebuild with torch.nn and compare outputs. For example, you will train a two-layer MLP on MNIST digits using CrossEntropyLoss and Adam.
Start here
A neural network is not a mysterious brain in code. It is a parameterised function built from layers. PyTorch tracks the operations used in the forward pass so automatic differentiation can calculate how every parameter contributed to the loss.
Big idea
Training a neural network repeats a disciplined loop: load a batch, compute logits in the forward pass, calculate loss, clear old gradients, backpropagate with loss.backward(), update parameters with optimizer.step(), and evaluate on separate data without gradient updates.
PyTorch training epoch
Compute logits and loss on a batch.
Learn one idea at a time
Read, explore, then mark each idea when you can explain it.
Idea 1 of 8
Subclass nn.Module to register learnable layers in __init__ and define how tensors flow through them in forward(). A simple image classifier may flatten 28×28 pixels, apply a Linear layer, a ReLU activation, and a final Linear layer that returns one logit for each class.
Choose a deep dive
Open the topics you want to explore. The detail stays folded until you need it.
Deep dive 1Worked example: MNIST from batch to update
A DataLoader returns 64 grayscale images shaped (64, 1, 28, 28) and 64 digit labels. The model returns logits shaped (64, 10). CrossEntropyLoss compares each row with its label and produces one scalar loss. backward() calculates gradients for every weight, and Adam updates them. At validation time, the team switches to eval mode, disables gradients, counts correct predictions, and saves the checkpoint only if validation accuracy improves.
Deep dive 2Debug the loop before changing the architecture
When loss does not fall, first inspect shapes, dtypes, label ranges, normalisation, and whether parameters receive non-zero finite gradients. Try to overfit a tiny batch: if the model cannot memorise 16 examples, the pipeline likely has a bug. Only after this sanity check should you increase depth, add regularisation, or launch a broad hyperparameter search.
Deep dive 3Activations introduce non-linearity
Without ReLU or similar, stacked linear layers collapse to one linear map. Non-linearities let networks approximate complex functions.
Deep dive 4Device and dtype matter
model.to(device) and inputs on the same device. Mixed precision (fp16) speeds training on GPU but needs care on CPU-only labs.
Deep dive 5The five-line training loop, annotated
Every PyTorch loop is variations on: optimizer.zero_grad() clears gradients accumulated from the previous step — forget it and gradients silently sum across batches, a classic bug. outputs = model(inputs) runs the forward pass and records the computation graph. loss = criterion(outputs, targets) reduces the batch to one scalar. loss.backward() sweeps the graph backwards filling every parameter's .grad. optimizer.step() applies the update rule. Wrap validation in torch.no_grad() and switch model.eval() so dropout and batch-norm behave correctly — then back to model.train(). Recite the five lines and the two mode switches; they diagnose half of all beginner training bugs.
Deep dive 6Debug training like an experimentalist
When a network refuses to learn, apply the standard ladder. First, overfit ten samples: a healthy model should drive loss to near zero on a tiny set — if it cannot, the bug is in the architecture, loss, or labels, not the data volume. Second, check the loss at initialisation: a ten-class classifier should start near ln(10) ≈ 2.3; anything else suggests wrong scaling or a broken loss. Third, watch gradient norms: all zeros means a detached graph; exploding values mean the learning rate or initialisation is wrong. Fourth, visualise a batch exactly as the model receives it — wrong normalisation and shuffled labels hide here. Ordered checks beat random hyperparameter twiddling every time.
Deep dive 7Training loop discipline
zero_grad → backward → step; eval() for validation.
One-minute challenge
Connect this lesson to real life
Name one situation where this idea could help, and one thing a person should still check.
Explore a real-world example
Use the arrows to connect the idea to a visible situation.
Photo example
Example: MNIST baseline
Log val accuracy every epoch before tuning architecture.
Key terms
Tap a term to flip and read the definition.
Optional further learningFree textbooks and trusted online resources
These sources informed the course structure. Use them to revisit a concept or study it in more depth.
Ready check
Tick each idea only when you could explain it without looking back.
Ready for practice? Train MNIST MLP; log train and validation accuracy each epoch.
Extra context (audience, logistics, curriculum notes)
Built for: Requires Modules 1–2; GPU optional (CPU training fine for labs).
Formats: PyTorch labs · Architecture sketching · Training-loop checklist · Quiz
Developer track Module 3 — neural nets, autograd, PyTorch training loop.
Next up
Ready for the next part?
When you've finished the reading, inline exercises, and knowledge check for this part, check the box to continue.