Quest 8 of 15
Calculus & optimisation for training
SGD, momentum, Adam, and learning-rate schedules all come from calculus on the loss surface — implement and compare optimisers on a toy loss landscape.
Start here
This module builds on earlier parts, but every important idea is explained in context. SGD, momentum, Adam, and learning-rate schedules all come from calculus on the loss surface — implement and compare optimisers on a toy loss landscape. As you read, connect each concept to the worked example and ask what a person must still decide.
Big idea
SGD, momentum, Adam, and learning-rate schedules all come from calculus on the loss surface — implement and compare optimisers on a toy loss landscape.
Learn one idea at a time
Read, explore, then mark each idea when you can explain it.
Idea 1 of 8
Define a differentiable loss L(θ) that connects model parameters θ to the error you care about. Plot a toy two-parameter surface so gradients become visible slopes rather than unexplained symbols.
Choose a deep dive
Open the topics you want to explore. The detail stays folded until you need it.
Deep dive 1Not all minima are equal
Flat minima sometimes generalise better — a research topic, but learning rate and batch size affect which basin you find.
Deep dive 2LR too high diverges
If loss spikes or becomes NaN, reduce learning rate and check for gradient explosion.
Deep dive 3Worked example: Calculus & optimisation for training
Fine-tuning a transformer with the same LR used for training from scratch. A responsible response is: Use a much smaller LR and fewer epochs with early stopping. Correct — fine-tuning needs gentle updates. Use this case to separate what the technology contributes from what people contribute. The team should compare the intended outcome with a baseline where applicable, record important assumptions, and keep a clear route to correct or stop the process.
Deep dive 4Local check for Calculus & optimisation for training
Ask whether the examples, data, and assumptions fit your school, company, or community. Generic demos often miss local names, laws, connectivity, and languages.
Deep dive 5Optimisers follow the loss landscape
Compare SGD, momentum, and Adam from the same initial weights; learning curves show when steps oscillate or diverge.
One-minute challenge
Connect this lesson to real life
Name one situation where this idea could help, and one thing a person should still check.
Explore a real-world example
Use the arrows to connect the idea to a visible situation.
Photo example
Optimiser comparison
Same model, same data — compare SGD, momentum, and Adam on validation loss.
Key terms
Tap a term to flip and read the definition.
Optional further learningFree textbooks and trusted online resources
These sources informed the course structure. Use them to revisit a concept or study it in more depth.
Ready check
Tick each idea only when you could explain it without looking back.
Ready for practice? Train the same model with lr=1e-1 and lr=1e-4; compare curves.
Extra context (audience, logistics, curriculum notes)
Built for: Software developers and technical learners — Python and coding required.
Formats: Learn scroll · Practice interactions · Quiz · Optional chat lab
Module 8 — Calculus & optimisation for training
Next up
Ready for the next part?
When you've finished the reading, inline exercises, and knowledge check for this part, check the box to continue.