Quest 4 of 15
Training, Evaluation & Model Improvement
Go deeper on making models reliable: learning rates, batch size, optimisers (SGD, Adam), schedulers, early stopping, confusion matrices, ROC curves, and error analysis. You will run hyperparameter sweeps on a small grid and document what improved validation performance without touching the test set. For example, you will inspect the worst-predicted examples and hypothesise whether more data, features, or architecture change would help.
Start here
A training run is evidence, not a verdict. Model improvement comes from controlled experiments that isolate one hypothesis, preserve comparable data splits, and inspect concrete failures instead of blindly increasing model size.
Big idea
Tune hyperparameters on validation data, keep experiment records detailed enough to reproduce results, and use error analysis to decide what to change. The goal is not the highest convenient score; it is dependable performance on the cases the system will actually face.
Learn one idea at a time
Read, explore, then mark each idea when you can explain it.
Idea 1 of 9
Write a baseline run before tuning: code commit, data version, split seed, preprocessing, architecture, optimiser, learning rate, batch size, runtime, and metrics. Change one main factor per experiment where possible so an improvement has a plausible explanation.
Choose a deep dive
Open the topics you want to explore. The detail stays folded until you need it.
Deep dive 1Worked example: improving a visual defect model
A factory classifier reaches 94% validation accuracy, but the error gallery shows most missed defects occur on the night shift under different lighting. Increasing network depth adds little. The team collects representative night-shift images, checks labels with inspectors, adds controlled brightness augmentation, and evaluates day and night slices separately. The improvement comes from understanding the failure mode, not from a random architecture change.
Deep dive 2Validation data can be overfit too
If a team tries hundreds of configurations and repeatedly chooses whichever scores highest on one validation set, design decisions gradually adapt to that set’s quirks. Keep a final test set untouched, limit unprincipled searching, and consider nested cross-validation for smaller classical datasets. Production monitoring remains necessary because even an honest test set represents the past.
Deep dive 3Test set is sacred
Touch it once at the end. Repeated peeking at test while tuning is the same as using it for validation.
Deep dive 4Learning rate finders
Increase LR until loss diverges, then pick a value below the knee — a practical trick before full grid search.
Deep dive 5Worked example: slicing a 94%-accurate model
A defect classifier reports 94% validation accuracy and the team nearly ships it. Slicing changes the story: accuracy is 97% on images from factory line A but 78% on line B, whose camera sits at a different angle; 96% in daytime batches but 81% at night under artificial light. The aggregate number hid two operating regimes. Fixes follow directly from the slices — augment training with line-B angles, add night-shift images — and the next evaluation reports per-slice accuracy with a minimum-slice threshold as the shipping criterion. Aggregate metrics describe the average case; production failures live in the worst slice. Slice by every operational dimension you can name: source, time, class, segment, geography.
Deep dive 6Hyperparameter search without wasted GPU-hours
Random search beats grid search in high dimensions because grid spends its budget re-testing the same few values of each parameter while random explores more distinct values. Search learning rate and regularisation strength on a log scale — the difference between 1e-3 and 1e-4 matters far more than between 0.003 and 0.004. Use successive halving: launch many configurations on short budgets, keep the top fraction, extend them. And log everything — the config that was second-best on this dataset is your warm start on the next one. Above all, fix data and evaluation first: no search recovers what leakage or a wrong metric has already destroyed.
Deep dive 7Experiments are evidence
Log LR, seed, and data version — not only final accuracy.
One-minute challenge
Connect this lesson to real life
Name one situation where this idea could help, and one thing a person should still check.
Explore a real-world example
Use the arrows to connect the idea to a visible situation.
Photo example
Example: error gallery
Worst mistakes guide data collection.
Key terms
Tap a term to flip and read the definition.
Optional further learningFree textbooks and trusted online resources
These sources informed the course structure. Use them to revisit a concept or study it in more depth.
Ready check
Tick each idea only when you could explain it without looking back.
Ready for practice? Run LR sweep and error gallery on your Module 3 model.
Extra context (audience, logistics, curriculum notes)
Built for: Builds on Module 3 training experience.
Formats: Experiment logging lab · Error analysis workshop · Quiz · Peer review
Developer track Module 4 — optimisation, evaluation, and error analysis.
Next up
Ready for the next part?
When you've finished the reading, inline exercises, and knowledge check for this part, check the box to continue.