Machine Learning from First Principles
Twelve weeks for a programmer with high-school math. Understand classical ML by writing regression, trees, boosting, PCA and k-means in NumPy, then ship an honest end-to-end tabular project with a model card.
Following gives you your own copy, dated from today. Free to start.

About this roadmap
For a programmer with high-school math who wants to understand classical machine learning, not just call it. Twelve weeks, four phases, four shipped projects. Six to eight hours a week: learn early in the week, build late in it.
What you will have built
- Week 3: linear regression solved two ways, with the normal equation and with gradient descent, matching scikit-learn to four decimals.
- Week 6: a small model library (linear, ridge, logistic) written in NumPy and benchmarked against scikit-learn with honest cross-validation.
- Week 9: a tree, a random forest, gradient boosting, k-NN and an SVM compared on one dataset under identical splits, plus your own tree and boosting code.
- Week 12: an end-to-end tabular project with a leakage-proof pipeline, calibrated probabilities, a chosen threshold and a written model card.
How the weeks work
- Learn in the first half of the week, build in the second. If you did not type it, you did not learn it.
- Every algorithm is written from scratch in NumPy before you touch scikit-learn. Scikit-learn then becomes your answer key.
- Every phase ends in a project week with a deliverable and a definition of done.
- From week 5 on, you never report a number without a baseline and a spread across folds.
- Derive one thing by hand per week. The derivations are short and they are what you will still remember in a year.
Failure modes to watch
- Lecture treadmill. Watching a derivation is not deriving it. Pause the video and write the next line first.
- Library first. Calling fit() before you can say what it minimises teaches you the API and nothing else.
- Test-set drift. Every time you look at the test score and then change something, it stops being a test set.
- Leaderboard thinking. A 0.3 point gain inside the fold-to-fold noise is not a gain.
Map
Every week on one canvas. Open a week to see its goals and resources.
What’s inside
12 weeks · 60 goals
Phase 1 · The math you will useWeeks 1–3 · 15 goals
WK 1Vectors, matrices and geometry5 goals
See vectors as arrows, matrices as transformations and the dot product as alignment, then do all of it in NumPy.
Goals
- Watch Essence of Linear Algebra chapters 1–4, 6, 9 and 14, predicting each transformation first
- Read MML 2.1–2.7 and 3.1–3.4, 3.8 and work two exercises from each chapter
- Implement norm, cosine similarity and projection in NumPy and test them against np.linalg
- Plot the unit square under three 2x2 matrices and mark the eigenvectors
- Write one paragraph on why the dot product measures alignment
WK 2Gradients, probability and MLE5 goals
Learn the two tools every model is built from: the gradient as the way downhill, and maximum likelihood as the principle that picks the loss.
Goals
- Watch the chain rule, gradient descent and maximum likelihood videos and write each rule from memory
- Derive the squared-error gradient and the Bernoulli MLE by hand
- Implement gradient_descent and plot paths for three learning rates on x^2 + 10y^2
- Verify every analytic gradient with a finite-difference check
- Recover a Gaussian's mean and variance by grid search over the log-likelihood
WK 3Project: linear regression two ways5 goals
Fit a line, then a plane, then a house-price model with the normal equation and with gradient descent, and see why they agree.
Goals
- Watch Caltech Lecture 3 and derive the normal equation by both routes
- Implement fit_normal with np.linalg.solve and fit_gd with batch gradient descent
- Match scikit-learn's weights to four decimals on California housing
- Plot loss curves for three learning rates, with and without feature scaling
- Write the README: results table and why scaling changes convergence speed
Phase 2 · Fit, then trust itWeeks 4–6 · 15 goals
WK 4Logistic regression and cross-entropy5 goals
Turn a line into a probability with the sigmoid, derive the loss from maximum likelihood and discover the gradient looks just like regression's.
Goals
- Watch Caltech Lecture 9 and the StatQuest maximum-likelihood video
- Derive the gradient X^T (p - y) / n from the negative log-likelihood
- Implement stable logistic regression and plot its decision boundary
- Extend to softmax regression and compare with scikit-learn within 1e-3
- Explain in four sentences why squared error is a poor loss for probabilities
WK 5Regularization and bias-variance5 goals
Understand why flexible models can fit worse on new data, and use ridge, lasso and validation to find the sweet spot.
Goals
- Watch Caltech Lectures 8 and 12 and derive the bias-variance decomposition at one point
- Measure bias squared and variance for polynomial degrees 1 to 12 over 2,000 noise draws
- Add ridge to your normal equation and plot validation error against lambda
- Compare ridge and lasso coefficient paths on a correlated feature set
- Write a paragraph on what changes when you double the training set
WK 6Project: scratch vs scikit-learn5 goals
Package your models into a small library and check it against scikit-learn with proper cross-validation and a sealed test set.
Goals
- Record baseline scores (mean predictor, majority class) for both datasets
- Package LinearRegression, Ridge and LogisticRegression with a fit/predict/score API
- Run 5-fold cross-validation for yours and scikit-learn's and tabulate mean and std
- Evaluate each final model once on the sealed test split
- Write the README: how each hyperparameter was chosen and what you would not claim
Resources
- Machine Learning Fundamentals: Cross Validationvideo
- Lecture 13: Validationvideo
- scikit-learn: cross-validationtool
- scikit-learn: common pitfalls and recommended practicestool
- A Few Useful Things to Know About Machine Learningpaper
- Model Evaluation, Model Selection, and Algorithm Selection in Machine Learningpaper
Phase 3 · Trees, boosting, kernelsWeeks 7–9 · 15 goals
WK 7Decision trees and random forests5 goals
Grow a tree by greedy splitting, see why one tree overfits and why averaging many noisy trees fixes it.
Goals
- Compute Gini and entropy gains by hand for a ten-row table and pick the first split
- Watch the StatQuest trees and forests videos and CS229 Lecture 9
- Implement best_split and a recursive tree with max_depth and min_samples_leaf
- Add bagging and feature subsampling and report out-of-bag error
- Plot validation error against max_depth for one tree and for the forest
WK 8Gradient boosting and XGBoost5 goals
Build a model as a sum of small trees, each fit to what the last one got wrong, and learn what XGBoost adds on top.
Goals
- Hand-run two rounds of depth-1 boosting on six points with learning rate 0.5
- Watch the Gradient Boost and XGBoost videos and read explained.ai
- Implement squared-error gradient boosting using your own tree
- Plot train and validation error per round and locate the early-stopping round
- Compare with XGBoost or HistGradientBoosting and explain the gap
WK 9Project: SVMs, k-NN and the bake-off5 goals
Meet the two classic non-tree models, margins with kernels and nearest neighbours, then run a fair bake-off of everything you have built.
Goals
- Watch the MIT SVM lecture and StatQuest's RBF video and sketch the margin by hand
- Implement k-NN and plot accuracy against k with and without scaling
- Tune SVC with linear and RBF kernels over a log grid of C and gamma
- Run the seven-model bake-off with identical 5-fold splits
- Write the report: which differences exceed the noise, and which model you would ship
Phase 4 · Structure and honest evaluationWeeks 10–12 · 15 goals
WK 10Unsupervised: k-means and PCA5 goals
Find structure without labels: cluster with k-means and compress with PCA, which turns out to be the SVD you met in week 1.
Goals
- Watch the k-means, PCA and SVD videos and predict k-means outcomes in the visualizer
- Implement k-means with k-means++ seeding and ten restarts
- Implement PCA by eigendecomposition and by SVD and check they agree up to sign
- Plot digits in two components and reconstructions at 5, 10 and 20 components
- State why you must fit PCA on training data only
Resources
WK 11Honest evaluation and pipelines5 goals
Measure a model the way the real world will: leakage-proof pipelines, the right metric, calibrated probabilities and a chosen threshold.
Goals
- Read the pitfalls page and ESL 7.10.2 and list five ways a pipeline can leak
- Build a ColumnTransformer and Pipeline and evaluate with stratified CV
- Reproduce a feature-selection leak on pure noise and show the inflated score
- Draw a reliability diagram and calibrate with CalibratedClassifierCV
- Choose a threshold from a written cost matrix using validation data only
Resources
WK 12Capstone: tabular project + model card5 goals
Take a real tabular problem from raw CSV to a defensible model, then write the model card an honest practitioner would.
Goals
- Write the problem statement, the metric and the baseline before training anything
- Build the pipeline: baseline, logistic regression, boosted trees, tuned with CV on train only
- Evaluate once on the sealed test set, calibrate and choose a threshold
- Read the 50 worst errors and compute metrics for at least two subgroups
- Write the model card and have one person reproduce the headline number
Notes
8 published notes
Data leakage: five ways to fool yourself
Nov 2, 2026- Nov 2, 2026
How gradient boosting fits residuals
Nov 2, 2026How a decision tree chooses a split
Nov 2, 2026Why cross-entropy and not squared error
Nov 2, 2026Gradient descent, derived and drawn
Nov 2, 2026