@omarUpdated Nov 3, 2026

Machine Learning from First Principles

Twelve weeks for a programmer with high-school math. Understand classical ML by writing regression, trees, boosting, PCA and k-means in NumPy, then ship an honest end-to-end tabular project with a model card.

Following gives you your own copy, dated from today. Free to start.

4 phases · 12 weeks · 1 follower

About this roadmap

For a programmer with high-school math who wants to understand classical machine learning, not just call it. Twelve weeks, four phases, four shipped projects. Six to eight hours a week: learn early in the week, build late in it.

What you will have built

  • Week 3: linear regression solved two ways, with the normal equation and with gradient descent, matching scikit-learn to four decimals.
  • Week 6: a small model library (linear, ridge, logistic) written in NumPy and benchmarked against scikit-learn with honest cross-validation.
  • Week 9: a tree, a random forest, gradient boosting, k-NN and an SVM compared on one dataset under identical splits, plus your own tree and boosting code.
  • Week 12: an end-to-end tabular project with a leakage-proof pipeline, calibrated probabilities, a chosen threshold and a written model card.

How the weeks work

  1. Learn in the first half of the week, build in the second. If you did not type it, you did not learn it.
  2. Every algorithm is written from scratch in NumPy before you touch scikit-learn. Scikit-learn then becomes your answer key.
  3. Every phase ends in a project week with a deliverable and a definition of done.
  4. From week 5 on, you never report a number without a baseline and a spread across folds.
  5. Derive one thing by hand per week. The derivations are short and they are what you will still remember in a year.

Failure modes to watch

  • Lecture treadmill. Watching a derivation is not deriving it. Pause the video and write the next line first.
  • Library first. Calling fit() before you can say what it minimises teaches you the API and nothing else.
  • Test-set drift. Every time you look at the test score and then change something, it stops being a test set.
  • Leaderboard thinking. A 0.3 point gain inside the fold-to-fold noise is not a gain.

Map

Every week on one canvas. Open a week to see its goals and resources.

What’s inside

12 weeks · 60 goals

Phase 1 · The math you will useWeeks 1–3 · 15 goals
WK 1Vectors, matrices and geometry5 goals

See vectors as arrows, matrices as transformations and the dot product as alignment, then do all of it in NumPy.

Goals

  • Watch Essence of Linear Algebra chapters 1–4, 6, 9 and 14, predicting each transformation first
  • Read MML 2.1–2.7 and 3.1–3.4, 3.8 and work two exercises from each chapter
  • Implement norm, cosine similarity and projection in NumPy and test them against np.linalg
  • Plot the unit square under three 2x2 matrices and mark the eigenvectors
  • Write one paragraph on why the dot product measures alignment
WK 2Gradients, probability and MLE5 goals

Learn the two tools every model is built from: the gradient as the way downhill, and maximum likelihood as the principle that picks the loss.

Goals

  • Watch the chain rule, gradient descent and maximum likelihood videos and write each rule from memory
  • Derive the squared-error gradient and the Bernoulli MLE by hand
  • Implement gradient_descent and plot paths for three learning rates on x^2 + 10y^2
  • Verify every analytic gradient with a finite-difference check
  • Recover a Gaussian's mean and variance by grid search over the log-likelihood
WK 3Project: linear regression two ways5 goals

Fit a line, then a plane, then a house-price model with the normal equation and with gradient descent, and see why they agree.

Goals

  • Watch Caltech Lecture 3 and derive the normal equation by both routes
  • Implement fit_normal with np.linalg.solve and fit_gd with batch gradient descent
  • Match scikit-learn's weights to four decimals on California housing
  • Plot loss curves for three learning rates, with and without feature scaling
  • Write the README: results table and why scaling changes convergence speed
Phase 2 · Fit, then trust itWeeks 4–6 · 15 goals
WK 4Logistic regression and cross-entropy5 goals

Turn a line into a probability with the sigmoid, derive the loss from maximum likelihood and discover the gradient looks just like regression's.

Goals

  • Watch Caltech Lecture 9 and the StatQuest maximum-likelihood video
  • Derive the gradient X^T (p - y) / n from the negative log-likelihood
  • Implement stable logistic regression and plot its decision boundary
  • Extend to softmax regression and compare with scikit-learn within 1e-3
  • Explain in four sentences why squared error is a poor loss for probabilities
WK 5Regularization and bias-variance5 goals

Understand why flexible models can fit worse on new data, and use ridge, lasso and validation to find the sweet spot.

Goals

  • Watch Caltech Lectures 8 and 12 and derive the bias-variance decomposition at one point
  • Measure bias squared and variance for polynomial degrees 1 to 12 over 2,000 noise draws
  • Add ridge to your normal equation and plot validation error against lambda
  • Compare ridge and lasso coefficient paths on a correlated feature set
  • Write a paragraph on what changes when you double the training set
WK 6Project: scratch vs scikit-learn5 goals

Package your models into a small library and check it against scikit-learn with proper cross-validation and a sealed test set.

Goals

  • Record baseline scores (mean predictor, majority class) for both datasets
  • Package LinearRegression, Ridge and LogisticRegression with a fit/predict/score API
  • Run 5-fold cross-validation for yours and scikit-learn's and tabulate mean and std
  • Evaluate each final model once on the sealed test split
  • Write the README: how each hyperparameter was chosen and what you would not claim
Phase 3 · Trees, boosting, kernelsWeeks 7–9 · 15 goals
WK 7Decision trees and random forests5 goals

Grow a tree by greedy splitting, see why one tree overfits and why averaging many noisy trees fixes it.

Goals

  • Compute Gini and entropy gains by hand for a ten-row table and pick the first split
  • Watch the StatQuest trees and forests videos and CS229 Lecture 9
  • Implement best_split and a recursive tree with max_depth and min_samples_leaf
  • Add bagging and feature subsampling and report out-of-bag error
  • Plot validation error against max_depth for one tree and for the forest
WK 8Gradient boosting and XGBoost5 goals

Build a model as a sum of small trees, each fit to what the last one got wrong, and learn what XGBoost adds on top.

Goals

  • Hand-run two rounds of depth-1 boosting on six points with learning rate 0.5
  • Watch the Gradient Boost and XGBoost videos and read explained.ai
  • Implement squared-error gradient boosting using your own tree
  • Plot train and validation error per round and locate the early-stopping round
  • Compare with XGBoost or HistGradientBoosting and explain the gap
WK 9Project: SVMs, k-NN and the bake-off5 goals

Meet the two classic non-tree models, margins with kernels and nearest neighbours, then run a fair bake-off of everything you have built.

Goals

  • Watch the MIT SVM lecture and StatQuest's RBF video and sketch the margin by hand
  • Implement k-NN and plot accuracy against k with and without scaling
  • Tune SVC with linear and RBF kernels over a log grid of C and gamma
  • Run the seven-model bake-off with identical 5-fold splits
  • Write the report: which differences exceed the noise, and which model you would ship
Phase 4 · Structure and honest evaluationWeeks 10–12 · 15 goals
WK 10Unsupervised: k-means and PCA5 goals

Find structure without labels: cluster with k-means and compress with PCA, which turns out to be the SVD you met in week 1.

Goals

  • Watch the k-means, PCA and SVD videos and predict k-means outcomes in the visualizer
  • Implement k-means with k-means++ seeding and ten restarts
  • Implement PCA by eigendecomposition and by SVD and check they agree up to sign
  • Plot digits in two components and reconstructions at 5, 10 and 20 components
  • State why you must fit PCA on training data only
WK 11Honest evaluation and pipelines5 goals

Measure a model the way the real world will: leakage-proof pipelines, the right metric, calibrated probabilities and a chosen threshold.

Goals

  • Read the pitfalls page and ESL 7.10.2 and list five ways a pipeline can leak
  • Build a ColumnTransformer and Pipeline and evaluate with stratified CV
  • Reproduce a feature-selection leak on pure noise and show the inflated score
  • Draw a reliability diagram and calibrate with CalibratedClassifierCV
  • Choose a threshold from a written cost matrix using validation data only
WK 12Capstone: tabular project + model card5 goals

Take a real tabular problem from raw CSV to a defensible model, then write the model card an honest practitioner would.

Goals

  • Write the problem statement, the metric and the baseline before training anything
  • Build the pipeline: baseline, logistic regression, boosted trees, tuned with CV on train only
  • Evaluate once on the sealed test set, calibrate and choose a threshold
  • Read the 50 worst errors and compute metrics for at least two subgroups
  • Write the model card and have one person reproduce the headline number