# Machine Learning from First Principles

By @omar · 12 weeks

> Twelve weeks for a programmer with high-school math. Understand classical ML by writing regression, trees, boosting, PCA and k-means in NumPy, then ship an honest end-to-end tabular project with a model card.

## About

**For a programmer with high-school math who wants to understand classical machine learning, not just call it.** Twelve weeks, four phases, four shipped projects. Six to eight hours a week: learn early in the week, build late in it.

### What you will have built

- **Week 3:** linear regression solved two ways, with the normal equation and with gradient descent, matching scikit-learn to four decimals.
- **Week 6:** a small model library (linear, ridge, logistic) written in NumPy and benchmarked against scikit-learn with honest cross-validation.
- **Week 9:** a tree, a random forest, gradient boosting, k-NN and an SVM compared on one dataset under identical splits, plus your own tree and boosting code.
- **Week 12:** an end-to-end tabular project with a leakage-proof pipeline, calibrated probabilities, a chosen threshold and a written model card.

### How the weeks work

1. Learn in the first half of the week, build in the second. If you did not type it, you did not learn it.
2. Every algorithm is written from scratch in NumPy before you touch scikit-learn. Scikit-learn then becomes your answer key.
3. Every phase ends in a project week with a deliverable and a definition of done.
4. From week 5 on, you never report a number without a baseline and a spread across folds.
5. Derive one thing by hand per week. The derivations are short and they are what you will still remember in a year.

### Failure modes to watch

- **Lecture treadmill.** Watching a derivation is not deriving it. Pause the video and write the next line first.
- **Library first.** Calling fit() before you can say what it minimises teaches you the API and nothing else.
- **Test-set drift.** Every time you look at the test score and then change something, it stops being a test set.
- **Leaderboard thinking.** A 0.3 point gain inside the fold-to-fold noise is not a gain.

## The math you will use

### Week 1: Vectors, matrices and geometry

Focus: See vectors as arrows, matrices as transformations and the dot product as alignment, then do all of it in NumPy.

Goals:

- Watch Essence of Linear Algebra chapters 1–4, 6, 9 and 14, predicting each transformation first
- Read MML 2.1–2.7 and 3.1–3.4, 3.8 and work two exercises from each chapter
- Implement norm, cosine similarity and projection in NumPy and test them against np.linalg
- Plot the unit square under three 2x2 matrices and mark the eigenvectors
- Write one paragraph on why the dot product measures alignment

Resources:

- [Essence of Linear Algebra](https://www.youtube.com/playlist?list=PLZHQObOWTQDPD3MizzM2xVFitgF8hE_ab) (course, Chapters 1–4, 6, 9, 14)
- [Dot products and duality](https://www.youtube.com/watch?v=LyGKycYT2v0) (video, The geometry behind projection)
- [Eigenvectors and eigenvalues](https://www.youtube.com/watch?v=PFDu9oVAE-g) (video)
- [Mathematics for Machine Learning](https://mml-book.github.io/) (book, Chapter 2 (2.1–2.7), chapter 3 (3.1–3.4, 3.8))
- [NumPy: the absolute basics for beginners](https://numpy.org/doc/stable/user/absolute_beginners.html) (article)
- [CS229 linear algebra review](https://cs229.stanford.edu/section/cs229-linalg.pdf) (article, Skim as a notation key)

### Week 2: Gradients, probability and MLE

Focus: Learn the two tools every model is built from: the gradient as the way downhill, and maximum likelihood as the principle that picks the loss.

Goals:

- Watch the chain rule, gradient descent and maximum likelihood videos and write each rule from memory
- Derive the squared-error gradient and the Bernoulli MLE by hand
- Implement gradient_descent and plot paths for three learning rates on x^2 + 10y^2
- Verify every analytic gradient with a finite-difference check
- Recover a Gaussian's mean and variance by grid search over the log-likelihood

Resources:

- [Visualizing the chain rule and product rule](https://www.youtube.com/watch?v=YG15m2VwSjA) (video)
- [Gradient descent, how neural networks learn](https://www.youtube.com/watch?v=IHZwWFHWa-w) (video, The cost-surface picture)
- [Gradient Descent, Step-by-Step](https://www.youtube.com/watch?v=sDv4f4s2SB8) (video)
- [Maximum Likelihood, clearly explained](https://www.youtube.com/watch?v=XepXtl9YKwc) (video)
- [Mathematics for Machine Learning](https://mml-book.github.io/) (book, Chapter 5 to the chain rule, chapter 6)
- [Seeing Theory: a visual introduction to probability](https://seeing-theory.brown.edu/) (article, Basic probability, Frequentist inference)

### Week 3: Project: linear regression two ways

Focus: Fit a line, then a plane, then a house-price model with the normal equation and with gradient descent, and see why they agree.

Goals:

- Watch Caltech Lecture 3 and derive the normal equation by both routes
- Implement fit_normal with np.linalg.solve and fit_gd with batch gradient descent
- Match scikit-learn's weights to four decimals on California housing
- Plot loss curves for three learning rates, with and without feature scaling
- Write the README: results table and why scaling changes convergence speed

Deliverable: A repo with ols.py (normal equation and gradient descent), tests that compare both solvers with scikit-learn's LinearRegression on California housing, and a one-page README with loss curves at three learning rates, with and without feature scaling, and a test RMSE table. Done when both solvers agree with scikit-learn to four decimals and the README explains why unscaled gradient descent is slow.

Resources:

- [Lecture 3: The Linear Model I](https://www.youtube.com/watch?v=FIbVs5GbBlQ) (video, Linear regression as projection)
- [Linear Regression, Clearly Explained](https://www.youtube.com/watch?v=nk2CQITm_eo) (video)
- [CS229 lecture notes](https://cs229.stanford.edu/main_notes.pdf) (article, Part I: linear regression)
- [An Introduction to Statistical Learning (ISLP)](https://www.statlearning.com/) (book, Chapter 3, sections 3.1–3.2)
- [California housing dataset](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.fetch_california_housing.html) (dataset)
- [scikit-learn: linear models](https://scikit-learn.org/stable/modules/linear_model.html) (tool, Ordinary least squares)

## Fit, then trust it

### Week 4: Logistic regression and cross-entropy

Focus: Turn a line into a probability with the sigmoid, derive the loss from maximum likelihood and discover the gradient looks just like regression's.

Goals:

- Watch Caltech Lecture 9 and the StatQuest maximum-likelihood video
- Derive the gradient X^T (p - y) / n from the negative log-likelihood
- Implement stable logistic regression and plot its decision boundary
- Extend to softmax regression and compare with scikit-learn within 1e-3
- Explain in four sentences why squared error is a poor loss for probabilities

Resources:

- [Lecture 9: The Linear Model II](https://www.youtube.com/watch?v=qSTHZvN8hzs) (video, Logistic regression and gradient descent)
- [Logistic Regression Details Pt 2: Maximum Likelihood](https://www.youtube.com/watch?v=BfKanl1aSG0) (video)
- [Neural Networks Part 6: Cross Entropy](https://www.youtube.com/watch?v=6ArSys5qHAU) (video)
- [Visual Information Theory](https://colah.github.io/posts/2015-09-Visual-Information/) (blog)
- [CS229 lecture notes](https://cs229.stanford.edu/main_notes.pdf) (article, Part II: classification and logistic regression)
- [Machine Learning Crash Course: logistic regression](https://developers.google.com/machine-learning/crash-course/logistic-regression) (course)

### Week 5: Regularization and bias-variance

Focus: Understand why flexible models can fit worse on new data, and use ridge, lasso and validation to find the sweet spot.

Goals:

- Watch Caltech Lectures 8 and 12 and derive the bias-variance decomposition at one point
- Measure bias squared and variance for polynomial degrees 1 to 12 over 2,000 noise draws
- Add ridge to your normal equation and plot validation error against lambda
- Compare ridge and lasso coefficient paths on a correlated feature set
- Write a paragraph on what changes when you double the training set

Resources:

- [Lecture 8: Bias-Variance Tradeoff](https://www.youtube.com/watch?v=zrEyxfl2-a8) (video)
- [Lecture 12: Regularization](https://www.youtube.com/watch?v=I-VfYXzC5ro) (video, First half)
- [Machine Learning Fundamentals: Bias and Variance](https://www.youtube.com/watch?v=EuBBz3bI-aA) (video)
- [Regularization Part 2: Lasso (L1) Regression](https://www.youtube.com/watch?v=NGf0voTMlcs) (video)
- [An Introduction to Statistical Learning (ISLP)](https://www.statlearning.com/) (book, Sections 2.2 and 6.2)
- [MLU-Explain: bias and variance](https://mlu-explain.github.io/bias-variance/) (article)

### Week 6: Project: scratch vs scikit-learn

Focus: Package your models into a small library and check it against scikit-learn with proper cross-validation and a sealed test set.

Goals:

- Record baseline scores (mean predictor, majority class) for both datasets
- Package LinearRegression, Ridge and LogisticRegression with a fit/predict/score API
- Run 5-fold cross-validation for yours and scikit-learn's and tabulate mean and std
- Evaluate each final model once on the sealed test split
- Write the README: how each hyperparameter was chosen and what you would not claim

Deliverable: A repo with your from-scratch models, a comparison script and a results table (baseline, yours, scikit-learn) giving mean and standard deviation over five folds plus one sealed test score. Done when each from-scratch score is within one standard deviation of scikit-learn's and the README says how every hyperparameter was chosen.

Resources:

- [Machine Learning Fundamentals: Cross Validation](https://www.youtube.com/watch?v=fSytzGwwBVw) (video)
- [Lecture 13: Validation](https://www.youtube.com/watch?v=o7zzaKd0Lkk) (video)
- [scikit-learn: cross-validation](https://scikit-learn.org/stable/modules/cross_validation.html) (tool)
- [scikit-learn: common pitfalls and recommended practices](https://scikit-learn.org/stable/common_pitfalls.html) (tool)
- [A Few Useful Things to Know About Machine Learning](https://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf) (paper)
- [Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning](https://arxiv.org/abs/1811.12808) (paper, Skim the cross-validation sections)

## Trees, boosting, kernels

### Week 7: Decision trees and random forests

Focus: Grow a tree by greedy splitting, see why one tree overfits and why averaging many noisy trees fixes it.

Goals:

- Compute Gini and entropy gains by hand for a ten-row table and pick the first split
- Watch the StatQuest trees and forests videos and CS229 Lecture 9
- Implement best_split and a recursive tree with max_depth and min_samples_leaf
- Add bagging and feature subsampling and report out-of-bag error
- Plot validation error against max_depth for one tree and for the forest

Resources:

- [MLU-Explain: decision trees](https://mlu-explain.github.io/decision-tree/) (article)
- [Decision and Classification Trees, Clearly Explained](https://www.youtube.com/watch?v=_L39rN6gz7Y) (video)
- [StatQuest: Random Forests Part 1](https://www.youtube.com/watch?v=J4Wdy0Wc_xQ) (video)
- [Stanford CS229 lecture playlist](https://www.youtube.com/playlist?list=PLoROMvodv4rMiGQp3WXShtMGgzqpfVfbU) (course, Lecture 9: decision trees and ensembles)
- [Random Forests (Breiman, 2001)](https://www.stat.berkeley.edu/~breiman/randomforest2001.pdf) (paper, Skim sections 1–3)
- [scikit-learn: decision trees](https://scikit-learn.org/stable/modules/tree.html) (tool, Mathematical formulation)

### Week 8: Gradient boosting and XGBoost

Focus: Build a model as a sum of small trees, each fit to what the last one got wrong, and learn what XGBoost adds on top.

Goals:

- Hand-run two rounds of depth-1 boosting on six points with learning rate 0.5
- Watch the Gradient Boost and XGBoost videos and read explained.ai
- Implement squared-error gradient boosting using your own tree
- Plot train and validation error per round and locate the early-stopping round
- Compare with XGBoost or HistGradientBoosting and explain the gap

Resources:

- [Gradient Boost Part 1: Regression Main Ideas](https://www.youtube.com/watch?v=3CC4N4z3GJc) (video)
- [Gradient Boost Part 2: Regression Details](https://www.youtube.com/watch?v=2xudPOBz-vs) (video)
- [XGBoost Part 1: Regression](https://www.youtube.com/watch?v=OtD8wVaFm6E) (video)
- [How to explain gradient boosting](https://explained.ai/gradient-boosting/) (article)
- [XGBoost: Introduction to Boosted Trees](https://xgboost.readthedocs.io/en/stable/tutorials/model.html) (tool)
- [XGBoost: A Scalable Tree Boosting System](https://arxiv.org/abs/1603.02754) (paper, Section 2)

### Week 9: Project: SVMs, k-NN and the bake-off

Focus: Meet the two classic non-tree models, margins with kernels and nearest neighbours, then run a fair bake-off of everything you have built.

Goals:

- Watch the MIT SVM lecture and StatQuest's RBF video and sketch the margin by hand
- Implement k-NN and plot accuracy against k with and without scaling
- Tune SVC with linear and RBF kernels over a log grid of C and gamma
- Run the seven-model bake-off with identical 5-fold splits
- Write the report: which differences exceed the noise, and which model you would ship

Deliverable: A bake-off report on one tabular classification dataset: seven models tuned with identical 5-fold splits, a table of mean and standard deviation for accuracy and log-loss, and a paragraph saying which differences are larger than the noise. Done when someone can rerun it with one command and reproduce the table.

Resources:

- [16. Learning: Support Vector Machines](https://www.youtube.com/watch?v=_PwhiWxHK8o) (video)
- [SVM Part 3: The Radial (RBF) Kernel](https://www.youtube.com/watch?v=Qc5IyLW_hns) (video)
- [Lecture 15: Kernel Methods](https://www.youtube.com/watch?v=XUj5JbQihlU) (video)
- [StatQuest: K-nearest neighbors](https://www.youtube.com/watch?v=HVXime0nQeI) (video)
- [scikit-learn: support vector machines](https://scikit-learn.org/stable/modules/svm.html) (tool, Tips on practical use)
- [scikit-learn: nearest neighbors](https://scikit-learn.org/stable/modules/neighbors.html) (tool)

## Structure and honest evaluation

### Week 10: Unsupervised: k-means and PCA

Focus: Find structure without labels: cluster with k-means and compress with PCA, which turns out to be the SVD you met in week 1.

Goals:

- Watch the k-means, PCA and SVD videos and predict k-means outcomes in the visualizer
- Implement k-means with k-means++ seeding and ten restarts
- Implement PCA by eigendecomposition and by SVD and check they agree up to sign
- Plot digits in two components and reconstructions at 5, 10 and 20 components
- State why you must fit PCA on training data only

Resources:

- [Visualizing K-Means Clustering](https://www.naftaliharris.com/blog/visualizing-k-means-clustering/) (article)
- [StatQuest: K-means clustering](https://www.youtube.com/watch?v=4b5d3muPQmA) (video)
- [StatQuest: Principal Component Analysis (PCA), Step-by-Step](https://www.youtube.com/watch?v=FgakZw6K1QQ) (video)
- [Principal Component Analysis, explained visually](https://setosa.io/ev/principal-component-analysis/) (article)
- [Singular Value Decomposition (SVD): Mathematical Overview](https://www.youtube.com/watch?v=nbBvuuNVfco) (video)
- [A Tutorial on Principal Component Analysis](https://arxiv.org/abs/1404.1100) (paper, Through the SVD section)

### Week 11: Honest evaluation and pipelines

Focus: Measure a model the way the real world will: leakage-proof pipelines, the right metric, calibrated probabilities and a chosen threshold.

Goals:

- Read the pitfalls page and ESL 7.10.2 and list five ways a pipeline can leak
- Build a ColumnTransformer and Pipeline and evaluate with stratified CV
- Reproduce a feature-selection leak on pure noise and show the inflated score
- Draw a reliability diagram and calibrate with CalibratedClassifierCV
- Choose a threshold from a written cost matrix using validation data only

Resources:

- [scikit-learn: common pitfalls and recommended practices](https://scikit-learn.org/stable/common_pitfalls.html) (tool, Data leakage)
- [scikit-learn: pipelines and composite estimators](https://scikit-learn.org/stable/modules/compose.html) (tool)
- [The Elements of Statistical Learning](https://hastie.su.domains/ElemStatLearn/) (book, Section 7.10.2: the wrong and right way to do cross-validation)
- [ROC and AUC, Clearly Explained](https://www.youtube.com/watch?v=4jRBRDbJemM) (video)
- [Machine Learning Crash Course: accuracy, precision and recall](https://developers.google.com/machine-learning/crash-course/classification/accuracy-precision-recall) (course)
- [scikit-learn: probability calibration](https://scikit-learn.org/stable/modules/calibration.html) (tool)

### Week 12: Capstone: tabular project + model card

Focus: Take a real tabular problem from raw CSV to a defensible model, then write the model card an honest practitioner would.

Goals:

- Write the problem statement, the metric and the baseline before training anything
- Build the pipeline: baseline, logistic regression, boosted trees, tuned with CV on train only
- Evaluate once on the sealed test set, calibrate and choose a threshold
- Read the 50 worst errors and compute metrics for at least two subgroups
- Write the model card and have one person reproduce the headline number

Deliverable: A public repo with a one-command pipeline and a model card: intended use, data, metrics with confidence intervals, subgroup results, calibration, limitations and failure modes. Done when a stranger can reproduce the headline number within noise and the card says what the model must not be used for.

Resources:

- [Model Cards for Model Reporting](https://arxiv.org/abs/1810.03993) (paper)
- [Hugging Face: model cards](https://huggingface.co/docs/hub/model-cards) (tool)
- [Rules of Machine Learning: best practices for ML engineering](https://developers.google.com/machine-learning/guides/rules-of-ml) (article, Before ML and Phase I)
- [Why do tree-based models still outperform deep learning on tabular data?](https://arxiv.org/abs/2207.08815) (paper)
- [Adult (census income) dataset on OpenML](https://www.openml.org/d/1590) (dataset)
- [Fairlearn](https://fairlearn.org/) (tool, MetricFrame for subgroup metrics)

## Notes

- [Reading a confusion matrix and choosing a threshold](https://ornix.io/u/omar/machine-learning-first-principles/reading-a-confusion-matrix-choosing-a-threshold.md)
- [Data leakage: five ways to fool yourself](https://ornix.io/u/omar/machine-learning-first-principles/data-leakage-five-ways-to-fool-yourself.md)
- [PCA is just an SVD](https://ornix.io/u/omar/machine-learning-first-principles/pca-is-just-an-svd.md)
- [How gradient boosting fits residuals](https://ornix.io/u/omar/machine-learning-first-principles/how-gradient-boosting-fits-residuals.md)
- [How a decision tree chooses a split](https://ornix.io/u/omar/machine-learning-first-principles/how-a-decision-tree-chooses-a-split.md)
- [The bias-variance decomposition with real numbers](https://ornix.io/u/omar/machine-learning-first-principles/bias-variance-with-real-numbers.md)
- [Why cross-entropy and not squared error](https://ornix.io/u/omar/machine-learning-first-principles/why-cross-entropy-not-squared-error.md)
- [Gradient descent, derived and drawn](https://ornix.io/u/omar/machine-learning-first-principles/gradient-descent-derived-and-drawn.md)

Follow this roadmap on Ornix: https://ornix.io/u/omar/machine-learning-first-principles
