Mike's
Archive
Deep notes on optimization, machine learning, and the mathematics behind modern AI. Seven algorithms, rigorously dissected.
Why subscribe
Paper-level rigor
Every note includes full mathematical derivations with KaTeX rendering. Convergence proofs, regret bounds, and eigenvalue analysis — presented as clearly as in the original papers.
Seven algorithms
From AdaGrad to SOAP. Each optimizer is examined through its preconditioner structure, update rule, and empirical behavior. Not summaries — deep technical walkthroughs.
One subscription
Full access to every article. New notes added as research evolves. Cancel anytime — keep access until the end of your billing period.
SOAP: Improving and Stabilizing Shampoo Using Adam
SOAP runs AdamW in the eigenbasis provided by Shampoo, outperforming both on language model pre-training.
Shampoo: Preconditioned Stochastic Tensor Optimization
An efficient Kronecker-product approximation of the full AdaGrad preconditioner for matrix-structured parameters.
PremiumSophia: A Scalable Stochastic Second-order Optimizer
A lightweight diagonal Hessian estimator with per-coordinate clipping, achieving 2x speedup over Adam.
PremiumOptimizing Neural Networks with Kronecker-factored Approximate Curvature
A block-wise Kronecker-factored approximation to the Fisher information matrix for efficient natural gradient descent.
PremiumAdaptive Subgradient Methods for Online Learning and Stochastic Optimization
The ADAGRAD algorithm that adapts learning rates per-feature based on historical gradient information.
PremiumStochastic Subgradient Method Converges on Tame Functions
Proves that the stochastic subgradient method converges on Whitney stratifiable and definable functions, including deep networks.
PremiumOptimization for Machine Learning
Gradient descent, accelerated methods, stochastic gradient, and adaptive methods (AdaGrad, RMSProp, Adam) with convergence guarantees.
PremiumMembership
Full access from $10/month
Unlock every article, including SOAP, Shampoo, Sophia, and more. Cancel anytime.
View plans