Loss Spikes in Gradient Descent
Loss spikes aren’t noise. They’re gradient descent briefly exceeding the edge of stability and snapping back. Here’s why.
- Optimization 6
- Numerical methods 4
- Iterative methods 3
- Python 3
- Interpolation 3
- Approximation theory 3
- Fixed-point arithmetic 3
- Numerical computing 3
- Least squares 2
- Regression 2
- Visualization 2
- Matplotlib 2
- Constrained optimization 2
- Machine Learning 2
- Machine learning 1
- Decision trees 1
- Linear algebra 1
- Duality 1
- Numerical integration 1
- Quadrature 1
- Rational approximation 1
- Trigonometry 1
- Finite differences 1
- Statistics 1
- Correlation 1
- Nonparametric methods 1
- Floating point 1
- Gradient Descent 1
Optimization
Robust Regression Without Gradients
L1 regression is more robust to outliers than least squares, but harder to solve. We walk through four algorithms, each addressing a limitation of the previo...
Golden Section Search for Robust Regression
Golden section search reuses objective evaluations to efficiently minimize 1D functions. Learn how this classical algorithm connects to the golden ratio and ...
Lagrange Duality
Derive and interpret the dual form of an optimization problem.
Quadratic Penalty Algo
Solve constrained optimization problems using your favorite unconstrained solver.
Nonlinear Least Squares
Fit nonlinear models using Gauss-Newton and Levenberg-Marquardt algorithms.
Numerical methods
Fixed Point vs Floating Point: A Numerical Representation Deep Dive
A ground-up look at how floating-point and fixed-point numbers are represented and how arithmetic works for each
Newton-Gregory Interpolation
Interpolate equally-spaced data efficiently and discover its connection to Taylor series.
Tanhsinh Quadrature
Tackle tricky integrals with endpoint singularities using a clever variable transformation.
Lagrange Interpolation
Approximate functions using polynomial interpolation without solving linear systems.
Iterative methods
BKM
Compute logarithms and exponentials without a floating point unit.
CORDIC
Compute sine, cosine, and exponentials using only addition, subtraction, and bit shifts.
Nonlinear Least Squares
Fit nonlinear models using Gauss-Newton and Levenberg-Marquardt algorithms.
Python
Plotting Ellipses
Plot ellipses using conic, quadratic, and parametric representations.
Numpy Regression Trees
Implement a regression tree from scratch using only numpy.
3D Plotting using Non-Rectangular Grids
Create cleaner 3D surface plots using radial and elliptical grids in matplotlib.
Interpolation
Newton-Gregory Interpolation
Interpolate equally-spaced data efficiently and discover its connection to Taylor series.
The AAA Algorithm
Fit rational functions to data with poles and discontinuities where polynomials fail.
Lagrange Interpolation
Approximate functions using polynomial interpolation without solving linear systems.
Approximation theory
Newton-Gregory Interpolation
Interpolate equally-spaced data efficiently and discover its connection to Taylor series.
The AAA Algorithm
Fit rational functions to data with poles and discontinuities where polynomials fail.
Lagrange Interpolation
Approximate functions using polynomial interpolation without solving linear systems.
Fixed-point arithmetic
Fixed Point vs Floating Point: A Numerical Representation Deep Dive
A ground-up look at how floating-point and fixed-point numbers are represented and how arithmetic works for each
BKM
Compute logarithms and exponentials without a floating point unit.
CORDIC
Compute sine, cosine, and exponentials using only addition, subtraction, and bit shifts.
Numerical computing
Fixed Point vs Floating Point: A Numerical Representation Deep Dive
A ground-up look at how floating-point and fixed-point numbers are represented and how arithmetic works for each
BKM
Compute logarithms and exponentials without a floating point unit.
CORDIC
Compute sine, cosine, and exponentials using only addition, subtraction, and bit shifts.
Least squares
The AAA Algorithm
Fit rational functions to data with poles and discontinuities where polynomials fail.
Nonlinear Least Squares
Fit nonlinear models using Gauss-Newton and Levenberg-Marquardt algorithms.
Regression
Robust Regression Without Gradients
L1 regression is more robust to outliers than least squares, but harder to solve. We walk through four algorithms, each addressing a limitation of the previo...
Nonlinear Least Squares
Fit nonlinear models using Gauss-Newton and Levenberg-Marquardt algorithms.
Visualization
Plotting Ellipses
Plot ellipses using conic, quadratic, and parametric representations.
3D Plotting using Non-Rectangular Grids
Create cleaner 3D surface plots using radial and elliptical grids in matplotlib.
Matplotlib
Plotting Ellipses
Plot ellipses using conic, quadratic, and parametric representations.
3D Plotting using Non-Rectangular Grids
Create cleaner 3D surface plots using radial and elliptical grids in matplotlib.
Constrained optimization
Lagrange Duality
Derive and interpret the dual form of an optimization problem.
Quadratic Penalty Algo
Solve constrained optimization problems using your favorite unconstrained solver.
Machine Learning
Loss Spikes in Gradient Descent
Loss spikes aren’t noise. They’re gradient descent briefly exceeding the edge of stability and snapping back. Here’s why.
Robust Regression Without Gradients
L1 regression is more robust to outliers than least squares, but harder to solve. We walk through four algorithms, each addressing a limitation of the previo...
Machine learning
Numpy Regression Trees
Implement a regression tree from scratch using only numpy.
Decision trees
Numpy Regression Trees
Implement a regression tree from scratch using only numpy.
Linear algebra
Plotting Ellipses
Plot ellipses using conic, quadratic, and parametric representations.
Duality
Lagrange Duality
Derive and interpret the dual form of an optimization problem.
Numerical integration
Tanhsinh Quadrature
Tackle tricky integrals with endpoint singularities using a clever variable transformation.
Quadrature
Tanhsinh Quadrature
Tackle tricky integrals with endpoint singularities using a clever variable transformation.
Rational approximation
The AAA Algorithm
Fit rational functions to data with poles and discontinuities where polynomials fail.
Trigonometry
CORDIC
Compute sine, cosine, and exponentials using only addition, subtraction, and bit shifts.
Finite differences
Newton-Gregory Interpolation
Interpolate equally-spaced data efficiently and discover its connection to Taylor series.
Statistics
Chatterjee’s Xi Coefficient
Detect nonlinear relationships that Pearson and Spearman miss.
Correlation
Chatterjee’s Xi Coefficient
Detect nonlinear relationships that Pearson and Spearman miss.
Nonparametric methods
Chatterjee’s Xi Coefficient
Detect nonlinear relationships that Pearson and Spearman miss.
Floating point
Fixed Point vs Floating Point: A Numerical Representation Deep Dive
A ground-up look at how floating-point and fixed-point numbers are represented and how arithmetic works for each
Gradient Descent
Loss Spikes in Gradient Descent
Loss spikes aren’t noise. They’re gradient descent briefly exceeding the edge of stability and snapping back. Here’s why.