Coordinate descent is an optimization method that minimizes an objective function by updating one variable or one block of variables at a time while holding the others fixed.
Intuition
Instead of moving in the full gradient direction, coordinate descent asks a simpler question: “If every other parameter stayed fixed, what is the best update for this one coordinate?” Repeating this across coordinates can solve high-dimensional problems using many small one-dimensional subproblems.
How It Works
- Start with an initial parameter vector.
- Pick one coordinate or block of coordinates.
- Optimize the objective with respect to that coordinate while all other coordinates are fixed.
- Move to another coordinate and repeat until the objective stops improving.
The coordinate choice can be cyclic, randomized, or based on which coordinate currently gives the largest improvement.
When It Is Useful
Coordinate descent is useful when each one-dimensional update is cheap or has a closed-form solution. It is common in sparse linear models, Lasso-style regularization, matrix factorization, and other problems where full-gradient updates are expensive.
In Machine Learning, coordinate descent is most useful when the model has many parameters but the update for one parameter can be computed cheaply.
Failure Modes
- Correlated features can make progress slow because each coordinate update only sees a narrow direction.
- Poor scaling across features can make some coordinates dominate the update path.
- Non-convex objectives can still get stuck in bad local minima.
One-line Interview Answer
Coordinate descent optimizes one parameter or block at a time while holding the rest fixed, which can make large optimization problems easier when each coordinate update is cheap.