Skip to content
All apps

Gradient Descent (3D)

About this app

Loss surface of a real line fit in 3D: θ ← θ − η∇L with stability limit 2/λmax, two valleys and a saddle, GD, momentum and Adam racing, SGD noise and the fitted line in the model panel.

How gradient descent works

Training a model means minimising a loss L(θ) that measures how badly the parameters θ fit the data. Gradient descent does this in small steps against the gradient: θ ← θ − η∇L. The learning rate η sets the step length. Too small and progress crawls; too large and every step overshoots the valley. On a quadratic surface, plain gradient descent stays stable only while η < 2/λmax, where λmax is the largest curvature. Local minima and saddle points also have zero gradient and can stall the method.

What the app shows

You can choose between three loss landscapes: a line fit with L(w, b) = 1/(2n) · Σ (w·xᵢ + b − yᵢ)², a surface with two valleys, and a saddle. A ball rolls downhill step by step, with an arrow showing the direction and length of each step. The readouts give loss, step count, gradient norm ‖∇L‖ and distance to the minimum, and a training curve plots the loss per step. For the line fit, the line in the model panel turns and shifts until it matches the least-squares solution.

Things to try

Move the learning rate from tiny to too large and watch the ball crawl, oscillate or fly out of the landscape; the app states the stability limit for GD. Race GD, momentum and Adam against each other. In the two-valley landscape start from the trap preset, then switch to high-dimensional mode: with a third weight direction z the local minimum turns into a saddle and the ball escapes. At the saddle, gradient noise (SGD) or a start just off the ridge helps.

Frequently asked questions

What is gradient descent?
Gradient descent is an optimisation method that minimises a function by repeatedly stepping against its gradient: θ ← θ − η∇L. The gradient points in the direction of steepest ascent, so its opposite leads downhill. Neural networks are almost always trained with variants of this method.
What happens if the learning rate is too high?
With a learning rate that is too high, every step jumps past the minimum. At first the parameters oscillate; beyond a stability limit the jumps grow, the loss rises and training diverges. For a quadratic loss surface this limit is η = 2/λmax.
What is the difference between momentum and Adam?
Momentum accumulates past gradients in a velocity v and carries that speed forward, which helps it cross flat valleys faster. Adam keeps running averages of the gradient and the squared gradient and divides each direction by its typical gradient size. As a result, Adam’s steps are almost independent of how steep the slope is.
What can I try in this app?
You pick a landscape, optimiser, learning rate and starting point and watch the ball roll across the 3D surface live. You can race all three optimisers, switch on SGD noise and see in high-dimensional mode how a local minimum becomes a saddle.

Subject: Neural networks

More from Neural networks

A VisuApp by heyprof: interactive, free in your browser, no sign-up. What is a VisuApp? · All apps