What is stochastic gradient descent (SGD)?

Stochastic Gradient Descent (SGD) is an optimization algorithm commonly used to train machine learning models, particularly in scenarios involving large datasets. Unlike standard gradient descent, which computes the gradient of the loss function using the entire dataset, SGD updates model parameters using a single randomly selected data point (or a small subset of data) at each iteration. This approach reduces computational overhead, making it feasible to handle datasets that are too large to process in one batch. The core idea is to approximate the true gradient of the loss function—which would require all data—using a noisy estimate from a subset, allowing the model to make incremental progress toward minimizing loss.

The process works as follows: At each training step, SGD randomly selects one data point (or a mini-batch), computes the gradient of the loss for that sample, and updates the model’s parameters in the opposite direction of the gradient, scaled by a learning rate. For example, in linear regression, the loss might be the squared error between predicted and actual values. Instead of calculating the average gradient over all training examples, SGD computes the gradient for just one example and adjusts the weights immediately. This frequent updating introduces noise, but it also enables faster iterations and the ability to escape shallow local minima. The learning rate is a critical hyperparameter—too high, and updates overshoot optimal values; too low, and training becomes sluggish.

SGD’s key advantage is efficiency, especially with large datasets. Processing one sample at a time avoids the memory and computation bottlenecks of batch methods. However, the noise in gradient estimates can cause unstable convergence, requiring careful tuning of the learning rate or techniques like learning rate schedules (e.g., gradually reducing the rate over time). Developers often use variants like mini-batch SGD, which balances noise and efficiency by processing small groups of samples, or momentum-based SGD, which smooths parameter updates. In practice, SGD is widely used in training neural networks, where datasets are massive, and its noisy updates can even help prevent overfitting. While it may require more iterations to converge compared to batch methods, the per-iteration speed makes it a practical choice for many real-world applications.