Stochastic Gradient Descent is the most basic optimization algorithm, applying the same fixed-size step, learning_rate multiplied by the gradient, to every parameter on every update, unlike adaptive optimizers such as Adam which scale each parameter's step individually. Setting a nonzero momentum accumulates a running average of past gradients, letting updates build up speed in a consistent direction and helping push through small local irregularities in the loss landscape — this closely mirrors the physical intuition of a ball rolling downhill and gathering momentum, rather than responding only to the immediate, current slope.
1Understanding optimizers.SGD()
Stochastic Gradient Descent is the most basic optimization algorithm, applying the same fixed-size step, learning_rate multiplied by the gradient, to every parameter on every update, unlike adaptive optimizers such as Adam which scale each parameter's step individually. Setting a nonzero momentum accumulates a running average of past gradients, letting updates build up speed in a consistent direction and helping push through small local irregularities in the loss landscape — this closely mirrors the physical intuition of a ball rolling downhill and gathering momentum, rather than responding only to the immediate, current slope.
Plain SGD with no momentum can converge quite slowly and get stuck oscillating in narrow valleys of the loss landscape — adding a moderate momentum value, commonly 0.9, is a simple, well-established way to meaningfully speed up and stabilize training.
import tensorflow as tf
optimizer = tf.keras.optimizers.SGD(learning_rate=0.01)
print(optimizer.learning_rate.numpy())2Practical Example
Here is a real-world application of optimizers.SGD() showing how it is used in production TensorFlow code.
import tensorflow as tf
from tensorflow.keras import layers, Sequential
model = Sequential([layers.Dense(1, input_shape=(1,))])
model.compile(optimizer=tf.keras.optimizers.SGD(learning_rate=0.01, momentum=0.9), loss='mse')
print(model.optimizer.momentum)3Best Practices
Follow these guidelines when working with optimizers.SGD():
1. Add momentum, commonly around 0.9, when using SGD, since plain SGD with momentum=0 typically converges much more slowly than momentum-based or adaptive alternatives
2. Expect to need more manual learning-rate tuning with SGD than with an adaptive optimizer like Adam, since it lacks any per-parameter adjustment
3. Consider SGD with momentum for the final fine-tuning stages of a model, since some research suggests it can reach slightly better final generalization than Adam in certain settings, despite typically converging more slowly
Tip: Plain SGD with no momentum can converge quite slowly and get stuck oscillating in narrow valleys of the loss landscape — adding a moderate momentum value, commonly 0.9, is a simple, well-established way to meaningfully speed up and stabilize training.
import tensorflow as tf
optimizer = tf.keras.optimizers.SGD(learning_rate=0.01)
print(optimizer.learning_rate.numpy())