Adam, Adaptive Moment Estimation, maintains a running average of both the gradient itself, the first moment, like momentum, and the squared gradient, the second moment, related to its variance, for every parameter individually, using these to automatically scale each parameter's effective learning rate — parameters with consistently large gradients get smaller effective steps, while parameters with small or infrequent gradients get comparatively larger ones. This per-parameter adaptivity is exactly why Adam tends to converge well with little manual tuning, making it the standard default optimizer choice for most deep learning models.
1Understanding optimizers.Adam()
Adam, Adaptive Moment Estimation, maintains a running average of both the gradient itself, the first moment, like momentum, and the squared gradient, the second moment, related to its variance, for every parameter individually, using these to automatically scale each parameter's effective learning rate — parameters with consistently large gradients get smaller effective steps, while parameters with small or infrequent gradients get comparatively larger ones. This per-parameter adaptivity is exactly why Adam tends to converge well with little manual tuning, making it the standard default optimizer choice for most deep learning models.
Adam's default learning_rate of 0.001 works reasonably well as a starting point for most models — before reaching for a different optimizer entirely, try adjusting Adam's learning rate first, since it's usually the single setting with the biggest effect on training behavior.
import tensorflow as tf
optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)
print(optimizer.learning_rate.numpy())2Practical Example
Here is a real-world application of optimizers.Adam() showing how it is used in production TensorFlow code.
import tensorflow as tf
from tensorflow.keras import layers, Sequential
model = Sequential([layers.Dense(1, input_shape=(1,))])
model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=0.01), loss='mse')
print(model.optimizer.learning_rate.numpy())3Best Practices
Follow these guidelines when working with optimizers.Adam():
1. Start with Adam and its default learning rate as the baseline optimizer for a new model, before experimenting with alternatives
2. Reduce the learning rate if training loss oscillates wildly or diverges to nan, a sign the steps are too large for the current loss landscape
3. Pass an actual Adam() object to compile() rather than the string shortcut 'adam' whenever you need a non-default learning rate
Tip: Adam's default learning_rate of 0.001 works reasonably well as a starting point for most models — before reaching for a different optimizer entirely, try adjusting Adam's learning rate first, since it's usually the single setting with the biggest effect on training behavior.
import tensorflow as tf
optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)
print(optimizer.learning_rate.numpy())