RMSprop, Root Mean Square Propagation, maintains a running average of the squared gradient for each parameter individually, and divides that parameter's update by the square root of this average — effectively giving parameters with consistently large gradients smaller effective steps, and parameters with small gradients comparatively larger ones, similar in spirit to Adam but without Adam's additional momentum-like first-moment tracking. It was originally designed to work well on recurrent neural networks and other settings with noisy, rapidly-changing gradients, and remains a solid, still commonly used alternative to Adam.
1Understanding optimizers.RMSprop()
RMSprop, Root Mean Square Propagation, maintains a running average of the squared gradient for each parameter individually, and divides that parameter's update by the square root of this average — effectively giving parameters with consistently large gradients smaller effective steps, and parameters with small gradients comparatively larger ones, similar in spirit to Adam but without Adam's additional momentum-like first-moment tracking. It was originally designed to work well on recurrent neural networks and other settings with noisy, rapidly-changing gradients, and remains a solid, still commonly used alternative to Adam.
RMSprop is a reasonable alternative to try if Adam's training curves look unusually unstable on a particular model — the two are closely related, both dividing each parameter's step by an estimate of its recent gradient magnitude, but RMSprop lacks Adam's extra momentum-like term.
import tensorflow as tf
optimizer = tf.keras.optimizers.RMSprop(learning_rate=0.001)
print(optimizer.learning_rate.numpy())2Practical Example
Here is a real-world application of optimizers.RMSprop() showing how it is used in production TensorFlow code.
import tensorflow as tf
from tensorflow.keras import layers, Sequential
model = Sequential([layers.Dense(1, input_shape=(1,))])
model.compile(optimizer=tf.keras.optimizers.RMSprop(learning_rate=0.0005), loss='mse')
print(model.optimizer.learning_rate.numpy())3Best Practices
Follow these guidelines when working with optimizers.RMSprop():
1. Try RMSprop as an alternative when a model trained with Adam shows unusually unstable or oscillating loss curves
2. Keep RMSprop's default learning rate of 0.001 as a reasonable starting point, similar to Adam's default
3. Recognize that RMSprop and Adam are closely related, both scaling each parameter's step by a running average of squared gradients, which is why switching between the two rarely dramatically changes results on typical models
Tip: RMSprop is a reasonable alternative to try if Adam's training curves look unusually unstable on a particular model — the two are closely related, both dividing each parameter's step by an estimate of its recent gradient magnitude, but RMSprop lacks Adam's extra momentum-like term.
import tensorflow as tf
optimizer = tf.keras.optimizers.RMSprop(learning_rate=0.001)
print(optimizer.learning_rate.numpy())