During training, a Dropout layer randomly zeroes out each input unit independently with probability rate, forcing the network to not rely too heavily on any single unit, since it might be dropped on any given training step, which encourages more robust, redundant feature representations. Critically, Dropout only behaves this way during training — during inference, evaluate() or predict(), it passes every value through unchanged, since introducing randomness into a model's actual predictions wouldn't make sense.
1Understanding tf.keras.layers.Dropout()
During training, a Dropout layer randomly zeroes out each input unit independently with probability rate, forcing the network to not rely too heavily on any single unit, since it might be dropped on any given training step, which encourages more robust, redundant feature representations. Critically, Dropout only behaves this way during training — during inference, evaluate() or predict(), it passes every value through unchanged, since introducing randomness into a model's actual predictions wouldn't make sense.
Dropout is automatically disabled during evaluate() and predict(), passing all values through unchanged — it only randomly zeroes units while actively training, so you never need to manually turn it off for inference.
import tensorflow as tf
from tensorflow.keras import layers
layer = layers.Dropout(0.5)
x = tf.ones([1, 10])
output = layer(x, training=True)
print(tf.reduce_sum(output).numpy() != 10.0)2Practical Example
Here is a real-world application of tf.keras.layers.Dropout() showing how it is used in production TensorFlow code.
import tensorflow as tf
from tensorflow.keras import layers
layer = layers.Dropout(0.5)
x = tf.ones([1, 10])
output = layer(x, training=False)
print(output.numpy())3Best Practices
Follow these guidelines when working with tf.keras.layers.Dropout():
1. Place Dropout layers after Dense (or other trainable) layers where overfitting is a concern, typically with a rate between 0.2 and 0.5
2. Trust that Dropout is automatically inactive during evaluate()/predict(), rather than trying to manually disable it for inference
3. Increase the dropout rate, or add more Dropout layers, if a model's validation loss is noticeably worse than its training loss, a sign of overfitting
Tip: Dropout is automatically disabled during evaluate() and predict(), passing all values through unchanged — it only randomly zeroes units while actively training, so you never need to manually turn it off for inference.
import tensorflow as tf
from tensorflow.keras import layers
layer = layers.Dropout(0.5)
x = tf.ones([1, 10])
output = layer(x, training=True)
print(tf.reduce_sum(output).numpy() != 10.0)