Inside a GradientTape's with block, every operation performed on a tracked tensor is recorded onto the tape, building up a record of the computation; afterward, calling tape.gradient(target, sources) uses that recorded computation to compute the derivative of target with respect to each tensor in sources, via automatic differentiation. Variables created with tf.Variable() are tracked automatically, while a plain tf.constant() tensor needs tape.watch() called on it explicitly if you need a gradient with respect to it. GradientTape is the fundamental building block underlying model.fit()'s automatic training loop, used directly whenever you need a custom training loop with full manual control.

1Understanding tf.GradientTape()

💡

tf.Variable() tensors are watched automatically by GradientTape, but a plain tf.constant() is not — call tape.watch(my_constant) explicitly inside the tape's context if you need a gradient with respect to a constant tensor.

editor.html

import tensorflow as tf

x = tf.Variable(3.0)
with tf.GradientTape() as tape:
    y = x ** 2
dy_dx = tape.gradient(y, x)
print(dy_dx.numpy())

localhost:3000

2Practical Example

Here is a real-world application of tf.GradientTape() showing how it is used in production TensorFlow code.

editor.html

import tensorflow as tf

x = tf.constant(3.0)
with tf.GradientTape() as tape:
    tape.watch(x)
    y = x ** 3
dy_dx = tape.gradient(y, x)
print(dy_dx.numpy())

localhost:3000

3Best Practices

Follow these guidelines when working with tf.GradientTape():

1. Use GradientTape directly when writing a custom training loop that needs full manual control over the forward pass, loss computation, and gradient application

2. Remember tf.Variable() tensors are watched automatically, while tf.constant() tensors need an explicit tape.watch() call to compute gradients with respect to them

3. Keep the with block as short as possible, containing only the forward pass and loss computation, since everything inside it is recorded and consumes extra memory

⚠️

Tip: tf.Variable() tensors are watched automatically by GradientTape, but a plain tf.constant() is not — call tape.watch(my_constant) explicitly inside the tape's context if you need a gradient with respect to a constant tensor.

editor.html

import tensorflow as tf

x = tf.Variable(3.0)
with tf.GradientTape() as tape:
    y = x ** 2
dy_dx = tape.gradient(y, x)
print(dy_dx.numpy())

localhost:3000

Examples

Example 01Basic Usage

import tensorflow as tf

x = tf.Variable(3.0)
with tf.GradientTape() as tape:
    y = x ** 2
dy_dx = tape.gradient(y, x)
print(dy_dx.numpy())

Example 02Advanced Example

import tensorflow as tf

x = tf.constant(3.0)
with tf.GradientTape() as tape:
    tape.watch(x)
    y = x ** 3
dy_dx = tape.gradient(y, x)
print(dy_dx.numpy())

Best Practices

Use GradientTape directly when writing a custom training loop that needs full manual control over the forward pass, loss computation, and gradient application
Remember tf.Variable() tensors are watched automatically, while tf.constant() tensors need an explicit tape.watch() call to compute gradients with respect to them
Keep the with block as short as possible, containing only the forward pass and loss computation, since everything inside it is recorded and consumes extra memory

Interview Question

Why does tf.GradientTape() automatically track tf.Variable() tensors but require an explicit tape.watch() call for a plain tf.constant()?▼

Hint: Think about what each type of tensor is typically used for, and whether you'd usually want a gradient with respect to it.

tf.Variable() is specifically designed to represent trainable, mutable state, most commonly a model's weights and biases, values you virtually always want a gradient with respect to during training, so automatically tracking every Variable used inside the tape's context is the sensible, convenient default. tf.constant(), by contrast, represents a fixed, immutable value, most commonly input data or a hyperparameter, something you typically don't need a gradient with respect to at all, so automatically tracking every constant tensor used inside a tape would waste memory recording unnecessary computation history for values that will practically never need a gradient. Requiring an explicit tape.watch() call for a constant makes tracking it an intentional, deliberate choice for the comparatively rare cases where you genuinely do need a gradient with respect to a fixed input, rather than silently tracking, and remembering, every constant by default.

Exercises

MediumPractice using tf.GradientTape() in a real scenario.

View Solution

import tensorflow as tf

x = tf.Variable(3.0)
with tf.GradientTape() as tape:
    y = x ** 2
dy_dx = tape.gradient(y, x)
print(dy_dx.numpy())

Frequently Asked Questions

Why does tf.GradientTape() automatically track tf.Variable() tensors but require an explicit tape.watch() call for a plain tf.constant()?▼

Related Functions

tape-gradientoptimizer-apply-gradientsmodel-fit