Stanford CS221 | Autumn 2025 | Lecture 4: Learning III

Why It Matters

Mastering PyTorch’s automatic differentiation and optimizer workflow enables faster, more reliable model development and prevents common training bugs, directly impacting productivity in industry‑scale machine‑learning projects.

Key Takeaways

• Transition from manual computation graphs to PyTorch simplifies deep learning.
• .requires_grad flag determines which tensors track gradients during training.
• Detaching tensors cuts gradient flow, useful for inference.
• Zeroing gradients each step prevents unwanted accumulation in PyTorch.
• Optimizer.step applies computed gradients to model parameters efficiently.

Summary

The lecture introduces deep learning fundamentals while guiding students from hand‑crafted computation graphs to the PyTorch ecosystem. After reviewing linear models, the professor emphasizes that modern frameworks like PyTorch and JAX handle forward evaluation, automatic differentiation, and graph management far more efficiently than custom implementations. Key concepts covered include the role of the requires_grad attribute for selecting parameters that need gradients, the use of tensor.detach() to sever gradient flow, and the torch.no_grad() context for inference‑only passes. The instructor also contrasts eager execution with symbolic graph construction, explaining why PyTorch evaluates operations immediately yet still retains a transparent graph for backpropagation. Illustrative code snippets walk through building a simple multilayer model, computing cross‑entropy loss, calling loss.backward(), and updating weights with an optimizer such as SGD or Adam. The example highlights the necessity of zeroing gradients each iteration to avoid unintended accumulation and shows how optimizer.step() applies the computed gradients to model parameters. Overall, the session equips students with practical PyTorch skills, reinforcing the standard training loop—forward pass, loss computation, backward pass, gradient reset, and parameter update—while stressing the importance of managing gradient flow for both training and deployment scenarios.

Original Description

For more information about Stanford’s Artificial Intelligence professional and graduate programs, visit: https://stanford.io/ai

To learn more about enrolling in this course, visit: https://online.stanford.edu/courses/cs221-artificial-intelligence-principles-and-techniques

Please follow along with the course schedule: https://stanford-cs221.github.io/autumn2025/

Follow the playlist: https://youtube.com/playlist?list=PLoROMvodv4rMeDqwS1yFl3j3sR_-MQNEN&si=bVivXjDfVEQKky1D

Teaching Team

Percy Liang, Associate Professor of Computer Science (and courtesy in Statistics)