AI Videos

Andrej Karpathy Built GPT in 243 Lines?! Meet MicroGPT

The video introduces MicroGPT, a minimalist implementation of a GPT‑style transformer written in just 243 lines of pure Python. Created by Andrej Karpathy, the project strips away all external dependencies—no PyTorch, TensorFlow, NumPy or other libraries—so that the entire model, from autograd to training loop, can be inspected line‑by‑line. MicroGPT builds a tiny autograd engine that mimics PyTorch’s tensor objects, tracking values and gradients automatically via the chain rule. The architecture includes token embeddings, positional embeddings, multi‑head self‑attention and a ReLU‑Squid activation, all coded from scratch. Training proceeds by tokenizing input text, predicting the next character, computing loss, and updating parameters with an “atom” optimizer. Karpathy emphasizes that the goal is comprehension, not performance, noting “It’s not about speed, it’s about understanding.” The demo shows the model generating coherent text despite its simplicity, proving that a functional transformer can be assembled without heavyweight frameworks. By exposing every component in a compact, dependency‑free script, MicroGPT lowers the barrier for students and hobbyists to explore transformer mechanics, potentially accelerating education and experimentation in the AI community.