Technology Videos

How to Run LLMs Locally - Full Guide

The video provides a step‑by‑step guide for developers who want to run large language models (LLMs) on their own hardware, focusing on two primary approaches: the open‑source Ollama tool and Docker’s model runner. It begins by positioning local inference as a solution for speed, privacy, and cost concerns that arise when relying on hosted services like ChatGPT, and then walks viewers through downloading, installing, and verifying the Ollama client across macOS, Windows, and Linux. Key insights include the mechanics of pulling models—using commands such as "ollama pull"—and the importance of matching model size to hardware capabilities. The presenter demonstrates running a tiny 271 MB model (small‑m‑2) interactively, highlights the latency advantage of local execution, and shows how to expose the model via an HTTP REST API (default port 11434) for programmatic access. Python examples illustrate both raw HTTP calls and the convenience of the "ollama" Python package, while the Docker model runner is presented as a more robust, GPU‑accelerated alternative that runs on port 12434 and integrates seamlessly with containerized workflows. Notable examples feature the model incorrectly answering a factual question (the capital of Canada) to underscore the limitations of very small models, and a successful generation of a 500‑word essay on the fall of Rome, retrieved via both Ollama and Docker endpoints. The speaker also points out practical UI differences—Ollama’s command‑line interface versus Docker Desktop’s graphical model browser—and provides concrete commands for listing, running, and inspecting models in both environments. The implications are clear: developers can replace external API calls with locally hosted LLMs, cutting subscription fees and eliminating data‑exfiltration risks while achieving near‑zero network latency. By leveraging either Ollama for quick CLI‑based experimentation or Docker for production‑grade container deployment, teams gain flexibility to integrate AI capabilities into existing stacks, from custom back‑end services to LangChain pipelines, fostering greater control over cost, compliance, and performance.