Inside an LLM Agent: A From-Scratch Walkthrough

Why It Matters

Understanding LLM agent internals empowers developers to create transparent, cost‑effective automation solutions, accelerating enterprise AI adoption.

Key Takeaways

• Build LLM agents from scratch using open‑source tools.
• Tool calling requires explicit context and post‑processing orchestration.
• Lightning AI Studio provides free L4 GPU for hands‑on demos.
• Ollama models enable cost‑effective, open‑source LLM integration seamlessly.
• Memory module upcoming chapter expands agent state management.

Summary

The ACM Tech Talk walked through the internals of a large‑language‑model (LLM) agent, previewing content from Dr. Andre Fajardo’s forthcoming book *Build a Multi‑Agent System from Scratch*. Using a Lightning AI Studio notebook and Ollama’s open‑source models, the presenter demonstrated how to construct a minimal framework that equips a backbone LLM with tool‑calling capabilities, memory handling, and orchestration logic without relying on third‑party SDKs.

Key insights included the distinction that LLMs generate text but do not execute code; they must be supplied with tool definitions and then parse their own output to issue tool calls. The demo used a simple "hailstone" function to illustrate the full cycle: prompt → tool‑call request → execution → result fed back to the LLM for final answer. The talk also highlighted practical resources—free L4 GPU time on Lightning AI, Ollama model integration, and a public GitHub repo—to let developers replicate the workflow.

A notable example was the manual tool‑call sequence where the LLM, given the query "next number in the hailstone sequence after six," generated a structured request, the system executed the function, and the LLM synthesized the correct answer. The presenter emphasized that this pattern scales to automated loops, enabling agents to perform multi‑step tasks autonomously.

The implications are clear: by exposing the underlying mechanics, engineers gain confidence to build custom agents, reduce dependence on opaque platforms, and leverage cost‑effective open‑source models. As memory modules and richer abstractions are added, such DIY frameworks could accelerate enterprise adoption of LLM‑driven automation.

Original Description

Title: Inside an LLM Agent: A From-Scratch Walkthrough

Date: June 4, 2026

Duration: 1HR

ABSTRACT

Everybody's talking about AI agents. But what's actually inside one when you strip away the framework?

In this talk, Val Andrei Fajardo (author of the upcoming Manning Publications book, Build a Multi-Agent System (From Scratch) and ex-founding engineer at LlamaIndex) walks through the internals of an LLM agent from scratch. No CrewAI, no LangGraph, no black boxes. Just the raw building blocks: tool abstraction, the agent processing loop, MCP integration, and the Agent Skills open standard (plus a sneak peek at the upcoming memory chapter).

Through live examples, you'll see exactly what's happening under the hood of the agent clients you use every day, and how to build it yourself.

SPEAKER

Val Andrei Fajardo

AI Engineering and Research Lead at Carlyle

MODERATOR

Garcia Liang

Principal Machine Learning Scientist at Integrate AI