Suffolk University Law School LIT Lab

How to Set up a Free, Open-Source, Local AI Assistant Using Ollama, Granite, and Open-WebUI

The Suffach Legal Innovation and Technology Labs workshop, hosted by IBM developer advocate JJ Asgar, walked participants through installing a free, open‑source AI stack on a personal computer. The tutorial focused on three components: Ollama, a Docker‑style container manager for LLMs; Granite, IBM’s fully open‑source foundational model; and Open‑WebUI, a feature‑rich web interface for querying the model. Asgar demonstrated how Ollama’s simple commands (e.g., "ollama pull", "ollama run") retrieve and launch Granite directly on a laptop, eliminating the need for remote cloud services. He highlighted Granite’s transparency—IBM provides the model’s weights and training data, and even offers legal defense if users face copyright claims. Open‑WebUI adds document upload and retrieval‑augmented generation (RAG) capabilities, allowing users to constrain the model’s knowledge to specific data sets. Key moments included a live run of Granite 3.3 and Granite 4 on Asgar’s machine, illustrating comparable performance to commercial APIs while consuming far less power. He used a librarian analogy to explain RAG: the model acts as a librarian who consults a curated library (the RAG) rather than the entire internet, ensuring more reliable, domain‑specific answers. He also warned about the unsustainable energy demands of large cloud‑based models and positioned local LLMs as a greener alternative. For businesses, this approach offers immediate cost savings, tighter data governance, and a clear legal safety net—critical factors as AI regulation tightens. By deploying open‑source models locally, organizations can experiment, prototype, and scale AI‑driven workflows without ceding control to third‑party providers, accelerating responsible AI adoption.