AI Playbooks Nov 3, 2026 · 7 min read

Run an LLM Locally: A Beginner’s Playbook for Ollama and Open Models

Learn how to run an LLM locally with Ollama in five simple plays: pick a model, install, chat, call it from code and know when to go bigger.

Meera is a developer in Pune. Every time she tests her small note-summarising app, it sends text to a paid AI service. Her bill creeps up, and her manager asks a fair question: “Do the customer notes really have to leave our laptops?”

Table of Contents

So Meera decides to run an LLM locally. That means the AI model lives and works on her own computer, with no outside service involved. She has never done it before. This playbook follows her first afternoon, in five small plays.

Last verified: 3 November 2026. AI tools change fast; check the linked sources for anything newer.

Who this is for: any beginner who can use a terminal and wants to try an AI model on their own machine. Time needed: about 30-45 minutes, mostly waiting for a download.

What does it mean to run an LLM locally, in simple words

An LLM (large language model) is the kind of AI that reads and writes text. Normally you rent one over the internet, like ordering food from a restaurant. Running it locally is like cooking at home: you need a kitchen and ingredients, but nobody else sees what you cook.

The “ingredients” are an open model, a model whose files you can download and run yourself. The “kitchen” is a tool that loads those files and answers your questions. The most popular beginner kitchen is Ollama.

If words like tokens and context window are new, read How LLMs Work first. It takes ten minutes.

The playbook at a glance

Decide why you want local, and check your machine.
Install Ollama and run your first model.
Start small and test with your own tasks.
Call the model from your own code.
Know when a bigger server tool is the better choice.

Play 1: Decide why you want local

Goal: avoid a download you do not need.

Write down your reason: privacy, cost, working offline, or just learning.
Check free disk space and memory. Models are large files, often several gigabytes, and a bigger model needs more memory.
Decide what “good enough” means for your task, for example “a clear three-line summary”.

Meera’s reason is privacy. Her “good enough” is a three-line summary of a customer note.

You are done when: you can say in one sentence why you are running locally.

Play 2: Install Ollama and run a model

Goal: see an answer appear from your own machine.

Install Ollama. On Linux and macOS the project README gives a one-line install script; Windows has its own installer. Use the exact command on the official page, since it can change.
Run a model with ollama run followed by a model name. The README example is ollama run gemma4.
The first run downloads the model, so wait. After that, type a question and read the answer.
Type /bye to leave the chat.

Meera types “Summarise this note in three lines” and pastes a fake note. Words stream back in seconds. Her laptop fan spins, and she smiles.

You are done when: the model answers a question with your internet switched off.

Play 3: Start small and test on your own tasks

Goal: find the smallest model that does your job.

Think of model sizes like vehicles. A scooter is cheap and quick in town, a truck carries more but needs fuel and space. Small models run fast on ordinary laptops. Bigger ones usually write better but need more memory and time.

Collect 5 real examples of your task.
Try the smallest model first and read each answer.
Move up one size only if the answers fall short.
Write down which model passed. This is a tiny version of an AI eval.

Meera finds that a small model handles four of her five notes well. She keeps it, and notes the one failure.

You are done when: you can name one model that passes your five examples.

Play 4: Call the model from your own code

Goal: let your app talk to the local model.

While Ollama runs, it answers requests at http://localhost:11434. “Localhost” just means “this same computer”. Your code sends a message and gets a reply, the same way it would with a paid service. This sketch comes from the README; check the API docs for current details.

curl http://localhost:11434/api/chat -d '{
  "model": "gemma4",
  "messages": [
    {"role": "user", "content": "Why is the sky blue?"}
  ],
  "stream": false
}'

In plain steps: name the model, send a list of messages, and ask for the whole answer at once instead of word by word (that is what "stream": false does).

Meera swaps the web address in her app and keeps the rest. Customer notes now stay on her machine.

You are done when: your app gets an answer from the local address.

Play 5: Know when to go bigger

Goal: avoid forcing a laptop tool to serve a whole company.

Ollama is a friendly way to start. When many users hit the model at once, teams usually look at serving tools built for servers, such as vLLM, which is a different kind of kitchen: a restaurant line instead of a home stove. Compare them using your own load, not opinions.

Count how many people will use it at the same time.
Measure how long one answer takes.
If one user is fine and many are slow, plan a server tool or a hosted service.

Meera’s tool serves only her team of four, so her laptop setup is enough for now.

You are done when: you know whether your use is “just me”, “my team” or “many customers”.

Local or hosted: which one?

Option	Good for	Trade-off
Local with Ollama	Privacy, learning, offline work, small teams	You supply the hardware and the upkeep
Hosted API	Best answer quality, many users, no setup	Per-use cost and data leaves your machine
Server tool such as vLLM	Many users on your own servers	More setup and operations work

Local models do not learn your documents by themselves. To make one answer from your files, see RAG vs fine-tuning.

Mistakes to avoid

Picking the biggest model first. Fix: start small and move up only when tests fail.
Ignoring memory. A model that does not fit becomes painfully slow. Fix: check free memory before downloading.
Assuming local means safe. A model that can call tools can still be tricked. Fix: read AI guardrails.
Judging by one answer. Fix: test at least five real examples.

How to know it is working

Answers arrive with the internet off.
Your five examples pass.
Response time feels acceptable for one user.

Your checklist

☐ Wrote down why I want local
☐ Checked disk and memory
☐ Installed Ollama and ran a model
☐ Tested five real examples
☐ Called the model from code
☐ Decided if a server tool is needed

Frequently asked questions

Is it free to run an LLM locally?

The software and many open models cost nothing to download, but you pay in hardware, electricity and your time. Check each model’s licence before using it at work.

Do I need a special graphics card?

Not to start. Small models can run on ordinary computers, only slower. A stronger machine makes bigger models practical.

Is my data really private?

When the model runs on your machine, your prompts are processed there. Still protect the machine and your logs like any other sensitive data.

Are local models as good as hosted ones?

It depends on the task and the model size. For simple summaries they can be enough. Test on your own examples instead of trusting rankings.

Meera’s evening, and your next step

By evening Meera tells her manager: the notes now stay on the laptop, the bill for testing is zero, and one tricky note still needs a bigger model. She did not need a data centre, only a clear reason and a small test.

Your next action: install Ollama, run one small model, and test five of your own tasks. Then read the AI agents playbook to see what a model can do once it has tools.

Sources: Ollama GitHub README and Ollama API docs. Meera’s story and her numbers are illustrative.

#ai for beginners #ai playbook #local llm #ollama #open models #run llm locally