LLM Observability for Beginners: The Playbook for Tracing and Logging AI Calls

On Monday morning, Meera, a developer in Pune, gets a message from a customer. Her company’s AI help assistant told them a refund takes “30 days”. The real rule is 7 days. Meera opens her code and stares at it. The answer is gone. She has no idea what the assistant was given, which steps it ran, or why it said that.

She is missing something every beginner misses at first: a record of what happened inside each AI call. That record is what LLM observability for beginners is all about. This playbook shows Meera building it in five small plays.

Last verified: 31 October 2026. AI tools change fast; check the linked sources for anything newer.

Who this is for: anyone who has built a small AI feature and cannot explain its wrong answers. No prior monitoring experience needed. Time: about 30-45 minutes.

LLM observability for beginners: what it is, in simple words

Think of a restaurant kitchen with a ticket for every order. The ticket shows what the customer asked for, who cooked it, how long it took and what went out. When a dish comes back wrong, you read the ticket instead of guessing.

LLM observability is that ticket for your AI app. It means saving enough detail about every AI call that you can later answer: what went in, what came out, how long it took and what it cost.

Two words come up everywhere. A trace is the full record of one request from start to finish. A span is one step inside it, such as a model call, a document search or a tool call. Many spans make one trace.

Ordinary logs are just lines of text. Tracing keeps the steps connected, so you can see the prompt, the model answer, tool calls, token counts and timing in one place. Guides from tools such as MLflow and PostHog describe the same idea.

You do not need a big platform to start. The five plays below work with a plain database table, and they teach the same habits that paid tools are built on.

The playbook at a glance

1. Give every request an ID.
2. Record each AI call as a span.
3. Track tokens, cost and time.
4. Protect private data in your logs.
5. Read the traces and turn failures into tests.

Play 1: Give every request an ID

Goal: be able to find one exact conversation later.

1. Create a unique ID when a user request arrives.
2. Pass that ID to every step that follows.
3. Show the ID in the support screen, so a customer message can be matched to a trace.

Meera adds a request ID. Now the angry refund message can be matched to one exact run.

You are done when: you can paste an ID and find that single request.

Play 2: Record each AI call as a span

Goal: see every step, not only the final answer.

1. List the steps in your feature: search documents, build the prompt, call the model, check the answer.
2. For each step, save its name, its input, its output, and the start and end time.
3. Save the model name and the prompt version, so you know which setup produced the answer.

Here is a small illustration. The exact names differ between tools, but the idea is the same: wrap each step, then save what happened.

def traced(step_name, request_id, fn, *args):
    start = time.time()
    try:
        result = fn(*args)
        save(request_id, step_name, args, result, start)
        return result
    except Exception as err:
        save(request_id, step_name, args, str(err), start)
        raise

In plain steps: note the start time, run the step, then save the input, the output (or the error) and the time taken.

Meera finds her answer: the document search returned an old policy page that said 30 days. The model only repeated what it was given. Without the span, she would have blamed the model.

You are done when: one request shows every step, in order, with inputs and outputs.

Play 3: Track tokens, cost and time

Goal: know what each answer costs and how long users wait.

1. Save the input and output tokens (small pieces of text the model counts) for every call.
2. Multiply by your model’s current price, read from the provider’s pricing page, to get a cost per call.
3. Save how long each step took, and look at the slowest 5 percent, not only the average.

Meera sees that a few requests send a huge prompt and take 12 seconds (an illustrative number). One average would have hidden them.

You are done when: you can say what yesterday’s AI calls cost and which were slowest.

Play 4: Protect private data in your logs

Goal: keep traces useful without leaking secrets.

1. Remove or mask names, emails, phone numbers and passwords before saving a prompt.
2. Decide how long traces are kept, and delete them after that.
3. Limit who can open them. Traces contain real customer words.

Meera adds a masking step that hides emails and phone numbers. Her trace tool is now a safe notebook, not a leak waiting to happen. This also pairs well with our prompt injection playbook, which suggests logging rejected outputs.

You are done when: a saved trace shows no raw personal details.

Play 5: Read traces and turn failures into tests

Goal: make every bad answer fix the app for good.

1. Once a week, open 10 traces: some bad, some random good ones.
2. For each bad one, find the first step that went wrong.
3. Copy that case into your test set, as described in our AI evals playbook.

Meera removes the old policy page and adds the refund question as a test. Next time that page sneaks back, the test fails before a customer sees it.

You are done when: every bad answer you found has a matching test.

Do you need a tool, or is your own log enough?

Mistakes to avoid

• Logging only the final answer. Fix: save every step, so you can find the first one that failed.
• Saving private data. Fix: mask it before it is stored.
• Trusting averages. Fix: look at the slowest and most expensive calls.
• Forgetting versions. Fix: save the model name and prompt version on every span.
• Collecting traces nobody reads. Fix: book a weekly 20-minute review.

How to know it is working

• A customer complaint can be traced to one request in under five minutes.
• You can name the cost and the slowest step of your feature.
• Your test set grows from real failures.

Your checklist

• ☐ Every request has an ID
• ☐ Every step is saved as a span with input, output and time
• ☐ Tokens, cost and slowest calls are visible
• ☐ Personal data is masked and old traces are deleted
• ☐ A weekly review turns failures into tests

Frequently asked questions

What is the difference between a trace and a span?

A trace is one whole request. A span is one step inside it. A trace is the order ticket; spans are the cooking steps.

How is observability different from logging?

Logs are loose lines of text. Observability keeps the steps linked together and searchable, so you can follow one request across all its steps.

Do I need a paid tool?

No. For one small feature, a database table with the fields from Play 2 is enough. Move to a tool when you have many features or many people.

How is this different from evals?

Evals test your app before release with known questions. Observability watches real requests after release. Use both: traces find new failures, evals stop them coming back.

Meera’s Monday, one week later

A new complaint arrives. This time Meera pastes the request ID, opens the trace and sees the faulty step in two minutes. She fixes it, adds a test and goes back to her tea. Next, read about how AI agents work, where traces matter even more because there are many steps.

Sources: MLflow, LLM tracing; PostHog, AI observability. The story and numbers are illustrative.