Why AI Agents Fail: LLM Limits in Python (Full Example)

This is the full implementation of the example from the article Why Agents Fail: LLM Limits.

If you have not read the article yet, start there. Here we focus on code: how to make agent behavior more stable despite the model's natural limits.

What This Example Demonstrates

LLM can fail on format or invent source citations
The agent must validate the response deterministically instead of trusting text "at a glance"
Low model confidence should route the task to human handoff
Step limits and bounded context are required even for a simple case

Project Structure

TEXT

foundations/
└── llm-limits-agents/
    └── python/
        ├── main.py           # agent loop with retry + handoff
        ├── llm.py            # model call
        ├── validator.py      # strict output validation
        ├── knowledge.py      # local KB for grounding
        └── requirements.txt

Module separation matters: the model generates, while the validator and policy make decisions.

How to Run

1. Clone the repository and go to the folder:

BASH

git clone https://github.com/AgentPatterns-tech/agentpatterns.git
cd foundations/llm-limits-agents/python

2. Install dependencies:

BASH

pip install -r requirements.txt

3. Set the API key:

BASH

export OPENAI_API_KEY="sk-..."

4. Run:

BASH

python main.py

What We Build in Code

We build a simple guarded loop for answering a customer question.

retrieve relevant snippets from a controlled knowledge base
provide the model with a strict JSON response contract
the validator checks format, citations, and confidence level
if the response fails validation, we retry; if confidence is low, we escalate to a human

This is a learning example where the key is not a "beautiful answer" but controlled and verifiable agent behavior.

Code

`knowledge.py` — controlled sources

PYTHON

from typing import Any

KB = [
    {
        "id": "KB-101",
        "title": "Refund Policy",
        "text": "Refunds are available within 14 days after payment. "
                "After 14 days, refunds are not available.",
    },
    {
        "id": "KB-102",
        "title": "Pro Plan",
        "text": "Pro customers have priority support and a 4-hour SLA.",
    },
    {
        "id": "KB-103",
        "title": "Free Plan",
        "text": "Free customers get business-hours support with no SLA.",
    },
]


def search_kb(question: str, limit: int = 2) -> list[dict[str, Any]]:
    q = question.lower()
    scored: list[tuple[int, dict[str, Any]]] = []
    for item in KB:
        score = 0
        text = f"{item['title']} {item['text']}".lower()
        for token in ("refund", "pro", "free", "sla", "support"):
            if token in q and token in text:
                score += 1
        scored.append((score, item))

    scored.sort(key=lambda x: x[0], reverse=True)
    return [item for _, item in scored[:limit]]


def build_context(snippets: list[dict[str, Any]], max_chars: int = 700) -> str:
    parts: list[str] = []
    total = 0
    for s in snippets:
        line = f"[{s['id']}] {s['title']}: {s['text']}\n"
        if total + len(line) > max_chars:
            break
        parts.append(line)
        total += len(line)
    return "".join(parts).strip()

LLM answers only based on this context. This reduces room for fabrication.

`validator.py` — strict validation of model output

PYTHON

import json
from dataclasses import dataclass
from typing import Any


@dataclass
class ValidationResult:
    ok: bool
    data: dict[str, Any] | None
    errors: list[str]


def validate_model_output(raw: str, allowed_sources: set[str]) -> ValidationResult:
    errors: list[str] = []

    try:
        data = json.loads(raw)
    except json.JSONDecodeError:
        return ValidationResult(False, None, ["invalid JSON"])

    if not isinstance(data, dict):
        return ValidationResult(False, None, ["output must be a JSON object"])

    answer = data.get("answer")
    citations = data.get("citations")
    confidence = data.get("confidence")
    needs_human = data.get("needs_human")

    if not isinstance(answer, str) or not answer.strip():
        errors.append("answer must be non-empty string")

    if not isinstance(citations, list) or not all(isinstance(x, str) for x in citations):
        errors.append("citations must be a list of strings")
    else:
        unknown = [x for x in citations if x not in allowed_sources]
        if unknown:
            errors.append(f"unknown citations: {unknown}")

    if not isinstance(confidence, (int, float)) or not (0 <= float(confidence) <= 1):
        errors.append("confidence must be a number in [0, 1]")

    if not isinstance(needs_human, bool):
        errors.append("needs_human must be boolean")

    return ValidationResult(len(errors) == 0, data if len(errors) == 0 else None, errors)

This protects against "confident garbage": an answer can look good but still be invalid.

`llm.py` — model call with a clear contract

PYTHON

import os
from openai import OpenAI

api_key = os.environ.get("OPENAI_API_KEY")

if not api_key:
    raise EnvironmentError(
        "OPENAI_API_KEY is not set.\n"
        "Run: export OPENAI_API_KEY='sk-...'"
    )

client = OpenAI(api_key=api_key)

SYSTEM_PROMPT = """
You are a support agent.
Reply with VALID JSON only in this format:
{
  "answer": "short answer",
  "citations": ["KB-101"],
  "confidence": 0.0,
  "needs_human": false
}
Use only sources that exist in the provided context.
If data is insufficient, set needs_human=true.
""".strip()


def ask_model(question: str, context: str, feedback: str | None = None) -> str:
    user_prompt = (
        f"Customer question:\n{question}\n\n"
        f"Context:\n{context}\n\n"
        "Return JSON only."
    )

    if feedback:
        user_prompt += f"\n\nFix your previous response using this error feedback: {feedback}"

    completion = client.chat.completions.create(
        model="gpt-4.1-mini",
        temperature=0,
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_prompt},
        ],
    )

    content = completion.choices[0].message.content
    return (content or "").strip()

We do not ask to "just explain". We define a strict contract and verify it.

`main.py` — agent loop with retry, confidence gate, and handoff

PYTHON

from knowledge import build_context, search_kb
from llm import ask_model
from validator import validate_model_output

MAX_STEPS = 4
MIN_CONFIDENCE = 0.65

QUESTION = "Can I get a refund for my subscription if 10 days have passed since payment?"


def run():
    snippets = search_kb(QUESTION, limit=2)
    allowed_sources = {s["id"] for s in snippets}
    context = build_context(snippets, max_chars=700)

    print("Allowed sources:", sorted(allowed_sources))
    print("Context:")
    print(context)

    feedback: str | None = None

    for step in range(1, MAX_STEPS + 1):
        print(f"\n=== STEP {step} ===")
        raw = ask_model(QUESTION, context, feedback=feedback)
        print("Model raw output:", raw)

        validation = validate_model_output(raw, allowed_sources)
        if not validation.ok:
            print("Validation failed:", validation.errors)
            feedback = "; ".join(validation.errors)
            continue

        data = validation.data
        assert data is not None

        if data["needs_human"] or data["confidence"] < MIN_CONFIDENCE:
            print("\nHandoff required:")
            print(
                "Model confidence is too low. Escalate this case to a human "
                f"(confidence={data['confidence']})."
            )
            return

        print("\nFinal answer:")
        print(data["answer"])
        print("Citations:", data["citations"])
        return

    print("\nStop: MAX_STEPS reached without a valid answer. Escalate to human.")


if __name__ == "__main__":
    run()

This is the key idea: the agent does not trust the model without validation, even when the text looks convincing.

`requirements.txt`

TEXT

openai>=1.0.0

Example Output

TEXT

python main.py
Allowed sources: ['KB-101', 'KB-102']
Context:
[KB-101] Refund Policy: Refunds are available within 14 days after payment. After 14 days, refunds are not available.
[KB-102] Pro Plan: Pro customers have priority support and a 4-hour SLA.

=== STEP 1 ===
Model raw output: {
  "answer": "Yes, you can get a refund for your subscription if 10 days have passed since payment, as refunds are available within 14 days after payment.",
  "citations": ["KB-101"],
  "confidence": 1.0,
  "needs_human": false
}

Final answer:
Yes, you can get a refund for your subscription if 10 days have passed since payment, as refunds are available within 14 days after payment.
Citations: ['KB-101']

Note: exact model raw output may vary between runs.
Example correctness criterion: the validator filters out invalid/hallucinated answers, and the final answer passes validation.

What You See in Practice

	Naive approach	Guarded approach
Accepts any model text	✅	❌
Validates format and sources	❌	✅
Has fallback on low confidence	❌	✅
Has a step limit	❌	✅

Where to Go Next

Add a separate max_tokens_per_run budget and log actual usage
Add a second reviewer model as an independent quality gate
Add needs_human_reason so the operator can see the escalation reason
Add replay tests with fixed model raw outputs

Full Code on GitHub

The repository contains the full version of this demo: retrieval context, quality-gate checks, and escalation to a human.

View full code on GitHub ↗

Why AI Agents Fail: LLM Limits in Python (Full Example)

What This Example Demonstrates

Project Structure

How to Run

What We Build in Code

Code

`knowledge.py` — controlled sources

`validator.py` — strict validation of model output

`llm.py` — model call with a clear contract

`main.py` — agent loop with retry, confidence gate, and handoff

`requirements.txt`

Example Output

What You See in Practice

Where to Go Next

Full Code on GitHub

Used by patterns

Related failures

Governance required

Author

Editorial note

Why AI Agents Fail: LLM Limits in Python (Full Example)

What This Example Demonstrates

Project Structure

How to Run

What We Build in Code

Code

knowledge.py — controlled sources

validator.py — strict validation of model output

llm.py — model call with a clear contract

main.py — agent loop with retry, confidence gate, and handoff

requirements.txt

Example Output

What You See in Practice

Where to Go Next

Full Code on GitHub

Used by patterns

Related failures

Governance required

Author

Editorial note

`knowledge.py` — controlled sources

`validator.py` — strict validation of model output

`llm.py` — model call with a clear contract

`main.py` — agent loop with retry, confidence gate, and handoff

`requirements.txt`