Llm-based Systems, Moving Beyond a Naive Loop to a Production-grade Architecture

The author argues that a basic loop is correct but insufficient, comparing it to a lone pilot versus a full air campaign with planners, parallel squadrons, fuel budgets, and after-action reviews. The post upgrades each component of a basic harness using composition of small, testable primitives: typed tools with Pydantic validation, a plan DAG for parallel execution, tiered memory under a retrieval budget, a verification hierarchy, multi-dimensional budgeting, and a tracer, all wired by a thin orchestrator. The design splits roles into Planner, Worker, and Critic to avoid overloading a single prompt.

To illustrate, the post builds a city comparison agent that takes a list of cities and produces a report on population, timezone, and narrative summaries. The task decomposes into independent lookups (nine tool calls for three cities) that can run in parallel, while the final report depends on all lookups completing. Tools have varying costs: in-memory dictionary reads for population and timezone, and LLM calls for summaries and aggregation, creating realistic budget pressure.

For reproducibility, lookup tools use a mocked dictionary, CITY_FACTS, and LLM-backed pieces can run against a real Anthropic model or a deterministic mock provider. The mock provider returns role-aware responses—a canonical plan, templated summaries, and rule-based verdicts—to separate orchestration bugs from model planning issues. Typed tools with Pydantic models provide runtime validation, JSON Schema for tool-use AP Is, documentation for the planner, and cost accounting hooks, enabling fail-fast validation before expensive calls. The post notes that an eval suite, retrieval benchmarks, and specialized worker pools will be covered in a future post.