Loop Engineering > Prompting: Why the Best Teams Stopped Tweaking Prompts

TL;DR

• Prompt engineering tunes one input. Loop engineering designs the entire generate → evaluate → refine → retry system the model runs inside.
• The bill is almost never set by your prompt — it's set by how many times the loop runs, how much context each turn carries, and what gets re-sent vs cached.
• Teams that shift from prompt-tuning to loop design routinely cut 40–70% of agent token spend with no quality loss — sometimes with quality *gains* because evaluation is now explicit.
• Works the same across OpenAI, Anthropic, Google, Meta, and xAI — loop structure is provider-agnostic, prompt tricks aren't.

The Mental Model Shift

A prompt is a single shot. A loop is a system: a model call wrapped in retrieval, evaluation, retry policy, memory, and termination conditions. Modern agent traffic is loops — coding agents, research agents, support copilots, long-running automations. The output quality and the cost are both emergent properties of the loop, not the prompt.

Once you see it this way, the optimization surface changes:

The first column has a ceiling. The second column has a 10× ceiling.

Where the Tokens Actually Go

In a typical agent loop, the user's question is a rounding error. The bill is built from four things, in order of impact:

1. Loop count — how many model calls happen before the loop terminates. A poorly-designed agent that takes 14 turns to do a 4-turn job burns 3.5× the budget on the same task.

2. Context carryover — what the agent re-sends every turn. Most agents naively append everything: the full conversation, all tool outputs, all retrieved docs. By turn 10, the input is 50K+ tokens of mostly-irrelevant history.

3. Tool-output bloat — a single `ls -R` or unfiltered SQL result can dump 20K tokens into the next turn's context. Multiply by every turn that doesn't trim it.

4. Retry / self-correction storms — when the loop has no explicit stopping condition, it retries on its own outputs until something looks "done." This is where stealth 6-figure bills come from.

Notice that none of these are fixed by a better prompt. They're fixed by loop architecture.

A Realistic Before/After

Take an autonomous coding agent on Claude Opus 4.8 running 10K tasks/month. The naive loop:

• Avg 11 turns per task
• Avg 50K input + 8K output per turn (full history re-sent, no caching, raw tool outputs)
• Monthly bill: ~$32,000

Now apply loop engineering — same model, same tasks, same target quality:

Same model, same outputs. New bill: ~$9,500/month — a 70% reduction. Run your own numbers in the Agent Loop Cost Estimator.

The kicker: teams report higher quality after this exercise, because the plan-and-verify structure forces the model to be deliberate instead of meandering.

The Five Loop-Engineering Disciplines

These are the moves, in priority order:

1. Cap the loop

Every loop needs an explicit termination condition that isn't "the model thinks it's done." Hard turn budgets, verifier checks, or a planner that pre-commits to a turn count. This single discipline kills the worst overruns.

2. Manage context like memory, not history

Stop re-sending the full conversation. Maintain a working memory — summarized older turns, pinned key facts, closed sub-tasks dropped entirely. Treat the context window as a workspace you actively curate, not a transcript.

3. Cache the stable prefix

System prompt, tool schemas, retrieved docs, and the plan rarely change within a task. Put them at the front of every call and turn on prompt caching. On OpenAI, Anthropic, Google, and most OpenRouter routes this drops the prefix to ~10% of its normal cost on cache hits — typically 40–60% off the whole bill on its own.

4. Filter tool outputs at the source

The tool layer — not the model — should decide what's worth sending back. A code-search tool returns the 3 relevant snippets, not 80. A SQL tool returns 10 rows + a row-count, not 10K rows. Every byte the tool trims is a byte the model doesn't bill you for, every subsequent turn.

5. Route by difficulty, not by reflex

Most loop turns don't need the frontier model. Use a small/cheap model (e.g. Gemini Flash, Claude Haiku, GPT-5 Mini, Llama, Grok-mini) for routing, classification, summarization, and tool-arg extraction; reserve the expensive model for actual reasoning steps. The Pareto code router is one productionized version of this.

Why This Works Across Every Provider

Prompt tricks are model-specific — what coaxes GPT-5.5 may confuse Claude Opus 4.8, and Gemini 3 ignores half of it. Loop engineering is provider-agnostic: turn budgets, context curation, caching, and routing apply identically whether you're on OpenAI, Anthropic, Google DeepMind, Meta Llama, or xAI Grok. That's why it survives model swaps — and why it's the right place to invest engineering time when the underlying models change every quarter. See current rates across providers in the Pricing Table.

The Cultural Shift

Prompt engineering felt like a craft you could do alone, in a text box. Loop engineering looks more like systems engineering: you instrument, you measure, you set budgets, you write evals, you ship dashboards. The skill that matters now isn't "the perfect prompt" — it's reading a trace and knowing which of the five disciplines is leaking your money.

The teams winning on AI economics in 2026 aren't the ones with the cleverest prompts. They're the ones whose loops are boringly disciplined.

Takeaway

If you're still A/B-testing system messages while your agent burns 11 turns and re-sends 50K tokens of history every call, you're optimizing the wrong layer. Stop tuning prompts. Start engineering loops. The token bill — and usually the quality — will tell you immediately.

For the cost math on your specific stack, plug your numbers into the Agent Loop Cost Estimator, then layer in prompt caching and batch APIs for the next round of savings.