Loop Economics: Cost Per Iteration

Why a Single Iteration’s Cost Is Not Fixed

The cost of a single LLM call is simple to compute: input tokens × input price + output tokens × output price. But in an agent loop this calculation changes with every iteration, because the message history accumulating in the context window grows larger with each pass.

In iteration 1 the model processes the system prompt plus the user message. In iteration 2 it receives all of that, plus the model’s response from iteration 1 and the tool result. In iteration 3 the messages from iteration 2 are also included. As this history accumulates, the input token count for iteration N equals the sum of all tokens from previous iterations plus the new content added in the current turn.

┌──────────────────────────────────────────────────────────────────┐
│  Cumulative Input Tokens per Iteration (simplified example)      │
├──────────────────────────────────────────────────────────────────┤
│  iter 1:  system(500) + user(200)                = 700 tokens    │
│  iter 2:  prev 700 + model resp(150) + tool(300) = 1,150 tokens  │
│  iter 3:  prev 1,150 + model(120) + tool(400)    = 1,670 tokens  │
│  iter N:  accelerating growth, not linear                        │
│                                                                  │
│  → Total cost of 10 iterations ≠ (iter 1 cost) × 10             │
│    The actual total is substantially higher                      │
└──────────────────────────────────────────────────────────────────┘

JSON Accumulation Drives Super-Linear Growth

The cost increase becomes even more pronounced when tool results return JSON payloads. A file-exploration tool returning a directory structure as JSON, or a web search returning long article bodies wrapped in JSON, injects that entire payload into the next iteration’s context — even after the agent has already processed the information. The data stays in the context window and is re-processed with every subsequent call.

OpenAI’s analysis of the Codex agent loop confirms this pattern. In a code-editing loop, tool results — file contents, test output, error messages — stack up with each iteration. By the later iterations, input token counts are often several times larger than at the start. Loop economics are decidedly nonlinear.

Modeling the Cost Structure Explicitly

This is a conceptual pseudocode example; actual API signatures may differ.

from dataclasses import dataclass, field

@dataclass
class LoopCostTracker:
    """Track per-iteration and cumulative costs for an agent loop."""
    input_price_per_1k: float   # e.g., 0.003 USD per 1K input tokens
    output_price_per_1k: float  # e.g., 0.015 USD per 1K output tokens
    iterations: list = field(default_factory=list)

    def record(self, input_tokens: int, output_tokens: int) -> float:
        cost = (
            (input_tokens / 1000) * self.input_price_per_1k
            + (output_tokens / 1000) * self.output_price_per_1k
        )
        self.iterations.append({
            "iteration": len(self.iterations) + 1,
            "input_tokens": input_tokens,
            "output_tokens": output_tokens,
            "cost_usd": cost,
        })
        return cost

    @property
    def total_cost(self) -> float:
        return sum(i["cost_usd"] for i in self.iterations)

    @property
    def total_input_tokens(self) -> int:
        return sum(i["input_tokens"] for i in self.iterations)

Embedding a tracker like this in the loop lets you check in real time whether a run is staying within budget. This is why a USD cost cap is a natural termination condition alongside iteration and token limits.

Cost Control Strategies

Controlling loop costs operates at several levels.

Tool result compression: Rather than inserting raw tool output directly into context, summarize or filter it first. Extract only the information the agent actually needs from a file-exploration result and store it in a concise form.

Context window cleanup: Remove processed tool results, keeping only the essential information. The compaction pattern from chapter 5-3 handles this: when context reaches a threshold, the entire history is summarized and replaced with a fresh, compact representation.

Iteration hard cap: Set an explicit maximum iteration count. Twenty iterations is sufficient for most complex tasks, and it prevents runaway cost from an infinite loop.

Cost-based termination: Check cumulative cost after each iteration and halt the loop when the budget is exceeded.

MAX_COST_USD = 1.0
MAX_ITERATIONS = 20
tracker = LoopCostTracker(input_price_per_1k=0.003, output_price_per_1k=0.015)

iteration = 0
while iteration < MAX_ITERATIONS:
    response = call_model(messages)
    cost = tracker.record(
        response.usage.input_tokens,
        response.usage.output_tokens
    )

    if tracker.total_cost > MAX_COST_USD:
        raise BudgetExceededError(
            f"Total cost {tracker.total_cost:.4f} USD exceeded limit"
        )

    if response.stop_reason == "end_turn":
        break

    # ... execute tools and append messages
    iteration += 1

Loop Cost vs. Single-Call Cost

Understanding that a loop is always more expensive than a single call enables deliberate strategy selection. Tasks with predictable answers that require no tool interaction are far more economical as single calls. Only tasks that genuinely require interaction with the external world and mid-course verification justify the compounding cost of a loop.

The next chapter covers the most effective technique for reducing loop cost: prompt caching — caching the system prompt and tool definitions that repeat unchanged across every iteration, so their token cost effectively disappears.