Evals, Retries, and Cost Limits: The Unglamorous Agent Stack

Flashy demos skip the boring layers. Production teams cannot. If you are staffing an agent initiative, budget for the unglamorous stack early — or you will pay for it as outages, surprise invoices, and quiet quality decay.

Evals before vibes

Ship a labeled set for the jobs the agent must do. Gate deploys on regressions. When a vendor drops a new model on a Friday, your question should be 'did evals pass?' not 'does it feel smarter?'

Retries with a brain

Blind retries amplify bad tool calls. Retry transient infrastructure errors. Do not retry policy denials, validation failures, or 'are you sure?' prompts. Cap attempts, backoff, and escalate to a human when the graph is stuck.

Cost limits as product features

• Per-run token and dollar budgets
• Per-customer daily caps
• Hard stops when enrichment loops detect duplicates
• Alerts when cost-per-successful-outcome drifts

Logging and tracing you can actually debug

You need the plan, the tool calls, the model versions, the retrieved context, and the final action in one trace. Without that, every incident becomes folklore.

None of this is glamorous. All of it is what separates an agent that survives Monday morning from one that looked great in the board deck.