Agents that know when to stop

Most agent failures we get asked to fix are not wrong answers. They are agents that keep going. A support agent that re-reads the same ticket eleven times. A research agent that opens a fortieth browser tab. A data-cleaning agent that “fixes” a column, notices the fix changed something else, and fixes that too.

An agent that cannot stop is a liability, and the stopping logic is usually the least-designed part of the system. This post is how we design it at Kairn Labs: explicit stop conditions, budgets, tool-call limits, escalation to a human, and evals that test all of it.

Every run should end for a reason we chose in advance, not a reason we discover in the logs.

Why agents don’t stop on their own

A language model inside a loop is asked, on every turn, “what next?” Nothing in that question invites the answer “nothing.” The model is biased toward producing another action, because the prompt, the tool list and the conversation history all describe work to be done.

Add a few common conditions and it gets worse:

• A tool returns an ambiguous result, so the model retries it.
• The goal is underspecified, so no result ever feels finished.
• An error message is vague enough that the model keeps guessing at fixes.
• The model is rewarded, in its own context, for looking busy: each new step reads like progress.

You cannot fix this with a sentence in the system prompt that says “stop when you are done.” That sentence helps. It is not a control. Controls live in code, outside the model, where a bad turn cannot talk its way past them.

Stop conditions are a type, not a vibe

The first thing we do on any agent project is write down every way a run can end. We encode it as an enum, and the loop is not allowed to return anything else.

type StopReason =
  | "goal_met"          // the agent called `finish` with a validated result
  | "step_budget"       // hit max steps
  | "token_budget"      // hit max tokens or cost ceiling
  | "tool_limit"        // one tool called too many times
  | "no_progress"       // same state seen repeatedly
  | "needs_human"       // agent asked for help, or policy demanded it
  | "tool_error"        // unrecoverable tool failure
  | "policy_block";     // an action was refused by a guard

interface RunResult<T> {
  stopReason: StopReason;
  output?: T;
  steps: number;
  trace: StepRecord[];
}

interface Budget {
  maxSteps: number;
  maxTokens: number;
  maxCallsPerTool: Record<string, number>;
}

async function runAgent<T>(
  task: Task,
  budget: Budget,
  deps: AgentDeps<T>
): Promise<RunResult<T>> {
  const trace: StepRecord[] = [];
  const toolCalls: Record<string, number> = {};
  const seen = new Set<string>();
  let tokens = 0;

  for (let step = 0; step < budget.maxSteps; step++) {
    const turn = await deps.model.next(task, trace);
    tokens += turn.usage.totalTokens;
    if (tokens > budget.maxTokens) return end("token_budget");

    if (turn.kind === "finish") {
      const checked = deps.validate(turn.result);
      if (checked.ok) return end("goal_met", checked.value);
      trace.push({ step, note: "finish rejected", errors: checked.errors });
      continue;
    }

    if (turn.kind === "escalate") return end("needs_human");

    const name = turn.call.name;
    toolCalls[name] = (toolCalls[name] ?? 0) + 1;
    if (toolCalls[name] > (budget.maxCallsPerTool[name] ?? 5)) {
      return end("tool_limit");
    }

    const verdict = deps.guard(turn.call);
    if (!verdict.allowed) return end("policy_block");

    const fingerprint = JSON.stringify([name, turn.call.args]);
    if (seen.has(fingerprint)) return end("no_progress");
    seen.add(fingerprint);

    const result = await deps.tools.run(turn.call);
    trace.push({ step, call: turn.call, result });
    if (result.fatal) return end("tool_error");
  }
  return end("step_budget");

  function end(stopReason: StopReason, output?: T): RunResult<T> {
    return { stopReason, output, steps: trace.length, trace };
  }
}

A few details in that sketch matter more than they look.

The finish action is validated. The model does not get to declare victory; it proposes a result, and code checks it against a schema and any business rules. A rejected finish goes back into the trace as feedback, and the loop continues until the step budget ends it.

The fallthrough at the bottom of the loop is step_budget. If every other guard fails to fire, the loop still ends. There is no while (true) anywhere in our agent code.

The no_progress check is crude on purpose. Repeating the same tool with the same arguments is almost never productive, and a fingerprint set catches it cheaply. More elaborate state hashing can come later if a project needs it.

Budgets: steps, tokens and tool calls

We set three budgets, and we set them from the task, not from a default.

Step budget. Count the steps a competent human would take, then give the agent a margin above that. A task that needs three lookups and one write gets a budget in the low teens, not a hundred. A tight budget turns a runaway into a cheap, visible failure instead of an expensive, invisible one.

Token or cost budget. Steps are not equal. One step with a huge document in context costs more than ten small ones. We track tokens per run and cap them, so cost has a ceiling per task that finance can reason about.

Per-tool call limits. The most dangerous tools get the lowest limits. A search tool might be allowed eight calls. A tool that sends an email or writes to a database is allowed one or two per run, and anything beyond that is a tool_limit stop, not a retry.

We also split tools by consequence. Read-only tools are cheap to call and cheap to get wrong. Tools with side effects require an idempotency key and pass through a guard before they execute. The guard is ordinary code: an allowlist, a maximum amount, a required approval flag. It does not ask the model whether the action is a good idea.

Escalation is a feature

“Needs human” is a successful outcome. We say this to every client, because teams tend to treat escalation as a failure of the agent, and then tune the agent to avoid it, which is how you get confident wrong answers.

We give the agent an explicit escalate action and tell it when to use it:

• The request is outside the scope the agent was given.
• Two sources of truth disagree.
• The action would be irreversible and the policy asks for approval.
• The agent has tried the obvious approaches and nothing changed.

When an agent escalates, it hands over a package: what it was asked, what it did, what it found, and the specific question it needs answered. A human should be able to resolve the case in a minute without reading the whole trace. If the handoff is poor, people stop trusting escalations and start ignoring them.

Some escalations are not the agent’s choice. Policy can force a handoff regardless of what the model wants: refunds above a threshold, anything touching a regulated field, a customer flagged as sensitive. These are needs_human stops produced by the guard, and they are often where the most value in an automation project sits, because the boring 80 percent of cases flow through and the judgment calls land with the right person.

What each stop reason triggers

A stop reason is only useful if it maps to a response. This is the table we put in every agent’s runbook.

Stop reason	What it means	What we do
goal_met	Validated result produced	Deliver the output, log the trace
step_budget	Ran out of steps	Return partial work, queue for review, add to the eval set
token_budget	Cost ceiling reached	Same as above, plus check for oversized context
tool_limit	One tool overused	Inspect that tool’s description and error messages
no_progress	Repeated identical action	Escalate; usually a tool is returning something ambiguous
needs_human	Agent or policy asked for help	Route to the owner with the handoff package
tool_error	Fatal failure in a tool	Alert engineering, not the end user
policy_block	A guard refused an action	Log and escalate; review whether the guard or the agent is wrong

Every non-goal_met stop teaches us something. A cluster of no_progress stops usually points at a tool whose output is confusing. A cluster of step_budget stops means either the budget is too tight or the task is badly scoped. We read these like error logs, because that is what they are.

Evals that test stopping

A normal eval asks whether the answer was right. For agents we add a second question: did the run end the right way?

Each eval case carries an expected stop reason alongside the expected output. Some cases are designed to be unanswerable. Some contain a tool that always fails. Some ask for an action the guard should refuse. The agent passes only if it stops cleanly and for the right reason.

interface AgentCase {
  id: string;
  task: Task;
  expectStop: StopReason;
  expectOutput?: (out: unknown) => boolean;
  maxSteps?: number;
}

async function runEvals(cases: AgentCase[], deps: AgentDeps<unknown>) {
  const failures: string[] = [];

  for (const c of cases) {
    const budget = { maxSteps: c.maxSteps ?? 12, maxTokens: 60_000, maxCallsPerTool: {} };
    const res = await runAgent(c.task, budget, deps);

    if (res.stopReason !== c.expectStop) {
      failures.push(`${c.id}: stopped with ${res.stopReason}, expected ${c.expectStop}`);
      continue;
    }
    if (c.expectOutput && !c.expectOutput(res.output)) {
      failures.push(`${c.id}: output failed check`);
    }
  }

  const passed = cases.length - failures.length;
  console.log(`${passed}/${cases.length} passed`);
  if (failures.length) {
    console.error(failures.join("\n"));
    process.exitCode = 1;
  }
}

The harness is small on purpose. It runs in CI, exits non-zero on any failure, and prints which case failed and why. The value is in the cases, not the code.

We build the case list in layers. Start with the happy paths that should end in goal_met. Add the cases that should escalate. Add adversarial cases: a tool that returns the same empty result forever, a prompt that tries to widen the agent’s scope, a task whose answer does not exist. Every production incident becomes a new case before the fix ships, so the same failure cannot come back quietly.¹

Tuning without guessing

When a case fails, the temptation is to edit the prompt and rerun until it passes. That can work, but it overfits fast. We try to fix problems in this order:

1. Fix the tool. Clearer descriptions, better error messages, narrower parameters. Many loops come from a tool that says “failed” without saying why.
2. Fix the guard or the budget. If the stop was correct but too early or too late, adjust the limit.
3. Fix the scope. Split one broad agent into two narrow ones with a clear handoff.
4. Only then, fix the prompt.

Narrow agents stop better than broad ones. An agent with three tools and a clear definition of done has few ways to wander. An agent with twenty tools and a vague mission has many.

Ship with a kill switch

One last control sits outside the loop entirely. Every agent we deploy has a flag that disables it, a per-run cost alert, and a daily ceiling on total runs and spend. Nobody should need a deploy to turn an agent off at two in the morning.

Stopping well is part of what makes an agent trustworthy enough to give real work to. Build the exits first, test them first, and the rest of the agent has somewhere safe to go wrong.

Footnotes

1. We keep failing traces as fixtures, with sensitive data removed, so a regression case can replay the exact tool outputs that caused the original loop. ↩

#agents#evals#reliability#typescript