Harness Engineering¶
A production agent is not the model. It is the model plus everything around it: retries, timeouts, tool permissions, context management, observability, and guardrails. That surrounding layer is the harness.
The term emerged in early 2026. Mitchell Hashimoto described "Engineer the Harness" as Step 5 of his AI adoption journey: anytime an agent makes a mistake, build a solution so it never makes that mistake again [Hashimoto, 2026]. Martin Fowler later published a synthesis that framed the harness as a system of guides and sensors around the model [Martin Fowler, 2026], and OpenAI's internal Harness team documented a similar environment-first approach [OpenAI Engineering, 2026].
Agent = Model + Harness¶
The model is replaceable. Providers release new versions every quarter. The harness is what stays:
- The retry policy around an LLM call.
- The circuit breaker that trips when a provider is down.
- The token budget and compaction logic.
- The tool-permission proxy.
- The traces and metrics that explain what happened.
If the model is the engine, the harness is the chassis, brakes, dashboard, and fuel gauge. A faster engine without a strong harness just crashes sooner.
Your backend skills are the missing piece¶
Senior backend engineers already know most of what a production agent needs:
- Circuit breakers and bulkheads for downstream calls.
- Retries with jitter and exponential backoff.
- Idempotency keys for state-changing operations.
- Timeouts and backpressure.
- Queues for slow, asynchronous work.
- Graceful degradation and failover.
- Observability through traces, metrics, and logs.
These patterns apply to LLM providers exactly as they apply to databases or microservices. The 429, 503, and 529 responses from Anthropic or OpenAI are no different from a slow or failing microservice.
What is new in the harness¶
Not everything is old backend work. Agents add a few specific concerns:
| Layer | Why it is agent-specific |
|---|---|
| Context window management | The context is append-only, cached, and billed on every loop. Mutations invalidate the KV cache and multiply cost. |
| Memory and retrieval | The agent needs working, episodic, semantic, and procedural memory, each with different consistency and cost requirements. |
| Tool permission policies | The LLM chooses tools dynamically, so permissions must be enforced at invocation time, not just declared in a prompt. |
| Progress tracking | A single request can span tens of model calls. The harness must expose state: current step, evidence, plan, cost. |
| Pipeline compaction | When the context grows, the harness decides when and how to compact without dropping critical instructions. |
These are the topics covered in Parts 1 and 2 of the baseline. They are not exotic AI research; they are production engineering with an agent-shaped twist.
Guides and sensors¶
Martin Fowler's harness model splits controls into two directions:
- Guides (feedforward) steer the agent before it acts. Examples: lint rules for generated code, typed tool schemas, system prompts that set scope, and context budgets that cap what the model can see.
- Sensors (feedback) observe after the agent acts and either self-correct or escalate. Examples: unit tests, type checkers, output validators, and LLM-as-judge evaluators that flag off-topic or dangerous outputs.
Both directions have two flavors:
- Computational — deterministic, fast, cheap. Tests, linters, regex checks, JSON schema validation.
- Inferential — semantic, slower, more expensive. LLM-as-judge, quality scoring, similarity checks.
A good harness uses cheap computational controls everywhere and reserves expensive inferential controls for the places where they matter.
The steering loop¶
The right response to a repeated agent failure is not to rephrase the prompt; it is to add or tighten a control in the harness. The human steers by iteratively adding guides and sensors, then letting the agent self-correct [Böckeler, 2026].
- A missing convention becomes an
AGENTS.mdpointer or a typed schema. - A repeated structural bug becomes a linter or a structural test.
- An ambiguous review becomes a checklist or an agent reviewer skill.
- A drift-prone invariant becomes a recurring "doc-gardening" or cleanup agent.
Over time the harness accumulates a portfolio of constraints that encode the team's intent. The agent then writes code inside that envelope rather than in an open-ended search space. OpenAI's Codex harness team put this into practice: when the agent struggled, they identified the missing tool, guardrail, or documentation and had Codex write the fix, continuously improving the environment [OpenAI, 2026].
Timing: keep quality left¶
Cheaper controls should run earlier; expensive or broad controls should run later [Böckeler, 2026].
| Phase | Typical controls | Cost / latency |
|---|---|---|
| Pre-commit | Linters, type checks, fast unit tests, basic code-review agent | ms–s, CPU |
| Post-integration | Integration tests, broader code review, mutation testing | s–min, CPU/GPU |
| Continuous / drift | Dead-code detection, dependency scanning, SLO anomaly detectors, doc-gardening agents | periodic, CPU/GPU |
| Runtime | LLM-as-judge sampling, trace-based quality scoring, incident feedback | sampled, GPU |
Earlier detection is cheaper because the agent has invested less context and less reviewer time. Reserve inferential sensors for the stages where their semantic judgment is worth the cost and latency.
Regulation categories¶
A harness regulates the codebase along several dimensions [Böckeler, 2026]:
Maintainability harness¶
Controls internal quality: duplicate code, cyclomatic complexity, test coverage, naming, and architectural drift. Computational sensors catch the structural issues reliably; inferential sensors can catch semantically redundant code or over-engineered solutions, but only probabilistically and at higher cost. Neither reliably catches misdiagnosed requirements, unnecessary features, or misunderstood instructions — those are human-specification failures.
Architecture fitness harness¶
Feedforward: skills that encode performance, reliability, and observability conventions (e.g., "always emit structured logs"). Feedback: performance tests, ArchUnit-style boundary checks, and post-change debugging reflections. These are fitness functions for the architecture.
Behaviour harness¶
The hardest category. Feedforward is the functional specification; feedback is the test suite, mutation testing, and approved fixtures. At current capability, green AI-generated tests alone are not sufficient evidence of correctness; manual testing and fixture-based approval remain necessary guardrails [Böckeler, 2026].
Harnessability¶
Not every codebase is equally easy to harness. Strong typing, clear module boundaries, stable conventions, and architectural fitness functions multiply the controls available. Conversely, a tangled monolith with implicit rules gives the agent few guide rails and few sensors, so it will wander [Böckeler, 2026].
Harness templates¶
Most organizations have a small set of recurring service topologies. A harness template bundles the guides, sensors, conventions, and AGENTS.md maps for one topology so teams do not rebuild them from scratch [Böckeler, 2026]. Examples:
| Topology | Bundled controls |
|---|---|
| API service | OpenAPI lint, request/response schema tests, observability conventions, authZ harness |
| Event processor | Schema registry checks, idempotency tests, dead-letter handling, DLQ monitors |
| Data dashboard | Query-cost guardrails, data freshness sensors, visualization linting |
Failure modes and metrics¶
Common harness failures and what to measure:
| Failure mode | Sensor or guide to add | Metric |
|---|---|---|
| Agent repeats a mistake | Add a linter, schema, or structural test | Time-to-harness (days from first failure to control) |
| Context overflow or budget blowout | Token budget, compaction policy, cost ceiling | Cost per task, tokens per loop |
| Agent ignores a guardrail | Enforce at invocation, not in prompt; add a validator | Guardrail bypass rate |
| Hallucinated tool calls | Typed tool schemas, output validators | Tool-call validation failure rate |
| Silent drift | Recurring doc-gardening / cleanup agent | Drift issues opened per week |
| Over-engineering | Complexity lint, approved-fixtures pattern | Review comments per PR |
Team shape¶
The strongest agent teams are not "ML engineers vs. backend engineers." They are a partnership:
- ML / AI engineer: owns model behavior, prompting, fine-tuning, and evaluation.
- Harness / backend engineer: owns the runtime, tooling, observability, and failure handling around the model.
Right now the market overweights the model side and underweights the harness side. That imbalance will not last, because production agents fail on the harness far more often than they fail on the model.
Case studies: harness failures in production¶
Two recent incident reconstructions show what happens when the harness is treated as an afterthought.
| Incident | What the agent did | Missing harness control | Cost / impact |
|---|---|---|---|
| Content-ops agent (Supergood, 2026) | Wrote to unapproved CMS fields, hallucinated a vendor contact, and entered a 4-hour retry loop during a 40-minute CMS outage | Field-level write allowlists, structured output with confidence/source, retry caps and circuit breakers |
$340 in unplanned API spend; three production records corrupted; manual rollback |
| $47K data-pipeline agent (Clyro, 2026) | Four LangChain-style agents kept generating new "approaches" to an unsolvable schema-drift problem, resetting per-approach retry counters each time | Loop detection, per-run cost ceilings, step limits, circuit breakers at 3× baseline deviation | $47,200 in API calls over 11 days; ~$12,000 in engineering time; two enterprise customer migrations delayed |
In both cases the model behaved exactly as prompted; the failure was in the harness around it. The Supergood team found that retrofitting guardrails cost more than building them upfront [Sandoval, 2026]. The Clyro reconstruction notes that the only alert was a monthly account-level budget, and the agent's status message — "Schema drift resolution in progress" — was technically correct yet operationally meaningless [Clyro Content Team, 2026].
Observable harness evolution¶
Manual harness tuning does not keep pace with quarterly model releases. Agentic Harness Engineering (AHE) treats the harness as an explicitly editable workspace and uses three observability pillars to evolve it automatically: component observability (every editable harness component has a file-level representation), experience observability (raw trajectory tokens are distilled into a layered evidence corpus), and decision observability (each edit is paired with a falsifiable prediction in a versioned manifest) [Lin et al., 2026].
On Terminal-Bench 2, ten AHE iterations lifted pass@1 from 69.7% to 77.0%, surpassing the human-designed Codex harness (71.9%) and self-evolving baselines. The same frozen harness transferred to SWE-bench-verified and used 12% fewer tokens than the seed, while producing +5.1 to +10.1 percentage-point gains across three alternate model families. Ablations showed the improvement came from tools, middleware, and long-term memory — not from the system prompt, suggesting that factual harness structure transfers while prose-level strategy does not.
A practical takeaway is the change manifest: every harness edit should state the failure evidence, root cause, targeted fix, predicted impact, and predicted regressions. The next evaluation round confirms or reverts the edit at file granularity. This makes the steering loop measurable rather than anecdotal.
Implementation patterns and anti-patterns¶
Deepset's Haystack experience frames the harness around four concrete responsibilities: context engineering, progressive tool disclosure, orchestration (sub-agent spawning, handoffs, routing), and guardrails/verification [deepset, 2026]. Externalizing memory, skills, and protocols turns recall problems into retrieval problems, improvised workflows into guided execution, and ad-hoc coordination into auditable contracts.
| Pattern | Anti-pattern | Why it matters |
|---|---|---|---|
| Field-level write allowlists with dry-run validation | Record-level write tokens and prompt-based permission reminders | The agent cannot "helpfully" edit fields it should never touch [Sandoval, 2026] |
| Structured output with confidence and source fields | Plain text outputs committed as fact | Low-confidence fields route to human review instead of being written as truth [Sandoval, 2026] |
| Retry cap + circuit breaker + per-run cost budget | Naive retry loops with no bounds | A transient outage becomes a runaway spend spiral instead of a bounded failure [Clyro Content Team, 2026; Sandoval, 2026] |
| Progressive tool disclosure (load tools only when needed) | Injecting the full tool catalog into every call | Keeps the context prefix stable, cheap, and cache-friendly [deepset, 2026] |
| File-level harness components under version control | Harness logic buried in prompts or hidden in agent loops | Each change is visible, testable, and revertible [deepset, 2026; Lin et al., 2026] |
Summary¶
- The harness is everything around the model: retries, routing, permissions, context, memory, observability.
- Most of the harness is standard backend engineering applied to LLM calls.
- The agent-specific parts are context management, memory, tool permissions, progress tracking, and compaction.
- A harness combines feedforward guides and feedback sensors, computational and inferential.
- The steering loop turns every repeated failure into a new control.
- Production agent quality is a team sport between model and harness engineers.
Deeper dive: the runtime harness around the model¶
The runtime harness is not packaging around the model; it is the active control plane that converts raw model output into safe, verifiable actions. Faros AI frames harness engineering as the third phase of AI engineering maturity—after prompt engineering and context engineering—because autonomy, accuracy, and control are now the bottleneck [Faros AI, 2026]. A field benchmark shows the delta starkly: the same 8B model fails 53% of multi-step tasks out of the box and succeeds 99% once a thin reliability layer handles rescue parsing, retry nudges, step enforcement, and context budget management [DEV Community, 2026]. No new weights, no new GPU—just runtime controls.
A production-grade harness is layered. Faros describes five runtime layers: tool orchestration, verification loops, context and memory, guardrails, and observability [Faros AI, 2026]. Tool orchestration decides which tool to call, in what order, and how to recover when a call fails. Verification loops insert unit tests, linters, and self-critique between steps so small errors do not compound. Guardrails enforce hard scope, security sandboxes, budget ceilings, and human-in-the-loop gates. Observability captures the exact tool inputs, outputs, and state so regressions become measurable rather than anecdotal.
| Runtime layer | What it controls at runtime |
|---|---|
| Tool orchestration | Tool selection, chaining, and dynamic error recovery |
| Verification loops | Intermediate QA: tests, linters, self-critique |
| Context and memory | Session history, codebase indexing, long-term skills |
| Guardrails | Scope limits, security sandboxes, budget ceilings, HITL gates |
| Observability | Traces, audit logs, telemetry, regression signals |
Some failure modes live in the harness, not the weights. Faros cites Anthropic research showing that agents "declare victory" before verifying outcomes, rush to finish as context fills ("context anxiety"), and overreach by one-shotting entire tasks [Faros AI, 2026]. The LangChain team moved from 30th to 5th on Terminal Bench 2.0 without changing the underlying model, purely by optimizing the harness [Faros AI, 2026]. That is evidence that runtime controls—prompted in the right place, enforced at invocation, and observed in traces—are the dominant production lever.
Default harnesses such as Claude Code or Codex are a starting point, not a finished product. Engineering teams still add compliance linters, migration-file sign-off gates, audit logging, and MCP servers that expose internal APIs [Faros AI, 2026]. The practical move this quarter is to baseline from existing systems—cost per merged PR, time-to-merge for agent-assisted PRs, review velocity relative to PR size, and compute spend per developer—then let that data decide which layer gets the next investment [Faros AI, 2026]. This keeps the runtime harness aligned with real engineering outcomes, not model benchmarks.
References¶
- Böckeler, B. "Harness engineering for coding agent users." Martin Fowler, Apr 2026. https://martinfowler.com/articles/harness-engineering.html
- Böckeler, B. "Harness Engineering - first thoughts." Martin Fowler, Feb 2026. https://martinfowler.com/articles/exploring-gen-ai/harness-engineering-memo.html
- Hashimoto, M. "My AI Adoption Journey." Feb 2026. https://mitchellh.com/writing/my-ai-adoption-journey
- OpenAI. "Harness engineering: leveraging Codex in an agent-first world." OpenAI, Feb 2026. https://openai.com/index/harness-engineering/
- OpenAI Engineering. "Harness engineering: leveraging Codex in an agent-first world." Engineering.fyi, Feb 2026. https://www.engineering.fyi/article/harness-engineering-leveraging-codex-in-an-agent-first-world
- Faros AI. "Harness Engineering: Making AI Coding Agents Work in 2026." Faros AI, May 2026. https://www.faros.ai/blog/harness-engineering
- DEV Community. "Harness Engineering: How to Build Production-Ready LLM Agents That Actually Work." DEV Community, May 2026. https://dev.to/monuminu/harness-engineering-how-to-build-production-ready-llm-agents-that-actually-work-20kc