Adding agentic harnesses to a codebase

This week I added a bunch of tooling to our TypeScript codebase, and it drastically improved our ability to ship with AI. A lot of these harnesses aren’t new to me. I know them from my platform days.

Making sure a codebase had unit, integration and e2e tests was something I made sure happened when I was in platform. So was making sure everything could be run locally. I pushed hard on CI as a way to gate PRs, because it let us improve our automation without needing everyone to run it on their own laptop. And I always made sure we had the best build tool available, because as a codebase gets bigger, a good build tool starts to matter a lot.

What’s different now is who’s consuming the harness. AI agents read documentation and find examples the same way humans do. They’re just faster at it. But humans tend to be fine working around a manual step. An agent will give up once it hits one. So pushing everything toward zero manual steps isn’t just tidiness, it’s what lets the agent keep going longer before it actually solves the problem.

The two-fold approach

The main thing I took from this week is that helping AI ship well is two-fold. You build a tool to catch bad stuff, like code coverage, and you add a skill to show what good looks like.

The catch half is the deterministic tooling: coverage, linting, the gates that fail when something is wrong. That part is tractable, and it’s most of what I got in this week. The show half is a skill that demonstrates what good looks like, so the AI has something to follow instead of only something to avoid.

You need both.

Where it broke down

The first time I tried the two-fold approach, it didn’t work, because I hadn’t actually shown what good looks like. I had the tool to catch bad stuff, but the AI would just write more bad tests to get past it.

The bad tests were mainly mocking external APIs instead of actually running any integration tests. Everything written to beat the code coverage was just unit tests, and nothing was covering any of the boundaries. Even when I caught the boundaries with heuristics and asked the AI to write integration tests, it wouldn’t write a test that actually hit the API. Mainly because hitting the API needs the real user to supply credentials, which the AI didn’t have access to.

It’s actually really hard to enforce good tests with AI. You can only really do it by showing what good looks like and improving a skill to do that. You might be able to catch it with a skill that looks for bad tests, but I haven’t done that yet.

Right now I’ve mainly got the deterministic tooling in place. Next I need to write some code that shows what good looks like, before I can write the skills that help follow convention and catch the bad tests.

This is exactly what the recorded-cassette harness below is for: record the real API interaction once, then replay it deterministically, so a test can actually cover the boundary without needing live credentials. There’s a tool built specifically for this that does it better, called Pact. I’ve looked into it but haven’t set it up yet.

The harnesses I added

Here’s the rest of what went in. Most of it is what you’d expect, and some of it people might not have run into before, like using ESLint to enforce architecture or adding a pass that checks the shape of a monorepo.

• Run it locally: I reproduce the CI checks locally with pre-commit and pre-push hooks, so the AI gets the same pass or fail before it ever opens a PR.
• Testing: a recorded-cassette e2e harness, cached so the suite is cheap to rerun, plus flaky-test detection through nightly seeded reruns and tracking on how long tests take.
• Architecture in ESLint: I enforce clean-architecture import boundaries and structural invariants with ESLint and dependency-cruiser, so a module can’t reach across a boundary it shouldn’t. I ratchet the boundaries in one module at a time.
• Style and lint: I upgraded ESLint to flat config and turned on every rule, with no eslint-disable left anywhere.
• Monorepo structure: a typed gate that makes sure the repo is well formed and packages are put where they should be. I looked at Projen but it didn’t quite meet our needs. I think there’s a better way to enforce specific monorepo guardrails, but I haven’t found it.
• Issues: a skill that creates good issues, and a check that enforces issue quality when they’re created. Every PR has to reference an issue.
• Documentation: I keep the CLAUDE.md from going stale, and added a plugin that audits it.
• AI skills: a create-issue skill and a new-script-or-package skill, all aligned to a skill-creator convention.
• Caching: Turbo-cached builds and test cassettes.
• Cheap builds: cut the Actions minutes by skipping CI on docs-only changes, and guard against oversized scripts.
• Automation: a changesets release bot that auto-rebases its version PR, and scheduled dependency updates aligned to a release-age cooldown.

Everything has to be automated

AI works best when everything is automated. You can’t introduce anything manually. If a step needs a human to run it, the AI can’t do it, and you’re back to doing it yourself. So you strive for the AI to do all of it, which means every check has to be something it can run on its own.

Keeping things deterministic also saves on tokens. A deterministic check gives the same answer every time, so you’re not paying a model to reason about it. That makes everything faster, and it lets you use cheaper models against the codebase.

Determinism means the model doesn’t need to figure out something you’ve already thought through and encoded into the harness. A cheap model is fine at finding a simple solution and implementing it with the tools already in front of it. The expensive thinking models are better spent on complex features that touch large parts of the codebase, where there isn’t a harness to lean on yet. It’s a bit like using a higher-order programming language: the harness does the thinking you’d otherwise pay a smarter model to redo every time.

What I walked back

Two things I put in this week I also took back out.

I built my own architecture boundary checker, then ripped it out and consolidated onto eslint-plugin-boundaries. The standard plugin covered what I needed, so there was no reason to keep maintaining my own.

I also added a coverage gate on changed lines, then removed it. The problem was we didn’t have great tests to begin with. AI can write tests, but it can’t write good ones on its own. It doesn’t know what’s actually failing. That mostly gets caught by running the system, in production or through acceptance testing, not by a coverage number. We can spot the specific seams that need tests, things like database calls, but we still have to show the AI what a good test for that seam looks like before a gate on top of it means anything.

I haven’t built that yet. I’ve been thinking about a code review agent that specifically judges how good a test is, rather than just whether one exists. My hesitation is cost: that could get expensive to run compared to just reaching for tooling that already does this, like mutation testing.

Building tooling for AI to ship well turns out to be a platform problem. Tests at every layer, things that run locally, CI as a gate, a good build tool: those are the same harnesses I built for a team, and they’re the same ones that make an agent productive. The difference is how far you take it. A team can live with a manual step here and there. An agent can’t. It gives up the moment it hits one. So the target is zero manual steps, because that’s what lets the agent keep going until it actually solves the problem.