Building with agents

Seneca Shield makes chart review software for hospice and home health; I’m CTO and co-founder. StoryArbor is a Mac app I’m building for writers.

Agents make it fast to produce and change code. The harder question is whether a change is correct, and the same question applies to what a product’s AI tells its users. In both cases an output can look finished, or pass its checks, and still be wrong.

Building both taught me that smaller models need a strong, robust harness, and that working on local hardware demands efficiency at every level. The code around a model decides what context it sees, which tools it can call, what gets checked before an answer is trusted, and what happens when it’s wrong.

I built my own testing harness for working with agents. It has built-in checks, retries, and a way to tune prompts against the results, and it made iterating with agents fast and efficient: I could try a change, see whether it held up, and adjust. That harness is a development tool. It’s separate from the code around a model inside each product, which is part of the product itself.

Both projects also had developer tools for trying out changes to the product AI. In both, those tools were reworked over time so that what a test reports matches what the product does.

Seneca Shield’s product has evolved as our needs and the models’ capabilities have changed. Good synthetic data and a custom harness gave us fast feedback loops, so we could try ideas quickly. That’s what made the product more capable and more reliable, especially against messy, inconsistent source data.

For both StoryArbor and Seneca Shield, I came up with ways to generate smart metadata up front. It makes processing faster and the results better.

Working at the bleeding edge has its challenges. Getting small models to perform well on limited hardware is hard. The frontier labs aren’t worried yet. But the right harness, plus curiosity and persistence, gets you something genuinely useful. A lot of the best work in this space right now is building the scaffolding that closes the gap between small models and frontier ones.

I prefer the guidance files that agents read to be hand-written. Seneca Shield’s went from a long generated overview to a short list of the things that tend to trip agents up.

StoryArbor’s development process started with heavy sign-off steps. It was later revised to checks proportionate to the project’s stage, with independent review kept for consequential changes. Review costs attention, so I put it where the risk is.