Agent-7
As agents and their surrounding ecosystems like Model Context Protocol (MCP) tooling have become more powerful, Codat has been following along and climbing the ladder of agentic engineering.
Over the past year we have been building complex skills and workflows locally to aid us in everyday engineering tasks. As we compared our usage patterns, it became clear that some tasks only required a human to kick them off and check their results at the end.
To use a common example: a ticket is marked ready. An engineer picks up the ticket and prompts an agent locally to produce an implementation. The agent goes off and comes back with a draft pull request. The human prompt is the only hurdle to starting the agent. What if we could trigger this automatically and free up our engineer to tackle something else?
We realised that if we wanted to run these agents autonomously, we would need to build a runtime to manage them.
Introducing our runtime
Agent-7 is our agent runtime. It lets us determine what runs, when, with which tools, and for how long, and leaves the how to the agent.
Each agent has its own definition, which consists of a trigger (e.g. a Slack webhook), a set of tools and the required credentials, and a prompt.
How a run works
A single agent run might look like this: a Slack webhook arrives; Agent-7 consumes the webhook, matches it to an agent definition, and then kicks off an isolated container with the agent's toolset and prompt. The agent then goes off, processes the request and actions an update like posting in Slack or updating a Jira ticket. Once the agent is finished, the container is spun down and the resources are cleaned up.
During a run we define a set of deterministic checks to ensure our agents aren't overloaded or performing redundant tasks. For example, we might check whether a Jira ticket is in flight, to avoid having multiple agents implement the same work.
- A triggerA message, an alert, a ticket update, or a set time.
- Agent-7 picks the agentIt matches the event to a job.
- Determinism checkThe agent worksAn agent is spun up.
- It gets the job donePosts in Slack, updates a ticket.
- Nothing is left behindThe container is spun down.
What we control
Through the runtime we are able to pull levers for each agent. For example, the toolset: does this agent need access to logs? Does it need Slack? This is useful because it gives us control over two key areas: agent context and security. With a well-defined toolset the agent is not pulling redundant information or accessing systems it shouldn't.
Authorization scopes allow us to narrow the toolset further. A given toolset will define what actions an agent can take, whereas an authorization scope limits where it can take them. For example, a GitHub token might only be allowed to create pull requests in the repository the agent is working on. This keeps every agent to the least privilege it needs.
A second lever is agent limits. We can limit how many times an agent can go back to the model, and how long it can run. This stops an optimistic agent from running away down a rabbit hole and burning through our token budget immediately.
How we test them
Agents are non-deterministic, which makes them difficult to test. We are unlikely to get the exact same result from our pr-review bot twice in a row. However, we do expect the results to be roughly the same and to meet a certain standard.
To evaluate our agents we've introduced Harbor, an agent evaluation harness, which grades the agent's output against known scenarios. The evaluation checks deterministic fields we expect every time (e.g. did it pick the correct Slack channel?), and uses LLM-as-judge grading for agent prose where necessary.
Results so far
Agent-7 has been running for about six weeks, across a small subset of our internal repositories as we prove out its effectiveness and bring new agents online. But in those six weeks our implementation agent has opened 129 pull requests, of which 105 have been merged and closed, and our review agent has posted just over 200 reviews across nine repositories.
- Raised129
- Merged105
| Week | Raised | Merged |
|---|---|---|
| 29 Jun | 4 | 0 |
| 6 Jul | 4 | 0 |
| 13 Jul | 7 | 1 |
| 20 Jul | 26 | 18 |
| 27 Jul | 57 | 36 |
| 3 Aug | 114 | 81 |
| 10 Aug | 129 | 105 |
The early signs are promising, and while it's great that the agents can take on this work, it's important to note that these agents are running simple, repeatable, and well-scoped tasks. Automating them has freed our engineers to spend their attention on the more complex (and exciting!) tasks.
Next steps
We have proven our agents can successfully handle work that engineers would have otherwise had to do. The next step is to roll out our agents across more of our estate.
Beyond that we are looking to develop deeper, more complex agent workflows by chaining our agents together.
If this work is of any interest to you, then please head over to our careers page.