This is the second of two posts about Admiral, the AI agent system we use to keep journalist profiles current. If you haven’t read part one, the short version is: v1 worked, ran in production, and taught us that an agent doing too many things at once is an agent you can’t improve.
By the third or fourth round of a research session, our AI agent was reading its own history like a novel. Most profiles complete in one or two iterations, but for the harder cases every search query, every result, and every decision the agent had made was still sitting in its context, attended to on every step, growing with no ceiling. That was the core problem, and it wasn’t only about cost. It was a design that made the whole system harder to tune, debug, and extend.
This post is about what we changed for v2, and why. The short version: by splitting that one overloaded agent into a pipeline of focused specialists, we cut cost per profile by around 40%, improved prediction accuracy by around 10%, made the output more consistent, and removed the ceiling that had kept us from running deeper research on the hardest profiles. And because each specialist is a self-contained piece, the same pipeline now powers research and enrichment well beyond journalist profiles.
The Problem We Were Solving
To understand the redesign, it helps to understand what v1’s agent was actually doing inside a single research session.
It was searching the web, checking social profiles, looking up publication records, deciding whether it had gathered enough information, producing structured output, getting that output reviewed, and deciding whether to accept the result or loop back. All of this happened inside one continuous loop, with one AI making all the decisions and carrying the full history of everything it had done so far.
That works, but it has costs.
The history never shrank. Every iteration added more messages to the agent’s context: the query it ran, the results it got, the decision it made. On harder profiles requiring three or four rounds, the agent was attending to a growing pile of raw data, most of which it had already processed and moved on from. Larger contexts mean higher API costs, and subtle quality problems as the model has to sift through more noise to find the signal.
One AI was doing two different jobs. Deciding what to search for next is exploratory work, while producing schema-compliant structured output is precise work. These benefit from different approaches, with different instructions and different model configurations. Mixing them together meant neither could be tuned without affecting the other.
One step was hard to isolate. Matching a publication name to a confirmed record in our database was critical to getting profiles right. But in v1 this happened inside the main loop, interleaved with everything else, so when a match was wrong, tracing the failure was hard.
The review happened too late. Our self-review step evaluated the proposed output after the agent decided it was finished. What works better is to catch evidence gaps earlier, during research rather than after.
The Redesign: Specialists Instead of One Generalist
The insight behind v2 was simple: if a task has distinct phases, give each phase its own specialist.
We broke the single agent loop into a sequence of focused components, each with a clear responsibility and clean inputs and outputs.
Research: just search
The research component has one job: gather evidence. It searches the web, checks social profiles, and follows leads. It doesn’t judge whether the evidence is sufficient, and it doesn’t produce any output, it just searches. Because that’s all it does, we can tune it independently, making it more thorough, improving its search strategies, and adjusting how it handles contradictory signals, without touching anything else in the pipeline.
Compress after every round
After each research pass, the accumulated findings are distilled into a concise structured summary. This isn’t the raw search results or the full sequence of queries and responses, just the actual findings: confirmed employer, recent articles, social handles, and any conflicting signals.
That summary is what every downstream step receives. Nothing after this point ever looks at the raw research history.
This was the highest-impact change in the entire redesign, and it had a compounding effect we didn’t fully anticipate when we designed it.
In v1, cost grew with every research iteration. Round one was cheap, but by round three or four (the harder profiles that needed the most depth) the cost of going one more round felt prohibitive. In practice, we were reluctant to allow deep research sessions because the cost curve was steep.
In v2, cost per iteration stays roughly constant. The research component always starts a new round with the same small input: the current summary plus any feedback from the review step. It doesn’t matter whether it’s the first round or the fourth, the context size is the same.
The practical result: we can afford more research iterations. Profiles that would have been cut short in v1 because another round was unjustifiable now get the depth they need. The compression step doesn’t just reduce cost, it removes the ceiling on research thoroughness.
Review the evidence, not the output
The review step in v2 asks a different question than v1. Instead of “does this output look right?” it asks “is the evidence sufficient to produce a reliable output?”
This is a better checkpoint, because an agent with thin evidence can produce a plausible-looking output that passes a superficial review. But if you evaluate the evidence directly, asking whether there are recent bylines, whether the employer is confirmed, and whether social profiles are verified, you catch gaps before they propagate into the final output.
When the review finds the research insufficient, it sends targeted instructions back to the research component. Rather than a vague “go search more,” it says something specific like “the employer isn’t confirmed, look for a staff page or LinkedIn profile directly,” so the next research round knows exactly what it’s looking for.
When a sub-task earns its own component
Some parts of a research pipeline are best kept entirely separate from the main loop. Matching publication names to confirmed records in our database is one example: a focused lookup task with clear inputs and outputs, distinct from open-ended research.
In v2, that match runs as a dedicated component, once, after research is complete and before enrichment begins. It receives the research summary, does its lookups, and returns a clean structured list of resolved records. That list is passed directly to the enrichment step.
The benefit is isolation. We can now run this component on any research summary and inspect its output independently. We can tune it, test it, and track its accuracy without touching the research logic or the enrichment logic. Accuracy on outlet matching improved, and more importantly, when it is wrong, we know exactly where to look.
This is a general pattern. Any sub-task that is well-scoped enough to operate on a defined input and produce a defined output can earn its own component. As we expand the schema to cover things like topic and beat classification, we expect to add specialists for those. One example would be a component that looks up an extended sample of recent articles, summarizes the journalist’s coverage, reviews whether the evidence supports a confident classification, and outputs the topics: a small pipeline of its own, sitting alongside the main one. The architecture supports this because the main pipeline coordinates discrete components, and new specialists slot in without disturbing the existing ones.
Enrich: just synthesize
The final journalist profile enrichment step currently receives three things: the compressed research summary, the resolved outlet records, and the original profile being updated. It has no search tools and no database access, it just synthesizes clean inputs into a structured output.
Because it isn’t attending to raw search history or interleaved tool calls, it can focus entirely on applying classification rules and filling fields correctly. Schema validation failures dropped noticeably, and the output became more consistent.
Crucially, because the enrichment step is its own isolated component, we can now define custom rules per field. How a social handle should be validated, what counts as a confirmed employer, how to handle conflicting signals on a beat: each field can carry its own logic without that logic bleeding into how the agent searches or how outlets get resolved. In v1, rules like these had to be baked into one monolithic prompt and applied universally. In v2, they live in the enrichment component’s specific prompt.
Two Cost Levers, One Result: More for Less
The first lever is smaller contexts. Compressing research history after every round means each iteration starts with the same compact input regardless of how many rounds have come before. Cost per iteration stays flat instead of growing.
The enrichment and summarization steps, which are precise, structured, and well-defined, perform just as well, and in some cases better, on a smaller, cheaper model. Swapping those steps to a lighter model delivered around 40% cost savings across the pipeline with no regression in output quality. When a task is well-scoped, a focused smaller model often beats a powerful one carrying irrelevant context.
Together these two levers changed the economics entirely. We now run more research iterations on average per enrichment, at a lower total cost per profile. The profiles that need depth get it, and the straightforward ones complete quickly.
Measuring Every Change: The Continuous Eval Pipeline
One of the hardest problems in AI systems isn’t building them, it’s knowing whether a change made them better or worse.
In v1, evaluation was manual and ad hoc. We’d run the agent on a handful of profiles we knew well, inspect the output, and make a judgment call. That works at small scale, but at production scale, with changes landing regularly and thousands of profiles in scope, it isn’t enough.
v2 ships with a continuous evaluation pipeline built on Langfuse. Every meaningful change (a prompt update, a configuration tweak, a new component) gets run against a curated dataset of profiles with known ground truth before it lands.
The dataset itself isn’t static. Journalists change jobs, switch outlets, and launch newsletters, so a ground truth captured a year ago would be wrong about a meaningful fraction of profiles today. We regenerate the curated dataset monthly from recent human reviews, keeping the benchmark in step with reality rather than frozen against a snapshot of the past.
The evaluation measures multiple dimensions per run:
Field-level accuracy: did the agent get the right answer on structured fields like role, social handles, and employer?
Coverage: for fields where the ground truth is filled, did the agent find a value? Missed opportunities are tracked separately from mismatches, because a wrong value is a different problem from a missing one, and we want to know both.
Mismatch detection: when both the ground truth and the agent produced a value, do they agree? A mismatch on a critical field is a different class of problem from a missed optional field.
Results are logged as named experiments in Langfuse, so we can compare runs side by side. A prompt change from last Tuesday can sit next to today’s version, on the same dataset and with the same metrics, visible in a single view.
This changed how we work. Before a prompt change ships, we know its effect on accuracy across every tracked field. Regressions get caught before they reach production. And because the evaluation is reproducible, we can revisit any previous experiment and understand exactly what the system was doing at that point in time.
The Payoff
The results after deploying v2 were meaningful across every dimension we tracked.
Token costs for the enrichment pipeline fell substantially, since the compression step alone removed the growth curve that had made deep research sessions feel expensive in v1. Output validation errors at the enrichment step decreased significantly, a direct result of the enrichment component receiving clean, pre-resolved inputs rather than raw research state.
The cost headroom this created let us increase research volume without changing infrastructure spend. And the quality improved too, across the fields we track, v2 improved prediction accuracy by around 10% over v1. That gain came from several changes working together, the greater research depth, the smaller, more focused context window and the specialist components each handling a narrower task.
There was one honest trade-off: speed. v1 was fast, with most profiles completing in under a minute. v2’s lighter, cheaper model trades some of that raw throughput for better cost efficiency and more consistent output quality. For a batch enrichment pipeline running at scale that’s the right call, and not really a trade-off at all, since we can run more enrichments in parallel to achieve the same throughput. But it’s worth naming: if latency is a constraint, the architectural gains come with a speed cost attached.
Beyond Journalist Profiles
v1 was purpose-built for one task: researching and enriching journalist profiles. v2 was designed from the start to research and enrich any kind of profile, not just journalists.
The pipeline structure is shared infrastructure. Individual teams register their own profile type by providing a schema, a set of prompts, tools, and a configuration, and the platform handles the rest. The research loop, the compression step, the review logic, and the enrichment step are all reused. Only the domain-specific inputs and outputs change.
Because each component in the pipeline is independently configurable, teams can go further than swapping prompts. They can define a separate workflow for each component: different search strategies for the research step, different validation rules for enrichment, and different review criteria depending on what fields matter most for their use case. A profile type that requires heavy social verification can run a research component tuned for that, while one that prioritizes publication history can weight its review step accordingly. The pipeline coordinates, and the components specialize.
The same architecture that keeps journalist profiles current in the Media Database now also researches and enriches other profile types across Meltwater’s data platform, each running through the same pipeline with its own configuration.
That extensibility is only possible because the components were cleanly separated. If research logic, output logic, and domain-specific rules were still entangled in one loop, none of them could be swapped out independently.
Wrap Up
The core lesson from v1 to v2 wasn’t really about AI, it was about software design.
Mixed responsibilities make systems hard to improve. State that grows without bound gets expensive. Components you can’t test in isolation are components you can only debug in production. None of these are new lessons. What surprised us was how quickly they surfaced in an AI context, and how much they compounded once we were running at scale.
Admiral v2 is live. Individual components have been improved, re-prompted, and replaced without touching the rest of the system. New entity types have been added without rebuilding anything. The architecture has held up, and that, more than any single metric, is the thing we’re most glad we got right.