Six Months of Agentic Development on Legacy Java: What Stuck

Six months ago I started seriously running coding agents on a 4,000-file Java 17 Spring Boot codebase. Not toy problems — real production tickets, security remediations, performance regressions, cross-repo migrations. This series documented the patterns that mattered along the way. This post closes it with a retrospective: what stuck, what I killed, and the numbers.

The one thing to take from this post: the patterns that stuck are the ones with measurable ROI. The ones I killed sounded clever but didn't survive a scoring harness. If you take one meta-lesson from six months of agentic development, it's this: measure, cull, keep.

The scorecard

Across the nine patterns in this series, three outcomes:

Kept (production-grade)

PatternMeasured ROI
Different critic (separate context)3× more defects caught at the gate
Context budget / 500-line skill limit36% median session token reduction
Rule layeringSilent overrides caught at CI, not in production
1,000-session eval harnessThree real regressions caught before rollout
2-retry limit42% median session cost drop, long-tail collapsed
Specialist checkpoints37% more real issues caught at review
Read-only subagents5× cheaper on exploration tasks, no accuracy loss
One topic per sessionTopics completed 3×, satisfaction nearly doubled
Rule compliance in-turnRule violations repeat-rate dropped to ~0

Every one of these has a measurement attached. The measurements are the reason they survived.

Killed (sounded clever, didn't work)

Three patterns I invested time in that I eventually deleted:

1. Multi-agent debate for every decision. The idea: two agents argue, a third moderates, best answer wins. Measured: 3× cost for 1.1× accuracy. For most tasks, the single-agent + critic pattern is strictly better. Debate helps on genuinely ambiguous open-ended questions; coding is rarely one of those.

2. Self-reflection loops. Agent produces output, agent reads it, agent iterates. Across measured sessions, the iterations usually didn't improve accuracy (Huang 2024 was right). I replaced self-reflection with the separate-context critic pattern.

3. Chain-of-thought-style scratchpads in every prompt. Making every agent "think step by step" sounded safe but doubled the output length on tasks where the answer was simple. I now reserve explicit CoT prompts for genuinely multi-step reasoning (planning, estimation).

Partially kept

Two patterns that work in some contexts and not others:

1. Specialist checkpoints for small codebases. Pays off at my scale (4,000+ files). On a 500-file project the generalist is still better; the specialists over-fire and nitpick. I keep the pattern but gate it on repo size.

2. Full-SDLC pipeline (investigate → analyze → architect → implement → test). Dramatic on large features. Overkill on bug fixes under 4 hours. I kept the pipeline but added a classifier up front that picks the recipe — bug fix vs feature vs hotfix — rather than running all stages every time.

The economics

Six months of running these patterns on a team of three developers:

MetricBaseline (no agents)After 6 months
Tickets completed per developer per sprint6.28.9
Agent token cost per developer per month$0$240
Average time to first PR on a new ticket2.1 days1.3 days
Code review turn-around (first comment)1.4 days0.4 days
Post-release defects per sprint2.31.1

Three things stand out:

  • Throughput up 44%. Not from "agents writing code"; from agents handling investigation, review, and documentation faster than humans. The humans still write most of the production code.
  • $240 per developer per month is the fully-loaded agent cost. For context: a quarter of the way to a day of senior developer time.
  • Defects down 52%. This surprised me. The pattern-review (specialists catching N+1s, the critic catching off-by-ones) does more for quality than I expected.

The unlock that mattered most

If I had to pick one pattern that enabled everything else, it's the eval harness. Every other pattern in this series was measurable only because the harness existed. Without it:

  • The "different critic" change would have been intuition, not evidence.
  • The context-budget trims would have been undetectable regressions.
  • The "2-retry limit" would have felt restrictive instead of saving 42%.

You can't optimize what you don't measure. The harness is the measurement surface. Build it first; everything else gets easier.

What I'd do differently if starting again

Three mistakes I'd avoid on a greenfield agent setup:

1. Build the eval harness in week one, not week six. I ran agents for six weeks without measurement and accumulated two months of unverified-by-anything intuitions. Half of them were wrong. I threw out a lot of work when I finally measured.

2. Keep AGENTS.md under 8 KB from day one. Mine grew to 24 KB before I audited. Trimming it back was painful because every section had been added for some real reason. Rejecting them in advance is cheaper than deleting them later.

3. Decide which model goes with which task from the start. I used the premium model for everything for the first two months. Switching to cheaper models for research cut costs in half overnight. Model choice is a decision; make it deliberately.

What might not transfer

Specific caveats before you copy the numbers into your own planning:

1. The 44% throughput number is on a team of three on legacy Java. It won't survive a cut-and-paste to a greenfield TypeScript shop. Patterns like context budget and the 2-retry limit transfer broadly; the specific throughput delta depends on how slow the previous baseline was.

2. The $240/dev/mo cost is at today's model prices and today's token efficiency. Both move. Any absolute cost number in this post is a snapshot — the ratio of improvement is more durable than the dollar amount.

3. Defects-down-52% rests on a specific review process. Teams that already run a strong code-review culture won't see the same lift from the specialist pattern — the baseline is closer. Teams coming from informal review see a bigger jump.

4. "Six months" is enough for me to form opinions, not enough to make them laws. Nothing in this series is settled. Each pattern has survived its measurements so far; the next six months may change the picture.

Where this is going next

The patterns in this series are stable; the ecosystem isn't. Three directions I'm watching:

  • Cheaper reasoning models. Haiku-class models already handle research. The next cost cliff is reasoning models cheap enough to use as the critic. That changes the economics of the gate pattern.
  • Code-local evals. Running the eval harness per-PR on the parts of the agent config the PR touches, not the whole thing. Faster feedback, less flake.
  • Deeper tool restriction. Read-only subagents at 5× cheaper is nice; what's a 50×-cheaper tool-restricted agent look like? I don't know yet, but it's where the next win probably is.

Closing

Agentic development isn't an experiment anymore for me. It's a toolchain, maintained like any other toolchain: measurements, budgets, specialists, retries, post-mortems. The weird part isn't that it works. The weird part is that nothing about making it work is magic — it's the same engineering practices that make everything else work. Measure, cull, keep.

If a specific pattern in this series was useful to you, tell me — especially if you've measured it in your own setup. The numbers above are mine; yours will differ. The ones that match are the ones to trust.

Attribution: The patterns and specific numbers in this post come from Devin with the ic-documentation agent framework my team maintains — not from an independent reproduction. See the series intro for the full disclosure. The patterns themselves generalize to any coding-agent setup.