I built a controlled workflow around local models, tested them on real application tasks and compared them with a hosted model completing the same tasks with human feedback. The models could produce useful code. They could not reliably close the software-engineering loop.
Local models are now big enough, fast enough and specialised enough that “run your whole engineering workflow on your own hardware” is now plausible. To find out whether this is indeed possible, I set out to actually build something using a local model… a real application called CoinVault: FastAPI backend, PostgreSQL, SQLAlchemy, real authentication, an external API integration, a Vue frontend, Docker-based development with automated tests throughout.
The plan was simple. Break the build into small, well-specified tasks. Let a local model drive the implementation. Write no application code by hand. See how far that gets me.
It got further than I expected on code. It got nowhere on autonomy.
Can a local model write code?
The first test was narrow and specific. A real, non-trivial application could be built end to end by Aider driving local 14B models (qwen3:14b, qwen-coder-16k) on 16GB VRAM, with no hand-written application code, when given tasks small enough for the model to actually hold in its head.
This test produced a negative result. On the auth test-backfill task, qwen-coder-16k was given the exact fix, verbatim, in the same message as the failing traceback. It didn’t apply the fix. It deleted the failing line rather than substituting the correction. Repeated attempts (architect model on, off, with a conventions doc, without one, with mechanical test-gating, with direct human diagnosis) all failed the same way, suggesting a capability ceiling rather than context or formatting problems.
That result changed the question I was asking. No longer: can a local model write code? This test answered that: mostly yes, for well-scoped tasks. End phase 1.
The next question became: can a local model close the software-engineering loop autonomously? Scope the work, implement it, run the tests, diagnose a failure, fix the actual cause rather than the symptom, retest and hand off something that survives an independent review. No human intervention required.
Redesigning the experiment
Aider had been doing too much orchestration in Phase 1 to answer that question cleanly. Phase 2 split the roles:
- OpenCode is the worker. It scopes, implements, tests, self-fixes, writes its own progress entry and stops before committing.
- Aider is the final gate, not the orchestrator. It re-runs the recorded test command independently and never trusts a reported PASS. It reviews the diff for scope creep or weakened assertions, scores the result against a fixed rubric and only then commits.
- The human’s job is defined narrowly: write the task specs, confirm install and migration commands, correct a task spec if it’s actually wrong and decide whether a task needs splitting. Not to write application code. Not to hand-fix a failing implementation. Not to quietly repair a test to get to green.
This last boundary is vital. If a human does end up fixing something, these are important data. They need to be logged, specifically what the model got wrong and what the fix was, and not just as a silently-corrected result. A local model getting help it doesn’t ask for and doesn’t record other than as “Human intervention: yes/no” isn’t the same thing as a local model working alone.
Why the testing architecture became the whole story
The backend tests should run against a real, permanently-running Dockerized backend-test service and not an in-process test client. Real PostgreSQL. Real authentication. pytest runs in one process. The uvicorn server runs in another. Two separate programs, running at the same time, each with its own loaded copy of the code. That sounds like a small implementation detail but it was actually the source of most of the bugs in this project.
Patching, overriding or resetting anything in the pytest process’s copy of a module has zero effect on the server actually answering the HTTP request. This was found independently four separate times across the project: app.dependency_overrides on get_current_user; unittest.mock.patch on route internals; monkeypatch on a service class; and a rate limiter’s .reset() that only ever reset pytest’s own imported copy. Every one of those produces a test that “passes” for the wrong reason. Or fails in a way that looks like something else entirely.
This is why “write a pytest” was a much more complex instruction than you might think. A model attempting one of these tasks wasn’t just generating assertions. It had to understand:
- which of two running processes it was actually talking to
- which fixture provided real authentication versus a fake
- why a mock that looks correct doesn’t touch the thing being tested at all
That failure of understanding is where the agentic weakness actually showed up, not in basic syntax or API calls.
The benchmark tasks
| Task | What it tested | Result |
|---|---|---|
backfill-03-auth-tests |
Writing tests against already-built auth functionality, reusing existing fixtures rather than inventing new implementation | No tested local model completed it |
04-storage-locations |
Implementing a CRUD feature plus a recursive-CTE tree query, with matching automated tests | Implementation was partially achievable. The test layer was the consistent failure point across every autonomous attempt |
backfill-03 was deliberately the “easier” task. Auth was already built, the acceptance checklist already existed in 03-auth.md and the job was to turn each bullet into a real assertion using fixtures that already existed. No new production code needed. Every autonomous attempt still failed: either by never resolving a fabricated import; or by drifting through the exact cross-process mocking anti-patterns that BACKEND-TEST-RULES-CORE.md explicitly bans.
04-storage-locations asked for CRUD routes plus a recursive CTE returning each node’s computed path, with a matching pytest file and a pytest-bdd scenario. Under OpenCode, the router and the CTE were written as if by two people who never spoke to one another. The CTE existed, but nothing called it. The test file used TestClient(app) and a backend.-prefixed import, both listed as bad examples in the rules doc it had been required to read. IDs were typed as plain integers against a schema that specifies UUIDs everywhere. Under Aider, running the identical model on the same task, the anti-pattern discipline actually improved. But the task’s actual point, the recursive CTE, was dropped entirely in favour of an invented Django-style .get_all() method that doesn’t exist anywhere in the codebase. Failure for different reasons but with the same underlying story. The model could produce plausible code and plausible tests without either one being correct. And without noticing.
Models tested
| Model | Role in the experiment | Observed result |
|---|---|---|
qwen3:14b |
Primary Phase 1/Phase 2 worker | Useful implementation, unreliable autonomous repair, hit a hard capability ceiling on test-writing tasks |
qwen-coder-16k |
Phase 1 editor model | Given the correct fix verbatim, still failed to apply it correctly |
deepseek-r1:14b |
Tested locally alongside Qwen | Same general limitation on multi-step diagnosis |
Devstral-Small-2-24B |
Selected as the Phase 2 “stronger” worker | Unusable in practice. Roughly 3 tokens/second on the available hardware. Stopped after 180 minutes across two loops once progress stalled. Never finished a task |
qwen3-coder:30b |
Reviewer role | Not the primary worker. Used for review, not autonomous implementation |
| A hosted model (Gemini 3 Flash Preview, via Junie) | Independent comparison point, same tasks | Reached a result close to the human+Claude-assisted outcome, with human feedback in the loop |
I’d like to be precise about what “24B was unusable in practice" means here. It’s a hardware and throughput finding, not a claim about the model’s reasoning ceiling. At roughly 3 tokens per second, Devstral ran for 180 minutes before I stopped it. I let it go as long as it kept showing visible progress, gave it two loops at that pace and then it was pointless to continue. A task a developer would finish by hand in under six hours simply never got done. The run was abandoned for exhausting my patience. Devstral and the 30B-class DeepSeek both hit this wall before they hit any interesting capability boundary at all.
What the 14B models could actually do
They were useful within a bounded scope:
- Generate application code for reasonably well-specified backend functionality.
- Produce a workable first-pass implementation from a precise task description.
- Operate effectively as a coding assistant on small, self-contained pieces of work.
But they struggled consistently with everything downstream of “the first attempt didn’t work”:
- Understanding an existing test architecture well enough to reuse it correctly.
- Distinguishing a fixable symptom from the actual underlying mistake.
- Correcting the real problem after a failed run, rather than patching around it.
- Holding a consistent approach across several repair iterations.
A failed session follows a consistent sequence. The model writes a test. The test fails. The model changes something. A new failure appears in a different place. The model changes something else. The original conceptual mistake, e.g. a mock that never touches the server, an ID type that doesn’t match the schema, a CTE nobody calls, is still sitting there untouched three iterations later. More instructions didn’t fix this reliably. They sometimes produced more iterations. They didn’t produce a more correct diagnosis.
Why more retries didn’t help
Runs were allowed roughly three automatic self-fix attempts, occasionally extended with additional instructions for another few rounds. Without this cap, a 14B model would stay in a self-fixing loop indefinitely.
A repaired attempt would often touch peripheral code while leaving the core mistake in place. Each new attempt carried its own chance of introducing a fresh error on top of the one that was never actually fixed. More retries did not translate into more agentic capability. In more than one case, retries simply increased the amount of wrong code sitting in the diff.
The model failed to recognise that its current approach was conceptually wrong.
The comparison that didn’t let the models off easy
I ran the same two tasks, backfill-03 and 04-storage-locations, through Junie using Gemini 3 Flash Preview, each iteration starting from the same clean point and its own branch, the same way every local run did. This wasn’t a same-conditions benchmark. Junie had a human in the loop giving feedback between iterations, where OpenCode was explicitly built to close its loop alone.
Each task took roughly 20 minutes. That included a handful of back-and-forth rounds where I pointed out what was wrong and redirected the model back to the rules it had already been given, rather than it finding the mistake unprompted. With that steering, it eventually got there.
That result was close to the outcome that eventually resolved 04-storage-locations for real. A human-assisted, interactive session (Claude plus a human, working the branch directly rather than running the autonomous OpenCode/Aider loop) that reached a real passing state. Along the way, a pre-existing security bug surfaced that the fully autonomous attempts never got close enough to find.
This bug is worth exploring as it’s the best evidence in the whole project that understanding the infrastructure and knowing how to test it are separate skills from coding. auth/routes.py imported verify_otp, then defined its own route handler with the exact same name, silently shadowing the import. The internal check if not verify_otp(code, hash) was therefore calling the route handler on itself (never awaited, a coroutine object, always truthy). This meant any OTP code was accepted for any pending login. This bug predates this entire experiment. Phase 1’s original auth tests never caught it because they mocked verify_otp directly instead of hitting the real endpoint so it was uncovered by its consequences rather than its mechanism.
No autonomous local run got far enough into the real test infrastructure to find this bug. It took someone actually understanding the fixture architecture well enough to rebuild it, twice, before it held.
It’s worth being precise about how badly this went, rather than just noting that it went badly. This wasn’t a case of the model lacking information. The task file, the rules doc and the real traceback were all available in context and none of that was enough for a local model to reason its way to the actual cause. Some sessions didn’t even fail in an interesting way. They failed in a mundane one, like a model repeatedly re-writing the identical import statement 20 times in a row because it had convinced itself a library wasn’t available, rather than checking. The model persisted in rephrasing the same wrong guess instead of stepping back to question its initial assumption.
Why “it can code” doesn’t mean “it can be the engineer”
The difference is in the number of steps, not just the difficulty.
A coding benchmark:
- Problem.
- Prompt.
- Expected output.
- Generated code passes. Done.
Agentic software engineering:
- Understand the current state of a real repository.
- Select the relevant files.
- Interpret an existing test architecture.
- Make a change.
- Run the real tests.
- Interpret a real failure.
- Identify the actual root cause.
- Change approach if the first one was wrong.
- Rerun.
- Verify.
- Document.
- Know when to stop.
Four steps against twelve. And steps 4 to 9 must repeat correctly.
A model can be good at coding and fail badly at software engineering. That’s not a contradiction. They’re different capabilities.
What actually works locally
The claim that the agentic models I tested can engineer didn’t hold up. The narrower claim that these models can code did. This is a useful result.
I found that the optimal working setup is an orchestrated chain of specialised 14B models, each with one bounded job, running in OpenWebUI:
- Planner (
qwen3:14b, 16K context, temperature 0.7). Turns an idea into an actionable plan. - Solution architect (
qwen3:14b, 16K, temperature 0.5). Designs the system before anything gets implemented. - Design assistant (
qwen3:14b, 16K, temperature 0.7). Icons, SVG, UI theming, visual decisions. - Architecture reviewer (
deepseek-r1:14b, 16K, temperature 0.4). Challenges the design before code exists. - Coder (
qwen-coder-16k, top_p 0.95, temperature 0.2). Small, well-defined, fire-and-forget coding tasks. Not autonomous full-system development. - Code reviewer (
qwen-coder-16k, same settings). Reviews what the coder produced. - Documentation assistant (
qwen3:14b, 16K, temperature 0.4). Project documentation.
Each stage gets a bounded problem and a structured input from the previous one. No stage has to maintain a long, evolving mental model of a repository. No stage has to find context it wasn’t given. A failure at one stage is visible before it reaches the next.
The same models, in the OpenCode/Aider harness, could not identify a failure at a specific stage. This is not because the harness is badly built. AGENTS.md and AIDER.md’s constraints exist precisely to make failures attributable to the model rather than to an uncontrolled workflow. It’s because the harness demands something the pipeline never asks for: reasoning, inspecting, modifying, executing, interpreting tool output, recovering and maintaining state, all in one continuous session, with much heavier demands on tool use and error recovery than a sequential pipeline ever makes.
What this project does and doesn’t show
It shows that:
- 14B local models can produce useful application code for appropriately scoped tasks.
- They struggle badly and consistently with autonomous debugging, test comprehension and recovering from a conceptually wrong first attempt. Increasing context and retry budget didn’t reliably fix this. It sometimes just produced more iterations of the same underlying mistake.
- 24B/30B-class models introduced a severe practical cost (throughput) on the tested hardware, independent of whatever capability gain they might have offered.
- A specialised, sequential, multi-model local pipeline is genuinely useful for real development work.
- A hosted model with human feedback in the loop reached roughly the same outcome as a human working directly with Claude. Both closed the loop. Nothing tested fully autonomously did.
It does not show that:
- All 14B models are incapable of agentic coding.
- OpenCode or Aider are bad tools.
- Any hosted model is universally better.
- There’s some universal size threshold above which this problem disappears.
This is simply one project, one codebase, one set of hardware constraints and a handful of models.
Some demonstrations of “running an entire company locally” quietly combine two different claims.
- That a sufficiently capable local model can write code and
- That the same model plus an agent framework can operate as the engineer running the whole process.
This project tested that second claim directly, with small, explicitly specified tasks, real automated tests, increased context and repeated repair opportunities. The most favourable conditions I could reasonably set up. It didn’t hold.
The actual bottleneck wasn’t first-pass code generation. Models implemented parts of 04-storage-locations without much trouble. The bottleneck was recovery. Existing fixtures were already there, the testing infrastructure was already documented, the correct pattern already existed in the codebase and the models still repeatedly failed to identify why their own attempt was wrong. That’s a stronger and more interesting finding than “the code was bad”, because in most of these runs the first-pass code wasn’t actually the main problem.
In summary
| Capability | Local models, in this experiment |
|---|---|
| Generate application code | Yes |
| Implement small, scoped tasks | Yes |
| Implement portions of a larger task | Yes |
| Produce architecture/design suggestions | Yes |
| Review code | Yes |
| Write documentation | Yes |
| Understand existing test architecture | No, unreliable |
| Correctly reuse complex fixtures | No, unreliable |
| Diagnose a failing test’s real cause | No, unreliable |
| Recover from a conceptually wrong attempt | No |
| Autonomously iterate to a real passing state | No |
| Run at usable speed at 24B+ on my hardware | No, abandoned on throughput alone |
| Sequential multi-model local pipeline | Yes, and genuinely practical |
| Replace the full engineering loop, alone | No, not demonstrated |
What I’d suggest
If you have local 14B-class models, use them for planning, architecture, design assistance, code review, documentation and small, bounded, fire-and-forget coding tasks. Don’t expect the same model to hold a large repository’s context, autonomously implement a feature end to end, write its own correct tests, diagnose why they failed and keep going until it’s actually right. That’s a different job requiring a set of skills that these models lack.
The CoinVault build is ongoing, using a mixture of agents doing the parts that they are actually good at.