Running 30 agents in production: what it actually takes

Anyone can demo an agent. Keeping a fleet of them alive for months, with real permissions and real consequences, is a different job, and almost nothing published about agents describes it honestly. This is the teardown of the Nerve Center, a 30-agent AI organization I designed, built, and operate in production: the architecture, three failures with root causes and the systemic fixes, how changes get reviewed, what it costs in real dollars, and the one constraint that makes the whole thing safe to hand authority.

The build itself stays private because it runs my actual life and work. Everything below is the real operating record, sanitized only where it has to be.

The architecture

The system in one sentence: a supervising daemon owns long-lived agent sessions, a message bus routes work between them and me, a shared memory graph gives them recall, Postgres holds durable state, and a watchdog layer assumes everything above it will eventually fail.

The pieces, briefly

• Supervising daemon and sessions. Every agent session runs as a PTY child of one supervising daemon. That gives clean lifecycle control and one place to enforce policy, and it creates the exact blast radius described in Failure 1. Architecture is a set of tradeoffs you choose on purpose; this one I keep because the failure is now machine-guarded.
• Message bus. A Python bridge routes inbound requests (chat, schedules, webhooks) to the right session and streams results back to the cockpit. Agents also message each other through it by name, so a domain specialist can ask the CEO agent for a decision instead of guessing.
• Memory graph. A vector store plus a graph store, queried together: semantic search finds candidate facts, one-hop graph expansion pulls in connected entities and decisions. Agents retrieve before they act. Without this, 30 agents are 30 goldfish.
• Durable state. Postgres on Supabase, 250+ tables: workflow runs, task ledgers, approval queues, cost records. If it matters, it lives in a table, not in a context window. Sessions rotate; state does not.
• Sync layer. The cockpit syncs open sessions across devices with per-entry tombstones and semantic writes rather than state blobs. Failure 2 explains why that distinction is load-bearing.
• Watchdog layer. Detectors that measure work completed, not process uptime: a death-cluster detector, an auto-unwedger for stuck prompts, stale-job alerting with catch-up semantics after reboots. The design assumption is that every layer above will eventually lie about its own health.

Failure 1: one restart killed 12 sessions at once

Every agent session runs as a child of the supervising daemon. One agent restarted that daemon with a raw process-manager command instead of the guarded reload script. The OS killed the whole process group, and all 12 live sessions died within the same two seconds, including the one that issued the command.

The root cause was not the command. It was that the safety check lived inside one script on the honor system, and nothing stopped an agent from going around it. An agent under token pressure or a confused plan will eventually take the shortest path, and prose instructions are not a barrier.

The fix came in three layers: a pre-execution hook that blocks raw restarts of protected services outright, reload scripts that refuse to run while sessions are live and print the casualty list if forced, and a detector that watches for multi-session death clusters and names the exact culprit session and command within five minutes. The postmortem is a file in the repo, and the rule it produced is enforced by a machine, not a memo.

Failure 2: a stale phone tab resurrected dead sessions

The cockpit syncs open sessions across devices. A phone tab left open overnight held an old copy of that state, and its writes were last-write-wins. So it kept reviving sessions I had already closed and killed: zombie entries that burned tokens, confused the fleet, and reappeared no matter how many times I killed them.

This is a classic distributed-systems bug wearing an AI costume. The moment two devices can write the same state, a full-blob write is a time machine: the staler device overwrites the newer truth. The fix was to stop syncing state and start syncing intent. Per-entry tombstones make a close a fact that cannot be overwritten by anything, and semantic writes mean a device can only add what it knows, never erase what it missed.

The regression suite now pins that exact interleaving, a stale reader racing a newer writer, so the bug class stays dead rather than the bug instance. UI changes run end-to-end on phone and desktop viewports, because this failure taught me the device matters.

Failure 3: a confirmation modal froze the org for 4.5 hours

The CLI under one daemon hit a usage limit and popped a confirmation modal. The daemon's turn loop sat waiting on a prompt no human would ever see, and inbound messages queued invisibly for four and a half hours. Nothing crashed, so nothing alerted.

That is the nastiest failure shape in agent systems: alive but wedged. Process supervision says healthy, the port answers, and no work is happening. Any liveness check based on uptime is blind to it by construction.

The fix was a watchdog that detects the stuck state and auto-confirms the modal, plus liveness checks that measure output: messages processed, work completed, state advanced. The same principle got applied fleet-wide, because a relay once wedged the same way on a dead database socket and silently swallowed outbound messages for hours. Now anything that can wedge has a detector that asks "did you do work?" instead of "are you running?"

How changes get in

Nothing merges on one agent's word. Every substantive change goes through adversarial review by three separate agents with different instructions, one of which is explicitly told to try to reject the change. Then CI gates: a QA gate, an architecture gate, and a unit-test suite. Any bug that reached production gets a fixture test reproducing the exact failure interleaving before the fix lands, which is how the failures above became regression suites instead of stories.

The same pipeline governs the autonomous work. An agent that fixes a bug does it in an isolated worktree, proves it with a failing test that passes after, and clears blast-radius caps and a deny-list before merge. I open-sourced that engine because the governance model is the interesting part.

What it costs

Real numbers, because "AI is expensive" and "AI is cheap" are both lazy. A heavy build session runs $30 to $35 in API cost; two recent ones came in at $34.67 across 201 model calls and $32.13 across 188. A five-reviewer audit panel is roughly 300k tokens. A rote heartbeat that formats a status report costs pennies on a small model.

So the system tiers cost by work type: frontier models for building, reviewing, and deciding; small models pinned to rote jobs like heartbeats and report formatting; local models for bulk extraction where latency and privacy beat quality. Cost is tracked per run in the same tables as everything else. The question is never the bill. It is whether the work earns the model.

The constraint I chose on purpose

The system can draft anything: email, messages, posts. It can send nothing. Every external message routes through one gated chokepoint with a global freeze switch and single-use approval tokens, and it transmits only after an explicit yes from me. I built it that way after watching how many places an autonomous system will grow an ungated send path if you let it; an audit found five separate paths that could have sent email without approval, all of them added for convenience, none of them malicious. They are all closed, and the gate is enforced at the same pre-execution layer as the restart guard.

That principle sits under every design decision here. Capability is easy to add. The stop conditions, the tombstones, the freeze switch, the deny-list, the liveness detectors: those are the difference between a demo and something you can hand real authority.

If you are building one of these

• State outlives sessions. Anything that matters goes to a table or a file, never to a context window.
• Sync intent, not state. The moment two writers exist, blob sync is a time machine. Tombstones and semantic writes.
• Liveness is work completed. Uptime checks are blind to the wedged state, and the wedged state is the common one.
• Enforce with machines. Every rule that matters gets a hook, a gate, or a detector. Prose is documentation, not enforcement.
• Gate the sends. One chokepoint for everything outbound, with a freeze switch you can hit from your phone.