Systematic Debugging: One Root-Cause Discipline, Two Very Different Patients

Writing code is where agent discipline shows off. Debugging is where it shows its true colors. The moment something fails unexpectedly — a test goes red, a container balloons, a deploy half-works — is precisely the moment the tempting shortcut appears: change something, anything, and see. The Superpowers skill library treats that moment as the load-bearing test of engineering discipline, and it handles it with a single root-cause philosophy pointed at two very different patients. systematic-debugging investigates your software. diagnosing-superpowers investigates the agent itself, using the same evidence-before-action rule. Together they form one of the most visible cross-skill themes in the library: no fix, no finding, no claim without evidence first.

This theme article is about how that discipline behaves under real conditions — on a live Docker host named SYNAS running 32 containers across 26 networks, and inside agent transcripts where the "system under test" is the agent's own behavior. The individual skills have their own deep dives (systematic-debugging and diagnosing-superpowers); what follows is what you see when you step back and look at the pattern they share.

One discipline, two targets

The two skills look unrelated at a glance. One is a four-phase debugging workflow with an iron law printed at the top of the file — NO FIXES WITHOUT ROOT CAUSE INVESTIGATION FIRST. The other is a forensic procedure for answering "what the hell did this session just do?" — locate the transcript files on disk, analyze them, and report findings where every claim cites path:line. "No citation, no finding" is its core principle.

But read both past the headers and the shared skeleton is unmistakable. Both forbid acting before understanding. Both insist that observations come from running commands and reading outputs, never from memory or vibes. Both treat a confident guess as a red flag rather than a virtue. And both terminate not in "I fixed it" or "I figured it out" but in evidence another party can check — a failing test that now passes, a report section that links to the exact transcript line.

The theme is this: the same epistemics, applied wherever something misbehaves — whether the misbehaving thing is a game server on your NAS or a subagent that burned forty thousand tokens re-reading a file it had already read. When a library applies one discipline across that many contexts, the discipline stops being a technique and starts being culture.

The four phases in practice: SYNAS under audit

The systematic-debugging skill prescribes four phases that must run in order: gather evidence (root cause investigation), find patterns, test hypotheses, implement fixes. Here is what that loop looked like on the SYNAS box — a home-lab Docker host whose raw stats output offered guessing plenty of appetizing false leads:

$ docker stats --no-stream
CONTAINER      CPU %     MEM USAGE / LIMIT
icarus         4.87%     4.3GB
checkmk        12.88%    1.75GB / 6GiB (29%)
dayz           110.88%   7.2GB / 62.82GiB

$ docker ps --format '{{.Names}} {{.Ports}}'
vaultwarden        10.10.10.12:8222->80/tcp
icarus             0.0.0.0:17777->17777/udp, 0.0.0.0:27017->27017/udp
checkmk            0.0.0.0:8100->5000/tcp, 0.0.0.0:8101->8000/tcp ...

Phase 1 — gather evidence before anyone proposes anything

The obvious stories wrote themselves: icarus is leaking memory, checkmk is spinning, the DayZ server is eating the box. A guessing agent would have restarted all three by minute five. Instead, the audit collected facts first: host memory totals (41 GiB used of 62.82 GiB, 21 GiB available), swap pressure (2.7 GiB of 32 GiB — nearly idle), restart counts across all 32 containers over 48 hours (zero), and uptime (most services 12–13 days; icarus only 3). The vaultwarden line got its own evidence pass, too: a service meant to serve two networks showed a binding to one explicit interface — 10.10.10.12:8222 — not the wildcard bind its siblings use.

Phase 2 — find the pattern of "normal" before judging "broken"

Phase 2 asks what the working comparison looks like, and on this host the neighbors answered. A DayZ headless server with hyperthreading routinely reports over 100% of one core — 110.88% CPU is its normal operating band, not a fire. A checkmk monitoring poller sits inside its declared limits: 12.88% CPU, 29% of its own 6 GiB cap. A monitoring system consuming resources while polling 30 other containers is behaving like a monitoring system. On the networking side, the pattern comparison was even sharper: 31 of 32 containers published on 0.0.0.0; vaultwarden was the single exception, bound to an explicit IP. That one-line difference between working neighbors and the odd container out is exactly the "list every difference, however small" step paying rent.

Phase 3 — falsifiable hypotheses, tested one variable at a time

The live hypothesis for icarus was a memory leak, and the skill demands a falsifiable version: "icarus's 4.3 GB grows without bound until it dies." The evidence ranked that low — a process that had run three days with zero restarts, light swap pressure, and a host still holding 21 GiB in reserve looks like a cache-hungry service, not a dying one. The audit also caught a classic collapse: icarus had a real disk anomaly (1.87 GB written into its writable layer because no volume was mounted), and it is tempting to fuse that with the RAM number into one scary story. Two measurements, two different problems, two different severities — keep them apart and both shrink to manageable.

For vaultwarden, the ranked hypothesis was structural, not behavioral: "the service is unreachable from the second network because the compose file publishes it to a single interface, and that binding — not the firewall, not the app — is the root cause." One command confirmed it: the port line in docker ps already showed the explicit 10.10.10.12 bind. No log-diving required once you had read what the evidence was already saying.

Phase 4 — implement fixes at the root cause, or don't implement

The most instructive part of a disciplined audit is what Phase 4 does not do. The correct fix for icarus and checkmk was: nothing. Zero code changes, zero restarts. icarus went on a watch list with a monitoring recommendation; checkmk stayed exactly where it was. The genuinely actionable items were chores, not emergencies — prune 531 MB of build cache, remove one orphaned volume, clean four orphaned networks. The vaultwarden finding produced one real change: a publish binding that names the interface(s) the service is actually meant to serve, made in the compose file, where the root cause lives — not a firewall rule poked at the symptom. Had guessing driven the session, someone would have restarted production services, tuned limits that were never the problem, and destroyed the evidence any future incident would have needed.

The Rule of Three: the exit ramp from debugging

Every debugging process eventually faces the uncomfortable case where investigation is honest and fixes still keep failing. The skill's answer is its most distinctive rule: three failed fixes is not a fourth hypothesis — it is an architecture complaint. The telltale signature is specific. Each fix reveals a new shared-state or coupling problem in a different place. Each fix requires "massive refactoring" to implement. Each fix creates new symptoms elsewhere. When that pattern shows up, attempt number four is forbidden until you have discussed fundamentals with your human partner: is this pattern sound at all, or are we sticking with it through sheer inertia? Refactor the architecture, or keep patching symptoms — but decide deliberately, not by accumulating scar tissue.

This rule matters to the theme because it defines where systematic debugging hands off. Root-cause discipline is not a claim that every bug is one bad line away; it is a promise to know which kind of problem you are in before spending more changes. The same instinct shows up in diagnosing-superpowers as the mirror-image rule: report what the transcripts prove, name your involvement level, and stop — whether the fix belongs in a skill is the triager's call, not the forensics pass's. Both skills draw the line between "I have established the facts" and "I am now freelancing."

Diagnosing the agent: the same phases pointed inward

diagnosing-superpowers takes the whole apparatus and aims it at the agent's own sessions, and the phase-for-phase correspondence is close enough to read like a translation:

• Phase 1 (evidence) becomes intake and locate. "It took too long" is a complaint, not a problem statement, so the skill asks one question at a time until it can name the session, the turn range, the expectation, and the observable. Then it resolves the transcript files to verified absolute paths on disk — and creates a workspace directory under your home folder where the case file and all cited evidence live. Every number in the eventual report must come from the transcripts or from a command that was actually run; a figure from memory is an invented figure.
• Phase 2 (patterns) becomes the analyst fan-out. One subagent per dimension — skill timeline, plan adherence, repeated work, stumbles, quality evidence, request conflicts, cost and time — dispatched in parallel over the transcript, an explicit use of dispatching-parallel-agents inside a debugging workflow. Any returned finding without a path:line citation is discarded on arrival, which is pattern analysis with a quality gate: conclusions must be traceable to the working example of the actual session.
• Phase 3 (hypotheses) becomes the report's involvement verdict. The report states what happened, with receipts, and — this is the deliberately hard rule — stops short of diagnosing the skill library itself. No naming a defect, no proposing a change, even when your human partner is pressing for one. The evidence goes on the table; the judgment call about what to change stays with a human.
• Phase 4 (fixes) becomes the gated exits. If the report says a Superpowers bug is possible or likely, the next step is a GitHub issue — searched for duplicates first, drafted exactly, and created only after approval. If a bundle is requested, transcripts are scrubbed at an agreed level (skeleton, evidence-only, or full), audited by a second pass, and re-scrubbed until the audit returns clean, with a human reviewing every file before anything ships. Read-only on session files, always: the forensic pass must not contaminate the evidence it is analyzing.

What the inward pointing version adds, and what makes the theme richer than "evidence is good," is the discipline of scope. The four phases tell you when your fix belongs at a different level of the system — a different component boundary, or the architecture itself. diagnosing-superpowers encodes the same lesson socially: establishing evidence and deciding the change are different jobs, and confusing them is how "root cause discipline" degenerates into freelancing with a citation format.

Related Deep Dives in the Series

The theme does not stand alone — it is bolted to the rest of the series at obvious seams. Phase 4's "create a failing test first" is a direct handoff to test-driven development: the RED-GREEN cycle is the same hypothesis test run before the bug exists. The claim "the bug is fixed" is not self-certifying — that is verification-before-completion's job, which is why the audit ended with a fresh docker ps -a showing 32 of 32 containers healthy rather than with the sentence "looks good." And the parallel analyst fan-out inside diagnosing-superpowers is the same dispatching pattern the subagent-driven development deep dive covers, pointed at transcripts instead of tasks. Systematic debugging is less a solo skill than a public utility the other skills plug into.

• Superpowers Dispatching Parallel Agents — when parallel agents earn their overhead, and how to brief each lane.

Where guessing leaks in: the red-flag list

Both skills ship a list of thoughts that mean stop and return to the evidence. They are worth reading as a set because they describe the psychology of the shortcut, not its mechanics: "quick fix for now, investigate later." "It's probably X, let me fix that." "Add multiple changes, run tests." "Skip the test, I'll manually verify." And the two most dangerous of all: "one more fix attempt" when you have already tried two or three — that is the Rule of Three ringing — and any finding you cannot cite. The human partner's redirections count too: "Stop guessing," "Is that not happening?", and "Will it show us...?" — which really means you should have added instrumentation instead of assuming.

The theme in one sentence

Whether the patient is a 4.3 GB game-server container or a subagent that went in circles, the Superpowers answer is the same ten words: find the root cause before touching anything, and cite your receipts. The individual mechanics live in the deep dives — Systematic Debugging and Diagnosing Superpowers — and the rest of the series is in The Skills We Use index.