The Superpowers Systematic Debugging Skill: A Deep Dive

Every engineer has watched the guessing spiral: a bug appears, someone changes a config value "just to see," nothing changes, then a second change, then a third — and an hour later the system is in a worse state than when you started, with three unexplained edits to unwind. The superpowers systematic debugging skill exists to end that spiral. It is a disciplined, four-phase root cause analysis workflow that forbids fixes until the root cause is understood — and in practice it is faster than guessing, even under deadline pressure.

In the Superpowers skill library for AI coding agents, this is the skill that governs every technical problem: test failures, production bugs, build breakage, performance regressions, and integration weirdness. Its one non-negotiable rule is printed at the top of the skill file:

NO FIXES WITHOUT ROOT CAUSE INVESTIGATION FIRST

If you have not completed Phase 1, you cannot propose fixes. Not "shouldn't" — cannot. The rest of the skill is machinery for making that rule practical rather than precious: a tight evidence-gathering loop, a comparison method, a hypothesis discipline, and a stop sign for the moment when the real problem turns out to be architectural.

What the skill actually does

The core principle is one sentence: always find root cause before attempting fixes; symptom fixes are failure. That sounds obvious written down, which is exactly why it needs to be enforced as process. Under time pressure, when "just one quick fix" looks obvious, guessing feels productive because it generates motion. The skill converts debugging from improvisation into the scientific method, with four phases that must be completed in order.

It applies to any technical issue, and the skill is explicit that you use it especially when you are tempted not to: when there is an emergency, when the issue "seems simple," when a manager wants it fixed now, or when you have already tried a couple of fixes and neither worked. Those are precisely the situations where thrashing is most expensive.

Phase 1: Root cause investigation

Phase 1 is a tight loop of evidence gathering, and it starts with the step everyone skips: reading the error message. All of it. Stack traces get read completely — line numbers, file paths, error codes — because error messages very often contain the exact solution in embryonic form.

Then three more questions, in order:

• Can you reproduce it consistently? If not, you gather more data, you do not guess. An unreproducible bug is an invitation to fix the wrong thing.
• What changed recently? Git diff, recent commits, new dependencies, config edits, environment drift. Most bugs have a birth date.
• Where exactly does it break? In multi-component systems — CI into build into signing, API into service into database — you add diagnostic instrumentation at every component boundary, run once, and let the evidence tell you which layer fails before you touch anything.

For a bug that surfaces deep in a call stack, the skill prescribes backward data-flow tracing: where did the bad value originate? What called this with the bad value? Keep walking up the chain until you reach the source, then fix at the source, not at the symptom.

Phase 2: Pattern analysis

Once you know what is broken and roughly where, Phase 2 asks what is working. Locate similar code in the same codebase that does what you want. If you are implementing a known pattern, read the reference implementation completely — not skimmed, every line — then diff working against broken and list every difference, however small. "That can't matter" is banned reasoning; partial understanding of a reference is exactly how pattern-imitation bugs are born. The phase ends when you can name the differences and state the dependencies the broken thing expects: settings, config, environment, assumptions.

Phase 3: Hypothesis and testing

This is the scientific method, applied without theater. You form hypotheses that are specific and falsifiable — "I think X is the root cause because Y" written down, never a vague vibe — keep them ranked by likelihood, and test one at a time with the smallest possible change. One variable, one test, read the result. If the top hypothesis dies, you move to the next ranked one; you do not stack a second fix on top of the failed first one, which destroys your ability to attribute anything.

The phase also licenses the most under-used sentence in engineering: "I don't understand X." The skill says say it, don't paper over it — research more or ask for help.

Phase 4: Implementation

Only now does anything get changed, and even here the discipline holds:

1. Write a failing test first. Simplest possible reproduction — an automated test, or a one-off script if there is no framework. This is where the skill hands off to test-driven development: the failing test is the proof you had a bug and the proof you fixed it.
2. Make a single fix at the root cause. One change. No "while I'm here" improvements, no bundled refactoring — attribution requires isolation.
3. Verify: the new test passes, no other tests broke, the issue is actually resolved. Then stop claiming and check, which is the job of the verification-before-completion skill.

And if the fix doesn't work: stop, count your attempts. Fewer than three means return to Phase 1 and re-analyze with what you just learned. Three or more means something else entirely.

The Rule of Three: when to question architecture

The most distinctive rule in the skill is the "question architecture" gate. If three or more fixes have failed, you are no longer in hypothesis territory — you likely have the wrong architecture. The telltale pattern is specific: each fix reveals a new shared-state or coupling problem in a different place; each fix requires "massive refactoring" to implement; each fix creates new symptoms elsewhere. When that pattern shows up, a fourth attempt is not a failed hypothesis, it is the architecture telling you the pattern itself is unsound. The skill requires you to stop and discuss fundamentals with your human partner — refactor versus continue patching symptoms — before anyone touches the code again.

A real example: the SYNAS Docker audit

Here is what the process looks like on a live system. During a Docker infrastructure audit of a home lab server ("SYNAS" — OpenMediaVault, 32 containers across 26 networks), the raw docker stats output raised two red flags that guessing would have "fixed" immediately:

$ docker stats --no-stream
CONTAINER      CPU %     MEM USAGE / LIMIT
icarus         4.87%     4.3GB
checkmk        12.88%    1.75GB / 6GiB (29%)
dayz           110.88%   7.2GB / 62.82GiB

$ docker inspect icarus --format '{{.SizeRw}}'
1.87GB   # writable layer, no volume mounted

Phase 1 evidence first. The tempting guesses were obvious: icarus has a memory leak, checkmk is spinning, the DayZ server is eating the box. Instead the investigation gathered facts before theories: memory totals (41 GiB used of 62.82 GiB, 21 GiB available), swap pressure (2.7 GiB of 32 GiB — very light), 48-hour restart counts (zero restarts on all 32 containers), and uptime (most services up 12-13 days; icarus up 3).

Phase 2 pattern analysis. Compare against working neighbors: what CPU do comparable workloads legitimately burn? A DayZ headless server with hyperthreading routinely reports over 100% per-core CPU. A checkmk monitoring poller at 12.88% CPU and 29% of its own 6 GiB limit sits squarely in its normal operating band. Game servers — not infrastructure — were the top three memory consumers, which is exactly what a game-server host should look like.

Phase 3 hypothesis testing. For icarus, the live hypothesis was a memory leak. Falsifiable version: "icarus's 4.3 GB is a leak that grows without bound." The evidence ranked that low — a leaking process that had been up 3 days with zero restarts and comfortable swap headroom was behaving like a cache-hungry service, not a dying one. The writable-layer finding (1.87 GB written into the container filesystem because no volume is mounted) was real but explained a disk-usage concern, not a RAM one — two symptoms people collapse into one.

Phase 4 outcome. Zero code changes. The audit's recorded rulings: "dayz at 110% CPU is expected — DayZ head server with hyperthreading reports CPU usage exceeding 100% per-core. Not a concern unless it stays near 100% consistently over days." icarus went on a watch list with a monitoring recommendation, and the genuinely actionable items were chores, not emergencies: prune 531 MB of build cache, remove one orphaned volume, prune four orphaned networks. Had the guessing instinct driven the "fix," someone would have restarted production services, tuned memory limits that were never the problem, and destroyed the very evidence a future investigation would need.

That is the skill working on infrastructure: 32 containers audited, two false alarms dismissed with receipts, and the real findings separated from the scary-looking ones.

Red flags and rationalizations

The skill ships a list of thoughts that mean stop and return to Phase 1: "quick fix for now, investigate later," "it's probably X, let me fix that," "add multiple changes, run tests," "skip the test, I'll manually verify," and the most dangerous of all — "one more fix attempt" when you have already tried two or three. It also lists the signals a human partner sends when an agent has started guessing: "Stop guessing," "Is that not happening?" and "Will it show us...?" — the last one meaning you should have added instrumentation instead of assuming.

When investigation finds "no root cause"

Occasionally the honest answer is that the issue is environmental, timing-dependent, or external. The skill allows that conclusion — but only after the full process, and with homework: document what you investigated, implement appropriate handling (retry, timeout, clear error message), and add monitoring so the next occurrence arrives with evidence attached. Its correction on this point is worth memorizing: 95% of "no root cause" cases are incomplete investigation.

Related Deep Dives in the Series

Systematic debugging does not travel alone. The test-driven-development skill supplies Phase 4's failing test — RED-GREEN-REFACTOR is the same scientific instinct applied before a bug exists. Subagent-driven development matters at audit scale: when a debugging task fans out into independent investigations (storage here, network topology there), each subagent gets a focused, self-contained brief and its findings come back as evidence, not vibes. And when the skill you are debugging is the agent system itself, diagnosing-superpowers applies the same four phases to skill installation and activation failures.

Guessing is the most expensive kind of speed. The superpowers systematic debugging skill is a reminder, enforced as process, that the shortest path between "it's broken" and "it's fixed" runs directly through "here is why it was broken." Browse the rest of the series in The Skills We Use index.