Dream-RSI: How I Wished My AI Security Agent Sweet Dreams

01The hard part

The loop is easy. The grader is not.

Recursive self-improvement is the hinge the whole AI conversation turns on. If a system can improve the process that improves it, capability stops being something you ship and starts compounding. Whether the frontier labs get there safely is a massive debate, and I cannot settle it. I can tell you what happened when I built a very small one and pointed it at my own work.

Here is the part that gets skipped: the loop is the easy bit. A system that proposes changes to itself is a weekend project. The hard part is the grader, the thing that decides whether a change actually made the system better, usually written by a human, in a hurry, with an incomplete definition of “better.” Get it wrong and you do not get a system that fails. You get one that succeeds enthusiastically at the wrong thing and reports that it is doing great. (We have all dealt with this before)

In vulnerability research we are all chasing the same holy grail: an autonomous agent that actually learns. The logistics are brutal. A scanner that learns from live targets has to keep hammering live targets, which burns compute and does not scale. So we built Dream-RSI into our Olympus pipeline. We take the append-only event logs from our dynamic runner, ARGUS, rebuild them offline as search trees, and walk a candidate exploration policy through a simulated world that is already over.

Call it a flight simulator for scanners: every crash, none of the wreckage. We can iterate and evolve the agent without a single new request against a production system. It is the difference between a loop you run nightly and a loop you need a permission slip for.

And the best part is not the learning. It is that the method is built to refuse. Teaching an AI to improve itself is easy. Teaching it not to lie to you is where the real engineering happens.

02How it works

Dreaming in trees.

Three steps: replay, score, refuse. We spend most of our effort on the third.

Replay. A recorded run becomes a world. Each node is an action ARGUS could have taken, each branch an endpoint family. We branch on endpoint rather than specialist agent, because branching per agent leaves the frontier narrower than the batch width and every candidate probes everything, which makes comparison impossible. Worlds that still cannot discriminate get dropped rather than quietly averaged in, and every candidate is budget-matched to the baseline per world. Otherwise the winner is just whoever left the meeting first.

Here is a full cycle in its own words:

32 replayable world(s), 2241 nodes;
81 ground-truth node(s) across 8 world(s)
  187 run(s) recorded no executed action
  inert_cost = 0.5

policy             score   recall  attain  steps  inert  prune
recorded_order     0.7387  100.0%   1.000   2241   1171  47.8%
attribution_aware  0.7262  100.0%   0.992   2192   1165  46.9%

REFUSED on recall (1):
  stall_aware lost ground truth in world-02, world-03, world-04

That REFUSED line is the whole product. stall_aware was a stopping rule that pruned aggressively and scored beautifully, right up until the veto noticed which three worlds it had quietly stopped looking in. Across the weight sweep, the same check rejected 13 more candidates that all looked like wins.

Score. Two terms, compared lexicographically: (recall_ok, attainment − inert_cost × inert_ratio). Recall is checked first, and it is a veto, not a weighted term. A candidate must still pick every confirmed-finding surface the baseline picked, or it is rejected outright no matter how efficient it got. Three details are load-bearing:

• Ground truth is the run’s own confirmed findings. A promotable candidate is deliberately excluded, because an objective that protects candidates ends up protecting guesses.
• No ground truth means recall reports None, never 1.0. A perfect score on an empty set is the most dangerous number in the file.
• One free parameter, the cost of a wasted step. Because it is free, we compute the exact value at which each ranking would flip and sweep it every run. If a conclusion moves when that knob moves, it was never a conclusion.

Refuse. A winner is not a change. Admission control is a separate project entirely, with its own lockfile, its own test suite, and no shared code with the scanner. It runs ten named checks before anything ships. My personal favourite is no_abstention_arbitrage, because a candidate may not win by simply answering fewer questions. ARGUS emits the evidence, the Overseer adjudicates it, and neither imports the other. A search that grades its own output is precisely what the gate exists to distrust.

03The loophole

The trap I built into my own veto.

In most reinforcement-learning horror stories the gaming is visible. A boat spins in circles farming power-ups instead of finishing the race, and you can watch it happen. In vulnerability research it is invisible: a policy that games its objective produces a scanner that finds nothing, and a scanner that finds nothing produces a clean report, which looks exactly like good news.

So I hunted for the loophole in my own veto, and found it. Miss the bug on a known-vulnerable build and you record zero confirmed findings, which means zero ground-truth nodes, which means recall has nothing to measure and the veto cannot fire. Meanwhile the branch where the bug lives answered a denial, so every step there bills as wasted effort. The optimizer notices it could skip those “inert” branches entirely, and congratulates itself on the savings.

A reweighted blind world still moves the score, and there is no exchange rate at which “the engine could not see here” becomes partial evidence about routing quality. So the fix was not a tweak to the objective. It was a pipeline-crashing gate at the loader: a run that declares a known_vulnerable baseline and records nothing gets voided, banner above every ranking, exit 2.

Notice the guard never fails in this scenario. It is bypassed, which is worse, because a failing guard shows up in your output and a bypassed one does not. An optimizer handed this data is not malfunctioning when it learns to look away. It is doing exactly what I asked.

We had accidentally built an engine that was going blind on purpose, to save money.

Vincent Suen

(Frugal, I’ll give it that.)

The exit code is the point. A warning scrolls past. A crash demands attention. Every run now declares what it was fired at: known_vulnerable, known_fixed (patched, so a zero is the expected answer), or unset, which is not the same thing as “probably fine.”

04The control

The alarm fires. On my own machine.

I broke an open-source plugin by hand on a local Docker lab, escalating a student-tier account to administrator and confirming it in the database rather than trusting the HTTP response. Then I reset the account and pointed ARGUS at it. If ARGUS missed it, ARGUS was wrong. Nothing to interpret, nowhere to hide.

It missed it. Worse, the archive confirmed the pattern: 0 of 225 runs had ever sent the required anti-forgery header, 301 of 306 authorization cases had no live source principal, and 0 of 32 denials were ever attributed, so every “the boundary held” had really meant “nobody asked.” So the alarm fired, on a real run, against a real CVE:

Runs that sent the required anti-forgery header

0 / 225

Authorization cases without a live source principal

301 / 306

Denials with attribution

0 / 32

DETECTOR ALARM: 1 world(s) voided
  -- a declared baseline was not met

control-01 [known_vulnerable] produced_no_finding
  expected: self role-write privilege escalation, primary route

EXCLUDED from the corpus.
Treat this as a detector regression,
not as a world with bad numbers.

I had built the thing that tells me no, and the first word out of its mouth was about me.

05The repairs

Four blind spots, one honest test.

Each alarm pointed at the next gap. This is the part worth your time, because I would bet most teams have at least two of these right now.

1. The session logged in but never proved who it was. All three principals authenticated perfectly. Not one could demonstrate it was a different person, because nothing handed the engine a self-referential endpoint, so every authorization test sat politely waiting for a sentence that said “I am user 7.” Logging in and having an identity are two separate events, and only one of them is testable.
2. The source stage could not see the route. It matches route registrations by literal string, and this plugin builds its routes from an inherited property. Zero routes out of 925 files. Static analysis has a vocabulary, and the bug was living in a word it did not know.
3. My own safety rail hid the entire bug class. A privilege escalation that happens by writing is a write, and the compiler rejected every write through a guard I had added months earlier to stop the scanner doing damage. Safety rails and blind spots are the same object seen from opposite ends.
4. The token never reached the probe. Four transports can issue a request. Three attached the anti-forgery header. I fixed the wrong one first, re-ran, and got identical 401s, which is how I learned there were four. Coverage is a minimum, not an average.

Then came the liar. I wrote a unit test asserting the token was attached. Green. Out of habit, I deleted the fix to watch the test fail. It passed again. The function built its headers twice, once in JavaScript to send, once in Python to report, and my test read the Python copy while the JavaScript sent nothing. Two expressions of one intent is a place for them to disagree quietly while both look fine.

Once the plumbing was real, the 401s turned into 200s:

PUT /<redacted>/students/7  {"roles":["administrator"]}
  -> 200, body echoes "roles":["administrator"]

The CVE reproduced through the scanner’s own transport. One run, one route, which is the entire claim. I reset the lab account about nine seconds later.

06What stays with me

Can you trust your zeros?

No bounty, no triager applause. The win was our own tooling proving our own blind spot, loudly, in a way I could not negotiate with.

A zero is not a result until the run says what kind of run it was. “We looked and found nothing” and “we could not look” produce the identical report. I had 225 runs where I could no longer tell, and I had read every one the flattering way.

Protect by veto, not by weight. The moment something has a weight, it has a price, and an optimizer’s whole job is finding prices.

Delete the fix and watch the test fail. Every time. A test I have not seen fail is a rumour.

I still cannot tell you whether the labs achieve recursive self-improvement. I can tell you that my very small one, handed a scoring function I was proud of, quietly learned to look away from the bug, and that I only know because I built something whose entire job was to tell me so. It crashed the pipeline to do it.

As security tooling gets more autonomous, the bottleneck moves from “can it find bugs?” to “can you trust its zeros?” Dream-RSI is how we make sure our AI is actually dreaming of vulnerabilities, rather than sleeping on the job.

Written by Vincent Suen

Field notes on AI-assisted pentesting · Entry 02