Benchmarks
What we can prove, and what we cannot.
Myrqen finds real vulnerabilities and rarely invents one. It also misses more than half of what it should find. Both halves are measured below, with the method, the corpora and the raw output, so you can disagree with the conclusion rather than with the numbers.
Three of the four gates failed on the blind run.
From holdout-7: 64 cases nobody had tuned against, scored once on 2026-08-25. This is the clean measurement of generalisation. Judge Myrqen on it.
- Recall
- 0.250
- Gate missed
- Gate 0.90. 8 of 32 real vulnerabilities were reported.
- Critical / high recall
- 0.250
- Gate missed
- Gate 0.95. Severity did not help: the misses were not the small ones.
- False-positive rate
- 0.125
- Gate missed
- Gate 0.03. 4 of 32 files written to be correct, with the guard in place, drew a finding.
- Precision
- n/a
- Not computable
- Not computable on this corpus: precision is derived from exact class-name matches, and thirty independent authors named the defects in their own vocabulary. The safe-case false-positive rate is the precision signal that survives.
- Sink is a runtime primitive
- 0.875
- Recall over 8 cases where the value reaches fetch, a filesystem path, a shell, or eval.
- Sink is framework vocabulary
- 0.042
- Recall over 24 cases where the value reaches a framework's own database or identity call.
Three gates failed and one could not be computed. Recall where the dangerous value reaches a language or runtime primitive — fetch, a filesystem path, a shell, eval — against recall where it reaches a framework's own database or identity call. Server-side request forgery and path traversal generalise to an unfamiliar stack because fetch and path.join are the same everywhere; a SQL injection reached through db.rawQuery, em.getConnection().execute or strapi.db.connection.raw does not, and three were missed on a corpus with SQL in it. Coverage of a framework's own vocabulary is the binding constraint, not the quality of the rules. Every figure here is the static pass alone, scored one file at a time with no agent involved.
Two different things are measured, and they are not interchangeable
An assessment has two halves. Conflating them is the easiest way to publish a flattering number that means nothing.
| Measurement | What runs | What it can say |
|---|---|---|
| The static pass | The engine alone, 49 rules, each case scanned alone in an empty directory. No agent. | Deterministic and repeatable, so it can be measured against a held-out corpus. It cannot say what the host agent adds. |
| End to end | Static pass, then agent validation against a running application, then dedupe and report. The path a user gets. | Only measured against fixtures the rules were developed against. A regression gate, not evidence about unseen code. |
Every held-out number on this page is the static pass alone. No harness scores the end-to-end path on code nobody tuned against: the corpus cases are isolated files with no application to exercise, and the deterministic stand-in agent needs a running one. That is a gap in our evidence, not a detail.
What "blind" means here
Five architecture families — Firebase Cloud Functions v2 over Firestore, Azure Functions Node v4 over Cosmos DB, Strapi 5 over its Document Service API, Medusa 2 over MikroORM, Encore.ts over its own SQLDatabase — written in parallel by independent authors, each blind to the analyser and to each other. No family repeated a stack any earlier corpus used. The engine was frozen at a recorded source hash before authoring began, verified byte-identical immediately before scoring, and packages/scan was never opened, listed, or grepped.
The scanner was at commit 8c9937f, and the corpus was scored once, before anything changed in response. Fixing what a corpus found burns it: the engine has now been tuned against it, so it can no longer measure generalisation.
Here is what that looks like on an earlier blind corpus, holdout-3, after the fixes it prompted, beside the current blind run:
| Measure | Blind, holdout-7 | holdout-3, after fixes | Gate |
|---|---|---|---|
| Recall | 0.250 | 0.500 | 0.90 |
| Critical / high recall | 0.250 | 0.500 | 0.95 |
| Precision | not computable | 1.000 | 0.90 |
| False positives on safe code | 0.125 | 0.000 | 0.03 |
The right-hand figures are not a generalisation claim. They are what the engine does on a corpus it has since been fitted to, published so the size of the improvement is visible and nobody has to take our word for which figure came from where.
Every corpus, including the ones that flatter us
There are 11 corpora. The first proves nothing at all. It is here so a perfect score is recognisable for what it is.
| Corpus | Cases | Recall | Precision | Safe-case FP | Standing |
|---|---|---|---|---|---|
corpus | 182 | 1.000 | 0.989 | 0.011 | Tuned against |
holdout | 60 (6 excluded) | 1.000 | 1.000 | 0.000 | Burned |
holdout-2 | 76 (38 excluded) | 0.421 | 1.000 | 0.000 | Burned |
holdout-3 | 68 | 0.500 | 1.000 | 0.000 | Burned |
holdout-4 | 60 | 0.067 | 0.667 | 0.033 | Burned |
holdout-5 | 60 | 0.000 | 0.000 | 0.067 | Burned |
holdout-6 | 59 | 0.000 | 0.000 | 0.069 | Burned |
holdout-7 | 64 | 0.000 | 0.000 | 0.125 | Burned |
corpus-python | 56 | 1.000 | 1.000 | 0.000 | Tuned against |
holdout-python | 58 | 1.000 | 1.000 | 0.000 | Burned |
holdout-python-2 | 62 | 1.000 | 1.000 | 0.000 | Burned |
corpus: The rules were developed against this corpus. A perfect score here is what a regression gate looks like, not evidence about unseen code.holdout: Scored, then tuned against. It no longer measures generalisation.holdout-2: Scored at 0.091, then tuned against. Half of its cases are in Java, Ruby or Go, which the engine does not parse, so 38 of them are excluded from scoring and named in the report. Its 16 Django cases have been scored since Python support arrived; Django itself is not modelled, so they mostly miss.holdout-3: Scored blind once, then tuned against in response, so the figure below is the after. The blind score is dated on /changelog.holdout-4: Its authors were invited to name the defects they saw and produced thirty class names the engine does not use, so its recall figure measured naming rather than detection. Class-blind recall on the same corpus is the honest reading. The lesson changed how every later corpus is scored.holdout-5: Scored blind, then tuned against across two passes of work. The figure below is the after and is not a generalisation claim.holdout-6: Five frameworks and five data layers no earlier corpus used. Scored once against a frozen engine, then tuned against, so the figure below is the after and its blind figure is in HOLDOUT-PROTOCOL.md rather than here.holdout-7: The corpus behind the current blind figure, which is stated separately and is NOT the number below. Firebase Functions over Firestore, Azure Functions over Cosmos DB, Strapi 5, Medusa 2 over MikroORM, and Encore.ts — five more frameworks and data layers no earlier corpus used. Scored once against an engine frozen and verified unchanged; the figure below is what the same corpus reports after the work that score prompted, so it is a development figure and no part of it is a generalisation claim.corpus-python: FastAPI and Flask cases the Python rules were developed against. A perfect score here is a regression gate, not evidence about unseen code.holdout-python: Scored blind once, then read on purpose and fixed against, so the figure below is the after and is not a generalisation claim. Its blind score is in HOLDOUT-PROTOCOL.md (Python Run 1).holdout-python-2: FastAPI and Flask, written by an author who never read the Python engine, and scored blind once (Python Run 2), which stays the most recent blind Python figure and is stated separately. It was then read on purpose and fixed against, so the figure below is the after and is not a generalisation claim. The next blind Python figure waits on a new held-out set.
"Excluded" counts cases in a language the engine does not parse. Each is named in its corpus report rather than quietly dropped, because a corpus with invisible exclusions can be made to say anything. holdout-2 is the extreme case: 38 of its 76 cases are excluded, so its recall covers a small population and deserves little weight.
Python support is newer and narrower: FastAPI and Flask only. On 62 held-out Python cases scored once, recall was 0.774 and precision 0.923, with 2 of 31 deliberately correct files flagged. Those cases have since been read and fixed against, so this is the most recent blind Python figure rather than one for the current engine; the next one is pending a new held-out set. They cover only the vulnerability classes the Python engine was built to find, so this is not comparable with the JavaScript figure.
What is missed, by vulnerability class
From holdout-3, after the fixes. Worst first, because the gaps are the point of publishing this. One cell per expected case, filled where it was detected.
| Class | Detected | Cases, filled when detected | Attributed to the wrong place |
|---|---|---|---|
| missing authentication | 0 of 2 | - | |
| broken function level authorization | 0 of 2 | - | |
| business logic price manipulation | 0 of 1 | - | |
| timing unsafe comparison | 0 of 1 | - | |
| secret exposed to client bundle | 0 of 1 | - | |
| stored cross site scripting | 0 of 1 | - | |
| nosql injection | 0 of 1 | - | |
| mass assignment | 0 of 1 | - | |
| broken object level authorization | 1 of 5 | 1 | |
| race condition double spend | 2 of 3 | - | |
| path traversal | 2 of 3 | - | |
| sql injection | 4 of 5 | - | |
| server side request forgery | 4 of 4 | - | |
| command injection | 2 of 2 | - | |
| prototype pollution | 1 of 1 | - | |
| insecure cors configuration | 1 of 1 | - |
12 of the 16 classes in this corpus are incompletely detected, and 8 were not detected at all: missing authentication, broken function level authorization, business logic price manipulation, timing unsafe comparison, secret exposed to client bundle, stored cross site scripting, nosql injection, mass assignment. Every miss is fixable work. Until it is done, it is a class Myrqen will not find in your code either.
"Attributed to the wrong place" means the defect was found but reported against a different line or route than the ground truth names. It is scored as a miss. That is strict, but a finding a developer cannot locate is not much better than none.
What the gap is made of
Each of the 34 vulnerable cases in holdout-3 was classified by the work it would take to catch it, one case at a time, by running the engine against a minimal variant of each miss until the deciding predicate was found. Cumulatively:
- Today0.500
- Reporting the right line for the one remaining attribution case0.529
- Recognising the last unclaimed entry points: a raw WebSocket upgrade, an extension command table0.647
- Twelve rules: three unmodelled sinks, two unmodelled classes, six predicates too narrow, one sanitizer wrongly assumed1.000
The vertical line on each bar is the 0.90 recall gate.
None of it is the module boundary. An earlier version of this table said six cases needed the static pass to follow a value from one module to a sink in another, which it did not do at the time. Checked case by case, that was wrong: each is stopped earlier, by a predicate inside a single function. Scanning each application whole recovers none of them. So the gap is work, not a design boundary, and this page no longer offers the boundary as an excuse.
Read the rows as a work order, not a forecast. A decomposition derived from one corpus reaches every case in that corpus by construction; the development corpus shows what that is worth. The corpus was deliberately not relabelled after scoring, because relabelling the corpus that produced your only clean number, to improve that number, is the one move the protocol forbids. None of this makes 0.500 acceptable.
The whole-product path, against fixtures
This is the path a user gets: static pass, an agent exercising each candidate against a running application under three identities, then dedupe and report. The rules were developed against these fixtures, so this is a regression gate, not evidence. It is published because it is the only measurement of the dynamic half that exists.
| Fixture | Expected | Reported | Recall | Precision | Verified | Unsettled | Unsafe actions blocked / allowed | Canary leaks |
|---|---|---|---|---|---|---|---|---|
vuln-shop | 17 | 17 | 1.000 | 1.000 | 9 | 5 | 4 / 0 | 0 |
safe-app | 0 | 0 | n/a | n/a | 0 | 0 | 4 / 0 | 0 |
vuln-fastapi | 13 | 13 | 1.000 | 1.000 | 9 | 4 | 4 / 0 | 0 |
vuln-flask | 14 | 14 | 1.000 | 1.000 | 10 | 4 | 4 / 0 | 0 |
The unsettled column is the honest part. On vuln-shop, 9 of 17 findings reached verified and 5 did not, because no safe runtime probe exists for them or two samples were not enough to settle the question. They stay needs_review and say why. Nothing is promoted to make the total look better.
safe-app is the control: a correct application, scanned the same way, produced 0 findings. A scanner that cannot stay silent on correct code is not usable, however much it finds.
Methodology
What the corpus is made of
Every corpus pairs each vulnerable case with a safe near-miss: the same route, framework and shape, with the defect fixed. That pairing makes the false-positive rate meaningful. A scanner can reach high recall by reporting everything; the safe half catches it.
Cases are grouped into architecture families, each with its own entry-point conventions. The Python corpora are FastAPI and Flask; every other case is JavaScript or TypeScript unless it is labelled otherwise, and a case in a language the engine does not parse is excluded from scoring and named.
What counts as detected
- Detected: a finding of the expected class, attributed to the file and line the ground truth names, within the expected severity band. The right file but the wrong place is recorded as misattributed and scored as a miss.
- False positive: any finding against a safe case. No partial credit.
- Verified: a candidate an agent exercised against a running application and observed. Only the fixture runs can produce this; corpus cases are isolated files.
- Unsettled: nobody exercised it. Reported as
needs_review, counted as a finding but not as a proven one.
Deduplication and severity
Finding identity, deduplication and redaction belong to the CLI, not the agent, so an agent submitting the same defect twice does not inflate the count. A resubmission merges as corroboration only on an exact summary match. Severity is capped by policy from the class and the evidence, so an agent cannot promote a finding by asserting it is critical.
What we do not measure
Token cost is unavailable, not omitted. The fixture benchmark drives a scripted stand-in agent rather than a model, and the static engine spends no tokens, so any figure would be invented. The session protocol accepts a token count from a real host agent (myrqen metrics token). When there is real data, it will appear here.
Reproduce it
Everything above comes out of commands in the public repository.
reproduce the benchmark
git clone https://github.com/stijnswapped/Myrqen.git
cd Myrqen && pnpm install && pnpm build
# every corpus, side by side, each with what it cannot say
pnpm detection:report
# one corpus, in full
node benchmark/score-corpus.mjs --corpus benchmark/holdout-3
# the whole-product path against the fixtures
node benchmark/run.mjsThe raw output of each run is committed as benchmark/*-last-run.json, beside each corpus's manifest and the expected-vulnerability list for every case. The scorer is score-corpus.mjs, and the harness has tests of its own, including one asserting that no case can leave the gate without a written reason.
The protocol that decides what counts as a clean measurement, and which corpora are burned and why, is HOLDOUT-PROTOCOL.md. The classification behind the ceiling is DETECTION-PLAN.md.
What none of this measures
Stated plainly, because a benchmark page that omits its own scope is the most misleading kind.
- Every corpus here is synthetic and first-party. We wrote the cases and the scorer. There is no independent evaluation and no third-party audit. Treat these numbers as the best evidence a first-party benchmark can offer, which is not the same as evidence.
- No real application has been measured. Corpus cases are small, isolated files. Recall on a large real codebase, with its own conventions, is unknown.
- What the host agent adds is unmeasured. The agent is the half that reasons across modules and exercises the running application, and no harness scores it on unseen code. It may close much of the gap. We cannot show that it does.
- In JavaScript and TypeScript, data flow is followed across functions and modules when a resolved call hands caller data from a request handler to another function, up to four calls deep, through ESM and CommonJS imports, re-exports, tsconfig paths and class methods, including the fields of an object argument. zod, yup, joi and valibot schemas are read field by field. A function handed caller data from more than two other modules is treated as a shared utility and not followed into. Values that travel through event emitters, queues, stored callbacks, dynamic dispatch or a call the engine cannot resolve are not connected.
- JavaScript, TypeScript and Python source is parsed. Other languages, configuration files, and infrastructure definitions are not analysed, and templates are read only where a Python view renders them.
- Runtime behaviour, authentication state, and business rules are not exercised by the static pass; dynamic validation is the host agent's part of the assessment.
- Dependency findings compare installed versions (from the lockfile, or exact pins in package.json without one) against an offline advisory snapshot. A finding is raised only where application code imports and uses an export the advisory names; a vulnerable package reached only through another package is listed as present but not reached, because this scan does not follow calls inside dependencies.
- A route is reported as the file declares it, or as another module in this scan registered the handler under. Where a router is itself mounted under a prefix further up the application, that prefix is not part of the reported path.
- 4 handler(s) were attributed to a route declared in a different module. The route is reported as that router spells it, without any prefix the router is mounted under.
- No entry point was recognised in 2 of 3 parsed file(s). Data-flow checks need an entry point to know which values a caller controls, so in those files they did not run — treat a clean result for this project as unproven rather than as evidence of safety.
If you need a number for a decision, use recall 0.250 from holdout-7, scored blind, beside precision 1.000 from holdout-3 after its fixes — two corpora, not one measurement: act on what Myrqen reports, and do not treat its silence as a result. The security and privacy model · the documentation.