Clira · research · 2026-08-13
We ran our full pipeline — scan, dependency install, sandboxed test run — against a random sample of public repositories built with AI coding tools. This is what came out, including the numbers that are bad for us.
Deterministic stride sample of 40 from a 198-repo corpus of AI/vibe-coded public projects, selected before we looked at any result — a hand-picked sample would tell you about our taste, not about the code. One repository failed to clone, leaving 39. Every repo was cloned, scanned, had its dependencies installed in a network-isolated sandbox, and had its test suite run offline where one existed.
| GO — nothing critical or high | 20 |
| HARDEN — high severity present | 11 |
| STOP — critical present | 8 |
24 critical and 41 high findings across the sample, against 323 medium and 207 low.
| Finding | Count |
|---|---|
| Supabase query — verify RLS policy exists | 47 |
| localhost URL in source (may indicate dev-only config) | 36 |
| AI pattern: dangerouslySetInnerHTML on dynamic data | 15 |
| .env file committed to repository | 8 |
| JWT token in source | 8 |
| Rust: std::process::Command — verify input sanitization | 6 |
| Hardcoded session/JWT secret | 4 |
| Private SSH/RSA key | 4 |
| AI pattern: security control left as TODO | 3 |
| CORS configured to allow all origins | 3 |
The head of that list is unglamorous and consistent: missing row-level-security
policies on Supabase queries, development URLs left in source, and unsanitised HTML injection. The
genuinely dangerous tail is small and real — 6 repositories had a
.env file committed, and the sample contained private SSH keys and JWTs in source.
Only 12 of 39 repositories (31%) had a test suite at all. Of those, we could run and parse 6 of 12.
Clira's central claim is that we run your tests to prove a fix did not break anything. On this sample that claim is available for roughly 15% of repositories. For the rest we report "could not verify" rather than implying a fix was proven — but the honest summary is that the strongest form of our guarantee applies to a minority of AI-built code today, because a minority of AI-built code has tests.
We publish this because you would find it out anyway, on your own repository, after paying.
| Repository | Passed | Failed | Time |
|---|---|---|---|
17-jd/cursor-boston | 123 | 0 | 12.7s |
anthusai/vibe-coding-workbench | 38 | 0 | 22.0s |
erikbjare/gptme-webui | 49 | 0 | 33.9s |
randytorres/cursor-boston | 123 | 0 | 22.2s |
vibecodes-org/vibe-coding-ideas | 2,713 | 0 | 194.7s |
yusei531642/vibe-editor | 605 | 0 | 77.2s |
All executed inside the gVisor sandbox with no network access, against dependencies installed in a separate networked stage that runs no package code. Of 12 dependency installs attempted, 9 succeeded; 9 were pinned by a lockfile whose digests the package manager enforces.
Running against real code found five classes of false positive that our own test fixtures could not, because every one was a context bug — what kind of file is this, who wrote it, is this prose:
Placeholders in .env.example read as leaked keys | fixed |
/re/.exec(s) read as code execution | fixed |
| Documentation scanned as executable source | fixed |
| Vendored third-party code scanned as the author's | fixed |
| A package.json with no test script counted as a test suite | fixed |
Each of the first four produced a STOP verdict — our entire deliverable — on repositories whose own code was clean. Before the fixes this sample reported an inflated 87% "has tests"; the true figure is 31%.
The section above reports what we found. It cannot report what we missed, because nobody knows the true answer for real code. So we built a corpus where we do know it: 34 vulnerabilities planted by us, at lines we recorded, across 9 realistic application skeletons — alongside 14 decoys, every one a real false positive we had previously produced.
A match only counts if it lands in the same file, within two lines, at or above the severity the vulnerability deserves — finding something nearby for an unrelated reason is not a detection. Decoys count against us only at high or critical, because that is the severity that produces a STOP and blocks a launch.
Our first version of this corpus scored 100%, and that was the problem. A corpus
you cannot fail is measuring your rules against themselves. We added a harder tier — path traversal,
SSRF, IDOR, weak crypto, prototype pollution, insecure deserialization, shell=True, XXE,
keys split across concatenation — deliberately including classes we suspected we would miss. Recall
fell to 76%, which is a number worth reporting.
Building it immediately exposed two real defects. Our Go command-execution rule matched
os/exec.Command — the import path — while real Go calls exec.Command(...),
so the rule had never fired on anything, ever. And three classes were absent outright:
deserialization, shell=True, and XML external entities. All are fixed, which is how
recall reached 92%.
What we still miss, named: idor-no-ownership, prototype-pollution, secret-concatenated, taint-interprocedural. Each needs semantics or
interprocedural dataflow — knowing which objects a user may see, folding a constant split across
concatenation, following taint across a function boundary. Attempting them with line-local heuristics
would cost precision, and precision is the one number that must not move.
Correction, 2026-08-13: this page previously reported 92% recall. That figure was wrong — our scorer credited a seeded vulnerability if any finding landed within two lines of it, and because seeds are written consecutively, one real detection was satisfying several neighbouring seeds. With exact matching the true figure was 73.5%; adding rules the corpus showed us to be missing brought it to 88%. We leave this note rather than quietly editing the number, because a measurement that flatters you is the failure mode this section exists to guard against, and we walked into it ourselves.
What our precision figure does and does not mean. Zero of 14 decoys were escalated. That is not the same as a 0% false-positive rate: with no failures in 14 trials, the 95% upper bound on the true rate is roughly 21%. Holding precision at 100% is a rule we impose on ourselves — no finding reaches STOP unless the evidence establishes it — not a property we have proven. Two things would close that gap and neither is built yet: hundreds of decoys rather than 14, and a held-out corpus we never look at while writing rules. Until then, treat the recall figure as training accuracy.
39 repositories is a small sample, drawn from public GitHub, and skews toward the kind of project people publish rather than the kind they keep private. We measured what our scanner finds, which is not the same as what is there.
The seeded corpus has its own limit, and it is the important one: we chose the vulnerabilities. Recall against a corpus you wrote is an upper bound on recall against code you have never seen. It is a floor for regression — it will catch us getting worse — not a promise about your repository.