Full-pool blind adjudication on a production Gin API: TrueMend 0.0% wrong-match vs. Semgrep's 50.0%, every one of 51 post-fix findings verified correct. The follow-up we promised, a second, independent Gin corpus, told a harder story: 40.4% wrong-match on first contact, an engine bug found underneath it, fixed back to 5.9%.
Gin's framework-specific detector surface hadn't been through real blind adjudication before: only synthetic fixtures. We ran a full campaign against a real production Gin codebase to find out whether that surface performs the same on real code, not only on test fixtures.
Judged completely, not sampled
The pool was small enough to judge every finding, not a stratified sample: 91 of 91 (51 TrueMend, 40 Semgrep), each ruled correct or wrong by an independent judge with no knowledge of which tool produced it or what the "right" answer should be.
Result
TrueMend: 0.0% wrong-match. Semgrep: 50.0%. Every single one of TrueMend's
51 post-fix findings checked out against the real source. Semgrep's 50% is
dominated by one 18-instance false-positive cluster: an HTML-escaping rule
firing on Gin's own render/*.go package (json.go, yaml.go, toml.go,
protobuf.go, ...) for writing correctly-serialized, correctly-content-typed
bytes: the rule doesn't check content type before applying an HTML-context
escaping assumption.
What it took to get there
Six systemic detector bugs, all genuine precision fixes: no detector was
disabled or withdrawn to reach this number. The largest: an "unchecked error"
check that allowlisted specific stdlib comma-ok patterns (strings.Cut,
utf8.Decode*) but not Go's own GetXxx() (T, bool) accessor convention,
so it fired on every c.GetQuery, c.GetPostForm, and similar call in Gin's
own context.go. Fixed by recognizing the GetXxx/Get naming convention
directly: two genuine true positives in the same category still fire
correctly post-fix. Full root-cause detail for all six: internal engineering
notes, available on request.
The follow-up we promised
This case study originally ended here, with one real corpus exhaustively judged, and an open question: would the surface hold up against a second, independent Gin codebase, or did these six fixes merely overfit to one repo's idioms? We ran that follow-up. It didn't go the way the first result would predict.
Round 2: first contact with a second corpus, and it's ugly
The second corpus is gin-gonic/gin's own source tree: its own test suite and
internal packages exercise gin.Context, c.Bind/c.ShouldBind, and error
handling the same way a consuming application does, though written by the
framework's own maintainers with their own idioms. A fresh, from-scratch full
pool sample (156 judged, TrueMend-only: golangci-lint wasn't installed this
round) came back at 40.4% wrong-match. Confirmed 2 of the 3 prior fixes
already held; the number is bad because eight Go/Gin-specific detectors were
sitting at 75–100% wrong on idioms this second corpus surfaces that the first
one never did: test-helper error-check wrappers (require.NoError), the
http.NewRequest-isn't-I/O distinction, and a batch of whole-construct claims
(named-return signatures, operand types) no line-scan can see. Publishing a
worse number two rounds later isn't a retraction of the 0.0% result above:
different corpus, different failure modes, but it's the honest version of
"does this generalize," and the honest answer was: not yet.
Round 3: an engine bug underneath the detector bugs
Eight detectors, each root-caused against the real gin-gonic/gin source. Three were genuine line-scan fixes. Five were quarantined: whole-construct properties (a named-return signature, an operand's type, a return type) that no bounded regex window can see, so we disabled and documented them rather than chase precision with a window tweak that would only trade false positives for silently dropped true positives.
The load-bearing finding wasn't a detector at all. One fix cluster wouldn't narrow no matter how the regex was adjusted, and root-causing it against source turned up an indexing bug in the detection engine itself: the per-function scan path computed each match's position using the wrong baseline, so any detector using a negative-context check on that path silently never suppressed anything: the check ran, found nothing, and kept every match, including the ones it existed to filter out. Confirmed directly against the corpus: one detector's raw findings dropped from 66 to 1 the moment the engine fix landed, with no change to that detector's own rule. It had been correct the whole time; it never ran correctly at all.
| Round | Judged | Wrong-match | Real-rate | Note |
|---|---|---|---|---|
| Round 1 (2026-07-26, above) | 51 | 0.0% | : | Production Gin API, vs. Semgrep 50.0% |
| Round 2 (fresh corpus) | 156 | 40.4% | 13.5% | gin-gonic/gin itself: 8 detectors newly exposed |
| Round 3 (post-fix) | 102 | 5.9% | 7.8% | 3 fixes, 5 quarantines, 1 engine bug |
Under our own 10% bar again, on a corpus that initially broke the surface far worse than round 1's clean result would have predicted. Raw findings on this corpus: 3049 → 2184 (−865 false positives) between rounds 2 and 3. No competitor ran this round, golangci-lint wasn't installed, so this isn't a head-to-head rerun of the 0.0%-vs-50.0% result above, it's the generalization check that result was missing.
Still open
The engine bug's fix applies everywhere the same code path runs, not only to Go/Gin, but this campaign only re-verified it here. Whether other language surfaces were quietly carrying the same silently-disabled negation checks, and how much their own certified numbers move once re-measured, is unverified and the next thing worth checking.