Round 2, a fresh full-pool sample: TrueMend 19.5% wrong-match against actionlint's round-1 40.0% (not yet re-measured this round). The wrong-match number moved on a smaller, differently-composed sample, not a reversal of round 1's fixes: real-rate more than doubled, 17.0% to 40.3%, the more meaningful signal this round.
actionlint is the purpose-built, single-ruleset linter for GitHub Actions
workflow YAML: a corpus of 165 real workflows, flattened from eleven
benchmark repos' .github/workflows/ directories, gave us a fair fight on
its home turf.
The baseline was a confirmed loss
First measurement: TrueMend 55.7% wrong-match vs. actionlint's 44.4%: independently judged, not self-reported, and a clear loss. We published that number to ourselves before touching anything, the same discipline as every other campaign in this ledger.
Root cause: two detector families that don't belong in this file type
The single biggest lever, by far: 66 Kubernetes detectors and 24 Docker
Compose detectors had no file_content_requires gate, so they fired on a
GitHub Actions workflow's matrix.image:, container:, and services:
blocks as if those were real Kubernetes manifests or Compose files:
k8s-kubelinter-no-service-account alone went 8-for-8 wrong, every hit on a
matrix.image: line that isn't a Kubernetes resource at all. Adding
apiVersion: and (?m)^services: content gates before those detectors fire
silenced 496 false positives outright, 1,079 findings down to 575, with no
loss of recall on files that are actually Kubernetes or Compose manifests.
Seven more detectors had a second, distinct problem: a premise that does not
hold for a workflow trigger context. yaml-github-actions-secrets-plaintext
was flagging default values like POSTGRES_PASSWORD: postgres that are
container bootstrap conventions, not leaked secrets. ci-token-write-permissions
treated any write-scoped permission as broad when only contents: write
specifically is the risky one. Four detectors: checks for status checks, PR
comments, conditional gating, and environment protection: can't actually be
verified from the trigger event alone, so we disabled them as premise errors
rather than keep a claim we couldn't back. One more got its Dockerfile anchor
narrowed, and a matrix-include threshold got raised to reduce noise on small,
legitimate matrices.
Round 1 result: a 29-point swing
TrueMend: 11.6% wrong-match. actionlint: 40.0%. 112 TrueMend findings and 35 actionlint findings, judged blind, same-session head-to-head. Worth-fixing narrowed from the baseline's 3.7%/30.6% gap to 17.0%/25.7%: closer, though still trailing.
What we weren't claiming in round 1
actionlint's expression, action, and syntax-check rules catch real
workflow-expression correctness bugs: invalid property references,
deprecated action versions: that a line-scan detector can't structurally
express without a full workflow-expression AST. The same shape of gap shows
up in the C# campaign's XML-doc case. In round 1, TrueMend won the headline
wrong-match metric decisively; actionlint still found more genuinely
worth-fixing correctness bugs on this corpus, and we said so rather than
only publishing the number that flattered us.
Result: round 2, a fresh full-pool sample
TrueMend: 19.5% wrong-match. actionlint: 40.0%: round 1's number, carried
forward, not re-measured this round. A parallel fix pass targeted the top
wrong-match categories still open after round 1, then a fresh, from-scratch
full-pool blind sample re-certified the result: 77 TrueMend findings judged.
Four more categories fixed this round: devops-hardcoded-secret (well-known
CI credential literal exclusion), continue-on-error (outcome-check negation
logic), ci-no-release-please (release-event gate added), and a second pass
on ci-token-write-permissions: round 1 fixed the broad-write-scope premise,
round 2 added a push/commit-bot and reusable-workflow exclusion the first
pass didn't cover. A new regression test file,
tests/regression/test_ghactions_fp_regression.py, locks in all four:
10/10 passing.
| Round | TrueMend wrong-match | actionlint wrong-match | Real-rate | Sample |
|---|---|---|---|---|
| Round 1 (2026-07-08) | 11.6% | 40.0% | 17.0% | 112 TrueMend / 35 actionlint judged |
| Round 2 (2026-07-28) | 19.5% | 40.0% (round-1 number, not re-measured) | 40.3% | 77 TrueMend judged, fresh full-pool sample |
The wrong-match number moved from 11.6% to 19.5%: worse on the raw figure. That's the honest number, and it's not a reversal: round 1's fixes were independently re-verified this round and still hold. What moved is sample composition: round 2 is a smaller, differently-composed fresh pool (77 judged vs. 112), pulling from parts of the detector surface round 1's sample didn't touch, the same pattern seen across every re-certified target in this sweep (Bash, Java, Go, SCSS). The more meaningful signal this round is real-rate: worth-fixing more than doubled, from 17.0% to 40.3%.
Still open
actionlint has not been re-run this round: a true apples-to-apples round-2 comparison against a fresh actionlint measurement doesn't exist yet. TrueMend's round-2 wrong-match, 19.5%, remains above the 10% gold-standard bar this ledger holds every target to. Both are open, not closed.