Boundary-mutation testing (arXiv 2609.02983): secret scanners miss edge-case credentials

What happened on this date

On 2026-09-02 (UTC), Shweta Mishra submitted v1 of Boundary-Mutation Testing for Pattern-Based Secret Detection to arXiv. The paper notes that regex-based secret scanners are usually tested with fixed examples that vary only the text around a credential. Its method generates credentials from each rule's own regular expression, places them in realistic source code, and classifies outcomes per rule rather than per tool. It evaluated a 43-rule open-source scanner, Gitleaks 8.21.2 and TruffleHog 3.82.13. On the primary subject, detection was near-perfect across ten contexts but fell to 0.5233 when a credential ended in a hyphen, affecting five rules; two rules with fixed-count quantifiers failed completely and deterministically. The author also reports that a plausible first fix silently broke two other rules, caught only by rerunning the full battery. The paper describes a source-confirmed Gitleaks defect, finds no boundary fragility but the narrowest coverage in TruffleHog, and reports that no recall difference reached statistical significance.

Why it matters for protecting app logic

Many teams treat a clean scan as evidence that no secret ships in the package. This paper shows that a scan only proves nothing was found under the rules and contexts that were actually exercised. For client apps the cost of a miss is direct: strings land verbatim in the APK, IPA or asar, and obfuscation usually does not remove them. To be honest about limits, the paper tested specific older versions, current releases may behave differently, and it does not support ranking the scanners; the author reports a null result. The real lesson is about process: a scanner is a check with blind spots, not a guarantee, and changing its rules is itself a change that can introduce regressions.

Checks you can run

First, scan build outputs as well as the source repository: the unpacked APK, IPA, asar or desktop install folder, because build steps can inject configuration. Second, consider running more than one scanner and record each version. Third, in a test branch, plant invalid test credentials with varied boundary formats and confirm your rule set catches them; rerun every case after editing custom rules. Fourth, if a real secret ever shipped in a release, revoke and rotate it first, before discussing how to hide it. Fifth, do not distribute with the client any credential that can live on the server.

Your checklist

• Scan unpacked release artifacts, not only source
• Validate your rule set with invalid test credentials in varied boundary formats
• Rerun the full suite after rule edits; revoke and rotate real leaks first