Short answer: Security regression testing checks that vulnerabilities you already fixed stay fixed. The same pentest runs again on a schedule and each run's findings are compared with the last run's. A bug that a reverted commit, copied code or a new endpoint brought back is then labelled regressed, instead of getting lost among new findings.
What is a security regression?
A security regression is a vulnerability that was found, fixed, confirmed as fixed, and then found again later. The code moved on and the fix didn't come with it.
Security regression testing is repeated security testing of the same app that checks whether vulnerabilities fixed earlier have come back. Each run is compared with the one before, and a finding that was fixed and is then found again is marked as regressed.
It's the security version of a normal software regression, with one difference. A broken feature gets noticed by users within hours. A broken access check usually doesn't get noticed by anyone until someone goes looking, and that someone may not be on your side.
Why do fixed vulnerabilities come back?
Almost never because someone decided to undo the fix. They come back as a side effect of ordinary work.
| Cause | How it happens | Typical example |
|---|---|---|
| Reverted commit | A release is rolled back, or a merge conflict is resolved in favour of the older branch, and the fix goes with it | The ownership check added to an order endpoint disappears when a hotfix branch is merged back |
| Copied code | Someone copies an old handler or query as a starting point, from before the fix | A new export feature is built from the pre-fix version of a search query, string concatenation included |
| New endpoint, old pattern | A new route is written the way the team used to write them | /api/v2/orders/{id} gets the ownership check, and the new /api/v3/orders/{id} doesn't |
| Dependency downgrade | A lockfile is regenerated, a version is pinned back to work around a bug, or a transitive dependency resolves to an older release | A library version that fixed a template-escaping bug is replaced by the one before it |
| Config drift | A new environment, a rebuilt gateway or a reset feature flag loses a setting that was part of the fix | A rate limit set on the old load balancer isn't carried over to the new one |
The last two are the hard ones, because the application code didn't change at all. A code review of the diff shows nothing. Only testing the running app does.
Why doesn't a one-off retest catch it?
A retest answers one question: is this fix in place today? That's worth knowing, and PCI DSS v4.0.1 asks for exactly that in Requirement 11.4.4, where exploitable findings are corrected and testing is repeated to verify the corrections.
But a regression happens after the retest. The retest passes in March, the hotfix merge in May takes the fix out, and nothing tests that endpoint again until next year's pentest. A retest is a point in time, and a regression needs a series.
Two more reasons a single retest misses regressions:
- It only looks at the old finding. A retest checks the exact endpoint that was reported. The same bug on a new endpoint (the
v3route above) is a new finding to a retest, if it's looked at at all. - Nobody compares reports. Two PDFs from two different engagements, written by different testers with different titles for the same bug, don't line up by themselves. Someone has to do that by hand, and usually nobody does.
How does run-over-run comparison work?
The same pentest runs again, with the same scope, depth and test users, and every finding gets a stable identity so it can be matched between runs. Then each finding in the new run is compared with what the tool already knows.
| Label | Rule | What to do |
|---|---|---|
| New | Not seen in any earlier run | Triage it like any fresh finding |
| Still open | Open in the last run and found again | Check that it's in someone's queue, and how long it's been there |
| Fixed | Open in the last run, not found in this one | The fix held for this run. Leave it in the history |
| Regressed | Fixed in an earlier run and found again now | Treat it as urgent. A fix came undone, so find out why |
Some tools and reports call the third label "resolved". Barrion's dashboard says "fixed", so that's what the example below uses.
The matching is the part to ask vendors about. If a finding's identity depends on its title, a slightly different wording in the next run turns a regression into a "new" finding and the signal is gone. A stable key built from what the finding is (the endpoint, the parameter, the class of bug) holds up better.
Example: a run-over-run report
Here's what a weekly run looks like when it's compared with the week before. The app, dates and findings are made up, but the layout and the labels are the ones a scheduled Barrion run uses.
How to read it
- Start with Regressed. The IDOR on
/api/v2/orders/{id}was fixed in run #11 and is back in run #14, so something shipped between 14 and 21 September took the fix out. Check what merged in that week. That's usually faster than debugging the endpoint from scratch. - New is a normal finding. The reflected XSS on
/searchwasn't there before. Triage it like any other. - Still open shows age. "First seen" on the missing rate limit says it has been open since 17 August, six runs in a row. That's a prioritisation question for the team, and the date makes it visible.
- Fixed means not found this time. The SQL injection didn't show up in run #14. "Last seen" records the last run that did find it, so if it comes back, you know exactly which window to look in.
- WSTG and CWE say what kind of bug each one is. The WSTG ID points to the OWASP test case that covers it, and the CWE is the weakness class your tracker or a customer's questionnaire may ask for.
How to stop regressions as well as catch them
Testing finds a regression. It doesn't stop the next one. NIST's Secure Software Development Framework (SP 800-218, practice RV.3) asks teams to analyze the root causes of vulnerabilities so the same kind doesn't recur. In practice that means a few habits:
- Write a test for each fix. When you fix a finding, add an automated test to your own suite that fails if the bug returns, such as "user B gets 403 on user A's order". A reverted commit then breaks the build.
- Fix the pattern as well as the endpoint. If one handler lacked an ownership check, look for its siblings. Put the check in shared middleware so new routes get it by default.
- Review dependency changes. Treat a lockfile diff that downgrades a package like any other change and review it.
- Keep config in code. A rate limit that lives in a console setting is easy to lose in a rebuild. One in the repository comes along with everything else.
- Keep testing the running app. Tests in your suite cover the cases you thought of. Repeated pentests of the live app catch the ones you didn't, including the config and dependency changes a code review can't see.
How often should regression testing run?
As often as the app changes in ways that could undo a fix. For an app that deploys weekly, a weekly run is the natural rhythm. Teams that ship daily often pair a lighter run that goes only when the app has changed with a deeper monthly run that always goes. The trade-offs are in how often to pentest, and the wider model is covered in what continuous pentesting is.
What regression testing doesn't tell you
"Fixed" means a run didn't find the bug at its depth. It doesn't prove the bug can never be reached. And a regression can only be detected in something the test reaches. If a route is out of scope, or sits behind a role no test user has, a regression there goes unseen. Keep the scope and the test users up to date as the app grows.
How Barrion does it
Barrion's AI agents run full pentests of web apps and APIs across 8 testing areas, covering all 97 OWASP WSTG v4.2 test cases and the OWASP API Security Top 10. You can save a pentest and run it again on a schedule or on demand. You can choose whether a scheduled run goes every time, or only when your app has changed. You can also start a run from your CI/CD pipeline via the Barrion API, so a release gets tested soon after it ships.
- Every run is compared with the previous one, and each finding is labelled new, still open, fixed or regressed. You can mark a finding as ignored and give a reason, so a colleague sees why.
- New and regressed findings trigger an email.
- Findings are checked against the live app before they're reported. Confirmed ones come with the request and response behind them. Anything we couldn't confirm is clearly marked and capped in severity.
- Retests of found issues are free. From Standard up, a security engineer reviews the findings, and deeper tests come with a report signed off by that engineer.
- Data is stored and hosted in Sweden, and AI processing runs in the EU.
See the schedule and the run comparison on continuous AI pentesting. Continuous programs are scoped to your apps, cadence and depth. Talk to us and we'll price it for your setup.
Sources
All sources checked 2026-09-26.
OWASP Web Security Testing Guide v4.2: WSTG-ATHZ-04 (insecure direct object references), WSTG-INPV-01 (reflected XSS), WSTG-ATHN-03 (weak lockout), WSTG-INPV-05 (SQL injection)
MITRE CWE: CWE-639, CWE-79, CWE-307, CWE-89
OWASP API Security Top 10 (2023), API1:2023 Broken Object Level Authorization
NIST SP 800-218, Secure Software Development Framework v1.1, practice RV.3
PCI DSS v4.0.1, PCI Security Standards Council, Requirement 11.4.4
Barrion product facts, facts page