All posts

Testing Basics · Part 15

Regression testing vs re-testing: what they are and why we do both

ProvaLab team · · 6 min read

A developer fixes a bug and says "done". Two questions follow, and they sound similar but are very different:

  1. Is the bug really fixed?
  2. Did the fix break anything else?

The first is answered by re-testing. The second is answered by regression testing. Every fix needs both, and mixing them up is one of the most common ways bugs reach customers.

What is re-testing?

Re-testing (also called confirmation testing) means running the exact test that failed again, after the bug has been fixed, to confirm the fix works.

Why do we re-test?

Because a fix is only a claim until someone checks it. In real teams, "fixed" turns out not to be fixed surprisingly often:

  • The fix is partial. It works for the case the developer tried, but not for the one in the bug report.
  • It works in one place but not another. Remember BUG-231: the expired-code fix worked on the web, but the Android app had its own copy of the check, so the bug was still there.
  • The fix never arrived. It's on the developer's machine or in a branch, but not in the build being tested.
  • The developer misunderstood the bug and fixed something else.
  • The environment hid the bug, so it passed on the developer's laptop but fails with real test data.

How to re-test

  • Use the same steps, the same data and the same kind of environment as the original bug report.
  • Test on every platform the bug affected (web, Android, iPhone).
  • Try one or two small variations around the bug: another expired code, a code that expired one minute ago.
  • Then update the bug: Verified if it passed, Reopened with details if not (see the bug life cycle).

The person who found the bug is usually the best one to re-test it.

What is regression testing?

Regression testing means re-running tests that used to pass, after a change, to make sure nothing that worked before is now broken.

A regression is something that worked yesterday and doesn't work today. Regressions happen because software is connected: code is shared, data flows between features, and a change in one place can ripple into others.

Why do we do regression testing?

Changes break things in places nobody expected:

  • Shared code: the fix for BUG-231 moved the discount check to the server. Every app now uses it, including flows nobody touched.
  • Side effects: a faster search query returns results in a different order, and the "recommended" section breaks.
  • Merges: two developers' correct changes conflict when combined.
  • Upgrades: a new version of a library, the database or the browser changes behaviour.
  • Configuration: a changed setting in one environment affects features far away.

Developers can't always predict these ripples. Regression tests catch them.

Re-testing vs regression testing for one fix

After the fix for BUG-231: re-testing runs the one failed test, TC-104, at the centre. Targeted regression checks the ring of nearby features the change could affect: valid codes, cart, checkout totals, free delivery, order history and the Android app. The wider regression suite covers the rest of the app, such as sign-in, search, menu, profile, tracking and the restaurant dashboard, nightly and before every release.

For the BUG-231 fix:

  • Re-test: TC-104 (expired code refused), on web and Android.
  • Targeted regression: valid codes still work, cart and checkout totals, free-delivery rule, order history totals, the Android checkout. These all touch the discount code that changed.
  • Wider regression: the full suite runs that night, and again before the release.

The differences at a glance

Re-testing Regression testing
Question Is this bug really fixed? Did the change break anything else?
What is run The test(s) that failed Tests that previously passed
When After a specific bug is fixed After any change: fixes, features, upgrades
Scope Narrow: the bug Wide: the areas around the change, or everything
Planned in advance? No, it depends on which bugs were found Yes, a regression suite is built and maintained
Good for automation? Sometimes; often quicker by hand Yes, it's the classic case for automation
Expected result The test that failed now passes Tests that passed still pass

Re-testing usually comes first: there's no point running a big regression suite on a fix that doesn't work. (A quick sanity check often comes before both.)

How to choose which regression tests to run

Running every test after every change is the safest choice, but rarely practical. There are three common strategies:

  1. Retest all: run the whole regression suite. Most thorough, slowest. Good before a release.
  2. Regression test selection: run only the tests related to what changed. A traceability matrix and good test tagging (by feature, by area of code) make this possible.
  3. Prioritisation: run tests in order of risk, with critical journeys first (risk-based testing), so the most important results come back first.

Most teams combine them with a schedule like this:

Which regression tests to run and when. Every commit: unit tests, thousands in about a minute. Every merge or pull request: integration tests and tests for the changed areas, in minutes. Every night: the full automated regression suite of UI, API and system tests on all browsers. Before each release: the full suite plus manual exploratory checks of the risky areas.

Building and maintaining a regression suite

A regression suite is a living thing. Some habits that keep it useful:

  • Add a test for every bug you fix. If a bug happened once, it can come back. The test from the BUG-231 fix joins the suite permanently.
  • Automate it. Regression tests are run again and again, which is exactly when automation pays off (see Part 3).
  • Tag tests by feature, priority and speed, so you can run "all checkout tests" or "critical tests only".
  • Remove or update obsolete tests. A test for a feature that no longer exists only slows everyone down.
  • Fix flaky tests quickly. If people learn to ignore red results, real regressions slip through.
  • Watch the run time. If the suite gets too slow to run often, split it into faster and slower layers.

Common mistakes

  • Re-testing only on one platform when the bug affected several.
  • Skipping regression for "small" fixes. Small changes to shared code cause big regressions.
  • Running the full suite on a fix that hasn't been re-tested, then wasting the run when the fix turns out not to work.
  • Never adding tests for fixed bugs, so the same bug returns months later.
  • Treating regression testing as manual work forever. It grows with every release; without automation it becomes impossible to keep up.

What's next

In Part 16 we look at performance testing and its types: load, stress, spike, soak, volume and scalability testing.

ProvaLab runs your regression suite on every change and every night, on real browsers and devices, and shows each test's history: when it last passed, and the build where it started failing. When a regression appears, your coding agent gets the failure with its logs and recording, and can fix it. See it in the two-minute product tour.

Prefer to read offline? The whole Testing Basics series is one free PDF: get the PDF.

See ProvaLab working on a real app

Your coding agent runs the tests, reads the failures and fixes them.