1. A load test at the expected peak: does it hold up?
2. A stress test past the peak: where does it slow down, and does it break?

Every command and every line of output below was run while writing this post.

Before you start: a load test is, by definition, a flood of traffic. Only ever point it at something you own, and never at production without a plan. Our target is the API from Part 4, running on our own machine.

Step 1: Get k6 (1 minute)

k6 is a single program. Install it from k6.io for your operating system, or run it from its container image, which is what we do here:

docker pull grafana/k6:latest
docker run --rm grafana/k6:latest version

k6 v2.3.0 (commit/e088784614, go1.27.1, linux/amd64)

Step 2: Start the target (1 minute)

In the Part 4 folder, with the fixed version of the API from Part 6:

uvicorn app:app

One request by hand, to know what "normal" looks like before we add load:

curl -s -o /dev/null -w "%{http_code} in %{time_total}s\n" -X POST http://127.0.0.1:8000/discounts/apply \
  -H "Authorization: Bearer token-alice" -H "Content-Type: application/json" \
  -d '{"order_total": 500, "code": "SAVE10"}'

200 in 0.039147s

Forty milliseconds for one request, on an idle machine. The question load testing answers is: what happens to that number when it's fifty requests at once?

Step 3: Write the load test (4 minutes)

Create a folder called k6 and in it load.js:

import http from "k6/http";
import { check, sleep } from "k6";

const BASE = __ENV.API_URL || "http://127.0.0.1:8000";
const HEADERS = { Authorization: "Bearer token-alice", "Content-Type": "application/json" };

export const options = {
  stages: [
    { duration: "30s", target: 50 }, // ramp up to 50 users
    { duration: "60s", target: 50 }, // hold the expected peak
    { duration: "15s", target: 0 },  // ramp down
  ],
  thresholds: {
    http_req_duration: ["p(95)<200"], // 95% of requests under 200 ms
    http_req_failed: ["rate<0.01"],   // less than 1% errors
  },
};

export default function () {
  // 7 in 10 users apply a discount, 3 in 10 look at an order
  if (Math.random() < 0.7) {
    const res = http.post(`${BASE}/discounts/apply`, JSON.stringify({ order_total: 500, code: "SAVE10" }), { headers: HEADERS });
    check(res, { "discount applied": (r) => r.status === 200 && r.json("total") === 450 });
  } else {
    const res = http.get(`${BASE}/orders/42`, { headers: HEADERS });
    check(res, { "order returned": (r) => r.status === 200 });
  }
  sleep(1); // think time: a real user pauses between actions
}

The four ideas in this file are the four ideas in any load test:

• Stages describe the shape of the load over time: ramp up, hold, ramp down. The ramp matters; starting 50 users in the same millisecond tests something different (that's a spike test).
• Thresholds are the pass/fail rules, decided before running: p95 under 200 ms, under 1% errors. Without them you get numbers but no verdict.
• The default function is what one virtual user (VU) does, over and over: here, a realistic mix of two actions.
• Think time (sleep(1)) is the pause a real person takes between actions. Without it, 50 virtual users behave like thousands of real ones.

The check lines are the load test's assertions: a 200 with the right total. A load test that only counts responses would happily report success while every response was an error page.

Step 4: Run it (2 minutes)

docker run --rm --network host -e API_URL=http://127.0.0.1:8000 -v "$PWD/k6:/scripts" \
  grafana/k6:latest run /scripts/load.js

(As in Part 6, on macOS and Windows use http://host.docker.internal:8000 and drop --network host.)

k6 prints a progress line every second, then the summary:

  █ TOTAL RESULTS

    checks_total.......: 4145    39.338342/s
    checks_succeeded...: 100.00% 4145 out of 4145
    checks_failed......: 0.00%   0 out of 4145

    ✓ discount applied
    ✓ order returned

    HTTP
    http_req_duration..............: avg=1.91ms min=975.39µs med=1.68ms max=16.09ms p(90)=2.7ms p(95)=3.15ms
    http_req_failed................: 0.00%  0 out of 4145
    http_reqs......................: 4145   39.338342/s

    EXECUTION
    iteration_duration.............: avg=1s     min=1s       med=1s     max=1.01s   p(90)=1s    p(95)=1s
    iterations.....................: 4145   39.338342/s
    vus............................: 1      min=1         max=50
    vus_max........................: 50     min=50        max=50

default ✓ [ 100% ] 00/50 VUs  1m45s

Passed. 4,145 requests over 1 minute 45, every check green, no failures, and a p95 of 3.15 ms against a target of 200 ms. At its expected peak this API isn't remotely stressed. (The "iteration_duration" of 1 s is just the think time.)

Note the tick next to default and the absence of any threshold error: that's k6's verdict. In CI, a crossed threshold makes k6 exit with a non-zero code, which fails the job.

Step 5: Stress it (5 minutes)

Passing at 50 users tells you it's fine today. It doesn't tell you how much headroom you have, or what failure looks like. For that, keep increasing the load past the expected peak until something gives. Create stress.js:

import http from "k6/http";
import { check } from "k6";

const BASE = __ENV.API_URL || "http://127.0.0.1:8000";
const HEADERS = { Authorization: "Bearer token-alice", "Content-Type": "application/json" };

export const options = {
  stages: [
    { duration: "20s", target: 100 },
    { duration: "20s", target: 300 },
    { duration: "20s", target: 600 },
    { duration: "20s", target: 1000 }, // keep going past the expected peak
    { duration: "10s", target: 0 },
  ],
  thresholds: {
    // we expect these to fail: the point is to see where
    http_req_duration: ["p(95)<500"],
    http_req_failed: ["rate<0.05"],
  },
};

export default function () {
  const res = http.post(`${BASE}/discounts/apply`, JSON.stringify({ order_total: 500, code: "SAVE10" }), { headers: HEADERS });
  check(res, { "status 200": (r) => r.status === 200 });
  // no think time: every virtual user hammers the API as fast as it can
}

Two deliberate changes: the load climbs in steps to 1,000 users, and there's no think time, so each virtual user sends requests as fast as the API answers. Run it the same way, with stress.js:

  █ TOTAL RESULTS

    checks_total.......: 70982   788.671898/s
    checks_succeeded...: 100.00% 70982 out of 70982
    checks_failed......: 0.00%   0 out of 70982

    ✓ status 200

    HTTP
    http_req_duration..............: avg=498.54ms min=731.6µs med=430.55ms max=1.79s p(90)=1.1s p(95)=1.51s
    http_req_failed................: 0.00% 0 out of 70982
    http_reqs......................: 70982 788.671898/s

    EXECUTION
    vus............................: 13    min=5          max=997
    vus_max........................: 1000  min=1000       max=1000

default ✓ [ 100% ] 0000/1000 VUs  1m30s
time="2026-10-29T18:31:28Z" level=error msg="thresholds on metrics 'http_req_duration' have been crossed"

Read it line by line:

• 70,982 requests in 90 seconds, 789 per second. Twenty times the load test's rate.
• Still 0% failed, every check green. The API never fell over. Every one of 70,982 requests got a correct 200.
• But p95 is 1.51 s, and the maximum 1.79 s. One in twenty customers would wait a second and a half for a discount to apply. That's the threshold k6 crossed, and the reason the run is marked failed.

So the honest answer from this stress test is: this API doesn't break under 1,000 hammering users; it slows down, from 3 ms to 1.5 s at the 95th percentile. That's the "breaking point" for a single-process server: the point where the queue of waiting requests grows faster than it drains. Whether that's acceptable depends on your product, which is exactly why thresholds are written down before the run.

What we'd do next

• Find the knee. Re-run with finer steps (200, 250, 300…) to find the user count where p95 first crosses 500 ms. That number is your capacity, and it belongs in the test plan.
• Watch the server, not just k6. CPU, memory and, for a real app, the database. Here the limit is one Python process on one core; uvicorn --workers 4 would roughly quadruple it. Load testing tells you that it slows; server metrics tell you why.
• Run it from a separate machine. We ran k6 on the same laptop as the API, so they competed for CPU. For real numbers the load generator must not share resources with the target, and for big loads it must be several machines.
• Add a soak run: 50 users for two hours, to catch memory leaks a 90-second test never will.

Putting it in the pipeline

Nightly, not on every commit (a load test takes minutes and needs a quiet environment):

      - run: uvicorn app:app &
      - run: >
          docker run --rm --network host -e API_URL=http://127.0.0.1:8000
          -v "$PWD/k6:/scripts" grafana/k6:latest run /scripts/load.js

If the thresholds are crossed, k6 exits non-zero and the job goes red. Over months, that catches the commit that made things slower.

Habits worth keeping

1. Decide the thresholds before you run. Numbers without a target are trivia.
2. Read p95 and p99, never the average. The average in the stress run (498 ms) hid a p95 three times worse.
3. Assert on the response, not just the status. A fast wrong answer is still wrong.
4. Model real behaviour: a mix of actions, with think time. Drop the think time only when you mean to stress.
5. Never load-test what you don't own. And never production without warning the people who run it.

Hands-On Testing: the complete series

That closes the series. Over seven parts you've built and run:

1. UI testing explained · 2. Your first Playwright test
2. API testing explained · 4. Your first API tests
3. Security testing basics: the OWASP Top 10 · 6. Your first security scan with OWASP ZAP
4. Your first load test with k6 (this post)

Together with the 23-part Testing Basics series, that's a complete foundation: the ideas, and the tools in your hands.

ProvaLab's load testing runs k6 tests like these from the cloud, on machines that don't compete with your app, shows p95 and throughput live, and can find the user count where your API first slows down, step by step. Your coding agent can write the script from your code and explain the slow spots afterwards. See it in the two-minute product tour.

This guide takes the three tests from Part 2 and grows them into a suite with ten tests that is cheap to change. Every command below was run as written, against provalab.io, with these versions:

What we're building

playwright-in-practice/
├── pytest.ini
├── conftest.py          # fixtures: saved state, viewport, mocked API
├── pages/
│   ├── __init__.py
│   ├── blog.py          # BlogPage, PostPage
│   └── guide.py         # GuidePage
├── test_blog.py
├── test_consent.py
└── test_guides.py

Four ideas, one each for the four problems above:

1. Page objects so a locator lives in one place.
2. Fixtures so a test asks for what it needs instead of building it.
3. Saved browser state so the cookie banner is dismissed once, not ten times.
4. A mocked API so the form test never creates a real lead.

Then we run the whole thing on four workers.

Setup (2 minutes)

Same folder and virtual environment routine as Part 2, plus the parallel runner:

mkdir playwright-in-practice && cd playwright-in-practice
python3 -m venv .venv && source .venv/bin/activate
pip install pytest-playwright pytest-xdist
playwright install chromium

pytest.ini points at the site and silences one warning:

[pytest]
addopts = --base-url https://provalab.io
filterwarnings = ignore::pytest.PytestRemovedIn10Warning

The second line is worth a sentence. pytest 9 deprecated a pattern that pytest-playwright 0.10 still uses for its base_url fixture, and prints a warning on every run. It is harmless, and the plugin will catch up; until then this keeps the output clean so that a warning you should read doesn't hide among ones you shouldn't.

Step 1: Page objects (10 minutes)

A page object is a small class that knows where things are on one page and what a user can do there. Tests use it instead of locators. Here is the blog index and a post:

# pages/blog.py
import re

from playwright.sync_api import Page


class PostPage:
    def __init__(self, page: Page):
        self.page = page
        self.title = page.get_by_role("heading", level=1)
        self.all_posts = page.get_by_role("link", name="All posts")

    URL = re.compile(r"/blog/[a-z0-9-]+/$")


class BlogPage:
    def __init__(self, page: Page):
        self.page = page
        self.heading = page.get_by_role("heading", level=1)
        self.cards = page.get_by_test_id("blog-card")
        self.pdf_banner = page.get_by_test_id("blog-pdf-banner")

    def open(self):
        self.page.goto("/blog/")
        return self

    def open_first_post(self) -> PostPage:
        self.cards.first.get_by_role("link").first.click()
        return PostPage(self.page)

    def pdf_link(self, series: str):
        return self.pdf_banner.get_by_role("link", name=f"{series} PDF")

Two things to notice. The attributes are locators, not elements: self.cards is a description of how to find the cards, evaluated fresh every time a test uses it, so it never goes stale after navigation. And an action that leads somewhere returns the page object for where you end up: open_first_post() hands back a PostPage.

The second page object covers the free-PDF pages, which have a form:

# pages/guide.py
from playwright.sync_api import Page


class GuidePage:
    """The free-PDF page for one series, e.g. /testing-basics/."""

    def __init__(self, page: Page, slug: str):
        self.page = page
        self.slug = slug
        self.parts = page.get_by_test_id("pdf-part")
        self.download_link = page.get_by_test_id("download-link")
        self.error = page.get_by_test_id("lead-error")

    def open(self):
        self.page.goto(f"/{self.slug}/")
        return self

    def request_pdf(self, name: str, email: str):
        self.page.get_by_test_id("lead-name").fill(name)
        self.page.get_by_test_id("lead-email").fill(email)
        self.page.get_by_test_id("lead-submit").click()

request_pdf is the kind of method page objects exist for: three low-level steps that every test would otherwise repeat, named after what the user is doing.

Now the three tests from Part 2, rewritten. They read like sentences, and there is not a single selector in them:

# test_blog.py
from playwright.sync_api import expect

from pages.blog import PostPage


def test_blog_lists_posts(blog):
    expect(blog.heading).to_have_text("ProvaLab Blog")
    expect(blog.cards.first).to_be_visible()


def test_a_post_opens_from_the_list(blog):
    post = blog.open_first_post()
    expect(post.page).to_have_url(PostPage.URL)
    expect(post.title).to_be_visible()
    expect(post.all_posts).to_be_visible()


def test_pdf_banner_links_to_both_guides(blog):
    expect(blog.pdf_link("Testing Basics")).to_have_attribute("href", "/testing-basics/")
    expect(blog.pdf_link("Hands-On Testing")).to_have_attribute("href", "/hands-on-testing/")

Where does blog come from? That is the next step.

Step 2: Fixtures (5 minutes)

A pytest fixture is a function that builds something a test needs; a test asks for it by naming it as a parameter. You have been using them since Part 2: page is a fixture that pytest-playwright provides. Yours go in conftest.py, which pytest loads automatically:

# conftest.py (first part)
import pytest

from pages.blog import BlogPage


@pytest.fixture
def blog(page) -> BlogPage:
    return BlogPage(page).open()

A fixture can ask for other fixtures. blog asks for page, wraps it and opens the blog. Every test that names blog gets a fresh page, already on /blog/, with no setup lines of its own.

pytest-playwright also lets you override its own fixtures. The most useful one is browser_context_args, the options every new browser context is created with. A fixed viewport and locale make screenshots and date formats the same on every machine:

@pytest.fixture(scope="session")
def browser_context_args(browser_context_args):
    return {
        **browser_context_args,
        "viewport": {"width": 1280, "height": 800},
        "locale": "en-GB",
    }

scope="session" means it runs once per test session, not once per test. Keep that word in mind for the next step.

Step 3: Dismiss the cookie banner once (10 minutes)

Every page on this site shows a small cookie notice until you close it. In Part 2 we ignored it, because none of the three tests clicked anything under it. In a bigger suite it will eventually sit on top of something you need, and the lazy fix, a click at the top of every test, is slow and gets copied everywhere.

Playwright has a better answer. A browser context can save its state (cookies and local storage) to a file, and a new context can start from that file. So: dismiss the banner once, save, and have every test's context begin with it already gone.

# conftest.py (second part)
import os
from pathlib import Path

# One state file per pytest-xdist worker, so parallel runs never write the same file.
STATE = Path(__file__).parent / ".state" / f"consent-{os.environ.get('PYTEST_XDIST_WORKER', 'main')}.json"


@pytest.fixture(scope="session")
def consent_state(browser, base_url) -> Path:
    """Dismiss the cookie banner once per session and save the browser state."""
    if not STATE.exists():
        context = browser.new_context(base_url=base_url)
        page = context.new_page()
        page.goto("/")
        page.get_by_role("button", name="Dismiss").click()
        STATE.parent.mkdir(exist_ok=True)
        context.storage_state(path=STATE)
        context.close()
    return STATE


@pytest.fixture(scope="session")
def browser_context_args(browser_context_args, consent_state):
    """Every test's browser context starts with the banner already dismissed."""
    return {
        **browser_context_args,
        "storage_state": str(consent_state),
        "viewport": {"width": 1280, "height": 800},
        "locale": "en-GB",
    }

consent_state is session-scoped, so the banner is dismissed once per run. browser_context_args now asks for it and passes the file as storage_state. Nothing else in the suite mentions the banner at all. The saved file is small and readable:

{"cookies": [], "origins": [{"origin": "https://provalab.io", "localStorage": [{"name": "cookie-consent", "value": "dismissed"}]}]}

What went wrong the first time

Our first version clicked the banner's OK button instead of Dismiss. The run was green, the state file was {"cookies": [], "origins": []}, and the banner was still there in every test. The home page has two buttons called "OK", and the click had landed on the other one. The lesson is the one from Part 1 again: a locator should describe one thing. "Dismiss" is the banner's only button with that accessible name, so that is the one the fixture clicks.

Two tests prove the state works, and show the difference between a context built from the fixture and a bare one:

# test_consent.py
from playwright.sync_api import expect

BANNER = "We use cookies for analytics"


def test_banner_is_gone_with_saved_state(page):
    page.goto("/")
    expect(page.get_by_text(BANNER)).to_have_count(0)


def test_banner_shows_for_a_fresh_visitor(browser, base_url):
    context = browser.new_context(base_url=base_url)   # no storage_state
    page = context.new_page()
    page.goto("/")
    expect(page.get_by_text(BANNER)).to_be_visible()
    context.close()

The same trick is how you handle login. Sign in once in a session fixture, save the state, and every test starts signed in. Put the state file in .gitignore: it holds whatever cookies the site set.

Step 4: Mock the API so no real lead is created (10 minutes)

The free-PDF pages have a form: name and email, then a download link appears. Behind it the page POSTs to a lead API. A test that fills the form for real creates a real lead every run, and someone has to delete "Test Reader" from the inbox each morning.

page.route intercepts requests inside the browser, before they leave it. The test decides what the server says.

As a fixture, so any test can ask for it:

# conftest.py (third part)
@pytest.fixture
def mock_leads(page):
    """Answer the lead API inside the browser, so tests never create a real lead.
    Returns the list of request bodies the page sent."""
    sent = []

    def handle(route):
        sent.append(route.request.post_data_json)
        route.fulfill(status=201, json={"id": 1})

    page.route("**/api/v1/public/leads", handle)
    return sent

The fixture returns the list it appends to, so a test can check what the page sent, not just what it showed afterwards. Here are the guide tests, parametrised over both PDF pages:

# test_guides.py
import pytest
from playwright.sync_api import expect

from pages.guide import GuidePage

GUIDES = [("testing-basics", 23), ("hands-on-testing", 7)]


@pytest.mark.parametrize("slug,parts", GUIDES)
def test_guide_lists_every_part(page, slug, parts):
    guide = GuidePage(page, slug).open()
    expect(guide.parts).to_have_count(parts)


@pytest.mark.parametrize("slug,parts", GUIDES)
def test_pdf_appears_after_the_form(page, mock_leads, slug, parts):
    guide = GuidePage(page, slug).open()
    guide.request_pdf("Test Reader", "reader@example.com")
    expect(guide.download_link).to_have_attribute("href", f"/downloads/{slug}.pdf")
    assert len(mock_leads) == 1
    assert mock_leads[0]["email"] == "reader@example.com"
    assert mock_leads[0]["source_path"] == f"/{slug}/"


def test_rejected_submission_shows_an_error(page):
    page.route("**/api/v1/public/leads", lambda route: route.fulfill(status=422, json={"detail": "invalid"}))
    guide = GuidePage(page, "testing-basics").open()
    guide.request_pdf("Test Reader", "reader@example.com")
    expect(guide.error).to_be_visible()
    expect(guide.download_link).to_have_count(0)

@pytest.mark.parametrize runs a test once per row of GUIDES, so two tests become four, each with its own name in the output. The last test skips the fixture and routes inline, because it wants a different answer: a 422. That is the real value of mocking. A real server gives you the happy path all day; the sad paths, a rejection, a 500, a timeout, you get on demand only when you are the server.

One rule: mock the things around what you are testing, never the thing itself. Here the subject is the page's behaviour after an answer, so faking the answer is fair. A test of the lead API would have to call the lead API.

Step 5: Run it, then run it in parallel (3 minutes)

The whole suite, ten tests, one browser:

$ pytest -v
test_blog.py::test_blog_lists_posts[chromium] PASSED                     [ 10%]
test_blog.py::test_a_post_opens_from_the_list[chromium] PASSED           [ 20%]
test_blog.py::test_pdf_banner_links_to_both_guides[chromium] PASSED      [ 30%]
test_consent.py::test_banner_is_gone_with_saved_state[chromium] PASSED   [ 40%]
test_consent.py::test_banner_shows_for_a_fresh_visitor[chromium] PASSED  [ 50%]
test_guides.py::test_guide_lists_every_part[chromium-testing-basics-23] PASSED [ 60%]
test_guides.py::test_guide_lists_every_part[chromium-hands-on-testing-7] PASSED [ 70%]
test_guides.py::test_pdf_appears_after_the_form[chromium-testing-basics-23] PASSED [ 80%]
test_guides.py::test_pdf_appears_after_the_form[chromium-hands-on-testing-7] PASSED [ 90%]
test_guides.py::test_rejected_submission_shows_an_error[chromium] PASSED [100%]
============================= 10 passed in 27.14s ==============================

Now the same suite on four workers. pytest-xdist starts four pytest processes and deals the tests out between them; each process gets its own browser and its own session fixtures, which is why the state file is named after the worker:

$ pytest -q -n 4
..........                                                               [100%]
10 passed in 11.56s

Ten tests don't need four workers. A hundred do, and parallel only works if the tests were written as if they would run in any order, each starting from its own goto() and sharing nothing but read-only state. That is why Part 2's habit, "no test depends on another", matters before the suite is big, not after.

Everything from Part 2 still applies on top: --headed to watch, --tracing retain-on-failure to keep a trace of anything red, --browser firefox to run the same ten tests in a second engine.

Habits worth keeping

1. Page objects hold locators and actions, never assertions. The test decides what should be true; the page object only knows where to look.
2. A fixture per recurring need. If two tests share three setup lines, that is a fixture.
3. Session scope for anything expensive and read-only: saved state, a signed-in session, test data that doesn't change.
4. Mock what creates real records, and the sad paths. Everything else, hit for real.
5. Write every test to run alone and in any order. Then -n 4 is free when you need it.

What's next

This suite runs on one laptop and one browser. The next step is to run it on the browsers and devices your users actually have, keep every trace somewhere the whole team can see it, and have failures land with the agent that wrote the code. That is what ProvaLab's QA Infrastructure does: the same Playwright tests connect to real browsers and devices in the cloud, every session is recorded, and a failing test's trace goes straight back to your coding agent through the ProvaLab plugin or MCP. See it in the two-minute product tour.

If you came here without doing Part 2 first, start at Your first Playwright test. The whole Hands-On series, seven parts, is also one free PDF.

Fundamentals

1. What is software testing? Checking that software does what it should, and doesn't do what it shouldn't. A test has a starting point, an action and an expected result; a mismatch is a bug. (Part 1)

2. Can testing prove software is bug-free? No. Testing shows the presence of defects, never their absence. Its job is to find the important bugs before users do, and to report the risk that remains. (The 7 principles)

3. Why is exhaustive testing impossible? The number of possible inputs and paths is astronomically large; a ten-character field alone has quadrillions of values. So we choose tests by risk and by technique. (The 7 principles)

4. What are the levels of testing? Unit (one piece of code), integration (pieces together), system (the whole application, end to end) and acceptance (does it meet the real need). Each catches what the others miss. (Part 2)

5. Explain the testing pyramid. Many fast unit tests at the base, fewer integration tests, and a few slow end-to-end tests at the top. The inverted version, mostly end-to-end tests, is slow and flaky. (Part 2)

6. What is the difference between QA, QC and testing? QA improves the process to prevent defects; QC checks the product to find them; testing is the main QC activity. Job titles blur this, but the definitions don't. (Part 20)

7. Verification vs validation? Verification: are we building the product right, against its specification? Validation: are we building the right product, for its users? A product can pass one and fail the other. (Part 20)

Process and planning

8. What is the STLC? The software testing life cycle: requirement analysis, test planning, test case design, environment setup, execution and closure, each with entry and exit criteria. It runs alongside development, not after it. (Part 4)

9. What are entry and exit criteria? Entry: what must be true before a phase starts, such as "build deployed, smoke test passed". Exit: what must be true to finish, such as "all high-priority tests passed, no open critical bugs". They stop testing starting too early or never ending. (Part 4)

10. What goes in a test plan? Objective, scope in and out, approach, environment, entry and exit criteria, people, schedule, deliverables and risks. The most underrated line is "out of scope". (Part 8)

11. Test plan vs test strategy? A strategy is the organisation's long-term approach; a plan applies it to one release or project. (Part 8)

12. What is risk-based testing? Spending the most testing effort where failure is both likely and harmful. Score each area on likelihood and impact, and test the top-right of that grid first and deepest. (Part 8)

13. What does "shift left" mean? Moving testing earlier: reviewing requirements, writing tests before code, running checks on every commit. Bugs found earlier are cheaper to fix. (Part 23)

Test design

14. What is a test case, and what does a good one contain? A set of steps with specific data that checks one thing and states the expected result. It has an ID, a title, preconditions, test data, steps, an expected result and a link to its requirement. (Part 6)

15. Test scenario vs test case? A scenario is what to test, in one line; a test case is exactly how to test one part of it. One scenario becomes several cases. (Part 6)

16. What is boundary value analysis? Testing at and around the edges of a valid range, because bugs like < instead of <= only show at the boundary. For a 0 to 100 field: -1, 0, 1, 99, 100, 101. (Part 6)

17. What is equivalence partitioning? Grouping inputs the software should treat the same, and testing one value from each group instead of all of them. (Part 6)

18. Black-box vs white-box vs grey-box testing? Black-box tests from the outside, against requirements. White-box designs tests from the code's structure. Grey-box tests from outside with some knowledge of the inside, such as the database or the API. (Part 7)

19. Statement coverage vs branch coverage? Statement: did every line run? Branch: did every decision go both true and false? One test can give 100% statement and only 50% branch coverage. (Part 7)

Bugs and reporting

20. What makes a good bug report? A title that says where, what and when; the environment; the fewest steps to reproduce; expected vs actual result; severity and priority; and evidence. Test: could a stranger reproduce it without asking you anything? (Part 9)

21. Severity vs priority? Severity is how badly the bug breaks the software; priority is how soon it must be fixed. A misspelled brand name on the home page is low severity, high priority. (Part 9)

22. Describe the bug life cycle. New, assigned, open, fixed, ready to retest, retest, verified, closed; or reopened if the retest fails. Side exits: rejected, duplicate, deferred, can't reproduce. (Part 10)

23. Who should verify a fix? Ideally the person who found the bug. They know exactly what was wrong, and they retest on every platform the bug affected. (Part 10)

24. What do you do when a developer says "works on my machine"? Add what the report was missing: exact data, environment, a recording. "Can't reproduce" usually means the report lacked something, not that the bug isn't real. (Part 9)

Test types

25. Smoke vs sanity testing? Smoke: a wide, shallow check of every main feature on each new build, to decide whether it's worth testing at all. Sanity: a narrow, deeper check of one area after a change. (Part 14)

26. Regression testing vs re-testing? Re-testing confirms a specific bug is fixed by running the failed test again. Regression testing runs previously passing tests to check the change broke nothing else. Every fix needs both. (Part 15)

27. What is integration testing, and what are stubs and drivers? Testing that separately built parts work together. A stub stands in for a part that is called; a driver stands in for a part that calls. Top-down integration uses stubs, bottom-up uses drivers. (Part 12)

28. System testing vs end-to-end testing? End-to-end follows one complete user journey; system testing covers those journeys plus non-functional checks of the whole system, in a production-like environment. (Part 13)

29. Name the types of performance testing. Load (expected peak), stress (past the peak, to the breaking point), spike (sudden rush), soak (hours, for leaks), volume (large data), scalability (does adding servers add capacity). (Part 16)

30. Why use percentiles instead of the average response time? The average hides the slow tail. A p95 of 1.8 s means one in twenty users waits that long, even if the average looks fine. (Part 16)

31. What is static testing? Finding defects without running the software: reviews, walkthroughs, inspections and static analysis tools. It catches missing requirements and design flaws that no executed test can. (Part 21)

Runs, coverage and traceability

32. Test case vs test run vs test result? A test case is written once; a run executes a set of cases on a specific build and environment; a result is what happened to one case in one run. (Part 17)

33. What do blocked and skipped mean, and why not count them as failed? Blocked: the test couldn't run because something else is broken. Skipped: deliberately not run. Neither tells you anything about the feature, so they're reported separately. (Part 17)

34. How do you calculate pass rate? Of executed tests: passed ÷ (passed + failed). Of planned tests: passed ÷ all planned. Report both; the gap is the blocked and skipped tests. (Part 17)

35. What is a requirements traceability matrix? A table linking each requirement to its test cases, results and bugs. It shows requirements with no tests, failing requirements, and tests that cover nothing. (Part 11)

36. What kinds of test coverage are there? Requirement coverage, execution coverage, code coverage (line, branch, function) and platform coverage. Each has a formula and none of them means bug-free at 100%. (Part 18)

Automation and tools

37. How do you decide what to automate? Three questions: will it run again and again, is the feature stable, is pass/fail clear? Yes to all three, automate; any no, keep it manual for now. (Part 3)

38. What is a flaky test, and why is it dangerous? One that passes and fails on different runs with no code change. Once people re-run red tests until they pass, real failures get re-run away too. (Hands-On 1)

39. What's the best way to locate elements in a UI test? By role and accessible name first, then test IDs, then visible text; CSS tied to styling and positional XPath last. The locator decides whether the test survives a redesign. (Hands-On 1)

40. Selenium vs Playwright vs Cypress? Selenium: the standard, most languages, widest browser and device coverage, manual waiting. Playwright: fast, auto-waiting, all three browser engines, best debugging. Cypress: best developer experience for JavaScript front-end teams. New projects usually pick Playwright. (Hands-On 1)

41. What is TDD? Write a failing test first, write just enough code to pass it, then clean up. The failing test proves the test can catch the bug. (Part 23)

API and security

42. Why test the API before the UI? API tests run in milliseconds, are stable, start before the screens exist, pinpoint the failing endpoint and cover every client at once. Test the business rules there; keep a few UI tests for journeys. (Hands-On 3)

43. 401 vs 403 vs 404? 401: not authenticated, who are you? 403: authenticated but not allowed. 404: not found. Assert the exact one; an API that returns 404 for someone else's record leaks that it exists. (Hands-On 4)

44. What is the most common serious API security bug? Broken access control: a user reading or changing another user's data. The test is four lines: sign in as one user, request another's record, expect 403. (Hands-On 5)

45. What is the OWASP Top 10? The ten most serious web application risks, updated from real-world data. The 2025 edition leads with broken access control and misconfiguration, and adds supply chain failures. (Hands-On 5)

46. SAST vs DAST? SAST reads the source code without running it, on every commit. DAST attacks the running application from outside, nightly or before release. They find different things. (Hands-On 6)

Agile, CI/CD and you

47. How does testing work in an agile team? Quality is the whole team's job, testing happens inside the sprint, acceptance criteria are the test cases, and regression is automated so people can explore. (Part 23)

48. What runs at each stage of a CI/CD pipeline? Every commit: static analysis and unit tests. Every pull request: integration and API tests. Merge: build and smoke test. Nightly: full regression and performance. Release candidate: system and exploratory testing. Production: smoke test and monitoring. (Part 23)

49. Tell me about a bug you found that the tests missed. Pick one that came from exploring beyond the script: an unusual sequence, an edge case, a state nobody considered. Explain how you found it, how you reported it, and what test you added so it stays fixed. (Part 22)

50. How do you decide testing is done? Against the exit criteria agreed in the plan, never against the deadline alone. Report what was tested, what wasn't, what's still open and the risk that remains. (Part 4)

How to use this before an interview

1. Read the short answers once. Mark any you couldn't explain to a colleague.
2. Follow the link for each marked one; every post is a ten-minute read.
3. For questions 7, 21, 25, 26 and 43, the pairs people confuse, practise saying the difference out loud.
4. Prepare one real story for question 49. Interviewers remember stories, not definitions.

Good luck. The whole Testing Basics series and the Hands-On Testing series are free, and every code sample in them actually runs.

What is an API?

An API (application programming interface) is the set of requests a program can make to another program. In a modern app, every screen talks to the server through an API: when you tap "Place order", the mobile app sends a request like POST /orders with the order details, and the server replies with the result.

Most web and mobile APIs are HTTP APIs, usually following the REST style:

• A request names a resource with a URL path, such as /orders/42.
• A method says what to do with it: GET (read), POST (create), PUT or PATCH (update), DELETE.
• Data travels as JSON, a plain-text format of fields and values.
• The reply carries a status code: 200 means OK, 201 created, 400 bad request, 401 not signed in, 403 not allowed, 404 not found, 500 the server broke.

The API is where the real logic lives. Whether a customer orders from the mobile app, the website or the restaurant dashboard, the same POST /discounts/apply decides what the discount is. A bug there shows up once, for everyone.

What is API testing?

API testing sends requests directly to the API and checks the responses, without any user interface.

A test is still locate, act, assert, with "locate" replaced by "build the request":

1. Build a request: method, URL, headers, body.
2. Send it.
3. Assert on the response: status, headers, body and timing.

In Python with the requests library it looks like this (we'll run tests like it for real in Part 4):

response = requests.post(
    "https://api.example.com/discounts/apply",
    json={"order_total": 500, "code": "SAVE10"},
    headers={"Authorization": f"Bearer {token}"},
)
assert response.status_code == 200
assert response.json()["total"] == 450

No browser, no page load, no waiting for elements. That test runs in a few milliseconds.

Why test the API before the UI

The pattern most teams settle on is the testing pyramid applied to the back end: test the business rules thoroughly at the API level, and keep a few UI tests for the journeys users see. The discount logic gets fifty API tests and one UI test, not the other way round.

A second reason that's easy to miss: APIs are products too. If partners or a mobile team build on your API, a change that breaks it breaks them, whether or not your own website still works.

What to check in every response

1. Status code. 200 for success, the right 4xx for bad input or missing permission, and never a 500. A 500 means the server crashed instead of handling the situation.
2. Headers. The content type is what the client expects (application/json); caching is right for the data; and nothing sensitive leaks in a header.
3. Body shape, the contract. The fields the API promises are present, with the right types: total is a number, not the string "450". Clients break on shape changes even when the values are right.
4. Values, the business rule. 500 with SAVE10 must give total: 450. This is the check that finds the bug that charges every customer the wrong amount.
5. Timing. The response arrived within the budget, say under 500 ms. One slow endpoint makes every screen that uses it slow.

Kinds of API tests

Functional tests check that each endpoint does its job: create an order, read it back, apply a discount. The bulk of your API suite.

Negative tests send wrong input on purpose, and check that the API refuses it cleanly: a missing field returns 400 with a helpful message, an unknown order returns 404, a discount of 150% is rejected. Many real bugs hide here, as we saw in test design.

Authentication and authorisation tests check that requests without a valid token get 401, and that a signed-in customer gets 403 when asking for someone else's order. The second kind, called broken object-level authorisation, is the most common serious API security bug, and it's easy to test: sign in as customer A, request customer B's order, expect 403.

Contract tests check the shape of requests and responses against an agreed definition, usually an OpenAPI (formerly Swagger) document. When several teams or partners depend on the API, contract tests catch a renamed field before it breaks anyone. Tools such as Pact go a step further, letting each client declare exactly what it needs.

Integration tests through the API check that a sequence works end to end: create an order, apply a code, pay, then read the order back and confirm the total. This is integration testing with the API as the entry point.

Performance tests send many requests at once to find the breaking point, as described in performance testing. Load tests are nearly always API tests.

Test data and authentication

Two practical problems every API tester meets:

• Tokens. Most APIs need a key or a bearer token. Tests get one in a setup step (sign in with a test account, or read a test key from an environment variable), then reuse it. Never put real credentials in the test code or the repository.
• Test data. POST /orders creates real rows. Tests should create what they need, use it, and clean up, or run against a test environment that's reset regularly. A test that depends on "order 42 already existing" will fail the day someone deletes order 42.

Tools

If you're starting from nothing, pytest with requests needs the least setup and produces tests that live next to your code.

API testing and the UI tests you already have

The two work together. The fastest reliable UI suites use the API to set the stage: create the test customer and the order through API calls in a few milliseconds, then open the browser only for the part that needs a browser. The sign-up form gets one UI test; the thousand ways an order can be wrong get API tests.

What's next

In Part 4 we build a small API of our own, so there's nothing to sign up for and nothing we don't own, and write a pytest suite against it: functional, negative, authorisation and contract checks, with the real output.

ProvaLab's QA AI Agent runs API test cases on its own and records each request and response against its test case, and AppSec fuzzes your API for the authorisation and input-handling bugs described above. See it in the two-minute product tour.

That gives three approaches: black-box, white-box and grey-box testing.

Black-box testing

In black-box testing you test without looking at the code. You know what goes in, what should come out, and what the requirements say. The inside is a closed box.

This is how users experience software, and it is how most QA testing is done. You need no programming knowledge, and the tests stay valid even when developers rewrite the code inside.

The main black-box techniques:

1. Equivalence partitioning

Split the inputs into groups the software should treat the same, and test one value from each group. A discount field that accepts 0–100% has three groups: below 0, 0 to 100, above 100. (More detail, with a diagram, in Part 6.)

2. Boundary value analysis

Test at and around every edge: -1, 0, 1, 99, 100 and 101 for that discount field. Mistakes like < instead of <= only show up at the exact boundary.

3. Decision table testing

When the result depends on combinations of conditions, list every combination in a table, and turn each row into a test. For example: new customer yes/no × order over 300 yes/no → free delivery yes/no.

4. State transition testing

Some features behave differently depending on what state they are in. A food order moves through clear states:

Draw the states and the allowed moves, then test:

• Every allowed move: placed → accepted, preparing → cancelled, and so on.
• The forbidden moves: can a customer cancel an order that is already out for delivery? They mustn't, and the app must say so clearly.

State transition testing is ideal for orders, payments, sign-in (locked after 3 wrong passwords?), subscriptions and any kind of workflow.

5. Use case testing

Test complete journeys the way a real user would: "A customer searches for pizza, adds two, applies a code, pays by card and tracks the delivery." Each use case has a main flow and alternative flows (the card is declined, the restaurant closes mid-order).

6. Pairwise testing

When many settings combine, testing every combination explodes. Three devices × three payment methods × two languages makes 18 combinations. Most bugs are triggered by one setting, or a pair of settings, so pairwise testing picks a smaller set where every pair appears at least once. Here, 9 tests cover every pair instead of all 18. Free tools such as PICT generate these sets for you.

White-box testing

In white-box testing you design tests by looking at the code itself. You read the logic and make sure every part of it is exercised. It is mostly done by developers, usually as unit tests (see Part 5).

The key white-box idea is code coverage: how much of the code your tests actually ran. Here is a small function from our delivery app:

def delivery_fee(order_total, is_member):
    fee = 40
    if order_total >= 300:
        fee = 0
    if is_member:
        fee = fee / 2
    return fee

Statement coverage

Has every line run at least once? One test is enough here:

def test_big_order_member():
    assert delivery_fee(400, is_member=True) == 0

Both if conditions are true, so every line runs. We measured it with pytest-cov:

Name          Stmts   Miss  Cover
delivery.py       7      0   100%

100% statement coverage, from one test. Sounds complete, but it isn't.

Branch coverage

Has every decision gone both ways? Each if can be true or false, and our single test only ever took the "true" side. Turn on branch coverage and the gap appears:

Name          Stmts   Miss Branch BrPart  Cover   Missing
delivery.py       7      0      4      2    82%   3->5, 5->7

3->5 means "the if on line 3 never skipped straight to line 5": we never tested a small order. Add one test where both conditions are false:

def test_small_order_non_member():
    assert delivery_fee(100, is_member=False) == 40

Name          Stmts   Miss Branch BrPart  Cover   Missing
delivery.py       7      0      4      0   100%

Now every decision has gone both ways. This is why branch coverage is a stronger measure than statement coverage.

Condition and path coverage

• Condition coverage checks each part of a combined condition separately, such as if is_member and order_total >= 300: has each half been both true and false?
• Path coverage checks every route through the code. Our function has two ifs, so four paths: big+member, big+non-member, small+member, small+non-member. Our two tests cover only two of them. The small-order member case (fee 20) has never been tested.

Path coverage grows very fast with real code, so teams usually aim for good branch coverage and add path tests only for critical logic.

One warning: coverage tells you what code ran, not whether the tests checked the right things. A test with no assert still counts as coverage. Treat coverage as a map of what you haven't tested, not as proof of quality.

Grey-box testing

Grey-box testing sits in between. You test from the outside, like a user, but you use some knowledge of the inside (the database design, the APIs, the logs or the architecture) to design sharper tests and check results more deeply.

Examples from our app:

• Check behind the screen: place an order in the app, then check the database. Is the order saved with the right total and status, or did the screen just show success?
• Go around the UI: the app only lets customers choose quantities 1–10, but what does the API accept if you send 1,000 or -5 directly? Knowing there is an API makes this test possible.
• Use the logs: trigger a failed payment and check the server logs for a clear error, with no card details written in plain text.
• Integration points: knowing that orders are passed to the restaurant through a message queue, test what happens when that queue is slow or down.

Grey-box testing is common in integration, API and security testing, where knowing a little about the inside finds bugs a pure black-box tester would never reach.

Experience-based techniques

Alongside the three boxes, testers rely on experience:

• Error guessing: "what usually breaks?" Empty fields, very long text, special characters, double clicks, the back button, a lost network.
• Exploratory testing: learning, designing and testing at the same time, guided by curiosity and a time limit.
• Checklists: a team's growing list of things that have gone wrong before.

Comparing the three

The last row matters most: each approach misses what the others catch. White-box testing can't find a feature nobody built, and black-box testing can't see a branch nobody reached. Good teams use all three.

What's next

In Part 8 we put it together in a test plan: what to test, how, by whom and when.

ProvaLab helps on every side of the box: Test Management keeps your black-box test cases, the coding agent writes and runs unit tests from the code, and API and security testing reach behind the UI. See it in the two-minute product tour.

That journey is the bug life cycle, also called the defect life cycle. Knowing it well means fewer bugs get lost, and nobody wonders "wait, who has this one?"

The bug life cycle at a glance

Every status, one by one

New

A tester (or a user, or an automated test) has reported the bug. Nobody has looked at it yet. This is where every bug starts.

Assigned

A test lead, product owner or developer lead has triaged it: confirmed it's a real bug, set its priority, and assigned it to a developer.

Open (or In Progress)

The developer has started working on it: reproducing it, finding the cause, writing the fix.

Fixed

The developer has changed the code and believes the bug is solved. Good practice is to also add an automated test that would have caught it, so it can't quietly come back.

Ready to retest

The fix has been deployed to the test environment, and the bug is waiting for a tester. A fix that only exists on the developer's laptop can't be retested yet. This status makes that clear.

Retest

The tester follows the original steps from the bug report again, on the new build.

Verified

The retest passed: the bug is gone. The tester also runs a few regression tests around it, to make sure the fix didn't break something nearby.

Closed

The bug is done. Usually the tester closes it after verifying, or it closes automatically when the fix is released.

Reopened

The retest failed: the bug is still there, or it only went away in some cases. The tester reopens it with a comment on what still happens, and it goes back to the developer.

The side exits

Not every bug follows the main path. During triage, a bug can also end up as:

"Can't reproduce" is usually a sign the report was missing something, which is why the habits in Part 9 matter so much.

Who moves the bug?

Each status change belongs to someone. If nobody owns a step, bugs get stuck there:

The rule worth remembering: the person who found the bug should be the one who verifies the fix. They know exactly what was wrong.

One bug, start to finish

Let's follow BUG-231 from Part 9, "Expired discount code accepted at checkout":

1. New (Mon 10:05): the tester reports it, with steps, a recording and a link to test case TC-104.
2. Assigned (Mon 11:30): the team lead sets priority High, because it costs money, and assigns it to a developer.
3. Open (Mon 14:00): the developer finds that the code checks the start date of a discount but never the end date.
4. Fixed (Mon 16:20): the developer adds the missing check and an automated test for an expired code.
5. Ready to retest (Tue 09:00): the fix is deployed to staging.
6. Retest → Reopened (Tue 10:15): OLD10 is now refused on the web, but still accepted in the Android app, which has its own copy of the check. The tester reopens the bug with that detail.
7. Fixed again (Tue 15:00): the check moves to the server, so every app uses the same rule.
8. Retest → Verified (Wed 09:30): refused on web and Android; regression tests for valid codes still pass.
9. Closed (Wed 09:35).

The reopen in step 6 is not a failure of the process. It is the process working. Without a proper retest, half the bug would have shipped.

Keeping the bug tracker healthy

• Triage regularly, ideally every day in busy periods, so no bug sits in "New" for a week.
• Watch for stuck bugs. A bug in "Fixed" for days probably never reached the test environment.
• Watch the reopen rate. If many bugs come back, fixes are being rushed or retests skipped.
• Don't let "Deferred" become a graveyard. Review deferred bugs before each release.
• Comment on every status change. "Reopened: still accepted in Android 2.4.0" is far more useful than a silent change.

Your tool's names may differ

Jira, Azure DevOps, GitHub Issues and other trackers each use their own names and allow teams to customise the flow. Some keep it as simple as To Do → In Progress → Done. The ideas stay the same: someone reports it, someone decides, someone fixes, someone checks, and only then is it closed.

What's next

In Part 11 we connect everything with the requirements traceability matrix: the table that proves every requirement has tests, and shows which ones are failing.

In ProvaLab, issues follow this life cycle with their status, priority and assignee, are linked to the test case and test run that found them, and can be raised straight from a failed automated test. See it in the two-minute product tour.

Exploratory testing is how skilled testers find those bugs. It's the most misunderstood kind of testing: often dismissed as "just clicking around", when done well it is one of the most effective techniques there is.

What is exploratory testing?

In exploratory testing, the tester designs, runs and learns from tests at the same time, using what they just discovered to decide what to try next.

In scripted testing, the thinking happens first (writing the test case) and the running happens later, often by someone else. In exploratory testing the two happen together: you try something, see what the software does, form a new idea, and try that.

Three things make it different from "just clicking around":

1. It has a goal. A written charter says what to explore and why.
2. It is time-boxed. A session lasts a fixed time, typically 60 to 120 minutes.
3. It is recorded. The tester keeps notes of what they tried, what they found and what questions came up, and reports them afterwards.

This structured form is called session-based test management, developed by James and Jonathan Bach.

Scripted vs exploratory testing

They are not rivals. Scripted tests, especially automated regression tests, guard against what you already know can go wrong. Exploratory testing finds what you don't know yet. Principle 5, tests wear out, is exactly why you need both.

A session, start to finish

1. Write the charter

A charter is one or two sentences describing the mission. A useful template, from Elisabeth Hendrickson's Explore It!:

Explore (the target) with (resources or approaches) to discover (the information you want).

For our food delivery app:

Explore discount codes at checkout with unusual sequences and inputs to discover ways the discount can be wrong.

A charter is narrow enough to finish in a session and open enough to leave room for discovery. "Test checkout" is too broad; "apply code SAVE10" is a test case, not a charter.

2. Run the session

Set a timer. Start from the charter, and let each result suggest the next idea. Keep notes as you go, in three groups:

• Ideas tried: so you and others know what was covered.
• Bugs found: with enough detail to write a proper bug report afterwards.
• Questions: things the requirements don't answer, and areas you didn't reach.

When something looks wrong, stop and reproduce it before moving on. A bug you can't reproduce is a bug you'll struggle to report.

3. Debrief

At the end, report briefly: what was covered, what was found, what's left. Then decide the next charter. In the example, the discount bugs on the web suggest an obvious follow-up: the same charter on Android.

Heuristics: where ideas come from

Experienced testers don't rely on inspiration. They use heuristics: rules of thumb that generate test ideas on demand. A few of the most useful:

Vary the input

• Boundaries: zero, one, the maximum, one more than the maximum (see Part 6).
• Nothing: empty fields, missing files, no network.
• Too much: very long text, huge files, thousands of items.
• Strange characters: spaces at the ends, emoji, quotes, <script>, text in another alphabet.
• Wrong type: letters in a number field, a date in the past.

Vary the sequence

• Do things in the wrong order. Pay, then add an item.
• Interrupt: go back, refresh, lose the network, close the app mid-way.
• Repeat: click twice, submit twice, apply the same code twice.
• Undo: add then remove, apply then cancel.

Vary the state and environment

• A new user vs a user with 500 orders.
• Two browser tabs doing the same thing at once.
• Switch language, time zone or currency mid-flow.
• Different devices, screen sizes, slow connections.

Follow the data

• Where does this value go after I enter it? Check the order history, the email, the restaurant's screen. Is it the same everywhere?

Think like different people

• A first-time user who misreads the screen. An impatient user. A restaurant owner. Someone trying to get a discount they shouldn't.

Keeping a personal list of heuristics, and adding to it every time a new kind of bug turns up, is one of the fastest ways to become a better tester.

When to use exploratory testing

• New features, before test cases exist or while they're being written.
• Right after a sprint's work lands, to find problems the stories didn't anticipate.
• Around a bug fix, alongside the re-test and regression run.
• Before a release, on the riskiest areas, after the scripted suite has passed.
• When requirements are thin and scripted tests would mostly be guesses.
• For usability: scripts can't tell you a flow is confusing.

Common mistakes

• No charter. Aimless clicking covers the easy paths and misses the rest.
• No notes. If nobody knows what was explored, the session can't be reported, repeated or built on.
• Only happy paths. The value is in the odd sequences and inputs.
• Treating it as a replacement for regression testing. Exploration doesn't guard against old bugs returning; automated scripts do.
• Only junior testers do it. It benefits most from experience; put your most knowledgeable people on the riskiest charters.
• Not reporting bugs properly. A finding in your notes isn't fixed until it's a bug report in the tracker.

What's next

In Part 23, the last part of Testing Basics, we look at testing in agile teams and CI/CD pipelines: how everything in this series fits together when the team ships every week, or every day.

ProvaLab keeps exploratory findings next to scripted results: a bug found in a session is filed with its steps and evidence, linked to the feature, and visible alongside the automated runs for the same release. See it in the two-minute product tour.

Every command and every line of output below was run while writing this post.

What you need: Python 3.10 or newer and a terminal. Nothing to sign up for.

Step 1: Set up (2 minutes)

mkdir first-api-test
cd first-api-test
python3 -m venv .venv
source .venv/bin/activate      # Windows: .venv\Scripts\Activate.ps1
pip install fastapi uvicorn requests pytest

FastAPI builds the API, uvicorn runs it, requests calls it, pytest runs the tests.

Step 2: Build the API (5 minutes)

Create app.py. It's the discount feature from our food delivery app, as an API:

from datetime import date

from fastapi import Depends, FastAPI, Header, HTTPException
from pydantic import BaseModel, Field

app = FastAPI(title="Discount API")

# Who is who: a bearer token maps to a customer (a real API would check a database)
TOKENS = {"token-alice": "alice", "token-bob": "bob"}

# Discount codes: percentage off and the day they expire
CODES = {
    "SAVE10": {"percent": 10, "expires": date(2099, 1, 1)},
    "OLD10": {"percent": 10, "expires": date(2025, 11, 1)},
}

# Orders and who owns them
ORDERS = {42: {"customer": "alice", "total": 500}, 43: {"customer": "bob", "total": 300}}


def current_customer(authorization: str = Header(default="")) -> str:
    token = authorization.removeprefix("Bearer ").strip()
    if token not in TOKENS:
        raise HTTPException(status_code=401, detail="sign in first")
    return TOKENS[token]


class ApplyDiscount(BaseModel):
    order_total: float = Field(gt=0)
    code: str


@app.post("/discounts/apply")
def apply_discount(body: ApplyDiscount, customer: str = Depends(current_customer)):
    code = CODES.get(body.code.upper())
    if code is None:
        raise HTTPException(status_code=404, detail="code not recognised")
    if code["expires"] < date.today():
        raise HTTPException(status_code=400, detail="this code has expired")
    discount = body.order_total * code["percent"] / 100
    return {"total": body.order_total - discount, "discount": discount, "code": body.code.upper()}


@app.get("/orders/{order_id}")
def get_order(order_id: int, customer: str = Depends(current_customer)):
    order = ORDERS.get(order_id)
    if order is None:
        raise HTTPException(status_code=404, detail="order not found")
    if order["customer"] != customer:
        raise HTTPException(status_code=403, detail="not your order")
    return {"id": order_id, **order}

Two endpoints:

• POST /discounts/apply takes an order total and a code, and returns the discounted total. Unknown codes get 404, expired ones 400, and a total of zero or less is rejected with 422 (FastAPI validates the body for us, because of Field(gt=0)).
• GET /orders/{id} returns an order, but only to the customer who owns it: 401 with no valid token, 403 for someone else's order, 404 if it doesn't exist.

Start it in a terminal and leave it running:

uvicorn app:app

Then make one request by hand, to see what a response looks like:

curl -i -X POST http://127.0.0.1:8000/discounts/apply \
  -H "Authorization: Bearer token-alice" \
  -H "Content-Type: application/json" \
  -d '{"order_total": 500, "code": "SAVE10"}'

HTTP/1.1 200 OK
content-length: 47
content-type: application/json

{"total":450.0,"discount":50.0,"code":"SAVE10"}

Status, headers, body: the three things from Part 3, in that order.

Step 3: Fixtures, so every test doesn't repeat itself (2 minutes)

Open a second terminal in the same folder, with the virtual environment activated. Create conftest.py, where pytest looks for shared setup:

import os

import pytest
import requests

BASE_URL = os.environ.get("API_URL", "http://127.0.0.1:8000")


@pytest.fixture
def api():
    """A session signed in as Alice. Every request through it carries her token."""
    session = requests.Session()
    session.headers["Authorization"] = "Bearer token-alice"
    session.base = BASE_URL
    yield session
    session.close()


@pytest.fixture
def anonymous():
    """A session with no token at all."""
    session = requests.Session()
    session.base = BASE_URL
    yield session
    session.close()

A fixture is setup that a test asks for by name. api gives a session already signed in as Alice; anonymous gives one with no token. The address comes from an environment variable, so the same tests can later point at a staging server: API_URL=https://staging.example.com pytest.

Step 4: The discount tests (3 minutes)

Create test_discounts.py:

def test_valid_code_reduces_the_total(api):
    r = api.post(f"{api.base}/discounts/apply", json={"order_total": 500, "code": "SAVE10"})
    assert r.status_code == 200
    assert r.json() == {"total": 450, "discount": 50, "code": "SAVE10"}


def test_response_matches_the_contract(api):
    r = api.post(f"{api.base}/discounts/apply", json={"order_total": 500, "code": "SAVE10"})
    body = r.json()
    assert r.headers["Content-Type"] == "application/json"
    assert set(body) == {"total", "discount", "code"}
    assert isinstance(body["total"], (int, float))
    assert isinstance(body["discount"], (int, float))
    assert isinstance(body["code"], str)
    assert r.elapsed.total_seconds() < 0.5


def test_code_is_not_case_sensitive(api):
    r = api.post(f"{api.base}/discounts/apply", json={"order_total": 500, "code": "save10"})
    assert r.status_code == 200
    assert r.json()["total"] == 450


def test_unknown_code_is_refused(api):
    r = api.post(f"{api.base}/discounts/apply", json={"order_total": 500, "code": "NOPE99"})
    assert r.status_code == 404
    assert r.json()["detail"] == "code not recognised"


def test_expired_code_is_refused(api):
    r = api.post(f"{api.base}/discounts/apply", json={"order_total": 500, "code": "OLD10"})
    assert r.status_code == 400
    assert "expired" in r.json()["detail"]


def test_zero_or_negative_total_is_rejected(api):
    r = api.post(f"{api.base}/discounts/apply", json={"order_total": 0, "code": "SAVE10"})
    assert r.status_code == 422


def test_missing_field_is_rejected(api):
    r = api.post(f"{api.base}/discounts/apply", json={"order_total": 500})
    assert r.status_code == 422

Notice how the five checks from Part 3 show up:

• Status code in every test, and the exact one: 404 for an unknown code, 400 for an expired one, 422 for bad input. Not "some 4xx".
• Headers and body shape in test_response_matches_the_contract: the content type, exactly the three promised fields, the right types. This is the test that catches a renamed field before it breaks the mobile app.
• Values in the first test: 500 with SAVE10 is 450, full stop.
• Timing with r.elapsed: under half a second.
• Negative cases are four of the seven tests. That's roughly the right proportion.

Step 5: The authorisation tests (2 minutes)

Create test_auth.py:

def test_no_token_means_401(anonymous):
    r = anonymous.post(f"{anonymous.base}/discounts/apply", json={"order_total": 500, "code": "SAVE10"})
    assert r.status_code == 401


def test_customer_can_read_their_own_order(api):
    r = api.get(f"{api.base}/orders/42")
    assert r.status_code == 200
    assert r.json()["customer"] == "alice"


def test_customer_cannot_read_someone_elses_order(api):
    r = api.get(f"{api.base}/orders/43")  # Bob's order, Alice's token
    assert r.status_code == 403


def test_unknown_order_is_404(api):
    r = api.get(f"{api.base}/orders/999")
    assert r.status_code == 404

test_customer_cannot_read_someone_elses_order is the most valuable test in this post. Signing in as one customer and asking for another's data is the single most common serious API security bug, and it takes four lines to check. Note the exact code: 403, not "any error". An API that answered 404 here would be quietly telling an attacker that order 43 exists.

Step 6: Run it (1 minute)

pytest -v

collected 11 items

test_auth.py::test_no_token_means_401 PASSED                             [  9%]
test_auth.py::test_customer_can_read_their_own_order PASSED              [ 18%]
test_auth.py::test_customer_cannot_read_someone_elses_order PASSED       [ 27%]
test_auth.py::test_unknown_order_is_404 PASSED                           [ 36%]
test_discounts.py::test_valid_code_reduces_the_total PASSED              [ 45%]
test_discounts.py::test_response_matches_the_contract PASSED             [ 54%]
test_discounts.py::test_code_is_not_case_sensitive PASSED                [ 63%]
test_discounts.py::test_unknown_code_is_refused PASSED                   [ 72%]
test_discounts.py::test_expired_code_is_refused PASSED                   [ 81%]
test_discounts.py::test_zero_or_negative_total_is_rejected PASSED        [ 90%]
test_discounts.py::test_missing_field_is_rejected PASSED                 [100%]

============================== 11 passed in 0.08s ==============================

Eleven tests, 0.08 seconds. The three browser tests in Part 2 took 10.68 seconds. That's the speed difference from Part 3 made concrete: about a hundred times faster per test.

Step 7: Plant a bug (2 minutes)

Change one line in app.py so the discount is subtracted as a flat amount instead of a percentage:

    discount = code["percent"]  # bug: flat amount, not a percentage

Restart uvicorn and run the tests again:

....F.F....                                                              [100%]
=================================== FAILURES ===================================
______________________ test_valid_code_reduces_the_total _______________________

    def test_valid_code_reduces_the_total(api):
        r = api.post(f"{api.base}/discounts/apply", json={"order_total": 500, "code": "SAVE10"})
        assert r.status_code == 200
>       assert r.json() == {"total": 450, "discount": 50, "code": "SAVE10"}
E       AssertionError: assert {'total': 490...de': 'SAVE10'} == {'total': 450...de': 'SAVE10'}
E         Differing items:
E         {'discount': 10} != {'discount': 50}
E         {'total': 490.0} != {'total': 450}

test_discounts.py:4: AssertionError
_______________________ test_code_is_not_case_sensitive ________________________
>       assert r.json()["total"] == 450
E       assert 490.0 == 450

test_discounts.py:21: AssertionError
========================= short test summary info ============================
FAILED test_discounts.py::test_valid_code_reduces_the_total - AssertionError:...
FAILED test_discounts.py::test_code_is_not_case_sensitive - assert 490.0 == 450
2 failed, 9 passed in 0.06s

Exactly the two tests that check values fail; the nine that check status codes, shape and permissions still pass, because the bug didn't touch those. pytest's diff says precisely what's wrong: discount 10 != 50. Put the line back, and you're green again.

What this suite doesn't cover, and what to add next

• A sequence: create an order, apply a code, read it back. That's integration testing through the API.
• Contract from a document: FastAPI generates an OpenAPI document at /openapi.json; tools like Schemathesis can generate hundreds of tests from it.
• Load: the same endpoints under 200 users at once, as described in performance testing types; we build one with k6 later in this series.
• Running in CI: the same pytest step as in Part 2's workflow, with uvicorn app:app & started first, or a test server fixture.

Habits worth keeping

1. Assert the exact status code. The difference between 401, 403 and 404 is the whole point.
2. Test the contract separately from the values. Shape changes break clients even when the maths is right.
3. Make negative tests at least a third of the suite. That's where the bugs live.
4. Always test "someone else's data". One test, four lines, the most common serious API bug.
5. Never put real tokens in the code. Read them from the environment; use test accounts.

What's next

In Part 5 we move to security testing: the OWASP Top 10 in plain language, and which of those risks the tests you just wrote already touch.

ProvaLab's QA AI Agent runs API test cases like these on its own and records every request and response against its test case; AppSec's API fuzzing sends thousands of malformed and unauthorised requests to find the 403-that-should-have-been and the 500-that-should-have-been-a-400. See it in the two-minute product tour.

We will use Python and pytest, the most popular way to write tests in Python. pytest needs almost no setup, and its tests are plain Python functions.

What you need: Python 3.10 or newer, and a terminal. Check your version with python3 --version (on Windows: python --version). No testing experience required.

Step 1: Set up a project (2 minutes)

Create a folder and a virtual environment: a private space for this project's Python packages, so they don't mix with anything else on your computer.

mkdir first-test
cd first-test
python3 -m venv .venv

Turn the environment on:

# macOS and Linux
source .venv/bin/activate

# Windows (PowerShell)
.venv\Scripts\Activate.ps1

Your prompt now starts with (.venv). Install pytest:

pip install pytest

Step 2: Write the code we want to test (2 minutes)

We will keep using the food delivery app from earlier parts. Create a file called shop.py with one function that adds up an order and applies a discount:

def order_total(prices, discount_percent=0):
    """Add up the prices and apply a percentage discount."""
    if not 0 <= discount_percent <= 100:
        raise ValueError("discount must be between 0 and 100")
    subtotal = sum(prices)
    return subtotal - subtotal * discount_percent / 100

This is exactly the kind of small, important piece of code that unit tests (from Part 2) are made for: if the total is wrong, every customer pays the wrong price.

Step 3: Write your first tests (3 minutes)

Create a second file called test_shop.py:

from shop import order_total


def test_total_adds_up_prices():
    assert order_total([200, 200, 50]) == 450


def test_discount_is_applied():
    assert order_total([200, 200, 50], discount_percent=10) == 405


def test_empty_cart_is_zero():
    assert order_total([]) == 0

That's it. Three things to notice:

• The file name starts with test_, and so does each function. That is how pytest finds your tests automatically.
• Each test checks one thing, and its name says what. When it fails, the name alone tells you what broke.
• assert is plain Python: "this must be true". If it isn't, the test fails.

Each test follows the pattern from Part 1: a starting point (some prices), an action (call order_total) and an expected result (the number after ==).

Step 4: Run the tests (1 minute)

pytest -v

You should see:

collected 3 items

test_shop.py::test_total_adds_up_prices PASSED                           [ 33%]
test_shop.py::test_discount_is_applied PASSED                            [ 66%]
test_shop.py::test_empty_cart_is_zero PASSED                             [100%]

============================== 3 passed in 0.01s ===============================

Three tests, a hundredth of a second. The -v (verbose) flag lists each test by name; without it, pytest prints one dot per passing test.

Step 5: Break something on purpose (3 minutes)

A test you have never seen fail proves very little. Let's plant a bug. In shop.py, change the last line so the discount is subtracted as a flat amount instead of a percentage:

    return subtotal - discount_percent   # bug!

Run pytest again:

collected 3 items

test_shop.py .F.                                                         [100%]

=================================== FAILURES ===================================
___________________________ test_discount_is_applied ___________________________

    def test_discount_is_applied():
>       assert order_total([200, 200, 50], discount_percent=10) == 405
E       assert 440 == 405
E        +  where 440 = order_total([200, 200, 50], discount_percent=10)

test_shop.py:9: AssertionError
=========================== short test summary info ============================
FAILED test_shop.py::test_discount_is_applied - assert 440 == 405
========================= 1 failed, 2 passed in 0.02s ==========================

How to read it:

• .F.: the first and third tests passed (.), the second failed (F).
• The > line shows exactly which check failed.
• assert 440 == 405: the code returned 440, but the test expected 405. A 10% discount on 450 should be 45, not 10.
• test_shop.py:9: the file and line number to look at.

The other two tests still pass, because the bug only affects discounts. That's why small, focused tests are so useful: the failing test's name points straight at the problem.

Now put the correct line back, run pytest again, and you're green.

Step 6: Test many cases at once (2 minutes)

Writing a new function for every number gets repetitive. pytest's parametrize runs one test with many inputs. Add this to the end of test_shop.py:

import pytest


@pytest.mark.parametrize(
    "prices, discount, expected",
    [
        ([100], 0, 100),
        ([100], 50, 50),
        ([100], 100, 0),
        ([99, 1], 25, 75),
    ],
)
def test_order_totals(prices, discount, expected):
    assert order_total(prices, discount_percent=discount) == expected

Each row is its own test: if one row fails, you see exactly which inputs broke.

Step 7: Test that bad input is refused (1 minute)

Good tests check what the code shouldn't do, too. Our function must refuse a discount over 100%:

def test_discount_over_100_is_refused():
    with pytest.raises(ValueError):
        order_total([100], discount_percent=150)

This test passes only if the code raises a ValueError. If someone removes that check one day, this test will catch it.

Run everything:

collected 8 items

test_shop.py::test_total_adds_up_prices PASSED                           [ 12%]
test_shop.py::test_discount_is_applied PASSED                            [ 25%]
test_shop.py::test_empty_cart_is_zero PASSED                             [ 37%]
test_shop.py::test_order_totals[prices0-0-100] PASSED                    [ 50%]
test_shop.py::test_order_totals[prices1-50-50] PASSED                    [ 62%]
test_shop.py::test_order_totals[prices2-100-0] PASSED                    [ 75%]
test_shop.py::test_order_totals[prices3-25-75] PASSED                    [ 87%]
test_shop.py::test_discount_over_100_is_refused PASSED                   [100%]

============================== 8 passed in 0.01s ===============================

Eight automated tests, done. Every time you change shop.py, one command tells you in a second whether anything broke.

Handy pytest commands

Five habits of good tests

1. One behaviour per test, with a name that says what it checks.
2. Test the edges: empty lists, zero, the maximum, and invalid input.
3. Watch every new test fail once, so you know it can actually catch a bug.
4. Keep tests fast, so people run them all the time.
5. Run them automatically on every code change, in a CI pipeline.

What's next

You have written unit tests. In Part 6 we step back to the craft behind every test: how to write good test cases, with a template, examples and the techniques testers use to choose them.

When your test suite grows, ProvaLab runs it for you on every change, on real browsers and devices, keeps every result next to its test case, and lets your coding agent read the failures and fix them. See it in the two-minute product tour.

Everything below was run while writing this post, and the output is pasted in unedited.

What you need: Python 3.10 or newer, a terminal, and about 400 MB of disk for a browser. We'll test provalab.io, our own site. Only ever run UI tests against sites you own or have permission to test; point the examples at your own app when you've got the hang of it.

Step 1: Install Playwright (3 minutes)

Create a folder and a virtual environment, as in Testing Basics Part 5:

mkdir first-ui-test
cd first-ui-test
python3 -m venv .venv
source .venv/bin/activate      # Windows: .venv\Scripts\Activate.ps1

Install the pytest plugin for Playwright, then the browser it will drive:

pip install pytest-playwright
playwright install chromium

The second command downloads a copy of Chromium that Playwright controls. It's separate from the Chrome you browse with, so tests never touch your own cookies or tabs.

Step 2: Point the tests at your site (1 minute)

Create pytest.ini so you don't have to repeat the site address in every test:

[pytest]
addopts = --base-url https://provalab.io

Now page.goto("/blog/") means https://provalab.io/blog/. To test a different site, change this one line.

Step 3: Write three tests (5 minutes)

Create test_blog.py:

import re

from playwright.sync_api import Page, expect


def test_blog_lists_posts(page: Page):
    page.goto("/blog/")
    expect(page.get_by_role("heading", level=1)).to_have_text("ProvaLab Blog")
    expect(page.get_by_test_id("blog-card").first).to_be_visible()


def test_a_post_opens_from_the_list(page: Page):
    page.goto("/blog/")
    page.get_by_test_id("blog-card").first.get_by_role("link").first.click()
    expect(page).to_have_url(re.compile(r"/blog/[a-z0-9-]+/$"))
    expect(page.get_by_role("heading", level=1)).to_be_visible()
    expect(page.get_by_role("link", name="All posts")).to_be_visible()


def test_tour_dialog_opens_and_closes(page: Page):
    page.goto("/")
    button = page.locator("#stage-tms").get_by_role("button", name="Watch the full tour")
    button.scroll_into_view_if_needed()
    button.click()
    dialog = page.get_by_role("dialog", name=re.compile("tour", re.I))
    expect(dialog).to_be_visible()
    page.keyboard.press("Escape")
    expect(dialog).to_be_hidden()

What's going on:

• page is a fixture the plugin provides: a fresh browser tab for every test, closed afterwards. Tests can't leak state into each other.
• Locate, act, assert, as in Part 1. get_by_role("button", name="Watch the full tour") finds the button the way a screen reader would; click() acts; expect(...).to_be_visible() asserts.
• expect waits for you. Each assertion retries for up to 5 seconds until the condition is true. That's why there are no sleep() calls, and why these tests won't flake when the network is slow.
• The third test checks a real interaction: a button opens a video dialog, and Escape closes it. Checking that something disappears is just as important as checking that it appears.

Step 4: Run them (1 minute)

pytest -v

collected 3 items

test_blog.py::test_blog_lists_posts[chromium] PASSED                     [ 33%]
test_blog.py::test_a_post_opens_from_the_list[chromium] PASSED           [ 66%]
test_blog.py::test_tour_dialog_opens_and_closes[chromium] PASSED         [100%]

======================== 3 passed, 1 warning in 10.68s =========================

Three real pages loaded in a real browser, a dialog opened and closed, in under 11 seconds. The browser ran headless (no window), which is the default and what CI uses.

(The warning is a deprecation notice inside the plugin itself, not in your code. It's safe to ignore.)

Watch it happen

To see the browser while the test runs, slowed down so you can follow:

pytest --headed --slowmo 500

Do this whenever a test confuses you. Nine times out of ten you'll see the problem within seconds: a cookie banner over the button, a page that hasn't finished loading, a locator pointing at the wrong thing.

Step 5: Break one on purpose (3 minutes)

As in Part 5, a test you have never seen fail proves little. Create test_broken.py with a wrong expectation:

from playwright.sync_api import Page, expect


def test_blog_heading_wrong_on_purpose(page: Page):
    page.goto("/blog/")
    expect(page.get_by_role("heading", level=1)).to_have_text("ProvaLab Articles")

pytest test_broken.py

    def test_blog_heading_wrong_on_purpose(page: Page):
        page.goto("/blog/")
>       expect(page.get_by_role("heading", level=1)).to_have_text("ProvaLab Articles")
E       AssertionError: Locator expected to have text 'ProvaLab Articles'
E       Actual value: ProvaLab Blog
E       Call log:
E         - Expect "to_have_text" get_by_role("heading", level=1) with timeout 5000ms
E         - waiting for get_by_role("heading", level=1)
E           14 × locator resolved to <h1 class="text-3xl sm:text-4xl lg:text-5xl font-bold text-gray-900">ProvaLab Blog</h1>
E              - unexpected value "ProvaLab Blog"
E
E       Aria snapshot:
E       - heading "ProvaLab Blog" [level=1]

test_broken.py:6: AssertionError
========================= 1 failed, 1 warning in 7.98s =========================

This is a much richer failure than a plain assert, and learning to read it is the core skill:

• Expected vs actual are on adjacent lines. Here the test is wrong, not the site.
• "14 × locator resolved to <h1 …>" says Playwright found the element, checked it 14 times over 5 seconds, and it never had the expected text. So this isn't a timing or locator problem; it's a real difference. If the element had not been found, you'd see "waiting for …" with no resolution, and you'd know to fix the locator.
• The aria snapshot shows the page the way a screen reader sees it. When your locator finds nothing, this list tells you what roles and names actually exist, so you can pick the right one.

Step 6: Record a trace (2 minutes)

A trace is a recording of everything the test did: every action, a screenshot before and after each one, the page's HTML, console messages and network calls. Keep one for every failure:

pytest test_broken.py --tracing retain-on-failure

1 failed, 1 warning in 6.59s

The trace lands in test-results/test-broken-py-test-blog-heading-wrong-on-purpose-chromium/trace.zip. Open it:

playwright show-trace test-results/test-broken-py-test-blog-heading-wrong-on-purpose-chromium/trace.zip

The trace viewer opens in a browser and lets you step through the test, see the page at each action, and inspect what was on screen when it failed. For a failure that only happens in CI, a trace is the difference between fixing it in five minutes and not being able to reproduce it at all.

Two related options, both worth adding to pytest.ini:

[pytest]
addopts = --base-url https://provalab.io --tracing retain-on-failure --screenshot only-on-failure

(--video retain-on-failure records a video too, at the cost of slower runs.)

Step 7: Run it in the pipeline (3 minutes)

The same tests, run on every change by GitHub Actions, with traces saved when anything fails:

name: ui-tests
on: [push, pull_request]

jobs:
  ui:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install pytest-playwright
      - run: playwright install --with-deps chromium
      - run: pytest --tracing retain-on-failure
      - uses: actions/upload-artifact@v4
        if: failure()
        with:
          name: playwright-traces
          path: test-results/

--with-deps installs the system libraries the browser needs on a fresh CI machine. The last step uploads the traces as a downloadable artifact whenever the run is red.

When you need more than one machine can give you, many browsers, operating systems and real phones at once, the same tests connect to a browser grid instead of a local Chromium. The test code doesn't change.

Handy options

codegen is a good way to learn the locator syntax: it records your clicks and suggests role-based locators, which you can then tidy up.

Habits worth keeping

1. Prefer get_by_role and get_by_test_id. The #stage-tms CSS selector in the third test is a reasonable exception: it's an ID set on purpose, and it narrows the search to one section.
2. Never time.sleep(). If you feel the need, you're missing an expect.
3. Each test starts from page.goto() and sets up what it needs. No test should depend on another.
4. Keep traces on failure, always. They cost nothing when tests pass.
5. Run headed when confused, headless everywhere else.

What's next

Our three tests all go through the user interface. In Part 3 we go one layer down: API testing, where the same checks run in milliseconds instead of seconds, and why many teams test the API before the UI.

ProvaLab's QA Infrastructure is the "third place" in the diagram above: your Playwright and Selenium tests run on real browsers and devices in the cloud, every session is recorded, and when a test fails the recording and logs go straight back to your coding agent. See it in the two-minute product tour.

We'll use OWASP ZAP, the most widely used free security scanner, against the discount API we built in Part 4. Then we'll plant a real vulnerability and watch the scanner find it.

Every command and every line of output below was run while writing this post.

Before you start: a DAST scan sends thousands of hostile requests. Only ever point it at something you own, and never at a production system: it will create records, trip rate limits and fill logs. Our target runs on our own laptop.

How a DAST scan works

The scanner has to discover the URLs first. For a website it crawls the pages; for an API it reads the OpenAPI document, which lists every endpoint and parameter. FastAPI generates one for us automatically at /openapi.json, which is why the Part 4 API is a convenient target.

Step 1: Get ZAP (2 minutes, plus a download)

ZAP ships as a container image, so there's nothing to install beyond Docker:

docker pull ghcr.io/zaproxy/zaproxy:stable

It's large (about 3.7 GB), because it carries a full Java runtime and a browser.

Step 2: Start the target (1 minute)

In the first-api-test folder from Part 4, start the API and leave it running:

uvicorn app:app

Step 3: Run the scan (5 minutes)

ZAP includes a ready-made API scan script. One command runs discovery, the passive scan and the active scan, and writes a report:

mkdir -p zap && chmod 777 zap
docker run --rm --network host -v "$PWD/zap:/zap/wrk:rw" ghcr.io/zaproxy/zaproxy:stable \
  zap-api-scan.py -t http://127.0.0.1:8000/openapi.json -f openapi \
  -J report.json -w report.md -I \
  -z "-config replacer.full_list(0).description=auth \
      -config replacer.full_list(0).enabled=true \
      -config replacer.full_list(0).matchtype=REQ_HEADER \
      -config replacer.full_list(0).matchstr=Authorization \
      -config replacer.full_list(0).regex=false \
      -config replacer.full_list(0).replacement=Bearer\ token-alice"

What the pieces mean:

• --network host lets the container reach the API on your machine. (On macOS and Windows, use http://host.docker.internal:8000 instead of 127.0.0.1 and drop --network host.)
• -t …/openapi.json -f openapi tells ZAP to import the endpoints from the OpenAPI document.
• -J and -w write JSON and Markdown reports into the zap folder.
• -I stops the script from returning a failure code on warnings, so it finishes cleanly.
• The long -z part configures ZAP's replacer to add Authorization: Bearer token-alice to every request. Without it, every endpoint would answer 401 and the scanner would test nothing but the sign-in wall.
• If the output ends with "Failed to start the main proxy: Address already in use", something on your machine is using port 8080, which ZAP wants for itself. Add -P 8099 (any free port) after zap-api-scan.py. This happened to us on the third run.

Three minutes later, the end of the output:

Number of Imported URLs: 6
Total of 35 URLs
PASS: Directory Browsing [0]
PASS: Vulnerable JS Library (Powered by Retire.js) [10003]
...
PASS: Remote OS Command Injection (Time Based) [90037]
WARN-NEW: X-Content-Type-Options Header Missing [10021] x 1
	http://127.0.0.1:8000/openapi.json (200 OK)
WARN-NEW: Cross-Origin-Resource-Policy Header Missing or Invalid [90004] x 1
	http://127.0.0.1:8000/openapi.json (200 OK)
FAIL-NEW: 0	FAIL-INPROG: 0	WARN-NEW: 2	WARN-INPROG: 0	INFO: 0	IGNORE: 0	PASS: 118

118 checks passed, 2 warnings, no failures. Both warnings are the same kind of thing: a recommended security header the server doesn't send, flagged on the OpenAPI document itself. ZAP rates them Low. For a tiny API with no browser pages they barely matter, but they're real and the fix is one line of middleware, so they go on the backlog.

(A note on the words: the API-scan script reports every finding as WARN unless you give it a rules file saying which ones should FAIL the build. The actual risk level, High to Informational, is in the report.)

What the scanner cannot see

Remember the most valuable test from Part 4: Alice asking for Bob's order and getting 403. ZAP can't find that bug. It has one token, Alice's, and no way of knowing that order 43 isn't hers. If the API wrongly returned Bob's order, the scanner would see a perfectly healthy 200.

This is the single most important thing to understand about security scanners: they find the generic mistakes, not the ones specific to your business rules. The number-one risk in the OWASP Top 10, broken access control, is mostly found by tests you write. The scanner and your test suite are partners, not alternatives.

Step 4: Plant a vulnerability (3 minutes)

To see what a real finding looks like, let's add two things a team might ship by accident. Append this to app.py:

# ---- two things a real team might ship by accident ----
import sqlite3
import traceback

from fastapi import Request
from fastapi.responses import PlainTextResponse

db = sqlite3.connect(":memory:", check_same_thread=False)
db.execute("create table restaurants(name text, cuisine text)")
db.executemany("insert into restaurants values (?, ?)",
               [("Pizza Palace", "pizza"), ("Curry House", "indian"), ("Sushi Go", "japanese")])


@app.get("/restaurants")
def search_restaurants(q: str = "", customer: str = Depends(current_customer)):
    # the query is built by gluing the user's text into the SQL
    rows = db.execute(f"select name, cuisine from restaurants where name like '%{q}%'").fetchall()
    return [{"name": n, "cuisine": c} for n, c in rows]


@app.exception_handler(Exception)
async def show_errors(request: Request, exc: Exception):
    # "debug mode": send the full stack trace back to whoever asked
    return PlainTextResponse("".join(traceback.format_exception(exc)), status_code=500)

The first is A05 Injection: the search text is glued straight into an SQL statement. The second is A10 Mishandling of Exceptional Conditions (and a bit of A02): any crash sends the full stack trace to the caller.

Restart uvicorn, and try the search with a single quote, the classic probe:

curl "http://127.0.0.1:8000/restaurants?q=%27" -H "Authorization: Bearer token-alice"

Traceback (most recent call last):
  ...
sqlite3.OperationalError: unrecognized token: "'"

One character broke the query, and the server told us exactly which database it uses and what went wrong. That's what an attacker's first probe looks like, and what their first clue looks like.

Step 5: Scan again

Run the same docker run command. This time:

WARN-NEW: A Server Error response code was returned by the server [100000] x 1
WARN-NEW: X-Content-Type-Options Header Missing [10021] x 2
WARN-NEW: SQL Injection [40018] x 1
WARN-NEW: Cross-Origin-Resource-Policy Header Missing or Invalid [90004] x 2
FAIL-NEW: 0	FAIL-INPROG: 0	WARN-NEW: 4	WARN-INPROG: 0	INFO: 0	IGNORE: 0	PASS: 117

ZAP found the injection. The detail is in the report:

[High (Low)] SQL Injection (plugin 40018) x1  CWE 89
   uri:      http://127.0.0.1:8000/restaurants?q=%27 | method: GET | param: q
   attack:   '
   evidence: HTTP/1.1 500 Internal Server Error
   solution: Do not trust client side input, even if there is client side validation
             in place. In general, type check all data on the server side. ...
             use PreparedStatement ... with parameters passed by '?'

[Low (High)] A Server Error response code was returned by the server (plugin 100000) x1
   uri:      http://127.0.0.1:8000/restaurants?q=5%3BURL%3D... | method: GET
   evidence: 500

How to read a finding:

• High (Low) means High risk, Low confidence. ZAP saw a 500 when it sent a quote, which is a strong hint but not proof. Confirm every finding by hand before you act on it. We already did: the curl above reproduced it.
• param: q, attack: ' tell you exactly where and how. That's your bug report's steps to reproduce.
• CWE 89 is the standard identifier for SQL injection, useful for looking up fixes and for compliance reports.
• The server-error finding is the symptom of the same bug plus the debug handler: rated Low on its own, but the stack trace it leaks is a gift to an attacker.

Step 6: Triage

A scan report is not a to-do list. Sort it:

• Fix now: the injection and the stack-trace leak. Confirmed, serious, same root cause.
• Backlog: the two headers. Real, minor, one line each.
• Not applicable: the 22 informational notes that the API returned 404 for things that don't exist, and the 4 about responses not being cacheable. An API is supposed to do both. Mark them as not applicable in your tracker so the next scan doesn't repeat them.

Thirty lines of report become two bugs and two backlog items. The bug report for the injection writes itself: the URL, the parameter, the payload ', the expected result (a normal empty list or a 400), the actual result (a 500 with a stack trace), and the CWE.

Step 7: Fix and re-scan

Two changes. The query takes a parameter instead of glued text, and the debug handler goes:

    rows = db.execute("select name, cuisine from restaurants where name like ?", (f"%{q}%",)).fetchall()

Restart, and the single quote now returns an empty list with a 200, as it should:

curl "http://127.0.0.1:8000/restaurants?q=%27" -H "Authorization: Bearer token-alice"

[]

Scan a third time:

Number of Imported URLs: 7
Total of 36 URLs
WARN-NEW: X-Content-Type-Options Header Missing [10021] x 2
WARN-NEW: Cross-Origin-Resource-Policy Header Missing or Invalid [90004] x 2
FAIL-NEW: 0	FAIL-INPROG: 0	WARN-NEW: 2	WARN-INPROG: 0	INFO: 0	IGNORE: 0	PASS: 118

The injection and the server error are gone; 118 checks pass again.

The two header warnings are still there, as expected; they're on the backlog, and a scanner will keep mentioning them until the header is added. That's the right behaviour: a scan is a regression test for security.

Putting it in the pipeline

The same command runs in CI. Start the API, run the scan against it, and keep the report as an artifact:

      - run: pip install -r requirements.txt
      - run: uvicorn app:app &
      - run: mkdir -p zap && chmod 777 zap
      - run: >
          docker run --rm --network host -v "$PWD/zap:/zap/wrk:rw"
          ghcr.io/zaproxy/zaproxy:stable zap-api-scan.py
          -t http://127.0.0.1:8000/openapi.json -f openapi -J report.json -w report.md
          -z "-config replacer.full_list(0).description=auth -config replacer.full_list(0).enabled=true -config replacer.full_list(0).matchtype=REQ_HEADER -config replacer.full_list(0).matchstr=Authorization -config replacer.full_list(0).regex=false -config replacer.full_list(0).replacement=Bearer\ ${{ secrets.TEST_API_TOKEN }}"
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: zap-report
          path: zap/

Without -I, the job fails when ZAP finds anything new, which is what you want once the backlog items are either fixed or listed in a rules file as ignored. Run it nightly rather than on every commit: a full active scan takes minutes, not seconds.

Habits worth keeping

1. Scan with a signed-in session. An unauthenticated scan tests your sign-in page and nothing else.
2. Confirm by hand before raising a bug. "High (Low)" is a lead, not a verdict.
3. Triage into fix-now, backlog, not-applicable, and record the last pile so it stops reappearing.
4. Keep writing access-control tests yourself. The scanner can't.
5. Never scan what you don't own.

What's next

In Part 7, the last of this series, we go hands-on with load testing: a k6 script against the same API, finding where it slows down and where it breaks.

ProvaLab AppSec runs this kind of scan, and API fuzzing, dependency, secrets, container and code scans, against targets you've verified you own, from the cloud, and presents the findings as a plain-language fix list ordered by severity, with your confirmed false positives remembered between runs. See it in the two-minute product tour.

Most developers have already made the switch. In the Stack Overflow Developer Survey 2025, 84% of developers said they use or plan to use AI tools in their work. But trust went the other way: 46% said they actively distrust the accuracy of what those tools produce.

So teams end up in an awkward place. They write code faster than ever, then either slow down to check it by hand, or ship it and hope.

Why AI-written code needs a different kind of checking

AI-written code is not worse than human code by default. It fails in different ways:

• It looks right. The code is tidy and confident, so reviewers skim it.
• It passes its own tests. If the agent wrote the code and the tests together, the tests often check what the code does, not what it should do.
• It handles the happy path. Empty lists, slow networks, expired sessions and two users at once are where it tends to break.
• There is a lot of it. A reviewer who could read 200 lines a day now gets 2,000.

The answer is not to read every line more carefully. It is to make checking as fast as writing.

A simple routine for every AI-written change

1. Write the test cases before the code

Ask the agent for the test cases first, in plain language: what the feature must do, and what must never happen. Read that list yourself. It is ten lines instead of five hundred, and it is where your judgement matters most.

2. Run the tests somewhere real

Tests that only pass on the developer's laptop prove little. Run them on real browsers and devices, against a running build, the way a user would meet the feature.

3. Keep a record of every run

When a test fails next week, you want to know whether it ever passed, on which build, and what changed. A test case and its run history should live in one place, not in a chat log.

4. Check what tests don't cover

Functional tests don't tell you whether an API leaks data or falls over when fifty people use it at once. Add a security scan and a short load test before anything important ships.

5. Send failures back to the agent

This is the step most teams skip. When a test fails, the agent that wrote the code should get the failure, the logs and a recording, and fix it itself. A person approves the result instead of doing the triage.

Where ProvaLab fits

We built ProvaLab so a coding agent can do this whole routine on its own. Through the Claude Code plugin or MCP, the agent runs your tests on cloud browsers and devices, records every result in test management, scans for security issues, runs load tests, and gets the failures back so it can fix them.

You can see it working in the two-minute product tour, or book a 30-minute demo and we will run it against your own app.

AI made writing code fast. Checking it should be just as fast.

A good bug report is one of the most valuable things a tester produces. This part shows you how to write one.

What is a bug report?

A bug report (also called a defect report or an issue) is a written record of a problem: what went wrong, how to make it happen again, and why it matters.

Its job is simple: let someone who has never seen the bug reproduce it, understand it and fix it, without asking you a single question.

Same bug, two reports

The bad report starts a conversation: which code, which app, what happened? The good one ends it.

The fields of a good bug report

Writing a good title

Most people only ever read the title, in a list of fifty bugs. Use this pattern:

[Where] + [what goes wrong] + [under what condition]

• Bad: "Checkout bug"
• Bad: "Discount doesn't work properly"
• Good: "Expired discount code accepted at checkout; total reduced by 10%"
• Good: "Order history page blank on Android 14 after switching language"

Expected vs actual: the heart of the report

Keep these two side by side and specific. "It doesn't work" is not an actual result. "The total becomes 450 and no error appears" is. If the expected behaviour comes from a requirement or test case, link to it: that shows it's a bug and not a matter of opinion.

Severity vs priority

These two get confused constantly, and they are not the same thing:

• Severity is about the software: how badly does the bug break it? Usually set by the tester.
• Priority is about the business: how soon must it be fixed? Usually set by the product owner or team lead.

They often match, but not always:

Common severity levels:

Attachments that save hours

• Screenshot, with the problem area highlighted.
• Screen recording for anything involving several steps or timing. A short video often replaces a page of text.
• Console log from the browser's developer tools (F12 → Console) for web bugs.
• Network log or HAR file when a server request fails. It shows exactly what was sent and what came back.
• Server or app logs, with the time the bug happened.
• Test data: the account, order number or file you used.

Never attach real passwords, card numbers or customers' personal data. Use test accounts, and blur anything sensitive in screenshots.

Ten habits of great bug reporters

1. Reproduce it first. Try at least twice before reporting.
2. Search for duplicates. Someone may already have reported it.
3. One bug per report. Two problems means two reports, because they may be fixed by different people at different times.
4. Use the fewest steps that still trigger the bug.
5. Find the boundaries. Does it happen in other browsers? With other codes? Only for new customers? Each answer narrows the cause.
6. Report facts, not guesses. "The total becomes 450", not "the discount logic is broken".
7. Keep the tone neutral. No blame, no capitals, no "!!!". Developers and testers are on the same side.
8. Write for a stranger. Assume the reader has never seen this part of the app.
9. Say if it's intermittent. "Happens about 2 in 5 tries" is useful; hiding it wastes time.
10. Keep it updated. If you learn more, add it to the report, not to a chat message that will be lost.

A copy-ready template

Title: [Where] [what goes wrong] [under what condition]

Environment: app version, browser/device, OS, server (staging/production)
Preconditions: account, data or settings needed first

Steps to reproduce:
1.
2.
3.

Expected result:
Actual result:

Severity: Critical / High / Medium / Low
Priority: High / Medium / Low
Reproducibility: e.g. 5/5, or 2 in 5 tries

Attachments: screenshot, recording, console log, network log
Links: test case, requirement, related bugs

What's next

Once a bug is reported, its journey has only just begun. In Part 10 we follow it through the bug life cycle: new, assigned, fixed, retested and closed (or reopened).

In ProvaLab, a failed automated test can raise a bug report with the steps, the error, the screen recording and the logs already attached, linked to its test case and test run. Your coding agent can read it and fix the code. See it in the two-minute product tour.

That agreement is the test plan. In the testing life cycle it is Phase 2, and everything after it follows from it.

What is a test plan?

A test plan is a document that describes the scope, approach, resources and schedule of the testing for a release or project.

It answers six questions:

1. Why are we testing? (the objective)
2. What will we test, and what won't we? (scope)
3. How will we test it? (approach)
4. Where will the tests run? (environment)
5. Who does what, and when? (people and schedule)
6. When do we start, and when are we done? (entry and exit criteria)

A good test plan is written for people, not for a filing cabinet. Developers, product managers and testers should all be able to read it in a few minutes and agree.

Test plan vs test strategy

• A test strategy is the long-term, organisation-wide approach: "We automate all regression tests, every release gets a security scan, and we test on the three most-used browsers." It rarely changes.
• A test plan applies that strategy to one release or project: "For release 2.4 we test discount codes on web and Android from 3 to 7 November."

Small teams often have no separate strategy document; its key rules simply live inside each plan.

The sections of a test plan

Formal templates exist. The IEEE 829 standard was the classic one, and it was later replaced by ISO/IEC/IEEE 29119-3. But most teams use a lighter version of the same ideas:

"Out of scope" is the most underrated line in any test plan. Writing down what you won't test avoids the worst surprise of all: a bug in an area everyone thought someone else had checked.

A one-page example

For most releases, the whole plan fits on one page:

If your plan doesn't fit on one or two pages, ask whether every section is really helping someone make a decision.

Risk-based testing: deciding what matters most

There is never time to test everything equally. Risk-based testing spends the most effort where a failure is both likely and harmful:

For each feature, ask two questions:

• How likely is it to break? New code, complex code, code that changed a lot, or code with a history of bugs.
• How bad is it if it breaks? Money, personal data, sign-in and anything that stops customers ordering.

Put the answers in the plan. They decide the order of testing and where the deepest test cases go, and they tell everyone what was tested lightly if time runs out.

How to write one, step by step

1. Read the requirements, and ask questions (STLC Phase 1).
2. Write the objective in one sentence.
3. List what's in and out of scope, with a reason for each "out".
4. Rank the risks, using the matrix above.
5. Choose the approach: which levels and types, and what to automate.
6. Define entry and exit criteria that can be checked, not "when it looks good".
7. Add people, environment and schedule.
8. Review it with developers and the product owner. A plan nobody agreed to isn't a plan.

Test plans in agile teams

Agile teams rarely write long plans for every sprint. They usually keep:

• A light test strategy for the product, updated a few times a year.
• Testing notes in each sprint: acceptance criteria on every user story, and a short checklist of what is risky this sprint.
• A short release plan for bigger releases: scope, risks and exit criteria.

The questions stay the same. Only the paperwork shrinks.

Common mistakes

• Copying last release's plan without changing the scope or risks.
• Vague exit criteria like "testing is complete". Use measurable ones.
• No "out of scope" section, so gaps stay invisible.
• Writing it alone. Developers know where the code is fragile; product owners know what matters most to customers.
• Never updating it. When the scope changes mid-release, the plan should too.

What's next

In Part 9 we look at what happens when a test fails: how to write a good bug report that developers can act on quickly.

In ProvaLab Test Management, a test plan has these fields built in: objective, test lead, dates, entry and exit criteria, and the features in scope. Test cases attach to the plan, and every test run reports back into it. See it in the two-minute product tour.

What is a test case?

A test case is a set of steps, with specific data, that checks one thing about the software and says exactly what result to expect.

Anyone on the team should be able to pick it up, follow it and get the same answer: pass or fail. If two testers can read the same test case and do different things, it isn't finished.

Test scenario vs test case

These two get mixed up a lot:

• A test scenario is what to test, in one line: "Check that discount codes work at checkout."
• A test case is exactly how to test one part of it: which code, which cart, which clicks, which result.

One scenario usually becomes several test cases: a valid code, an expired code, two codes at once, a code on an empty cart, and so on.

The anatomy of a test case

Example: discount codes in the food delivery app

Notice the mix: one positive case (it works when used correctly) and several negative cases (it fails safely when used wrongly). Most real bugs live in the negative cases, so don't skip them.

Techniques for choosing test cases

You can't test every possible input. These techniques help you pick the few cases that catch the most bugs.

Equivalence partitioning

Group inputs that the software should treat the same way, then test one value from each group. If a discount field accepts 0 to 100, there are three groups: below 0, 0 to 100, and above 100. Testing 50 tells you about the whole valid group; you don't need to test 51, 52 and 53.

Boundary value analysis

Bugs love edges. A developer might write < where they meant <=, and only the exact edge value shows it. So test at and around each boundary:

Six well-chosen values catch most of the mistakes a hundred random ones would.

Decision tables

When the result depends on combinations of conditions, list every combination in a table so none is missed. For free delivery:

Each row becomes one test case.

Error guessing

Use experience to guess where things break: empty fields, very long text, special characters like ' or <, double-clicking a button, going back in the browser, or losing the network mid-payment. Experienced testers keep a personal list of these.

Seven habits of good test cases

1. One test, one purpose. If the title needs "and", split it.
2. A clear, specific title. "Apply an expired discount code", not "Discount test 3".
3. Exact test data. "Code OLD10", not "an expired code".
4. Short steps anyone can follow, without knowing the app.
5. A checkable expected result. "Total stays 500", not "it works correctly".
6. Independent. Each test sets up what it needs, so the order doesn't matter.
7. Linked to a requirement, so you know why it exists and what it covers.

Common mistakes

• Vague expected results like "the page should work fine". Nobody can fail that test.
• Only happy paths. Users mistype, double-click and go back. Test that too.
• Giant test cases with 30 steps. When step 17 fails, the rest are useless.
• Copy-paste duplicates that drift apart over time. Reuse and update instead.
• Never reviewing them. A quick review by another tester catches missing cases early.

What's next

In Part 7 we go deeper into test design techniques: black-box, white-box and grey-box testing, and how each one finds different bugs.

In ProvaLab Test Management, every test case has these fields built in, is grouped into suites, links to its requirement, and keeps the history of every run, manual or automated. See it in the two-minute product tour.

Finding those bugs is the job of integration testing.

What is integration testing?

Integration testing checks that separately built parts of the software work correctly together. The parts might be code modules, services, a database, or an outside system like a payment provider.

Each part may pass all its own unit tests and the whole thing can still fail, because the parts disagree about something: a field name, a date format, a unit of money, or what happens when one of them is slow.

Where integration bugs hide

Here is our food delivery app, broken into its main parts:

Unit tests check the boxes. Integration tests check the arrows.

A classic example: the order service sends the payment amount as 450 meaning rupees, but the payment service expects the amount in paise (hundredths of a rupee), so it charges 4.50 instead of 450. Both services pass their unit tests perfectly. Only a test that runs them together catches the bug.

Other typical integration bugs:

• Different field names or formats: one side sends orderId, the other expects order_id; one sends dates as 29/10/2026, the other expects 2026-10-29.
• Missing error handling: the payment service is down, and the order service crashes instead of showing "payment failed, please try again".
• Timeouts: the restaurant notifier takes 30 seconds and the app gives up after 10.
• Wrong assumptions: the order service assumes the database always returns orders newest first, but nobody told the database that.
• Changes on one side: a team renames a field in their API, and the services that depend on it break.

Approaches to integration testing

When there are many parts, in what order do you connect and test them?

Big bang integration

Connect everything at once, then test the whole thing.

• Good: simple; nothing extra to build.
• Bad: when a test fails, the bug could be in any of the connections, so finding it is slow. And you can't start until every part is finished.

It is only reasonable for very small systems.

Incremental integration

Connect and test the parts a few at a time, so when something fails you know it is in the piece you just added. There are three ways to do this.

Top-down

Start with the top-level part (for example, the order service) and add the parts below it one by one. Parts that aren't ready yet are replaced by stubs.

• Good: the main flow of the application is tested early.
• Bad: many stubs to write; the lowest-level parts are tested last.

Bottom-up

Start with the lowest-level parts (for example, the database layer and the payment client) and work upwards. The parts above that don't exist yet are replaced by drivers.

• Good: the foundations are solid before anything is built on them.
• Bad: the full user flow can only be seen at the end.

Sandwich (hybrid)

Do top-down and bottom-up at the same time, meeting in the middle. It is the most common choice for large systems, because different teams can work in parallel.

Stubs and drivers (and other test doubles)

When you test part of a system, the parts it depends on may not be ready, or may be slow, costly or unreliable (no one wants a test that charges a real card). So we replace them with stand-ins, called test doubles:

Test doubles make tests fast and repeatable. But remember: a test against a stub only proves your code works with the stub. At some point you also need tests against the real thing, or against a realistic copy in a test environment.

Two kinds of integration testing

• Component integration testing checks parts inside your system: the order service with your own database and payment service.
• System integration testing (SIT) checks your system with outside systems: the real payment provider's sandbox, an SMS gateway, a maps API.

System integration tests are usually slower and run less often, because outside systems can be slow, rate-limited or briefly unavailable.

What to check at every integration point

For each arrow in your diagram, check:

1. The happy path: a normal request gets the right response, and data arrives correctly on the other side.
2. Data format and units: names, types, dates, currencies, time zones and character encoding all match.
3. Errors: what happens when the other side returns an error, is down, or sends something unexpected.
4. Timeouts and slowness: the caller waits a sensible time and recovers cleanly.
5. Retries and duplicates: if a request is retried, does the customer get charged twice?
6. Security: the call is authenticated, and no secrets or personal data leak in the messages or logs.

Teams with many services also use contract testing (with tools such as Pact): each service writes down what it expects from the others, and both sides test against that agreement, so a change on one side can't silently break the other.

Integration testing in practice

• Integration tests usually run in the CI pipeline, after the unit tests pass.
• Tools like Testcontainers start a real database or message queue in a container just for the tests, then throw it away.
• API tests (with tools like Postman or pytest with requests) are a natural way to test integration points directly.
• Keep them fewer than unit tests and more than end-to-end tests, as the testing pyramid suggests.

What's next

In Part 13 we move up a level to system testing: testing the whole application, end to end, the way a real user would.

ProvaLab runs integration and API tests on every change, records each result next to its test case, and sends failures back to your coding agent with the request, the response and the logs. See it in the two-minute product tour.

What is manual testing?

In manual testing, a person uses the software and checks the result themselves. They click, type, look and decide whether what they see is right.

There are two main styles:

• Scripted manual testing: the tester follows written test cases step by step. "Open the cart. Apply code SAVE10. Check the total drops by 10%."
• Exploratory testing: the tester explores freely, using experience and curiosity. "What happens if I apply two discount codes? Or change the quantity to minus one?"

Manual testing is quick to start. You need no code, just the app and a sharp eye.

What is automated testing?

In automated testing, code performs the checks. A test script opens the app, does the steps, and compares the result with what was expected, without a person watching.

def test_discount_code_reduces_total(cart):
    cart.add("Margherita pizza", price=200)
    cart.apply_code("SAVE10")
    assert cart.total == 180

Once written, that check runs in a second, as often as you like: on every code change, every night, on every browser. That repeatability is the whole point.

Where each one shines

The key idea: automated tests mostly guard what you already know works. They are excellent at catching regressions, things that used to work and broke. Most new kinds of bugs are found by people, exploring in ways nobody wrote a script for.

Side by side

What should you automate?

Ask three questions about any test:

Good candidates for automation:

• Regression tests: everything that must keep working after each change.
• Smoke tests: "does the app start, can people sign in?" before every release.
• Critical journeys: sign-up, sign-in, checkout and payment.
• Cross-browser and device checks: the same flow on Chrome, Safari, Android and iPhone.
• Data-heavy tests: the same form with a hundred different inputs.
• API tests: fast, stable and easy to check.

Better kept manual:

• Brand-new features still changing every week.
• Usability and design: "is this confusing?" needs a person.
• Exploratory testing: looking for the bugs nobody imagined.
• One-off checks you will never repeat.

Automation isn't free

Two hidden costs catch many teams out:

1. Maintenance. When the app changes, tests that depend on it must change too. A button renamed from "Buy" to "Checkout" can break dozens of tests at once.
2. Flaky tests. A flaky test sometimes passes and sometimes fails without any code change, often because of timing or test data. Flaky tests are dangerous: once people stop trusting red results, they start ignoring real failures too.

Good automation is written like good code: readable, small and reliable. A few trusted tests beat hundreds that nobody believes.

Three myths about automation

1. "Automation will replace manual testers." It replaces repetitive checking, not thinking. Exploring, questioning and judging still need people.
2. "We should automate 100% of our tests." Some tests cost more to automate and maintain than they will ever save. Automate where it pays back.
3. "Once written, automated tests run forever." They need the same care as the product code they test.

Common automation tools

Automated tests usually run in a CI pipeline (continuous integration): every time someone changes the code, the tests run automatically and report the result before the change is merged.

Where AI fits in

AI coding agents can now write automated tests from a plain description, run them and read the failures. That makes automation much cheaper to start. It doesn't change the rules above: you still decide what is worth automating, and a person still explores and judges.

What's next

In Part 4 we put it all in order with the Software Testing Life Cycle (STLC): the six phases of testing, from the first requirement to the release.

ProvaLab runs your automated tests on real browsers and devices, keeps every result next to its test case, and sends failures back to your coding agent with a video and logs. See it in the two-minute product tour.

What is MCP?

The Model Context Protocol (MCP) is an open standard, introduced by Anthropic in November 2024, for connecting AI apps to the tools and data they need.

Before MCP, every AI app needed its own custom integration for every tool. MCP replaces that with one standard plug. A tool is wrapped once as an MCP server, and any app that speaks MCP, an MCP client, can use it. People often compare it to USB-C: one shape of connector for many devices.

An MCP server can offer three kinds of things:

• Tools: actions the model can take, such as "run this test suite" or "create an issue".
• Resources: data the model can read, such as a file or a test report.
• Prompts: ready-made instructions a user can pick, such as "summarise last night's run".

The important part: the model decides which tool to call and when, based on what you asked. You don't click buttons in another dashboard; you ask in plain language, and the agent does the work through MCP.

What is a skill?

MCP gives an agent access to a tool. It does not teach the agent how to use it well. That is what a skill is for.

A skill is a folder with a SKILL.md file: a short name, a one-line description of when to use it, and step-by-step instructions. The agent only sees the name and description until your request matches. Then it loads the full instructions. So you can have many skills installed without filling up the agent's memory.

For example, ProvaLab's triage-failures skill tells the agent: read the actual error text before describing a failure, group failures by cause rather than by test, and flag skipped tests. Without it, an agent with the same MCP access might just list twenty red tests that all come from one broken selector.

What is a plugin?

A plugin is the package that ships it all in one install. A Claude Code plugin can bundle:

• Skills: the know-how.
• Slash commands: shortcuts like /provalab-test.
• MCP servers: the connection, configured in a .mcp.json file.
• Optionally, sub-agents and hooks.

In one line: MCP is the access, skills are the know-how, and the plugin is the box they come in.

"MCP 1.0" vs "MCP 2.0": what changed?

Officially, MCP versions are named by date, not by number. The original specification is 2024-11-05, and it was revised in 2025-03-26, 2025-06-18 and 2025-11-25. The 2026-07-28 revision changed the protocol's core design so much that many people call it the next generation of MCP, or "MCP 2.0". Here, "MCP 1.0" means the original design that ran through 2025-11-25.

The headline change: MCP became stateless. In the original design, a client opened a session with an initialize handshake, got a session id, and the server had to remember every connected client. In the 2026-07-28 revision there is no handshake and no session id. Every request carries its own protocol version and capabilities, so any server instance can answer any request. That makes MCP servers much easier to run at scale, behind load balancers and gateways.

Deprecated features keep working for at least twelve months under MCP's new deprecation policy, so existing servers don't break overnight. Teams that relied on session ids will have some migration work. The details are in the official changelog and the release announcement.

What this means for you as a user: very little changes day to day. You still install a server or plugin and ask in plain language. The difference shows up as faster, more reliable connections, because the servers behind them are simpler to run.

How to use ProvaLab through MCP

ProvaLab's features are exposed as tools on one MCP server at https://mcp.provalab.io/mcp. Running tests, reading results, triaging failures, load tests and security scans can all be driven by your coding agent.

Option 1: the Claude Code plugin (recommended)

The plugin gives you the MCP connection plus the skills and slash commands.

/plugin marketplace add provalab-dev/claude-code-plugin
/plugin install provalab@provalab-marketplace

Then sign in once. The sign-in works on any machine, including SSH sessions and CI, because you approve it in a browser on any device:

curl -fsSL https://plugins.provalab.io/install.sh | sh
provalab-auth login

Restart Claude Code, then check the connection with claude mcp list. You should see the ProvaLab server marked as connected.

Option 2: the claude.ai connector

If you work in claude.ai rather than Claude Code, add the ProvaLab connector instead. It points at the same server and gives you the same tools. Use either the connector or the plugin, not both.

What you can ask

Once connected, just ask. The skills pick up your intent:

• "Run the login tests for the checkout project": runs the suite on ProvaLab's cloud browsers and records the run in Test Management.
• "Generate tests for https://staging.example.com/signup and run them": writes the tests from the live page, then runs them.
• "Why did last night's run fail?": reads the failures, groups them by cause and explains them.
• "Load test our API up to 200 users": writes the load test from your repo, shows the cost, and runs it when you say go.

Or use the slash commands: /provalab-test, /provalab-status, /provalab-results and /provalab-load.

A few tips

• Set your project. If your organization has more than one project, set PROVALAB_PROJECT_ID (and PROVALAB_ORG_ID if you belong to several organizations) so results land in the right place.
• Let the agent fix, but you approve. The real gain comes when failures go straight back to the agent that wrote the code. Keep a person on the final approval.
• Allow the tools by name. Plugin tools are named mcp__plugin_provalab_api__<tool>. Use that pattern in your permission rules if you want to skip repeated approval prompts.

The short version

• MCP is the standard plug that lets AI agents use tools.
• Skills teach the agent how to use those tools well.
• Plugins package both into one install.
• The 2026-07-28 revision ("MCP 2.0") made MCP stateless and easier to scale, without changing how you use it.
• With the ProvaLab plugin, your coding agent can run tests, read results, triage failures and run load tests, all from a plain-language request.

Want to see it working? Watch the two-minute product tour or book a 30-minute demo.

The app worked. It just didn't work fast enough, for enough people at once. Finding that out before your customers do is the job of performance testing.

What is performance testing?

Performance testing checks how fast, stable and scalable a system is under different amounts of use. It is a non-functional type of testing: it doesn't ask "does checkout work?" but "does checkout still work, quickly, when 2,000 people use it at the same time?"

It answers questions like:

• How fast does the app respond with a normal number of users?
• How many users can it handle before it slows down or fails?
• What happens during a sudden rush?
• Does it stay healthy after running for hours or days?
• If we add servers, can we serve more users?

The six types of performance testing

Each type answers a different question, and each has its own shape of load over time:

1. Load testing

Can the system handle the load we expect?

Ramp up to the expected number of users, for example the Friday dinner peak, hold it for a while, and check that response times and error rates stay within your targets. This is the most common performance test, and the one to run before every major release.

Example: 2,000 users browsing menus and placing orders for 15 minutes; 95% of pages must load in under 1 second, with less than 1% errors.

2. Stress testing

Where does the system break, and how does it break?

Keep increasing the load beyond the expected peak until something fails. The goal isn't to pass; it's to learn the breaking point and the failure mode. Does the app slow down gracefully and show a friendly "busy, please wait" message, or does it crash and lose orders? Does it recover by itself when the load drops?

3. Spike testing

Can the system survive a sudden rush?

Jump from a small load to a very large one in seconds, then back down. Real spikes happen: a push notification about a 50% discount, a TV advert, a cricket match ending. Auto-scaling systems in particular need this test, because adding servers takes time and the spike arrives first.

4. Soak testing (endurance testing)

Does the system stay healthy over a long time?

Run a moderate, realistic load for many hours, or even days. Some problems only appear slowly: memory that is never freed (a memory leak), database connections that aren't returned, log files that fill the disk, or caches that grow forever. A one-hour test will never find these.

5. Volume testing

Does the system stay fast with a lot of data?

Keep the number of users normal, but fill the system with a large amount of data: millions of past orders, a customer with 5,000 orders in their history, a restaurant with a huge menu. Queries that are instant on a small test database can be very slow on a full-sized one.

6. Scalability testing

Does adding resources let us serve more users?

Increase the load in steps while adding servers or capacity, and check that the system's capacity grows with them. If doubling the servers doesn't come close to doubling the capacity, something, often the database, is a bottleneck that more servers can't fix.

The six types at a glance

You may also hear capacity testing (finding the maximum number of users that still meets your targets), which sits between load and stress testing.

The metrics that matter

Why averages lie: use percentiles

The average response time hides your slowest users:

That's why performance targets use percentiles:

• p50 (median): half of requests are faster than this.
• p95: 95% of requests are faster than this; 1 in 20 users waits longer.
• p99: 99% are faster; 1 in 100 waits longer.

When thousands of people use your app, "1 in 20" is a lot of unhappy customers. A good target sounds like: "p95 under 1 second, error rate under 1%, at 2,000 users."

How to run a performance test, step by step

1. Set clear goals. Which pages and APIs, how many users, and what response time and error rate count as a pass.
2. Model realistic behaviour. Real users don't all hit "Pay" at the same millisecond. Mix the journeys (browse, search, order) in realistic proportions, and add think time: the pause while a user reads a menu.
3. Use a production-like environment. Same server sizes, same configuration and a realistic amount of data. Results from a tiny test server don't predict production.
4. Prepare test data. Enough test accounts, restaurants and codes that thousands of virtual users don't all fight over the same row in the database.
5. Write the test script. Here is a load test written for k6, a popular open-source tool:

import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  stages: [
    { duration: '2m', target: 200 },  // ramp up to 200 users
    { duration: '5m', target: 200 },  // hold
    { duration: '1m', target: 0 },    // ramp down
  ],
  thresholds: {
    http_req_duration: ['p(95)<1000'], // 95% of requests under 1 second
    http_req_failed: ['rate<0.01'],     // less than 1% errors
  },
};

export default function () {
  const res = http.get('https://staging.example.com/api/restaurants?q=pizza');
  check(res, { 'status is 200': (r) => r.status === 200 });
  sleep(1); // think time
}

6. Run it, and watch the servers too. Response times tell you that it's slow; CPU, memory and database graphs tell you why.
7. Find the bottleneck, fix it, and run it again. Performance work is a loop.

Only ever test systems you own or have permission to test. Sending heavy load to someone else's website or API can look like an attack, and can take their service down.

Common bottlenecks

• Slow database queries, often a missing index.
• The "N+1" problem: loading a list of orders, then making a separate database call for each one.
• Too few database connections in the connection pool.
• No caching of data that rarely changes, like menus.
• Slow or rate-limited outside services, such as a payment provider or a maps API.
• Large pages and images that take long to download on mobile networks.

Common mistakes

• Only looking at averages instead of p95 and p99.
• Testing from an overloaded load generator. If the machine sending the traffic is at 100% CPU, you're measuring it, not your app.
• Unrealistic scenarios: no think time, every user doing the same thing, or a test database with ten rows.
• Testing once, just before launch. Run load tests regularly, so you see when a change makes things slower.
• Ignoring errors because response times looked good. Fast errors are still errors.

Popular tools

What's next

In Part 17 we look at test runs and test results: what happens each time a set of tests is run, and how to read the results.

ProvaLab's load testing runs k6 tests from the cloud, shows response time percentiles, throughput and errors live, and can find your app's breaking point step by step. Your coding agent can write the test from your code and explain the slow spots afterwards. See it in the two-minute product tour.

The short version:

• QA improves the process to prevent defects. QC checks the product to find them. Testing is the main way QC finds them.
• Verification asks "are we building the product right?" Validation asks "are we building the right product?"

Now the longer version, with examples.

QA vs QC vs testing

Quality assurance (QA)

Quality assurance is about the process. Its goal is to prevent defects by making sure the team works in a way that produces good software in the first place.

QA activities in our food delivery app team might include:

• Agreeing coding standards and a definition of done ("a story is done when it has tests, a code review and updated docs").
• Using templates for test plans and bug reports.
• Making code review mandatory for every change.
• Learning from defects: "we had five payment bugs this quarter; let's add a payment checklist to code reviews."
• Training the team, and choosing good tools.

QA is proactive, runs through the whole project, and is everyone's responsibility.

Quality control (QC)

Quality control is about the product. Its goal is to find defects in what has been built, before it reaches users.

QC activities include:

• Reviewing finished work: requirements, designs and code.
• Testing the software: everything from unit tests to system tests.
• Checking releases against exit criteria before they ship.

QC is reactive: it looks at a product that already exists and asks "is this right?"

Testing

Testing is one part of quality control: running and examining the software to find defects and to provide information about its quality. Everything in this series so far, from test cases to regression runs, is testing.

The differences at a glance

A simple analogy from a restaurant kitchen: QA is the recipe, the hygiene rules and the chef's training. QC is tasting each dish before it leaves the kitchen.

"But my job title is QA engineer..."

In everyday industry use, "QA" often just means "testing": a QA engineer or QA team usually spends most of their time testing. That's fine in practice. But in interviews and in quality-management standards (such as ISO 9001), QA and QC have the distinct meanings above, so it's worth knowing the difference.

Verification vs validation

The classic definitions, from software engineer Barry Boehm:

• Verification: "Are we building the product right?" Does the product meet its specification: the requirements, designs and standards written for it?
• Validation: "Are we building the right product?" Does the product meet the real needs of its users and the business?

Verification

Verification checks each piece of work against what it was supposed to be. Much of it happens without running the software:

• Reviewing requirements for gaps and contradictions.
• Reviewing designs against requirements.
• Code review and static analysis tools, which check code without running it.
• Testing against the specification: does the discount function return exactly what the requirement says?

Example: the requirement says "expired codes must be refused with a clear message". A tester checks that the app does exactly that. That's verification.

Validation

Validation checks the product against what users actually need. It usually happens by running the software with real or realistic use:

• Acceptance testing with the product owner or customers.
• Usability testing with real users.
• Beta releases to a small group of real customers.
• Demos to stakeholders at the end of each sprint.

Example: the new restaurant dashboard matches its specification perfectly, which is verified. But restaurant owners find it too slow to use during the dinner rush. It fails validation. (This is principle 7: bug-free can still be wrong.)

The differences at a glance

You need both. Verification without validation can build a perfect version of the wrong product. Validation without verification finds the right product, full of bugs.

The V-model: where they meet

The V-model shows how each step of building software has a matching level of testing:

• Each step down the left is verified against the one before it, through reviews.
• Each test level up the right checks the matching step on the left: unit tests check the detailed design, integration tests the architecture, system tests the specification.
• Acceptance testing, at the top, validates the product against the business's real needs.

The V-model also explains why testing should start early: the acceptance tests can be designed as soon as the business requirements are written, long before any code exists.

Quick interview answers

• QA vs QC? QA prevents defects by improving the process; QC finds defects by checking the product. Testing is the main part of QC.
• Verification vs validation? Verification: are we building the product right (against the spec)? Validation: are we building the right product (for the users)?
• Is a code review QA or QC? Reviewing a specific piece of code is QC (and verification). Requiring code reviews for every change, and improving how they're done, is QA.
• Can something pass verification but fail validation? Yes: a product that exactly matches a specification that missed what users need.

What's next

In Part 21 we look at static testing: reviews, walkthroughs, inspections and static analysis, the cheapest way to find bugs, because it happens before the software even runs.

ProvaLab supports both sides: Test Management links requirements to test cases and results for verification, the PR Bot reviews every code change before it's merged, and acceptance and exploratory results live alongside automated ones, so you can see the whole picture of quality in one place. See it in the two-minute product tour.

1. Is the bug really fixed?
2. Did the fix break anything else?

The first is answered by re-testing. The second is answered by regression testing. Every fix needs both, and mixing them up is one of the most common ways bugs reach customers.

What is re-testing?

Re-testing (also called confirmation testing) means running the exact test that failed again, after the bug has been fixed, to confirm the fix works.

Why do we re-test?

Because a fix is only a claim until someone checks it. In real teams, "fixed" turns out not to be fixed surprisingly often:

• The fix is partial. It works for the case the developer tried, but not for the one in the bug report.
• It works in one place but not another. Remember BUG-231: the expired-code fix worked on the web, but the Android app had its own copy of the check, so the bug was still there.
• The fix never arrived. It's on the developer's machine or in a branch, but not in the build being tested.
• The developer misunderstood the bug and fixed something else.
• The environment hid the bug, so it passed on the developer's laptop but fails with real test data.

How to re-test

• Use the same steps, the same data and the same kind of environment as the original bug report.
• Test on every platform the bug affected (web, Android, iPhone).
• Try one or two small variations around the bug: another expired code, a code that expired one minute ago.
• Then update the bug: Verified if it passed, Reopened with details if not (see the bug life cycle).

The person who found the bug is usually the best one to re-test it.

What is regression testing?

Regression testing means re-running tests that used to pass, after a change, to make sure nothing that worked before is now broken.

A regression is something that worked yesterday and doesn't work today. Regressions happen because software is connected: code is shared, data flows between features, and a change in one place can ripple into others.

Why do we do regression testing?

Changes break things in places nobody expected:

• Shared code: the fix for BUG-231 moved the discount check to the server. Every app now uses it, including flows nobody touched.
• Side effects: a faster search query returns results in a different order, and the "recommended" section breaks.
• Merges: two developers' correct changes conflict when combined.
• Upgrades: a new version of a library, the database or the browser changes behaviour.
• Configuration: a changed setting in one environment affects features far away.

Developers can't always predict these ripples. Regression tests catch them.

Re-testing vs regression testing for one fix

For the BUG-231 fix:

• Re-test: TC-104 (expired code refused), on web and Android.
• Targeted regression: valid codes still work, cart and checkout totals, free-delivery rule, order history totals, the Android checkout. These all touch the discount code that changed.
• Wider regression: the full suite runs that night, and again before the release.

The differences at a glance

Re-testing usually comes first: there's no point running a big regression suite on a fix that doesn't work. (A quick sanity check often comes before both.)

How to choose which regression tests to run

Running every test after every change is the safest choice, but rarely practical. There are three common strategies:

1. Retest all: run the whole regression suite. Most thorough, slowest. Good before a release.
2. Regression test selection: run only the tests related to what changed. A traceability matrix and good test tagging (by feature, by area of code) make this possible.
3. Prioritisation: run tests in order of risk, with critical journeys first (risk-based testing), so the most important results come back first.

Most teams combine them with a schedule like this:

Building and maintaining a regression suite

A regression suite is a living thing. Some habits that keep it useful:

• Add a test for every bug you fix. If a bug happened once, it can come back. The test from the BUG-231 fix joins the suite permanently.
• Automate it. Regression tests are run again and again, which is exactly when automation pays off (see Part 3).
• Tag tests by feature, priority and speed, so you can run "all checkout tests" or "critical tests only".
• Remove or update obsolete tests. A test for a feature that no longer exists only slows everyone down.
• Fix flaky tests quickly. If people learn to ignore red results, real regressions slip through.
• Watch the run time. If the suite gets too slow to run often, split it into faster and slower layers.

Common mistakes

• Re-testing only on one platform when the bug affected several.
• Skipping regression for "small" fixes. Small changes to shared code cause big regressions.
• Running the full suite on a fix that hasn't been re-tested, then wasting the run when the fix turns out not to work.
• Never adding tests for fixed bugs, so the same bug returns months later.
• Treating regression testing as manual work forever. It grows with every release; without automation it becomes impossible to keep up.

What's next

In Part 16 we look at performance testing and its types: load, stress, spike, soak, volume and scalability testing.

ProvaLab runs your regression suite on every change and every night, on real browsers and devices, and shows each test's history: when it last passed, and the build where it started failing. When a regression appears, your coding agent gets the failure with its logs and recording, and can fix it. See it in the two-minute product tour.

• Is every requirement tested?
• Which requirements are failing right now?
• This requirement just changed: which tests do we re-run?

The tool that answers all three is the requirements traceability matrix, usually just called the RTM or traceability matrix. In the final part of this series, we build one.

What is a traceability matrix?

A traceability matrix is a table that links each requirement to the test cases that check it, and to their latest results and bugs.

"Traceability" means you can trace a line from any requirement to its tests, and from any test or bug back to its requirement. The matrix makes those lines visible.

An example

Here are the discount-code requirements from our food delivery app, matched against the test cases from Part 6:

The same information as a table, the form most teams keep it in:

At a glance, the matrix shows three problems that would otherwise stay hidden:

1. A coverage gap: REQ-14 has no tests at all. Remember the Android bug from Part 10? Without the matrix, nobody would notice that Android was never tested.
2. A failing requirement: REQ-12 is tested, but its test fails. The release isn't ready until BUG-231 is fixed.
3. An orphan test: TC-106 ("removing all items removes the discount") isn't linked to any requirement. Either the behaviour matters and a requirement is missing, or the test is out of date.

Forward, backward and bidirectional traceability

• Forward traceability goes from requirements to tests: is every requirement covered, and is it passing? This finds gaps.
• Backward traceability goes from tests and bugs back to requirements: why does this test exist, and which requirement does this bug affect? This finds orphans and shows the business impact of each bug.
• Bidirectional traceability is both at once, and it is what a good RTM gives you.

Measuring coverage

The matrix turns "are we done?" into numbers:

• Requirement coverage = requirements with at least one test ÷ all requirements. In our example: 4 of 5 = 80%. REQ-14 is the missing 20%.
• Passing requirements = requirements whose tests all pass ÷ all requirements. In our example: 3 of 5 = 60%. REQ-12 is failing and REQ-14 is untested.

Weight these by priority. Three of the five requirements are high priority, and two of those three have a problem: REQ-12 is failing and REQ-14 is untested. That tells you far more than the percentage does.

Requirement coverage is not the same as code coverage from Part 7. Code coverage asks "which lines of code ran?" Requirement coverage asks "which promises to the user did we check?" You need both.

Impact analysis: when a requirement changes

Requirements change all the time. Say the business decides expired codes should show a "renew your code" link instead of just an error. The RTM tells you instantly:

• Which tests to update: TC-104.
• Which bugs are affected: BUG-231 (the fix must include the new link).
• What to re-run before release: everything linked to REQ-12.

Without the matrix, someone has to search through hundreds of test cases and hope they found them all.

How to build an RTM, step by step

1. List the requirements (or user stories), each with a unique ID such as REQ-12.
2. Write test cases, each with its own ID (TC-104), as in Part 6.
3. Link each test case to the requirements it covers. Do this when you write the test, not weeks later.
4. Add the latest result of each test after every run.
5. Link bugs to the failing test and the requirement.
6. Review the gaps before each release: requirements with no tests, failing high-priority requirements, and orphan tests.

Where the RTM fits in the life cycle

The RTM isn't a one-time document. It grows through the whole testing life cycle:

In agile teams, the "requirements" are usually user stories and their acceptance criteria, and the matrix links tests to stories.

Common mistakes

• A spreadsheet nobody updates. A stale RTM is worse than none, because it gives false confidence. Link tests and results where they live, so the matrix updates itself.
• Counting tests instead of requirements. Two hundred tests on one feature and none on another is not good coverage.
• Linking after the fact. Link each test to its requirement the moment you write it.
• Ignoring orphan tests. They often point to missing requirements, or to wasted effort on tests nobody needs.
• Only checking coverage, never results. "Tested" is not the same as "passing".

Testing Basics: the whole series

That completes our Testing Basics series. You now know:

1. What software testing is, and why every team needs it
2. The types and levels of testing, and the testing pyramid
3. Manual vs automated testing, and what to automate
4. The software testing life cycle (STLC)
5. How to write your first automated test with pytest
6. How to write good test cases
7. Black-box, white-box and grey-box testing
8. How to write a test plan
9. How to write a good bug report
10. The bug life cycle
11. The requirements traceability matrix (this post)

Next we go deeper into specific kinds of testing, starting with Part 12: integration testing, then system testing, smoke and sanity testing, regression testing and performance testing.

ProvaLab Test Management keeps this matrix up to date for you: requirements and features link to test cases, every manual or automated run records its results, and issues link back to the test and requirement they affect, so coverage reports are always current. See it in the two-minute product tour.

Users mistype. Attackers don't. They send exactly the input that breaks a check, ask for exactly the record they're not allowed to see, and try the default password first. Security testing means doing that before they do.

What is security testing?

Security testing checks that the software protects its data and its users against misuse: that people can only see and do what they're allowed to, that data is protected in storage and in transit, and that the system fails safely when something unexpected happens.

It's a non-functional kind of testing, and the one with the highest cost of failure. A functional bug annoys a customer. A security bug can expose every customer at once.

The OWASP Top 10

OWASP (the Open Worldwide Application Security Project) is a non-profit that publishes free security guidance. Its best-known document is the OWASP Top 10: the ten most serious risks to web applications, updated every few years from real-world data. The current edition is 2025. If you learn one thing about security testing, learn these ten.

Here is each one, with an example from our food delivery app.

A01 · Broken Access Control

Users doing things they're not allowed to. Alice reads Bob's order by changing 42 to 43 in the URL. A customer reaches the restaurant dashboard. A regular user calls an admin-only endpoint.

This has been number one for years, because it's common, easy to get wrong and devastating. The 2025 edition also folds in server-side request forgery (SSRF), where an attacker tricks your server into fetching internal resources on their behalf.

You've already tested for it: test_customer_cannot_read_someone_elses_order in Part 4 is an access-control test.

A02 · Security Misconfiguration

The software is fine; the settings aren't. Debug mode left on in production, so errors show internal details. Default admin passwords never changed. Directory listings enabled. Cloud storage buckets open to the world. Missing security headers.

It climbed from fifth to second in 2025, because modern systems have so many settings to get wrong.

A03 · Software Supply Chain Failures

A vulnerability in something you installed becomes your vulnerability. Your app may be perfect, but it runs on hundreds of libraries, a build tool, a container image and some CI plugins. Any of them can carry a known flaw, or be tampered with. New in 2025, replacing and widening the old "vulnerable and outdated components".

A04 · Cryptographic Failures

Data that should be protected isn't. Passwords stored in plain text (or with weak hashing). Card numbers kept when they didn't need to be. Traffic sent without TLS. Home-made encryption. The fix is mostly "use the standard, well-tested thing and don't invent your own".

A05 · Injection

Untrusted input gets run as code. The classic is SQL injection: a search box that turns ' OR 1=1 -- into a database query that returns everything. Also OS command injection, and cross-site scripting (XSS), where a comment containing <script> runs in every other user's browser.

Injection is almost always prevented the same way: never build queries or commands by gluing strings together; use parameterised queries and output encoding.

A06 · Insecure Design

A flaw in the plan, not the code. The discount feature works exactly as specified, but the specification never limited how many codes one order can use, so a customer stacks ten. Password reset by "mother's maiden name". No limit on how many OTPs can be requested. No amount of bug-free code fixes a design that's wrong. This is what threat modelling during design is for.

A07 · Authentication Failures

Proving who you are, done badly. Weak passwords accepted. No rate limit on sign-in, so an attacker tries a million passwords. Session tokens that never expire, or that stay valid after sign-out. "Remember me" implemented with the password in a cookie.

Already tested: test_no_token_means_401 in Part 4 is the simplest authentication test.

A08 · Software or Data Integrity Failures

Trusting code or data without checking it. Auto-updates with no signature check. Deserialising objects from an untrusted source. A CI pipeline anyone can push to. Closely related to supply chain, but about verification: did this come from where it claims, unchanged?

A09 · Security Logging & Alerting Failures

Attacks nobody notices. Failed sign-ins aren't logged. Logs exist but nobody looks, and nothing alerts. A breach is discovered months later, from the outside. Renamed in 2025 to stress that logging without alerting isn't enough.

A10 · Mishandling of Exceptional Conditions

Errors that fail the wrong way. An exception in the permission check gets caught and the request continues as if allowed ("fail open"). A crash shows users a stack trace with file paths and library versions. An out-of-memory condition leaves half an order saved. New in 2025.

Already tested: every "never a 500" assertion in Part 4 is a small check of this. An API that answers 422 to bad input, instead of crashing, is handling the exceptional condition.

The six kinds of security testing

Different risks need different kinds of testing. Three run on the code and three on the running system:

The first three are static testing applied to security: cheap, fast, on every change. DAST and fuzzing are dynamic and mostly black-box or grey-box. A penetration test adds the one thing none of the tools have: creativity.

Which kind finds which risk

Notice how many rows say "API tests you write". Ordinary functional and negative tests, written with security in mind, are the first and cheapest layer of security testing. The 403 test, the 401 test, the "never a 500" assertion: you don't need a security tool to write those.

Three checks you can do this week

1. Add one access-control test per resource. For every kind of record (orders, profiles, addresses), sign in as one user and request another user's record. Expect 403 or 404, never 200. This single habit covers the number-one risk.
2. Run a dependency scan. pip-audit for Python, npm audit for JavaScript, or Trivy for containers. It takes a minute and often finds something.
3. Do a 30-minute threat model. Draw the app's data flows on a whiteboard (what goes where, who can reach it) and ask, for each arrow: what's the worst thing someone could do here? This is the design-stage activity that catches A06, and no tool replaces it.

Rules of the road

• Only test what you own or have written permission to test. Running a scanner or a fuzzer against someone else's site is illegal in most countries, and it looks exactly like an attack because it is one.
• Test against a test environment, with test data. A scanner will try to submit forms, create records and trip rate limits.
• Fix in order of severity, not in order found. A scanner's 200 findings usually contain 3 that matter.
• Verify, then report. Scanners produce false positives. Confirm a finding by hand before raising it, and write it up as a bug report with steps, not as "the scanner said so".

What's next

In Part 6 we run a real DAST scan with OWASP ZAP against the little API from Part 4, read the findings, and sort the real ones from the noise.

ProvaLab AppSec runs web app (DAST), API fuzzing, dependency, secrets, container and code scans against targets you've verified you own, and turns the findings into a plain-language fix list ordered by severity, with the false positives you've checked marked as such. See it in the two-minute product tour.

Sources: the OWASP Top 10:2025 introduction.

They come from the ISTQB Certified Tester Foundation Level syllabus, the most widely used testing certification, and they're a favourite interview topic. More importantly, they change how you test.

1. Testing shows the presence of defects, not their absence

Testing can show that bugs are there. It can never prove that there are none.

If 500 tests pass, you know the software works in those 500 situations. You don't know about the 501st. Every passing test reduces the risk of an unknown bug, but never removes it.

Example: the discount-code tests all pass. A week after release, a customer finds that applying a code, changing the delivery address and going back to the cart doubles the discount. No test tried that exact sequence.

What it means for you: never promise "bug-free". Report what was tested, what wasn't, and the risk that remains, as in a good test summary report.

2. Exhaustive testing is impossible

Testing every possible input and combination is impossible, except in trivially small cases.

A search box that accepts 10 characters of letters and digits has 36¹⁰ possible inputs: over 3.6 quadrillion. At a thousand tests per second, trying them all would take more than 100,000 years. And that's one field.

Example: the food app has thousands of restaurants, many browsers and phones, two languages, dozens of payment options and endless orders of steps. Testing every combination isn't hard; it's impossible.

What it means for you: choose tests wisely, using risk and test design techniques like equivalence partitioning, boundary values and pairwise testing. Good choices find most bugs with a small fraction of the tests.

3. Early testing saves time and money

The earlier a defect is found, the cheaper it is to fix.

A mistake caught while reading the requirements is a five-minute conversation. The same mistake found by customers in production means support tickets, an urgent fix, a new release, and lost trust.

Example: while reviewing the requirement "codes give 10% off", a tester asks, "before or after the delivery fee?" Nobody had decided. Settling it then costs nothing; finding it after release means refunds.

What it means for you: test from day one. Review requirements, write test cases before the code, and run tests on every change. This is the "shift left" idea from Part 1, and the reason the STLC runs alongside development.

4. Defects cluster together

A small number of areas usually contain most of the defects. This is often linked to the 80/20 (Pareto) rule, though the exact split varies.

Bugs cluster where code is complex, changes often, was written in a hurry, or connects to outside systems.

What it means for you: use your bug history. Where bugs were found before, more are likely hiding. Test those areas more deeply, and ask why they're fragile: often the real fix is simpler code or better tests, not just one more bug report.

5. Tests wear out (the pesticide paradox)

If you run the same tests over and over, they eventually stop finding new bugs.

The older name, the pesticide paradox, compares this to insects becoming resistant to a pesticide used again and again. Once the bugs your tests can find are fixed, the same tests just keep passing, while new bugs appear in places they don't look.

Example: the checkout regression suite has passed every night for six months. Meanwhile a new "pay later" option was added, and no test touches it.

What it means for you: keep regression tests (they still protect against old bugs returning), but review and extend them regularly. Add tests for new features and new risks, vary the test data, and mix in exploratory testing to find what scripts miss.

6. Testing is context dependent

There is no single right way to test. The right approach depends on what you're testing and why.

A banking app, a mobile game, a hospital system and a startup's first prototype need very different testing: different risks, rules, depth and speed.

Example: for the food app, a bug that charges the wrong amount is critical and gets deep testing; a slightly misaligned icon on the help page is not. A medical device would need formal documentation and strict regulation that a food app doesn't.

What it means for you: don't copy another team's process blindly. Choose your test plan, level of documentation, amount of automation and coverage targets based on your product's risks.

7. The absence-of-defects fallacy

Finding and fixing lots of bugs doesn't guarantee success. Software with no known bugs can still be the wrong software.

If the system is hard to use, too slow, or simply doesn't do what users need, a clean bug list doesn't help.

Example: the new restaurant dashboard passes every test, with zero open bugs. But restaurants refuse to use it, because it takes six clicks to accept an order during the dinner rush.

What it means for you: test against users' real needs, not only the written requirements. That's what acceptance testing and usability testing are for, along with talking to real users early.

The seven principles at a glance

How the principles connect to this series

Almost everything in earlier parts follows from these principles:

• Test design techniques (Parts 6 and 7) exist because of principle 2.
• The STLC and test planning (Parts 4 and 8) put principle 3 into practice.
• Risk-based testing (Part 8) uses principles 4 and 6.
• Regression maintenance (Part 15) answers principle 5.
• Acceptance testing (Part 2) protects against principle 7.
• Coverage (Part 18) is limited by principles 1 and 2.

What's next

In Part 20 we clear up three terms that are often mixed up: QA, QC and testing, and verification vs validation.

ProvaLab helps put the principles into practice: tests run on every change (principle 3), each test case keeps a history so you can see where failures cluster (principle 4), and coverage reports show which requirements have gone untested (principles 1 and 2). See it in the two-minute product tour.

Two quick checks exist to stop that from happening: smoke testing and sanity testing. They are among the most confused terms in testing, and a favourite interview question. This post clears them up.

What is smoke testing?

A smoke test is a quick, shallow check of the most important features of a new build, to decide whether it is stable enough to test further.

It answers one question: "Is this build worth testing at all?"

The name comes from hardware: switch a new device on, and if smoke comes out, there's no point testing anything else. Smoke testing is also called build verification testing (BVT) or confidence testing.

Key traits:

• Runs on every new build, before any deeper testing.
• Covers the main features, lightly: wide, not deep.
• Takes minutes, not hours.
• Is usually scripted and automated.
• Fail = reject the build. Testing doesn't start until it's fixed.

Example smoke test checklist

For our food delivery app, a smoke test might be just ten checks:

1. The app opens and the home page loads.
2. A test customer can sign in.
3. Searching for "pizza" returns restaurants.
4. A restaurant menu opens.
5. An item can be added to the cart.
6. Checkout opens and shows the right total.
7. A test payment in the sandbox succeeds.
8. The order appears in "My orders".
9. The restaurant dashboard receives the order.
10. The customer can sign out.

Notice that none of these go deep. There's no discount-code edge case, no closed-restaurant scenario. That's the job of the full test cycle that comes after the smoke test passes.

What is sanity testing?

A sanity test is a quick, focused check of one area after a small change or bug fix, to confirm the change makes sense before investing in further testing.

It answers a different question: "Did this specific change work, and does this area still behave sensibly?"

Key traits:

• Runs after a small change: a bug fix, a minor feature, a configuration change.
• Covers one area, a little deeper: narrow, not wide.
• Takes minutes.
• Is often unscripted: a tester who knows the area checks it quickly.
• Fail = back to the developer before wasting time on full re-testing and regression.

Example sanity test

The developer has just fixed BUG-231 from Part 9: expired discount codes were being accepted. A quick sanity check of the discount area:

• An expired code (OLD10) is now refused with the right message.
• A valid code (SAVE10) still works.
• Removing a code restores the full total.
• The checkout page still loads normally.

If that looks sensible, the tester moves on to the formal re-test of the bug and the regression run (both covered in Part 15). If the valid code is now broken too, the build goes straight back. No point running 200 regression tests on it.

Smoke vs sanity: the difference at a glance

A simple way to remember it: smoke checks the whole house has power; sanity checks that the one room you just rewired has working lights.

A word of warning: teams and textbooks don't all use these terms the same way. Some glossaries treat "sanity test" as just another name for a smoke test. In day-to-day practice, most teams use the distinction above. If you join a new team, ask what they mean by each.

Where they fit in the testing process

In the testing life cycle, a passing smoke test is usually part of the entry criteria for test execution: "build deployed and smoke test passed".

Automating smoke tests

Smoke tests are ideal for automation: few in number, run on every build, and with a clear pass or fail.

• Run them automatically after every deployment to a test environment, as a step in the CI/CD pipeline.
• Keep them fast: a few minutes at most. If the smoke suite takes an hour, it has turned into a regression suite.
• Keep them stable: a flaky smoke test blocks every build, so these need to be the most reliable tests you have.
• Run them in production too: after each release, a small set of safe, read-only smoke checks (the home page loads, search works) confirms the release went out cleanly.

Sanity tests are usually quicker to do by hand, because each one targets whatever just changed.

Common mistakes

• Skipping the smoke test "because the change was small". Small changes break builds too.
• Letting the smoke suite grow until it takes as long as full testing.
• Testing deeply during a smoke test. Its job is go / no-go, not finding every bug.
• Treating a sanity pass as "done". Sanity only says the change is worth testing properly: re-testing and regression still follow.

What's next

After the sanity check, two more things happen to every fix: the bug is re-tested, and regression tests check that nothing else broke. In Part 15 we explain regression testing vs re-testing, and how to keep regression suites fast.

ProvaLab can run your smoke suite automatically after every deployment, on real browsers and devices, and report the result in seconds, so a broken build is caught before anyone starts testing it. See it in the two-minute product tour.

The answer is the Software Testing Life Cycle, or STLC: six phases that take testing from "what should we test?" to "we're done, here's what we found". It is also one of the most common topics in QA job interviews.

What is the STLC?

The STLC is the sequence of activities a team follows to plan, design, run and finish testing. Each phase has:

• Entry criteria: what must be true before the phase can start.
• Activities: the work done in the phase.
• Deliverables: what the phase produces.
• Exit criteria: what must be true before you move on.

Here are the six phases at a glance:

We will use the same food delivery app from Part 2, and follow one new feature through all six phases: discount codes at checkout.

Phase 1: Requirement analysis

Goal: understand exactly what needs to be tested.

The testers read the requirements and ask questions before any code is written. For discount codes:

• Can a customer use two codes on one order?
• Does the code apply before or after the delivery fee?
• What message appears for an expired code?

Every unanswered question here is a bug avoided later. Testers also decide which requirements can be tested and how (manual, automated, or both).

Deliverables: a list of testable requirements, open questions, and often a Requirements Traceability Matrix (RTM): a table that links every requirement to the tests that cover it, so nothing is forgotten.

Phase 2: Test planning

Goal: decide how the testing will be done.

A test lead writes the test plan, which answers:

• Scope: what will be tested, and what won't. ("Discount codes on web and Android. iPhone next release.")
• Approach: which test levels and types, and what to automate.
• Resources: who tests what, and with which tools.
• Schedule and effort: how long it will take.
• Risks: what could go wrong, and the backup plan.
• Entry and exit criteria for the whole cycle.

Deliverables: the test plan and effort estimate.

Phase 3: Test case design

Goal: write the actual tests.

Testers turn the requirements into test cases, each with steps and an expected result, plus the test data they need:

Test cases are reviewed by another tester or a developer. Automation engineers start writing automated scripts for the cases chosen in planning.

Deliverables: reviewed test cases, test data and automation scripts. The RTM is updated so every requirement maps to its test cases.

Phase 4: Test environment setup

Goal: get a place ready to run the tests.

Tests need somewhere to run that behaves like production: servers, a database with test data, browsers, phones and the right app version. Often a separate team prepares it, and testers do a quick smoke test to check it actually works before starting.

This phase often runs at the same time as Phase 3.

Deliverables: a working test environment and smoke test results.

Phase 5: Test execution

Goal: run the tests and record what happens.

Testers run the test cases, manual and automated, and mark each one passed, failed, blocked (can't run, because something else is broken) or not run.

When a test fails, they log a defect (bug report) with the steps to reproduce it, what was expected, what actually happened, and screenshots or a video. When developers fix it, testers re-test that fix, then run regression tests to make sure the fix didn't break anything else.

Deliverables: test results, defect reports and an updated RTM showing which requirements passed.

Phase 6: Test closure

Goal: finish properly and learn for next time.

The team checks the exit criteria. For example: "all critical tests passed, no open critical bugs, at least 95% of planned tests run". Then they write a test summary report:

• What was tested, and what wasn't.
• How many tests passed, failed and were blocked.
• Bugs found, fixed and still open, by severity.
• Risks the release still carries.
• Lessons learned for the next cycle.

Deliverables: the test summary report, and the test cases and scripts saved for reuse.

Entry and exit criteria

Entry and exit criteria stop two classic problems: starting too early (testing a build that doesn't even start) and never finishing (testing forever, or stopping on a deadline without knowing the risk).

How the STLC fits the SDLC

The SDLC (Software Development Life Cycle) covers building the whole product. The STLC is the testing part of it, and it runs alongside development, not after it:

This is the "test early" idea from Part 1. When testers start on day one, the test cases are ready the moment the build is.

STLC in agile teams

In agile teams, the STLC doesn't disappear. It gets smaller and faster, and repeats every sprint:

• Requirement analysis happens during backlog refinement, when testers ask questions about each user story.
• Planning is part of sprint planning, often with an overall test strategy instead of a long plan per sprint.
• Design, execution and closure happen inside the sprint, story by story.
• Automation carries the regression checks from sprint to sprint.

Same six steps, just in days instead of months.

Quick interview recap

1. STLC = the six phases of testing: requirement analysis, test planning, test case design, environment setup, test execution, test closure.
2. Each phase has entry criteria, activities, deliverables and exit criteria.
3. The RTM links requirements to test cases, to prove everything is covered.
4. The STLC runs in parallel with the SDLC, and repeats every sprint in agile.

What's next

In Part 5 we get hands-on: your first automated test in 15 minutes, with Python and pytest.

ProvaLab's Test Management follows the same life cycle: requirements linked to test cases, test plans, test runs with every result recorded, and defects linked back to the failing test. See it in the two-minute product tour.

That's static testing, and principle 3 explains why it matters so much: the earlier a defect is found, the cheaper it is to fix.

Static vs dynamic testing

The two find different defects. A requirement that was never written can't be caught by running tests, because no test was written for it either. Only a review notices what's missing.

What can be statically tested?

Almost anything the team writes down:

• Requirements and user stories: are they complete, consistent, testable?
• Designs and architecture documents
• Code
• Test plans and test cases (yes, tests get reviewed too)
• User guides, API documentation, configuration files

Reviews: static testing by people

A review is a person, or a group, examining a piece of work to find defects and improve it. Reviews range from a quick "can you look at this?" to a formal procedure with defined roles.

Informal review

A colleague reads the work and comments. No defined process, nothing recorded. Fast and cheap, and far better than no review. Most pair programming and quick desk checks fall here.

Walkthrough

The author leads: they walk the group through the document or code, explaining it, while the group asks questions and spots problems. The main goals are shared understanding and learning, as well as finding defects. Walkthroughs are good for designs and for bringing new team members up to speed.

Technical review

A more structured discussion led by a moderator rather than the author, with technical experts checking the work against standards, designs and best practice. Findings are recorded and followed up. Typical for architecture and API designs.

Inspection

The most formal type. Inspections have defined roles (moderator, author, reader, reviewers, scribe), entry criteria (the work is complete enough to inspect), checklists, metrics (defects found per page, time spent) and exit criteria. They're expensive, so they're reserved for critical work: payment logic, safety-related code, legal text.

Where code review fits

The code review that happens on every pull request is the most common review in modern teams. Depending on the team it sits somewhere between an informal review and a technical review: a reviewer reads the change, comments, and the author fixes issues before the change is merged.

Good code reviews check:

• Correctness: does the change do what the ticket says, including error cases?
• Tests: are there tests, and do they test the right things?
• Readability: could someone else maintain this?
• Security: any injection risks, secrets in the code, missing permission checks?
• Performance: anything that will be slow with real data?

The review process

A formal review follows roughly these steps:

1. Planning: decide what to review, who reviews it, and the entry criteria.
2. Kick-off: share the work and the goals with the reviewers.
3. Individual review: each reviewer reads the work alone first and notes potential defects. This step is where most defects are found.
4. Review meeting (or discussion): reviewers share findings, decide which are real defects, and agree on actions.
5. Rework: the author fixes the agreed defects.
6. Follow-up: someone checks the fixes were made and the exit criteria are met.

Even for a pull request, step 3 matters: a reviewer who reads carefully on their own finds more than a group skimming together.

A requirements review checklist

Reviewing requirements is the highest-value static testing there is, because every later step builds on them. For each requirement, ask:

• Complete: does it say what happens on errors and edge cases?
• Unambiguous: could two people read it differently? ("Codes give 10% off": before or after the delivery fee?)
• Consistent: does it contradict another requirement?
• Testable: could you write a test case with a clear pass or fail? ("The app should be fast" is not testable; "search results appear within 1 second" is.)
• Necessary and traceable: does anyone need this, and can it be linked to a business goal?

Every question a tester asks here is a bug that will never be written.

Static analysis: static testing by tools

Static analysis tools read code without running it and flag problems automatically. They are built into most modern development setups and run in seconds.

Here is a small, deliberately flawed Python function from our food delivery app:

import os
import datetime


def apply_discount(total, code, codes):
    discount = 0
    if code in codes:
        discount = codes[code]
    if discount > 100:
        pass
    final = total - total * discount / 100
    return finall

Running Ruff, a popular Python linter, on it:

discount.py:1:8: F401 [*] `os` imported but unused
discount.py:2:8: F401 [*] `datetime` imported but unused
discount.py:11:5: F841 Local variable `final` is assigned to but never used
discount.py:12:12: F821 Undefined name `finall`
Found 5 errors.

In under a second, without running anything, the tool found:

• two unused imports (clutter, and sometimes a sign of leftover code);
• a variable that is assigned but never used;
• an undefined name: finall is a typo for final. This function would crash the first time it ran. Dynamic testing would catch it too, but only after someone wrote and ran a test; static analysis caught it the moment the file was saved.

(The fifth finding was about import ordering, a style issue.) Notice what the tool did not find: the if discount > 100: pass block does nothing, which a human reviewer would question. Tools and people complement each other.

What static analysis tools check

Run them in the CI pipeline, so every change is checked automatically before review.

Why static testing pays off

• It's early. Requirements can be reviewed before a line of code exists.
• It finds the cause, not just the symptom. A review finds the ambiguous sentence; a failing test only shows that two developers read it differently.
• It finds what dynamic testing can't: missing requirements, poor design, unreadable code, and security weaknesses that no functional test would trigger.
• It's cheap to repeat. Static analysis runs on every commit in seconds.
• It spreads knowledge. Reviews teach the team the codebase and each other's reasoning.

Common mistakes

• Skipping requirement reviews and discovering ambiguity during system testing, when it's expensive.
• Rubber-stamp reviews: approving a pull request without reading it. A review that finds nothing on a 500-line change probably wasn't a review.
• Reviewing too much at once. Defect detection drops sharply on huge changes. Smaller changes get better reviews.
• Treating linter warnings as noise and letting hundreds pile up, so the important ones are lost.
• Making reviews personal. Review the work, not the author. The goal is a better product, not a verdict on a person.

What's next

In Part 22 we look at exploratory testing: how skilled testers find the bugs that no script or checklist anticipated.

ProvaLab's PR Bot performs static testing on every pull request, combining static analysis, security checks and an AI review of the change, and feeds its findings back to the coding agent that wrote the code, so problems are fixed before a human reviewer looks. See it in the two-minute product tour.

That's system testing, the third of the four levels of testing.

What is system testing?

System testing checks the complete, integrated system against its requirements, from the outside, the way a real user would use it.

Three things make it different from the levels below it:

• The whole system is tested, not parts of it: apps, servers, database, notifications and outside services, all running together.
• From the outside, mostly as black-box testing: through the app, the website or the API, without looking at the code.
• In a production-like environment: set up like the live system, with realistic data.

System testing vs integration and acceptance testing

A system can pass system testing and still fail acceptance, if it does exactly what the requirements said but the requirements missed what users needed.

System testing vs end-to-end testing

The two terms are often used as if they meant the same thing, but there is a small difference:

• End-to-end (E2E) testing follows one complete user journey through every part of the system.
• System testing is broader: it covers those journeys plus non-functional checks like performance, security and compatibility of the whole system.

So E2E tests are one important part of system testing.

Types of system testing

Functional system testing

Does the whole system do the right thing?

• Complete journeys: sign up, search, order, pay, track and rate.
• Business rules: free delivery over 300, one discount code per order, no orders when the restaurant is closed.
• Data end to end: the order shown to the customer, the restaurant and the delivery partner has the same items and the same total.

Non-functional system testing

How well does the whole system do it?

• Performance: does it stay fast with many users at once? (We cover this in Part 16.)
• Security: can users see only their own data? Can someone order without paying?
• Usability: can a first-time user order without help?
• Accessibility: does it work with a screen reader, with only a keyboard, with large text?
• Compatibility: does it work on the browsers, phones and operating systems your users have?
• Recovery: if a server restarts mid-payment, is the order lost, or charged twice?
• Installation and upgrade: does the new app version install cleanly and keep the user's saved addresses?
• Localisation: are languages, dates, currencies and phone numbers right in every country you serve?

Example system test cases

Notice how each one exercises many parts of the system at once, and how several test what happens when something goes wrong.

Entry and exit criteria for system testing

Start system testing when:

• Integration testing is complete, with no open critical integration bugs.
• The full system is deployed to a production-like environment.
• Test data and test accounts are ready.
• A smoke test of the build has passed (Part 14).

Finish system testing when:

• All planned system tests have been run.
• All high-priority tests pass.
• No critical or high-severity bugs are open (or the remaining ones are accepted risks).
• The test summary report is shared.

Doing system testing well

1. Use a production-like environment. Bugs that only appear with the real configuration, a real-sized database or the real network setup won't show up on a developer's laptop.
2. Use realistic test data. Long names, addresses with special characters, customers with a hundred past orders. Never use real customers' personal data; create test data that looks like it.
3. Test the unhappy paths. Payment failures, closed restaurants, lost network, expired sessions. Users meet these every day.
4. Think like different users: a first-time customer, a regular, a restaurant owner, a delivery partner, an admin.
5. Automate the critical journeys. Sign-up, sign-in and checkout should run automatically on every release, with tools like Playwright or Selenium. Keep the rest manual or exploratory, as the testing pyramid suggests.
6. Trace every test to a requirement, using the traceability matrix, so you know the whole specification is covered.

Common mistakes

• Testing on a developer environment that differs from production in small but important ways.
• Only testing happy paths because "the requirements didn't mention errors".
• Skipping non-functional testing until after release, then discovering the app is slow or inaccessible.
• Too many automated end-to-end tests, making the suite slow and flaky. Automate the journeys that matter most.
• Starting before integration is stable, so system testers spend their days reporting integration bugs.

What's next

Before deep testing like this begins, teams run a quick check to make sure the build is even worth testing. In Part 14 we cover smoke testing and sanity testing: what each one is, and when to use which.

ProvaLab runs your end-to-end and system tests on real browsers and devices, records a video of every run, and keeps each result linked to its test case and requirement, so system test reports are always up to date. See it in the two-minute product tour.

In this part we look at the main kinds of test coverage, how to calculate each one, and how to use them without being fooled.

What is test coverage?

Test coverage measures how much of something your tests have checked, as a percentage. That "something" can be:

• the requirements (what the software should do),
• the test cases you planned to run,
• the code itself,
• the platforms your users have: browsers, devices and operating systems.

Each kind answers a different question:

1. Requirement coverage

Question: did we test everything the software is supposed to do?

Requirement coverage = (requirements with at least one test ÷ all requirements) × 100

Example: from the traceability matrix in Part 11, the discount-code feature has 5 requirements, and 4 of them have test cases (REQ-14, Android, has none).

4 ÷ 5 × 100 = 80%

A stricter version only counts requirements whose tests pass:

Passing requirement coverage = (requirements whose tests all pass ÷ all requirements) × 100

In the same example, REQ-12 is tested but failing, so: 3 ÷ 5 × 100 = 60%.

Requirement coverage is the one business people care about most: it speaks the language of features, not code.

2. Execution coverage

Question: did we actually run the tests we planned?

Execution coverage = (test cases run ÷ test cases planned) × 100

"Run" means the test produced a real verdict: passed or failed. Blocked, skipped and not-run tests don't count, because we learned nothing from them.

Example: in Run 42 from Part 17, 120 test cases were planned: 104 passed, 9 failed, 4 were blocked and 3 skipped.

(104 + 9) ÷ 120 × 100 = 94%

Don't confuse this with the pass rate, which asks how many of the tests that ran passed: 104 ÷ 113 = 92%. Execution coverage tells you how much you tested; pass rate tells you how well it did.

3. Code coverage

Question: which parts of the code did the tests actually run?

Code coverage is measured by a tool while the tests run: coverage.py for Python, JaCoCo for Java, Istanbul/nyc or Jest's built-in coverage for JavaScript, and so on. There are several kinds:

Line (statement) coverage = (lines executed ÷ all executable lines) × 100

Branch coverage = (branch outcomes taken ÷ all branch outcomes) × 100

Function coverage = (functions called ÷ all functions) × 100

The difference between line and branch coverage matters a lot. Here is the delivery-fee function from Part 7, with a single test:

• Line coverage: all 7 lines ran → 7 ÷ 7 = 100%.
• Branch coverage: each if has two outcomes (true and false), so there are 4 outcomes. The test only took the 2 "true" ones → 2 ÷ 4 = 50%.

100% line coverage, yet half the decisions in the code were never tested. That's why branch coverage is the more honest number.

A note on tools: some tools combine lines and branches into one figure. In Part 7, coverage.py reported 82% for this same test: it counts the 7 lines plus the 4 branch outcomes, and the test covered 7 + 2 = 9 of those 11. (9 ÷ 11 = 82%.) When you compare coverage numbers, check exactly what each tool is counting.

Path coverage goes further still: every route through the code. With two if statements there are four paths (big order + member, big + non-member, small + member, small + non-member). Path coverage is very thorough, but grows so fast in real code that it is usually only used for critical logic.

4. Platform coverage

Question: did we test on the browsers and devices our users actually have?

Platform coverage = (supported platform combinations tested ÷ all supported combinations) × 100

Example: the food app officially supports Chrome, Safari and Firefox on desktop, plus the Android app, the iPhone app, and Chrome on Android phones: 6 combinations. This release was tested on 4 of them.

4 ÷ 6 × 100 = 67%

Use your analytics to weight this: missing a browser that 2% of users have matters far less than missing the one that 50% use.

Other coverage measures you may hear

• Risk coverage: how many of the identified high-risk areas have tests.
• Feature coverage: like requirement coverage, but counted per feature or user story.
• Mutation score: a test of your tests. A tool makes small deliberate changes to the code (for example, >= becomes >) and checks whether any test fails. Mutation score = changes caught ÷ changes made × 100. Tests with no real assertions score badly here, even with high code coverage.

What coverage target should you aim for?

There is no universal right number. Some sensible rules of thumb:

• Requirement coverage: aim for 100% of high-priority requirements before release. Any gap should be a deliberate, documented decision, not a surprise.
• Execution coverage: aim for 100% of planned tests run, and explain every blocked or skipped one in the test report.
• Code coverage: many teams aim for around 70–80% line or branch coverage, with higher targets for critical code like payments. Chasing 100% everywhere often produces low-value tests written just to touch lines.
• Platform coverage: cover the combinations used by most of your real users, plus anything business-critical.

More useful than any single target: watch the trend, and make sure coverage doesn't drop when new code is added.

Why 100% coverage doesn't mean bug-free

Coverage tells you what your tests touched. It does not tell you whether they checked the right things:

• A test with no assert still gives coverage.
• 100% requirement coverage only means each requirement has a test, not a good one.
• Code coverage can't find a feature that was never built. You can't cover code that doesn't exist (see Part 7).
• You can't test every possible input, so coverage will always be of a chosen sample.

So use coverage as a map of what you haven't tested, not as proof of quality. Low coverage is a clear warning; high coverage is a good sign, not a guarantee.

Quick reference

What's next

That completes this batch of the Testing Basics series, from what testing is to coverage. Coming up next: the 7 principles of testing, QA vs QC, static testing, exploratory testing and testing in agile and CI/CD.

ProvaLab Test Management keeps requirement and execution coverage up to date automatically: requirements link to test cases, every manual or automated run records its results, and coverage reports show the gaps by feature, test plan and run. See it in the two-minute product tour.

This part explains test runs and test results: what they are, what each result status means, how to calculate a pass rate correctly, and how to read a run like an experienced tester.

Test case, test run and test result

These three are easy to confuse, but each has a different job:

• Test case: what to check. Written once and kept in the catalogue. "TC-104: an expired discount code must be refused."
• Test run: one execution of a set of test cases, at a specific time, on a specific build and environment. "Run 42: release 2.4 regression, build 2.4.0, staging, 4 November."
• Test result: what happened to one test case in one run. "TC-104 in Run 42: Failed. The total became 450 instead of showing 'code expired'."

A test case is written once. It is run many times. Every run gives it a new result. Over months, those results become the test case's history: "TC-104 passed on every build until 2.4.0".

You may also hear test execution (the act of running tests) and test cycle (one round of testing, often made up of several runs, such as "the release 2.4 test cycle").

What a test run records

The build and environment are what make a run meaningful. "TC-104 failed" is not useful on its own; "TC-104 failed on build 2.4.0 on staging, and passed on 2.3.9" tells a developer exactly where to look.

What a test result records

For a passed test, the status is often enough. For a failure, the evidence is what turns a red mark into something a developer can fix.

Every result status explained

Blocked is not the same as failed. If sign-in is broken, every test that needs a signed-in user is blocked: we learned nothing about those features, good or bad. Counting them as failed exaggerates the damage; ignoring them hides how much is untested.

Calculating the pass rate correctly

Here is the summary of Run 42:

There are two different pass rates, and both are useful:

• Pass rate of executed tests = passed ÷ (passed + failed) = 104 ÷ 113 = 92%. How good is the software in the parts we actually tested?
• Pass rate of planned tests = passed ÷ all planned tests = 104 ÷ 120 = 87%. How much of the plan has been proven to work?

The gap between them is the blocked and skipped tests: 7 tests, nearly 6% of the plan, that told us nothing. A report that only says "92% passed" hides that.

How to read a test run

A run with 9 red results is not 9 problems. Read it like this:

1. Check the run is valid. Right build? Right environment? Did it finish, or stop halfway?
2. Look at blocked and skipped first. They show what you didn't learn.
3. Group failures by cause, not by test. In Run 42, six failures come from one cause: a renamed button that the tests still look for by its old name. That's one fix to the test code, not six bugs.
4. Separate real bugs from test or environment problems.
   • Product bug: the app is wrong (BUG-231).
   • Test problem: the test is outdated or wrong (the renamed button).
   • Environment problem: something around the app failed (the payment sandbox timeout).
5. Compare with the previous run. A test that passed yesterday and fails today on a new build is a likely regression (see Part 15). A test that fails on some runs and passes on others with no change is probably flaky.
6. Read the actual error before describing a failure. Don't guess from a test's name what went wrong; open the result, its message and its recording.
7. Link every real failure to a bug (see Part 9), so the run, the test case and the bug can all be traced to each other.

Manual and automated runs

• Manual runs: a tester works through the test cases and records each result by hand, with screenshots for failures.
• Automated runs: a tool runs the tests (often every night, or on every code change) and records the results itself, often with logs and video.

Most teams have both, and the best practice is to keep their results in the same place, so a release report covers everything tested, however it was run.

Common mistakes

• Only reporting "X% passed", without the blocked and skipped tests.
• Re-running failed tests until they pass, then reporting green. That hides flaky tests and real intermittent bugs.
• Counting failures instead of causes, so one broken selector looks like a disaster.
• Not recording the build and environment, so nobody can tell what the result applies to.
• Results scattered across spreadsheets, chat messages and CI logs, so nobody has the full picture.

What's next

In Part 18, the last of this set, we look at test coverage: the different kinds, and how to calculate each one.

In ProvaLab Test Management, every run, manual or automated, records its build, environment and a result for each test case, with the error, screen recording and logs attached to failures. Each test case keeps its full history across runs, and failures can be grouped by cause and linked to bugs. See it in the two-minute product tour.

This final part of Testing Basics shows how everything we've covered fits together when a team works like that.

Testing in agile teams

Agile methods such as Scrum and Kanban build software in short cycles (sprints of one to four weeks, or a continuous flow of work). Four things change for testing:

1. Quality is the whole team's job

There's no separate testing phase, so there can't be a separate testing team that gets the software at the end. Developers write unit tests and review each other's code. Testers join the planning conversations, ask the awkward questions early, and spend their time on the things only people can do: exploratory testing, test design and risk. This is often called the whole-team approach.

2. Testing happens inside the sprint

Each user story is tested in the same sprint it's built. A story isn't "done" when the code is written; it's done when it meets the team's definition of done, which usually includes: tests written and passing, code reviewed, acceptance criteria checked, and no open bugs of high severity.

3. Acceptance criteria are the test cases

Each user story carries acceptance criteria agreed before development starts. They play the role that requirements and test cases played in the STLC: they say what "working" means, and each one becomes at least one test. A common format is Given / When / Then:

Given a signed-in customer with 500 in the cart When they apply the expired code OLD10 Then they see "This code has expired" and the total stays 500

4. Automation is not optional

If the regression suite takes a week to run by hand, you can't release every week. Agile teams automate their regression tests and run them continuously, so the humans are free to explore.

The agile testing quadrants

Brian Marick's agile testing quadrants, made popular by Lisa Crispin and Janet Gregory, are a map of all the testing an agile team does. They answer the question "are we doing all the kinds of testing we need, or only the easy ones?"

• Q1 and Q2 support the team while building: they tell developers quickly whether the code does what the story asked.
• Q3 and Q4 critique the finished product: they find what the stories didn't anticipate.

Most teams are strong in Q1 and Q2 and weak in Q3 and Q4. If your team has thousands of unit tests but nobody has done a load test or an exploratory session this quarter, the quadrants show the gap.

Shift left, shift right

Shift left means moving testing earlier in the process: reviewing requirements, writing tests before code, running tests on every commit. Everything from principle 3 (early testing) and static testing is shifting left.

Shift right means continuing to test after release, in production: monitoring, alerts, feature flags that release a change to 1% of users first, and small smoke checks that run against the live site. Shift right accepts that some problems only show up with real users and real data, and makes sure you learn about them in minutes rather than weeks.

Modern teams do both: catch what you can early, and watch closely for what you couldn't.

TDD and BDD

Two practices shape how tests are written in agile teams:

Test-driven development (TDD) is a developer practice: write a failing unit test first, write just enough code to make it pass, then clean up, and repeat. The tests become a design tool, and every line of code exists because a test asked for it. (You saw the value of a test you've watched fail in Part 5.)

Behaviour-driven development (BDD) applies the same idea at the story level, using the Given/When/Then format above. The acceptance criteria are written so that tools such as Cucumber, Behave or SpecFlow can run them directly as automated tests. The main benefit is shared language: product owners, developers and testers all read the same examples.

Neither is mandatory for agile, but both make "acceptance criteria are the test cases" literally true.

Continuous integration and continuous delivery

Continuous integration (CI) means every change is merged into the shared codebase frequently (at least daily) and automatically built and tested each time. Continuous delivery (CD) extends the pipeline so that every change that passes the tests is ready to release, and continuous deployment releases it automatically.

The pipeline is where all the test types in this series take their places:

Three rules make a pipeline work:

1. Each stage is a gate. A failure stops the change from going further, and the team fixes it before doing anything else. A red pipeline that everyone ignores protects nobody.
2. Fast first, deep later. The tests that run on every commit must be fast, or developers stop committing often. The slow, thorough tests run less frequently.
3. Every result is recorded. Which build, which environment, which tests, which result (see test runs), so a failure can be traced and a release can be reported on.

What a pipeline looks like

Here is a minimal GitHub Actions workflow that runs the static analysis and unit tests from Parts 5 and 21 on every push:

name: tests
on: [push, pull_request]

jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install pytest ruff
      - run: ruff check .        # static analysis
      - run: pytest              # unit tests

If either command fails, the workflow fails, the pull request shows a red cross, and the change can't be merged until it's fixed. Real pipelines add more stages, but they all start like this.

Where AI coding agents fit

The newest change to this picture is that much of the code is now written by AI coding agents such as Claude Code and Cursor. That makes two things in this series more important, not less:

• Shift left matters more, because code is produced faster than people can review it. Static analysis, unit tests and automated review on every pull request are what keep up.
• The pipeline becomes the agent's feedback loop. When a test fails, the agent that wrote the code can read the failure, the logs and the recording, fix the code and run the tests again, with a person approving the result rather than doing the checking by hand.

That is the idea ProvaLab is built around, and the subject of our first post, how to test code your AI coding agent wrote.

Common mistakes

• Keeping a separate "testing sprint" at the end. That's a waterfall with shorter phases.
• Ignoring red pipelines. Once one failure is tolerated, every failure is.
• A slow pipeline. If the commit stage takes 30 minutes, people batch up changes, and the fast feedback is lost.
• Only Q1 and Q2 testing. Unit tests and story tests don't tell you the product is slow, insecure or confusing.
• No shift right. Releasing without monitoring means customers are your alerting system.

Testing Basics: the complete series

That's the end of Testing Basics. Over 23 parts you've learned:

Foundations: what testing is · types and levels · manual vs automated · the STLC · the 7 principles · QA vs QC, verification vs validation

The craft: your first automated test · writing test cases · test design techniques · test plans · bug reports · the bug life cycle · the traceability matrix · test runs and results · test coverage

The test types: integration · system · smoke and sanity · regression and re-testing · performance · static testing · exploratory testing · agile and CI/CD (this post)

Next on the blog, the Hands-On Testing series: deeper guides to UI testing, API testing, security testing and load testing, each with examples you can run. It starts with Part 1: UI testing explained.

ProvaLab is built for the pipeline described here: the PR Bot checks every pull request, QA Infrastructure runs the tests on real browsers and devices, Test Management records every run, and AppSec and load testing cover Q4, with your coding agent able to drive all of it. See it in the two-minute product tour.

Testers answer that with four levels of testing. Each level looks at the software from a different distance, from one small piece of code up to the whole product in a customer's hands. We will walk through all four using one example: a food delivery app.

The four levels at a glance

1. Unit testing

A unit test checks one small piece of code on its own, usually a single function.

In our app, a unit test might check the function that adds up an order:

• Two pizzas at 200 and a drink at 50 should total 450.
• A 10% discount code should bring it to 405.
• An empty cart should total 0, not crash.

Developers write unit tests while they write the code. They run in milliseconds, so a project can have thousands of them and still check everything on every change.

What it can't tell you: whether the pieces work together. A perfect total function is no use if the checkout page never calls it.

2. Integration testing

An integration test checks that pieces work together: your code with the database, with another service, or with an outside system like a payment provider.

In our app: when a customer places an order, does the order service save it to the database and notify the restaurant? Each piece may pass its own unit tests and still fail here. A field could be named differently on each side, or a date could be sent in the wrong format.

Integration tests are slower than unit tests, because they need real databases or services (or realistic stand-ins). Teams usually have fewer of them.

3. System testing

A system test checks the whole app, end to end, the way a user would use it.

In our app: open the app, search for "pizza", add two to the cart, pay, and follow the delivery on the map. Every part of the system takes part: the screens, the servers, the database and the payment provider.

System testing also covers the non-functional questions from Part 1: is it fast enough, is it secure, is it easy to use? This is usually where QA engineers and automated UI tests do most of their work. You will often hear these called end-to-end (E2E) tests.

4. Acceptance testing

An acceptance test checks that the software actually solves the business need, and is ready to release.

The question changes from "does it work?" to "is this what we needed?" In our app, a restaurant owner might try the new order screen and confirm that orders arrive with everything their kitchen needs. If the app works perfectly but shows orders in a way the kitchen can't use, it fails acceptance.

Acceptance testing is often done by customers, product managers or a small group of real users. You will hear names like UAT (user acceptance testing) and beta testing (a release to a small group of real users first).

The testing pyramid: how many of each?

If end-to-end tests are closest to what users do, why not write only those? Because they are slow, costly to maintain, and when one fails it can take a while to find out why.

The testing pyramid, an idea popularised by Mike Cohn, gives a simple rule of thumb:

• Many unit tests at the base: fast, cheap, and they point straight at the broken line.
• Some integration tests in the middle.
• A few end-to-end tests at the top, for the journeys that matter most, such as sign-up and checkout.

The opposite shape, lots of slow end-to-end tests and few unit tests, is sometimes called the ice-cream cone. Its test runs are slow, flaky and hard to debug.

Other testing words you will hear

Not every testing word is a level. Some describe why or when you run tests, at any level:

Putting it together

A healthy project has all four. Each level catches problems the others miss.

What's next

In Part 3 we look at manual vs automated testing: what to automate, what not to, and why.

Running end-to-end tests is where ProvaLab helps most: your coding agent runs them on real browsers and devices, every result is kept in one place, and failures come back with a video and logs. See it in the two-minute product tour.

We start with the kind of testing most people picture when they hear the word: a program that opens a browser and uses your app like a person would.

What is UI testing?

UI testing (user interface testing) checks an application through the same interface users see: pages, buttons, forms and screens, in a real browser or on a real device.

It sits at the top of the testing pyramid: the closest to a real user, and also the slowest and most expensive kind of test to write and maintain. That's why the pyramid says to have a few UI tests, for the journeys that matter most, not hundreds.

Two terms you'll hear alongside it:

• End-to-end (E2E) testing: a UI test that follows a complete journey through the whole system, such as search → add to cart → pay → see confirmation. Most UI tests of any value are end-to-end tests.
• Visual testing: comparing screenshots against a known-good version to catch layout and styling changes. A specialised branch of UI testing.

What UI tests are good at, and bad at

The rule of thumb: test the logic with unit tests, the joins with integration tests, and the journeys with UI tests. A UI test that checks fifty discount-code combinations is doing a unit test's job, fifty times slower.

How a UI test works

Every UI test, in every tool, is the same three steps repeated:

1. Locate an element on the page: the email field, the "Sign in" button.
2. Act on it: type, click, select, scroll.
3. Assert what the page shows afterwards: the heading, the URL, a message, the absence of an error.

Here is a complete sign-in test in Playwright for Python. Even if you've never seen Playwright, you can read it:

def test_customer_can_sign_in(page):
    page.goto("https://staging.example.com/sign-in")
    page.get_by_label("Email").fill("test-user-2@example.com")
    page.get_by_label("Password").fill("correct-horse-battery")
    page.get_by_role("button", name="Sign in").click()
    expect(page.get_by_role("heading", name="Your orders")).to_be_visible()

Locate (get_by_label), act (fill, click), assert (expect … to_be_visible). We'll build and run this for real in Part 2.

Between the script and the browser sits a driver or automation protocol. Selenium uses the WebDriver standard, a W3C protocol every major browser supports. Playwright and Cypress talk to browsers more directly, which is a large part of why they feel faster.

Locators: the skill that decides whether your tests survive

The single biggest cause of broken UI tests is the way they find elements. A test that looks for div.col-md-6 > form > button.btn-primary breaks the day a designer changes a class name, even though the app works perfectly.

The order of preference most teams settle on:

1. Role and accessible name (get_by_role("button", name="Sign in")). This is how a screen reader sees the page, so it matches what users see and ignores styling. It also quietly pushes your app towards being accessible.
2. Test IDs (data-testid="checkout-total"), attributes added to the HTML purely for tests. Very stable; the only cost is asking developers to add them.
3. Visible text and labels. Readable, but tied to wording and language.
4. CSS selectors. Fine for structure that won't change; fragile when tied to styling classes.
5. XPath by position. Avoid. It breaks whenever anything moves.

A good test reads like a description of what a user does. If you can't tell what a locator points at without opening the page, it's the wrong locator.

Why UI tests go flaky, and what to do about it

A flaky test passes and fails on different runs with no code change. UI tests are the flakiest kind, for predictable reasons:

Flaky tests are dangerous, not just annoying: once people learn to re-run red tests until they pass, real failures get re-run away too (see Part 17). Treat a flaky test as a bug in the test suite and fix it the same week.

Selenium vs Playwright vs Cypress

The three tools almost every team chooses between:

Some honest observations:

• Selenium is the industry standard with the largest ecosystem and the most job listings. Its WebDriver protocol is why it works everywhere, including real mobile devices through Appium. Its main cost is that you manage waiting yourself, which is where many flaky Selenium suites come from.
• Playwright was built by the team that originally created Puppeteer at Google, after they moved to Microsoft. It's fast, waits automatically, tests all three browser engines with one API, and its trace viewer (a recording of every step, with screenshots and network calls) is the best debugging tool of the three. It is the default choice for new projects today.
• Cypress has the most pleasant developer experience for front-end engineers: tests run in the browser with a live, time-travelling view. The trade-offs are JavaScript-only, weaker multi-tab and cross-origin support, and a paid service for parallel runs and recordings.

If you're starting from nothing, pick Playwright. If your team already has a Selenium suite that works, there is no need to rewrite it. If your team is all front-end JavaScript developers, Cypress will make them happiest.

Where UI tests run

• On your laptop while writing them.
• In the CI pipeline on every change, usually headless (no visible browser window), as described in Part 23 of Testing Basics.
• On a browser grid or device cloud when you need many browsers, operating systems and real phones at once, far more than one machine can hold. This is what services like ProvaLab's QA Infrastructure, BrowserStack and Sauce Labs provide.

Seven habits of good UI tests

1. Test journeys, not pages. "A new customer can sign up and place an order" beats "the sign-up page has an email field".
2. Keep them few and valuable. Push everything else down the pyramid.
3. Use role-based locators or test IDs.
4. Make every test independent: its own data, its own sign-in.
5. Never sleep. Wait for a condition, not a number of seconds.
6. Assert on what the user sees, not on internal state.
7. Record on failure. A screenshot, video or trace turns a red test into a bug report in seconds.

What's next

In Part 2 we build real tests like the one above: install Playwright, write the test, run it headless and headed, watch it fail, read the trace, and add it to a pipeline.

ProvaLab's QA Infrastructure runs Playwright and Selenium tests on real browsers and devices in the cloud, records every session, and hands the recording and logs back to your coding agent when a test fails. See it in the two-minute product tour.

This is Part 1 of our Testing Basics series. No prior knowledge needed. By the end you will know what software testing is, what it checks, when it happens and who does it.

What is software testing?

Software testing is checking that software does what it should, and doesn't do what it shouldn't.

That sentence has two halves, and both matter:

• Does what it should: a user can sign up, add items to a cart and pay.
• Doesn't do what it shouldn't: a user can't see someone else's orders, the app doesn't crash on an empty search, and a wrong password doesn't let anyone in.

A test is one small check of that kind. It has three parts:

1. A starting point: "A signed-in user with one item in their cart."
2. An action: "They click Pay."
3. An expected result: "They see the order confirmation, and the cart is empty."

If what actually happens matches the expected result, the test passes. If not, it fails, and you have found a bug: a difference between what the software does and what it should do.

What does testing check?

Testing answers four big questions about your software:

The first question, "does it work?", is called functional testing. The other three are non-functional testing: they check how well the software works, not just whether it does. Each one gets its own posts later in this series.

Why does every team need testing?

• Users leave quickly. If the checkout breaks, most people don't report it. They just go to a competitor.
• Bugs get more expensive the later you find them. A mistake caught while writing the code takes minutes to fix. The same mistake found by a customer means support tickets, an urgent fix, a new release and lost trust.
• Security problems are business problems. One leaked customer record can cost far more than years of testing.
• Changes break old things. Every new feature can break an old one. Tests that run on every change catch that, so the team can move fast without fear.
• Code is written faster than ever. With AI coding assistants, teams produce far more code. Checking it has to keep up.

When does testing happen?

A common mistake is to test only at the end, just before release. Good teams test at every stage:

This idea is often called "shift left": move testing earlier (to the left) on the timeline, where bugs are cheapest to fix.

Who does the testing?

• Developers write unit tests that check small pieces of their own code.
• QA engineers (quality assurance) plan what to test, test like a real user would, and look for the cases nobody thought of.
• Automation engineers turn repeated checks into code that runs by itself, on every change.
• Security and performance specialists test the harder questions: can it be attacked, and does it hold up under heavy traffic?
• AI agents are the newest member of the team: they can now write and run tests, and read the results.

In a small team, one person may do all of these. What matters is that each job gets done, not who has which title.

Manual and automated testing

Testing can be done in two ways:

• Manual testing: a person uses the app and checks the result by eye. Best for new features, design and "does this feel right?"
• Automated testing: code performs the checks. Best for anything you need to check again and again, like "can users still sign in?" on every release.

Most teams use both. We cover automation in detail in Part 3.

Four myths about testing

1. "Testing proves there are no bugs." It can't. Testing shows bugs are there; it never proves they are all gone. Its job is to find the important ones before users do.
2. "Testing is the QA team's job." Quality is everyone's job. Developers, reviewers and testers each catch different problems.
3. "We'll test at the end." By the end, bugs are tangled together and expensive. Test as you go.
4. "100% test coverage means we're safe." Coverage tells you which lines of code ran during tests, not whether the tests checked the right things.

Key terms

What's next

In Part 2 we look at the types of testing (unit, integration, system and acceptance tests) and when to use each.

Want to see testing done by an AI agent on a real app? Watch the two-minute product tour. ProvaLab keeps your test cases, runs them on real browsers and devices, and records every result in one place.