The test summary report (hands-on): 96.8% passed, not ready to release, and the metrics that tell you why

At the end of a test cycle, someone has to decide: release or not? The test summary report is what they decide from. ISTQB now calls it a test completion report, and the ISO/IEC/IEEE 29119-3 standard has a template for it. Whatever its name, it has one job: give the people who decide an honest picture of what was tested, what was found, what wasn't tested, and what that means for the release.

Most summary reports fail that job the same way. They lead with a big number, such as "96.8% of tests passed", and the big number hides the one thing that matters.

This post builds a summary report from real test results. We ran the actual test suites from earlier parts of this series as one FoodNow release candidate, with five tools' results in one place. A short Python script turns the results into a report by requirement and risk. Every result and every line of output below is real. Two things were deliberate:

• We put one bug from Part 12 back into the release candidate.
• We wrote the list of requirements and their risks for this example. In a real project, they come from your test plan.

What goes in a test summary report

Section	What it says
What was tested	The build, the environments, the dates, and the scope from the test plan
Results by requirement and risk	Which requirements pass, which fail, which have no tests
Open defects	By severity, with what each one means for a user
What wasn't tested, and why	Scope cuts, blocked tests, environments you didn't have
Changes from the plan	Dates, scope, approach
Metrics, with context	A few numbers that help the decision
Recommendation	Release, release with conditions, or don't. And why.

The order matters. Lead with what affects the decision, not with what's easiest to count.

Step 1: Collect the results in one format

Every test tool can write JUnit XML, a simple format that most CI systems and reporting tools read. We ran five suites from this series against the release candidate:

# Parts 8, 11, 7 and 9: pytest suites
pytest tests --junitxml=results/pricing-bdd.xml           # BDD pricing scenarios
pytest tests --junitxml=results/order-page-selenium.xml   # Selenium order page
pytest tests --junitxml=results/orders-contract.xml       # Pact contract
pytest tests --junitxml=results/order-for-later.xml       # order for later

# Part 12: Cypress, with its built-in JUnit reporter
npx cypress run --spec cypress/e2e/order.cy.js \
  --reporter junit --reporter-options "mochaFile=results/order-page-cypress.xml"

9 passed in 0.26s
4 passed in 14.59s
6 passed, 1 warning in 6.30s
1 failed, 4 passed, 1 warning in 1.88s
 │ Passing: 5 │
 │ Failing: 1 │

The Cypress failure we expected: it's the bug we put back. The order-for-later failure we didn't. We'll come to it in Step 4.

Step 2: The number everyone asks for

Save this as summary.py. It reads every JUnit file in a folder, and a list of requirements:

"""Build a test summary from JUnit XML files, by requirement and risk, not just by count."""
import csv
import glob
import re
import sys
import xml.etree.ElementTree as ET
from pathlib import Path

results_dir, req_file = sys.argv[1], sys.argv[2]
triage_file = sys.argv[3] if len(sys.argv) > 3 else None

# 1. Every test result, as "suite::test name".
tests = {}
for f in sorted(glob.glob(f"{results_dir}/*.xml")):
    suite = Path(f).stem
    for tc in ET.parse(f).iter("testcase"):
        problem = tc.find("failure")
        if problem is None:
            problem = tc.find("error")
        message = ""
        if problem is not None:
            message = (problem.get("message") or problem.text or "").strip().splitlines()[0][:90]
        tests[f"{suite}::{tc.get('name')}"] = ("failed" if problem is not None else "passed", message)

triage = {}
if triage_file:
    for row in csv.DictReader(open(triage_file, encoding="utf-8")):
        triage[row["test"]] = (row["kind"], row["note"])

total = len(tests)
passed = sum(1 for s, _ in tests.values() if s == "passed")
print(f"Tests run: {total}   passed: {passed}   failed: {total - passed}   pass rate: {passed / total:.1%}\n")

# 2. By requirement.
print(f"{'req':<4}{'risk':<8}{'tests':>6}{'failed':>8}  status       requirement")
blockers, mapped = [], set()
for r in csv.DictReader(open(req_file, encoding="utf-8")):
    ids = [t for t in tests if r["tests"] and re.search(r["tests"], t, flags=re.IGNORECASE)]
    mapped.update(ids)
    failed = [t for t in ids if tests[t][0] == "failed"]
    product_failures = [t for t in failed if triage.get(t, ("product bug",))[0] != "test bug"]
    if not ids:
        status = "NOT TESTED"
    elif product_failures:
        status = "FAILING"
    elif failed:
        status = "test issue"
    else:
        status = "passing"
    print(f"{r['id']:<4}{r['risk']:<8}{len(ids):>6}{len(failed):>8}  {status:<12} {r['requirement']}")
    if r["risk"] == "high" and status in ("NOT TESTED", "FAILING"):
        blockers.append(f"{r['id']} ({r['risk']} risk) {status.lower()}: {r['requirement']}")

unmapped = [t for t in tests if t not in mapped]
if unmapped:
    print(f"\n{len(unmapped)} test(s) not linked to any requirement:")
    for t in unmapped:
        print("  ", t)

# 3. Failures, with their triage.
print("\nFailures:")
for t, (status, msg) in tests.items():
    if status == "failed":
        kind, note = triage.get(t, ("not triaged", ""))
        print(f"  [{kind}] {t}\n      {msg}" + (f"\n      triage: {note}" if note else ""))

# 4. Against the exit criterion: every high-risk requirement tested, with no product failures.
print("\nReady to release?", "NO" if blockers else "YES")
for b in blockers:
    print("  -", b)

The first line of its output is the number most reports lead with:

Tests run: 30   passed: 28   failed: 2   pass rate: 93.3%

93.3% sounds nearly ready. It tells you nothing about what failed, or what was never tested.

Step 3: Link every test to a requirement and its risk

A release decision is about requirements, not tests. Which things that customers need are known to work, which are known to be broken, and which nobody checked? This requirements.csv lists FoodNow's requirements for this release, each with its risk and a pattern that picks out its tests:

id,requirement,risk,tests
R1,Totals and delivery fee are right,high,pricing-bdd::.*(delivery|big_order)|order-page-.*::.*(delivery|500)
R2,Promo codes work and are capped,medium,pricing-bdd::.*(first50|unknown_code|discount)
R3,The basket can be edited,medium,order-page-.*::.*(remov|empty)
R4,Customers are told when the price can't be worked out,high,order-page-cypress::.*price service
R5,Order for later offers only future slots in kitchen time,high,order-for-later::.*
R6,The app and the orders API agree,high,orders-contract::.*
R7,Group order: colleagues add dishes and one person pays,high,

R7 has no pattern, because no tests exist for it yet. Group ordering is in the release, but its tests weren't written in time.

Our first run of the script printed this:

R1  high         4       0  passing      Totals and delivery fee are right
R2  medium       0       0  NOT TESTED   Promo codes work and are capped
R3  medium       0       0  NOT TESTED   The basket can be edited
R4  high         0       0  NOT TESTED   Customers are told when the price can't be worked out
...
15 test(s) not linked to any requirement:
   order-page-cypress::FoodNow order charges delivery on a small order
   ...

Half the tests weren't linked to anything. That was a bug in our first version of the script: it used re.fullmatch, so a pattern had to match the whole test name. The fixed version above uses re.search. The lesson is the check itself: always list the tests that aren't linked to any requirement. Without that list, a mistake in the linking would quietly turn "tested" into "NOT TESTED", or the other way round, and the report would be wrong without anyone knowing.

With the fix, every test is linked:

$ python summary.py results requirements.csv
Tests run: 30   passed: 28   failed: 2   pass rate: 93.3%

req risk     tests  failed  status       requirement
R1  high         9       0  passing      Totals and delivery fee are right
R2  medium       4       0  passing      Promo codes work and are capped
R3  medium       4       0  passing      The basket can be edited
R4  high         2       1  FAILING      Customers are told when the price can't be worked out
R5  high         5       1  FAILING      Order for later offers only future slots in kitchen time
R6  high         6       0  passing      The app and the orders API agree
R7  high         0       0  NOT TESTED   Group order: colleagues add dishes and one person pays

Failures:
  [not triaged] order-for-later::test_page_offers_slots_from_now
      IndexError: list index out of range
  [not triaged] order-page-cypress::when the price service misbehaves says so when the price service is down
      Timed out retrying after 4000ms: expected '<span#total>' to contain text 'couldn\'t work o

Ready to release? NO
  - R4 (high risk) failing: Customers are told when the price can't be worked out
  - R5 (high risk) failing: Order for later offers only future slots in kitchen time
  - R7 (high risk) not tested: Group order: colleagues add dishes and one person pays

Now the picture is different. Two failures, each in a high-risk requirement, and a high-risk feature that's in the release with no tests at all.

Step 4: Triage every failure

A failed test means something went wrong. It doesn't yet say whether it's the product, the test, or the environment. Every failure needs a person to look at it before it goes into a report.

The Cypress failure is the bug we put back: when the price service fails, the page shows "Total: ₹undefined". A product bug.

The order-for-later failure we didn't plan. The test is from Part 9:

def test_page_offers_slots_from_now():
    html = client.get("/").text
    first = fn.slots(fn.now())[0]
    assert f"{first:%H:%M} –" in html

We ran the suites at 23:09. The kitchen closes at 23:00, so there are no slots, and [0] fails with IndexError. The page itself was right: it said the kitchen was closed. We checked exactly when this test fails:

time   slots  first
21:44  1      22:30
21:45  1      22:30
21:46  0      -
07:59  0      -
08:00  1      11:00

It fails from 21:46 to 07:59, every day: more than ten hours out of every twenty-four. When we wrote and ran it for Part 9, it was evening, and it passed. Anyone running the nightly build would have seen it fail every night. This is a test bug: the test depends on the real clock.

The triage decisions go in triage.csv, so the report shows them:

test,kind,note
order-for-later::test_page_offers_slots_from_now,test bug,"Fails from 21:46 to 07:59 every day: no slots are left, and the test reads slot [0]. The page correctly says the kitchen is closed."
order-page-cypress::when the price service misbehaves says so when the price service is down,product bug,"Regression of the Part 12 fix: the page shows 'Total: ₹undefined' when the price service returns 500."

Step 5: Fix the test, and run the report again

The fix is to give the test a fixed time instead of the real clock, and to add a test for the closed kitchen, which is real behaviour worth checking:

def test_page_offers_slots_from_now(monkeypatch):
    monkeypatch.setattr(fn, "now", lambda: datetime(2026, 11, 11, 19, 10))
    html = client.get("/").text
    assert "20:00 – 20:30" in html


def test_page_says_when_the_kitchen_is_closed(monkeypatch):
    monkeypatch.setattr(fn, "now", lambda: datetime(2026, 11, 11, 23, 9))
    assert "The kitchen is closed" in client.get("/").text

6 passed, 1 warning in 0.51s

And the report, with the triage:

$ python summary.py results requirements.csv triage.csv
Tests run: 31   passed: 30   failed: 1   pass rate: 96.8%

req risk     tests  failed  status       requirement
R1  high         9       0  passing      Totals and delivery fee are right
R2  medium       4       0  passing      Promo codes work and are capped
R3  medium       4       0  passing      The basket can be edited
R4  high         2       1  FAILING      Customers are told when the price can't be worked out
R5  high         6       0  passing      Order for later offers only future slots in kitchen time
R6  high         6       0  passing      The app and the orders API agree
R7  high         0       0  NOT TESTED   Group order: colleagues add dishes and one person pays

Failures:
  [product bug] order-page-cypress::when the price service misbehaves says so when the price service is down
      Timed out retrying after 4000ms: expected '<span#total>' to contain text 'couldn\'t work o
      triage: Regression of the Part 12 fix: the page shows 'Total: ₹undefined' when the price service returns 500.

Ready to release? NO
  - R4 (high risk) failing: Customers are told when the price can't be worked out
  - R7 (high risk) not tested: Group order: colleagues add dishes and one person pays

The pass rate went up, from 93.3% to 96.8%. The answer is still no. One high-risk requirement fails, and one isn't tested at all. Notice also that R7 doesn't appear anywhere in the pass rate: a feature with no tests can't fail a test. That's why "what wasn't tested" needs its own section.

Step 6: Write the report

The script gives the facts. The report adds what they mean and what you recommend. Here's the report we'd write for this release candidate. The build name, dates and fix times are made up for the example; the results are the real ones above:

FoodNow release 26.11, test summary

Tested: build 26.11-rc2 on the staging environment, 10–11 November. Five suites, 31 automated tests, linked to 7 requirements.

Recommendation: don't release yet. Two high-risk problems:

1. When the price service fails, customers see "Total: ₹undefined" and can still place the order (R4). This is a regression: it was fixed in the previous release. Fix and retest: about half a day.
2. Group ordering (R7) has no tests. It's new in this release, and it handles payment for several people. We don't know whether it works. Testing it needs about 3 days, or the feature can be switched off for this release.

Working: totals and delivery fees, promo codes, basket editing, order for later, and the app–API contract all pass (29 tests).

Fixed during testing: one test failed every night between 21:46 and 07:59 because it used the real clock. It's been fixed, and a test for the closed kitchen was added.

If we release anyway: customers whose price check fails will see a broken total, and group orders go out untested.

Short, specific and honest about what's not known. Someone who reads only the first three lines knows what to decide.

Metrics: which ones help

Metric	Helps when...	Misleads when...
Pass rate	Shown per requirement and risk, with the failures named	Shown alone. 96.8% hid a broken price display and an untested feature.
Requirement coverage by risk	It lists the high-risk requirements with no tests	Tests are linked carelessly, so the list is wrong. Check the unlinked tests.
Open defects by severity	Each one says what it means for a user	It's only a count. Ten cosmetic bugs are not worse than one payment bug.
Flaky tests	You track tests that fail and pass without a code change	They're quietly re-run until green. Our clock test would have looked flaky.
Number of tests	Rarely	Used as a target. More tests isn't better testing.
Defect detection percentage	Measured after release: how many bugs did testing find, compared with users?	Measured before release, when users haven't found theirs yet
Code coverage	It shows code that no test runs	Read as "how well tested". It isn't.

A warning that applies to every metric: when a measure becomes a target, people improve the measure instead of the thing. This is known as Goodhart's law. Make a pass rate a target, and tests that fail get deleted or skipped. Use metrics to ask questions, not to grade people.

Common mistakes

• Leading with the pass rate. Lead with the recommendation and the high-risk problems.
• Not reporting what wasn't tested. An untested feature never shows up in a pass rate. Give it its own line.
• Reporting failures without triage. A test bug, an environment problem and a product bug look the same in a raw count.
• No link from tests to requirements. Without it, you can't say which customer needs are safe, only how many tests passed.
• Not checking the linking. List the unlinked tests. Our first run had 15.
• Tests that depend on the real clock. Ours failed for 10 hours a day. Give tests a fixed time.
• Writing for testers. The reader is deciding whether to release. Say what each problem means for a customer.

What's next

In Part 15 of Testing in Practice we look at localization testing: the FoodNow app in Hindi, German and Arabic, with Playwright switching languages and number formats, and the bugs that only appear when the text isn't in English.

ProvaLab's test management system builds this kind of report from your test runs without a script. Requirements are linked to the test cases that check them, the requirement coverage view shows which ones pass, fail or have no tests, and issues are linked to the failing results with their severity. Results from pytest, Cypress, Playwright and other frameworks arrive through ProvaLab's plugins, so five tools end up in one report. See it in the two-minute product tour.