pytest in practice: fixtures, parametrize, raises, markers and the plugins worth knowing
ProvaLab team · · 14 min read
Most people learn pytest from a tutorial like our first one: write a function, write assert, run pytest. That gets you a long way. Then the suite grows, and the problems that show up are rarely about assert. They come from how you use the rest of pytest: a pytest.raises that passes for the wrong reason, a setting you patch too late, a marker with a typo, a coverage number that looks fine.
This guide starts with a small suite of the kind that builds up after a busy sprint. It has six tests, 81% line coverage, and catches none of the three bugs in the code. Then we rebuild it, one pytest feature at a time, and look at what each one catches. Every command was run as written, with these versions:
| Package | Version |
|---|---|
| Python | 3.12.3 |
| pytest | 9.2.0 |
| pytest-cov | 7.1.0 (coverage 7.17.3) |
| pytest-xdist | 3.8.0 |
The code: FoodNow's bill (5 minutes)
FoodNow is the food delivery app we use as the example across our posts. This is its bill maths. Money is kept in paise (₹1 = 100 paise) as integers, so there are no float rounding surprises. The business rules:
- GST is 5% of the food.
- Delivery is ₹40, and free from ₹499.
- A coupon like
SAVE50takes ₹50 off, but only on orders of ₹199 or more. Below that it is ignored and a warning is logged.
The code has two bugs we planted on purpose. Try to spot them before the tests do.
# foodnow/bill.py
"""FoodNow bill maths. All money is in paise (Rs 1 = 100 paise)."""
import json
import logging
import os
from dataclasses import dataclass, field
from pathlib import Path
log = logging.getLogger("foodnow.bill")
GST_RATE = float(os.environ.get("FOODNOW_GST", "0.05"))
DELIVERY_FEE = 4_000 # Rs 40
FREE_DELIVERY_FROM = 49_900 # Rs 499 and above: free delivery
COUPON_MIN_ORDER = 19_900 # coupons need Rs 199 of food
@dataclass
class Line:
name: str
price: int # paise, for one
qty: int
@dataclass
class Basket:
lines: list[Line] = field(default_factory=list)
coupon_off: int = 0
def add(self, name: str, price: int, qty: int = 1) -> None:
if price <= 0:
raise ValueError("price must be above zero")
self.lines.append(Line(name, price, qty))
def apply_coupon(self, code: str) -> None:
if not code.startswith("SAVE"):
raise ValueError(f"unknown coupon {code!r}")
self.coupon_off = int(code[4:]) * 100
def subtotal(self) -> int:
return sum(line.price * line.qty for line in self.lines)
def gst(self) -> int:
return round(self.subtotal() * GST_RATE)
def delivery(self) -> int:
return 0 if self.subtotal() > FREE_DELIVERY_FROM else DELIVERY_FEE
def discount(self) -> int:
if self.coupon_off and self.subtotal() < COUPON_MIN_ORDER:
log.warning("coupon ignored: subtotal %d below %d",
self.subtotal(), COUPON_MIN_ORDER)
return 0
return min(self.coupon_off, self.subtotal())
def total(self) -> int:
return self.subtotal() + self.gst() + self.delivery() - self.discount()
def total_rupees(self) -> float:
return self.total() / 100
def write_receipt(basket: Basket, folder: Path, order_id: str) -> Path:
path = folder / f"receipt-{order_id}.json"
path.write_text(json.dumps({
"order_id": order_id,
"lines": [[l.name, l.price, l.qty] for l in basket.lines],
"total": basket.total(),
}))
return path
Set up a project around it:
mkdir pyguide && cd pyguide
mkdir foodnow tests && touch foodnow/__init__.py
# save the code above as foodnow/bill.py
python -m venv .venv && source .venv/bin/activate
pip install pytest==9.2.0 pytest-cov==7.1.0 pytest-xdist==3.8.0
Version 1: the suite after a busy sprint
Here is a suite like many real ones. Nothing in it is unusual.
# tests/test_bill_v1.py
import pytest
from foodnow.bill import Basket
def test_small_order_pays_delivery():
basket = Basket()
basket.add("Veg thali", 17_900)
assert basket.delivery() == 4_000
def test_big_order_free_delivery():
basket = Basket()
basket.add("Veg thali", 17_900, qty=3)
assert basket.delivery() == 0
def test_total_with_gst_and_delivery():
basket = Basket()
basket.add("Veg thali", 17_900)
basket.add("Butter naan", 6_500, qty=2)
assert basket.total_rupees() == pytest.approx(364.45)
def test_coupon_takes_money_off():
basket = Basket()
basket.add("Veg thali", 17_900, qty=2)
basket.apply_coupon("SAVE50")
assert basket.discount() == 5_000
def test_negative_quantity_is_refused():
basket = Basket()
with pytest.raises(ValueError):
basket.apply_coupon("SAVE")
basket.add("Veg thali", 17_900, qty=-1)
def test_gst_rate_comes_from_settings(monkeypatch):
monkeypatch.setenv("FOODNOW_GST", "0.12")
basket = Basket()
basket.add("Veg thali", 10_000)
assert basket.gst() == 1_200
Run it with plain pytest:
$ pytest -q tests/test_bill_v1.py
ImportError while importing test module '.../tests/test_bill_v1.py'.
E ModuleNotFoundError: No module named 'foodnow'
1 error in 0.09s
That error is the first lesson. python -m pytest adds the current folder to the import path; plain pytest does not. Rather than remembering which command to type, tell pytest where your code is in a pytest.ini at the project root:
# pytest.ini
[pytest]
testpaths = tests
pythonpath = .
Now it runs:
$ pytest -q tests/test_bill_v1.py
=================================== FAILURES ===================================
______________________ test_gst_rate_comes_from_settings _______________________
def test_gst_rate_comes_from_settings(monkeypatch):
monkeypatch.setenv("FOODNOW_GST", "0.12")
basket = Basket()
basket.add("Veg thali", 10_000)
> assert basket.gst() == 1_200
E AssertionError: assert 500 == 1200
1 failed, 5 passed in 0.03s
Five pass, one fails, and with coverage turned on (more on that below) the suite reaches 81% of the lines. The team's verdict would be "one flaky settings test, otherwise fine". It is not fine. Three things are wrong with this suite.
The raises test passes for the wrong reason. A line copied from the coupon test, basket.apply_coupon("SAVE"), raises ValueError first (the code does int("")), so basket.add(..., qty=-1) never runs. pytest sees a ValueError and is happy. The basket in fact accepts a quantity of −1.
The settings test can never pass. GST_RATE is read once, when foodnow.bill is imported. By the time the test sets the environment variable, the module has already read it. The product works as written; the test is wrong.
The boundary is never tested. The two delivery tests use ₹179 and ₹537. The rule changes at ₹499, and that is where the bug is.
Step 1: Parametrize the boundary, with names (5 minutes)
Boundary tests are where @pytest.mark.parametrize pays for itself: one test body, one row per value either side of the line. Give each row an id, so a failure tells you which row in words:
# tests/test_bill_v2.py
import json
import logging
import pytest
import foodnow.bill as bill
THALI = 17_900 # Rs 179
NAAN = 6_500 # Rs 65
@pytest.mark.parametrize(
"subtotal, fee",
[
pytest.param(49_800, 4_000, id="Rs 498 pays"),
pytest.param(49_900, 0, id="Rs 499 free"),
pytest.param(50_000, 0, id="Rs 500 free"),
],
)
def test_delivery_fee_boundary(make_basket, subtotal, fee):
basket = make_basket(("Party platter", subtotal, 1))
assert basket.delivery() == fee
make_basket is a fixture, which we write in step 3. The failure reads like a bug report:
FAILED tests/test_bill_v2.py::test_delivery_fee_boundary[Rs 499 free] - Asser...
> assert basket.delivery() == fee
E AssertionError: assert 4000 == 0
E + where 4000 = delivery()
A ₹499 order is charged ₹40 for delivery. The code says > where the rule says "from ₹499", which is >=. That is planted bug number one.
Step 2: pytest.raises with match, and nothing else in the block (5 minutes)
Two rules make pytest.raises trustworthy. Put only the one call you expect to fail inside the with block. And always pass match=, a regular expression the error message must contain, so the test fails if the right type of error is raised for the wrong reason.
def test_negative_quantity_is_refused(make_basket):
basket = make_basket()
with pytest.raises(ValueError, match="quantity"):
basket.add("Veg thali", THALI, qty=-1)
@pytest.mark.parametrize("code", ["SAVE", "SAVEFIFTY", "FREE50"])
def test_bad_coupon_is_refused(make_basket, code):
basket = make_basket(("Veg thali", THALI, 2))
with pytest.raises(ValueError, match="coupon"):
basket.apply_coupon(code)
> with pytest.raises(ValueError, match="quantity"):
E Failed: DID NOT RAISE ValueError
Planted bug number two: a negative quantity goes straight into the basket, and the total goes down.
The coupon test found something we did not plant:
FAILED tests/test_bill_v2.py::test_bad_coupon_is_refused[SAVE]
E AssertionError: Regex pattern did not match.
E Expected regex: 'coupon'
E Actual message: "invalid literal for int() with base 10: ''"
FAILED tests/test_bill_v2.py::test_bad_coupon_is_refused[SAVEFIFTY]
E Expected regex: 'coupon'
E Actual message: "invalid literal for int() with base 10: 'FIFTY'"
FREE50 gets our "unknown coupon" message. SAVE and SAVEFIFTY start with SAVE, get past the check, and crash in int(). If the app shows that message to a customer, they see Python internals. A bare pytest.raises(ValueError) passes all three and never tells you. This is the same blind spot that hid the quantity bug in version 1.
Step 3: Fixtures: a factory, a teardown, and conftest.py (5 minutes)
Version 1 starts every test with basket = Basket() and a few add calls. A factory fixture returns a function, so each test still says what is in its basket, in one line:
# tests/conftest.py
import pytest
from foodnow.bill import Basket
@pytest.fixture
def make_basket():
"""Factory: make_basket(("Veg thali", 17_900, 2), ...) -> Basket."""
def _make(*lines):
basket = Basket()
for name, price, qty in lines:
basket.add(name, price, qty)
return basket
return _make
@pytest.fixture
def receipts_dir(tmp_path):
folder = tmp_path / "receipts"
folder.mkdir()
yield folder
print(f"\nteardown: {len(list(folder.iterdir()))} receipt(s) left in {folder.name}/")
Fixtures in conftest.py are available to every test in that folder and below, without an import. Don't import from conftest.py yourself; if tests share constants, put them in a normal module or keep them in the test file, as we did with THALI and NAAN.
receipts_dir shows the other half of fixtures: everything after yield is teardown, and it runs even when the test fails. It builds on tmp_path, a fixture pytest provides: a fresh, empty folder for each test, so no test ever writes into your project or sees another test's files.
def test_receipt_is_written(make_basket, receipts_dir):
basket = make_basket(("Veg thali", THALI, 1))
path = bill.write_receipt(basket, receipts_dir, "A1001")
assert json.loads(path.read_text())["total"] == basket.total()
When you're not sure which fixture runs when, --setup-show prints the order. S is session scope, F is function scope:
$ pytest -q --setup-show -s -k receipt
SETUP S tmp_path_factory
SETUP F make_basket
SETUP F tmp_path (fixtures used: tmp_path_factory)
SETUP F receipts_dir (fixtures used: tmp_path)
tests/test_bill_v2.py::test_receipt_is_written (fixtures used: make_basket, receipts_dir, request, tmp_path, tmp_path_factory) .
teardown: 1 receipt(s) left in receipts/
TEARDOWN F receipts_dir
TEARDOWN F tmp_path
TEARDOWN F make_basket
TEARDOWN S tmp_path_factory
Setup runs in dependency order, and teardown in reverse.
Step 4: monkeypatch and caplog: patch the right thing, check the log (5 minutes)
The version 1 settings test set an environment variable the code had already read. monkeypatch can set an attribute instead, and puts the original back after the test:
def test_gst_rate_comes_from_settings(make_basket, monkeypatch):
monkeypatch.setattr(bill, "GST_RATE", 0.12)
basket = make_basket(("Veg thali", 10_000, 1))
assert basket.gst() == 1_200
The general rule: patch the name where the code looks it up, at the time it looks it up. Here that is the module attribute bill.GST_RATE, not the environment. (If the rate must change without a restart, that is a product change: read the variable inside gst(). Then monkeypatch.setenv works.)
caplog captures log records, so you can test the "ignored, with a warning" rule instead of only the "ignored" half:
def test_small_order_coupon_is_ignored_with_a_warning(make_basket, caplog):
basket = make_basket(("Butter naan", NAAN, 1))
basket.apply_coupon("SAVE50")
with caplog.at_level(logging.WARNING, logger="foodnow.bill"):
assert basket.discount() == 0
assert "coupon ignored" in caplog.text
A word on pytest.approx and money
Version 1 compared rupees as floats with pytest.approx. It worked there, but look at what approx allows by default, a relative tolerance of one in a million:
>>> 0.1 + 0.2 == 0.3
False
>>> 100_000.09 == pytest.approx(100_000.00)
True
>>> 350.01 == pytest.approx(350.00)
False
On a ₹1,00,000 catering order, approx accepts a total that is 9 paise off. Use approx for real measurements (an average rating, a percentage, a time). For money, keep integers in paise and compare with ==, as test_total_in_paise does in the full file.
Step 5: Markers, and why --strict-markers belongs in every project (3 minutes)
Markers tag tests so you can choose what to run. We register slow in pytest.ini and mark the one heavy test. It checks that no coupon value makes a total negative, over about 118,000 baskets:
@pytest.mark.slwo
def test_total_never_negative_for_any_coupon(make_basket):
for rupees in range(1, 2_000):
for qty in range(1, 60):
basket = make_basket(("Butter naan", NAAN, qty))
basket.coupon_off = rupees * 100
assert basket.total() >= 0
Spot the typo: slwo. The fast lane, -m "not slow", still runs the test, because nothing is marked slow:
$ pytest -q -m "not slow" --durations=2
PytestUnknownMarkWarning: Unknown pytest.mark.slwo - is this a typo?
0.57s call tests/test_bill_v2.py::test_total_never_negative_for_any_coupon
4 failed, 9 passed, 1 warning in 0.62s
Nothing deselected, and a warning most people scroll past. On a real suite, that is how the "quick" job slowly stops being quick. Make unknown markers an error:
# pytest.ini
[pytest]
testpaths = tests
pythonpath = .
addopts = --strict-markers
markers =
slow: takes more than a second; skip with -m "not slow"
$ pytest -q -m "not slow"
ERROR tests/test_bill_v2.py - Failed: 'slwo' not found in `markers` configuration option
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
Step 6: Fix the code, and rerun only what failed (3 minutes)
Three small changes to bill.py:
def add(self, name: str, price: int, qty: int = 1) -> None:
if price <= 0:
raise ValueError("price must be above zero")
+ if qty < 1:
+ raise ValueError("quantity must be 1 or more")
def apply_coupon(self, code: str) -> None:
- if not code.startswith("SAVE"):
+ rupees = code.removeprefix("SAVE")
+ if rupees == code or not rupees.isdigit():
raise ValueError(f"unknown coupon {code!r}")
- self.coupon_off = int(code[4:]) * 100
+ self.coupon_off = int(rupees) * 100
def delivery(self) -> int:
- return 0 if self.subtotal() > FREE_DELIVERY_FROM else DELIVERY_FEE
+ return 0 if self.subtotal() >= FREE_DELIVERY_FROM else DELIVERY_FEE
With the typo fixed to @pytest.mark.slow, --lf (last failed) runs only the tests that failed last time. On a big suite, that is the difference between seconds and minutes while you fix things:
$ pytest -q --lf # before the fix
4 failed, 9 deselected in 0.03s
$ pytest -q --lf # after the fix
4 passed, 9 deselected in 0.01s
$ pytest -q --lf # nothing failed last time, so it runs everything
13 passed in 0.59s
$ pytest -q -m "not slow"
12 passed, 1 deselected in 0.02s
Step 7: Coverage shows what you didn't run, not what is right (3 minutes)
pytest-cov measures which lines and branches ran. --cov-branch also counts each side of an if:
$ pytest -q --cov=foodnow --cov-branch --cov-report=term-missing tests/test_bill_v1.py # buggy code
Name Stmts Miss Branch BrPart Cover Missing
foodnow/bill.py 46 7 6 3 81% 30, 35, 49-51, 62-68
$ pytest -q --cov=foodnow --cov-branch --cov-report=term-missing tests/test_bill_v2.py # fixed code
foodnow/bill.py 49 2 8 1 95% 30, 61
Look at what the version 1 report does not list: the delivery() line. It ran, so it counts as covered, and it held a bug the whole time. Coverage only tells you a line ran, not that any test checked its answer.
The version 2 report is still useful: line 30 is the "price must be above zero" check, which no test reaches. One more parametrized test closes it:
@pytest.mark.parametrize("price", [0, -100])
def test_zero_or_negative_price_is_refused(make_basket, price):
with pytest.raises(ValueError, match="price"):
make_basket(("Veg thali", price, 1))
foodnow/bill.py 49 1 8 0 98% 61
15 passed in 2.85s
The one line left is total_rupees(), which only version 1 used. Two side notes from the same runs: coverage slowed the slow test from about 0.55 s to about 2.7 s, because tracing every line costs time; and the report reads 95%, not 100%, and that is fine. Chase the lines that matter, not the number.
Step 8: Parallel runs: measure before you switch them on (2 minutes)
pytest-xdist runs tests across processes with -n. It helps big, slow suites. On this one, it doesn't:
serial 0.82 s 0.82 s 0.86 s
-n 4 1.39 s 1.45 s 1.45 s
Starting four worker processes costs more than the whole 15-test run. The Playwright guide shows the other side: browser tests that went from 27.1 s to 11.6 s with -n 4. Measure on your suite. And before you turn on -n, make sure every test passes alone and in any order; the flaky tests post shows what happens when they don't.
The commands worth remembering
| Command | What it does |
|---|---|
pytest -x |
Stop at the first failure |
pytest --lf |
Rerun only last run's failures |
pytest -k "boundary or coupon" |
Run tests whose names match |
pytest -m "not slow" |
Skip tests marked slow |
pytest --setup-show |
Show fixture setup and teardown order |
pytest --durations=5 |
List the five slowest tests |
pytest --cov=pkg --cov-branch --cov-report=term-missing |
Coverage, with the missing lines |
pytest -n 4 |
Run on four processes (pytest-xdist) |
Habits worth keeping
- Test both sides of every boundary, with
parametrizeand anidper row that reads like the rule. pytest.raisesgets one call and amatch=. Withoutmatch, any error of that type passes.- Patch where the code looks, when it looks. If the value is read at import, set the attribute.
- Keep money in integers.
approxis for measurements. - Turn on
--strict-markerson day one, inpytest.ini, so a typo fails instead of warning. - Treat coverage as a list of untested lines, not as a score. The buggy line in this guide was covered.
What's next
This suite gives one answer on one laptop. On a team, you want every run's results kept, each test linked to the requirement it checks, and the history of a test across runs, so "it failed once last week" is something you can look up. ProvaLab's pytest plugin sends each test's result to a test run in the ProvaLab test management system, and a marker such as @pytest.mark.tms_test_case(id=123) links a test to its test case, so requirement coverage comes from real runs rather than a spreadsheet. See it in the two-minute product tour.
New to pytest? Start with Your first automated test in 15 minutes. For the same ideas applied to browser tests, read Playwright with Python in practice.
See ProvaLab working on a real app
Your coding agent runs the tests, reads the failures and fixes them.