pytest-testmon: Run Only the Tests Affected by Your Code Change

pytest-testmon is a pytest plugin that runs only the tests your change can affect. It watches which lines of code each test executes, using coverage.py, and stores a checksum of every block of code a test touched. On the next run it compares those checksums with the code on disk and runs only the tests whose blocks changed, plus anything that failed last time, so it also does the job of pytest --lf. The general technique is called test impact analysis, and testmon is a free, open-source way to get it in Python. Nothing about it depends on git: it compares against what it saw on the last run, committed or not.

I added it to my marketplace app’s Python tests this week, and the reason wasn’t speed for its own sake.

Why I added it

#

A feature request is usually small: one screen, one endpoint, one rule that should work differently. Running the whole test suite made every one of them bigger than it was. The suite is 2,248 Django tests, about three and a half minutes with four processes and 11 in one, and every edit ran all of it, though almost all of it checks things the request never touched. When something unrelated failed, or the run simply took its time, the feature waited. It was being held up by circumstances outside the feature request, and the thing holding it up was my own test suite.

The alternative wasn’t better. Skip the full run, guess which test file covers the change, and now and then guess wrong. That’s quick, but it isn’t precise, and an untested change I ship is a feature request I’ll be doing twice.

What I want is to stay agile without getting sloppy: take a request, make the change, know it works, ship it, and move to the next one. For that, the tests I wait for should be exactly the tests the change can reach, no more and no fewer. testmon gives me that set from a record instead of a guess. The full suite still runs, in CI, where it doesn’t hold up whatever I’m working on next.

How it decides what to run

#

testmon splits each Python file into blocks: the module’s top level, each class, each function, with the bodies of nested functions cut out of the block around them. It stores a checksum of each block, never the code itself. While a test runs, coverage.py records the lines it executes, and testmon notes which blocks those lines belong to. There’s no static analysis and no guessing from imports. A test depends on exactly the blocks it ran.

Take a module like this:

# pricing.py
def subtotal(items):
    return sum(item.price * item.quantity for item in items)

def discount(total, code):
    if code == "WELCOME10":
        return total * 0.9
    return total

A test that only calls subtotal() depends on the module’s top level and on subtotal’s body. Rewrite discount() however you like and testmon won’t rerun that test, because it never executed a line of discount. Change one character inside subtotal and it runs again, even if the change can’t alter the result. testmon doesn’t try to be clever about side effects: if a block you executed changed, you run.

It also tracks what’s around the code. The record includes the Python version, the installed packages and their versions, and environment variables such as DJANGO_SETTINGS_MODULE. Upgrade Django and everything that touched Django is affected again, which in a Django app is everything.

What it can’t see

#

testmon tracks Python that a test executes. It doesn’t track the other files your tests read: templates, JSON, text, anything else on disk. It doesn’t track services reached over the network. And it can’t pin anything that runs once before the tests start on a particular test. In a Django project that means:

• Migrations. They run once, when the test database is created, so no test’s record includes them. The wrapper below rebuilds the test database when there’s a new one.
• Settings and start-up code. They run while pytest starts up, outside any one test.
• Templates and data files. A template isn’t Python, so editing one reruns nothing that renders it.

My suite has a few more of its own. Some tests read a JSON file of fixtures that the mobile app’s tests share. Some check the built web bundle, and some check the output of a static site generated in the same repository. Change any of those and testmon will happily tell you nothing needs to run. That’s why the how-to below wraps testmon in a script that notices these files and runs everything when one has changed.

How to add it to a Django project

#

These steps assume pytest and pytest-django, and uv for the dependencies. With pip, swap uv add --dev for pip install and drop uv run.

1. Install it.

uv add --dev pytest-testmon

2. Make sure pytest can run the suite. testmon is a pytest plugin, so it does nothing under python manage.py test. pytest-django needs the settings module and the file patterns your tests use:

# pytest.ini
[pytest]
DJANGO_SETTINGS_MODULE = myproject.settings
python_files = tests.py test_*.py *_tests.py
python_classes = Test* *Tests *TestCase

If the suite has only ever run under Django’s runner, run it under pytest once and fix whatever differs before adding testmon. Otherwise you’ll be debugging two things at once.

3. Keep the record out of git. testmon writes .testmondata in the pytest root directory. It’s a SQLite database, so you’ll also see -shm and -wal files next to it, and the script below adds a stamp file. One pattern covers all of them:

# .gitignore
.testmondata*

4. Record the whole suite once. testmon can’t select anything until it has seen every test run:

uv run pytest --testmon

In a fresh checkout that first run is the whole suite in one process. Here it takes about 12 minutes for 2,248 tests. From then on, the same command runs only the affected tests. With nothing changed, it runs nothing.

Two catches. testmon needs pytest’s cache plugin: my first attempt ran with -p no:cacheprovider and crashed with KeyError: 'lf'. And on Python 3.14 or later, set COVERAGE_CORE=ctrace before you record. “The bug that looked like a green run”, further down, explains why. It’s the most important line in the script below.

5. Wrap it for the files it can’t see. This is the part that makes it safe to use every day. The script keeps a stamp file from the start of the last run. If any file testmon can’t trace is newer than the stamp, it runs everything and records afresh. A new migration only rebuilds the test database. Otherwise it lets testmon choose. A cut-down example:

#!/usr/bin/env bash
# scripts/test-changed.sh: the tests a change can affect, or all when in doubt.
#   scripts/test-changed.sh            # affected tests
#   scripts/test-changed.sh --all      # everything, re-recorded
#   scripts/test-changed.sh -k orders  # anything else goes to pytest
set -euo pipefail
cd "$(dirname "$0")/.."

# coverage's C tracer: from Python 3.14 coverage defaults to sys.monitoring,
# which reports a line only the first time a process runs it, so testmon
# would credit each line to one test. Changing this means --all.
export COVERAGE_CORE=ctrace

STAMP=.testmondata.started
full=false
rebuild=false
if [ "${1:-}" = "--all" ]; then
    full=true; shift
elif [ ! -f .testmondata ] || [ ! -f "$STAMP" ]; then
    echo "No record yet: running everything once."
    full=true
else
    # Changed since the last run started, and not traceable per test:
    # anything that isn't Python, plus settings and conftest.
    newer="$(find myapp myproject fixtures pytest.ini \
        -type f -newer "$STAMP" ! -path '*/migrations/*' \
        \( ! -name '*.py' -o -path 'myproject/settings*' -o -name 'conftest.py' \) \
        ! -path '*/__pycache__/*' | head -5)"
    if [ -n "$newer" ]; then
        echo "Changed, and not traceable per test:"; printf '  %s\n' $newer
        full=true
    fi
    # A new migration needs a fresh test database, not a full run.
    [ -n "$(find myapp -path '*/migrations/*.py' -newer "$STAMP" | head -1)" ] && rebuild=true
fi

# Stamp the start, so an edit made while the tests run counts next time.
started="$(mktemp .testmondata.run.XXXXXX)"
args=(--testmon --reuse-db)
# --create-db alone deletes the database when the run ends; with
# --reuse-db as well, the rebuilt one is kept for next time.
[ "$rebuild" = true ] && args=(--testmon --reuse-db --create-db)
[ "$full" = true ] && args=(--testmon-noselect --reuse-db --create-db)

status=0
uv run pytest "${args[@]}" -q "$@" || status=$?
# 0: passed, or nothing selected. 1: some failed (still recorded).
# 5: nothing collected at all.
case "$status" in 0|1|5) mv "$started" "$STAMP" ;; *) rm -f "$started" ;; esac
[ "$status" = 5 ] && exit 0
exit "$status"

Two details in there are worth explaining. --reuse-db is pytest-django’s flag for keeping the test database between runs, which matters once a run takes seconds: creating it again each time can take longer than the tests themselves. A full run, or one after a new migration, adds --create-db so the database has the current schema. Keep --reuse-db next to it: --create-db on its own makes pytest-django delete the database when the run ends, so the next run would build it all over again. And when testmon deselects everything, the run exits 0, so “nothing changed” counts as a pass. Exit code 5 is pytest’s “no tests collected”, which the script treats as fine too.

Notice what’s not in the list: pyproject.toml and the lock file. My first version ran everything whenever the lock file changed, which made the slow fallback fire on most days and the whole thing worse than before. testmon already tracks installed package versions itself, so it doesn’t need the help.

6. Give it its own database and cache. If other things run tests on the same machine, like a second terminal, a coding agent or manage.py test, point the script at its own test database and its own cache number so a kept database never meets someone else’s run:

export DATABASE_URL="postgresql://myapp:myapp@localhost:5432/myapp_changed"
export REDIS_URL="redis://localhost:6379/8"

7. Empty that cache before each run. I learnt this one the hard way. Subset runs started failing with KeyErrors that full runs didn’t. Some category labels are cached with no expiry, and a test database kept from one run met a cache still warm from an earlier one. Clearing the cache at the top of the script fixed it:

uv run python -c "import os, redis; redis.Redis.from_url(os.environ['REDIS_URL']).flushdb()"

That’s testmon doing what its docs say it will: running a different subset each time exposes tests that only passed because of what ran before them. The fix belongs in the test setup, not in testmon.

8. Know the flags for when selection is wrong.

• --testmon-noselect runs every test but puts the likely failures first. testmon switches to it by itself when you also pass -k, -m, --lf or a test’s node ID.
• --testmon-forceselect runs only affected tests that also match your -k or -m.
• --testmon-nocollect uses the existing record but doesn’t add to it. It’s forced under a debugger or when coverage is already running.
• --no-testmon turns it off for one run, for when it’s in addopts.
• --testmon-env=<name> keeps separate records in one file, for a suite you run under two settings modules.

If the project has a JavaScript side, Jest has its own version built in. jest --changedSince=main runs the tests that import anything changed since main, following imports rather than coverage. I added it to the app’s package.json as npm run test:changed so both halves have the same habit. An edit to one utility module selected 7 of the app’s 121 test files.

What it saves

#

I timed the wrapper once it was in place, start-up included, after fixing the bug in the next section:

For an everyday edit that’s about 25 times faster than the parallel suite, and when nothing has changed it’s about two hundred times faster. The record says the same thing more broadly. Of the 307 non-test Python files the suite runs, a change to the median one reaches 22 tests, about 8 seconds of test time. A few core modules sit at the other end: almost every test runs them, so a change there reaches around 1,500 tests, and testmon reruns most of the suite. That’s the right answer, not a failure. Those figures are upper bounds per file: testmon works per block, so changing one function selects only the tests that ran that function.

The cost shows up when the wrapper falls back. A full run is the same single-process recording as the first one, about 12 minutes, which is slower than the parallel suite it replaces. That’s why the list of files that trigger it should be as short as it can honestly be.

The bug that looked like a green run

#

My first numbers were better than these: one edit, 6 tests, 3.3 seconds. They were wrong, and nothing on screen said so.

From Python 3.14, coverage.py uses a new core by default, built on the interpreter’s sys.monitoring. That core reports a line only the first time a process runs it. testmon builds its record of which test ran which line from coverage, so each line was credited to the first test that happened to run it, and to no other. Edit a line that eight tests share, and testmon selects one of them.

It was caught on a second, smaller project of mine that copied this setup. Part of adopting it there was breaking a line on purpose and comparing the results, and the numbers didn’t match. Back in this suite I ran the same check. I broke one line in the home-screen banner code, the title of a slide. Plain pytest reported 8 failing tests. testmon selected 4. The other 4 broken tests never ran, and nothing in the output said they’d been left out.

The fix is one line, export COVERAGE_CORE=ctrace, which puts coverage back on its classic C tracer. A different tracer means a different record, so I re-recorded: 12 minutes 3 seconds for 2,248 tests. Afterwards, the same kind of breaking edit ran all 11 tests it broke.

That’s the lesson I’d pass on before anything else here. A test selector that selects too little looks exactly like success: fewer tests, a faster run, all green. So when you set one up, break something shared on purpose, run the plain suite and the selector, and check they fail the same tests. Then do it again whenever Python or coverage.py changes version.

Where a coding agent comes in

#

I don’t make every change in that repository myself any more. A coding agent, Claude Code in my case, makes a lot of the small ones, and a good agent runs the tests after each edit. Faced with an 11-minute suite, it has the same two bad options I had: wait, or guess which test file matters.

So the repository’s agent instructions now say the wrapper script is the default way to run tests while working, with --all for a full run. The agent gets the same precise set of tests I do, after every edit, in seconds. It’s also why step 6 gives the script its own database and cache: on a busy day there’s more than one session running tests on the same machine, and a kept database must never meet somebody else’s run.

None of this needs anything AI-specific from testmon. It’s the same record and the same selection. It just matters more when whatever is making the changes makes a great many of them.

Why CI got something coarser

#

testmon has no CI mode. It behaves the same on a runner as on a laptop, and its author writes that sharing or caching .testmondata across CI runners is far less effective than their hosted service, testmon.net. A cached record would also be blind in all the ways above, and my CI runs the suite with Django’s runner in four processes, so I left testmon out.

What CI got instead is the same idea at file level. The checks workflow now starts with a small job that diffs the push and decides which half of the repository it touched: the Python API, the mobile app, both or neither. Before this, CI ran both the Python and the app jobs on every push, docs-only ones included. Now a docs-only push runs nothing, and a push to the app alone skips the Django suite. I dry-ran the job against five real commits: the docs-only one ran nothing, the app-only one skipped the Python job, and the backend-only one skipped the app job. A cut-down example:

jobs:
  changes:
    runs-on: ubuntu-latest
    outputs:
      python: ${{ steps.diff.outputs.python }}
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - id: diff
        env:
          BEFORE: ${{ github.event.before }}
          HEAD: ${{ github.sha }}
        run: |
          FILES="$(git diff --name-only "$BEFORE" "$HEAD")"
          # Python's inputs include the non-Python files its tests read.
          PY='^(myapp/|myproject/|fixtures/shared\.json$|web/dist/|pyproject\.toml$|uv\.lock$)'
          if printf '%s\n' "$FILES" | grep -qE "$PY"; then
            echo "python=true" >> "$GITHUB_OUTPUT"
          else
            echo "python=false" >> "$GITHUB_OUTPUT"
          fi

  python:
    needs: changes
    if: needs.changes.outputs.python == 'true'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: uv run python manage.py test --parallel 4 --noinput

The real one also runs everything when there’s no range to compare: a new branch, a force push or a manual run.

The blind spot is the same one in a different form. testmon can’t see a JSON file, and neither can a path filter unless you list it. So the pattern names the shared fixtures, the built bundle and the generated site explicitly, with a comment asking whoever edits it to keep the list honest. The file-level version is cruder, but it fails in the safe direction: if anything on the Python side changed, the whole suite runs.

Is it worth it?

#

For a suite that takes minutes, yes. The point was never the minutes themselves: it’s that a change gets judged by the tests it can affect, so a small request stays small. It’s one dependency, one flag and one environment variable, it changes nothing about how the tests are written, and with the C tracer set, the worst it does is run too much after a change to something everything imports. Without it, on Python 3.14 or later, it can run too little and stay green, which is why the break-it-on-purpose check matters. The real cost is what it can’t see, and that’s what the wrapper script is for. Even so, the laptop loop is where testmon belongs. CI still runs the whole suite whenever the Python side changes, and that’s the run I trust before a deploy.