What the gate does not run¶

Note

Written for the next release manager. It assumes none of the context of the release it came from, and it exists to answer one question: what does a green gate on this project mean, and what does it not mean?

The numbers are from the 0.10 cut, measured at d5323a6d. Treat them as a worked example. Re-measure before your own cut, because the point of this document is the method, not the figures.

The thesis, in one example¶

Under the default test matrix, HARP’s MySQL tests are not red. They are absent.

$ uv run pytest harp_apps/storage -k mysql -q --collect-only
1/101 tests collected (100 deselected)

The one test that collects compiles SQL and never opens a connection. make qa does not fail on MySQL. It silently has no MySQL, and reports success.

That is not a broken test or a failing gate. It is a whole supported backend quietly not present in the run, counted as a pass. Everything else in this document is a variation on it.

A backend that fails loudly gets fixed. A backend that is never collected gets counted as passing.

Theme one: the gates report what they decided to do, not what they observed¶

make test decided it had run the frontend. make qa-full decided the backend result was the whole result. bin/build_wheel decided an absent unzip meant absent assets. misc/cache-tests/run-tests.sh decided that writing a file was the same as reading it.

In each case the number printed at the end was true about the runner’s intentions and silent about reality. Three report success without observing the result. One reports failure without observing it. Same root, opposite signs, and the pair makes the point better than either alone: the gate does not distinguish between what it measured and what it assumed.

Theme two: a measurement can mean something other than what it appears to say¶

Five times during one release, on five different tools, output was taken at face value and meant something else.

The output	Read as	Actually
`unzip` absent, stderr discarded	frontend assets missing from the wheel	the tool was missing; the wheel held all 32
a warning attributed to an `asyncmy` test	asyncmy has a teardown defect	a garbage collector finalizer, attributed to whatever ran when the collector woke
`FakeDatetime` rejected by MariaDB	HARP cannot write transactions on MariaDB	freezegun’s test double, never constructed in production
the first screen of a failure list	the failure list	462 errors, only countable with `grep -c`
escape sequences in piped output	the CLI ignores that stdout is a pipe	`FORCE_COLOR` was set in the shell doing the measuring

Two rules follow.

Attribution is not evidence. A test runner will happily name a test that had nothing to do with the failure. Any per-test count inherits that noise.

A count omits what it did not do. 2480 passed is the same number whether the MySQL matrix ran and passed or never existed.

The discipline that separated a release scare from a footnote in the MariaDB case was not cleverness. It was reproducing the failure outside the harness that reported it, and finding that it did not reproduce.

The last row is the one to read twice, because it is the one that got past everybody. The claim was checked by a second person, who ran the same reproduction, on the same machine, in the same shell, and confirmed it. A verification that inherits the conditions of the thing it is verifying is not a verification. It has the shape of rigour and none of the content, and in this case it promoted a wrong assertion into a user-facing report before anyone looked again.

So “reproduce it outside the harness” is broader than it first sounds. Outside pytest was enough for the freezegun case. It was not enough here, because the harness was the shell. When you cannot change machines, at least change the environment: env -i, an unset of the variables you did not choose, a different user.

The same rule has a second form. Asked what the #929 migration’s downgrade() destroys, two people read the source carefully and both concluded the wrong thing, in the same direction. One query against a real 0.9.1 database settled it:

len |   content_type   | count
| application/json |    11
| http/headers     |     9
| cache/meta       |     4

Two of us reasoned from the source and got the direction wrong; one distribution from a real database settled it in a line.

Reading the code tells you what it does. Only the data tells you what is there. And two careful readers agreeing is not evidence when both are reading the same source: that is the same failure as a second reproduction in the same shell, and as three measurement setups that were all pointed away from the production path. Agreement between observers only counts when something differs between them.

The evidence¶

Deliberately skipped¶

What	Count	Reason recorded
`tests/test_uv_integration.py::TestUVIntegration`	8	none
`test_application_filtering_integration.py`	3	yes, “out of scope”
`@pytest.mark.xfail`	4	per marker
`@pytest.mark.skipif`	8	per marker
`pytest.skip()` at runtime	13	per call

TestUVIntegration is why this category is worth writing down. Eight tests whose purpose is to prove uv run harp-proxy, uvx harp-proxy and python -m harp work, disabled by a bare class-level @pytest.mark.skip with no reason string, in the release whose headline packaging change was the migration to UV. They were born disabled in the commit that introduced UV and shipped disabled through the release before.

Each method still carried @pytest.mark.skipif(not is_uv_available(), ...). Those guards became dead code the moment the blanket skip landed on top, and nothing reported that they had stopped meaning anything.

Run them and six of eight pass; the two failures were assertions about a command that had been renamed. Everything the class was written to prove worked the whole time.

This is the unknown category, and it is the one this document exists for. For two releases the honest answer to “why are these off” was that nobody knew, and nothing in the repository asked.

Deselected by marker¶

All three CI backend jobs run -m 'not subprocess', excluding 35 tests from every CI run: 27 cookiecutter integration tests and the 8 above, which are therefore excluded twice. Removing either lock alone would have changed nothing.

make qa does not filter, so the release manager is the only person who ever meets them. At the 0.10 cut all 24 failing ones shared a single cause, documented in Chores: the generated project pins harp-proxy>=X.Y.Z and PEP 440 excludes pre-releases, so nothing resolves until the version is published.

Dead at fixture setup¶

462 errors, one cause, and no MySQL or MariaDB test body had ever executed on any release.

Every MySQL parametrisation died in harp_apps/storage/conftest.py with ModuleNotFoundError: No module named 'MySQLdb' before its body ran. 474 of 499 non-passing results were that one thing.

An error count that dwarfs the failure count is almost always one problem wearing many hats. Check that before reading anything else into the numbers.

Two stacked defects, neither sufficient alone: the test URL builder rewrote a string format that testcontainers had changed in a version inside our own dependency range, so the driver never reached the URL; and the mysql dependency group was never synced, so no driver was installed either. A consequence worth noticing on its own: because the driver never reached the URL, the two driver legs of the matrix produced an identical connection string, so one of them had never been executed by anything while appearing in every test id.

Repaired locally, the suite ran and HARP worked on MySQL and MariaDB, including the full-text search path that a fix in that release had repaired and that no gate had ever executed.

Never reached¶

The frontend, whenever the backend fails. make test runs test-backend first, and a failure stops make before test-frontend. On the qa-full run no frontend suite ran at all: not “ran and failed”, not reported as skipped, absent, with nothing in the log saying so.

Every RFC 9111 cache test. make test-e2e-cache starts its origin server, then HARP fails to boot because misc/cache-tests/config.yml hardcodes a PostgreSQL URL that nothing provisions. The suite ends having executed nothing. The credentials in that file are transposed between username and password, which dates the last successful run: a typo like that cannot survive one.

In no gate at all¶

Suite	Where it lives	Where it runs
`make qa-full` (all databases)	Makefile	nowhere; `TEST_ALL_DATABASES` defaults false
`-m subprocess` (35 tests)	pytest marker	nowhere
`make test-e2e-cache` (RFC 9111)	Makefile	nowhere
`pnpm test:browser`	package.json	nowhere
visual suite `test:ui:dev`	`make test-frontend`	not in CI
`pnpm lint`	package.json	nowhere

CI runs three backend jobs, pnpm test:unit, and a documentation build. That is the entire automated gate. Everything else in the Makefile runs only when a human runs it.

Observed but not reported¶

Success reported without observing the result. make qa converts a frontend failure into the word Skipped. and a green exit, because ... && $(MAKE) test-frontend || echo "Skipped." cannot tell a deliberate skip from a failure. make test-e2e-cache writes its RFC 9111 results to a JSON file that nothing reads. The second is worse than the first: a swallowed failure at least leaves a number on screen to scroll back to, while a file nobody opens leaves nothing at all.

Failure reported without observing the result, the mirror. bin/build_wheel counted wheel assets by shelling to unzip and discarding its stderr, so a missing tool was indistinguishable from an empty archive. The build died with “frontend assets were not included in the wheel” while the wheel held all 32. A check whose failure mode impersonates the condition it guards is worse than no check, and this one fired at the exact moment someone was cutting a release.

Incapable of reporting¶

Every other entry here is a gate that failed to report something it did observe. This one is an instrument that cannot report at all, and it went unnoticed for an unknown number of releases.

make docs emits zero warnings whatever it is fed. Importing harp runs logging.config.dictConfig at module import time with disable_existing_loggers: True (harp/_logging.py, lines 71 and 100). The documentation build imports harp through the two extensions in docs/_extensions/, which disables the sphinx logger that Sphinx created moments earlier. Its handlers stay correctly attached; the logger itself is dead.

A build that reports no warnings and a build that cannot report warnings look identical from the exit code.

It was not found by reading the output, which had looked clean for as long as anyone remembered. It was found by feeding the gate a known failure: a title underline shorter than its title, an undefined :ref: label, and a :doc: reference to a document that does not exist, appended to a page that was definitely being built. The build rendered all three onto the page and reported nothing, with an empty sphinx-build -w file to match. Re-enabling the logger produced 160 warnings on the same tree, including 13 dead documentation links.

That technique is the generalisable part, and it is cheaper than any of the investigations recorded above. A gate you have never seen fail is a gate you have never tested. See issue #944.

Note also what made this one hard to see: a child logger obtained through sphinx.util.logging.getLogger still works, because propagation reaches the ancestor’s handlers without consulting the ancestor’s disabled flag. So some Sphinx output survived and some did not, and three people reported three different things about the same warning without any of them being wrong.

Unknown¶

Kept deliberately, because the temptation to drop an entry once its alarming version turns out to be wrong is itself the failure mode this document is about.

Three consoles in the codebase are constructed with force_terminal=True. Whether that produces styled output in a pipe for any command a user would pipe is not known. The commands that could be exercised were clean once FORCE_COLOR was unset, so there is no evidence of a user-visible problem, and no evidence there is not one.

Audit a migration’s diff for what it turned off¶

A large migration is where suppressions get introduced under pressure, so its diff is worth reading separately for what it silenced rather than for what it changed.

Two of the entries above turned out to come from the same commit, the migration from Poetry to UV. That was enough of a coincidence to go and read the whole diff. It had silenced five things, none of them with a recorded reason, and all five were still in place two releases later:

@pytest.mark.skip on the TestUVIntegration class, eight tests, no reason string.
-$(UV_RUN) pre-commit in preqa, whose - makes make ignore the exit status.
continue-on-error: true on the Build Documentation CI job.
continue-on-error: true on the Build Storybook CI job.
The entire test-frontend-visual CI job commented out, including the visual suite.

The pattern is ordinary and forgivable: someone migrating a build system meets a wall of unrelated failures, silences them to get the migration through, intends to come back, and does not. It cost two releases and a full day of investigation to rediscover two of them from their symptoms. Reading one diff found the other three in minutes.

Both continue-on-error jobs currently pass, so those two are dormant rather than actively hiding a failure. That is worth stating precisely: dormant is not the same as harmless, because the whole point of the flag is that the day one of them does fail, nothing will say so.

The search is cheap and bounded. In the diff of any large migration, look for skip, xfail, continue-on-error, || true, || echo, --no-verify, noqa, leading - on make recipes, new exclude entries in tool configuration, and commented-out jobs or steps.

What to do with this at your cut¶

Every entry here was invisible for the same reason: the runner prints a number that omits what it did not do. A gate that printed

35 selected, 8 unconditionally skipped, 462 fixture errors,
2 driver legs identical, frontend not reached

would have surfaced all of it without anyone writing a single new test.

Until a gate prints that by itself, do it by hand. Before the tag:

Read the suites’ summary lines, not the exit code. See Chores for the specific traps.
Count what was not run, not only what failed. Skips, deselections, and collection counts.
Ask what an error count means when it dwarfs the failure count. It is usually one fixture.
Check that each suite you believe in actually executed. A suite that cannot start reports nothing, and nothing looks like success.
Feed each gate a known failure and confirm it says so. Break something the gate is supposed to catch, watch it complain, put it back. This one line would have caught the documentation build, the unzip guard and the frontend skip, and it costs a minute per gate.
Re-measure this document. If a number here has changed, that change is itself the finding.

Read the categories in this order, because it is the order in which they hide things:

Incapable of reporting, where the instrument is silent no matter what you feed it.
Unknown, where a real defect sits looking like an accepted one.
Absent, where nothing was collected and the count says nothing.
Dead at fixture setup, where one problem wears many hats.
Never reached, which leaves no trace in the output at all.