Surefire receipt for coding agents: the tests-ok that never wrote XML

The coding agent finishes the task, writes "tests ok" in the chat, and you open the PR. CI breaks two minutes later because Maven never ran — or it ran against the wrong module. Nothing on disk proved the suite had actually executed; there was just an optimistic sentence in the transcript.

This post is about one small, annoying discipline: require a verifiable artifact before treating a task as tested. It's not about trusting the agent more or less — it's about no longer accepting text as proof of execution when the build tool already produces a file for exactly that purpose. Maven Surefire's target/surefire-reports/TEST-*.xml is that file. If it doesn't exist, is stale, or shows a failure, the task isn't done, no matter what the chat says.

All the code and numbers below come from a real run on this host: Java 25.0.4 Temurin, Maven 3.9.12, maven-surefire-plugin 3.6.0, maven-compiler-plugin 3.16.0, and JUnit BOM/Jupiter 6.1.3. I'm not using maven-compiler-plugin 4.0.0-beta-5 because it's still a beta on Central; 3.16.0 is the newest non-beta release available today.

The problem: "tests ok" is not an artifact

A coding agent can return whatever text it wants, including a cheerful summary of a suite it never ran. To make that concrete, the script below simulates exactly that behavior — it never calls Maven at all:

#!/usr/bin/env bash
# Simulates a coding agent that prints a pass in chat without running Maven.
set -euo pipefail
echo "testes ok"
exit 0

Running it at 2026-09-22T05:20:45Z:

CMD: ./agent_says_ok.sh
testes ok
EXIT=0

Clean output, EXIT=0, all the trappings of success. Except target/surefire-reports/ is still empty — no TEST-*.xml to inspect. That's the fake receipt: a free-text claim with no matching artifact on the filesystem. In practice it's not that different from a human writing "ran it here, passed" on a PR comment without pasting the log.

The gate: only accepts Surefire's report, with the official attributes

The fix isn't asking the agent to "be more careful." It's putting a verifier between the claim and the merge decision, one that only accepts a real report. When Surefire runs the test goal, it writes one TEST-*.xml per test class under target/surefire-reports/ (the default path is documented in the plugin reference and the test mojo). The root testsuite element of that XML follows the official report XSD, with required attributes tests, errors, skipped, and failures, plus an optional timestamp.

The gate reads exactly those attributes — nothing invented:

#!/usr/bin/env python3
"""Local receipt gate: only a recent Surefire TEST-*.xml with failures=0 and errors=0 counts.

Attributes read from the official Surefire XML Report Schema (testsuite):
  tests, errors, skipped, failures (required); timestamp (optional xs:dateTime).
Recency uses file mtime always, plus testsuite/@timestamp when the attribute is present.
"""
...
    for path in files:
        age = now - path.stat().st_mtime
        if age > args.max_age_seconds:
            problems.append(
                f"{path.name}: file mtime age {age:.0f}s exceeds window {args.max_age_seconds}s"
            )
            continue
        root = ET.parse(path).getroot()
        failures = int(root.attrib["failures"])
        errors = int(root.attrib["errors"])
        tests = int(root.attrib["tests"])
        skipped = int(root.attrib["skipped"])
        xml_ts = root.attrib.get("timestamp")
        if failures != 0 or errors != 0:
            problems.append(
                f"{path.name}: testsuite failures={failures} errors={errors} "
                f"tests={tests} skipped={skipped}"
            )
            continue
        print(f"OK {path.name}: tests={tests} failures={failures} errors={errors} skipped={skipped} ...")

The rule is short: no TEST-*.xml under reportsDirectory, fail. XML present but failures or errors nonzero, fail. Valid, zeroed-out XML that's older than the task window, fail. Only the intersection of "exists" + "recent" + "zeroed" clears the task.

Running the gate right after agent_says_ok.sh, with no mvn test having happened:

CMD: check_surefire_receipt.py --reports-dir target/surefire-reports --max-age-seconds 120
FAIL: no TEST-*.xml under .../target/surefire-reports
EXIT=1

The agent said green; the gate said EXIT=1. That's exactly the behavior you want: the text claim doesn't move the gate's result, because the gate never reads the chat.

Old report on disk doesn't count either

An XML file existing isn't enough — it has to belong to the current run. Two planted cases make that clear.

Case with a two-hour-old XML. I planted a valid TEST-academy.devdojo.ReceiptTest.xml (failures="0" errors="0" tests="1", no timestamp attribute) with its mtime forced to 2026-09-22 03:20:45Z, while the gate ran at 05:20:45Z — 7200 seconds apart against a 120s window:

FAIL: TEST-academy.devdojo.ReceiptTest.xml: file mtime age 7200s exceeds window 120s
EXIT=1

This is the "yesterday's report still sitting in target/" case — it passed when someone ran the suite that morning, nobody cleaned the directory, and now it sits there as a ghost receipt for any new task. The gate rejects it on the file's mtime, which is always available, and would also use the XML's testsuite/@timestamp if that attribute were present — Surefire only writes that timestamp when reportTestTimestamp is turned on, and the plugin's default is false (documented in the test mojo).

Case with a fresh XML but a real failure. I planted another TEST-*.xml with a current mtime, but failures="1":

FAIL: TEST-academy.devdojo.ReceiptTest.xml: testsuite failures=1 errors=0 tests=1 skipped=0
EXIT=1

Fresh isn't a synonym for green. You need all three at once: it exists, it's recent, and failures=0 errors=0.

What this is not

Worth separating this from a similar mechanism this blog already covered: MCP server tool gating with an execute-verify-stop policy, published on 08/31. That post refuses a tools/call inside the MCP server before a side effect happens — the server says "no" at the protocol level, with a versioned policy and an audit trail, without negotiating with the model.

None of that applies here. The agent may never have spoken MCP at all; it could have run in a plain terminal, with no structured tool-calling. The only checkpoint is afterward: does a Surefire file exist on disk, produced by Maven's test goal, with the right attributes and the right age? A gate at a tool's boundary decides what's allowed to run. This gate decides what's allowed to be called "tested" — two different problems that just happen to both be local, deterministic checks.

CI still runs Maven

The local gate doesn't replace the pipeline — it's a quick check before opening a PR, meant to catch the "forgot to run it" or "ran the wrong module" before it turns into a wasted CI cycle. The real decision still belongs to mvn test (or mvn verify) running on a clean checkout in CI, because only there can you be sure there's no planted XML or stale module cache.

A common trap here is -DskipTests or -Dmaven.test.skip=true sneaking into a profile or a build script "to speed things up." Both skip execution — the second even skips compiling the tests — and neither produces a trustworthy receipt; best case, no XML is left at all, and the gate fails the right way (case A). Worst case, the previous run's XML is still sitting in target/, and it's the age check that keeps it from being accepted as proof of the current task.

The real green case

For contrast, the actual suite: one test class, one assertEquals, nothing exotic.

package academy.devdojo;

import static org.junit.jupiter.api.Assertions.assertEquals;

import org.junit.jupiter.api.Test;

class ReceiptTest {

    @Test
    void onePlusOneIsTwo() {
        assertEquals(2, 1 + 1);
    }
}

With the POM set to maven.compiler.release=25, the JUnit 6.1.3 BOM, and Surefire 3.6.0 — plus reportTestTimestamp=true turned on explicitly, since the default is false and I wanted the optional @timestamp present in the XML for this example:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>3.6.0</version>
  <configuration>
    <reportTestTimestamp>true</reportTestTimestamp>
  </configuration>
</plugin>

Running mvn -B test: Surefire 3.6.0 uses the JUnitPlatformProvider (the official JUnit Platform example confirms that's the unified execution path today), reports Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, and finishes with BUILD SUCCESS, EXIT=0. The generated XML:

<testsuite ... name="academy.devdojo.ReceiptTest" time="0.049"
           timestamp="2026-09-22T05:20:48.951Z"
           tests="1" errors="0" skipped="0" failures="0" flakes="0">
  ...
  <testcase name="onePlusOneIsTwo" classname="academy.devdojo.ReceiptTest"
            time="0.031" timestamp="2026-09-22T05:20:48.970Z"/>
</testsuite>

And the gate, run right after:

OK TEST-academy.devdojo.ReceiptTest.xml: tests=1 failures=0 errors=0 skipped=0 mtime_age=0.1s timestamp=2026-09-22T05:20:48.951Z
EXIT=0

Four runs, four results that match what each one was supposed to prove:

The only case that passes is the only one where Maven actually ran, generated the XML on the spot, and the suite closed without a failure.

Where this fits in practice

The gate fits anywhere a task gets marked done before it turns into a PR: a local hook, a pre-commit step, or a script the agent's own flow calls before reporting success. It doesn't need new infrastructure — it's a Python script reading an XML that Maven already produces. Production observability stays exactly what it always was: CI's mvn test/mvn verify running on a clean checkout, with its archived build report as the final source of truth. The local gate just cuts down how often that source of truth has to reject a PR for a silly reason.

It's worth adopting once the team already relies on agents to open PRs and has seen at least one "chat said green, CI said red" case — writing and maintaining the gate costs little compared to a wasted CI cycle. It's not worth hard-coding into projects where no agent decides on its own when a task is done, or into builds where target/ already gets wiped before every attempt — in that case, case B doesn't even exist, and half the complexity disappears with it. And if the team ever notices PRs clearing the local gate but failing CI for a reason unrelated to tests — compilation, lint, packaging — that's a signal the receipt's scope needs to grow past Surefire, not that the gate should be removed.