mvn test in the coding agent: the *IT that Failsafe never ran

The coding agent finishes its task, runs mvn test, gets exit 0 and writes "all tests passed". For the unit tests, that's true. But the module also has an *IT.java class, and that class never ran. It didn't fail, and it wasn't skipped with a warning. Maven never got to the phase where it runs.

This post covers why that happens and what you'll find in target/ afterwards, plus what you won't find. You'll also get an agent gate that won't accept a half-green build. Everything here ran on a minimal module: no container, no framework.

Where this touches what we've already published

We've already written about using the Surefire XML as a receipt that the tests actually ran, about a diff gate that catches weakened assertions and @Disabled, and about the repository gates to set up before the first PR. Today's case is a different kind of false green. The Surefire XML is there, it's green, and it's legitimate. What's missing is the receipt from a second plugin, and nobody checked for it.

Environment

• Temurin JDK 25.0.4, Maven 3.9.12, maven.compiler.release=25, no --enable-preview
• maven-surefire-plugin 3.6.0, maven-failsafe-plugin 3.6.0
• org.junit.jupiter:junit-jupiter 6.1.3
• jar packaging, no Spring, no Testcontainers, no HTTP

Two plugins, two naming patterns, different phases

Surefire and Failsafe come from the same plugin family, but each one picks up classes by a different naming pattern.

Surefire 3.6.0 default includes (Surefire inclusion docs):

• **/Test*.java
• **/*Test.java
• **/*Tests.java
• **/*TestCase.java

Failsafe 3.6.0 default includes (Failsafe inclusion docs):

• **/IT*.java
• **/*IT.java
• **/*ITCase.java

IntegrationProbeIT.java doesn't match any of the Surefire patterns. Surefire 3.x still doesn't include *IT.java by default.

The two plugins also run in different phases of the Maven lifecycle. surefire:test binds to the test phase. failsafe:integration-test binds to integration-test, and failsafe:verify binds to verify. The order is:

... test → package → pre-integration-test → integration-test → post-integration-test → verify

mvn test stops at test. Everything after it gets left out, Failsafe included.

There's one more catch, and it gets a lot of people. Even with mvn verify, Failsafe only runs if the POM declares it. In Maven 3.9.12 the Super POM doesn't include Failsafe, and the default bindings for jar packaging only bind Surefire (at version 3.2.5) to the test phase. Nothing is bound to integration-test or verify. That's why the example POM pins both versions and declares the Failsafe executions:

<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>devdojo</groupId>
  <artifactId>failsafe-it-probe</artifactId>
  <version>1.0.0-SNAPSHOT</version>
  <packaging>jar</packaging>
  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <maven.compiler.release>25</maven.compiler.release>
  </properties>
  <dependencies>
    <dependency>
      <groupId>org.junit.jupiter</groupId>
      <artifactId>junit-jupiter</artifactId>
      <version>6.1.3</version>
      <scope>test</scope>
    </dependency>
  </dependencies>
  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>3.6.0</version>
      </plugin>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-failsafe-plugin</artifactId>
        <version>3.6.0</version>
        <executions>
          <execution>
            <goals>
              <goal>integration-test</goal>
              <goal>verify</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
</project>

The goals don't declare a <phase> because they inherit their default phases (integration-test and verify). Without this block, mvn verify walks right past the *IT, just like mvn test does.

Both tests are small on purpose. Here's the unit test:

package devdojo;

import static org.junit.jupiter.api.Assertions.assertEquals;

import org.junit.jupiter.api.Test;

class UnitProbeTest {

  @Test
  void unitProbeOnePlusOneEqualsTwo() {
    assertEquals(2, 1 + 1, "unitProbeUniqueString");
  }
}

And here's the integration test. It writes a temp file and reads it back, with no network and no server:

package devdojo;

import static org.junit.jupiter.api.Assertions.assertEquals;

import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import org.junit.jupiter.api.Test;

class IntegrationProbeIT {

  @Test
  void integrationProbeTempFileRoundTrip() throws Exception {
    Path tmp = Files.createTempFile("failsafe-it-probe-", ".txt");
    Files.writeString(tmp, "integrationProbeUniqueString", StandardCharsets.UTF_8);
    assertEquals(
        "integrationProbeUniqueString",
        Files.readString(tmp, StandardCharsets.UTF_8));
  }
}

Failsafe doesn't need a servlet container. The Jetty example in the docs is optional. Any class whose name matches the pattern and that has a JUnit test will do.

What shows up in target/ after each command

We ran both commands on the same module and then looked at the report directories:

After mvn -q test:

$ ls target/failsafe-reports
ls: cannot access 'target/failsafe-reports': No such file or directory

The Failsafe directory isn't empty. It doesn't exist at all. In quiet mode, the only output from mvn -q test was a few sun.misc.Unsafe warnings from the Guava that Maven itself loads. Nothing in there suggests that anything is missing.

After mvn -q clean verify, each plugin records only its own tests:

<!-- target/surefire-reports/TEST-devdojo.UnitProbeTest.xml -->
<testcase name="unitProbeOnePlusOneEqualsTwo" classname="devdojo.UnitProbeTest" time="0.035"/>

<!-- target/failsafe-reports/TEST-devdojo.IntegrationProbeIT.xml -->
<testcase name="integrationProbeTempFileRoundTrip" classname="devdojo.IntegrationProbeIT" time="0.05"/>

The Surefire XML never mentions IntegrationProbeIT, and the Failsafe XML never mentions UnitProbeTest. Failsafe also writes a failsafe-summary.xml:

<completed>1</completed>
<errors>0</errors>
<failures>0</failures>
<skipped>0</skipped>

The receipt for the *IT lives in target/failsafe-reports. Reading only surefire-reports tells you nothing about the integration test, however green it looks.

The agent script and the skip flags

This is the gate that usually shows up in an agent harness:

#!/usr/bin/env bash
set -euo pipefail
mvn -q test
echo "all tests passed"

The set -euo pipefail line makes the script look strict, and it is strict. It's just strict about the wrong command. The exit 0 covers Surefire. The echo line claims more than the command proved, and the agent copies that line straight into the PR summary.

Two things about Failsafe change how this gate should be written:

• Failures only show up at verify. According to the Failsafe docs, the plugin doesn't fail the build during the integration-test phase, so that post-integration-test still gets to run. failsafe:verify is the goal that fails the build. That's why the docs recommend running mvn verify instead of calling the integration-test phase directly.
• Running the goals by themselves isn't a shortcut. In the same experiment, running failsafe:integration-test failsafe:verify after clean exited 0 and produced no TEST-*.xml at all, because no tests had been compiled. One more green with nothing inside it.

Then there are the skip flags, which changed in 3.6.0 (skipping tests in Failsafe):

• -DskipITs skips Failsafe's integration tests.
• Since Failsafe 3.6.0, -DskipTests skips only Surefire. It no longer skips the Failsafe ITs.
• -Dmaven.test.skip=true is honored by Surefire, Failsafe and the Compiler plugin. It skips compiling the tests as well as running them.

If the agent's command carries any of these three flags, it isn't testing what it says it's testing.

A gate you can show in review

The honest gate runs the lifecycle through verify and checks for the receipt whenever the module has an *IT:

#!/usr/bin/env bash
set -euo pipefail

mvn -q clean verify

has_it="$(find src/test/java \( -name '*IT.java' -o -name 'IT*.java' -o -name '*ITCase.java' \) -print -quit)"

if [ -n "$has_it" ]; then
  compgen -G 'target/failsafe-reports/TEST-*.xml' > /dev/null || {
    echo "module has *IT, but target/failsafe-reports has no TEST-*.xml" >&2
    exit 1
  }
fi

echo "unit tests (Surefire) and integration tests (Failsafe) executed"

The find patterns are the same as Failsafe's default includes. If your POM changes the includes, the script has to change with it. In a multi-module build, run the check once per module.

The same rule applies in CI. Publish target/failsafe-reports as an artifact, and fail the job if the module has an *IT but the directory has no TEST-*.xml. That's the signal you can actually observe in production. Maven's exit code alone can't tell "the ITs passed" apart from "the ITs never ran". The missing directory can.

Next step

If the module has *IT.java, the agent's gate is mvn verify, with Failsafe declared in the POM and the receipt in target/failsafe-reports checked. mvn test won't do. If the module has neither *IT nor Failsafe, mvn test is still the right command for unit tests. Just don't let the agent summarize that as "all tests".

To get started, search the repository for *IT.java and check whether the POM declares the Failsafe executions. Then open the agent's gate script and change test to verify. If target/failsafe-reports shows up for the first time after that, you've just found out how long those tests had gone without running.