JMH benchmarks: why a System.nanoTime loop measures the wrong thing

Most Java performance discussions start the same way. Someone puts two System.nanoTime() calls around a block of code, runs it in a loop ten million times, divides, and pastes the number into the pull request. Nobody asks what the JIT did to that loop while the clock was running.

Nobody is cheating. The JVM optimizes your code while you measure it, and three things happen at once. Warmup pollutes the first rounds. Code whose result nobody reads disappears. A constant calculation gets replaced by its precomputed result. The timed loop blends all of that into a single number.

Below, you'll see all three effects in a naive loop. Then you'll build a proper JMH measurement in a Maven project on Java 25, learn to read Score and Error without fooling yourself, and see when JMH is the wrong tool.

The measurement environment

Everything ran on September 25, 2026, on a shared VM with 4 vCPUs (AMD EPYC-Rome Processor, first core reporting 2445.406 MHz) and 7.6 GiB of RAM, running Ubuntu 26.04 LTS on kernel 7.0.0-27-generic. Versions: Temurin 25.0.4+7 LTS, Maven 3.9.12 and JMH 1.37.

Java 25 is the current LTS according to Oracle's support roadmap, with GA on September 16, 2025 (JDK 25 page on OpenJDK). According to the Maven Central metadata, 1.37 is the latest JMH release.

The numbers hold for this machine, on this day. They are not a basis for estimating the capacity of anything in production.

The System.nanoTime loop, the way everyone writes it

The program has three columns, each with 10 million iterations and 6 repetitions. It has no warmup, no fork and no Blackhole:

package academy.devdojo.jmh;

import java.util.Locale;

/**
 * The naive microbenchmark: System.nanoTime() around a hot loop, no warmup,
 * no forking, no blackhole. Run it three times and compare with the JMH run.
 *
 * Columns:
 *   dead_ns_op      -> result of the hot path is never used (dead code candidate)
 *   foldable_ns_op  -> loop-invariant add, result used (constant folding candidate)
 *   used_ns_op      -> loop-variant add, result used (the only "honest" column)
 *   ops_per_ns      -> throughput implied by the column; > ~4 ops/ns is physically
 *                      impossible on a single core, so the number is an artifact
 */
public class NaiveNanoTime {

    private static final int ITERATIONS = 10_000_000;
    private static final int REPS = 6;

    static long hotPath(long x) {
        return x * 31 + 7;
    }

    public static void main(String[] args) {
        System.out.println("naive-nanotime iterations=" + ITERATIONS + " reps=" + REPS
                + " jvm=" + System.getProperty("java.vm.version"));
        System.out.println("rep,dead_ns_op,dead_ops_per_ns,foldable_ns_op,foldable_ops_per_ns,used_ns_op,used_ops_per_ns");

        for (int rep = 1; rep <= REPS; rep++) {
            // A) result discarded: the compiler is free to delete the whole loop
            long t0 = System.nanoTime();
            for (int i = 0; i < ITERATIONS; i++) {
                long ignored = hotPath(i);
            }
            long t1 = System.nanoTime();
            double dead = (t1 - t0) / (double) ITERATIONS;

            // B) loop-invariant computation with a used result: constant folding candidate
            long t2 = System.nanoTime();
            long folded = 0;
            for (int i = 0; i < ITERATIONS; i++) {
                folded += 3 * 4;
            }
            long t3 = System.nanoTime();
            double foldable = (t3 - t2) / (double) ITERATIONS;

            // C) loop-variant computation with a used result
            long t4 = System.nanoTime();
            long sum = 0;
            for (int i = 0; i < ITERATIONS; i++) {
                sum += hotPath(i);
            }
            long t5 = System.nanoTime();
            double used = (t5 - t4) / (double) ITERATIONS;

            System.out.printf(Locale.ROOT, "%d,%.6f,%.3f,%.6f,%.3f,%.6f,%.3f%n",
                    rep, dead, opsPerNs(dead), foldable, opsPerNs(foldable), used, opsPerNs(used));

            if (folded == Long.MIN_VALUE || sum == Long.MIN_VALUE) {
                System.out.println("unreachable sink " + folded + " " + sum);
            }
        }
    }

    private static double opsPerNs(double nsPerOp) {
        return nsPerOp <= 0.0 ? Double.POSITIVE_INFINITY : 1.0 / nsPerOp;
    }
}

It ran three times, each in a fresh JVM, with java -cp target/classes academy.devdojo.jmh.NaiveNanoTime. This is the first run:

naive-nanotime iterations=10000000 reps=6 jvm=25.0.4+7-LTS
rep,dead_ns_op,dead_ops_per_ns,foldable_ns_op,foldable_ops_per_ns,used_ns_op,used_ops_per_ns
1,0.778754,1.284,0.305308,3.275,1.004656,0.995
2,1.152258,0.868,0.000003,333333.333,0.338771,2.952
3,0.000003,322580.645,0.000003,333333.333,0.338985,2.950
4,0.000003,333333.333,0.000003,333333.333,0.339685,2.944
5,0.000003,344827.586,0.000003,322580.645,0.333577,2.998
6,0.000005,196078.431,0.000003,333333.333,0.335848,2.978

The table shows three different mistakes.

Warmup ignored. In the used column, repetition 1 came in at 1.004656 ns/op and repetition 6 at 0.335848. The first round was 2.99× slower than the last. The other two JVMs repeated the pattern (1.058541 → 0.337724 and 1.002936 → 0.341853). Measure once, with a cold JVM, and you are measuring the JVM warming up, not the method.

Dead code eliminated. In the dead column, repetitions 1 and 2 came in at 0.778754 and 1.152258. From repetition 3 on, the value sits between 0.000003 and 0.000005 ns/op. The code didn't change: nobody reads ignored, the JIT noticed before any reviewer did, and it removed the loop.

Constant folded. In the foldable column, repetition 1 came in at 0.305308 ns/op and, from repetition 2 on, at 0.000003. The same shape showed up in all three JVMs. Adding 3 * 4 ten million times gives a result you can compute without running the loop, and the JIT did exactly that.

The output gives itself away. The ops_per_ns column prints 333333.333 operations per nanosecond. Even being generous with this host's reported clock (around 2.4 GHz) and assuming 4 instructions per cycle, a core lands in the order of 10 operations per nanosecond. The collapsed value is about four orders of magnitude beyond the hardware. That 0.000003 is not a rate. It is the clock's resolution divided by 10 million, meaning the whole timed region took about 30 ns. At that point the benchmark is measuring the machine's clock. That makes a handy sanity check: if a per-operation number implies more than a few operations per nanosecond, it came from the instrument, not from your code.

Setting up the Maven project with JMH on Java 25

The project has pom.xml at the root and the sources in src/main/java/academy/devdojo/jmh/ (NaiveNanoTime.java and HotPathBenchmark.java).

JMH generates the harness code from your annotations at compile time, using an annotation processor. That's why the pom.xml needs more than a dependency:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>academy.devdojo.jmh</groupId>
  <artifactId>jmh-benchmark</artifactId>
  <version>1.0-SNAPSHOT</version>
  <packaging>jar</packaging>
  <name>devdojo-jmh-benchmark</name>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <maven.compiler.release>25</maven.compiler.release>
    <jmh.version>1.37</jmh.version>
  </properties>

  <dependencies>
    <dependency>
      <groupId>org.openjdk.jmh</groupId>
      <artifactId>jmh-core</artifactId>
      <version>${jmh.version}</version>
    </dependency>
    <dependency>
      <groupId>org.openjdk.jmh</groupId>
      <artifactId>jmh-generator-annprocess</artifactId>
      <version>${jmh.version}</version>
      <scope>provided</scope>
    </dependency>
  </dependencies>

  <build>
    <finalName>benchmarks</finalName>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.14.0</version>
        <configuration>
          <release>25</release>
          <!-- explicit processor path: annotation processing is NOT auto-discovered
               from the compile classpath on modern JDKs / plugin versions -->
          <annotationProcessorPaths>
            <path>
              <groupId>org.openjdk.jmh</groupId>
              <artifactId>jmh-generator-annprocess</artifactId>
              <version>${jmh.version}</version>
            </path>
          </annotationProcessorPaths>
        </configuration>
      </plugin>

      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <version>3.6.1</version>
        <executions>
          <execution>
            <phase>package</phase>
            <goals><goal>shade</goal></goals>
            <configuration>
              <transformers>
                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                  <mainClass>org.openjdk.jmh.Main</mainClass>
                </transformer>
              </transformers>
              <filters>
                <filter>
                  <artifact>*:*</artifact>
                  <excludes>
                    <exclude>META-INF/*.SF</exclude>
                    <exclude>META-INF/*.DSA</exclude>
                    <exclude>META-INF/*.RSA</exclude>
                  </excludes>
                </filter>
              </filters>
            </configuration>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
</project>

The maven-shade-plugin builds target/benchmarks.jar with org.openjdk.jmh.Main as the main class. It also writes a dependency-reduced-pom.xml to the project root, which is expected.

The block that matters most is <annotationProcessorPaths>. In a control project with the same POM minus that block, Maven compiles, packages and reports success. The error only shows up when you try to run it:

[INFO] BUILD SUCCESS
MVN_EXIT=0
Exception in thread "main" java.lang.RuntimeException: ERROR: Unable to find the resource: /META-INF/BenchmarkList
	at org.openjdk.jmh.runner.AbstractResourceReader.getReaders(AbstractResourceReader.java:98)
	at org.openjdk.jmh.runner.BenchmarkList.find(BenchmarkList.java:124)
	at org.openjdk.jmh.runner.Runner.internalRun(Runner.java:252)
	at org.openjdk.jmh.runner.Runner.run(Runner.java:208)
	at org.openjdk.jmh.Main.main(Main.java:71)
JAVA_EXIT=1

The build passes and the jar doesn't run. What separates a green build from a runnable jar is these two commands, after package:

jar tf target/benchmarks.jar | grep META-INF/BenchmarkList
find target/generated-sources -name "*jmh*"

The first must list META-INF/BenchmarkList. The second must find the generated classes in target/generated-sources/annotations/academy/devdojo/jmh/jmh_generated/. If either comes back empty, the annotation processor didn't run.

This is the benchmark class, exactly as it was executed:

package academy.devdojo.jmh;

import java.util.concurrent.TimeUnit;

import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.BenchmarkMode;
import org.openjdk.jmh.annotations.Fork;
import org.openjdk.jmh.annotations.Measurement;
import org.openjdk.jmh.annotations.Mode;
import org.openjdk.jmh.annotations.OutputTimeUnit;
import org.openjdk.jmh.annotations.Scope;
import org.openjdk.jmh.annotations.State;
import org.openjdk.jmh.annotations.Threads;
import org.openjdk.jmh.annotations.Warmup;
import org.openjdk.jmh.infra.Blackhole;

/**
 * Three benchmarks, three lessons:
 *
 *  sumLoop                        -> the measured hot path, consumed through Blackhole,
 *                                    thread-local state (the correct instrument)
 *  counterThreadScope             -> same mutable field increment, @State(Scope.Thread)
 *  counterBenchmarkScope          -> same mutable field increment, @State(Scope.Benchmark)
 *
 * The last two differ only in State scope: the second writes to a field shared by all
 * four threads, so the measured cost is cache-line contention, not the code.
 */
@BenchmarkMode(Mode.Throughput)
@OutputTimeUnit(TimeUnit.SECONDS)
@Warmup(iterations = 3, time = 1)
@Measurement(iterations = 5, time = 1)
@Fork(2)
public class HotPathBenchmark {

    private static final int LOOPS = 10_000;

    @State(Scope.Thread)
    public static class ThreadLocalState {
        long acc;          // one instance per thread: no sharing
    }

    @State(Scope.Benchmark)
    public static class SharedState {
        long shared;       // one instance for the whole run: every thread writes here
    }

    @Benchmark
    public void sumLoop(ThreadLocalState state, Blackhole blackhole) {
        long local = state.acc;
        for (int i = 0; i < LOOPS; i++) {
            local = local * 31 + 7;
        }
        state.acc = local;
        blackhole.consume(local);       // without this, the JIT may delete the loop
    }

    @Benchmark
    @Threads(4)
    public void counterThreadScope(ThreadLocalState state, Blackhole blackhole) {
        for (int i = 0; i < LOOPS; i++) {
            state.acc++;
        }
        blackhole.consume(state.acc);
    }

    @Benchmark
    @Threads(4)
    public void counterBenchmarkScope(SharedState state, Blackhole blackhole) {
        for (int i = 0; i < LOOPS; i++) {
            state.shared++;
        }
        blackhole.consume(state.shared);
    }
}

Each annotation answers one of the naive loop's mistakes:

To build and run:

mvn -q -DskipTests package
java -jar target/benchmarks.jar -l
java -jar target/benchmarks.jar

Always run from the jar, not from the IDE: the JMH README says that in an IDE the setup is "more complex and the results are less reliable".

The cold build took 13.2 s, downloads included. -l lists the three benchmarks. The full run finished with # Run complete. Total time: 00:00:49, about 50 s of wall-clock time. The header confirms the configuration: # Warmup: 3 iterations, 1 s each, # Measurement: 5 iterations, 1 s each and # Blackhole mode: compiler (auto-detected, use -Djmh.blackhole.autoDetect=false to disable).

On JDK 25, each fork prints a WARNING saying that sun.misc.Unsafe::objectFieldOffset was called by org.openjdk.jmh.util.Utils. The warning comes from JMH's own internals and does not indicate an incompatibility with Java 25.

Three traps JMH won't fix for you

Loop unrolling inside the benchmark

JMH doesn't stop you from writing a loop inside a @Benchmark. JMHSample_11_Loops, in the official samples, warns: "you will see there is more magic happening when we allow optimizers to merge the loop iterations". The naive loop's dead and foldable columns show that magic taken to the extreme.

sumLoop has a 10,000-round loop on purpose. Each round depends on the previous one and the result is consumed. Even so, Score counts method calls, not loop rounds. Dividing by 10,000 to get the cost of one round assumes you know what the JIT did with the loop. Also, don't compare sumLoop with the naive loop's used column: sum += hotPath(i) and local = local * 31 + 7 are different recurrences.

If the question is the cost of one round, take the loop out of the @Benchmark and let JMH repeat the call. If the loop has to stay, put @OperationsPerInvocation(10_000) on the method: with it, Score counts loop rounds instead of calls.

State shared between threads

counterThreadScope and counterBenchmarkScope run the same increment with @Threads(4). The only difference is who owns the field:

That's a 10.43× gap. The second number doesn't measure the increment. It measures four cores fighting over the same cache line (see also JMHSample_22_FalseSharing). That factor belongs to this configuration: 4 threads incrementing one long on 4 shared vCPUs. Keep mutable state in Scope.Thread, unless contention is precisely the question.

Results that only exist on your laptop

In the contended benchmark, the two forks averaged 135,617,169 and 159,710,711 ops/s, 16.32% apart. In the uncontended benchmarks, the forks agree within 0.51% (sumLoop) and 0.06% (counterThreadScope). With a single fork, a "20% faster" can be pure JVM-startup luck, which is what JMHSample_12_Forking and JMHSample_13_RunToRun are about. So use at least two forks and never @Fork(0). And only compare runs with the same JVM, the same hardware and the same Blackhole mode: JMH's closing note warns that the difference between modes "can be very significant".

Reading the output: Score, Error and ops/s

The final table, as JMH printed it:

Benchmark                                Mode  Cnt           Score          Error  Units
HotPathBenchmark.counterBenchmarkScope  thrpt   10   147663940.288 ± 19890606.627  ops/s
HotPathBenchmark.counterThreadScope     thrpt   10  1540289479.933 ± 15442130.687  ops/s
HotPathBenchmark.sumLoop                thrpt   10      152219.898 ±     1245.595  ops/s

• Mode and Units: thrpt in ops/s, so higher is better. In ns/op, lower would be better. Always state the mode alongside the number.
• Cnt: 10 is 5 measurement iterations × 2 forks. It's the number of samples behind the Score.
• Score: the mean of all measured iterations, across forks.
• Error: half the width of the 99.9% confidence interval. JMH also prints (min, avg, max), stdev and the full interval. For the contended benchmark, the interval is CI (99.9%): [127773333.661, 167554546.916], a band 27% wide in relative terms.

Error is 13.47% of Score for the contended benchmark, 1.00% for counterThreadScope and 0.82% for sumLoop. That gives you the working rule: a difference only counts when the two variants' intervals don't overlap, which in this configuration means it exceeds the sum of both Error values. For sumLoop, at about 0.82% on each side, that puts the practical floor at about 1.6% for this configuration. For the contended benchmark, a 5% or 10% "win" is still inside the error bar.

Error depends on the configuration. A smoke run with -wi 1 -i 1 -f 1 printed 150487.920 ops/s with an empty Error column. That is a sample, not a result. The naive loop returns six decimal places and no error bar at all, and the number still ends up in the pull request.

JMH itself closes the run by saying the numbers "are just data" and asking: "Do not assume the numbers tell you what you want them to tell." To understand why a number came out the way it did, use the harness's profilers (-lprof lists the available ones). This run collected no profiler data.

When JMH is the wrong tool

In Mode.Throughput, ops/s measures a hot path running inside the process. The README says JMH covers "nano/micro/milli/macro benchmarks", so it will happily run code that does I/O. The problem is something else: when the answer depends on a disk, a socket, a connection pool or a remote service, the numerator stops measuring CPU work and starts measuring the other system. And before you write any benchmark, confirm with JFR or async-profiler that the code is actually hot.

For network calls, integration tests and database access, the question is usually about latency under concurrency. That calls for a different instrument: p50/p95/p99 percentiles measured end to end, with a load generator and realistic timeouts, retries and backpressure. In production, those percentiles come from the service's own p95/p99 metrics or the team's APM stack. The article on Reactor, virtual threads and parallelism in Spring Boot measured end-to-end latency, CPU and queueing for the whole pipeline. A JMH benchmark of one method from that pipeline would answer a different question.

The recommendation

Put both variants as @Benchmark methods in the same class, with at least two forks, mutable state in Scope.Thread and the result consumed. Run on the same JVM and the same hardware, and report Score ± Error with the unit.

As a next step, copy the project, replace sumLoop with your own hot path, add the alternative as a second @Benchmark and run java -jar target/benchmarks.jar. Only open the PR if the difference counts: the two variants' intervals don't overlap, meaning it exceeds the sum of both Error values.