Java Benchmarking With JMH: What Each Parameter Does and Why

When I started the Fastest MS SQL Inserts series, the first question I had to answer wasn't "which strategy is fastest?" It was: how do I measure it reliably? Java has a well-known answer to this — JMH (Java Microbenchmark Harness) — but knowing that JMH exists doesn't tell you why it works, or when it's the right tool.

This post walks through JMH from first principles: what happens when Java code gets "warm," why naively timing a loop gives wrong answers, what each JMH parameter controls, and when a simpler approach (timed loop + median) is good enough.

The one thing to take from this post: JMH isn't magic. It's a controlled way to deal with JIT compilation, GC pauses, and OS noise. Understanding why it does what it does lets you decide when you need it and when you don't.

Why you can't just time a loop

The simplest benchmark anyone writes:

1long start = System.nanoTime();
2for (int i = 0; i < 1_000_000; i++) {
3    someMethod();
4}
5long elapsed = System.nanoTime() - start;

Three problems with this:

1. JIT compilation changes what you're measuring. The JVM starts by interpreting bytecode. After ~10,000 invocations, the C1 compiler kicks in with a fast, basic compilation. After ~40,000, C2 recompiles with full optimizations (inlining, escape analysis, loop unrolling). Your first million iterations execute progressively faster code — the loop is measuring a mix of interpreted, C1, and C2 performance. Production runs on C2-compiled code only.

2. Dead-code elimination. If someMethod() returns a value you never use, the JIT compiler may eliminate the call entirely. Your benchmark measures... nothing. You get ~0 ns/op and wonder why your code is infinitely fast.

3. GC pauses. A garbage collection pause in the middle of the loop inflates the elapsed time. One GC adds 10-100 ms; if your loop takes 500 ms total, that's a 20% error from a single pause.

How JMH solves each problem

JMH was created by Aleksey Shipilev (OpenJDK performance engineer) specifically to produce reliable Java benchmarks. It handles the three problems above, plus several more subtle ones.

Warm-up iterations (solving JIT compilation)

JMH runs your benchmark in two phases:

1@Warmup(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS)
2@Measurement(iterations = 10, time = 1, timeUnit = TimeUnit.SECONDS)
  • Warm-up iterations: run the benchmark but throw away the results. This gives the JIT compiler time to optimize your code to the C2-compiled steady state. By the end of warm-up, the hot path is fully compiled.
  • Measurement iterations: run the benchmark and record the results. By now, the code is stable — you're measuring production-quality compiled code, not the compilation process.

How many warm-up iterations? The default (5 × 1 second) is usually sufficient for most Java code to reach C2 compilation. If your benchmark involves complex class loading or framework initialization (like Spring Boot), you may need 10-20 warm-up iterations. The way to tell: if the first two measurement iterations are consistently faster or slower than the rest, you need more warm-up.

Blackhole (solving dead-code elimination)

1@Benchmark
2public void measure(Blackhole bh) {
3    bh.consume(someMethod());  // compiler cannot eliminate this
4}

Blackhole.consume() accepts the return value of your method and does just enough with it that the JIT compiler can't prove the method is unused. It's a carefully designed sink — it doesn't actually do meaningful work, but the compiler can't prove that at compile time.

Without Blackhole: the JIT may inline someMethod(), see that the result is unused, and eliminate the call. You'd be benchmarking an empty loop.

Forking (solving process-level interference)

1@Fork(value = 3, warmups = 1)

JMH runs each benchmark in a separate JVM process (a "fork"). This prevents:

  • Profile pollution: JIT optimizations from one benchmark method leaking into another. If method A and method B share a call site, running A first can cause the JIT to de-optimize B.
  • Class-loading side effects: loading classes in one benchmark shouldn't affect the timing of another.

Fork(3) means: run the benchmark in 3 separate JVM processes, each starting fresh. The results are aggregated across forks.

How many forks? 3 is the default and usually sufficient. For benchmarks where you suspect OS-level variance (disk I/O, thread scheduling), more forks smooth out the noise. For SQL insert benchmarks where each iteration takes seconds, 1 fork is often enough.

The key JMH annotations explained

@BenchmarkMode

1@BenchmarkMode(Mode.AverageTime)  // default: report avg time per operation
2// or Mode.Throughput       — operations per second
3// or Mode.SampleTime       — distribution of times (p50, p99, etc.)
4// or Mode.SingleShotTime   — time a single invocation (no looping)

For SQL insert benchmarks: Mode.AverageTime or Mode.SingleShotTime. Each insert of 100k rows takes 0.3-10 seconds — one invocation is a full benchmark. Mode.Throughput (ops/sec) is more natural for sub-microsecond operations.

@OutputTimeUnit

1@OutputTimeUnit(TimeUnit.MILLISECONDS)

Controls the unit of the reported numbers. For insert benchmarks measured in seconds, MILLISECONDS is the natural choice. For nanosecond-level operations, use NANOSECONDS.

@State

1@State(Scope.Thread)     // one state instance per thread
2@State(Scope.Benchmark)  // shared across all threads

State objects hold the data your benchmark operates on (the list of persons, the database connection, the configuration). Scope.Thread creates a separate instance per thread — safe for mutable state. Scope.Benchmark shares one instance across all threads — use for read-only shared resources (a connection pool, a configuration object).

For SQL insert benchmarks: Scope.Benchmark for the DataSource (shared pool), Scope.Thread for the List<Person> (each thread gets its own data to insert).

@Param

1@Param({"100", "1000", "5000", "10000"})
2int batchSize;

Runs the benchmark once for each parameter value. JMH generates a separate benchmark method for each combination. This is how the batch-size sweep in Part 5 would look in JMH — one annotation instead of a manual loop.

@Threads

1@Threads(4)

Runs the benchmark concurrently with N threads. Combined with @State(Scope.Thread), each thread gets its own state object. This is how the parallel inserts in Part 6 would look in JMH.

When to use JMH vs. a simple timed loop

This is the question I get most often. My rule:

Operation under testJMH?Why
Nanosecond-level (String concat, collection lookup, math)YesJIT, dead-code, and GC effects dominate. You'll get wrong answers without JMH.
Microsecond-level (JSON parse, regex, small SQL query)ProbablyJIT compilation matters less but dead-code elimination still bites.
Millisecond-level (HTTP call, medium SQL query, file I/O)MaybeThe operation is long enough that JIT and GC are noise, not signal. A timed loop with warm-up might suffice.
Second-level (bulk insert, batch ETL, report generation)NoJMH's overhead and complexity don't buy proportional reliability. A simple timed loop (1 warm-up + 10 runs + median) works.

The Fastest MS SQL Inserts series uses the second-level approach — each insert of 100k rows takes 0.3-10 seconds. JMH's warm-up, forking, and blackholing are designed for nanosecond measurements where the JIT and GC are the main sources of noise. For seconds-level operations, the noise sources are different (disk I/O, SQL Server lock contention, network latency), and JMH doesn't help with those.

A minimal JMH setup for Java 17

 1<!-- pom.xml -->
 2<dependency>
 3    <groupId>org.openjdk.jmh</groupId>
 4    <artifactId>jmh-core</artifactId>
 5    <version>1.37</version>
 6</dependency>
 7<dependency>
 8    <groupId>org.openjdk.jmh</groupId>
 9    <artifactId>jmh-generator-annprocess</artifactId>
10    <version>1.37</version>
11    <scope>provided</scope>
12</dependency>
 1@BenchmarkMode(Mode.AverageTime)
 2@OutputTimeUnit(TimeUnit.MICROSECONDS)
 3@Warmup(iterations = 5, time = 1)
 4@Measurement(iterations = 10, time = 1)
 5@Fork(3)
 6@State(Scope.Thread)
 7public class MyBenchmark {
 8
 9    private List<String> data;
10
11    @Setup
12    public void setup() {
13        data = IntStream.range(0, 10_000)
14            .mapToObj(i -> "item-" + i)
15            .collect(Collectors.toList());
16    }
17
18    @Benchmark
19    public int measureFilter(Blackhole bh) {
20        int count = 0;
21        for (String s : data) {
22            if (s.startsWith("item-5")) count++;
23        }
24        bh.consume(count);
25        return count;
26    }
27}

Run with:

1mvn clean install
2java -jar target/benchmarks.jar

What the output means

Benchmark                    Mode  Cnt   Score   Error  Units
MyBenchmark.measureFilter    avgt   30   42.3 ±  1.2   us/op
  • Mode: avgt = average time per operation.
  • Cnt: 30 = 10 measurement iterations × 3 forks.
  • Score: 42.3 = average time per invocation.
  • Error: ± 1.2 = 99.9% confidence interval. If the error is >10% of the score, your benchmark is unstable — increase iterations or forks.
  • Units: us/op = microseconds per operation.

The error column is the most underrated part. A benchmark showing 42.3 ± 25.0 is telling you the measurement is unreliable — something is varying wildly between runs (GC, background process, cold cache). Fix the environment before trusting the number.

The parameters I'd use for our insert benchmarks (if I used JMH)

For the Fastest MS SQL Inserts series, here's what a JMH configuration would look like — and why I didn't use it:

 1@BenchmarkMode(Mode.SingleShotTime)   // each op is one full 100k-row insert
 2@OutputTimeUnit(TimeUnit.MILLISECONDS)
 3@Warmup(iterations = 1)               // one untimed insert to warm cache
 4@Measurement(iterations = 10)         // ten timed inserts, report median
 5@Fork(1)                              // one JVM — the DB is the bottleneck, not the JIT
 6@State(Scope.Benchmark)
 7public class InsertBenchmark {
 8
 9    @Param({"bulkApi", "jdbcTemplate", "hibernate"})
10    String strategy;
11
12    // ... setup DataSource, generate persons ...
13
14    @Benchmark
15    public void insert() {
16        // delegate to the strategy under test
17    }
18}

This is functionally identical to the FastestInsertApplicationTests class I actually wrote — 1 warm-up, 10 timed runs, report the median. JMH adds forking (unnecessary when the DB is the bottleneck) and Blackhole (unnecessary when the insert has a real side effect — it writes to the database). The manual loop is simpler and equally reliable for this use case.

The takeaway: use JMH when the JVM is the noise source. Use a timed loop when the database, network, or disk is the noise source. Know which one you're fighting.