All articles

// Knowledge.log — 技術記事

Resilience4j Bulkhead on Spring Boot 4.1: Keep One Slow Dependency From Starving the Rest

Configure Resilience4j Bulkhead per dependency on Boot 4.1 and prove with an in-process WireMock test that a slow payment gateway never starves inventory calls.

The payment gateway starts answering in two seconds instead of two hundred milliseconds. Nobody touched the inventory service, but inventory lookups start failing too. That's the classic symptom of a shared pool: the threads (or the concurrency budget) that should be free for inventory are all parked waiting on payment to respond.

Bulkhead exists precisely for this: each outbound dependency gets its own limit on concurrent calls, so one dependency maxing out its budget doesn't touch anyone else's. Here we set up two independent bulkheads with Resilience4j on Spring Boot 4.1, bring up a slow dependency with WireMock in-process (no Docker involved), and prove the isolation with a real concurrency test — not a Thread.sleep pretending to be a pass criterion.

This is a different animal from what we covered in rate limiting with Resilience4j on Boot 4.1, no Redis required (English twin: in-memory rate limiting on Boot 4.1). RateLimiter governs inbound admission: how many requests per second your API agrees to process. Bulkhead governs outbound isolation: how many concurrent calls each downstream dependency is allowed to receive from your service. Opposite problems, and the two get confused often enough to earn their own pitfall further down.

Prerequisites and versions used

Everything below ran on this host, nothing invented:

  • Java 25.0.4 Temurin.
  • Spring Boot 4.1.1 (the official system requirements page confirms compatibility up to Java 26; 4.2.0-M1 is a milestone and stays out of this).
  • io.github.resilience4j:resilience4j-spring-boot4:2.4.0 — the Boot 4 starter, not to be confused with resilience4j-spring-boot3.
  • AOP via org.springframework.boot:spring-boot-starter-aspectj:4.1.1. Resilience4j's getting-started guide still points at spring-boot-starter-aop, but that coordinate 404s for Boot 4.1.1 — swap in starter-aspectj and move on.
  • Test dependency org.wiremock:wiremock-standalone:3.13.2. The slimmer org.wiremock:wiremock:3.13.2 failed on this host because it expects a Jetty 11 that isn't on Boot 4.1's test classpath; the standalone jar ships its own Jetty and sidesteps the issue.

The full pom.xml for the demo project:

<parent>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-parent</artifactId>
  <version>4.1.1</version>
</parent>
<properties>
  <java.version>25</java.version>
  <resilience4j.version>2.4.0</resilience4j.version>
</properties>
<dependencies>
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-webmvc</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-aspectj</artifactId>
  </dependency>
  <dependency>
    <groupId>io.github.resilience4j</groupId>
    <artifactId>resilience4j-spring-boot4</artifactId>
    <version>${resilience4j.version}</version>
  </dependency>
  <dependency>
    <groupId>org.wiremock</groupId>
    <artifactId>wiremock-standalone</artifactId>
    <version>3.13.2</version>
    <scope>test</scope>
  </dependency>
</dependencies>

ThreadPoolBulkhead vs SemaphoreBulkhead

Resilience4j ships two real bulkhead flavors, and they are not interchangeable:

SemaphoreBulkhead (io.github.resilience4j.bulkhead.Bulkhead, BulkheadRegistry) runs on a plain java.util.concurrent.Semaphore, executed on the calling thread. It spins up no new threads and has no queue of its own — it just gates how many executions are in flight at once. Once the semaphore is saturated and the configured maxWaitDuration has already elapsed, the call gets a BulkheadFullException immediately. This is the default behind the @Bulkhead annotation (Type.SEMAPHORE), and it's the right fit for a blocking method built on RestClient: the limit protects the dependency without requiring a dedicated thread pool, and overflow fails fast instead of piling up callers waiting for a slot.

ThreadPoolBulkhead (io.github.resilience4j.bulkhead.ThreadPoolBulkhead, ThreadPoolBulkheadRegistry) owns its own ThreadPoolExecutor with a bounded queue. You submit work with executeSupplier(...) and get back a CompletionStage/CompletableFuture; a full pool plus a full queue raises RejectedExecutionException, which Resilience4j wraps as BulkheadFullException. This makes sense when the method already returns CompletableFuture and you actually want the slow dependency to have its own reserved threads, isolated from the rest of the app — but if the method is blocking and just calls .join() on the result, you're back to holding the servlet thread hostage, only now with an extra hop in the middle.

A quick way to decide: blocking call through RestClient, no CompletableFuture in the picture? SEMAPHORE. Method already returns CompletableFuture and you want a dedicated pool? THREADPOOL. Reactive call with WebClient/Mono? Also SEMAPHORE — Resilience4j's own docs state that reactive support only uses the semaphore type, since spinning up an extra thread pool inside a reactive pipeline defeats the point of not blocking in the first place.

Worth flagging: this is the opposite of what Spring Cloud CircuitBreaker defaults to, which uses FixedThreadPoolBulkhead by default for non-reactive calls — the reverse of @Bulkhead's default. This piece stays on plain resilience4j-spring-boot4; it's not about Spring Cloud's abstraction layer, so its YAML namespace stays out of scope.

Configuring one bulkhead per dependency on Boot 4.1

The actual isolation trick isn't "use Bulkhead" — it's using two separate instance names, one per dependency. A single bulkhead shared between payment and inventory just recreates the original problem under a nicer label.

resilience4j:
  bulkhead:
    instances:
      payment:
        maxConcurrentCalls: 2
        maxWaitDuration: 0
      inventory:
        maxConcurrentCalls: 8
        maxWaitDuration: 0

management:
  endpoints:
    web:
      exposure:
        include: health,metrics

Notice maxWaitDuration: 0 on both: with zero wait, a call that can't get a permit fails with BulkheadFullException right away instead of blocking on the semaphore's queue hoping a slot frees up. That's the setup you want for a deterministic isolation test; a maxWaitDuration above zero still uses the calling thread, only now it sits there waiting — a fine call for production (a short queue absorbing bursts), but not something a zero-wait test would match.

Also note that management.endpoints.web.exposure.include only lists health,metrics. Never add env to that include: it exposes environment variables and configuration properties — potentially secrets, depending on what's on the classpath — through the Actuator endpoint.

Each dependency gets its own annotated method:

@Component
public class DownstreamClients {

    private final RestClient paymentRestClient;
    private final RestClient inventoryRestClient;

    public DownstreamClients(
            @Qualifier("paymentRestClient") RestClient paymentRestClient,
            @Qualifier("inventoryRestClient") RestClient inventoryRestClient) {
        this.paymentRestClient = paymentRestClient;
        this.inventoryRestClient = inventoryRestClient;
    }

    @Bulkhead(name = "payment")
    public String pay() {
        return paymentRestClient.get().uri("/pay").retrieve().body(String.class);
    }

    @Bulkhead(name = "inventory")
    public String stock() {
        return inventoryRestClient.get().uri("/stock").retrieve().body(String.class);
    }
}

With no explicit type, both default to SEMAPHORE — consistent with what we just covered for blocking RestClient calls. The name on each annotation is what ties the method to its matching instance in the YAML; that pairing, repeated twice with different names, is the isolation — not some inherent magic in Bulkhead itself.

WireMock in-process, no Docker in sight

To simulate the slow dependency without spinning up a container, wiremock-standalone runs as an ordinary test dependency, embedded right in the test JVM:

static final WireMockServer WM = new WireMockServer(options().dynamicPort());

static {
    WM.start();
    WM.stubFor(get(urlEqualTo("/pay"))
            .willReturn(aResponse().withStatus(200).withBody("paid").withFixedDelay(2000)));
    WM.stubFor(get(urlEqualTo("/stock"))
            .willReturn(aResponse().withStatus(200).withBody("ok")));
}

/pay answers with withFixedDelay(2000) — a fixed two-second delay standing in for the slow payment gateway. /stock answers instantly, no delay attached. dynamicPort() keeps runs from fighting over a fixed port, and each dependency's RestClient points at whatever port WireMock picked, via @DynamicPropertySource wiring downstream.payment and downstream.inventory to http://127.0.0.1:{port}.

That covers everything the test needs to exercise without touching external network, Testcontainers, or any infrastructure dependency at all.

The concurrency test that proves the isolation

Proof of isolation can't be "it worked here, looks fine." It needs to count accepted calls, rejected calls, and the actual concurrency peak, with the delay coming from WireMock — not a Thread.sleep faking contention that isn't really there.

The test launches 6 payment workers and 8 inventory workers, all released at the same instant by a CountDownLatch, each one calling its respective annotated method:

int paymentWorkers = 6;
int inventoryWorkers = 8;
ExecutorService pool = Executors.newFixedThreadPool(paymentWorkers + inventoryWorkers);
CountDownLatch start = new CountDownLatch(1);

Every payment worker lands in one of three buckets: paymentAccepted if clients.pay() returned "paid", paymentRejected if it hit BulkheadFullException (directly or as the cause of another exception), paymentOther for anything else. Inventory workers follow the same pattern. A listener on paymentBh.getEventPublisher().onCallPermitted(...) / onCallFinished(...) also tracks the real peak of concurrently permitted calls.

Result from this run, via mvn -q -Dtest=BulkheadIsolationTest test:

RESULT paymentAccepted=2 paymentRejected=4 paymentOther=0 paymentMaxPermitted=2 inventoryAccepted=8 inventoryRejected=0 inventoryOther=0 paymentAvailable=2 inventoryAvailable=8

Out of 6 payment workers, exactly 2 got through (matching the maxConcurrentCalls configured for payment), and the other 4 hit BulkheadFullException — none fell into "other," and the observed peak of simultaneous permitted calls never went above 2. Out of 8 inventory workers, all 8 were accepted, zero rejected — even while payment sat locked up for two full seconds in the middle of the test window. Once the burst is done, getAvailableConcurrentCalls() goes back to 2 for payment and 8 for inventory: the credits consumed during the test came back correctly, nothing leaked.

That's the central demonstration: the two-second delay on /pay never once ate into the capacity available for /stock. The two bulkheads are independent counters, full stop.

Saturation metrics for production

In production you're not going to be tailing exception counts in a log to know whether the payment bulkhead is saturated. Resilience4j exposes this via Micrometer, per instance (tagged by name):

  • resilience4j.bulkhead.available.concurrent.calls — how many calls still fit right now.
  • resilience4j.bulkhead.max.allowed.concurrent.calls — the configured ceiling, handy as a reference for computing usage percentage.

If available drops to zero while max stays at 2, the payment instance is saturated at that moment — that's the alert signal you want on a dashboard, not a latency average papering over the problem.

One thing worth stating outright because it's easy to assume otherwise: there is no documented Micrometer counter for bulkhead rejections. No resilience4j.bulkhead.rejected.calls, no official equivalent. To count rejections, wire the bulkhead's EventPublisher (onCallRejected) into your own counter, or catch BulkheadFullException at the call site and bump a metric yourself. Inventing a meter name the library doesn't actually expose just gets you an empty dashboard panel in production.

And a distinction that separates a ten-minute debugging session from an hour-long one: rejection is not a timeout. A call rejected by the bulkhead never occupied a permit — it failed at the gate, before touching the network. A timeout happens after a call already got in (permit granted) and the dependency simply took too long, getting cut off by the TimeLimiter or by the HTTP client's read timeout. If your dashboard lumps both into one generic "error" metric, you'll end up chasing a network problem when the actual cause is concurrency, or the other way around.

To expose these metrics safely: management.endpoints.web.exposure.include: health,metrics (add a Prometheus endpoint too, if you already have that registry wired up). Don't add env to that list: it hands back configuration properties and, depending on the classpath, secrets.

Bulkhead is not a RateLimiter

Worth tying off the distinction we opened with, because confusing the two mechanisms is the kind of thing that survives code review unchallenged. RateLimiter, as covered in rate limiting with Resilience4j on Boot 4.1, no Redis required, decides how many requests per second your own API agrees to process — it protects you from whoever calls you. Bulkhead decides how many concurrent calls each outbound dependency can receive from you — it protects your dependencies (and the rest of your app) from one of them turning slow. A service can perfectly well run both at once, covering different directions of the same traffic, and one is no substitute for the other.

Common pitfalls

  • One bulkhead covering two dependencies. If payment and inventory share the same name, inventory goes back to competing for payment's slow-consumed budget — zero isolation, despite the code technically "having Bulkhead." Cross-check the annotation's name against the YAML's name any time a new dependency joins the service.
  • maxWaitDuration above zero changing behavior under saturation. With a configured wait, a call that couldn't get a permit sits parked until the wait times out, still occupying the calling thread — different from failing instantly. That's a legitimate production call (a short queue absorbing bursts), but it isn't the same as zero, and a test written for zero won't match that configuration.
  • THREADPOOL on a method that calls .join(). Switching the type to THREADPOOL without also changing the method signature to return CompletableFuture still leaves the servlet thread parked waiting for the result — you've added an extra pool without removing the original contention from anywhere.
  • Following the getting-started page literally on the AOP dependency. It still cites spring-boot-starter-aop; on Boot 4.1.1 that coordinate no longer exists on Maven Central. Use spring-boot-starter-aspectj.
  • Assuming Semaphore "spawns threads." It doesn't create any — it only limits how many concurrent executions pass through the gate on the caller's own thread. Tomcat's pool sizing remains a separate concern; keeping payment's maxConcurrentCalls well below Tomcat's total thread count is what guarantees the bulkhead limit itself never becomes the new bottleneck.

Recommendation

If your application calls two or more external dependencies with different latency and criticality profiles from the same process, configuring Bulkhead per dependency from day one is worth it — the cost is a YAML section and one annotation per method, and the payoff is the guarantee that service C doesn't go down because service A decided to have a bad day. Default to SEMAPHORE for blocking calls through RestClient; save THREADPOOL for cases where the method is already asynchronous and you genuinely want a dedicated, isolated thread pool for that specific dependency.

Back off from this if you only have one external dependency worth worrying about (there's nothing to isolate from what), or if the application already runs on Spring Cloud CircuitBreaker as its abstraction layer — in that case the configuration and defaults are different, and mixing the two property namespaces causes more bugs than it prevents. Outside those cases, the WireMock concurrency test carries over almost word for word to validate any new pair of dependencies you decide to isolate.

References

javaspring-bootresilience4jmicroservices

// Continue.training — 次のステップ

Knowledge only counts when it becomes practice.

Go back to the article, run the examples, and share what you learned.

Explore more articles