[
https://issues.apache.org/jira/browse/CXF-9250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Freeman Yue Fang resolved CXF-9250.
-----------------------------------
Fix Version/s: 4.0.12
4.1.9
4.2.4
3.6.13
Resolution: Fixed
> HttpClientHTTPConduit: failed exchange without request body is re-sent
> endlessly from the exceptionally callback (e.g. remote WSDL fetch)
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CXF-9250
> URL: https://issues.apache.org/jira/browse/CXF-9250
> Project: CXF
> Issue Type: Bug
> Components: Transports
> Affects Versions: 4.2.0, 4.1.2, 4.0.8, 3.6.7, 4.1.3, 4.0.9, 3.6.8, 4.1.4,
> 3.6.9, 4.0.10, 4.1.5, 4.0.11, 4.2.1, 3.6.11, 4.1.6, 4.1.7, 3.6.12, 4.2.2,
> 4.2.3, 4.1.8, 4.2.4
> Reporter: Sebastian Schubert
> Assignee: Freeman Yue Fang
> Priority: Major
> Fix For: 4.0.12, 4.1.9, 4.2.4, 3.6.13
>
>
> When an exchange sent through {{HttpClientHTTPConduit}} fails (TLS handshake
> failure, connection reset, connection refused, ...) and the request had no
> body (e.g. a GET), the conduit sends the same request again from the
> {{future.exceptionally(...)}} callback. That request fails too, its callback
> sends it again, and so on. The loop only ends when an exchange eventually
> succeeds.
> The most common trigger is the remote WSDL download done by
> {{TransportURIResolver}} when a JAX-WS client is created with an http(s) WSDL
> URL ({{{}Service.create(url, qname){}}} or a generated {{Service}} subclass).
> The caller gets a normal {{{}WebServiceException{}}}, but a background resend
> loop keeps running for every failed attempt.
> h3. Impact (observed in production)
> A JAX-WS client whose remote WSDL host started failing TLS handshakes (server
> certificate chain not in the client truststore) produced ~150,000-200,000
> requests/min to that host from a handful of application calls. Side effects:
> * ~250 {{ForkJoinPool.commonPool}} workers blocked in
> {{HttpClientHTTPConduit$HttpClientWrappedOutputStream.getResponse}} ->
> {{{}CompletableFuture.get{}}}, called from
> {{{}lambda$setProtocolHeaders$...{}}}; {{compensatedBlock}} spawning
> replacement workers
> * high CPU from repeated PKIX path building
> * no log output: {{TransportURIResolver}} logs the failure at FINEST, and
> the future returned by {{exceptionally}} is never observed
> * application-level retry/circuit-breaker logic does not see the resends,
> because they happen after the original call has already failed
> Traffic stopped immediately once the certificate was added to the truststore,
> i.e. once the next resend succeeded.
> h3. Root cause
> {{HttpClientHTTPConduit.HttpClientWrappedOutputStream.setProtocolHeaders()}}
> builds the request, calls {{sendAsync}} and registers:
> {code:java}
> future = cl.sendAsync(request, handler);
> future.exceptionally(ex -> {
> if (pout != null) {
> synchronized (pout) {
> pout.notifyAll();
> }
> }
> try {
> close();
> } catch (IOException e) {
> ex.addSuppressed(e);
> }
> return null;
> });{code}
> {{close()}} is the full {{{}HTTPConduit.WrappedOutputStream.close(){}}}:
> {code:java}
> boolean exceptionSet = outMessage.getContent(Exception.class) != null;
> if (!written && !exceptionSet) {
> handleHeadersTrustCaching(); // -> setProtocolHeaders() ->
> sendAsync(...) again
> }
> ...
> handleResponse();{code}
> For a request without body, {{written}} is never set (it is only set by
> {{{}AbstractWrappedOutputStream.write{}}}), and on the synchronous path the
> out message never gets an {{Exception}} content. So every invocation of
> {{close()}} from the callback runs {{handleHeadersTrustCaching()}} ->
> {{{}setProtocolHeaders(){}}}, which sends a new request and registers a new
> callback. The callback thread then blocks in {{handleResponse()}} ->
> {{getResponse()}} on the new future. Each failed attempt therefore starts a
> self-sustaining chain with roughly one blocked thread per chain.
> With {{{}URLConnectionHTTPConduit{}}}, {{setProtocolHeaders()}} only sets
> headers on the connection object, so running it twice has no network effect.
> In {{HttpClientHTTPConduit}} the same template hook performs the send, which
> makes the stream's {{close()}} non-idempotent.
> The {{close()}} call in the callback was added in 3.6.7 / 4.0.8 / 4.1.2 (not
> present in 3.6.6 / 4.0.7 / 4.1.1), likely as part of CXF-9115.
> {{HttpClientWrappedOutputStream.exception}} is read in
> {{setupWrappedStream()}} and {{handleNoOutput()}} but never assigned.
> Requests with a body (e.g. SOAP POST) are mostly unaffected because
> {{written}} becomes true after the first write. However, {{written = true}}
> is only set after {{onFirstWrite()}} returns, while the callback is already
> registered inside {{onFirstWrite()}} -> {{{}setProtocolHeaders(){}}}. So an
> exchange that fails inside that window can be re-sent as well.
> h3. Reproducer
> A server that accepts and immediately closes every connection, and one JAX-WS
> {{Service.create}} with a WSDL URL pointing to it:
> {code:java}
> import java.net.*;
> import java.util.concurrent.atomic.AtomicInteger;
> import javax.xml.namespace.QName;
> import jakarta.xml.ws.Service; // javax.xml.ws.Service on 3.6.x
> public class ResendLoopRepro {
> public static void main(String[] args) throws Exception {
> AtomicInteger requests = new AtomicInteger();
> ServerSocket server = new ServerSocket(0, 200,
> InetAddress.getLoopbackAddress());
> Thread acceptor = new Thread(() -> {
> while (true) {
> try (Socket s = server.accept()) {
> requests.incrementAndGet(); // close without
> responding
> } catch (Exception e) {
> return;
> }
> }
> });
> acceptor.setDaemon(true);
> acceptor.start();
> URL wsdl = new URL("http://localhost:"; + server.getLocalPort() +
> "/service.wsdl");
> try {
> Service.create(wsdl, new QName("urn:test", "TestService"));
> } catch (Exception expected) {
> System.out.println("Service.create failed as expected; requests
> so far: " + requests.get());
> }
> Thread.sleep(2000);
> System.out.println("Requests 2s later: " + requests.get()); //
> keeps growing
> System.exit(0);
> }
> }
> {code}
> Observed with one {{Service}} creation:
> ||CXF / Java||requests when the call failed||requests after 2 s||
> |3.6.11 / Java 17|102|460|
> |4.1.7 / Java 21|143|442|
> Expected: 1 request (or a small bounded number); nothing after the call has
> failed.
> A TLS server with an untrusted certificate shows the same behaviour with
> {{{}SSLHandshakeException: PKIX path building failed{}}}.
> h3. Workarounds
> * {{{}-Dorg.apache.cxf.transport.http.forceURLConnection=true{}}}, or the
> contextual property {{force.urlconnection.http.conduit=true}} (note: a
> client/request-context property does not reach the WSDL fetch in
> {{{}TransportURIResolver{}}}, which uses a bare message)
> * load the WSDL from the classpath instead of http(s)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
