[
https://issues.apache.org/jira/browse/CXF-9250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sebastian Schubert updated CXF-9250:
------------------------------------
Description:
When an exchange sent through {{HttpClientHTTPConduit}} fails (TLS handshake
failure, connection reset, connection refused, ...) and the request had no body
(e.g. a GET), the conduit sends the same request again from the
{{future.exceptionally(...)}} callback. That request fails too, its callback
sends it again, and so on. The loop only ends when an exchange eventually
succeeds.
The most common trigger is the remote WSDL download done by
{{TransportURIResolver}} when a JAX-WS client is created with an http(s) WSDL
URL ({{{}Service.create(url, qname){}}} or a generated {{Service}} subclass).
The caller gets a normal {{{}WebServiceException{}}}, but a background resend
loop keeps running for every failed attempt.
h3. Impact (observed in production)
A JAX-WS client whose remote WSDL host started failing TLS handshakes (server
certificate chain not in the client truststore) produced ~150,000-200,000
requests/min to that host from a handful of application calls. Side effects:
* ~250 {{ForkJoinPool.commonPool}} workers blocked in
{{HttpClientHTTPConduit$HttpClientWrappedOutputStream.getResponse}} ->
{{{}CompletableFuture.get{}}}, called from
{{{}lambda$setProtocolHeaders$...{}}}; {{compensatedBlock}} spawning
replacement workers
* high CPU from repeated PKIX path building
* no log output: {{TransportURIResolver}} logs the failure at FINEST, and the
future returned by {{exceptionally}} is never observed
* application-level retry/circuit-breaker logic does not see the resends,
because they happen after the original call has already failed Traffic stopped
immediately once the certificate was added to the truststore, i.e. once the
next resend succeeded.
h3. Root cause
{{HttpClientHTTPConduit.HttpClientWrappedOutputStream.setProtocolHeaders()}}
builds the request, calls {{sendAsync}} and registers:
{code:java}
future = cl.sendAsync(request, handler);
future.exceptionally(ex -> {
if (pout != null) {
synchronized (pout) {
pout.notifyAll();
}
}
try {
close();
} catch (IOException e) {
ex.addSuppressed(e);
}
return null;
});{code}
{{close()}} is the full {{{}HTTPConduit.WrappedOutputStream.close(){}}}:
{code:java}
boolean exceptionSet = outMessage.getContent(Exception.class) != null;
if (!written && !exceptionSet) {
handleHeadersTrustCaching(); // -> setProtocolHeaders() ->
sendAsync(...) again
}
...
handleResponse();{code}
For a request without body, {{written}} is never set (it is only set by
{{{}AbstractWrappedOutputStream.write{}}}), and on the synchronous path the out
message never gets an {{Exception}} content. So every invocation of {{close()}}
from the callback runs {{handleHeadersTrustCaching()}} ->
{{{}setProtocolHeaders(){}}}, which sends a new request and registers a new
callback. The callback thread then blocks in {{handleResponse()}} ->
{{getResponse()}} on the new future. Each failed attempt therefore
starts a self-sustaining chain with roughly one blocked thread per chain.
With {{{}URLConnectionHTTPConduit{}}}, {{setProtocolHeaders()}} only sets
headers on the connection object, so running it twice has no network effect. In
{{HttpClientHTTPConduit}} the same template hook performs the send, which makes
the stream's {{close()}} non-idempotent.
The {{close()}} call in the callback was added in 3.6.7 / 4.0.8 / 4.1.2 (not
present in 3.6.6 / 4.0.7 / 4.1.1), likely as part of CXF-9115.
{{HttpClientWrappedOutputStream.exception}} is read in {{setupWrappedStream()}}
and {{handleNoOutput()}} but never assigned.
Requests with a body (e.g. SOAP POST) are mostly unaffected because {{written}}
becomes true after the first write. However, {{written = true}} is only set
after {{onFirstWrite()}} returns, while the callback is already registered
inside {{onFirstWrite()}} -> {{{}setProtocolHeaders(){}}}. So an exchange that
fails inside that window can be re-sent as well.
h3. Reproducer
A server that accepts and immediately closes every connection, and one JAX-WS
{{Service.create}} with a WSDL URL pointing to it:
{code:java}
import java.net.*;
import java.util.concurrent.atomic.AtomicInteger;
import javax.xml.namespace.QName;
import jakarta.xml.ws.Service; // javax.xml.ws.Service on 3.6.x
public class ResendLoopRepro {
public static void main(String[] args) throws Exception {
AtomicInteger requests = new AtomicInteger();
ServerSocket server = new ServerSocket(0, 200,
InetAddress.getLoopbackAddress());
Thread acceptor = new Thread(() -> {
while (true) {
try (Socket s = server.accept()) {
requests.incrementAndGet(); // close without
responding
} catch (Exception e) {
return;
}
}
});
acceptor.setDaemon(true);
acceptor.start();
URL wsdl = new URL("http://localhost:"; + server.getLocalPort() +
"/service.wsdl");
try {
Service.create(wsdl, new QName("urn:test", "TestService"));
} catch (Exception expected) {
System.out.println("Service.create failed as expected; requests
so far: " + requests.get());
}
Thread.sleep(2000);
System.out.println("Requests 2s later: " + requests.get()); //
keeps growing
System.exit(0);
}
}
{code}
Observed with one {{Service}} creation:
||CXF / Java||requests when the call failed||requests after 2 s||
|3.6.11 / Java 17|102|460|
|4.1.7 / Java 21|143|442|
Expected: 1 request (or a small bounded number); nothing after the call has
failed.
A TLS server with an untrusted certificate shows the same behaviour with
{{{}SSLHandshakeException: PKIX path building failed{}}}.
h3. Workarounds
* {{{}-Dorg.apache.cxf.transport.http.forceURLConnection=true{}}}, or the
contextual property {{force.urlconnection.http.conduit=true}} (note: a
client/request-context property does not reach the WSDL fetch in
{{{}TransportURIResolver{}}}, which uses a bare message)
* load the WSDL from the classpath instead of http(s)
was:
When an exchange sent through {{HttpClientHTTPConduit}} fails (TLS handshake
failure, connection reset, connection refused, ...) and the request had no body
(e.g. a GET), the conduit sends the same request again from the
{{future.exceptionally(...)}} callback. That request fails too, its callback
sends it again, and so on. The loop only ends when an exchange eventually
succeeds.
The most common trigger is the remote WSDL download done by
{{TransportURIResolver}} when a JAX-WS client is created with an http(s) WSDL
URL ({{{}Service.create(url, qname){}}} or a generated {{Service}} subclass).
The caller gets a normal {{{}WebServiceException{}}}, but a background resend
loop keeps running for every failed attempt.
h3. Impact (observed in production)
A JAX-WS client whose remote WSDL host started failing TLS handshakes (server
certificate chain not in the client truststore) produced ~150,000-200,000
requests/min to that host from a handful of application calls. Side effects:
* ~250 {{ForkJoinPool.commonPool}} workers blocked in
{{HttpClientHTTPConduit$HttpClientWrappedOutputStream.getResponse}} ->
{{{}CompletableFuture.get{}}}, called from
{{{}lambda$setProtocolHeaders$...{}}}; {{compensatedBlock}} spawning
replacement workers
* high CPU from repeated PKIX path building
* no log output: {{TransportURIResolver}} logs the failure at FINEST, and the
future returned by {{exceptionally}} is never observed
* application-level retry/circuit-breaker logic does not see the resends,
because they happen after the original call has already failed Traffic stopped
immediately once the certificate was added to the truststore, i.e. once the
next resend succeeded.
h3.
h3. Root cause
{{HttpClientHTTPConduit.HttpClientWrappedOutputStream.setProtocolHeaders()}}
builds the request, calls {{sendAsync}} and registers:
{code:java}
future = cl.sendAsync(request, handler);
future.exceptionally(ex -> {
if (pout != null) {
synchronized (pout) {
pout.notifyAll();
}
}
try {
close();
} catch (IOException e) {
ex.addSuppressed(e);
}
return null;
});
{code}
{{close()}} is the full {{{}HTTPConduit.WrappedOutputStream.close(){}}}:
{code:java}
boolean exceptionSet = outMessage.getContent(Exception.class) != null;
if (!written && !exceptionSet) {
handleHeadersTrustCaching(); // -> setProtocolHeaders() ->
sendAsync(...) again
}
...
handleResponse();
{code}
For a request without body, {{written}} is never set (it is only set by
{{{}AbstractWrappedOutputStream.write{}}}), and on the synchronous path the out
message never gets an {{Exception}} content. So every invocation of {{close()}}
from the callback runs {{handleHeadersTrustCaching()}} ->
{{{}setProtocolHeaders(){}}}, which sends a new request and registers a new
callback. The callback thread then blocks in {{handleResponse()}} ->
{{getResponse()}} on the new future. Each failed attempt therefore
starts a self-sustaining chain with roughly one blocked thread per chain.
With {{{}URLConnectionHTTPConduit{}}}, {{setProtocolHeaders()}} only sets
headers on the connection object, so running it twice has no network effect. In
{{HttpClientHTTPConduit}} the same template hook performs the send, which makes
the stream's {{close()}} non-idempotent.
The {{close()}} call in the callback was added in 3.6.7 / 4.0.8 / 4.1.2 (not
present in 3.6.6 / 4.0.7 / 4.1.1), likely as part of CXF-9115.
{{HttpClientWrappedOutputStream.exception}} is read in {{setupWrappedStream()}}
and {{handleNoOutput()}} but never assigned.
Requests with a body (e.g. SOAP POST) are mostly unaffected because {{written}}
becomes true after the first write. However, {{written = true}} is only set
after {{onFirstWrite()}} returns, while the callback is already registered
inside {{onFirstWrite()}} -> {{{}setProtocolHeaders(){}}}. So an exchange that
fails inside that window can be re-sent as well.
h3. Reproducer
A server that accepts and immediately closes every connection, and one JAX-WS
{{Service.create}} with a WSDL URL pointing to it:
{code:java}
import java.net.*;
import java.util.concurrent.atomic.AtomicInteger;
import javax.xml.namespace.QName;
import jakarta.xml.ws.Service; // javax.xml.ws.Service on 3.6.x
public class ResendLoopRepro {
public static void main(String[] args) throws Exception {
AtomicInteger requests = new AtomicInteger();
ServerSocket server = new ServerSocket(0, 200,
InetAddress.getLoopbackAddress());
Thread acceptor = new Thread(() -> {
while (true) {
try (Socket s = server.accept()) {
requests.incrementAndGet(); // close without
responding
} catch (Exception e) {
return;
}
}
});
acceptor.setDaemon(true);
acceptor.start();
URL wsdl = new URL("http://localhost:"; + server.getLocalPort() +
"/service.wsdl");
try {
Service.create(wsdl, new QName("urn:test", "TestService"));
} catch (Exception expected) {
System.out.println("Service.create failed as expected; requests
so far: " + requests.get());
}
Thread.sleep(2000);
System.out.println("Requests 2s later: " + requests.get()); //
keeps growing
System.exit(0);
}
}
{code}
Observed with one {{Service}} creation:
||CXF / Java||requests when the call failed||requests after 2 s||
|3.6.11 / Java 17|102|460|
|4.1.7 / Java 21|143|442|
Expected: 1 request (or a small bounded number); nothing after the call has
failed.
A TLS server with an untrusted certificate shows the same behaviour with
{{{}SSLHandshakeException: PKIX path building failed{}}}.
h3. Workarounds
* {{{}-Dorg.apache.cxf.transport.http.forceURLConnection=true{}}}, or the
contextual property {{force.urlconnection.http.conduit=true}} (note: a
client/request-context property does not reach the WSDL fetch in
{{{}TransportURIResolver{}}}, which uses a bare message)
* load the WSDL from the classpath instead of http(s)
> HttpClientHTTPConduit: failed exchange without request body is re-sent
> endlessly from the exceptionally callback (e.g. remote WSDL fetch)
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CXF-9250
> URL: https://issues.apache.org/jira/browse/CXF-9250
> Project: CXF
> Issue Type: Bug
> Components: Transports
> Affects Versions: 4.2.0, 4.1.2, 4.0.8, 3.6.7, 4.1.3, 4.0.9, 3.6.8, 4.1.4,
> 3.6.9, 4.0.10, 4.1.5, 4.0.11, 4.2.1, 3.6.11, 4.1.6, 4.1.7, 3.6.12, 4.2.2,
> 4.2.3, 4.1.8, 4.2.4
> Reporter: Sebastian Schubert
> Priority: Major
>
> When an exchange sent through {{HttpClientHTTPConduit}} fails (TLS handshake
> failure, connection reset, connection refused, ...) and the request had no
> body (e.g. a GET), the conduit sends the same request again from the
> {{future.exceptionally(...)}} callback. That request fails too, its callback
> sends it again, and so on. The loop only ends when an exchange eventually
> succeeds.
> The most common trigger is the remote WSDL download done by
> {{TransportURIResolver}} when a JAX-WS client is created with an http(s) WSDL
> URL ({{{}Service.create(url, qname){}}} or a generated {{Service}} subclass).
> The caller gets a normal {{{}WebServiceException{}}}, but a background resend
> loop keeps running for every failed attempt.
> h3. Impact (observed in production)
> A JAX-WS client whose remote WSDL host started failing TLS handshakes (server
> certificate chain not in the client truststore) produced ~150,000-200,000
> requests/min to that host from a handful of application calls. Side effects:
> * ~250 {{ForkJoinPool.commonPool}} workers blocked in
> {{HttpClientHTTPConduit$HttpClientWrappedOutputStream.getResponse}} ->
> {{{}CompletableFuture.get{}}}, called from
> {{{}lambda$setProtocolHeaders$...{}}}; {{compensatedBlock}} spawning
> replacement workers
> * high CPU from repeated PKIX path building
> * no log output: {{TransportURIResolver}} logs the failure at FINEST, and
> the future returned by {{exceptionally}} is never observed
> * application-level retry/circuit-breaker logic does not see the resends,
> because they happen after the original call has already failed Traffic
> stopped immediately once the certificate was added to the truststore, i.e.
> once the next resend succeeded.
> h3. Root cause
> {{HttpClientHTTPConduit.HttpClientWrappedOutputStream.setProtocolHeaders()}}
> builds the request, calls {{sendAsync}} and registers:
> {code:java}
> future = cl.sendAsync(request, handler);
> future.exceptionally(ex -> {
> if (pout != null) {
> synchronized (pout) {
> pout.notifyAll();
> }
> }
> try {
> close();
> } catch (IOException e) {
> ex.addSuppressed(e);
> }
> return null;
> });{code}
> {{close()}} is the full {{{}HTTPConduit.WrappedOutputStream.close(){}}}:
> {code:java}
> boolean exceptionSet = outMessage.getContent(Exception.class) != null;
> if (!written && !exceptionSet) {
> handleHeadersTrustCaching(); // -> setProtocolHeaders() ->
> sendAsync(...) again
> }
> ...
> handleResponse();{code}
> For a request without body, {{written}} is never set (it is only set by
> {{{}AbstractWrappedOutputStream.write{}}}), and on the synchronous path the
> out message never gets an {{Exception}} content. So every invocation of
> {{close()}} from the callback runs {{handleHeadersTrustCaching()}} ->
> {{{}setProtocolHeaders(){}}}, which sends a new request and registers a new
> callback. The callback thread then blocks in {{handleResponse()}} ->
> {{getResponse()}} on the new future. Each failed attempt therefore
> starts a self-sustaining chain with roughly one blocked thread per chain.
> With {{{}URLConnectionHTTPConduit{}}}, {{setProtocolHeaders()}} only sets
> headers on the connection object, so running it twice has no network effect.
> In {{HttpClientHTTPConduit}} the same template hook performs the send, which
> makes the stream's {{close()}} non-idempotent.
> The {{close()}} call in the callback was added in 3.6.7 / 4.0.8 / 4.1.2 (not
> present in 3.6.6 / 4.0.7 / 4.1.1), likely as part of CXF-9115.
> {{HttpClientWrappedOutputStream.exception}} is read in
> {{setupWrappedStream()}} and {{handleNoOutput()}} but never assigned.
> Requests with a body (e.g. SOAP POST) are mostly unaffected because
> {{written}} becomes true after the first write. However, {{written = true}}
> is only set after {{onFirstWrite()}} returns, while the callback is already
> registered inside {{onFirstWrite()}} -> {{{}setProtocolHeaders(){}}}. So an
> exchange that fails inside that window can be re-sent as well.
> h3. Reproducer
> A server that accepts and immediately closes every connection, and one JAX-WS
> {{Service.create}} with a WSDL URL pointing to it:
> {code:java}
> import java.net.*;
> import java.util.concurrent.atomic.AtomicInteger;
> import javax.xml.namespace.QName;
> import jakarta.xml.ws.Service; // javax.xml.ws.Service on 3.6.x
> public class ResendLoopRepro {
> public static void main(String[] args) throws Exception {
> AtomicInteger requests = new AtomicInteger();
> ServerSocket server = new ServerSocket(0, 200,
> InetAddress.getLoopbackAddress());
> Thread acceptor = new Thread(() -> {
> while (true) {
> try (Socket s = server.accept()) {
> requests.incrementAndGet(); // close without
> responding
> } catch (Exception e) {
> return;
> }
> }
> });
> acceptor.setDaemon(true);
> acceptor.start();
> URL wsdl = new URL("http://localhost:"; + server.getLocalPort() +
> "/service.wsdl");
> try {
> Service.create(wsdl, new QName("urn:test", "TestService"));
> } catch (Exception expected) {
> System.out.println("Service.create failed as expected; requests
> so far: " + requests.get());
> }
> Thread.sleep(2000);
> System.out.println("Requests 2s later: " + requests.get()); //
> keeps growing
> System.exit(0);
> }
> }
> {code}
> Observed with one {{Service}} creation:
> ||CXF / Java||requests when the call failed||requests after 2 s||
> |3.6.11 / Java 17|102|460|
> |4.1.7 / Java 21|143|442|
> Expected: 1 request (or a small bounded number); nothing after the call has
> failed.
> A TLS server with an untrusted certificate shows the same behaviour with
> {{{}SSLHandshakeException: PKIX path building failed{}}}.
> h3. Workarounds
> * {{{}-Dorg.apache.cxf.transport.http.forceURLConnection=true{}}}, or the
> contextual property {{force.urlconnection.http.conduit=true}} (note: a
> client/request-context property does not reach the WSDL fetch in
> {{{}TransportURIResolver{}}}, which uses a bare message)
> * load the WSDL from the classpath instead of http(s)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
