Jonny Carter created HTTPCLIENT-2430:
----------------------------------------
Summary: Brotli decompression fails for large payloads
Key: HTTPCLIENT-2430
URL: https://issues.apache.org/jira/browse/HTTPCLIENT-2430
Project: HttpComponents HttpClient
Issue Type: Bug
Components: HttpClient (async)
Affects Versions: 5.6.1
Reporter: Jonny Carter
h2. Symptom
With transparent async content decompression (new in 5.6) and brotli4j on the
classpath, any response with {{Content-Encoding: br}} whose *compressed* body
is larger than 8 KB never completes. {{InflatingBrotliDataConsumer.consume()}}
stops making progress once the decoder's fixed 8 KB input buffer is full: the
{{xfer == 0}} branch calls {{decoder.push(0)}} / {{pump()}} in a loop that can
neither advance nor fail, so {{while (src.hasRemaining())}} never exits.
In a real async client the consequences are severe and silent: the exchange
never completes, no response timeout fires, and the pooled connection is never
released. Under load the pool drains connection by connection until every
request to that route hangs. There is no exception and no log line anywhere.
We found this on 5.6.4; the code is unchanged on the 5.6.x branch head.
HTTPCLIENT-2428 (fixed for 5.7-alpha1) touches the same class but only replaces
the capacity plumbing; the {{consume()}} loop is untouched, so 5.7-alpha1
appears to be affected as well.
A second, smaller defect in the same class: a single spurious byte after a
complete, valid brotli stream fails the exchange with {{{}IOException: Brotli
stream corrupted{}}}, although the content is fully recoverable and the trailer
could be ignored.
h2. How to reproduce
Failing unit tests against {{InflatingBrotliDataConsumerTest}} are attached and
available on a GitHub branch in my fork
[https://github.com/jonnybot0/httpcomponents-client/tree/bugfix-brotli] (commit
402e689774fe76ae9385e9b2e7b603a3ac11f5bc).
* {{inflateBrotliLargerThanInputBufferSingleBuffer}} – well-formed brotli
stream, ~64 KB of incompressible (random) payload so the compressed form
exceeds 8 KB, delivered as one buffer. Fails with {{{}execution timed out after
5000 ms{}}}; the timeout stack pins the busy-loop at
{{InflatingBrotliDataConsumer.consume}} ({{{}decoder.push{}}} in the {{xfer ==
0}} branch).
* {{inflateBrotliLargerThanInputBufferChunked}} – the same stream in 1 KB
chunks, i.e. identical in shape to the existing passing test; the only
difference is the compressed size. Same 5 s timeout.
* {{inflateBrotliWithTrailingByte}} – valid small stream plus one trailing
byte; fails with {{{}IOException: Brotli stream corrupted{}}}.
Note the existing happy-path test passes only because its highly compressible
payload stays well under the 8 KB buffer.
Core of the repro (the harness drives the consumer directly, no network):
{code:java}
final byte[] original = new byte[64 * 1024];
new java.util.Random(42).nextBytes(original); // incompressible =>
compressed > 8 KB
final byte[] compressed = Encoder.compress(original,
new Encoder.Parameters().setQuality(6).setWindow(22));
final InflatingBrotliDataConsumer inflating = new
InflatingBrotliDataConsumer(rawByteCollector);
inflating.consume(ByteBuffer.wrap(compressed)); // never returns
inflating.streamEnd(Collections.emptyList());
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
