shashank created CAMEL-25573:
--------------------------------
Summary: camel-netty-http - unmarshal().gzipDeflater() of a
request body fails with "Input is not in the .gz format", and a byte-by-byte
read of the body ends at the first 0xFF byte or throws at its end
Key: CAMEL-25573
URL: https://issues.apache.org/jira/browse/CAMEL-25573
Project: Camel
Issue Type: Bug
Components: camel-netty-http
Reporter: shashank
Assignee: shashank
The body of a request received by a netty-http consumer is a
{{NettyChannelBufferStreamCache}} ({{DefaultNettyHttpBinding.toCamelMessage}},
lines 119-123 of main 2a0a2b54f4ba; not with {{disableStreamCache=true}} or
{{httpProxy=true}}). The class is a {{java.io.InputStream}}, and a route step
that asks for the body as an {{InputStream}} gets this object. Its {{read()}}
does not keep the {{InputStream}} contract. With the default configuration:
# Loud: {{from("netty-http:...").unmarshal().gzipDeflater()}} does not work.
Every request fails with HTTP 500 and {{java.io.IOException: Input is not in
the .gz format.}} in the log. The same holds for any reader that reads the
first bytes one by one (to sniff a header, for example): it gets wrong values
for bytes from 0x80, or an {{IndexOutOfBoundsException}} at the end of the body.
# Silent, and narrower: user code or a library that reads the body byte by byte
({{while ((b = in.read()) != -1)}}) gets only the bytes before the first 0xFF,
without any error, and negative values for bytes from 0x80. A byte 0xFF is
common in binary payloads and never occurs in UTF-8 text, so text bodies are
not cut; for them the loop ends with the {{IndexOutOfBoundsException}} (loud).
Not affected: everything that reads in bulk. The type converters of Camel to
{{byte[]}} and {{String}} ({{StreamCacheConverter.convertToByteArray}} calls
{{writeTo}}) and {{NettyHttpConverter.toByteBuf}} read with {{read(byte[], int,
int)}}, which is why most routes do not see this. Silent loss is not the
general case.
The causes in the class:
* {{read()}} (line 56) is {{return buffer.readByte();}}. {{ByteBuf.readByte()}}
returns a Java {{byte}} (-128 to 127) and throws {{IndexOutOfBoundsException}}
when nothing is left. {{InputStream.read()}} must return "an int in the range 0
to 255" and -1 at the end of the stream. So a byte 0xFF is returned as -1,
which every reader takes for the end of the stream, bytes 0x80 to 0xFE are
returned as negative values, and at the end of the body the method throws
instead of returning -1.
* {{markSupported()}} (line 51) returns true, but {{mark(int)}} is not
overridden (it is the empty method of {{InputStream}}). The gzip reader of
commons-compress uses a stream with {{markSupported()}} as it is, so it reads
the magic number with {{read()}} and gets -117 for the byte 0x8b.
({{java.util.zip.GZIPInputStream}} fails with
{{NettyChannelBufferStreamCache.read() returned value out of range -1..255:
-117}}.)
* {{read(byte[], int, int)}} (lines 65-78) takes {{buffer.capacity()}} for the
end of the content; for a buffer whose capacity is larger than its writer index
it throws {{IndexOutOfBoundsException}} at the end of the content. (The
aggregated content of a request has no spare capacity, so this one is not
reached by the consumer today.)
h3. Reproduction
New {{NettyHttpGzipRequestBodyTest}} (4 tests, real consumer and producer) and
{{NettyChannelBufferStreamCacheTest}} (7 tests on the class). With the class of
main 10 of the 11 fail:
* {{gzipDataFormatUnmarshalsTheRequestBody}},
{{resetReturnsTheWholeRequestBodyAfterTheGzipUnmarshal}},
{{writeToReturnsTheWholeRequestBodyAfterTheGzipUnmarshal}}:
{{NettyHttpOperationFailedException ... statusCode: 500}}, in the log of the
consumer {{java.io.IOException: Input is not in the .gz format.}} (all three
fail at the unmarshal)
* {{processorReadsTheRequestBodyByteByByte}} (body {{10 20 FF 30 40}}, the
route counts the bytes with {{read()}}): {{expected: <5> but was: <2>}}
* {{readReturnsEveryByteAsUnsignedValue}}: {{expected: <128> but was: <-128>}}
* {{readLoopCopiesBinaryContent}}: {{array lengths differ, expected: <5> but
was: <2>}}
* {{readLoopEndsAtTheEndOfText}}: {{IndexOutOfBounds readerIndex(5) + length(1)
exceeds writerIndex(5)}}
* {{markIsNotSupported}}: {{expected: <false> but was: <true>}}
* {{readStopsAtTheLastWrittenByte}}: {{IndexOutOfBounds readerIndex(0) +
length(16) exceeds writerIndex(3)}}
* {{gzipInputStreamReadsItsHeader}}: {{IOException:
...NettyChannelBufferStreamCache.read() returned value out of range -1..255:
-117}}
{{resetAndWriteToReturnTheWholeBodyAfterAReaderThatMarks}} passes on main too
(there {{mark}} does nothing either); it guards the fix against a {{mark(int)}}
that moves the reset position, see below.
h3. Proposed fix
* {{read()}} returns {{buffer.readByte() & 0xFF}}, and -1 when the buffer has
no readable byte.
* {{markSupported()}} returns false. {{mark(int)}} stays the empty method and
{{reset()}} is not changed: it goes back to the start of the body, as
{{StreamCache.reset()}} needs. A reader that needs mark support wraps the
stream (the gzip reader of commons-compress does that by itself with a
{{BufferedInputStream}}).
* {{read(byte[], int, int)}} uses {{buffer.readableBytes()}}.
Why not implement {{mark(int)}}: {{reset()}}, {{writeTo()}} (used by the
converters to {{byte[]}} and {{String}}) and {{NettyHttpConverter.toByteBuf}}
all go back to the marked reader index of the buffer, which today is always the
start because only the constructor marks. If {{mark(int)}} moved it, a reader
that marks in the middle (the gzip reader marks before every buffer of 8192
bytes) would leave the stream cache returning only its tail: with that variant
the tests got 3634 of 20028 bytes from {{reset()}} and {{writeTo()}} after the
gzip unmarshal. A stream cache without mark support whose {{reset()}} goes back
to the start exists already: {{FileInputStreamCache}} (line 59, no
{{markSupported}} or {{mark}} override; {{reset()}} at line 89 drops the open
stream, so the next read starts at the beginning of the file). The in-memory
stream caches of camel-support differ: they are, or delegate to, a
{{ByteArrayInputStream}}, which supports mark.
Behaviour change: code that reads the body byte by byte now gets values from 0
to 255 and -1 at the end, and {{markSupported()}} is false. An upgrade note for
4.24 is included (the fix creates {{camel-4x-upgrade-guide-4_24.adoc}}). In the
main sources of Camel the only caller of {{markSupported()}} on a body stream
that can be this class is {{BlobClientWrapper}} of camel-azure-storage-blob
(line 137; read, not run). camel-netty-http with the fix: 312 tests, 0
failures, 8 skipped (skipped before as well).
Not run: a byte-by-byte reader of a text body in a route (the exception at the
end is shown on the class only), a dead letter channel with
{{useOriginalMessage}}.
Related, and independent of this ticket: the gzip data format of
camel-zip-deflater moving the mark of the in-memory stream caches of
camel-support has its own ticket. This fix does not rely on that one.
Found with a Lean 4 model of {{read()}} and of the copy loop of a reader: "the
reader gets the whole body" fails for every body with a byte 0xFF (the reader
ends normally with the bytes before it); a body without one ends with the
exception in the tested case. For the fixed {{read()}} the copy loop is proved
to return every body, and main and the fix are proved equal for bytes below
0x80. For {{reset()}} the model shows that no sequence of read, mark and reset
calls of a reader changes where it goes back to (by construction: {{mark}} does
nothing), and gives the 3634 of 20028 bytes for the variant with a marking
{{mark(int)}}. Then confirmed with the consumer as above.
Affected: main; camel-4.22.x, camel-4.18.x and camel-4.14.x have the same lines
(read, not run).
Duplicate check (2026-10-12): JIRA text "NettyChannelBufferStreamCache" (5:
CAMEL-7884, CAMEL-8284, CAMEL-8408, CAMEL-13967, CAMEL-14166, none about
{{read()}} or {{mark}}), "netty-http markSupported" (none), "not in the .gz
format" (none), "netty-http gzip" (1, unrelated), "netty-http InputStream read
negative" (1, unrelated); GitHub pull requests "NettyChannelBufferStreamCache":
1 merged (#25247, locks), none open.
_Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
