[
https://issues.apache.org/jira/browse/KAFKA-17825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17909418#comment-17909418
]
Matthias J. Sax edited comment on KAFKA-17825 at 1/2/25 8:20 PM:
-----------------------------------------------------------------
I also stumbled across https://issues.apache.org/jira/browse/KAFKA-4852 again –
it tried to make similar optimizations as KIP-863, but w/o a KIP and it also
broke stuff – for this case, we did actually revert it:
https://issues.apache.org/jira/browse/KAFKA-15602
I am now wondering, if we should actually extend KIP-863 to include the
deserializer part, which is also a breaking change, and ship with 4.0 release?
As a major release, it would be ok to have a breaking change? (Not sure if it
would be worth to roll back KIP-863 for 3.x versions in addition?)
was (Author: mjsax):
I also stumbled across https://issues.apache.org/jira/browse/KAFKA-4852 again –
it tried to make similar optimizations as KIP-863, but w/o a KIP and it also
broke stuff – for this case, we did actually revert it:
https://issues.apache.org/jira/browse/KAFKA-15602
I am now wondering, if we should actually extend KIP-863 to include the
deserializer part, which is also a breaking change, and ship with 4.0 release?
As a major release, it would be ok to have a breaking change?
> ByteBufferDeserializaer's array size can be inconsistent with the older
> version
> -------------------------------------------------------------------------------
>
> Key: KAFKA-17825
> URL: https://issues.apache.org/jira/browse/KAFKA-17825
> Project: Kafka
> Issue Type: Bug
> Components: consumer
> Affects Versions: 3.6.0
> Reporter: Philip Nee
> Assignee: LinShunkang
> Priority: Blocker
> Fix For: 4.0.0, 3.9.1, 3.8.2
>
>
> We've noticed that using the ByteBufferDeserializer can yield a different
> byte array length compare to the deserializer from 3.5.2. This is attributed
> by KIP-863, in particular, the old deserializer truncated the byte array
> starting from
> `buffer.position() + buffer.arrayOffset() + offset` using `Utils.toArray`
>
> Whereas the current implementation is a passthrough.
>
> This can be reproduced using the
> {code:java}
> KafkaConsumerProducerDemo.java{code}
> by changing the type to <Integer, ByteBuffer> and perform a print after poll.
> For example, the producer produces a record [0, test0] (key is an int,
> "test0" is a 5 bytes long string, converted to byte buffer using
> {code:java}
> ByteBuffer.wrap(value.getBytes()){code}
> Prior to KIP-863 we see the following after polling the record from consumer:
> 3.5.2: test0
> 3.6.0:
> {code:java}
> ?$���y�NNޅ�-p�=�����NAc�����8D���8D���������������
> test0{code}
>
> And if you analyze the ByteBuffer post 3.6.0, we can see the current offset
> is at 140 with array length of 149.
>
> [~LSK] - since you wrote the kip and did the implementation, can you address
> this ?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
