[ https://issues.apache.org/jira/browse/KAFKA-4852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17611345#comment-17611345 ]
Werner Daehn commented on KAFKA-4852: ------------------------------------- Thanks! > ByteBufferSerializer not compatible with offsets > ------------------------------------------------ > > Key: KAFKA-4852 > URL: https://issues.apache.org/jira/browse/KAFKA-4852 > Project: Kafka > Issue Type: Bug > Components: clients > Affects Versions: 0.10.1.1 > Environment: all > Reporter: Werner Daehn > Assignee: LinShunkang > Priority: Minor > Fix For: 3.4.0 > > > Quick intro: A ByteBuffer.rewind() resets the position to zero. What if the > ByteBuffer was created with an offset? new ByteBuffer(data, 3, 10)? The > ByteBufferSerializer will send from pos=0 and not from pos=3 onwards. > Solution: No rewind() but flip() for reading a ByteBuffer. That's what the > flip is meant for. > Story: > Imagine the incoming data comes from a byte[], e.g. a network stream > containing topicname, partition, key, value, ... and you want to create a new > ProducerRecord for that. As the constructor of ProducerRecord requires > (topic, partition, key, value) you have to copy from above byte[] the key and > value. That means there is a memcopy taking place. Since the payload can be > potentially large, that introduces a lot of overhead. Twice the memory. > A nice solution to this problem is to simply wrap the network byte[] into new > ByteBuffers: > ByteBuffer key = ByteBuffer.wrap(data, keystart, keylength); > ByteBuffer value = ByteBuffer.wrap(data, valuestart, valuelength); > and then use the ByteBufferSerializer instead of the ByteArraySerializer. > But that does not work as the ByteBufferSerializer does a rewind(), hence > both, key and value, will start at position=0 of the data[]. > public class ByteBufferSerializer implements Serializer<ByteBuffer> { > public byte[] serialize(String topic, ByteBuffer data) { > data.rewind(); -- This message was sent by Atlassian Jira (v8.20.10#820010)