Werner Daehn created KAFKA-4852:
-----------------------------------
Summary: ByteBufferSerializer not compatible with offsets
Key: KAFKA-4852
URL: https://issues.apache.org/jira/browse/KAFKA-4852
Project: Kafka
Issue Type: Bug
Components: clients
Affects Versions: 0.10.1.1
Environment: all
Reporter: Werner Daehn
Priority: Minor
Quick intro: A ByteBuffer.rewind() resets the position to zero. What if the
ByteBuffer was created with an offset? new ByteBuffer(data, 3, 10)? The
ByteBufferSerializer will send from pos=0 and not from pos=3 onwards.
Solution: No rewind() but flip() for reading a ByteBuffer. That's what the flip
is meant for.
Story:
Imagine the incoming data comes from a byte[], e.g. a network stream containing
topicname, partition, key, value, ... and you want to create a new
ProducerRecord for that. As the constructor of ProducerRecord requires (topic,
partition, key, value) you have to copy from above byte[] the key and value.
That means there is a memcopy taking place. Since the payload can be
potentially large, that introduces a lot of overhead. Twice the memory.
A nice solution to this problem is to simply wrap the network byte[] into new
ByteBuffers:
ByteBuffer key = ByteBuffer.wrap(data, keystart, keylength);
ByteBuffer value = ByteBuffer.wrap(data, valuestart, valuelength);
and then use the ByteBufferSerializer instead of the ByteArraySerializer.
But that does not work as the ByteBufferSerializer does a rewind(), hence both,
key and value, will start at position=0 of the data[].
public class ByteBufferSerializer implements Serializer<ByteBuffer> {
public byte[] serialize(String topic, ByteBuffer data) {
data.rewind();
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
