[ https://issues.apache.org/jira/browse/KAFKA-544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472511#comment-13472511 ]
Jun Rao commented on KAFKA-544: ------------------------------- This plan sounds good to me. > Retain key in producer > ---------------------- > > Key: KAFKA-544 > URL: https://issues.apache.org/jira/browse/KAFKA-544 > Project: Kafka > Issue Type: Bug > Affects Versions: 0.8 > Reporter: Jay Kreps > Assignee: Jay Kreps > Priority: Blocker > Labels: bugs > > KAFKA-506 added support for retaining a key in the messages, however this > field is not yet set by the producer. > The proposal for doing this is to change the producer api to change > ProducerData to allow only a single key/value pair so it has a one-to-one > mapping to Message. That is change from > ProducerData(topic: String, key: K, data: Seq[V]) > to > ProducerData(topic: String, key: K, data: V) > The key itself needs to be encoded. There are several ways this could be > handled. A few of the options: > 1. Change the Encoder and Decoder to be MessageEncoder and MessageDecoder and > have them take both a key and value. > 2. Another option is to change the type of the encoder/decoder to not refer > to Message so it could be used for both the key and value. > I favor the second option but am open to feedback. > One concern with our current approach to serialization as well as both of > these proposals is that they are inefficient. We go from > Object=>byte[]=>Message=>MessageSet with a copy at each step. In the case of > compression there are a bunch of intermediate steps. We could theoretically > clean this up by instead having an interface for the encoder that was > something like > Encoder.writeTo(buffer: ByteBuffer, object: AnyRef) > and > Decoder.readFrom(buffer:ByteBuffer): AnyRef > However there are two problems with this. The first is that we don't actually > know the size of the data until it is serialized so we can't really allocate > the bytebuffer properly and might need to resize it. The second is that in > the case of compression there is a whole other path to consider. Originally I > thought maybe it would be good to try to fix this, but now I think it should > be out-of-scope and we should revisit the efficiency issue in a future > release in conjunction with our internal handling of compression. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira