[
https://issues.apache.org/jira/browse/KAFKA-544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jay Kreps updated KAFKA-544:
----------------------------
Attachment: KAFKA-544-joel-comments.patch
Hey Joel, those are good points. Here is a follow-up patch that adresses these
issues. Specifically:
- KeylessMessageDecoder makes no sense it was a hold over from an older, lower
impact approach to defining decoder that would have let us leave the consumer
api more or less unchanged. It is irrelevant now. Deleted it.
- This does make sense. I had planned to leave this as a follow-up future item
since it is essentially a new feature. However it is not hard. I implemented
this on both the producer and consumer side.
- Null vs Options. This is controversial but I am one of the ones who think
option is not so good for us. Two reasons: (1) You have to create a new object
and if the value is a number, box it, (2) the resulting match statement is a
lot less readable than if(x == null) imho. It is true that
NullPointerExceptions are a drag but in my experience these are just not that
common in our kind of software as a production issue because our testing and
load is relatively heavy so these issues are flushed out very quickly.
- ProducerSendThread: good call, changed.
- Yeah I thought about this. I basically just thought it was simpler. The
complication is what do you do if there is both a no-arg constructor and one
that takes options--the user has to know which we will invoke. Seems simpler to
just require the argument even if you don't use it.
> Retain key in producer and expose it in the consumer
> ----------------------------------------------------
>
> Key: KAFKA-544
> URL: https://issues.apache.org/jira/browse/KAFKA-544
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Jay Kreps
> Assignee: Jay Kreps
> Priority: Blocker
> Labels: bugs
> Attachments: KAFKA-544-joel-comments.patch, KAFKA-544-v1.patch,
> KAFKA-544-v2.patch, KAFKA-544-v3.patch, KAFKA-544-v4.patch,
> KAFKA-544-v5.patch, KAFKA-544-v6.patch
>
>
> KAFKA-506 added support for retaining a key in the messages, however this
> field is not yet set by the producer.
> The proposal for doing this is to change the producer api to change
> ProducerData to allow only a single key/value pair so it has a one-to-one
> mapping to Message. That is change from
> ProducerData(topic: String, key: K, data: Seq[V])
> to
> ProducerData(topic: String, key: K, data: V)
> The key itself needs to be encoded. There are several ways this could be
> handled. A few of the options:
> 1. Change the Encoder and Decoder to be MessageEncoder and MessageDecoder and
> have them take both a key and value.
> 2. Another option is to change the type of the encoder/decoder to not refer
> to Message so it could be used for both the key and value.
> I favor the second option but am open to feedback.
> One concern with our current approach to serialization as well as both of
> these proposals is that they are inefficient. We go from
> Object=>byte[]=>Message=>MessageSet with a copy at each step. In the case of
> compression there are a bunch of intermediate steps. We could theoretically
> clean this up by instead having an interface for the encoder that was
> something like
> Encoder.writeTo(buffer: ByteBuffer, object: AnyRef)
> and
> Decoder.readFrom(buffer:ByteBuffer): AnyRef
> However there are two problems with this. The first is that we don't actually
> know the size of the data until it is serialized so we can't really allocate
> the bytebuffer properly and might need to resize it. The second is that in
> the case of compression there is a whole other path to consider. Originally I
> thought maybe it would be good to try to fix this, but now I think it should
> be out-of-scope and we should revisit the efficiency issue in a future
> release in conjunction with our internal handling of compression.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
