Not yet. It will be when 0.8.2 is release. Thanks,
Jun On Wed, Dec 17, 2014 at 5:24 PM, Rajiv Kurian <ra...@signalfuse.com> wrote: > > Has the mvn repo been updated too? > > On Wed, Dec 17, 2014 at 4:31 PM, Jun Rao <j...@confluent.io> wrote: > > > > Thanks everyone for the feedback and the discussion. The proposed changes > > have been checked into both 0.8.2 and trunk. > > > > Jun > > > > On Tue, Dec 16, 2014 at 10:43 PM, Joel Koshy <jjkosh...@gmail.com> > wrote: > > > > > > Jun, > > > > > > Thanks for summarizing this - it helps confirm for me that I did not > > > misunderstand anything in this thread so far; and that I disagree with > > > the premise that the steps in using the current byte-oriented API is > > > cumbersome or inflexible. It involves instantiating the K-V > > > serializers in code (as opposed to config) and a extra (but explicit > > > - i.e., making it very clear to the user) but simple call to serialize > > > before sending. > > > > > > The point about downstream queries breaking can happen just as well > > > with the implicit serializers/deserializers - since ultimately people > > > have to instantiate the specific type in their code and if they want > > > to send it they will. > > > > > > I think adoption is also equivalent since people will just instantiate > > > whatever serializer/deserializer they want in one line. Plugging in a > > > new serializer implementation does require a code change, but that can > > > also be avoided via a config driven factory. > > > > > > So I'm still +0 on the change but I'm definitely not against moving > > > forward with the changes. i.e., unless there is any strong -1 on the > > > proposal from anyone else. > > > > > > Thanks, > > > > > > Joel > > > > > > > With a byte array interface, of course there is nothing that one > can't > > > do. > > > > However, the real question is that whether we want to encourage > people > > to > > > > use it this way or not. Being able to flow just bytes is definitely > > > easier > > > > to get started. That's why many early adopters choose to do it that > > way. > > > > However, it's often the case that they start feeling the pain later > > when > > > > some producers change the data format. Their Hive/Pig queries start > to > > > > break and it's a painful process to have the issue fixed. So, the > > purpose > > > > of this api change is really to encourage people to standardize on a > > > single > > > > serializer/deserializer that supports things like data validation and > > > > schema evolution upstream in the producer. Now, suppose there is an > > Avro > > > > serializer/deserializer implementation. How do we make it easy for > > people > > > > to adopt? If the serializer is part of the api, we can just say, wire > > in > > > > the Avro serializer for key and/or value in the config and then you > can > > > > start sending Avro records to the producer. If the serializer is not > > part > > > > of the api, we have to say, first instantiate a key and/or value > > > serializer > > > > this way, send the key to the key serializer to get the key bytes, > send > > > the > > > > value to the value serializer to get the value bytes, and finally > send > > > the > > > > bytes to the producer. The former will be simpler and likely makes > the > > > > adoption easier. > > > > > > > > Thanks, > > > > > > > > Jun > > > > > > > > On Mon, Dec 15, 2014 at 7:20 PM, Joel Koshy <jjkosh...@gmail.com> > > wrote: > > > > > > > > > > Documentation is inevitable even if the serializer/deserializer is > > > > > part of the API - since the user has to set it up in the configs. > So > > > > > again, you can only encourage people to use it through > documentation. > > > > > The simpler byte-oriented API seems clearer to me because anyone > who > > > > > needs to send (or receive) a specific data type will _be forced to_ > > > > > (or actually, _intuitively_) select a serializer (or deserializer) > > and > > > > > will definitely pick an already available implementation if a good > > one > > > > > already exists. > > > > > > > > > > Sorry I still don't get it and this is really the only sticking > point > > > > > for me, albeit a minor one (which is why I have been +0 all along > on > > > > > the change). I (and I think many others) would appreciate it if > > > > > someone can help me understand this better. So I will repeat the > > > > > question: What "usage pattern" cannot be supported by easily by the > > > > > simpler API without adding burden on the user? > > > > > > > > > > Thanks, > > > > > > > > > > Joel > > > > > > > > > > On Mon, Dec 15, 2014 at 11:59:34AM -0800, Jun Rao wrote: > > > > > > Joel, > > > > > > > > > > > > It's just that if the serializer/deserializer is not part of the > > > API, you > > > > > > can only encourage people to use it through documentation. > However, > > > not > > > > > > everyone will read the documentation if it's not directly used in > > the > > > > > API. > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Jun > > > > > > > > > > > > On Mon, Dec 15, 2014 at 2:11 AM, Joel Koshy <jjkosh...@gmail.com > > > > > wrote: > > > > > > > > > > > > > (sorry about the late follow-up late - I'm traveling most of > this > > > > > > > month) > > > > > > > > > > > > > > I'm likely missing something obvious, but I find the following > to > > > be a > > > > > > > somewhat vague point that has been mentioned more than once in > > this > > > > > > > thread without a clear explanation. i.e., why is it hard to > > share a > > > > > > > serializer/deserializer implementation and just have the > clients > > > call > > > > > > > it before a send/receive? What "usage pattern" cannot be > > supported > > > by > > > > > > > the simpler API? > > > > > > > > > > > > > > > 1. Can we keep the serialization semantics outside the > Producer > > > > > interface > > > > > > > > and have simple bytes in / bytes out for the interface (This > is > > > what > > > > > we > > > > > > > > have today). > > > > > > > > > > > > > > > > The points for this is to keep the interface simple and usage > > > easy to > > > > > > > > understand. The points against this is that it gets hard to > > share > > > > > common > > > > > > > > usage patterns around serialization/message validations for > the > > > > > future. > > > > > > > > > > > > > > > > > > > > > On Tue, Dec 09, 2014 at 03:51:08AM +0000, Sriram Subramanian > > wrote: > > > > > > > > Thank you Jay. I agree with the issue that you point w.r.t > > paired > > > > > > > > serializers. I also think having mix serialization types is > > > rare. To > > > > > get > > > > > > > > the current behavior, one can simply use a > ByteArraySerializer. > > > This > > > > > is > > > > > > > > best understood by talking with many customers and you seem > to > > > have > > > > > done > > > > > > > > that. I am convinced about the change. > > > > > > > > > > > > > > > > For the rest who gave -1 or 0 for this proposal, does the > > answers > > > > > for the > > > > > > > > three points(updated) below seem reasonable? Are these > > > explanations > > > > > > > > convincing? > > > > > > > > > > > > > > > > > > > > > > > > 1. Can we keep the serialization semantics outside the > Producer > > > > > interface > > > > > > > > and have simple bytes in / bytes out for the interface (This > is > > > what > > > > > we > > > > > > > > have today). > > > > > > > > > > > > > > > > The points for this is to keep the interface simple and usage > > > easy to > > > > > > > > understand. The points against this is that it gets hard to > > share > > > > > common > > > > > > > > usage patterns around serialization/message validations for > the > > > > > future. > > > > > > > > > > > > > > > > 2. Can we create a wrapper producer that does the > serialization > > > and > > > > > have > > > > > > > > different variants of it for different data formats? > > > > > > > > > > > > > > > > The points for this is again to keep the main API clean. The > > > points > > > > > > > > against this is that it duplicates the API, increases the > > surface > > > > > area > > > > > > > and > > > > > > > > creates redundancy for a minor addition. > > > > > > > > > > > > > > > > 3. Do we need to support different data types per record? The > > > current > > > > > > > > interface (bytes in/bytes out) lets you instantiate one > > producer > > > and > > > > > use > > > > > > > > it to send multiple data formats. There seems to be some > valid > > > use > > > > > cases > > > > > > > > for this. > > > > > > > > > > > > > > > > > > > > > > > > Mixed serialization types are rare based on interactions with > > > > > customers. > > > > > > > > To get the current behavior, one can simply use a > > > > > ByteArraySerializer. > > > > > > > > > > > > > > > > On 12/5/14 5:00 PM, "Jay Kreps" <j...@confluent.io> wrote: > > > > > > > > > > > > > > > > >Hey Sriram, > > > > > > > > > > > > > > > > > >Thanks! I think this is a very helpful summary. > > > > > > > > > > > > > > > > > >Let me try to address your point about passing in the serde > at > > > send > > > > > > > time. > > > > > > > > > > > > > > > > > >I think the first objection is really to the paired > key/value > > > > > serializer > > > > > > > > >interfaces. This leads to kind of a weird combinatorial > thing > > > where > > > > > you > > > > > > > > >would have an avro/avro serializer a string/avro > serializer, a > > > pb/pb > > > > > > > > >serializer, and a string/pb serializer, and so on. But your > > > proposal > > > > > > > would > > > > > > > > >work as well with separate serializers for key and value. > > > > > > > > > > > > > > > > > >I think the downside is just the one you call out--that this > > is > > > a > > > > > corner > > > > > > > > >case and you end up with two versions of all the apis to > > > support it. > > > > > > > This > > > > > > > > >also makes the serializer api more annoying to implement. I > > > think > > > > > the > > > > > > > > >alternative solution to this case and any other we can give > > > people > > > > > is > > > > > > > just > > > > > > > > >configuring ByteArraySerializer which gives you basically > the > > > api > > > > > that > > > > > > > you > > > > > > > > >have now with byte arrays. If this is incredibly common then > > > this > > > > > would > > > > > > > be > > > > > > > > >a silly solution, but I guess the belief is that these cases > > are > > > > > rare > > > > > > > and > > > > > > > > >a > > > > > > > > >really well implemented avro or json serializer should be > 100% > > > of > > > > > what > > > > > > > > >most > > > > > > > > >people need. > > > > > > > > > > > > > > > > > >In practice the cases that actually mix serialization types > > in a > > > > > single > > > > > > > > >stream are pretty rare I think just because the consumer > then > > > has > > > > > the > > > > > > > > >problem of guessing how to deserialize, so most of these > will > > > end up > > > > > > > with > > > > > > > > >at least some marker or schema id or whatever that tells you > > > how to > > > > > read > > > > > > > > >the data. Arguable this mixed serialization with marker is > > > itself a > > > > > > > > >serializer type and should have a serializer of its own... > > > > > > > > > > > > > > > > > >-Jay > > > > > > > > > > > > > > > > > >On Fri, Dec 5, 2014 at 3:48 PM, Sriram Subramanian < > > > > > > > > >srsubraman...@linkedin.com.invalid> wrote: > > > > > > > > > > > > > > > > > >> This thread has diverged multiple times now and it would > be > > > worth > > > > > > > > >> summarizing them. > > > > > > > > >> > > > > > > > > >> There seems to be the following points of discussion - > > > > > > > > >> > > > > > > > > >> 1. Can we keep the serialization semantics outside the > > > Producer > > > > > > > > >>interface > > > > > > > > >> and have simple bytes in / bytes out for the interface > (This > > > is > > > > > what > > > > > > > we > > > > > > > > >> have today). > > > > > > > > >> > > > > > > > > >> The points for this is to keep the interface simple and > > usage > > > > > easy to > > > > > > > > >> understand. The points against this is that it gets hard > to > > > share > > > > > > > common > > > > > > > > >> usage patterns around serialization/message validations > for > > > the > > > > > > > future. > > > > > > > > >> > > > > > > > > >> 2. Can we create a wrapper producer that does the > > > serialization > > > > > and > > > > > > > have > > > > > > > > >> different variants of it for different data formats? > > > > > > > > >> > > > > > > > > >> The points for this is again to keep the main API clean. > The > > > > > points > > > > > > > > >> against this is that it duplicates the API, increases the > > > surface > > > > > area > > > > > > > > >>and > > > > > > > > >> creates redundancy for a minor addition. > > > > > > > > >> > > > > > > > > >> 3. Do we need to support different data types per record? > > The > > > > > current > > > > > > > > >> interface (bytes in/bytes out) lets you instantiate one > > > producer > > > > > and > > > > > > > use > > > > > > > > >> it to send multiple data formats. There seems to be some > > > valid use > > > > > > > cases > > > > > > > > >> for this. > > > > > > > > >> > > > > > > > > >> I have still not seen a strong argument against not having > > > this > > > > > > > > >> functionality. Can someone provide their views on why we > > don't > > > > > need > > > > > > > this > > > > > > > > >> support that is possible with the current API? > > > > > > > > >> > > > > > > > > >> One possible approach for the per record serialization > would > > > be to > > > > > > > > >>define > > > > > > > > >> > > > > > > > > >> public interface SerDe<K,V> { > > > > > > > > >> public byte[] serializeKey(); > > > > > > > > >> > > > > > > > > >> public K deserializeKey(); > > > > > > > > >> > > > > > > > > >> public byte[] serializeValue(); > > > > > > > > >> > > > > > > > > >> public V deserializeValue(); > > > > > > > > >> } > > > > > > > > >> > > > > > > > > >> This would be used by both the Producer and the Consumer. > > > > > > > > >> > > > > > > > > >> The send APIs can then be > > > > > > > > >> > > > > > > > > >> public Future<RecordMetadata> send(ProducerRecord<K,V> > > > record); > > > > > > > > >> public Future<RecordMetadata> send(ProducerRecord<K,V> > > record, > > > > > > > Callback > > > > > > > > >> callback); > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> public Future<RecordMetadata> send(ProducerRecord<K,V> > > record, > > > > > > > > >>SerDe<K,V> > > > > > > > > >> serde); > > > > > > > > >> > > > > > > > > >> public Future<RecordMetadata> send(ProducerRecord<K,V> > > record, > > > > > > > > >>SerDe<K,V> > > > > > > > > >> serde, Callback callback); > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> A default SerDe can be set in the config. The producer > would > > > use > > > > > the > > > > > > > > >> default from the config if the non-serde send APIs are > used. > > > The > > > > > > > > >>downside > > > > > > > > >> to this approach is that we would need to have four > variants > > > of > > > > > Send > > > > > > > API > > > > > > > > >> for the Producer. > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> On 12/5/14 3:16 PM, "Jun Rao" <j...@confluent.io> wrote: > > > > > > > > >> > > > > > > > > >> >Jiangjie, > > > > > > > > >> > > > > > > > > > >> >The issue with adding the serializer in ProducerRecord is > > > that > > > > > you > > > > > > > > >>need to > > > > > > > > >> >implement all combinations of serializers for key and > > value. > > > So, > > > > > > > > >>instead > > > > > > > > >> >of > > > > > > > > >> >just implementing int and string serializers, you will > have > > > to > > > > > > > > >>implement > > > > > > > > >> >all 4 combinations. > > > > > > > > >> > > > > > > > > > >> >Adding a new producer constructor like Producer<K, > > > > > > > V>(KeySerializer<K>, > > > > > > > > >> >ValueSerializer<V>, Properties properties) can be useful. > > > > > > > > >> > > > > > > > > > >> >Thanks, > > > > > > > > >> > > > > > > > > > >> >Jun > > > > > > > > >> > > > > > > > > > >> >On Thu, Dec 4, 2014 at 10:33 AM, Jiangjie Qin > > > > > > > > >><j...@linkedin.com.invalid> > > > > > > > > >> >wrote: > > > > > > > > >> > > > > > > > > > >> >> > > > > > > > > >> >> I'm just thinking instead of binding serialization with > > > > > producer, > > > > > > > > >> >>another > > > > > > > > >> >> option is to bind serializer/deserializer with > > > > > > > > >> >> ProducerRecord/ConsumerRecord (please see the detail > > > proposal > > > > > > > below.) > > > > > > > > >> >> The arguments for this option is: > > > > > > > > >> >> A. A single producer could send different > message > > > > > types. > > > > > > > > >>There > > > > > > > > >> >>are > > > > > > > > >> >> several use cases in LinkedIn for per record serializer > > > > > > > > >> >> - In Samza, there are some in-stream > > > order-sensitive > > > > > > > control > > > > > > > > >> >> messages > > > > > > > > >> >> having different deserializer from other messages. > > > > > > > > >> >> - There are use cases which need support for > > > sending > > > > > both > > > > > > > > >>Avro > > > > > > > > >> >> messages > > > > > > > > >> >> and raw bytes. > > > > > > > > >> >> - Some use cases needs to deserialize some Avro > > > > > messages > > > > > > > into > > > > > > > > >> >> generic > > > > > > > > >> >> record and some other messages into specific record. > > > > > > > > >> >> B. In current proposal, the > > serializer/deserilizer > > > is > > > > > > > > >> >>instantiated > > > > > > > > >> >> according to config. Compared with that, binding > > serializer > > > > > with > > > > > > > > >> >> ProducerRecord and ConsumerRecord is less error prone. > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > > >> >> This option includes the following changes: > > > > > > > > >> >> A. Add serializer and deserializer interfaces > to > > > > > replace > > > > > > > > >> >>serializer > > > > > > > > >> >> instance from config. > > > > > > > > >> >> Public interface Serializer <K, V> { > > > > > > > > >> >> public byte[] serializeKey(K > > key); > > > > > > > > >> >> public byte[] serializeValue(V > > > value); > > > > > > > > >> >> } > > > > > > > > >> >> Public interface deserializer <K, V> { > > > > > > > > >> >> Public K deserializeKey(byte[] > > > key); > > > > > > > > >> >> public V > deserializeValue(byte[] > > > > > value); > > > > > > > > >> >> } > > > > > > > > >> >> > > > > > > > > >> >> B. Make ProducerRecord and ConsumerRecord > > abstract > > > > > class > > > > > > > > >> >> implementing > > > > > > > > >> >> Serializer <K, V> and Deserializer <K, V> respectively. > > > > > > > > >> >> Public abstract class ProducerRecord > <K, > > V> > > > > > > > > >>implements > > > > > > > > >> >> Serializer <K, V> > > > > > > > > >> >> {...} > > > > > > > > >> >> Public abstract class ConsumerRecord > <K, > > V> > > > > > > > > >>implements > > > > > > > > >> >> Deserializer <K, > > > > > > > > >> >> V> {...} > > > > > > > > >> >> > > > > > > > > >> >> C. Instead of instantiate the > > > serializer/Deserializer > > > > > from > > > > > > > > >> >>config, > > > > > > > > >> >> let > > > > > > > > >> >> concrete ProducerRecord/ConsumerRecord extends the > > abstract > > > > > class > > > > > > > and > > > > > > > > >> >> override the serialize/deserialize methods. > > > > > > > > >> >> > > > > > > > > >> >> Public class AvroProducerRecord extends > > > > > > > > >>ProducerRecord > > > > > > > > >> >> <String, > > > > > > > > >> >> GenericRecord> { > > > > > > > > >> >> ... > > > > > > > > >> >> @Override > > > > > > > > >> >> Public byte[] > serializeKey(String > > > key) > > > > > {Š} > > > > > > > > >> >> @Override > > > > > > > > >> >> public byte[] > > > > > serializeValue(GenericRecord > > > > > > > > >> >>value); > > > > > > > > >> >> } > > > > > > > > >> >> > > > > > > > > >> >> Public class AvroConsumerRecord extends > > > > > > > > >>ConsumerRecord > > > > > > > > >> >> <String, > > > > > > > > >> >> GenericRecord> { > > > > > > > > >> >> ... > > > > > > > > >> >> @Override > > > > > > > > >> >> Public K deserializeKey(byte[] > > > key) {Š} > > > > > > > > >> >> @Override > > > > > > > > >> >> public V > deserializeValue(byte[] > > > > > value); > > > > > > > > >> >> } > > > > > > > > >> >> > > > > > > > > >> >> D. The producer API changes to > > > > > > > > >> >> Public class KafkaProducer { > > > > > > > > >> >> ... > > > > > > > > >> >> > > > > > > > > >> >> Future<RecordMetadata> send > > > > > (ProducerRecord > > > > > > > > >><K, > > > > > > > > >> >>V> > > > > > > > > >> >> record) { > > > > > > > > >> >> ... > > > > > > > > >> >> K key = > > > > > > > > >>record.serializeKey(record.key); > > > > > > > > >> >> V value = > > > > > > > > >> >> record.serializedValue(record.value); > > > > > > > > >> >> BytesProducerRecord > > > > > > > > >>bytesProducerRecord > > > > > > > > >> >>= > > > > > > > > >> >> new > > > > > > > > >> >> BytesProducerRecord(topic, partition, key, value); > > > > > > > > >> >> ... > > > > > > > > >> >> } > > > > > > > > >> >> ... > > > > > > > > >> >> } > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > > >> >> We also had some brainstorm in LinkedIn and here are > the > > > > > feedbacks: > > > > > > > > >> >> > > > > > > > > >> >> If the community decide to add the serialization back > to > > > new > > > > > > > > >>producer, > > > > > > > > >> >> besides current proposal which changes new producer API > > to > > > be a > > > > > > > > >> >>template, > > > > > > > > >> >> there are some other options raised during our > > discussion: > > > > > > > > >> >> 1) Rather than change current new producer API, > > we > > > can > > > > > > > > >>provide a > > > > > > > > >> >> wrapper > > > > > > > > >> >> of current new producer (e.g. KafkaSerializedProducer) > > and > > > > > make it > > > > > > > > >> >> available to users. As there is value in the simplicity > > of > > > > > current > > > > > > > > >>API. > > > > > > > > >> >> > > > > > > > > >> >> 2) If we decide to go with tempalated new > > producer > > > API, > > > > > > > > >> >>according > > > > > > > > >> >> to > > > > > > > > >> >> experience in LinkedIn, it might worth considering to > > > > > instantiate > > > > > > > the > > > > > > > > >> >> serializer in code instead of from config so we can > avoid > > > > > runtime > > > > > > > > >>errors > > > > > > > > >> >> due to dynamic instantiation from config, which is more > > > error > > > > > > > prone. > > > > > > > > >>If > > > > > > > > >> >> that is the case, the producer API could be changed to > > > > > something > > > > > > > > >>like: > > > > > > > > >> >> producer = new Producer<K, > > > V>(KeySerializer<K>, > > > > > > > > >> >> ValueSerializer<V>) > > > > > > > > >> >> > > > > > > > > >> >> --Jiangjie (Becket) Qin > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > > >> >> On 11/24/14, 5:58 PM, "Jun Rao" <jun...@gmail.com> > > wrote: > > > > > > > > >> >> > > > > > > > > >> >> >Hi, Everyone, > > > > > > > > >> >> > > > > > > > > > >> >> >I'd like to start a discussion on whether it makes > sense > > > to > > > > > add > > > > > > > the > > > > > > > > >> >> >serializer api back to the new java producer. > Currently, > > > the > > > > > new > > > > > > > > >>java > > > > > > > > >> >> >producer takes a byte array for both the key and the > > > value. > > > > > While > > > > > > > > >>this > > > > > > > > >> >>api > > > > > > > > >> >> >is simple, it pushes the serialization logic into the > > > > > application. > > > > > > > > >>This > > > > > > > > >> >> >makes it hard to reason about what type of data is > being > > > sent > > > > > to > > > > > > > > >>Kafka > > > > > > > > >> >>and > > > > > > > > >> >> >also makes it hard to share an implementation of the > > > > > serializer. > > > > > > > For > > > > > > > > >> >> >example, to support Avro, the serialization logic > could > > be > > > > > quite > > > > > > > > >> >>involved > > > > > > > > >> >> >since it might need to register the Avro schema in > some > > > remote > > > > > > > > >>registry > > > > > > > > >> >> >and > > > > > > > > >> >> >maintain a schema cache locally, etc. Without a > > > serialization > > > > > api, > > > > > > > > >>it's > > > > > > > > >> >> >impossible to share such an implementation so that > > people > > > can > > > > > > > easily > > > > > > > > >> >> >reuse. > > > > > > > > >> >> >We sort of overlooked this implication during the > > initial > > > > > > > > >>discussion of > > > > > > > > >> >> >the > > > > > > > > >> >> >producer api. > > > > > > > > >> >> > > > > > > > > > >> >> >So, I'd like to propose an api change to the new > > producer > > > by > > > > > > > adding > > > > > > > > >> >>back > > > > > > > > >> >> >the serializer api similar to what we had in the old > > > producer. > > > > > > > > >> >>Specially, > > > > > > > > >> >> >the proposed api changes are the following. > > > > > > > > >> >> > > > > > > > > > >> >> >First, we change KafkaProducer to take generic types K > > > and V > > > > > for > > > > > > > the > > > > > > > > >> >>key > > > > > > > > >> >> >and the value, respectively. > > > > > > > > >> >> > > > > > > > > > >> >> >public class KafkaProducer<K,V> implements > > Producer<K,V> { > > > > > > > > >> >> > > > > > > > > > >> >> > public Future<RecordMetadata> > > send(ProducerRecord<K,V> > > > > > record, > > > > > > > > >> >> >Callback > > > > > > > > >> >> >callback); > > > > > > > > >> >> > > > > > > > > > >> >> > public Future<RecordMetadata> > > send(ProducerRecord<K,V> > > > > > > > record); > > > > > > > > >> >> >} > > > > > > > > >> >> > > > > > > > > > >> >> >Second, we add two new configs, one for the key > > > serializer and > > > > > > > > >>another > > > > > > > > >> >>for > > > > > > > > >> >> >the value serializer. Both serializers will default to > > the > > > > > byte > > > > > > > > >>array > > > > > > > > >> >> >implementation. > > > > > > > > >> >> > > > > > > > > > >> >> >public class ProducerConfig extends AbstractConfig { > > > > > > > > >> >> > > > > > > > > > >> >> > .define(KEY_SERIALIZER_CLASS_CONFIG, Type.CLASS, > > > > > > > > >> >> > >"org.apache.kafka.clients.producer.ByteArraySerializer", > > > > > > > > >> >>Importance.HIGH, > > > > > > > > >> >> >KEY_SERIALIZER_CLASS_DOC) > > > > > > > > >> >> > .define(VALUE_SERIALIZER_CLASS_CONFIG, Type.CLASS, > > > > > > > > >> >> > >"org.apache.kafka.clients.producer.ByteArraySerializer", > > > > > > > > >> >>Importance.HIGH, > > > > > > > > >> >> >VALUE_SERIALIZER_CLASS_DOC); > > > > > > > > >> >> >} > > > > > > > > >> >> > > > > > > > > > >> >> >Both serializers will implement the following > interface. > > > > > > > > >> >> > > > > > > > > > >> >> >public interface Serializer<T> extends Configurable { > > > > > > > > >> >> > public byte[] serialize(String topic, T data, > > boolean > > > > > isKey); > > > > > > > > >> >> > > > > > > > > > >> >> > public void close(); > > > > > > > > >> >> >} > > > > > > > > >> >> > > > > > > > > > >> >> >This is more or less the same as what's in the old > > > producer. > > > > > The > > > > > > > > >>slight > > > > > > > > >> >> >differences are (1) the serializer now only requires a > > > > > > > > >>parameter-less > > > > > > > > >> >> >constructor; (2) the serializer has a configure() and > a > > > > > close() > > > > > > > > >>method > > > > > > > > >> >>for > > > > > > > > >> >> >initialization and cleanup, respectively; (3) the > > > serialize() > > > > > > > method > > > > > > > > >> >> >additionally takes the topic and an isKey indicator, > > both > > > of > > > > > which > > > > > > > > >>are > > > > > > > > >> >> >useful for things like schema registration. > > > > > > > > >> >> > > > > > > > > > >> >> >The detailed changes are included in KAFKA-1797. For > > > > > > > completeness, I > > > > > > > > >> >>also > > > > > > > > >> >> >made the corresponding changes for the new java > consumer > > > api > > > > > as > > > > > > > > >>well. > > > > > > > > >> >> > > > > > > > > > >> >> >Note that the proposed api changes are incompatible > with > > > > > what's in > > > > > > > > >>the > > > > > > > > >> >> >0.8.2 branch. However, if those api changes are > > > beneficial, > > > > > it's > > > > > > > > >> >>probably > > > > > > > > >> >> >better to include them now in the 0.8.2 release, > rather > > > than > > > > > > > later. > > > > > > > > >> >> > > > > > > > > > >> >> >I'd like to discuss mainly two things in this thread. > > > > > > > > >> >> >1. Do people feel that the proposed api changes are > > > > > reasonable? > > > > > > > > >> >> >2. Are there any concerns of including the api changes > > in > > > the > > > > > > > 0.8.2 > > > > > > > > >> >>final > > > > > > > > >> >> >release? > > > > > > > > >> >> > > > > > > > > > >> >> >Thanks, > > > > > > > > >> >> > > > > > > > > > >> >> >Jun > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >