I have some practical experience with this from past projects. My opinion is that if you are going to have dozens or hundreds of message types you definitely need an intermediate data definition format like pb or avro or thrift. However, the downside is that these intermediate forms have spotty language support and create somewhat awkward apis. Also, as Jun mentions they typically also do involve another round of byte copying.
For our case since we only need to have a handful of request apis, just standardizing the format rather than the intermediate definition language isn't much harder and gives broader compatibility. Parsing two or three formats is probably not the biggest burden in implementing the client. Right now, imo, the biggest burden is the thickness of the consumer protocol which requires a lot of client side logic. We are definitely interested in thinning this out and moving that logic to the server if possible to reduce the burden on client implementation. -Jay On Wed, Sep 19, 2012 at 5:37 PM, Jun Rao <jun...@gmail.com> wrote: > One of the difficulties in adopting something like avro/thrift is that we > optimize sending messages from brokers to consumers by using the sendfile > api (sending data directly from file channel to socket). avro/thrift would > require copying messages into the application buffer first. > > Thanks, > > Jun > > On Wed, Sep 19, 2012 at 11:32 AM, Vito Sabella <vsabe...@msn.com> wrote: > > > Thanks! Are there any plans to move to Thrift/Avro/similar/... to make > > writing and maintaining new clients in other languages easier? Would > that > > be a desired contribution? > > > > -Vito > > > > ---------------------------------------- > > > Date: Wed, 19 Sep 2012 11:26:39 -0700 > > > Subject: Re: Kafka 0.8 > > > From: neha.narkh...@gmail.com > > > To: kafka-dev@incubator.apache.org > > > > > > The design of Kafka 0.8 is here - > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Replication > > > The client protocol changes are here - > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/New+Wire+Format+Proposal > > > > > > Thanks, > > > Neah > > > > > > On Wed, Sep 19, 2012 at 11:16 AM, Vito Sabella <vsabe...@msn.com> > wrote: > > > > Is there a quick place where I can get up to speed on the client > > protocol and other changes in 0.8? > > > > > > > > Thanks > > > > Vito > > >
