Linkedin currently just forces everyone using kafka to: 1. use avro as payload 2. add company-wide headers into their avro schema
this leads to problems with teams that would like to produce arbitrary blobs while still benefitting from company-wide infra. it also means every change to these fields is a company-wide version bump across literally hundreds of source repositories. moving to "standard" headers would allow much more rapid development iteration and roll-out of features and capabilities (not to mention enable arbitrary binary payloads) why not strings - space efficiency. some of our payloads are very small (<50 bytes) which would make the headers dominate the bandwidth. header ordering - its mostly a micro-optimization. coupled with the namespacing suggestion it would make it faster to check is any "system headers" (low ids) exist. this could later be combined with lazy parsing of the headers blob to minimize the overhead of server side plugins. On Wed, Oct 5, 2016 at 2:20 PM, Gwen Shapira <g...@confluent.io> wrote: > Since LinkedIn has some kind of wrapper thingy that adds the headers, > where they could have added them to Apache Kafka - I'm very curious to > hear what drove that decision and the pros/cons of managing the > headers outside Kafka itself. > > On Wed, Oct 5, 2016 at 2:16 PM, K Burstev <k.burs...@yandex.com> wrote: > > +1 > > > > This is a much needed feature, one I also have been waiting for, and that > > Kafka has been too long without. > > > > Especially since compaction the custom wrapper solution does not work > where > > you want a null payload but need the record to have headers. > > > > (It actually made me sign up to the mailing list so I could reply, as up > > until now I just browse the archives and forums) > > > > > > In general the KIP looks great. The solution address's all my core needs. > > Really hope this makes it to the next release after the current one. > > > > > > Questions: > > > > 1) Why not String,String headers? > > > > I assume reading the KIP it is for message size but surely compression > would > > greatly reduce this overhead with Strings. > > > > Many systems in the eco-sphere that kafka sits in, like JMS and Flume use > > String,String headers as such it would be easier to integrate with these > > frameworks/systems, as they can simply map across the headers. > > > > > > 2) Key Allocation > > > > If you do keep with int keys why not make the key allocation the proposed > > why is it an example. The example makes sense after all, and seems very > > sensible and clean. > > > > 3) Header Ordering > > > > I would avoid this as per your proposed between the two options and keep > > them un-ordered. > > There are many clients not maintained by the core community and also > > internally in many companies, that would need to implement it. Whilst > > trivial it complicates matters, its easier to just expect an unordered > set > > as will be converted to a map client side anyhow. > > > > Kostya > > > > > > > > On 04/10/2016 23:35, radai wrote: > >> > >> another potential benefit of headers is it would reduce the number of > API > >> changes required to support future features (as they could be > implemented > >> as plugins). > >> that would greatly accelerate the rate with which kafka can be extended. > >> > >> On Mon, Oct 3, 2016 at 12:46 PM, Michael Pearce <michael.pea...@ig.com> > >> wrote: > >> > >>> Opposite way around v4 instead of v3 ;) > >>> ________________________________________ > >>> From: Michael Pearce > >>> Sent: Monday, October 3, 2016 8:45 PM > >>> To: dev@kafka.apache.org > >>> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers > >>> > >>> Thanks guys for spotting this, i have updated KIP-82 to state v3 > instead > >>> of v4. > >>> > >>> Mike. > >>> ________________________________________ > >>> From: Becket Qin <becket....@gmail.com> > >>> Sent: Monday, October 3, 2016 6:18 PM > >>> To: dev@kafka.apache.org > >>> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers > >>> > >>> Yes, KIP-74 has already been checked in. The new FetchRequest/Response > >>> version should be V4. :) > >>> > >>> On Mon, Oct 3, 2016 at 10:14 AM, Sean McCauliff < > >>> smccaul...@linkedin.com.invalid> wrote: > >>> > >>>> Change to public interfaces: > >>>> > >>>> "Add ProduceRequest/ProduceResponse V3 which uses the new message > >>>> format. > >>>> Add FetchRequest/FetchResponse V3 which uses the new message format." > >>>> > >>>> When I look at org.apache.kafka.common.requests.FetchResponse on > >>>> master I see that there is already a version 3. Seems like this is > >>>> from a recent commit about implementing KIP-74. Do we need to > >>>> coordinate these changes with KIP-74? > >>>> > >>>> > >>>> "The serialisation of the [int, bye[]] header set will on the wire > >>>> using a strict format" bye[] -> byte[] > >>>> > >>>> Sean > >>>> -- > >>>> Sean McCauliff > >>>> Staff Software Engineer > >>>> Kafka > >>>> > >>>> smccaul...@linkedin.com > >>>> linkedin.com/in/sean-mccauliff-b563192 > >>>> > >>>> > >>>> On Fri, Sep 30, 2016 at 3:43 PM, radai <radai.rosenbl...@gmail.com> > >>> > >>> wrote: > >>>>> > >>>>> I think headers are a great idea. > >>>>> > >>>>> Right now, people who are trying to implement any sort of org-wide > >>>>> functionality like monitoring, tracing, profiling etc pretty much > have > >>> > >>> to > >>>>> > >>>>> define their own wrapper layers, which probably leads to everyone > >>>>> implementing their own variants of the same underlying functionality. > >>>>> > >>>>> I think a common base for headers would allow implementing a lot of > >>> > >>> this > >>>>> > >>>>> functionality only one in a way that different header-based > >>> > >>> capabilities > >>>>> > >>>>> could be shared and composed and open the door the a wide range of > >>>> > >>>> possible > >>>>> > >>>>> Kafka middleware that's simply impossible to write against the > current > >>>> > >>>> API. > >>>>> > >>>>> Here's a list of things that could be implemented as "plugins" on top > >>> > >>> of > >>>> > >>>> a > >>>>> > >>>>> header mechanism (full list here - > >>>>> https://cwiki.apache.org/confluence/display/KAFKA/A+ > >>>> > >>>> Case+for+Kafka+Headers). > >>>>> > >>>>> A lot of these already exist within LinkedIn and could for example be > >>>> > >>>> open > >>>>> > >>>>> sourced if Kafka had headers. I'm fairly certain the same is true in > >>>> > >>>> other > >>>>> > >>>>> organizations using Kafka > >>>>> > >>>>> > >>>>> > >>>>> On Thu, Sep 22, 2016 at 12:31 PM, Michael Pearce < > >>> > >>> michael.pea...@ig.com> > >>>>> > >>>>> wrote: > >>>>> > >>>>>> Hi All, > >>>>>> > >>>>>> > >>>>>> I would like to discuss the following KIP proposal: > >>>>>> > >>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP- > >>>>>> 82+-+Add+Record+Headers > >>>>>> > >>>>>> > >>>>>> > >>>>>> I have some initial ?drafts of roughly the changes that would be > >>> > >>> needed. > >>>>>> > >>>>>> This is no where finalized and look forward to the discussion > >>>> > >>>> especially as > >>>>>> > >>>>>> some bits I'm personally in two minds about. > >>>>>> > >>>>>> https://github.com/michaelandrepearce/kafka/tree/ > >>>> > >>>> kafka-headers-properties > >>>>>> > >>>>>> > >>>>>> > >>>>>> Here is a link to a alternative option mentioned in the kip but one > i > >>>>>> would personally would discard (disadvantages mentioned in kip) > >>>>>> > >>>>>> https://github.com/michaelandrepearce/kafka/tree/kafka-headers-full > ? > >>>>>> > >>>>>> > >>>>>> Thanks > >>>>>> > >>>>>> Mike > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> The information contained in this email is strictly confidential and > >>> > >>> for > >>>>>> > >>>>>> the use of the addressee only, unless otherwise indicated. If you > are > >>>> > >>>> not > >>>>>> > >>>>>> the intended recipient, please do not read, copy, use or disclose to > >>>> > >>>> others > >>>>>> > >>>>>> this message or any attachment. Please also notify the sender by > >>>> > >>>> replying > >>>>>> > >>>>>> to this email or by telephone (+44(020 7896 0011) and then delete > the > >>>> > >>>> email > >>>>>> > >>>>>> and any copies of it. Opinions, conclusion (etc) that do not relate > to > >>>> > >>>> the > >>>>>> > >>>>>> official business of this company shall be understood as neither > given > >>>> > >>>> nor > >>>>>> > >>>>>> endorsed by it. IG is a trading name of IG Markets Limited (a > company > >>>>>> registered in England and Wales, company number 04008957) and IG > Index > >>>>>> Limited (a company registered in England and Wales, company number > >>>>>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate > Hill, > >>>>>> London EC4R 2YA. Both IG Markets Limited (register number 195355) > and > >>> > >>> IG > >>>>>> > >>>>>> Index Limited (register number 114059) are authorised and regulated > by > >>>> > >>>> the > >>>>>> > >>>>>> Financial Conduct Authority. > >>>>>> > >>> The information contained in this email is strictly confidential and > for > >>> the use of the addressee only, unless otherwise indicated. If you are > not > >>> the intended recipient, please do not read, copy, use or disclose to > >>> others > >>> this message or any attachment. Please also notify the sender by > replying > >>> to this email or by telephone (+44(020 7896 0011) and then delete the > >>> email > >>> and any copies of it. Opinions, conclusion (etc) that do not relate to > >>> the > >>> official business of this company shall be understood as neither given > >>> nor > >>> endorsed by it. IG is a trading name of IG Markets Limited (a company > >>> registered in England and Wales, company number 04008957) and IG Index > >>> Limited (a company registered in England and Wales, company number > >>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, > >>> London EC4R 2YA. Both IG Markets Limited (register number 195355) and > IG > >>> Index Limited (register number 114059) are authorised and regulated by > >>> the > >>> Financial Conduct Authority. > >>> > > > > > > -- > Gwen Shapira > Product Manager | Confluent > 650.450.2760 | @gwenshap > Follow us: Twitter | blog >
