Hi All, In case you hadn't noticed re the compaction issue for non-null values i have created a separate KIP-87, if you could all contribute to its discussion would be much appreciated.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-87+-+Add+Compaction+Tombstone+Flag Secondly, focussing back on KIP-82, one of the actions agreed from the KIP call was for some additional alternative solution proposals on top of those already detailed in the KIP wiki and subsequent linked wiki pages by others in the group in the meeting. I haven't seen any activity on this, does this mean there isn't any further and everyone in hindsight actually thinks the current proposed solution in the KIP is the front runner? (i assume this isn't the case, just want to nudge everyone) Also just copying across the kip call thread to keep everything in one thread to avoid a divergence of the discussion into multiple threads. Cheers Mike ________________________________________ From: Mayuresh Gharat <gharatmayures...@gmail.com> Sent: Monday, October 24, 2016 6:17 PM To: dev@kafka.apache.org Subject: Re: Kafka KIP meeting Oct 19 at 11:00am PST I agree with Nacho. +1 for the KIP. Thanks, Mayuresh On Fri, Oct 21, 2016 at 11:46 AM, Nacho Solis <nso...@linkedin.com.invalid> wrote: > I think a separate KIP is a good idea as well. Note however that potential > decisions in this KIP could affect the other KIP. > > Nacho > > On Fri, Oct 21, 2016 at 10:23 AM, Jun Rao <j...@confluent.io> wrote: > > > Michael, > > > > Yes, doing a separate KIP to address the null payload issue for compacted > > topics is a good idea. > > > > Thanks, > > > > Jun > > > > On Fri, Oct 21, 2016 at 12:57 AM, Michael Pearce <michael.pea...@ig.com> > > wrote: > > > > > I had noted that what ever the solution having compaction based on null > > > payload was agreed isn't elegant. > > > > > > Shall we raise another kip to : as discussed propose using a attribute > > bit > > > for delete/compaction flag as well/or instead of null value and > updating > > > compaction logic to look at that delelete/compaction attribute > > > > > > I believe this is less contentious, so that at least we get that done > > > alleviating some concerns whilst the below gets discussed further? > > > > > > ________________________________________ > > > From: Jun Rao <j...@confluent.io> > > > Sent: Wednesday, October 19, 2016 8:56:52 PM > > > To: dev@kafka.apache.org > > > Subject: Re: Kafka KIP meeting Oct 19 at 11:00am PST > > > > > > The following are the notes from today's KIP discussion. > > > > > > > > > - KIP-82 - add record header: We agreed that there are use cases for > > > third-party vendors building tools around Kafka. We haven't reached > > the > > > conclusion whether the added complexity justifies the use cases. We > > will > > > follow up on the mailing list with use cases, container format > people > > > have > > > been using, and details on the proposal. > > > > > > > > > The video will be uploaded soon in https://cwiki.apache.org/ > > > confluence/display/KAFKA/Kafka+Improvement+Proposals . > > > > > > Thanks, > > > > > > Jun > > > > > > On Mon, Oct 17, 2016 at 10:49 AM, Jun Rao <j...@confluent.io> wrote: > > > > > > > Hi, Everyone., > > > > > > > > We plan to have a Kafka KIP meeting this coming Wednesday at 11:00am > > PST. > > > > If you plan to attend but haven't received an invite, please let me > > know. > > > > The following is the tentative agenda. > > > > > > > > Agenda: > > > > KIP-82: add record header > > > > > > > > Thanks, > > > > > > > > Jun > > > > > > > The information contained in this email is strictly confidential and > for > > > the use of the addressee only, unless otherwise indicated. If you are > not > > > the intended recipient, please do not read, copy, use or disclose to > > others > > > this message or any attachment. Please also notify the sender by > replying > > > to this email or by telephone (+44(020 7896 0011) and then delete the > > email > > > and any copies of it. Opinions, conclusion (etc) that do not relate to > > the > > > official business of this company shall be understood as neither given > > nor > > > endorsed by it. IG is a trading name of IG Markets Limited (a company > > > registered in England and Wales, company number 04008957) and IG Index > > > Limited (a company registered in England and Wales, company number > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, > > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and > IG > > > Index Limited (register number 114059) are authorised and regulated by > > the > > > Financial Conduct Authority. > > > > > > > > > -- > Nacho (Ignacio) Solis > Kafka > nso...@linkedin.com > -- -Regards, Mayuresh R. Gharat (862) 250-7125 ________________________________________ From: Michael Pearce <michael.pea...@ig.com> Sent: Monday, October 17, 2016 7:48 AM To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-82 - Add Record Headers Hi Jun, Sounds good. Look forward to the invite. Cheers, Mike ________________________________________ From: Jun Rao <j...@confluent.io> Sent: Monday, October 17, 2016 5:55:57 AM To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-82 - Add Record Headers Hi, Michael, We do have online KIP discussion meetings from time to time. How about we discuss this KIP Wed (Oct 19) at 11:00am PST? I will send out an invite (we typically do the meeting through Zoom and will post the video recording to Kafka wiki). Thanks, Jun On Wed, Oct 12, 2016 at 1:22 AM, Michael Pearce <michael.pea...@ig.com> wrote: > @Jay and Dana > > We have internally had a few discussions of how we may address this if we > had a common apache kafka message wrapper for headers that can be used > client side only to, and address the compaction issue. > I have detailed this solution separately and linked from the main KIP-82 > wiki. > > Here’s a direct link – > https://cwiki.apache.org/confluence/display/KAFKA/ > Headers+Value+Message+Wrapper > > We feel this solution though doesn’t manage to address all the use cases > being mentioned still and also has some compatibility drawbacks e.g. > backwards forwards compatibility especially on different language clients > Also we still require with this solution, as still need to address > compaction issue / tombstones, we need to make server side changes and as > many message/record version changes. > > We believe the proposed solution in KIP-82 does address all these needs > and is cleaner still, and more benefits. > Please have a read, and comment. Also if you have any improvements on the > proposed KIP-82 or an alternative solution/option your input is appreciated. > > @All > As Joel has mentioned to get this moving along, and able to discuss more > fluidly, it would be great if we can organize to meet up virtually online > e.g. webex or something. > I am aware, that the majority are based in America, myself is in the UK. > @Kostya I assume you’re in Eastern Europe or Russia based on your email > address (please correct this assumption), I hope the time difference isn’t > too much that the below would suit you if you wish to join > > Can I propose next Wednesday 19th October at 18:30 BST , 10:30 PST, 20:30 > MSK we try meetup online? > > Would this date/time suit the majority? > Also what is the preferred method? I can host via Adobe Connect style > webex (which my company uses) but it isn’t the best IMHO, so more than > happy to have someone suggest a better alternative. > > Best, > Mike > > > > > On 10/8/16, 7:26 AM, "Michael Pearce" <michael.pea...@ig.com> wrote: > > >> I agree with the critique of compaction not having a value. I think > we should consider fixing that directly. > > > Agree that the compaction issue is troubling: compacted "null" > deletes > are incompatible w/ headers that must be packed into the message > value. Are there any alternatives on compaction delete semantics that > could address this? The KIP wiki discussion I think mostly assumes > that compaction-delete is what it is and can't be changed/fixed. > > This KIP is about dealing with quite a few use cases and issues, > please see both the KIP use cases detailed by myself and also the > additional use cases wiki added by LinkedIn linked from the main KIP. > > The compaction is something that happily is addressed with headers, > but most defiantly isn't the sole reason or use case for them, headers > solves many issues and use cases. Thus their elegance and simplicity, and > why they're so common in transport mechanisms and so succesfull, as stated > like http, tcp, jms. > > ________________________________________ > From: Dana Powers <dana.pow...@gmail.com> > Sent: Friday, October 7, 2016 11:09 PM > To: dev@kafka.apache.org > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers > > > I agree with the critique of compaction not having a value. I think > we should consider fixing that directly. > > Agree that the compaction issue is troubling: compacted "null" deletes > are incompatible w/ headers that must be packed into the message > value. Are there any alternatives on compaction delete semantics that > could address this? The KIP wiki discussion I think mostly assumes > that compaction-delete is what it is and can't be changed/fixed. > > -Dana > > On Fri, Oct 7, 2016 at 1:38 PM, Michael Pearce <michael.pea...@ig.com> > wrote: > > > > Hi Jay, > > > > Thanks for the comments and feedback. > > > > I think its quite clear that if a problem keeps arising then it is > clear that it needs resolving, and addressing properly. > > > > Fair enough at linkedIn, and historically for the very first use > cases addressing this maybe not have been a big priority. But as Kafka is > now Apache open source and being picked up by many including my company, it > is clear and evident that this is a requirement and issue that needs to be > now addressed to address these needs. > > > > The fact in almost every transport mechanism including networking > layers in the enterprise ive worked in, there has always been headers i > think clearly shows their need and success for a transport mechanism. > > > > I understand some concerns with regards to impact for others not > needing it. > > > > What we are proposing is flexible solution that provides no overhead > on storage or network traffic layers if you chose not to use headers, but > does enable those who need or want it to use it. > > > > > > On your response to 1), there is nothing saying that it should be > put in any faster or without diligence and the same KIP process can still > apply for adding kafka-scope headers, having headers, just makes it easier > to add, without constant message and record changes. Timestamp is a clear > real example of actually what should be in a header (along with other > fields) but as such the whole message/record object needed to be changed to > add this, as will any further headers deemed needed by kafka. > > > > On response to 2) why within my company as a platforms designer > should i enforce that all teams use the same serialization for their > payloads? But what i do need is some core cross cutting concerns and > information addressed at my platform level and i don't want to impose onto > my development teams. This is the same argument why byte[] is the exposed > value and key because as a messaging platform you dont want to impose that > on my company. > > > > On response to 3) Actually this isnt true, there are many 3rd party > tools, we need to hook into our messaging flows that they only build onto > standardised interfaces as obviously the cost to have a custom > implementation for every company would be very high. > > APM tooling is a clear case in point, every enterprise level APM > tool on the market is able to stitch in transaction flow end 2 end over a > platform over http, jms because they can stitch in some "magic" data in a > uniform/standardised for the two mentioned they stitch this into the > headers. It is current form they cannot do this with Kafka. Providing a > standardised interface will i believe actually benefit the project as > commercial companies like these will now be able to plugin their tooling > uniformly, making it attractive and possible. > > > > Some of you other concerns as Joel mentions these are more > implementation details, that i think should be agreed upon, but i think can > be addressed. > > > > e.g. re your concern on the hashmap. > > it is more than possible not to have every record have to have a > hashmap unless it actually has a header (just like we have managed to do on > the serialized meesage) so if theres a concern on the in memory record size > for those using kafka without headers. > > > > On your second to last comment about every team choosing their own > format, actually we do want this a little, as very first mentioned, no we > don't want a free for all, but some freedom to have different serialization > has different benefits and draw backs across our business. I can iterate > these if needed. One of the use case for headers provided by linkedIn on > top of my KIP even shows where headers could be beneficial here as a header > could be used to detail which data format the message is serialized to > allowing me to consume different formats. > > > > Also we have some systems that we need to integrate that pretty near > impossible to wrap or touch their binary payloads, or we’re not allowed to > touch them (historic system, or inter/intra corporate) > > > > Headers really gives as a solution to provide a pluggable platform, > and standardisation that allows users to build platforms that adapt to > their needs. > > > > > > Cheers > > Mike > > > > > > ________________________________________ > > From: Jay Kreps <j...@confluent.io> > > Sent: Friday, October 7, 2016 4:45 PM > > To: dev@kafka.apache.org > > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers > > > > Hey guys, > > > > This discussion has come up a number of times and we've always > passed. > > > > One of things that has helped keep Kafka simple is not adding in new > > abstractions and concepts except when the proposal is really elegant > and > > makes things simpler. > > > > Consider three use cases for headers: > > > > 1. Kafka-scope: We want to add a feature to Kafka that needs a > > particular field. > > 2. Company-scope: You want to add a header to be shared by > everyone in > > your company. > > 3. World-wide scope: You are building a third party tool and want > to add > > some kind of header. > > > > For the case of (1) you should not use headers, you should just add > a field > > to the record format. Having a second way of encoding things doesn't > make > > sense. Occasionally people have complained that adding to the record > format > > is hard and it would be nice to just shove lots of things in > quickly. I > > think a better solution would be to make it easy to add to the record > > format, and I think we've made progress on that. I also think we > should be > > insanely focused on the simplicity of the abstraction and not adding > in new > > thingies often---we thought about time for years before adding a > timestamp > > and I guarantee you we would have goofed it up if we'd gone with the > > earlier proposals. These things end up being long term commitments > so it's > > really worth being thoughtful. > > > > For case (2) just use the body of the message. You don't need a > globally > > agreed on definition of headers, just standardize on a header you > want to > > include in the value in your company. Since this is just used by > code in > > your company having a more standard header format doesn't really > help you. > > In fact by using something like Avro you can define exactly the > types you > > want, the required header fields, etc. > > > > The only case that headers help is (3). This is a bit of a niche > case and i > > think is easily solved just making the reading and writing of given > > required fields pluggable to work with the header you have. > > > > A couple of specific problems with this proposal: > > > > 1. A global registry of numeric keys is super super ugly. This > seems > > silly compared to the Avro (or whatever) header solution which > gives more > > compact encoding, rich types, etc. > > 2. Using byte arrays for header values means they aren't really > > interoperable for case (3). E.g. I can't make a UI that displays > headers, > > or allow you to set them in config. To work with third party > headers, the > > only case I think this really helps, you need the union of all > > serialization schemes people have used for any tool. > > 3. For case (2) and (3) your key numbers are going to collide like > > crazy. I don't think a global registry of magic numbers > maintained either > > by word of mouth or checking in changes to kafka source is the > right thing > > to do. > > 4. We are introducing a new serialization primitive which makes > fields > > disappear conditional on the contents of other fields. This > breaks the > > whole serialization/schema system we have today. > > 5. We're adding a hashmap to each record > > 6. This proposes making the ProducerRecord and ConsumerRecord > mutable > > and adding setters and getters (which we try to avoid). > > > > For context on LinkedIn: I set up the system there, but it may have > changed > > since i left. The header is maintained with the record schemas in > the avro > > schema registry and is required for all records. Essentially all > messages > > must have a field named "header" of type EventHeader which is itself > a > > record schema with a handful of fields (time, host, etc). The header > > follows the same compatibility rules as other avro fields, so it can > be > > evolved in a compatible way gradually across apps. Avro is typed and > > doesn't require deserializing the full record to read the header. The > > header information is (timestamp, host, etc) is important and needs > to > > propagate into other systems like Hadoop which don't have a concept > of > > headers for records, so I doubt it could move out of the value in > any case. > > Not allowing teams to chose a data format other than avro was > considered a > > feature, not a bug, since the whole point was to be able to share > data, > > which doesn't work if every team chooses their own format. > > > > I agree with the critique of compaction not having a value. I think > we > > should consider fixing that directly. > > > > -Jay > > > > On Thu, Sep 22, 2016 at 12:31 PM, Michael Pearce < > michael.pea...@ig.com> > > wrote: > > > >> Hi All, > >> > >> > >> I would like to discuss the following KIP proposal: > >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP- > >> 82+-+Add+Record+Headers > >> > >> > >> > >> I have some initial ?drafts of roughly the changes that would be > needed. > >> This is no where finalized and look forward to the discussion > especially as > >> some bits I'm personally in two minds about. > >> > >> https://github.com/michaelandrepearce/kafka/tree/ > kafka-headers-properties > >> > >> > >> > >> Here is a link to a alternative option mentioned in the kip but one > i > >> would personally would discard (disadvantages mentioned in kip) > >> > >> https://github.com/michaelandrepearce/kafka/tree/kafka-headers-full > ? > >> > >> > >> Thanks > >> > >> Mike > >> > >> > >> > >> > >> > >> The information contained in this email is strictly confidential > and for > >> the use of the addressee only, unless otherwise indicated. If you > are not > >> the intended recipient, please do not read, copy, use or disclose > to others > >> this message or any attachment. Please also notify the sender by > replying > >> to this email or by telephone (+44(020 7896 0011) and then delete > the email > >> and any copies of it. Opinions, conclusion (etc) that do not relate > to the > >> official business of this company shall be understood as neither > given nor > >> endorsed by it. IG is a trading name of IG Markets Limited (a > company > >> registered in England and Wales, company number 04008957) and IG > Index > >> Limited (a company registered in England and Wales, company number > >> 01190902). Registered address at Cannon Bridge House, 25 Dowgate > Hill, > >> London EC4R 2YA. Both IG Markets Limited (register number 195355) > and IG > >> Index Limited (register number 114059) are authorised and regulated > by the > >> Financial Conduct Authority. > >> > > The information contained in this email is strictly confidential and > for the use of the addressee only, unless otherwise indicated. If you are > not the intended recipient, please do not read, copy, use or disclose to > others this message or any attachment. Please also notify the sender by > replying to this email or by telephone (+44(020 7896 0011) and then delete > the email and any copies of it. Opinions, conclusion (etc) that do not > relate to the official business of this company shall be understood as > neither given nor endorsed by it. IG is a trading name of IG Markets > Limited (a company registered in England and Wales, company number > 04008957) and IG Index Limited (a company registered in England and Wales, > company number 01190902). Registered address at Cannon Bridge House, 25 > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number > 195355) and IG Index Limited (register number 114059) are authorised and > regulated by the Financial Conduct Authority. > > The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.