@Kostya Regarding "To get around this we have an awful *cough* solution whereby we have to send our message wrapper with the headers and null content, and then we have an application that has to consume from all the compacted topics and when it sees this message it produces back in a null payload record to make the broker compact it out."
---> This has a race condition, right? Suppose the producer produces a message with headers and null content at time To to Kafka. Then the producer, at time To + 1, sends another message with headers and actual content to Kafka. What we expect is that the application that is consuming and then producing same message with null payload should happen at time To + 0.5, so that the message at To + 1 is not deleted. But there is no guarantee here. If the null payload goes in to Kafka at time To + 2, then essentially you loose the second message produced by the producer at time To + 1. Thanks, Mayuresh On Wed, Oct 5, 2016 at 6:13 PM, Joel Koshy <jjkosh...@gmail.com> wrote: > @Nacho > > > > - Brokers can't see the headers (part of the "V" black box)> > > > > > > (Also, it would be nice if we had a way to access the headers from the > > > brokers, something that is not trivial at this time with the current > > broker > > > architecture). > > > > > > I think this can be addressed with broker interceptors which we touched on > in KIP-42 > <https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 42%3A+Add+Producer+and+Consumer+Interceptors> > . > > @Gwen > > You are right that the wrapper thingy “works”, but there are some drawbacks > that Nacho and Radai have covered in detail that I can add a few more > comments to. > > At LinkedIn, we *get by* without the proposed Kafka record headers by > dumping such metadata in one or two places: > > - Most of our applications use Avro, so for the most part we can use an > explicit header field in the Avro schema. Topic owners are supposed to > include this header in their schemas. > - A prefix to the payload that primarily contains the schema’s ID so we > can deserialize the Avro. (We could use this for other use-cases as > well - > i.e., move some of the above into this prefix blob.) > > Dumping headers in the Avro schema pollutes the application’s data model > with data/service-infra-related fields that are unrelated to the underlying > topic; and forces the application to deserialize the entire blob whether or > not the headers are actually used. Conversely from an infrastructure > perspective, we would really like to not touch any application data. Our > infiltration of the application’s schema is a major reason why many at > LinkedIn sometimes assume that we (Kafka folks) are the shepherds for all > things Avro :) > > Another drawback is that all this only works if everyone in the > organization is a good citizen and includes the header; and uses our > wrapper libraries - which is a good practice IMO - but may not always be > easy for open source projects that wish to directly use the Apache > producer/client. If instead we allow these headers to be inserted via > suitable interceptors outside the application payloads it would remove such > issues of separation in the data model and choice of clients. > > Radai has enumerated a number of use-cases > <https://cwiki.apache.org/confluence/display/KAFKA/A+ > Case+for+Kafka+Headers> > and > I’m sure the broader community will have a lot more to add. The feature as > such would enable an ecosystem of plugins from different vendors that users > can mix and match in their data pipelines without requiring any specific > payload formats or client libraries. > > Thanks, > > Joel > > > > > > > > > > > > On Wed, Oct 5, 2016 at 2:20 PM, Gwen Shapira <g...@confluent.io> > wrote: > > > > > > > Since LinkedIn has some kind of wrapper thingy that adds the headers, > > > > where they could have added them to Apache Kafka - I'm very curious > to > > > > hear what drove that decision and the pros/cons of managing the > > > > headers outside Kafka itself. > > > > > > > -- -Regards, Mayuresh R. Gharat (862) 250-7125
