Sorry for the delay, you should have access now. Ismael
On Thu, Sep 22, 2016 at 8:15 AM, Michael Pearce <michael.pea...@ig.com> wrote: > Hi again, > > Sorry to be nudging this, but it seems I'm still unable to create a page > in the KIP Proposal area. Just don't want to be forgotten about between all > the emails. > > Cheers > Mike > ________________________________________ > From: Michael Pearce <michael.pea...@ig.com> > Sent: Monday, September 19, 2016 6:19 PM > To: dev@kafka.apache.org > Subject: Re: ProducerRecord/Consumer MetaData/Headers > > Hi Again, > > I went to the wiki this afternoon to start writing it up, and seems I > cannot create a page under the KIP area still. If someone could assist. > > Cheers > Mike > > > On 9/18/16, 7:07 PM, "Michael Pearce" <michael.pea...@ig.com> wrote: > > Hi Ismaelj > > Thanks, my wiki user is michael.andre.pearce. > > Re the link thanks again, actually indeed we started off trying to do > this after we lost the ability to use the key to hold metadata once the > compaction feature came, but it actually abusing the payload isn't imo a > great solution, and has some issues that cannot be overcome and stopping us > from using in some of our data / message flows. As such I think a solution > in the broker/message/client needs to be made and formalised. Also then an > ecosystems of tools could rely on such. > > I will add all details in KIP proposal, once I have access. > > Cheers > Mike > > > ________________________________________ > From: isma...@gmail.com <isma...@gmail.com> on behalf of Ismael Juma < > ism...@juma.me.uk> > Sent: Sunday, September 18, 2016 9:01:22 AM > To: dev@kafka.apache.org > Subject: Re: ProducerRecord/Consumer MetaData/Headers > > Hi Mike, > > If you give me your wiki user name, I can give you the required > permissions > to post a KIP. This is definitely a big change and there is no clear > consensus if changing the Kafka message format is the right way (it > would > be good not to pay the cost if you don't need it) or if it should be > done > via schemas, for example. Gwen shared some thoughts in the following > message: > > http://search-hadoop.com/m/uyzND1OXS8EoGCU2 > > Ismael > > On Sun, Sep 18, 2016 at 7:11 AM, Michael Pearce <michael.pea...@ig.com > > > wrote: > > > Hi All, (again) > > > > If it helps the discussion, and almost ready patch implementing this > is > > available here: > > > > https://github.com/michaelandrepearce/kafka > > > > > > The biggest/most core change is obviously the kafka.message.Message > object. > > > > > > > > Some key bits in this implementation is the server side, and > submodules > > (connect, mirrormaker, streams) all updated to be aware of the new > “headers” > > > > > > > > As a big API change have to use new feature on the client side, you > use > > the ConsumerRecord and ProducerRecord (which now extend the new > Enhanced > > versions) for K,V records without any code changes, to use the > headers you > > use the Enhanced versions HeadersConsumerRecord and > HeadersProducerRecord. > > This was needed to avoid causing code compilation failure just by > > upgrading. If the patch were accepted I would imagine it as a way to > > transition. > > > > > > > > I am guessing this needs a KIP rather than just myself raising a > JIRA as > > fairly substantial api change but unsure whom can raise these so > assistance > > in the process would be gratefully accepted.. > > > > > > > > Cheers > > > > Mike > > > > > > > > > > > > > > From: Michael Pearce <michael.pea...@ig.com> > > Date: Saturday, September 17, 2016 at 6:40 AM > > To: "dev@kafka.apache.org" <dev@kafka.apache.org> > > Subject: ProducerRecord/Consumer MetaData/Headers > > > > Hi All, > > > > First of all apologies if this has been previously discussed I have > just > > joined the mail list (I cannot find a JIRA or KIP related nor via > good old > > google search) > > > > In our company we are looking to replace some of our more traditional > > message flows with Kafka. > > > > One thing we have found lacking though compared with most messaging > > systems is the ability to set header/metadata separate from our > payload. We > > did think about the key, but as this is used for compaction we > cannot have > > changing values here which metadata/header values will obviously be. > > > > e.g. these headers/metadata are useful for audit data or platform > data > > that is not business payload related e.g. storing > > the clientId that generated the message, correlation id of a > > request/response, cluster id where the message was generate (case for > > MirrorMakers), message uuid etc this list is endless. > > > > We would like to propose extending the Record from > > ProducerRecord/ConsumerRecord<K, V> to > ProducerRecord/ConsumerRecord<K,V,M> > > where M is metadata/header again being like the key and value a > simple > > byte[] so that it is completely upto the end users how to serialize / > > deserialize it. > > > > What our people’s thoughts? > > Any other ideas how to add headers/metadata. > > > > How can I progress this? > > > > Cheers > > Mike > > The information contained in this email is strictly confidential and > for > > the use of the addressee only, unless otherwise indicated. If you > are not > > the intended recipient, please do not read, copy, use or disclose to > others > > this message or any attachment. Please also notify the sender by > replying > > to this email or by telephone (+44(020 7896 0011) and then delete > the > > email and any copies of it. Opinions, conclusion (etc) that do not > relate > > to the official business of this company shall be understood as > neither > > given nor endorsed by it. IG is a trading name of IG Markets Limited > (a > > company registered in England and Wales, company number 04008957) > and IG > > Index Limited (a company registered in England and Wales, company > number > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate > Hill, > > London EC4R 2YA. Both IG Markets Limited (register number 195355) > and IG > > Index Limited (register number 114059) are authorised and regulated > by the > > Financial Conduct Authority. > > > The information contained in this email is strictly confidential and > for the use of the addressee only, unless otherwise indicated. If you are > not the intended recipient, please do not read, copy, use or disclose to > others this message or any attachment. Please also notify the sender by > replying to this email or by telephone (+44(020 7896 0011) and then delete > the email and any copies of it. Opinions, conclusion (etc) that do not > relate to the official business of this company shall be understood as > neither given nor endorsed by it. IG is a trading name of IG Markets > Limited (a company registered in England and Wales, company number > 04008957) and IG Index Limited (a company registered in England and Wales, > company number 01190902). Registered address at Cannon Bridge House, 25 > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number > 195355) and IG Index Limited (register number 114059) are authorised and > regulated by the Financial Conduct Authority. > > >
