Does this mean that starting with V4 requests we would allow storing null messages in compacted topics? The KIP should probably clarify the behavior for null messages where the tombstone flag is not net.
On Wed, Oct 26, 2016 at 1:32 AM Magnus Edenhill <mag...@edenhill.se> wrote: > 2016-10-25 21:36 GMT+02:00 Nacho Solis <nso...@linkedin.com.invalid>: > > > I think you probably require a MagicByte bump if you expect correct > > behavior of the system as a whole. > > > > From a client perspective you want to make sure that when you deliver a > > message that the broker supports the feature you're expecting > > (compaction). So, depending on the behavior of the broker on > encountering > > a previously undefined bit flag I would suggest making some change to > make > > certain that flag-based compaction is supported. I'm going to guess that > > the MagicByte would do this. > > > > I dont believe this is needed since it is already attributed through the > request's API version. > > Producer: > * if a client sends ProduceRequest V4 then attributes.bit5 indicates a > tombstone > * if a clients sends ProduceRequest <V4 then attributes.bit5 is ignored > and value==null indicates a tombstone > * in both cases the on-disk messages are stored with attributes.bit5 (I > assume?) > > Consumer: > * if a clients sends FetchRequest V4 messages are sendfile():ed directly > from disk (with attributes.bit5) > * if a client sends FetchRequest <V4 messages are slowpathed and > translated from attributes.bit5 to value=null as required. > > > That's my understanding anyway, please correct me if I'm wrong. > > /Magnus > > > > > On Tue, Oct 25, 2016 at 10:17 AM, Magnus Edenhill <mag...@edenhill.se> > > wrote: > > > > > It is safe to assume that a previously undefined attributes bit will be > > > unset in protocol requests from existing clients, if not, such a client > > is > > > already violating the protocol and needs to be fixed. > > > > > > So I dont see a need for a MagicByte bump, both broker and client has > the > > > information it needs to construct or parse the message according to > > request > > > version. > > > > > > > > > 2016-10-25 18:48 GMT+02:00 Michael Pearce <michael.pea...@ig.com>: > > > > > > > Hi Magnus, > > > > > > > > I was wondering if I even needed to change those also, as technically > > > > we’re just making use of a non used attribute bit, but im not 100% > that > > > it > > > > be always false currently. > > > > > > > > If someone can say 100% it will already be set false with current and > > > > historic bit wise masking techniques used over the time, we could do > > away > > > > with both, and simply just start to use it. Unfortunately I don’t > have > > > that > > > > historic knowledge so was hoping it would be flagged up in this > > > discussion > > > > thread ☺ > > > > > > > > Cheers > > > > Mike > > > > > > > > On 10/25/16, 5:36 PM, "Magnus Edenhill" <mag...@edenhill.se> wrote: > > > > > > > > Hi Michael, > > > > > > > > With the version bumps for Produce and Fetch requests, do you > > really > > > > need > > > > to bump MagicByte too? > > > > > > > > Regards, > > > > Magnus > > > > > > > > > > > > 2016-10-25 18:09 GMT+02:00 Michael Pearce <michael.pea...@ig.com > >: > > > > > > > > > Hi All, > > > > > > > > > > I would like to discuss the following KIP proposal: > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > > > 87+-+Add+Compaction+Tombstone+Flag > > > > > > > > > > This is off the back of the discussion on KIP-82 / KIP meeting > > > > where it > > > > > was agreed to separate this issue and feature. See: > > > > > http://mail-archives.apache.org/mod_mbox/kafka-dev/201610. > > > > > mbox/%3cCAJS3ho8OcR==EcxsJ8OP99pD2hz=iiGecWsv- > > > > > EZsBsNyDcKr=g...@mail.gmail.com%3e > > > > > > > > > > Thanks > > > > > Mike > > > > > > > > > > The information contained in this email is strictly > confidential > > > and > > > > for > > > > > the use of the addressee only, unless otherwise indicated. If > you > > > > are not > > > > > the intended recipient, please do not read, copy, use or > disclose > > > to > > > > others > > > > > this message or any attachment. Please also notify the sender > by > > > > replying > > > > > to this email or by telephone (+44(020 7896 0011) and then > delete > > > > the email > > > > > and any copies of it. Opinions, conclusion (etc) that do not > > relate > > > > to the > > > > > official business of this company shall be understood as > neither > > > > given nor > > > > > endorsed by it. IG is a trading name of IG Markets Limited (a > > > company > > > > > registered in England and Wales, company number 04008957) and > IG > > > > Index > > > > > Limited (a company registered in England and Wales, company > > number > > > > > 01190902). Registered address at Cannon Bridge House, 25 > Dowgate > > > > Hill, > > > > > London EC4R 2YA. Both IG Markets Limited (register number > 195355) > > > > and IG > > > > > Index Limited (register number 114059) are authorised and > > regulated > > > > by the > > > > > Financial Conduct Authority. > > > > > > > > > > > > > > > > > The information contained in this email is strictly confidential and > > for > > > > the use of the addressee only, unless otherwise indicated. If you are > > not > > > > the intended recipient, please do not read, copy, use or disclose to > > > others > > > > this message or any attachment. Please also notify the sender by > > replying > > > > to this email or by telephone (+44(020 7896 0011) and then delete the > > > email > > > > and any copies of it. Opinions, conclusion (etc) that do not relate > to > > > the > > > > official business of this company shall be understood as neither > given > > > nor > > > > endorsed by it. IG is a trading name of IG Markets Limited (a company > > > > registered in England and Wales, company number 04008957) and IG > Index > > > > Limited (a company registered in England and Wales, company number > > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate > Hill, > > > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and > > IG > > > > Index Limited (register number 114059) are authorised and regulated > by > > > the > > > > Financial Conduct Authority. > > > > > > > > > > > > > > > -- > > Nacho (Ignacio) Solis > > Kafka > > nso...@linkedin.com > > >
