Hi Ismael, Thanks for the explanation. Specially I like this part where in you mentioned we can get rid of the older null value support for log compaction later on, here : We can't change semantics of the message format without having a long transition period. And we can't rely on people reading documentation or acting on a warning for something so fundamental. As such, my take is that we need to bump the magic byte. The good news is that we don't have to support all versions forever. We have said that we will support direct upgrades for 2 years. That means that message format version n could, in theory, be removed 2 years after the it's introduced.
Just a heads up, I would like to mention that even without bumping magic byte, we will *NOT* loose zero copy as in the client(x+1) in my explanation above will convert internally a null value to have a tombstone bit set and a tombstone bit set to have a null value automatically internally and by the time we move to version (x+2), the clients would have upgraded. Obviously if we support a request from consumer(x), we will loose zero copy but that is the same case with magic byte. But if magic byte bump makes life easier for transition for the above reasons that you explained, I am OK with it since we are going to meet the end goal down the road :) On a side note can we update the doc here on magic byte to say that "*it should be bumped whenever the message format is changed or the interpretation of message format (usage of the reserved bits as well) is changed*". Hi Michael, Here is the update plan that we discussed offline yesterday : Currently the magic-byte which corresponds to the "message.format.version" is set to 1. 1) On broker it will be set to 1 initially. 2) When a producer client sends a message with magic-byte = 2, since the broker is on magic-byte = 1, we will down convert it, which means if the tombstone bit is set, the value will be set to null. A consumer understanding magic-byte = 1, will still work with this. A consumer working with magic-byte =2 will also be able to understand this, since it understands the tombstone. Now there is still the question of supporting a non-tombstone and null value from producer client with magic-byte = 2.* (I am not sure if we should support this. Ismael/Becket can comment here)* 3) When almost all the clients have upgraded, the message.format.version on the broker can be changed to 2, where in the down conversion in the above step will not happen. If at this point we get a consumer request from a older consumer, we might have to down convert where in we loose zero copy, but these cases should be rare. Becket can you review this plan and add more details if I have missed/wronged something, before we put it on KIP. Thanks, Mayuresh On Wed, Nov 16, 2016 at 11:07 PM, Michael Pearce <michael.pea...@ig.com> wrote: > Thanks guys, for discussing this offline and getting some consensus. > > So its clear for myself and others what is proposed now (i think i > understand, but want to make sure) > > Could i ask either directly update the kip to detail the migration > strategy, or (re-)state your offline discussed and agreed migration > strategy based on a magic byte is in this thread. > > > The main original driver for the KIP was to support compaction where value > isn't null, based off the discussions on KIP-82 thread. > > We should be able to support non-tombstone + null value by the completion > of the KIP, as we noted when discussing this kip, having logic based on a > null value isn't very clean and also separates the concerns. > > As discussed already though we can split this into KIP-87a and KIP-87b > > Where we look to deliver KIP-87a on a compacted topic (to address the > immediate issues) > * tombstone + null value > * tombstone + non-null value > * non-tombstone + non-null value > > Then we can discuss once KIP-87a is completed options later and how we > support the second part KIP-87b to deliver: > * non-tombstone + null value > > Cheers > Mike > > > > ________________________________________ > From: Becket Qin <becket....@gmail.com> > Sent: Thursday, November 17, 2016 1:43 AM > To: dev@kafka.apache.org > Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag > > Renu, Mayuresh and I had an offline discussion, and following is a brief > summary. > > 1. We agreed that not bumping up magic value may result in losing zero copy > during migration. > 2. Given that bumping up magic value is almost free and has benefit of > avoiding potential performance issue. It is probably worth doing. > > One issue we still need to think about is whether we want to support a > non-tombstone message with null value. > Currently it is not supported by Kafka. If we allow a non-tombstone null > value message to exist after KIP-87. The problem is that such message will > not be supported by the consumers prior to KIP-87. Because a null value > will always be interpreted to a tombstone. > > One option is that we keep the current way, i.e. do not support such > message. It would be good to know if there is a concrete use case for such > message. If there is not, we can probably just not support it. > > Thanks, > > JIangjie (Becket) Qin > > > > On Wed, Nov 16, 2016 at 1:28 PM, Mayuresh Gharat < > gharatmayures...@gmail.com > > wrote: > > > Hi Ismael, > > > > This is something I can think of for migration plan: > > So the migration plan can look something like this, with up conversion : > > > > 1) Currently lets say we have Broker at version x. > > 2) Currently we have clients at version x. > > 3) a) We move the version to Broker(x+1) : supports both tombstone and > null > > for log compaction. > > b) We upgrade the client to version client(x+1) : if in the producer > > client(x+1) the value is set to null, we will automatically set the > > Tombstone bit internally. If the producer client(x+1) sets the tombstone > > itself, well and good. For producer client(x), the broker will up convert > > to have the tombstone bit. Broker(x+1) is supporting both. Consumer > > client(x+1) will be aware of this and should be able to handle this. For > > consumer client(x) we will down convert the message on the broker side. > > c) At this point we will have to specify a warning or clearly specify > > in docs that this behavior is about to be changed for log compaction. > > 4) a) In next release of the Broker(x+2), we say that only Tombstone is > > used for log compaction on the Broker side. Clients(x+1) still is > > supported. > > b) We upgrade the client to version client(x+2) : if value is set to > > null, tombstone will not be set automatically. The client will have to > call > > setTombstone() to actually set the tombstone. > > > > We should compare this migration plan with the migration plan for magic > > byte bump and do whatever looks good. > > I am just worried that if we go down magic byte route, unless I am > missing > > something, it sounds like kafka will be stuck with supporting both null > > value and tombstone bit for log compaction for life long, which does not > > look like a good end state. > > > > Thanks, > > > > Mayuresh > > > > > > > > > > On Wed, Nov 16, 2016 at 9:32 AM, Mayuresh Gharat < > > gharatmayures...@gmail.com > > > wrote: > > > > > Hi Ismael, > > > > > > That's a very good point which I might have not considered earlier. > > > > > > Here is a plan that I can think of: > > > > > > Stage 1) The broker from now on, up converts the message to have the > > > tombstone marker. The log compaction thread does log compaction based > on > > > both null and tombstone marker. This is our transition period. > > > Stage 2) The next release we only say that log compaction is based on > > > tombstone marker. (Open source kafka makes this as a policy). By this > > time, > > > the organization which is moving to this release will be sure that they > > > have gone through the entire transition period. > > > > > > My only goal of doing this is that Kafka clearly specifies the end > state > > > about what log compaction means (is it null value or a tombstone > marker, > > > but not both). > > > > > > What do you think? > > > > > > Thanks, > > > > > > Mayuresh > > > . > > > > > > On Wed, Nov 16, 2016 at 9:17 AM, Ismael Juma <ism...@juma.me.uk> > wrote: > > > > > >> One comment below. > > >> > > >> On Wed, Nov 16, 2016 at 5:08 PM, Mayuresh Gharat < > > >> gharatmayures...@gmail.com > > >> > wrote: > > >> > > >> > - If we don't bump up the magic byte, on the broker side, the > > broker > > >> > will always have to look at both tombstone bit and the value when > > do > > >> the > > >> > compaction. Assuming we do not bump up the magic byte, > > >> > imagine the broker sees a message which does not have a tombstone > > bit > > >> > set. The broker does not know when the message was produced (i.e. > > >> > whether > > >> > the message has been up converted or not), it has to take a > further > > >> > look at > > >> > the value to see if it is null or not in order to determine if it > > is > > >> a > > >> > tombstone. The same logic has to be put on the consumer as well > > >> because > > >> > the > > >> > consumer does not know if the message has been up converted or > not. > > >> > - If we upconvert while appending, this is not the case, > right? > > >> > > >> > > >> If I understand you correctly, this is not sufficient because the log > > may > > >> have messages appended before it was upgraded to include KIP-87. > > >> > > >> Ismael > > >> > > > > > > > > > > > > -- > > > -Regards, > > > Mayuresh R. Gharat > > > (862) 250-7125 > > > > > > > > > > > -- > > -Regards, > > Mayuresh R. Gharat > > (862) 250-7125 > > > The information contained in this email is strictly confidential and for > the use of the addressee only, unless otherwise indicated. If you are not > the intended recipient, please do not read, copy, use or disclose to others > this message or any attachment. Please also notify the sender by replying > to this email or by telephone (+44(020 7896 0011) and then delete the email > and any copies of it. Opinions, conclusion (etc) that do not relate to the > official business of this company shall be understood as neither given nor > endorsed by it. IG is a trading name of IG Markets Limited (a company > registered in England and Wales, company number 04008957) and IG Index > Limited (a company registered in England and Wales, company number > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG > Index Limited (register number 114059) are authorised and regulated by the > Financial Conduct Authority. > -- -Regards, Mayuresh R. Gharat (862) 250-7125
