Not at all.  This only acts on compacted topics just as what occurs today

Sent using OWA for iPhone
________________________________________
From: Jun Rao <j...@confluent.io>
Sent: Tuesday, December 6, 2016 6:25:28 PM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

Hi, Michael,

4. Hmm, does that mean the new client library can never send a null message
even to a regular topic? This seems like a change of the existing behavior.

Thanks,

Jun

On Tue, Dec 6, 2016 at 9:51 AM, Michael Pearce <michael.pea...@ig.com>
wrote:

> Hi Jun,
>
> Re 4) That's because we expect the tombstone value to be set correctly if
> message bit is 2, as such if an older client sends in on old message the
> message is upcast and the bit is set correctly. And such no longer need to
> check the value. Mayuresh can you confirm my thinking and understanding of
> what we've discussed?
>
> The second point I understand what you're getting at now my apologies. Yes
> this makes sense to save on touching the message, if we're the only kip
> going in, in this release.
>
> Cheers
> Mike
>
> Sent using OWA for iPhone
> ________________________________________
> From: Jun Rao <j...@confluent.io>
> Sent: Tuesday, December 6, 2016 5:22:13 PM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>
> Hi, Michael,
>
> 4. Is this updated in the wiki? The text "If the magic byte on message is
> 2, the broker should use the tombstone bit for log compaction." doesn't
> seem to have changed.
>
> 2. My point is that if we change the message format just for this KIP, we
> should consider whether it's worth optimizing the down conversion path
> (i.e., decide whether a conversion is needed by just looking at the
> tombstone bit in the wrapper message) since tombstone will be used rarely.
> However, if the message format change here is combined with other KIPs,
> then this optimization likely won't be needed. The latter probably makes
> the code simpler. Jiangjie, Mayuresh, what do you think?
>
> Other than those, +1 from me,
>
> Thanks,
>
> Jun
>
> On Tue, Dec 6, 2016 at 8:54 AM, Michael Pearce <michael.pea...@ig.com>
> wrote:
>
> > Hi Jun
> >
> > do we have your vote on this now?
> >
> > Any other concerns?
> >
> > Cheers
> > Mike
> >
> > Sent using OWA for iPhone
> > ________________________________________
> > From: Michael Pearce <michael.pea...@ig.com>
> > Sent: Saturday, December 3, 2016 1:37:45 AM
> > To: dev@kafka.apache.org
> > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> >
> > Hi Jun,
> >
> > Have updated. Thanks again for the feedback.
> >
> > Agree yes we should align up when it gets to that, I assume you’ve
> flagged
> > the same in the other KIP?
> >
> > I think for now let’s get this KIP approved, then we can start the work
> to
> > get an initial PR, then we can discuss how to align the two if needed.
> >
> > Cheers,
> > Mike
> >
> > On 02/12/2016, 18:26, "Jun Rao" <j...@confluent.io> wrote:
> >
> >     Hi, Michael,
> >
> >     For 2), this is fine. Could you update the KIP wiki to make this and
> > other
> >     points clearer? Other than that, the KIP looks good to me.
> >
> >     An orthogonal thing is that there are other KIPs such as KIP-98 that
> > also
> >     intend to change the message format. If they all get approved, we
> > should
> >     think about whether it's better to just bump up the magic byte once
> to
> >     incorporate multiple format changes like we did in KIP-31/KIP-32.
> >
> >     Thanks,
> >
> >     Jun
> >
> >
> >     On Fri, Dec 2, 2016 at 3:19 AM, Michael Pearce <
> michael.pea...@ig.com>
> >     wrote:
> >
> >     > Hi Jao
> >     >
> >     > Thanks for the response. Sorry for slow reply, both with personal
> > sickness
> >     > and also battling some critical issues encountered since upgrading
> to
> >     > 0.10.1.0
> >     >
> >     > 1) Thans for spotting, Document error where we branched this KIP
> from
> >     > KIP-82, will get that removed.
> >     > 2) Intent is to do this just at the record message level.
> >     > 3) Thanks for spotting, Will ensure this is corrected.
> >     > 4) As per discussion thread we will support tombstone + null value,
> >     > tombstone + non null value, no tombstone + null value.
> >     > 5) I believe this was in the discussion thread, @Mayuresh is this
> >     > something we’ve overlooked? I thought we would down convert and
> > remove the
> >     > value so the old consumer had existing behavior, or is there
> > something we
> >     > haven’t thought about?
> >     >
> >     > Cheers
> >     > Mike
> >     >
> >     > On 30/11/2016, 18:12, "Jun Rao" <j...@confluent.io> wrote:
> >     >
> >     >     Hi, Michael,
> >     >
> >     >     Thanks for the KIP. A few comments below.
> >     >
> >     >     1. The message format change contains "HeadersLength Headers".
> > Is that
> >     >     intended?
> >     >
> >     >     2. For compressed messageset, is the tombstone bit only set at
> > the
> >     > shallow
> >     >     level? Do we always leave that bit in the wrapper message
> unset?
> > An
> >     >     alternative is to set the tombstone bit in the wrapper if at
> > least one
> >     >     inner message has the tombstone bit set. This makes things a
> bit
> > more
> >     >     complicated, but we could potentially exploit that for
> > optimizing down
> >     >     conversion. For example, we only need to convert messages with
> > magic 2
> >     > to
> >     >     magic 1 if the wrapper's tombstone bit is set (conversion is
> > always
> >     > needed
> >     >     from magic 2 to magic 0). Not sure if the optimization is worth
> > the
> >     >     complexity though.
> >     >
> >     >     3. The referencing of the new version of
> > ProducerRequest/FetchRequest
> >     > is
> >     >     inconsistent (v4 vs v3). Since our convention starts at version
> > at 0, I
> >     >     think the new version would be 3.
> >     >
> >     >     4. "If the magic byte on message is 2, the broker should use
> the
> >     > tombstone
> >     >     bit for log compaction." What about null value? My
> understanding
> > is
> >     > that
> >     >     null value will be treated the same as setting the tombstone
> bit.
> >     >
> >     >     5. For the migration path, it would be useful to describe the
> > down
> >     >     conversion path to consumers (i.e., brokers on message format
> > 0.10.2
> >     > and
> >     >     consumers on older version).
> >     >
> >     >     Thanks,
> >     >
> >     >     Jun
> >     >
> >     >
> >     >     On Tue, Nov 29, 2016 at 3:18 AM, Michael Pearce <
> > michael.pea...@ig.com
> >     > >
> >     >     wrote:
> >     >
> >     >     > Hi All,
> >     >     >
> >     >     > We have been discussing in the below thread and final changes
> > have
> >     > been
> >     >     > made to the KIP wiki based on these discussions.
> >     >     >
> >     >     > We would now like to put to the vote the following KIP:
> >     >     > https://cwiki.apache.org/confluence/display/KAFKA/KIP-87+-+
> >     >     > Add+Compaction+Tombstone+Flag
> >     >     >
> >     >     > This kip is for having a distinct compaction attribute
> > “tombstone”
> >     > flag
> >     >     > instead of relying on null value, allowing non-null value
> > delete
> >     > messages.
> >     >     >
> >     >     > Many thanks,
> >     >     > Michael
> >     >     >
> >     >     >
> >     >     >
> >     >     > On 22/11/2016, 15:52, "Michael Pearce" <
> michael.pea...@ig.com>
> >     > wrote:
> >     >     >
> >     >     >     Hi Mayuresh,
> >     >     >
> >     >     >     LGTM. Ive just made one small adjustment updating the
> wire
> >     > protocol to
> >     >     > show the magic byte bump.
> >     >     >
> >     >     >     Do we think we’re good to put to a vote? Is there any
> > other bits
> >     >     > needing discussion?
> >     >     >
> >     >     >     Cheers
> >     >     >     Mike
> >     >     >
> >     >     >     On 21/11/2016, 18:26, "Mayuresh Gharat" <
> >     > gharatmayures...@gmail.com>
> >     >     > wrote:
> >     >     >
> >     >     >         Hi Michael,
> >     >     >
> >     >     >         I have updated the migration section of the KIP. Can
> > you
> >     > please
> >     >     > take a look?
> >     >     >
> >     >     >         Thanks,
> >     >     >
> >     >     >         Mayuresh
> >     >     >
> >     >     >         On Fri, Nov 18, 2016 at 9:07 AM, Mayuresh Gharat <
> >     >     > gharatmayures...@gmail.com
> >     >     >         > wrote:
> >     >     >
> >     >     >         > Hi Michael,
> >     >     >         >
> >     >     >         > That whilst sending tombstone and non null value,
> the
> >     > consumer
> >     >     > can expect
> >     >     >         > only to receive the non-null message only in step
> > (3) is
> >     > this
> >     >     > correct?
> >     >     >         > ---> I do agree with you here.
> >     >     >         >
> >     >     >         > Becket, Ismael : can you guys review the migration
> > plan
> >     > listed
> >     >     > above using
> >     >     >         > magic byte?
> >     >     >         >
> >     >     >         > Thanks,
> >     >     >         >
> >     >     >         > Mayuresh
> >     >     >         >
> >     >     >         > On Fri, Nov 18, 2016 at 8:58 AM, Michael Pearce <
> >     >     > michael.pea...@ig.com>
> >     >     >         > wrote:
> >     >     >         >
> >     >     >         >> Many thanks for this Mayuresh. I don't have any
> >     > objections.
> >     >     >         >>
> >     >     >         >> I assume we should state:
> >     >     >         >>
> >     >     >         >> That whilst sending tombstone and non null value,
> > the
> >     > consumer
> >     >     > can expect
> >     >     >         >> only to receive the non-null message only in step
> > (3) is
> >     > this
> >     >     > correct?
> >     >     >         >>
> >     >     >         >> Cheers
> >     >     >         >> Mike
> >     >     >         >>
> >     >     >         >>
> >     >     >         >>
> >     >     >         >> Sent using OWA for iPhone
> >     >     >         >> ________________________________________
> >     >     >         >> From: Mayuresh Gharat <gharatmayures...@gmail.com
> >
> >     >     >         >> Sent: Thursday, November 17, 2016 5:18:41 PM
> >     >     >         >> To: dev@kafka.apache.org
> >     >     >         >> Subject: Re: [DISCUSS] KIP-87 - Add Compaction
> > Tombstone
> >     > Flag
> >     >     >         >>
> >     >     >         >> Hi Ismael,
> >     >     >         >>
> >     >     >         >> Thanks for the explanation.
> >     >     >         >> Specially I like this part where in you mentioned
> > we can
> >     > get
> >     >     > rid of the
> >     >     >         >> older null value support for log compaction later
> > on,
> >     > here :
> >     >     >         >> We can't change semantics of the message format
> > without
> >     > having
> >     >     > a long
> >     >     >         >> transition period. And we can't rely
> >     >     >         >> on people reading documentation or acting on a
> > warning for
> >     >     > something so
> >     >     >         >> fundamental. As such, my take is that we need to
> > bump the
> >     > magic
> >     >     > byte. The
> >     >     >         >> good news is
> >     >     >         >> that we don't have to support all versions
> forever.
> > We
> >     > have
> >     >     > said that we
> >     >     >         >> will support direct upgrades for 2 years. That
> > means that
> >     >     > message format
> >     >     >         >> version n could, in theory, be removed 2 years
> > after the
> >     > it's
> >     >     > introduced.
> >     >     >         >>
> >     >     >         >> Just a heads up, I would like to mention that even
> > without
> >     >     > bumping magic
> >     >     >         >> byte, we will *NOT* loose zero copy as in the
> > client(x+1)
> >     > in my
> >     >     >         >> explanation
> >     >     >         >> above will convert internally a null value to
> have a
> >     > tombstone
> >     >     > bit set and
> >     >     >         >> a tombstone bit set to have a null value
> > automatically
> >     >     > internally and by
> >     >     >         >> the time we move to version (x+2), the clients
> > would have
> >     >     > upgraded.
> >     >     >         >> Obviously if we support a request from
> consumer(x),
> > we
> >     > will
> >     >     > loose zero
> >     >     >         >> copy
> >     >     >         >> but that is the same case with magic byte.
> >     >     >         >>
> >     >     >         >> But if magic byte bump makes life easier for
> > transition
> >     > for the
> >     >     > above
> >     >     >         >> reasons that you explained, I am OK with it since
> > we are
> >     > going
> >     >     > to meet the
> >     >     >         >> end goal down the road :)
> >     >     >         >>
> >     >     >         >> On a side note can we update the doc here on magic
> > byte
> >     > to say
> >     >     > that "*it
> >     >     >         >> should be bumped whenever the message format is
> > changed
> >     > or the
> >     >     >         >> interpretation of message format (usage of the
> > reserved
> >     > bits as
> >     >     > well) is
> >     >     >         >> changed*".
> >     >     >         >>
> >     >     >         >>
> >     >     >         >> Hi Michael,
> >     >     >         >>
> >     >     >         >> Here is the update plan that we discussed offline
> >     > yesterday :
> >     >     >         >>
> >     >     >         >> Currently the magic-byte which corresponds to the
> >     >     > "message.format.version"
> >     >     >         >> is set to 1.
> >     >     >         >>
> >     >     >         >> 1) On broker it will be set to 1 initially.
> >     >     >         >>
> >     >     >         >> 2) When a producer client sends a message with
> > magic-byte
> >     > = 2,
> >     >     > since the
> >     >     >         >> broker is on magic-byte = 1, we will down convert
> > it,
> >     > which
> >     >     > means if the
> >     >     >         >> tombstone bit is set, the value will be set to
> > null. A
> >     > consumer
> >     >     >         >> understanding magic-byte = 1, will still work with
> > this. A
> >     >     > consumer
> >     >     >         >> working
> >     >     >         >> with magic-byte =2 will also be able to understand
> > this,
> >     > since
> >     >     > it
> >     >     >         >> understands the tombstone.
> >     >     >         >> Now there is still the question of supporting a
> >     > non-tombstone
> >     >     > and null
> >     >     >         >> value from producer client with magic-byte = 2.*
> (I
> > am
> >     > not sure
> >     >     > if we
> >     >     >         >> should support this. Ismael/Becket can comment
> > here)*
> >     >     >         >>
> >     >     >         >> 3) When almost all the clients have upgraded, the
> >     >     > message.format.version
> >     >     >         >> on
> >     >     >         >> the broker can be changed to 2, where in the down
> >     > conversion in
> >     >     > the above
> >     >     >         >> step will not happen. If at this point we get a
> > consumer
> >     >     > request from a
> >     >     >         >> older consumer, we might have to down convert
> where
> > in we
> >     > loose
> >     >     > zero copy,
> >     >     >         >> but these cases should be rare.
> >     >     >         >>
> >     >     >         >> Becket can you review this plan and add more
> > details if I
> >     > have
> >     >     >         >> missed/wronged something, before we put it on KIP.
> >     >     >         >>
> >     >     >         >> Thanks,
> >     >     >         >>
> >     >     >         >> Mayuresh
> >     >     >         >>
> >     >     >         >> On Wed, Nov 16, 2016 at 11:07 PM, Michael Pearce <
> >     >     > michael.pea...@ig.com>
> >     >     >         >> wrote:
> >     >     >         >>
> >     >     >         >> > Thanks guys, for discussing this offline and
> > getting
> >     > some
> >     >     > consensus.
> >     >     >         >> >
> >     >     >         >> > So its clear for myself and others what is
> > proposed now
> >     > (i
> >     >     > think i
> >     >     >         >> > understand, but want to make sure)
> >     >     >         >> >
> >     >     >         >> > Could i ask either directly update the kip to
> > detail the
> >     >     > migration
> >     >     >         >> > strategy, or (re-)state your offline discussed
> and
> >     > agreed
> >     >     > migration
> >     >     >         >> > strategy based on a magic byte is in this
> thread.
> >     >     >         >> >
> >     >     >         >> >
> >     >     >         >> > The main original driver for the KIP was to
> > support
> >     >     > compaction where
> >     >     >         >> value
> >     >     >         >> > isn't null, based off the discussions on KIP-82
> > thread.
> >     >     >         >> >
> >     >     >         >> > We should be able to support non-tombstone +
> null
> > value
> >     > by the
> >     >     >         >> completion
> >     >     >         >> > of the KIP, as we noted when discussing this
> kip,
> > having
> >     >     > logic based on
> >     >     >         >> a
> >     >     >         >> > null value isn't very clean and also separates
> the
> >     > concerns.
> >     >     >         >> >
> >     >     >         >> > As discussed already though we can split this
> into
> >     > KIP-87a
> >     >     > and KIP-87b
> >     >     >         >> >
> >     >     >         >> > Where we look to deliver KIP-87a on a compacted
> > topic
> >     > (to
> >     >     > address the
> >     >     >         >> > immediate issues)
> >     >     >         >> > * tombstone + null value
> >     >     >         >> > * tombstone + non-null value
> >     >     >         >> > * non-tombstone + non-null value
> >     >     >         >> >
> >     >     >         >> > Then we can discuss once KIP-87a is completed
> > options
> >     > later
> >     >     > and how we
> >     >     >         >> > support the second part KIP-87b to deliver:
> >     >     >         >> > * non-tombstone + null value
> >     >     >         >> >
> >     >     >         >> > Cheers
> >     >     >         >> > Mike
> >     >     >         >> >
> >     >     >         >> >
> >     >     >         >> >
> >     >     >         >> > ________________________________________
> >     >     >         >> > From: Becket Qin <becket....@gmail.com>
> >     >     >         >> > Sent: Thursday, November 17, 2016 1:43 AM
> >     >     >         >> > To: dev@kafka.apache.org
> >     >     >         >> > Subject: Re: [DISCUSS] KIP-87 - Add Compaction
> >     > Tombstone Flag
> >     >     >         >> >
> >     >     >         >> > Renu, Mayuresh and I had an offline discussion,
> > and
> >     > following
> >     >     > is a brief
> >     >     >         >> > summary.
> >     >     >         >> >
> >     >     >         >> > 1. We agreed that not bumping up magic value may
> > result
> >     > in
> >     >     > losing zero
> >     >     >         >> copy
> >     >     >         >> > during migration.
> >     >     >         >> > 2. Given that bumping up magic value is almost
> > free and
> >     > has
> >     >     > benefit of
> >     >     >         >> > avoiding potential performance issue. It is
> > probably
> >     > worth
> >     >     > doing.
> >     >     >         >> >
> >     >     >         >> > One issue we still need to think about is
> whether
> > we
> >     > want to
> >     >     > support a
> >     >     >         >> > non-tombstone message with null value.
> >     >     >         >> > Currently it is not supported by Kafka. If we
> > allow a
> >     >     > non-tombstone null
> >     >     >         >> > value message to exist after KIP-87. The problem
> > is
> >     > that such
> >     >     > message
> >     >     >         >> will
> >     >     >         >> > not be supported by the consumers prior to
> KIP-87.
> >     > Because a
> >     >     > null value
> >     >     >         >> > will always be interpreted to a tombstone.
> >     >     >         >> >
> >     >     >         >> > One option is that we keep the current way, i.e.
> > do not
> >     >     > support such
> >     >     >         >> > message. It would be good to know if there is a
> >     > concrete use
> >     >     > case for
> >     >     >         >> such
> >     >     >         >> > message. If there is not, we can probably just
> not
> >     > support it.
> >     >     >         >> >
> >     >     >         >> > Thanks,
> >     >     >         >> >
> >     >     >         >> > JIangjie (Becket) Qin
> >     >     >         >> >
> >     >     >         >> >
> >     >     >         >> >
> >     >     >         >> > On Wed, Nov 16, 2016 at 1:28 PM, Mayuresh
> Gharat <
> >     >     >         >> > gharatmayures...@gmail.com
> >     >     >         >> > > wrote:
> >     >     >         >> >
> >     >     >         >> > > Hi Ismael,
> >     >     >         >> > >
> >     >     >         >> > > This is something I can think of for migration
> > plan:
> >     >     >         >> > > So the migration plan can look something like
> > this,
> >     > with up
> >     >     >         >> conversion :
> >     >     >         >> > >
> >     >     >         >> > > 1) Currently lets say we have Broker at
> version
> > x.
> >     >     >         >> > > 2) Currently we have clients at version x.
> >     >     >         >> > > 3) a) We move the version to Broker(x+1) :
> > supports
> >     > both
> >     >     > tombstone and
> >     >     >         >> > null
> >     >     >         >> > > for log compaction.
> >     >     >         >> > >     b) We upgrade the client to version
> > client(x+1) :
> >     > if in
> >     >     > the
> >     >     >         >> producer
> >     >     >         >> > > client(x+1) the value is set to null, we will
> >     > automatically
> >     >     > set the
> >     >     >         >> > > Tombstone bit internally. If the producer
> > client(x+1)
> >     > sets
> >     >     > the
> >     >     >         >> tombstone
> >     >     >         >> > > itself, well and good. For producer client(x),
> > the
> >     > broker
> >     >     > will up
> >     >     >         >> convert
> >     >     >         >> > > to have the tombstone bit. Broker(x+1) is
> > supporting
> >     > both.
> >     >     > Consumer
> >     >     >         >> > > client(x+1) will be aware of this and should
> be
> > able
> >     > to
> >     >     > handle this.
> >     >     >         >> For
> >     >     >         >> > > consumer client(x) we will down convert the
> > message
> >     > on the
> >     >     > broker
> >     >     >         >> side.
> >     >     >         >> > >     c) At this point we will have to specify a
> >     > warning or
> >     >     > clearly
> >     >     >         >> specify
> >     >     >         >> > > in docs that this behavior is about to be
> > changed for
> >     > log
> >     >     > compaction.
> >     >     >         >> > > 4) a) In next release of the Broker(x+2), we
> > say that
> >     > only
> >     >     > Tombstone
> >     >     >         >> is
> >     >     >         >> > > used for log compaction on the Broker side.
> >     > Clients(x+1)
> >     >     > still is
> >     >     >         >> > > supported.
> >     >     >         >> > >     b) We upgrade the client to version
> > client(x+2) :
> >     > if
> >     >     > value is set
> >     >     >         >> to
> >     >     >         >> > > null, tombstone will not be set automatically.
> > The
> >     > client
> >     >     > will have to
> >     >     >         >> > call
> >     >     >         >> > > setTombstone() to actually set the tombstone.
> >     >     >         >> > >
> >     >     >         >> > > We should compare this migration plan with the
> >     > migration
> >     >     > plan for
> >     >     >         >> magic
> >     >     >         >> > > byte bump and do whatever looks good.
> >     >     >         >> > > I am just worried that if we go down magic
> byte
> > route,
> >     >     > unless I am
> >     >     >         >> > missing
> >     >     >         >> > > something, it sounds like kafka will be stuck
> > with
> >     >     > supporting both
> >     >     >         >> null
> >     >     >         >> > > value and tombstone bit for log compaction for
> > life
> >     > long,
> >     >     > which does
> >     >     >         >> not
> >     >     >         >> > > look like a good end state.
> >     >     >         >> > >
> >     >     >         >> > > Thanks,
> >     >     >         >> > >
> >     >     >         >> > > Mayuresh
> >     >     >         >> > >
> >     >     >         >> > >
> >     >     >         >> > >
> >     >     >         >> > >
> >     >     >         >> > > On Wed, Nov 16, 2016 at 9:32 AM, Mayuresh
> > Gharat <
> >     >     >         >> > > gharatmayures...@gmail.com
> >     >     >         >> > > > wrote:
> >     >     >         >> > >
> >     >     >         >> > > > Hi Ismael,
> >     >     >         >> > > >
> >     >     >         >> > > > That's a very good point which I might have
> > not
> >     >     > considered earlier.
> >     >     >         >> > > >
> >     >     >         >> > > > Here is a plan that I can think of:
> >     >     >         >> > > >
> >     >     >         >> > > > Stage 1) The broker from now on, up converts
> > the
> >     > message
> >     >     > to have the
> >     >     >         >> > > > tombstone marker. The log compaction thread
> > does log
> >     >     > compaction
> >     >     >         >> based
> >     >     >         >> > on
> >     >     >         >> > > > both null and tombstone marker. This is our
> >     > transition
> >     >     > period.
> >     >     >         >> > > > Stage 2) The next release we only say that
> log
> >     > compaction
> >     >     > is based
> >     >     >         >> on
> >     >     >         >> > > > tombstone marker. (Open source kafka makes
> > this as a
> >     >     > policy). By
> >     >     >         >> this
> >     >     >         >> > > time,
> >     >     >         >> > > > the organization which is moving to this
> > release
> >     > will be
> >     >     > sure that
> >     >     >         >> they
> >     >     >         >> > > > have gone through the entire transition
> > period.
> >     >     >         >> > > >
> >     >     >         >> > > > My only goal of doing this is that Kafka
> > clearly
> >     >     > specifies the end
> >     >     >         >> > state
> >     >     >         >> > > > about what log compaction means (is it null
> > value
> >     > or a
> >     >     > tombstone
> >     >     >         >> > marker,
> >     >     >         >> > > > but not both).
> >     >     >         >> > > >
> >     >     >         >> > > > What do you think?
> >     >     >         >> > > >
> >     >     >         >> > > > Thanks,
> >     >     >         >> > > >
> >     >     >         >> > > > Mayuresh
> >     >     >         >> > > > .
> >     >     >         >> > > >
> >     >     >         >> > > > On Wed, Nov 16, 2016 at 9:17 AM, Ismael
> Juma <
> >     >     > ism...@juma.me.uk>
> >     >     >         >> > wrote:
> >     >     >         >> > > >
> >     >     >         >> > > >> One comment below.
> >     >     >         >> > > >>
> >     >     >         >> > > >> On Wed, Nov 16, 2016 at 5:08 PM, Mayuresh
> > Gharat <
> >     >     >         >> > > >> gharatmayures...@gmail.com
> >     >     >         >> > > >> > wrote:
> >     >     >         >> > > >>
> >     >     >         >> > > >> >    - If we don't bump up the magic byte,
> > on the
> >     > broker
> >     >     > side, the
> >     >     >         >> > > broker
> >     >     >         >> > > >> >    will always have to look at both
> > tombstone
> >     > bit and
> >     >     > the value
> >     >     >         >> when
> >     >     >         >> > > do
> >     >     >         >> > > >> the
> >     >     >         >> > > >> >    compaction. Assuming we do not bump up
> > the
> >     > magic
> >     >     > byte,
> >     >     >         >> > > >> >    imagine the broker sees a message
> which
> > does
> >     > not
> >     >     > have a
> >     >     >         >> tombstone
> >     >     >         >> > > bit
> >     >     >         >> > > >> >    set. The broker does not know when the
> >     > message was
> >     >     > produced
> >     >     >         >> (i.e.
> >     >     >         >> > > >> > whether
> >     >     >         >> > > >> >    the message has been up converted or
> > not), it
> >     > has
> >     >     > to take a
> >     >     >         >> > further
> >     >     >         >> > > >> > look at
> >     >     >         >> > > >> >    the value to see if it is null or not
> in
> >     > order to
> >     >     > determine
> >     >     >         >> if it
> >     >     >         >> > > is
> >     >     >         >> > > >> a
> >     >     >         >> > > >> >    tombstone. The same logic has to be
> put
> > on the
> >     >     > consumer as
> >     >     >         >> well
> >     >     >         >> > > >> because
> >     >     >         >> > > >> > the
> >     >     >         >> > > >> >    consumer does not know if the message
> > has
> >     > been up
> >     >     > converted or
> >     >     >         >> > not.
> >     >     >         >> > > >> >       - If we upconvert while appending,
> > this is
> >     > not
> >     >     > the case,
> >     >     >         >> > right?
> >     >     >         >> > > >>
> >     >     >         >> > > >>
> >     >     >         >> > > >> If I understand you correctly, this is not
> >     > sufficient
> >     >     > because the
> >     >     >         >> log
> >     >     >         >> > > may
> >     >     >         >> > > >> have messages appended before it was
> > upgraded to
> >     > include
> >     >     > KIP-87.
> >     >     >         >> > > >>
> >     >     >         >> > > >> Ismael
> >     >     >         >> > > >>
> >     >     >         >> > > >
> >     >     >         >> > > >
> >     >     >         >> > > >
> >     >     >         >> > > > --
> >     >     >         >> > > > -Regards,
> >     >     >         >> > > > Mayuresh R. Gharat
> >     >     >         >> > > > (862) 250-7125
> >     >     >         >> > > >
> >     >     >         >> > >
> >     >     >         >> > >
> >     >     >         >> > >
> >     >     >         >> > > --
> >     >     >         >> > > -Regards,
> >     >     >         >> > > Mayuresh R. Gharat
> >     >     >         >> > > (862) 250-7125
> >     >     >         >> > >
> >     >     >         >> > The information contained in this email is
> > strictly
> >     >     > confidential and for
> >     >     >         >> > the use of the addressee only, unless otherwise
> >     > indicated. If
> >     >     > you are
> >     >     >         >> not
> >     >     >         >> > the intended recipient, please do not read,
> copy,
> > use or
> >     >     > disclose to
> >     >     >         >> others
> >     >     >         >> > this message or any attachment. Please also
> > notify the
> >     > sender
> >     >     > by
> >     >     >         >> replying
> >     >     >         >> > to this email or by telephone (+44(020 7896
> 0011)
> > and
> >     > then
> >     >     > delete the
> >     >     >         >> email
> >     >     >         >> > and any copies of it. Opinions, conclusion (etc)
> > that
> >     > do not
> >     >     > relate to
> >     >     >         >> the
> >     >     >         >> > official business of this company shall be
> > understood as
> >     >     > neither given
> >     >     >         >> nor
> >     >     >         >> > endorsed by it. IG is a trading name of IG
> Markets
> >     > Limited (a
> >     >     > company
> >     >     >         >> > registered in England and Wales, company number
> >     > 04008957) and
> >     >     > IG Index
> >     >     >         >> > Limited (a company registered in England and
> > Wales,
> >     > company
> >     >     > number
> >     >     >         >> > 01190902). Registered address at Cannon Bridge
> > House, 25
> >     >     > Dowgate Hill,
> >     >     >         >> > London EC4R 2YA. Both IG Markets Limited
> (register
> >     > number
> >     >     > 195355) and IG
> >     >     >         >> > Index Limited (register number 114059) are
> > authorised
> >     > and
> >     >     > regulated by
> >     >     >         >> the
> >     >     >         >> > Financial Conduct Authority.
> >     >     >         >> >
> >     >     >         >>
> >     >     >         >>
> >     >     >         >>
> >     >     >         >> --
> >     >     >         >> -Regards,
> >     >     >         >> Mayuresh R. Gharat
> >     >     >         >> (862) 250-7125
> >     >     >         >> The information contained in this email is
> strictly
> >     >     > confidential and for
> >     >     >         >> the use of the addressee only, unless otherwise
> >     > indicated. If
> >     >     > you are not
> >     >     >         >> the intended recipient, please do not read, copy,
> > use or
> >     >     > disclose to others
> >     >     >         >> this message or any attachment. Please also notify
> > the
> >     > sender
> >     >     > by replying
> >     >     >         >> to this email or by telephone (+44(020 7896 0011)
> > and then
> >     >     > delete the email
> >     >     >         >> and any copies of it. Opinions, conclusion (etc)
> > that do
> >     > not
> >     >     > relate to the
> >     >     >         >> official business of this company shall be
> > understood as
> >     >     > neither given nor
> >     >     >         >> endorsed by it. IG is a trading name of IG Markets
> >     > Limited (a
> >     >     > company
> >     >     >         >> registered in England and Wales, company number
> > 04008957)
> >     > and
> >     >     > IG Index
> >     >     >         >> Limited (a company registered in England and
> Wales,
> >     > company
> >     >     > number
> >     >     >         >> 01190902). Registered address at Cannon Bridge
> > House, 25
> >     >     > Dowgate Hill,
> >     >     >         >> London EC4R 2YA. Both IG Markets Limited (register
> > number
> >     >     > 195355) and IG
> >     >     >         >> Index Limited (register number 114059) are
> > authorised and
> >     >     > regulated by the
> >     >     >         >> Financial Conduct Authority.
> >     >     >         >>
> >     >     >         >
> >     >     >         >
> >     >     >         >
> >     >     >         > --
> >     >     >         > -Regards,
> >     >     >         > Mayuresh R. Gharat
> >     >     >         > (862) 250-7125
> >     >     >         >
> >     >     >
> >     >     >
> >     >     >
> >     >     >         --
> >     >     >         -Regards,
> >     >     >         Mayuresh R. Gharat
> >     >     >         (862) 250-7125
> >     >     >
> >     >     >
> >     >     >     The information contained in this email is strictly
> > confidential
> >     > and
> >     >     > for the use of the addressee only, unless otherwise
> indicated.
> > If
> >     > you are
> >     >     > not the intended recipient, please do not read, copy, use or
> >     > disclose to
> >     >     > others this message or any attachment. Please also notify the
> > sender
> >     > by
> >     >     > replying to this email or by telephone (+44(020 7896 0011)
> and
> > then
> >     > delete
> >     >     > the email and any copies of it. Opinions, conclusion (etc)
> > that do
> >     > not
> >     >     > relate to the official business of this company shall be
> > understood
> >     > as
> >     >     > neither given nor endorsed by it. IG is a trading name of IG
> > Markets
> >     >     > Limited (a company registered in England and Wales, company
> > number
> >     >     > 04008957) and IG Index Limited (a company registered in
> > England and
> >     > Wales,
> >     >     > company number 01190902). Registered address at Cannon Bridge
> > House,
> >     > 25
> >     >     > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited
> > (register
> >     > number
> >     >     > 195355) and IG Index Limited (register number 114059) are
> > authorised
> >     > and
> >     >     > regulated by the Financial Conduct Authority.
> >     >     >
> >     >     >
> >     >     >
> >     >
> >     >
> >     > The information contained in this email is strictly confidential
> and
> > for
> >     > the use of the addressee only, unless otherwise indicated. If you
> > are not
> >     > the intended recipient, please do not read, copy, use or disclose
> to
> > others
> >     > this message or any attachment. Please also notify the sender by
> > replying
> >     > to this email or by telephone (+44(020 7896 0011) and then delete
> > the email
> >     > and any copies of it. Opinions, conclusion (etc) that do not relate
> > to the
> >     > official business of this company shall be understood as neither
> > given nor
> >     > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> >     > registered in England and Wales, company number 04008957) and IG
> > Index
> >     > Limited (a company registered in England and Wales, company number
> >     > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> > Hill,
> >     > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> > and IG
> >     > Index Limited (register number 114059) are authorised and regulated
> > by the
> >     > Financial Conduct Authority.
> >     >
> >
> >
> > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by replying
> > to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> > official business of this company shall be understood as neither given
> nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> > Index Limited (register number 114059) are authorised and regulated by
> the
> > Financial Conduct Authority.
> >
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
>
The information contained in this email is strictly confidential and for the 
use of the addressee only, unless otherwise indicated. If you are not the 
intended recipient, please do not read, copy, use or disclose to others this 
message or any attachment. Please also notify the sender by replying to this 
email or by telephone (+44(020 7896 0011) and then delete the email and any 
copies of it. Opinions, conclusion (etc) that do not relate to the official 
business of this company shall be understood as neither given nor endorsed by 
it. IG is a trading name of IG Markets Limited (a company registered in England 
and Wales, company number 04008957) and IG Index Limited (a company registered 
in England and Wales, company number 01190902). Registered address at Cannon 
Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited 
(register number 195355) and IG Index Limited (register number 114059) are 
authorised and regulated by the Financial Conduct Authority.

Reply via email to