Hi Chaps,
Can we either get one more +1 binding (we have 2 already) on the existing?
Or have some response on the below possible alternative. We are keen to get
working on this, so we make next feature release.
Cheers
Mike
On 13/12/2016, 16:32, "Michael Pearce" <michael.pea...@ig.com> wrote:
Hi Ismael
Did you see our email this morning, what's your thoughts on this approach
to instead we simply have a brand new policy?
Cheers
Mike
Sent using OWA for iPhone
________________________________________
From: isma...@gmail.com <isma...@gmail.com> on behalf of Ismael Juma
<ism...@juma.me.uk>
Sent: Tuesday, December 13, 2016 11:30:05 AM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
Yes, this is actually tricky to do in a way where we both have the desired
semantics and maintain compatibility. When someone creates a
`ProducerRecord` with a `null` value today, the producer doesn't know if
it's meant to be a tombstone or not. For V3 messages, it's easy when the
constructor that takes a tombstone is used. However, if any other
constructor is used, it's not clear. A couple of options I can think of,
none of them particularly nice:
1. Have a third state where tombstone = unknown and the broker would set
the tombstone bit if the value was null and the topic was compacted. People
that wanted to pass a non-null value for the tombstone would have to use
the constructor that takes a tombstone. The drawbacks: third state for
tombstone in message format, message conversion at the broker for a common
case.
2. Extend MetadataResponse to optionally include topic configs, which would
make it possible for the producer to be smarter about setting the
tombstone. It would only do it if a tombstone was not passed explicitly,
the value was null and the topic was compacted. The main drawback is that
the producer would be getting a bit more data for each topic even though it
probably won't use it most of the time. Extending MetadataResponse to
return topic configs would be useful for other reasons as well, so that
part seems OK.
In addition, for both proposals, we could consider adding warnings to the
documentation that the behaviour of the constructors that don't take a
tombstone would change in the next major release so that tombstone = false.
Not sure if this would be worth it though.
Ismael
On Sun, Dec 11, 2016 at 11:15 PM, Ewen Cheslack-Postava <e...@confluent.io>
wrote:
> Michael,
>
> It kind of depends on how you want to interpret the tombstone flag. If
it's
> purely a producer-facing Kafka-level thing that we treat as internal to
the
> broker and log cleaner once the record is sent, then your approach makes
> sense. You're just moving copying the null-indicates-delete behavior of
the
> old constructor into the tombstone flag.
>
> However, if you want this change to more generally decouple the idea of
> deletion and null values, then you are sometimes converting what might be
a
> completely valid null value that doesn't indicate deletion into a
> tombstone. Downstream applications could potentially handle these cases
> differently given the separation of deletion from value.
>
> I guess the question is if we want to try to support the latter even for
> topics where we have older produce requests. An example where this could
> come up is in something like a CDC Connector. If we try to support the
> semantic difference, a connector might write changes to Kafka using the
> tombstone flag to indicate when a row was truly deleted (vs an update that
> sets it to null but still present; this probably makes more sense for CDC
> from document stores or extracting single columns). There are various
> reasons we might want to maintain the full log and not turn compaction on
> (or just use a time-based retention policy), but downstream applications
> might care to know the difference between a delete and a null value. In
> fact both versions of the same log (compacted and time-retention) could be
> useful and I don't think it'll be uncommon to maintain both or use KIP-71
> to maintain a hybrid compacted/retention topic.
>
> -Ewen
>
> On Sun, Dec 11, 2016 at 1:18 PM, Michael Pearce <michael.pea...@ig.com>
> wrote:
>
> > Hi Jay,
> >
> > Why wouldn't that work, the tombstone value is only looked at by the
> > broker, on a topic configured for compaction as such is benign on non
> > compacted topics. This is as much as sending a null value currently
> >
> >
> > Regards
> > Mike
> >
> >
> >
> > Sent using OWA for iPhone
> > ________________________________________
> > From: Jay Kreps <j...@confluent.io>
> > Sent: Sunday, December 11, 2016 8:58:53 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> >
> > Hey Michael,
> >
> > I'm not quite sure that works as that would translate ALL null values to
> > tombstones, even for non-compacted topics that use null as an acceptable
> > value sent by the producer and expected by the consumer.
> >
> > -Jay
> >
> > On Sun, Dec 11, 2016 at 3:26 AM, Michael Pearce <michael.pea...@ig.com>
> > wrote:
> >
> > > Hi Ewen,
> > >
> > > I think the easiest way to show this is with code.
> > >
> > > As you can see we keep the existing behaviour for code/binaries
calling
> > > the pre-existing constructors, whereby if the value is null the
> tombstone
> > > is set to true.
> > >
> > > Regards
> > > Mike
> > >
> > >
> > >
> > > /**
> > > * Creates a record with a specified timestamp to be sent to a
> > > specified topic and partition
> > > *
> > > * @param topic The topic the record will be appended to
> > > * @param partition The partition to which the record should be
> sent
> > > * @param timestamp The timestamp of the record
> > > * @param tombstone if the record should be treated as a tombstone
> if
> > > the topic is compacted
> > > * @param key The key that will be included in the record
> > > * @param value The record contents
> > > */
> > > public ProducerRecord(String topic, Integer partition, Boolean
> > > tombstone, Long timestamp, K key, V value) {
> > > if (topic == null)
> > > throw new IllegalArgumentException("Topic cannot be
> null.");
> > > if (timestamp != null && timestamp < 0)
> > > throw new IllegalArgumentException(
> > > String.format("Invalid timestamp: %d. Timestamp
> > should
> > > always be non-negative or null.", timestamp));
> > > if (partition != null && partition < 0)
> > > throw new IllegalArgumentException(
> > > String.format("Invalid partition: %d. Partition
> > number
> > > should always be non-negative or null.", partition));
> > > if (!tombstone && value == null){
> > > throw new IllegalArgumentException(
> > > String.format("Tombstone must be true if null
> > value"));
> > > }
> > > this.topic = topic;
> > > this.partition = partition;
> > > this.tombstone = tombstone;
> > > this.key = key;
> > > this.value = value;
> > > this.timestamp = timestamp;
> > > }
> > >
> > > public ProducerRecord(String topic, Integer partition, Long
> > timestamp,
> > > K key, V value) {
> > > this(topic, partition, value == null, timestamp, key, value);
> > > }
> > > ________________________________________
> > > From: Ewen Cheslack-Postava <e...@confluent.io>
> > > Sent: Sunday, December 11, 2016 5:45 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> > >
> > > It seemed like this was addressed in the migration section, wasn't it?
> > The
> > > V2 vs V3 requests and the fact that the broker will downgrade the
> message
> > > format (losing zero copy) if you issues a V2 request to a broker using
> V3
> > > format handles compatibility, does it not? The existing consumers
won't
> > see
> > > the extra metadata in the value, but they will get a null instead and
> > treat
> > > it as a tombstone. Certainly there is a performance impact, but it
> seemed
> > > compatible.
> > >
> > > I'm worried about this though:
> > >
> > >
> > > - *NOTE *: With the new version of producer client using
> > ProduceRequest
> > > V3 (magic byte = 2), a non tombstone (tombstone bit not set) and
> null
> > > value
> > > should not be allowed as the older version of consumer using
> > > FetchRequest
> > > V2 will think of this as a tombstone when its actually not.
> > >
> > >
> > > Unless I'm misunderstanding, this ends up breaking binary
compatibility
> > for
> > > the Java API. It sounds like this suggests that passing a null value
to
> > the
> > > existing ProducerRecord constructors would generate an exception since
> > you
> > > didn't explicitly enable the tombstone (via whatever new constructor
is
> > > provided). But that means you can't swap in a newer client jar without
> > > recompiling and get the same behavior if your app deletes keys using
> the
> > > current approach because your app will start throwing exceptions.
Maybe
> > > this is a tradeoff we're ok with, but we've tried pretty hard to avoid
> > > breaking compatibility like this so far -- it makes picking up bug
> fixes
> > in
> > > the clients harder for users of frameworks that have to pin to earlier
> > > default versions for compatibility.
> > >
> > > But then later the KIP says:
> > >
> > >
> > > - The old constructors for ProducerRecord without this parameter
> will
> > be
> > > kept but updated so that their default behaviour if setting null
> value
> > > will
> > > be the tombstone will be set to true to keep existing behaviour.
> > >
> > >
> > > So maybe I am misinterpreting something.
> > >
> > > And just a nit re: motivation section, item 6 would be clearer for a
> > union
> > > schema with null (or optional schemas in other formats), e.g. [null,
> > > string], in which case losing the schema truly is losing information
> > > (whereas null is already the only valid value for a pure null schema).
> > >
> > > -Ewen
> > >
> > >
> > > On Sat, Dec 10, 2016 at 9:24 PM, Michael Pearce <michael.pea...@ig.com
> >
> > > wrote:
> > >
> > > > Hi Jay,
> > > >
> > > > Good point this detail is missing in the KIP write up. Ive added
this
> > > now.
> > > >
> > > > Essentially simply just upgrading the clients we do not expect any
> > client
> > > > app code change needed.
> > > >
> > > > Cheers
> > > > Mike
> > > > ________________________________________
> > > > From: Jay Kreps <j...@confluent.io>
> > > > Sent: Saturday, December 10, 2016 10:51 PM
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> > > >
> > > > Michael,
> > > >
> > > > The compatibility section goes through the migration path, but isn't
> > the
> > > > bigger compatibility issue with existing apps? There are many
> (probably
> > > > thousands) of apps in production that use this feature and send null
> to
> > > > mean delete. It seems like this would break compatibility with them,
> > and
> > > > they would have to be rewritten to right?
> > > >
> > > > -Jay
> > > >
> > > > On Thu, Dec 8, 2016 at 12:12 AM, Michael Pearce <
> michael.pea...@ig.com
> > >
> > > > wrote:
> > > >
> > > > > Hi Jun,
> > > > >
> > > > > 4) On v3 we honour the tombstone. As such we expect it to be set
> > > > correctly
> > > > > as per the KIP.
> > > > >
> > > > > 4.1) why would we want to produce an error when its v3? This is
the
> > > exact
> > > > > purpose to support non-null tombstone’s
> > > > > 4.2) again here im unclear on the question on the v3, produce
> request
> > > we
> > > > > expect the tombstone flag to be set correctly.
> > > > >
> > > > > 4.4) compaction only occurs on compacted topics, the bit makes no
> > > > > difference and not looked at on un-compacted (time/size based
> > > eviction).
> > > > >
> > > > >
> > > > > On 06/12/2016, 20:08, "Jun Rao" <j...@confluent.io> wrote:
> > > > >
> > > > > Hi, Michael,
> > > > >
> > > > > 4. Then, I think I misunderstood this point. Could you
document
> > the
> > > > > following points in the wiki?
> > > > > 4.1 If producer V3 sets tombstone, but provides a non-null
> value,
> > > > does
> > > > > the
> > > > > send() get an error or does the producer automatically set the
> > > value
> > > > to
> > > > > null?
> > > > > 4.2 If producer V3 doesn't set tombstone, but provides a null
> > > value,
> > > > > does
> > > > > the send() get an error or does the producer automatically
sets
> > the
> > > > > tombstone?
> > > > > 4.3 Does the broker only expect messages that (a) have no
> > tombstone
> > > > and
> > > > > non-null value; (b) have tombstone and null value and reject
> the
> > > > > messages
> > > > > with an error code otherwise?
> > > > > 4.4 Do 4.1, 4.2, 4.3 depend on whether the topic is compacted
> on
> > > > not?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Tue, Dec 6, 2016 at 10:35 AM, Michael Pearce <
> > > > michael.pea...@ig.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Not at all. This only acts on compacted topics just as what
> > > occurs
> > > > > today
> > > > > >
> > > > > > Sent using OWA for iPhone
> > > > > > ________________________________________
> > > > > > From: Jun Rao <j...@confluent.io>
> > > > > > Sent: Tuesday, December 6, 2016 6:25:28 PM
> > > > > > To: dev@kafka.apache.org
> > > > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> > > > > >
> > > > > > Hi, Michael,
> > > > > >
> > > > > > 4. Hmm, does that mean the new client library can never send
> a
> > > null
> > > > > message
> > > > > > even to a regular topic? This seems like a change of the
> > existing
> > > > > behavior.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Tue, Dec 6, 2016 at 9:51 AM, Michael Pearce <
> > > > > michael.pea...@ig.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Jun,
> > > > > > >
> > > > > > > Re 4) That's because we expect the tombstone value to be
> set
> > > > > correctly if
> > > > > > > message bit is 2, as such if an older client sends in on
> old
> > > > > message the
> > > > > > > message is upcast and the bit is set correctly. And such
no
> > > > longer
> > > > > need
> > > > > > to
> > > > > > > check the value. Mayuresh can you confirm my thinking and
> > > > > understanding
> > > > > > of
> > > > > > > what we've discussed?
> > > > > > >
> > > > > > > The second point I understand what you're getting at now
my
> > > > > apologies.
> > > > > > Yes
> > > > > > > this makes sense to save on touching the message, if we're
> > the
> > > > > only kip
> > > > > > > going in, in this release.
> > > > > > >
> > > > > > > Cheers
> > > > > > > Mike
> > > > > > >
> > > > > > > Sent using OWA for iPhone
> > > > > > > ________________________________________
> > > > > > > From: Jun Rao <j...@confluent.io>
> > > > > > > Sent: Tuesday, December 6, 2016 5:22:13 PM
> > > > > > > To: dev@kafka.apache.org
> > > > > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
> > > > > > >
> > > > > > > Hi, Michael,
> > > > > > >
> > > > > > > 4. Is this updated in the wiki? The text "If the magic
byte
> > on
> > > > > message is
> > > > > > > 2, the broker should use the tombstone bit for log
> > compaction."
> > > > > doesn't
> > > > > > > seem to have changed.
> > > > > > >
> > > > > > > 2. My point is that if we change the message format just
> for
> > > this
> > > > > KIP, we
> > > > > > > should consider whether it's worth optimizing the down
> > > conversion
> > > > > path
> > > > > > > (i.e., decide whether a conversion is needed by just
> looking
> > at
> > > > the
> > > > > > > tombstone bit in the wrapper message) since tombstone will
> be
> > > > used
> > > > > > rarely.
> > > > > > > However, if the message format change here is combined
with
> > > other
> > > > > KIPs,
> > > > > > > then this optimization likely won't be needed. The latter
> > > > probably
> > > > > makes
> > > > > > > the code simpler. Jiangjie, Mayuresh, what do you think?
> > > > > > >
> > > > > > > Other than those, +1 from me,
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > > On Tue, Dec 6, 2016 at 8:54 AM, Michael Pearce <
> > > > > michael.pea...@ig.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Jun
> > > > > > > >
> > > > > > > > do we have your vote on this now?
> > > > > > > >
> > > > > > > > Any other concerns?
> > > > > > > >
> > > > > > > > Cheers
> > > > > > > > Mike
> > > > > > > >
> > > > > > > > Sent using OWA for iPhone
> > > > > > > > ________________________________________
> > > > > > > > From: Michael Pearce <michael.pea...@ig.com>
> > > > > > > > Sent: Saturday, December 3, 2016 1:37:45 AM
> > > > > > > > To: dev@kafka.apache.org
> > > > > > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone
> Flag
> > > > > > > >
> > > > > > > > Hi Jun,
> > > > > > > >
> > > > > > > > Have updated. Thanks again for the feedback.
> > > > > > > >
> > > > > > > > Agree yes we should align up when it gets to that, I
> assume
> > > > > you’ve
> > > > > > > flagged
> > > > > > > > the same in the other KIP?
> > > > > > > >
> > > > > > > > I think for now let’s get this KIP approved, then we can
> > > start
> > > > > the work
> > > > > > > to
> > > > > > > > get an initial PR, then we can discuss how to align the
> two
> > > if
> > > > > needed.
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Mike
> > > > > > > >
> > > > > > > > On 02/12/2016, 18:26, "Jun Rao" <j...@confluent.io>
> wrote:
> > > > > > > >
> > > > > > > > Hi, Michael,
> > > > > > > >
> > > > > > > > For 2), this is fine. Could you update the KIP wiki
> to
> > > make
> > > > > this
> > > > > > and
> > > > > > > > other
> > > > > > > > points clearer? Other than that, the KIP looks good
> to
> > > me.
> > > > > > > >
> > > > > > > > An orthogonal thing is that there are other KIPs
such
> > as
> > > > > KIP-98
> > > > > > that
> > > > > > > > also
> > > > > > > > intend to change the message format. If they all get
> > > > > approved, we
> > > > > > > > should
> > > > > > > > think about whether it's better to just bump up the
> > magic
> > > > > byte once
> > > > > > > to
> > > > > > > > incorporate multiple format changes like we did in
> > > > > KIP-31/KIP-32.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Dec 2, 2016 at 3:19 AM, Michael Pearce <
> > > > > > > michael.pea...@ig.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Jao
> > > > > > > > >
> > > > > > > > > Thanks for the response. Sorry for slow reply,
both
> > > with
> > > > > personal
> > > > > > > > sickness
> > > > > > > > > and also battling some critical issues encountered
> > > since
> > > > > > upgrading
> > > > > > > to
> > > > > > > > > 0.10.1.0
> > > > > > > > >
> > > > > > > > > 1) Thans for spotting, Document error where we
> > branched
> > > > > this KIP
> > > > > > > from
> > > > > > > > > KIP-82, will get that removed.
> > > > > > > > > 2) Intent is to do this just at the record message
> > > level.
> > > > > > > > > 3) Thanks for spotting, Will ensure this is
> > corrected.
> > > > > > > > > 4) As per discussion thread we will support
> > tombstone +
> > > > > null
> > > > > > value,
> > > > > > > > > tombstone + non null value, no tombstone + null
> > value.
> > > > > > > > > 5) I believe this was in the discussion thread,
> > > @Mayuresh
> > > > > is this
> > > > > > > > > something we’ve overlooked? I thought we would
down
> > > > > convert and
> > > > > > > > remove the
> > > > > > > > > value so the old consumer had existing behavior,
or
> > is
> > > > > there
> > > > > > > > something we
> > > > > > > > > haven’t thought about?
> > > > > > > > >
> > > > > > > > > Cheers
> > > > > > > > > Mike
> > > > > > > > >
> > > > > > > > > On 30/11/2016, 18:12, "Jun Rao"
<j...@confluent.io>
> > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi, Michael,
> > > > > > > > >
> > > > > > > > > Thanks for the KIP. A few comments below.
> > > > > > > > >
> > > > > > > > > 1. The message format change contains
> > > "HeadersLength
> > > > > > Headers".
> > > > > > > > Is that
> > > > > > > > > intended?
> > > > > > > > >
> > > > > > > > > 2. For compressed messageset, is the tombstone
> > bit
> > > > > only set
> > > > > > at
> > > > > > > > the
> > > > > > > > > shallow
> > > > > > > > > level? Do we always leave that bit in the
> wrapper
> > > > > message
> > > > > > > unset?
> > > > > > > > An
> > > > > > > > > alternative is to set the tombstone bit in the
> > > > wrapper
> > > > > if at
> > > > > > > > least one
> > > > > > > > > inner message has the tombstone bit set. This
> > makes
> > > > > things a
> > > > > > > bit
> > > > > > > > more
> > > > > > > > > complicated, but we could potentially exploit
> > that
> > > > for
> > > > > > > > optimizing down
> > > > > > > > > conversion. For example, we only need to
> convert
> > > > > messages
> > > > > > with
> > > > > > > > magic 2
> > > > > > > > > to
> > > > > > > > > magic 1 if the wrapper's tombstone bit is set
> > > > > (conversion is
> > > > > > > > always
> > > > > > > > > needed
> > > > > > > > > from magic 2 to magic 0). Not sure if the
> > > > optimization
> > > > > is
> > > > > > worth
> > > > > > > > the
> > > > > > > > > complexity though.
> > > > > > > > >
> > > > > > > > > 3. The referencing of the new version of
> > > > > > > > ProducerRequest/FetchRequest
> > > > > > > > > is
> > > > > > > > > inconsistent (v4 vs v3). Since our convention
> > > starts
> > > > at
> > > > > > version
> > > > > > > > at 0, I
> > > > > > > > > think the new version would be 3.
> > > > > > > > >
> > > > > > > > > 4. "If the magic byte on message is 2, the
> broker
> > > > > should use
> > > > > > > the
> > > > > > > > > tombstone
> > > > > > > > > bit for log compaction." What about null
value?
> > My
> > > > > > > understanding
> > > > > > > > is
> > > > > > > > > that
> > > > > > > > > null value will be treated the same as setting
> > the
> > > > > tombstone
> > > > > > > bit.
> > > > > > > > >
> > > > > > > > > 5. For the migration path, it would be useful
> to
> > > > > describe the
> > > > > > > > down
> > > > > > > > > conversion path to consumers (i.e., brokers on
> > > > message
> > > > > format
> > > > > > > > 0.10.2
> > > > > > > > > and
> > > > > > > > > consumers on older version).
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Nov 29, 2016 at 3:18 AM, Michael
> Pearce <
> > > > > > > > michael.pea...@ig.com
> > > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi All,
> > > > > > > > > >
> > > > > > > > > > We have been discussing in the below thread
> and
> > > > final
> > > > > > changes
> > > > > > > > have
> > > > > > > > > been
> > > > > > > > > > made to the KIP wiki based on these
> > discussions.
> > > > > > > > > >
> > > > > > > > > > We would now like to put to the vote the
> > > following
> > > > > KIP:
> > > > > > > > > > https://cwiki.apache.org/
> > > > > confluence/display/KAFKA/KIP-
> > > > > > 87+-+
> > > > > > > > > > Add+Compaction+Tombstone+Flag
> > > > > > > > > >
> > > > > > > > > > This kip is for having a distinct compaction
> > > > > attribute
> > > > > > > > “tombstone”
> > > > > > > > > flag
> > > > > > > > > > instead of relying on null value, allowing
> > > non-null
> > > > > value
> > > > > > > > delete
> > > > > > > > > messages.
> > > > > > > > > >
> > > > > > > > > > Many thanks,
> > > > > > > > > > Michael
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On 22/11/2016, 15:52, "Michael Pearce" <
> > > > > > > michael.pea...@ig.com>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hi Mayuresh,
> > > > > > > > > >
> > > > > > > > > > LGTM. Ive just made one small adjustment
> > > > > updating the
> > > > > > > wire
> > > > > > > > > protocol to
> > > > > > > > > > show the magic byte bump.
> > > > > > > > > >
> > > > > > > > > > Do we think we’re good to put to a vote?
> Is
> > > > > there any
> > > > > > > > other bits
> > > > > > > > > > needing discussion?
> > > > > > > > > >
> > > > > > > > > > Cheers
> > > > > > > > > > Mike
> > > > > > > > > >
> > > > > > > > > > On 21/11/2016, 18:26, "Mayuresh Gharat"
<
> > > > > > > > > gharatmayures...@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hi Michael,
> > > > > > > > > >
> > > > > > > > > > I have updated the migration section
> of
> > > the
> > > > > KIP.
> > > > > > Can
> > > > > > > > you
> > > > > > > > > please
> > > > > > > > > > take a look?
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Mayuresh
> > > > > > > > > >
> > > > > > > > > > On Fri, Nov 18, 2016 at 9:07 AM,
> > Mayuresh
> > > > > Gharat <
> > > > > > > > > > gharatmayures...@gmail.com
> > > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Michael,
> > > > > > > > > > >
> > > > > > > > > > > That whilst sending tombstone and
> non
> > > > null
> > > > > value,
> > > > > > > the
> > > > > > > > > consumer
> > > > > > > > > > can expect
> > > > > > > > > > > only to receive the non-null
> message
> > > only
> > > > > in step
> > > > > > > > (3) is
> > > > > > > > > this
> > > > > > > > > > correct?
> > > > > > > > > > > ---> I do agree with you here.
> > > > > > > > > > >
> > > > > > > > > > > Becket, Ismael : can you guys
> review
> > > the
> > > > > > migration
> > > > > > > > plan
> > > > > > > > > listed
> > > > > > > > > > above using
> > > > > > > > > > > magic byte?
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > >
> > > > > > > > > > > Mayuresh
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Nov 18, 2016 at 8:58 AM,
> > > Michael
> > > > > Pearce <
> > > > > > > > > > michael.pea...@ig.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >> Many thanks for this Mayuresh. I
> > don't
> > > > > have any
> > > > > > > > > objections.
> > > > > > > > > > >>
> > > > > > > > > > >> I assume we should state:
> > > > > > > > > > >>
> > > > > > > > > > >> That whilst sending tombstone and
> > non
> > > > null
> > > > > > value,
> > > > > > > > the
> > > > > > > > > consumer
> > > > > > > > > > can expect
> > > > > > > > > > >> only to receive the non-null
> message
> > > > only
> > > > > in
> > > > > > step
> > > > > > > > (3) is
> > > > > > > > > this
> > > > > > > > > > correct?
> > > > > > > > > > >>
> > > > > > > > > > >> Cheers
> > > > > > > > > > >> Mike
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >> Sent using OWA for iPhone
> > > > > > > > > > >> ______________________________
> > > > __________
> > > > > > > > > > >> From: Mayuresh Gharat <
> > > > > > gharatmayures...@gmail.com
> > > > > > > >
> > > > > > > > > > >> Sent: Thursday, November 17, 2016
> > > > 5:18:41
> > > > > PM
> > > > > > > > > > >> To: dev@kafka.apache.org
> > > > > > > > > > >> Subject: Re: [DISCUSS] KIP-87 -
> Add
> > > > > Compaction
> > > > > > > > Tombstone
> > > > > > > > > Flag
> > > > > > > > > > >>
> > > > > > > > > > >> Hi Ismael,
> > > > > > > > > > >>
> > > > > > > > > > >> Thanks for the explanation.
> > > > > > > > > > >> Specially I like this part where
> in
> > > you
> > > > > > mentioned
> > > > > > > > we can
> > > > > > > > > get
> > > > > > > > > > rid of the
> > > > > > > > > > >> older null value support for log
> > > > > compaction
> > > > > > later
> > > > > > > > on,
> > > > > > > > > here :
> > > > > > > > > > >> We can't change semantics of the
> > > message
> > > > > format
> > > > > > > > without
> > > > > > > > > having
> > > > > > > > > > a long
> > > > > > > > > > >> transition period. And we can't
> rely
> > > > > > > > > > >> on people reading documentation
or
> > > > acting
> > > > > on a
> > > > > > > > warning for
> > > > > > > > > > something so
> > > > > > > > > > >> fundamental. As such, my take is
> > that
> > > we
> > > > > need to
> > > > > > > > bump the
> > > > > > > > > magic
> > > > > > > > > > byte. The
> > > > > > > > > > >> good news is
> > > > > > > > > > >> that we don't have to support all
> > > > versions
> > > > > > > forever.
> > > > > > > > We
> > > > > > > > > have
> > > > > > > > > > said that we
> > > > > > > > > > >> will support direct upgrades for
2
> > > > years.
> > > > > That
> > > > > > > > means that
> > > > > > > > > > message format
> > > > > > > > > > >> version n could, in theory, be
> > > removed 2
> > > > > years
> > > > > > > > after the
> > > > > > > > > it's
> > > > > > > > > > introduced.
> > > > > > > > > > >>
> > > > > > > > > > >> Just a heads up, I would like to
> > > mention
> > > > > that
> > > > > > even
> > > > > > > > without
> > > > > > > > > > bumping magic
> > > > > > > > > > >> byte, we will *NOT* loose zero
> copy
> > as
> > > > in
> > > > > the
> > > > > > > > client(x+1)
> > > > > > > > > in my
> > > > > > > > > > >> explanation
> > > > > > > > > > >> above will convert internally a
> null
> > > > > value to
> > > > > > > have a
> > > > > > > > > tombstone
> > > > > > > > > > bit set and
> > > > > > > > > > >> a tombstone bit set to have a
null
> > > value
> > > > > > > > automatically
> > > > > > > > > > internally and by
> > > > > > > > > > >> the time we move to version
(x+2),
> > the
> > > > > clients
> > > > > > > > would have
> > > > > > > > > > upgraded.
> > > > > > > > > > >> Obviously if we support a request
> > from
> > > > > > > consumer(x),
> > > > > > > > we
> > > > > > > > > will
> > > > > > > > > > loose zero
> > > > > > > > > > >> copy
> > > > > > > > > > >> but that is the same case with
> magic
> > > > byte.
> > > > > > > > > > >>
> > > > > > > > > > >> But if magic byte bump makes life
> > > easier
> > > > > for
> > > > > > > > transition
> > > > > > > > > for the
> > > > > > > > > > above
> > > > > > > > > > >> reasons that you explained, I am
> OK
> > > with
> > > > > it
> > > > > > since
> > > > > > > > we are
> > > > > > > > > going
> > > > > > > > > > to meet the
> > > > > > > > > > >> end goal down the road :)
> > > > > > > > > > >>
> > > > > > > > > > >> On a side note can we update the
> doc
> > > > here
> > > > > on
> > > > > > magic
> > > > > > > > byte
> > > > > > > > > to say
> > > > > > > > > > that "*it
> > > > > > > > > > >> should be bumped whenever the
> > message
> > > > > format is
> > > > > > > > changed
> > > > > > > > > or the
> > > > > > > > > > >> interpretation of message format
> > > (usage
> > > > > of the
> > > > > > > > reserved
> > > > > > > > > bits as
> > > > > > > > > > well) is
> > > > > > > > > > >> changed*".
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >> Hi Michael,
> > > > > > > > > > >>
> > > > > > > > > > >> Here is the update plan that we
> > > > discussed
> > > > > > offline
> > > > > > > > > yesterday :
> > > > > > > > > > >>
> > > > > > > > > > >> Currently the magic-byte which
> > > > > corresponds to
> > > > > > the
> > > > > > > > > > "message.format.version"
> > > > > > > > > > >> is set to 1.
> > > > > > > > > > >>
> > > > > > > > > > >> 1) On broker it will be set to 1
> > > > > initially.
> > > > > > > > > > >>
> > > > > > > > > > >> 2) When a producer client sends a
> > > > message
> > > > > with
> > > > > > > > magic-byte
> > > > > > > > > = 2,
> > > > > > > > > > since the
> > > > > > > > > > >> broker is on magic-byte = 1, we
> will
> > > > down
> > > > > > convert
> > > > > > > > it,
> > > > > > > > > which
> > > > > > > > > > means if the
> > > > > > > > > > >> tombstone bit is set, the value
> will
> > > be
> > > > > set to
> > > > > > > > null. A
> > > > > > > > > consumer
> > > > > > > > > > >> understanding magic-byte = 1,
will
> > > still
> > > > > work
> > > > > > with
> > > > > > > > this. A
> > > > > > > > > > consumer
> > > > > > > > > > >> working
> > > > > > > > > > >> with magic-byte =2 will also be
> able
> > > to
> > > > > > understand
> > > > > > > > this,
> > > > > > > > > since
> > > > > > > > > > it
> > > > > > > > > > >> understands the tombstone.
> > > > > > > > > > >> Now there is still the question
of
> > > > > supporting a
> > > > > > > > > non-tombstone
> > > > > > > > > > and null
> > > > > > > > > > >> value from producer client with
> > > > > magic-byte = 2.*
> > > > > > > (I
> > > > > > > > am
> > > > > > > > > not sure
> > > > > > > > > > if we
> > > > > > > > > > >> should support this.
Ismael/Becket
> > can
> > > > > comment
> > > > > > > > here)*
> > > > > > > > > > >>
> > > > > > > > > > >> 3) When almost all the clients
> have
> > > > > upgraded,
> > > > > > the
> > > > > > > > > > message.format.version
> > > > > > > > > > >> on
> > > > > > > > > > >> the broker can be changed to 2,
> > where
> > > in
> > > > > the
> > > > > > down
> > > > > > > > > conversion in
> > > > > > > > > > the above
> > > > > > > > > > >> step will not happen. If at this
> > point
> > > > we
> > > > > get a
> > > > > > > > consumer
> > > > > > > > > > request from a
> > > > > > > > > > >> older consumer, we might have to
> > down
> > > > > convert
> > > > > > > where
> > > > > > > > in we
> > > > > > > > > loose
> > > > > > > > > > zero copy,
> > > > > > > > > > >> but these cases should be rare.
> > > > > > > > > > >>
> > > > > > > > > > >> Becket can you review this plan
> and
> > > add
> > > > > more
> > > > > > > > details if I
> > > > > > > > > have
> > > > > > > > > > >> missed/wronged something, before
> we
> > > put
> > > > > it on
> > > > > > KIP.
> > > > > > > > > > >>
> > > > > > > > > > >> Thanks,
> > > > > > > > > > >>
> > > > > > > > > > >> Mayuresh
> > > > > > > > > > >>
> > > > > > > > > > >> On Wed, Nov 16, 2016 at 11:07 PM,
> > > > Michael
> > > > > > Pearce <
> > > > > > > > > > michael.pea...@ig.com>
> > > > > > > > > > >> wrote:
> > > > > > > > > > >>
> > > > > > > > > > >> > Thanks guys, for discussing
this
> > > > > offline and
> > > > > > > > getting
> > > > > > > > > some
> > > > > > > > > > consensus.
> > > > > > > > > > >> >
> > > > > > > > > > >> > So its clear for myself and
> others
> > > > what
> > > > > is
> > > > > > > > proposed now
> > > > > > > > > (i
> > > > > > > > > > think i
> > > > > > > > > > >> > understand, but want to make
> sure)
> > > > > > > > > > >> >
> > > > > > > > > > >> > Could i ask either directly
> update
> > > the
> > > > > kip to
> > > > > > > > detail the
> > > > > > > > > > migration
> > > > > > > > > > >> > strategy, or (re-)state your
> > offline
> > > > > discussed
> > > > > > > and
> > > > > > > > > agreed
> > > > > > > > > > migration
> > > > > > > > > > >> > strategy based on a magic byte
> is
> > in
> > > > > this
> > > > > > > thread.
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> > The main original driver for
the
> > KIP
> > > > > was to
> > > > > > > > support
> > > > > > > > > > compaction where
> > > > > > > > > > >> value
> > > > > > > > > > >> > isn't null, based off the
> > > discussions
> > > > on
> > > > > > KIP-82
> > > > > > > > thread.
> > > > > > > > > > >> >
> > > > > > > > > > >> > We should be able to support
> > > > > non-tombstone +
> > > > > > > null
> > > > > > > > value
> > > > > > > > > by the
> > > > > > > > > > >> completion
> > > > > > > > > > >> > of the KIP, as we noted when
> > > > discussing
> > > > > this
> > > > > > > kip,
> > > > > > > > having
> > > > > > > > > > logic based on
> > > > > > > > > > >> a
> > > > > > > > > > >> > null value isn't very clean and
> > also
> > > > > separates
> > > > > > > the
> > > > > > > > > concerns.
> > > > > > > > > > >> >
> > > > > > > > > > >> > As discussed already though we
> can
> > > > > split this
> > > > > > > into
> > > > > > > > > KIP-87a
> > > > > > > > > > and KIP-87b
> > > > > > > > > > >> >
> > > > > > > > > > >> > Where we look to deliver
KIP-87a
> > on
> > > a
> > > > > > compacted
> > > > > > > > topic
> > > > > > > > > (to
> > > > > > > > > > address the
> > > > > > > > > > >> > immediate issues)
> > > > > > > > > > >> > * tombstone + null value
> > > > > > > > > > >> > * tombstone + non-null value
> > > > > > > > > > >> > * non-tombstone + non-null
value
> > > > > > > > > > >> >
> > > > > > > > > > >> > Then we can discuss once
KIP-87a
> > is
> > > > > completed
> > > > > > > > options
> > > > > > > > > later
> > > > > > > > > > and how we
> > > > > > > > > > >> > support the second part KIP-87b
> to
> > > > > deliver:
> > > > > > > > > > >> > * non-tombstone + null value
> > > > > > > > > > >> >
> > > > > > > > > > >> > Cheers
> > > > > > > > > > >> > Mike
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> > ______________________________
> > > > > __________
> > > > > > > > > > >> > From: Becket Qin <
> > > > becket....@gmail.com>
> > > > > > > > > > >> > Sent: Thursday, November 17,
> 2016
> > > 1:43
> > > > > AM
> > > > > > > > > > >> > To: dev@kafka.apache.org
> > > > > > > > > > >> > Subject: Re: [DISCUSS] KIP-87 -
> > Add
> > > > > Compaction
> > > > > > > > > Tombstone Flag
> > > > > > > > > > >> >
> > > > > > > > > > >> > Renu, Mayuresh and I had an
> > offline
> > > > > > discussion,
> > > > > > > > and
> > > > > > > > > following
> > > > > > > > > > is a brief
> > > > > > > > > > >> > summary.
> > > > > > > > > > >> >
> > > > > > > > > > >> > 1. We agreed that not bumping
up
> > > magic
> > > > > value
> > > > > > may
> > > > > > > > result
> > > > > > > > > in
> > > > > > > > > > losing zero
> > > > > > > > > > >> copy
> > > > > > > > > > >> > during migration.
> > > > > > > > > > >> > 2. Given that bumping up magic
> > value
> > > > is
> > > > > almost
> > > > > > > > free and
> > > > > > > > > has
> > > > > > > > > > benefit of
> > > > > > > > > > >> > avoiding potential performance
> > > issue.
> > > > > It is
> > > > > > > > probably
> > > > > > > > > worth
> > > > > > > > > > doing.
> > > > > > > > > > >> >
> > > > > > > > > > >> > One issue we still need to
think
> > > about
> > > > > is
> > > > > > > whether
> > > > > > > > we
> > > > > > > > > want to
> > > > > > > > > > support a
> > > > > > > > > > >> > non-tombstone message with null
> > > value.
> > > > > > > > > > >> > Currently it is not supported
by
> > > > Kafka.
> > > > > If we
> > > > > > > > allow a
> > > > > > > > > > non-tombstone null
> > > > > > > > > > >> > value message to exist after
> > KIP-87.
> > > > The
> > > > > > problem
> > > > > > > > is
> > > > > > > > > that such
> > > > > > > > > > message
> > > > > > > > > > >> will
> > > > > > > > > > >> > not be supported by the
> consumers
> > > > prior
> > > > > to
> > > > > > > KIP-87.
> > > > > > > > > Because a
> > > > > > > > > > null value
> > > > > > > > > > >> > will always be interpreted to a
> > > > > tombstone.
> > > > > > > > > > >> >
> > > > > > > > > > >> > One option is that we keep the
> > > current
> > > > > way,
> > > > > > i.e.
> > > > > > > > do not
> > > > > > > > > > support such
> > > > > > > > > > >> > message. It would be good to
> know
> > if
> > > > > there is
> > > > > > a
> > > > > > > > > concrete use
> > > > > > > > > > case for
> > > > > > > > > > >> such
> > > > > > > > > > >> > message. If there is not, we
can
> > > > > probably just
> > > > > > > not
> > > > > > > > > support it.
> > > > > > > > > > >> >
> > > > > > > > > > >> > Thanks,
> > > > > > > > > > >> >
> > > > > > > > > > >> > JIangjie (Becket) Qin
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> > On Wed, Nov 16, 2016 at 1:28
PM,
> > > > > Mayuresh
> > > > > > > Gharat <
> > > > > > > > > > >> > gharatmayures...@gmail.com
> > > > > > > > > > >> > > wrote:
> > > > > > > > > > >> >
> > > > > > > > > > >> > > Hi Ismael,
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > This is something I can think
> of
> > > for
> > > > > > migration
> > > > > > > > plan:
> > > > > > > > > > >> > > So the migration plan can
look
> > > > > something
> > > > > > like
> > > > > > > > this,
> > > > > > > > > with up
> > > > > > > > > > >> conversion :
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > 1) Currently lets say we have
> > > Broker
> > > > > at
> > > > > > > version
> > > > > > > > x.
> > > > > > > > > > >> > > 2) Currently we have clients
> at
> > > > > version x.
> > > > > > > > > > >> > > 3) a) We move the version to
> > > > > Broker(x+1) :
> > > > > > > > supports
> > > > > > > > > both
> > > > > > > > > > tombstone and
> > > > > > > > > > >> > null
> > > > > > > > > > >> > > for log compaction.
> > > > > > > > > > >> > > b) We upgrade the client
> to
> > > > > version
> > > > > > > > client(x+1) :
> > > > > > > > > if in
> > > > > > > > > > the
> > > > > > > > > > >> producer
> > > > > > > > > > >> > > client(x+1) the value is set
> to
> > > > null,
> > > > > we
> > > > > > will
> > > > > > > > > automatically
> > > > > > > > > > set the
> > > > > > > > > > >> > > Tombstone bit internally. If
> the
> > > > > producer
> > > > > > > > client(x+1)
> > > > > > > > > sets
> > > > > > > > > > the
> > > > > > > > > > >> tombstone
> > > > > > > > > > >> > > itself, well and good. For
> > > producer
> > > > > > client(x),
> > > > > > > > the
> > > > > > > > > broker
> > > > > > > > > > will up
> > > > > > > > > > >> convert
> > > > > > > > > > >> > > to have the tombstone bit.
> > > > > Broker(x+1) is
> > > > > > > > supporting
> > > > > > > > > both.
> > > > > > > > > > Consumer
> > > > > > > > > > >> > > client(x+1) will be aware of
> > this
> > > > and
> > > > > should
> > > > > > > be
> > > > > > > > able
> > > > > > > > > to
> > > > > > > > > > handle this.
> > > > > > > > > > >> For
> > > > > > > > > > >> > > consumer client(x) we will
> down
> > > > > convert the
> > > > > > > > message
> > > > > > > > > on the
> > > > > > > > > > broker
> > > > > > > > > > >> side.
> > > > > > > > > > >> > > c) At this point we will
> > have
> > > to
> > > > > > specify a
> > > > > > > > > warning or
> > > > > > > > > > clearly
> > > > > > > > > > >> specify
> > > > > > > > > > >> > > in docs that this behavior is
> > > about
> > > > > to be
> > > > > > > > changed for
> > > > > > > > > log
> > > > > > > > > > compaction.
> > > > > > > > > > >> > > 4) a) In next release of the
> > > > > Broker(x+2), we
> > > > > > > > say that
> > > > > > > > > only
> > > > > > > > > > Tombstone
> > > > > > > > > > >> is
> > > > > > > > > > >> > > used for log compaction on
the
> > > > Broker
> > > > > side.
> > > > > > > > > Clients(x+1)
> > > > > > > > > > still is
> > > > > > > > > > >> > > supported.
> > > > > > > > > > >> > > b) We upgrade the client
> to
> > > > > version
> > > > > > > > client(x+2) :
> > > > > > > > > if
> > > > > > > > > > value is set
> > > > > > > > > > >> to
> > > > > > > > > > >> > > null, tombstone will not be
> set
> > > > > > automatically.
> > > > > > > > The
> > > > > > > > > client
> > > > > > > > > > will have to
> > > > > > > > > > >> > call
> > > > > > > > > > >> > > setTombstone() to actually
set
> > the
> > > > > > tombstone.
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > We should compare this
> migration
> > > > plan
> > > > > with
> > > > > > the
> > > > > > > > > migration
> > > > > > > > > > plan for
> > > > > > > > > > >> magic
> > > > > > > > > > >> > > byte bump and do whatever
> looks
> > > > good.
> > > > > > > > > > >> > > I am just worried that if we
> go
> > > down
> > > > > magic
> > > > > > > byte
> > > > > > > > route,
> > > > > > > > > > unless I am
> > > > > > > > > > >> > missing
> > > > > > > > > > >> > > something, it sounds like
> kafka
> > > will
> > > > > be
> > > > > > stuck
> > > > > > > > with
> > > > > > > > > > supporting both
> > > > > > > > > > >> null
> > > > > > > > > > >> > > value and tombstone bit for
> log
> > > > > compaction
> > > > > > for
> > > > > > > > life
> > > > > > > > > long,
> > > > > > > > > > which does
> > > > > > > > > > >> not
> > > > > > > > > > >> > > look like a good end state.
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > Thanks,
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > Mayuresh
> > > > > > > > > > >> > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > On Wed, Nov 16, 2016 at 9:32
> AM,
> > > > > Mayuresh
> > > > > > > > Gharat <
> > > > > > > > > > >> > > gharatmayures...@gmail.com
> > > > > > > > > > >> > > > wrote:
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > > Hi Ismael,
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > That's a very good point
> > which I
> > > > > might
> > > > > > have
> > > > > > > > not
> > > > > > > > > > considered earlier.
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > Here is a plan that I can
> > think
> > > > of:
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > Stage 1) The broker from
now
> > on,
> > > > up
> > > > > > converts
> > > > > > > > the
> > > > > > > > > message
> > > > > > > > > > to have the
> > > > > > > > > > >> > > > tombstone marker. The log
> > > > compaction
> > > > > > thread
> > > > > > > > does log
> > > > > > > > > > compaction
> > > > > > > > > > >> based
> > > > > > > > > > >> > on
> > > > > > > > > > >> > > > both null and tombstone
> > marker.
> > > > > This is
> > > > > > our
> > > > > > > > > transition
> > > > > > > > > > period.
> > > > > > > > > > >> > > > Stage 2) The next release
we
> > > only
> > > > > say that
> > > > > > > log
> > > > > > > > > compaction
> > > > > > > > > > is based
> > > > > > > > > > >> on
> > > > > > > > > > >> > > > tombstone marker. (Open
> source
> > > > > kafka makes
> > > > > > > > this as a
> > > > > > > > > > policy). By
> > > > > > > > > > >> this
> > > > > > > > > > >> > > time,
> > > > > > > > > > >> > > > the organization which is
> > moving
> > > > to
> > > > > this
> > > > > > > > release
> > > > > > > > > will be
> > > > > > > > > > sure that
> > > > > > > > > > >> they
> > > > > > > > > > >> > > > have gone through the
entire
> > > > > transition
> > > > > > > > period.
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > My only goal of doing this
> is
> > > that
> > > > > Kafka
> > > > > > > > clearly
> > > > > > > > > > specifies the end
> > > > > > > > > > >> > state
> > > > > > > > > > >> > > > about what log compaction
> > means
> > > > (is
> > > > > it
> > > > > > null
> > > > > > > > value
> > > > > > > > > or a
> > > > > > > > > > tombstone
> > > > > > > > > > >> > marker,
> > > > > > > > > > >> > > > but not both).
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > What do you think?
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > Thanks,
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > Mayuresh
> > > > > > > > > > >> > > > .
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > On Wed, Nov 16, 2016 at
9:17
> > AM,
> > > > > Ismael
> > > > > > > Juma <
> > > > > > > > > > ism...@juma.me.uk>
> > > > > > > > > > >> > wrote:
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > >> One comment below.
> > > > > > > > > > >> > > >>
> > > > > > > > > > >> > > >> On Wed, Nov 16, 2016 at
> 5:08
> > > PM,
> > > > > Mayuresh
> > > > > > > > Gharat <
> > > > > > > > > > >> > > >> gharatmayures...@gmail.com
> > > > > > > > > > >> > > >> > wrote:
> > > > > > > > > > >> > > >>
> > > > > > > > > > >> > > >> > - If we don't bump up
> > the
> > > > > magic
> > > > > > byte,
> > > > > > > > on the
> > > > > > > > > broker
> > > > > > > > > > side, the
> > > > > > > > > > >> > > broker
> > > > > > > > > > >> > > >> > will always have to
> look
> > > at
> > > > > both
> > > > > > > > tombstone
> > > > > > > > > bit and
> > > > > > > > > > the value
> > > > > > > > > > >> when
> > > > > > > > > > >> > > do
> > > > > > > > > > >> > > >> the
> > > > > > > > > > >> > > >> > compaction. Assuming
> we
> > do
> > > > > not bump
> > > > > > up
> > > > > > > > the
> > > > > > > > > magic
> > > > > > > > > > byte,
> > > > > > > > > > >> > > >> > imagine the broker
> sees
> > a
> > > > > message
> > > > > > > which
> > > > > > > > does
> > > > > > > > > not
> > > > > > > > > > have a
> > > > > > > > > > >> tombstone
> > > > > > > > > > >> > > bit
> > > > > > > > > > >> > > >> > set. The broker does
> not
> > > > know
> > > > > when
> > > > > > the
> > > > > > > > > message was
> > > > > > > > > > produced
> > > > > > > > > > >> (i.e.
> > > > > > > > > > >> > > >> > whether
> > > > > > > > > > >> > > >> > the message has been
> up
> > > > > converted or
> > > > > > > > not), it
> > > > > > > > > has
> > > > > > > > > > to take a
> > > > > > > > > > >> > further
> > > > > > > > > > >> > > >> > look at
> > > > > > > > > > >> > > >> > the value to see if
it
> > is
> > > > > null or
> > > > > > not
> > > > > > > in
> > > > > > > > > order to
> > > > > > > > > > determine
> > > > > > > > > > >> if it
> > > > > > > > > > >> > > is
> > > > > > > > > > >> > > >> a
> > > > > > > > > > >> > > >> > tombstone. The same
> > logic
> > > > has
> > > > > to be
> > > > > > > put
> > > > > > > > on the
> > > > > > > > > > consumer as
> > > > > > > > > > >> well
> > > > > > > > > > >> > > >> because
> > > > > > > > > > >> > > >> > the
> > > > > > > > > > >> > > >> > consumer does not
know
> > if
> > > > the
> > > > > > message
> > > > > > > > has
> > > > > > > > > been up
> > > > > > > > > > converted or
> > > > > > > > > > >> > not.
> > > > > > > > > > >> > > >> > - If we upconvert
> > while
> > > > > > appending,
> > > > > > > > this is
> > > > > > > > > not
> > > > > > > > > > the case,
> > > > > > > > > > >> > right?
> > > > > > > > > > >> > > >>
> > > > > > > > > > >> > > >>
> > > > > > > > > > >> > > >> If I understand you
> > correctly,
> > > > > this is
> > > > > > not
> > > > > > > > > sufficient
> > > > > > > > > > because the
> > > > > > > > > > >> log
> > > > > > > > > > >> > > may
> > > > > > > > > > >> > > >> have messages appended
> before
> > > it
> > > > > was
> > > > > > > > upgraded to
> > > > > > > > > include
> > > > > > > > > > KIP-87.
> > > > > > > > > > >> > > >>
> > > > > > > > > > >> > > >> Ismael
> > > > > > > > > > >> > > >>
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > --
> > > > > > > > > > >> > > > -Regards,
> > > > > > > > > > >> > > > Mayuresh R. Gharat
> > > > > > > > > > >> > > > (862) 250-7125
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > --
> > > > > > > > > > >> > > -Regards,
> > > > > > > > > > >> > > Mayuresh R. Gharat
> > > > > > > > > > >> > > (862) 250-7125
> > > > > > > > > > >> > >
> > > > > > > > > > >> > The information contained in
> this
> > > > email
> > > > > is
> > > > > > > > strictly
> > > > > > > > > > confidential and for
> > > > > > > > > > >> > the use of the addressee only,
> > > unless
> > > > > > otherwise
> > > > > > > > > indicated. If
> > > > > > > > > > you are
> > > > > > > > > > >> not
> > > > > > > > > > >> > the intended recipient, please
> do
> > > not
> > > > > read,
> > > > > > > copy,
> > > > > > > > use or
> > > > > > > > > > disclose to
> > > > > > > > > > >> others
> > > > > > > > > > >> > this message or any attachment.
> > > Please
> > > > > also
> > > > > > > > notify the
> > > > > > > > > sender
> > > > > > > > > > by
> > > > > > > > > > >> replying
> > > > > > > > > > >> > to this email or by telephone
> > > (+44(020
> > > > > 7896
> > > > > > > 0011)
> > > > > > > > and
> > > > > > > > > then
> > > > > > > > > > delete the
> > > > > > > > > > >> email
> > > > > > > > > > >> > and any copies of it. Opinions,
> > > > > conclusion
> > > > > > (etc)
> > > > > > > > that
> > > > > > > > > do not
> > > > > > > > > > relate to
> > > > > > > > > > >> the
> > > > > > > > > > >> > official business of this
> company
> > > > shall
> > > > > be
> > > > > > > > understood as
> > > > > > > > > > neither given
> > > > > > > > > > >> nor
> > > > > > > > > > >> > endorsed by it. IG is a trading
> > name
> > > > of
> > > > > IG
> > > > > > > Markets
> > > > > > > > > Limited (a
> > > > > > > > > > company
> > > > > > > > > > >> > registered in England and
Wales,
> > > > company
> > > > > > number
> > > > > > > > > 04008957) and
> > > > > > > > > > IG Index
> > > > > > > > > > >> > Limited (a company registered
in
> > > > > England and
> > > > > > > > Wales,
> > > > > > > > > company
> > > > > > > > > > number
> > > > > > > > > > >> > 01190902). Registered address
at
> > > > Cannon
> > > > > Bridge
> > > > > > > > House, 25
> > > > > > > > > > Dowgate Hill,
> > > > > > > > > > >> > London EC4R 2YA. Both IG
Markets
> > > > Limited
> > > > > > > (register
> > > > > > > > > number
> > > > > > > > > > 195355) and IG
> > > > > > > > > > >> > Index Limited (register number
> > > 114059)
> > > > > are
> > > > > > > > authorised
> > > > > > > > > and
> > > > > > > > > > regulated by
> > > > > > > > > > >> the
> > > > > > > > > > >> > Financial Conduct Authority.
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >> --
> > > > > > > > > > >> -Regards,
> > > > > > > > > > >> Mayuresh R. Gharat
> > > > > > > > > > >> (862) 250-7125
> > > > > > > > > > >> The information contained in this
> > > email
> > > > is
> > > > > > > strictly
> > > > > > > > > > confidential and for
> > > > > > > > > > >> the use of the addressee only,
> > unless
> > > > > otherwise
> > > > > > > > > indicated. If
> > > > > > > > > > you are not
> > > > > > > > > > >> the intended recipient, please do
> > not
> > > > > read,
> > > > > > copy,
> > > > > > > > use or
> > > > > > > > > > disclose to others
> > > > > > > > > > >> this message or any attachment.
> > Please
> > > > > also
> > > > > > notify
> > > > > > > > the
> > > > > > > > > sender
> > > > > > > > > > by replying
> > > > > > > > > > >> to this email or by telephone
> > (+44(020
> > > > > 7896
> > > > > > 0011)
> > > > > > > > and then
> > > > > > > > > > delete the email
> > > > > > > > > > >> and any copies of it. Opinions,
> > > > > conclusion (etc)
> > > > > > > > that do
> > > > > > > > > not
> > > > > > > > > > relate to the
> > > > > > > > > > >> official business of this company
> > > shall
> > > > be
> > > > > > > > understood as
> > > > > > > > > > neither given nor
> > > > > > > > > > >> endorsed by it. IG is a trading
> name
> > > of
> > > > IG
> > > > > > Markets
> > > > > > > > > Limited (a
> > > > > > > > > > company
> > > > > > > > > > >> registered in England and Wales,
> > > company
> > > > > number
> > > > > > > > 04008957)
> > > > > > > > > and
> > > > > > > > > > IG Index
> > > > > > > > > > >> Limited (a company registered in
> > > England
> > > > > and
> > > > > > > Wales,
> > > > > > > > > company
> > > > > > > > > > number
> > > > > > > > > > >> 01190902). Registered address at
> > > Cannon
> > > > > Bridge
> > > > > > > > House, 25
> > > > > > > > > > Dowgate Hill,
> > > > > > > > > > >> London EC4R 2YA. Both IG Markets
> > > Limited
> > > > > > (register
> > > > > > > > number
> > > > > > > > > > 195355) and IG
> > > > > > > > > > >> Index Limited (register number
> > 114059)
> > > > are
> > > > > > > > authorised and
> > > > > > > > > > regulated by the
> > > > > > > > > > >> Financial Conduct Authority.
> > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > -Regards,
> > > > > > > > > > > Mayuresh R. Gharat
> > > > > > > > > > > (862) 250-7125
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > -Regards,
> > > > > > > > > > Mayuresh R. Gharat
> > > > > > > > > > (862) 250-7125
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > The information contained in this email
> is
> > > > > strictly
> > > > > > > > confidential
> > > > > > > > > and
> > > > > > > > > > for the use of the addressee only, unless
> > > otherwise
> > > > > > > indicated.
> > > > > > > > If
> > > > > > > > > you are
> > > > > > > > > > not the intended recipient, please do not
> read,
> > > > > copy, use
> > > > > > or
> > > > > > > > > disclose to
> > > > > > > > > > others this message or any attachment.
Please
> > > also
> > > > > notify
> > > > > > the
> > > > > > > > sender
> > > > > > > > > by
> > > > > > > > > > replying to this email or by telephone
> (+44(020
> > > > 7896
> > > > > 0011)
> > > > > > > and
> > > > > > > > then
> > > > > > > > > delete
> > > > > > > > > > the email and any copies of it. Opinions,
> > > > conclusion
> > > > > (etc)
> > > > > > > > that do
> > > > > > > > > not
> > > > > > > > > > relate to the official business of this
> company
> > > > > shall be
> > > > > > > > understood
> > > > > > > > > as
> > > > > > > > > > neither given nor endorsed by it. IG is a
> > trading
> > > > > name of
> > > > > > IG
> > > > > > > > Markets
> > > > > > > > > > Limited (a company registered in England and
> > > Wales,
> > > > > company
> > > > > > > > number
> > > > > > > > > > 04008957) and IG Index Limited (a company
> > > > registered
> > > > > in
> > > > > > > > England and
> > > > > > > > > Wales,
> > > > > > > > > > company number 01190902). Registered address
> at
> > > > > Cannon
> > > > > > Bridge
> > > > > > > > House,
> > > > > > > > > 25
> > > > > > > > > > Dowgate Hill, London EC4R 2YA. Both IG
> Markets
> > > > > Limited
> > > > > > > > (register
> > > > > > > > > number
> > > > > > > > > > 195355) and IG Index Limited (register
number
> > > > > 114059) are
> > > > > > > > authorised
> > > > > > > > > and
> > > > > > > > > > regulated by the Financial Conduct
Authority.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > The information contained in this email is
strictly
> > > > > confidential
> > > > > > > and
> > > > > > > > for
> > > > > > > > > the use of the addressee only, unless otherwise
> > > > indicated.
> > > > > If you
> > > > > > > > are not
> > > > > > > > > the intended recipient, please do not read, copy,
> use
> > > or
> > > > > disclose
> > > > > > > to
> > > > > > > > others
> > > > > > > > > this message or any attachment. Please also notify
> > the
> > > > > sender by
> > > > > > > > replying
> > > > > > > > > to this email or by telephone (+44(020 7896 0011
> <020%207896%200011>) and
> > > > then
> > > > > delete
> > > > > > > > the email
> > > > > > > > > and any copies of it. Opinions, conclusion (etc)
> that
> > > do
> > > > > not
> > > > > > relate
> > > > > > > > to the
> > > > > > > > > official business of this company shall be
> understood
> > > as
> > > > > neither
> > > > > > > > given nor
> > > > > > > > > endorsed by it. IG is a trading name of IG Markets
> > > > Limited
> > > > > (a
> > > > > > > company
> > > > > > > > > registered in England and Wales, company number
> > > 04008957)
> > > > > and IG
> > > > > > > > Index
> > > > > > > > > Limited (a company registered in England and
Wales,
> > > > company
> > > > > > number
> > > > > > > > > 01190902). Registered address at Cannon Bridge
> House,
> > > 25
> > > > > Dowgate
> > > > > > > > Hill,
> > > > > > > > > London EC4R 2YA. Both IG Markets Limited (register
> > > number
> > > > > 195355)
> > > > > > > > and IG
> > > > > > > > > Index Limited (register number 114059) are
> authorised
> > > and
> > > > > > regulated
> > > > > > > > by the
> > > > > > > > > Financial Conduct Authority.
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > The information contained in this email is strictly
> > > > confidential
> > > > > and
> > > > > > for
> > > > > > > > the use of the addressee only, unless otherwise
> indicated.
> > If
> > > > > you are
> > > > > > not
> > > > > > > > the intended recipient, please do not read, copy, use or
> > > > > disclose to
> > > > > > > others
> > > > > > > > this message or any attachment. Please also notify the
> > sender
> > > > by
> > > > > > replying
> > > > > > > > to this email or by telephone (+44(020 7896 0011
> <020%207896%200011>) and then
> > > > > delete the
> > > > > > > email
> > > > > > > > and any copies of it. Opinions, conclusion (etc) that do
> > not
> > > > > relate to
> > > > > > > the
> > > > > > > > official business of this company shall be understood as
> > > > neither
> > > > > given
> > > > > > > nor
> > > > > > > > endorsed by it. IG is a trading name of IG Markets
> Limited
> > (a
> > > > > company
> > > > > > > > registered in England and Wales, company number
04008957)
> > and
> > > > IG
> > > > > Index
> > > > > > > > Limited (a company registered in England and Wales,
> company
> > > > > number
> > > > > > > > 01190902). Registered address at Cannon Bridge House, 25
> > > > Dowgate
> > > > > Hill,
> > > > > > > > London EC4R 2YA. Both IG Markets Limited (register
number
> > > > > 195355) and
> > > > > > IG
> > > > > > > > Index Limited (register number 114059) are authorised
and
> > > > > regulated by
> > > > > > > the
> > > > > > > > Financial Conduct Authority.
> > > > > > > >
> > > > > > > The information contained in this email is strictly
> > > confidential
> > > > > and for
> > > > > > > the use of the addressee only, unless otherwise indicated.
> If
> > > you
> > > > > are not
> > > > > > > the intended recipient, please do not read, copy, use or
> > > disclose
> > > > > to
> > > > > > others
> > > > > > > this message or any attachment. Please also notify the
> sender
> > > by
> > > > > replying
> > > > > > > to this email or by telephone (+44(020 7896 0011
> <020%207896%200011>) and then
> > > delete
> > > > > the
> > > > > > email
> > > > > > > and any copies of it. Opinions, conclusion (etc) that do
> not
> > > > > relate to
> > > > > > the
> > > > > > > official business of this company shall be understood as
> > > neither
> > > > > given
> > > > > > nor
> > > > > > > endorsed by it. IG is a trading name of IG Markets Limited
> (a
> > > > > company
> > > > > > > registered in England and Wales, company number 04008957)
> and
> > > IG
> > > > > Index
> > > > > > > Limited (a company registered in England and Wales,
company
> > > > number
> > > > > > > 01190902). Registered address at Cannon Bridge House, 25
> > > Dowgate
> > > > > Hill,
> > > > > > > London EC4R 2YA. Both IG Markets Limited (register number
> > > 195355)
> > > > > and IG
> > > > > > > Index Limited (register number 114059) are authorised and
> > > > > regulated by
> > > > > > the
> > > > > > > Financial Conduct Authority.
> > > > > > >
> > > > > > >
> > > > > > The information contained in this email is strictly
> > confidential
> > > > and
> > > > > for
> > > > > > the use of the addressee only, unless otherwise indicated.
If
> > you
> > > > > are not
> > > > > > the intended recipient, please do not read, copy, use or
> > disclose
> > > > to
> > > > > others
> > > > > > this message or any attachment. Please also notify the
sender
> > by
> > > > > replying
> > > > > > to this email or by telephone (+44(020 7896 0011
> <020%207896%200011>) and then
> > delete
> > > > > the email
> > > > > > and any copies of it. Opinions, conclusion (etc) that do not
> > > relate
> > > > > to the
> > > > > > official business of this company shall be understood as
> > neither
> > > > > given nor
> > > > > > endorsed by it. IG is a trading name of IG Markets Limited
(a
> > > > company
> > > > > > registered in England and Wales, company number 04008957)
and
> > IG
> > > > > Index
> > > > > > Limited (a company registered in England and Wales, company
> > > number
> > > > > > 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate
> > > > > Hill,
> > > > > > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355)
> > > > > and IG
> > > > > > Index Limited (register number 114059) are authorised and
> > > regulated
> > > > > by the
> > > > > > Financial Conduct Authority.
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > The information contained in this email is strictly confidential
> and
> > > for
> > > > > the use of the addressee only, unless otherwise indicated. If you
> are
> > > not
> > > > > the intended recipient, please do not read, copy, use or disclose
> to
> > > > others
> > > > > this message or any attachment. Please also notify the sender by
> > > replying
> > > > > to this email or by telephone (+44(020 7896 0011
> <020%207896%200011>) and then delete the
> > > > email
> > > > > and any copies of it. Opinions, conclusion (etc) that do not
relate
> > to
> > > > the
> > > > > official business of this company shall be understood as neither
> > given
> > > > nor
> > > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> > > > > registered in England and Wales, company number 04008957) and IG
> > Index
> > > > > Limited (a company registered in England and Wales, company number
> > > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> > Hill,
> > > > > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> and
> > > IG
> > > > > Index Limited (register number 114059) are authorised and
regulated
> > by
> > > > the
> > > > > Financial Conduct Authority.
> > > > >
> > > > The information contained in this email is strictly confidential and
> > for
> > > > the use of the addressee only, unless otherwise indicated. If you
are
> > not
> > > > the intended recipient, please do not read, copy, use or disclose to
> > > others
> > > > this message or any attachment. Please also notify the sender by
> > replying
> > > > to this email or by telephone (+44(020 7896 0011
<020%207896%200011>)
> and then delete the
> > > email
> > > > and any copies of it. Opinions, conclusion (etc) that do not relate
> to
> > > the
> > > > official business of this company shall be understood as neither
> given
> > > nor
> > > > endorsed by it. IG is a trading name of IG Markets Limited (a
company
> > > > registered in England and Wales, company number 04008957) and IG
> Index
> > > > Limited (a company registered in England and Wales, company number
> > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
> > > > London EC4R 2YA. Both IG Markets Limited (register number 195355)
and
> > IG
> > > > Index Limited (register number 114059) are authorised and regulated
> by
> > > the
> > > > Financial Conduct Authority.
> > > >
> > > The information contained in this email is strictly confidential and
> for
> > > the use of the addressee only, unless otherwise indicated. If you are
> not
> > > the intended recipient, please do not read, copy, use or disclose to
> > others
> > > this message or any attachment. Please also notify the sender by
> replying
> > > to this email or by telephone (+44(020 7896 0011 <020%207896%200011>)
> and then delete the
> > email
> > > and any copies of it. Opinions, conclusion (etc) that do not relate to
> > the
> > > official business of this company shall be understood as neither given
> > nor
> > > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > > registered in England and Wales, company number 04008957) and IG Index
> > > Limited (a company registered in England and Wales, company number
> > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> IG
> > > Index Limited (register number 114059) are authorised and regulated by
> > the
> > > Financial Conduct Authority.
> > >
> > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are
not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by
replying
> > to this email or by telephone (+44(020 7896 0011 <020%207896%200011>)
> and then delete the email
> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> > official business of this company shall be understood as neither given
> nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> > Index Limited (register number 114059) are authorised and regulated by
> the
> > Financial Conduct Authority.
> >
>
The information contained in this email is strictly confidential and for the
use of the addressee only, unless otherwise indicated. If you are not the
intended recipient, please do not read, copy, use or disclose to others this
message or any attachment. Please also notify the sender by replying to this
email or by telephone (+44(020 7896 0011) and then delete the email and any
copies of it. Opinions, conclusion (etc) that do not relate to the official
business of this company shall be understood as neither given nor endorsed by
it. IG is a trading name of IG Markets Limited (a company registered in England
and Wales, company number 04008957) and IG Index Limited (a company registered
in England and Wales, company number 01190902). Registered address at Cannon
Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited
(register number 195355) and IG Index Limited (register number 114059) are
authorised and regulated by the Financial Conduct Authority.
