my personal opinion - a log compacted topic is basically a kv-store, so a map API. map.put(key, null) is not the same as map.remove(key), which to me means a null value should not represent a delete. a delete should be explicit (meaning flag).
On Wed, Nov 9, 2016 at 11:01 AM, Mayuresh Gharat <gharatmayures...@gmail.com > wrote: > I see the reasoning and might be inclined to agree a bit : > If we go to stage 2, the only difference is that we can theoretically > support a null value non-tombstone message in a log compacted topic, but I > am not sure if that has any use case. > > But as an end goal I see that kafka should clearly specify what it means by > a tombstone : is it the attribute flag OR is it the null value. If we just > do stage 1, I don't think we are defining the end-goal completely. > Again this is more about semantics of correctness of end state. > > Thanks, > > Mayuresh > > On Wed, Nov 9, 2016 at 10:49 AM, Becket Qin <becket....@gmail.com> wrote: > > > I am not sure if we need the second stage. Wouldn't it be enough to say > > that a message is a tombstone if one of the following is true? > > 1. tombstone flag is set. > > 2. value is null. > > > > If we go to stage 2, the only difference is that we can theoretically > > support a null value non-tombstone message in a log compacted topic, but > I > > am not sure if that has any use case. > > > > Thanks, > > > > Jiangjie (Becket) Qin > > > > > > On Wed, Nov 9, 2016 at 9:23 AM, Mayuresh Gharat < > > gharatmayures...@gmail.com> > > wrote: > > > > > I think it will be a good idea. +1 > > > > > > Thanks, > > > > > > Mayuresh > > > > > > On Wed, Nov 9, 2016 at 9:13 AM, Michael Pearce <michael.pea...@ig.com> > > > wrote: > > > > > > > +1 Mayuresh, I think this is a good solution/strategy. > > > > > > > > Shall we update the KIP with this? Becket/Jun/Joel any comments to > add > > > > before we do? > > > > > > > > On 08/11/2016, 17:29, "Mayuresh Gharat" <gharatmayures...@gmail.com> > > > > wrote: > > > > > > > > I think the migration can be done in 2 stages : > > > > > > > > 1) In first stage the broker should understand the attribute flag > > as > > > > well > > > > as Null for the value for log compaction. > > > > 2) In second stage we move on to supporting only the attribute > flag > > > > for log > > > > compaction. > > > > > > > > I agree with Becket that for older clients (consumers) the broker > > > might > > > > have to down convert a message that has the attribute flag set > for > > > log > > > > compacting but has a non null value. But this should be in first > > > stage. > > > > Once all the clients have upgraded (clients start recognizing the > > > > attribute > > > > flag), we can move the broker to stage 2. > > > > > > > > Thanks, > > > > > > > > Mayuresh > > > > > > > > On Tue, Nov 8, 2016 at 12:06 AM, Michael Pearce < > > > michael.pea...@ig.com > > > > > > > > > wrote: > > > > > > > > > Also we can add further guidance: > > > > > > > > > > To avoid the below caveat to organisations by promoting of > > > > upgrading all > > > > > consumers first before relying on producing tombstone messages > > with > > > > data > > > > > > > > > > Sent using OWA for iPhone > > > > > ________________________________________ > > > > > From: Michael Pearce > > > > > Sent: Tuesday, November 8, 2016 8:03:32 AM > > > > > To: dev@kafka.apache.org > > > > > Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag > > > > > > > > > > Thanks Jun on the feedback, I think I understand the > issue/point > > > now. > > > > > > > > > > We def can add that on older client version if tombstone marker > > > make > > > > the > > > > > value null to preserve behaviour. > > > > > > > > > > There is one caveats to this: > > > > > > > > > > * we have to be clear that data is lost if reading via old > > > > client/message > > > > > format - I don't think this is a big issue as mostly the > idea/use > > > > case is > > > > > around meta data transport as such would only be as bad as > > current > > > > situation > > > > > > > > > > Re having configurable broker this was to handle cases like you > > > > described > > > > > but in another way by allowing organisation choose the > behaviour > > of > > > > the > > > > > compaction per broker or per topic so they could manage their > > > > transition to > > > > > using tombstone markers. > > > > > > > > > > On hind sight it maybe easier to just upgrade and downgrade the > > > > messages > > > > > on version as you propose. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sent using OWA for iPhone > > > > > ________________________________________ > > > > > From: Jun Rao <j...@confluent.io> > > > > > Sent: Tuesday, November 8, 2016 12:34:41 AM > > > > > To: dev@kafka.apache.org > > > > > Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag > > > > > > > > > > For the use case, one potential use case is for schema > > > registration. > > > > For > > > > > example, in Avro, a null value corresponds to a Null schema. > So, > > if > > > > you > > > > > want to be able to keep the schema id in a delete message, the > > > value > > > > can't > > > > > be null. We could get around this issue by specializing null > > value > > > > during > > > > > schema registration though. > > > > > > > > > > Now for the proposed changes. We probably should preserve > client > > > > > compatibility. If a client application is sending a null value > > to a > > > > > compacted topic, ideally, it should work the same after the > > client > > > > > upgrades. > > > > > > > > > > I am not sure about making the tombstone marker configurable, > > > > especially at > > > > > the topic level. Should we allow users to change the config > > values > > > > back and > > > > > forth, and what would be the implication? > > > > > > > > > > Thanks, > > > > > > > > > > Jun > > > > > > > > > > On Mon, Nov 7, 2016 at 10:48 AM, Becket Qin < > > becket....@gmail.com> > > > > wrote: > > > > > > > > > > > Hi Michael, > > > > > > > > > > > > Yes, changing the logic in the log cleaner makes sense. There > > > > could be > > > > > some > > > > > > other thing worth thinking (e.g. the message size change > after > > > > > conversion), > > > > > > though. > > > > > > > > > > > > The scenario I was thinking is the following: > > > > > > Imagine a distributed caching system built on top of Kafka. A > > > user > > > > is > > > > > > consuming from a topic and it is guaranteed that if the user > > > > consume to > > > > > the > > > > > > log end it will get the latest value for all the keys. > > Currently > > > > if the > > > > > > consumer sees a null value it knows the key has been removed. > > Now > > > > let's > > > > > say > > > > > > we rolled out this change. And the producer applies a message > > > with > > > > the > > > > > > tombstone flag set, but the value was not null. When we > append > > > that > > > > > message > > > > > > to the log I suppose we will not do the down conversion if > the > > > > broker has > > > > > > set the message.format.version to the latest. Because the log > > > > cleaner > > > > > won't > > > > > > touch the active log segment, so that message will be sitting > > in > > > > the > > > > > active > > > > > > segment as is. Now when a consumer that hasn't upgraded yet > > > > consumes that > > > > > > tombstone message in the active segment, it seems that the > > broker > > > > will > > > > > need > > > > > > to down convert that message to remove the value, right? In > > this > > > > case, we > > > > > > cannot wait for the log cleaner to do the down conversion > > because > > > > that > > > > > > message may have already been consumed before the log > > compaction > > > > happens. > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Nov 7, 2016 at 9:59 AM, Michael Pearce < > > > > michael.pea...@ig.com> > > > > > > wrote: > > > > > > > > > > > > > Hi Becket, > > > > > > > > > > > > > > We were thinking more about having the logic that’s in the > > > method > > > > > > > shouldRetainMessage configurable via > > http://kafka.apache.org/ > > > > > > > documentation.html#brokerconfigs at a broker/topic level. > > And > > > > then > > > > > > scrap > > > > > > > auto converting the message, and allow organisations to > > manage > > > > the > > > > > > rollout > > > > > > > of enabling of the feature. > > > > > > > (this isn’t in documentation but in response to the > > discussion > > > > thread > > > > > as > > > > > > > an alternative approach to roll out the feature) > > > > > > > > > > > > > > Does this make any more sense? > > > > > > > > > > > > > > Thanks > > > > > > > Mike > > > > > > > > > > > > > > On 11/3/16, 2:27 PM, "Becket Qin" <becket....@gmail.com> > > > wrote: > > > > > > > > > > > > > > Hi Michael, > > > > > > > > > > > > > > Do you mean using a new configuration it is just the > > > exiting > > > > > > > message.format.version config? It seems the > > > > message.format.version > > > > > > > config > > > > > > > is enough in this case. And the default value would > > always > > > > be the > > > > > > > latest > > > > > > > version. > > > > > > > > > > > > > > > Message version migration would be handled as like in > > > > KIP-32 > > > > > > > > > > > > > > Also just want to confirm on this. Today if an old > > consumer > > > > > consumes > > > > > > a > > > > > > > log > > > > > > > compacted topic and sees an empty value, it knows that > > is a > > > > > > tombstone. > > > > > > > After we start to use the attribute bit, a tombstone > > > message > > > > can > > > > > > have a > > > > > > > non-empty value. So by "like in KIP-32" you mean we > will > > > > remove the > > > > > > > value > > > > > > > to down convert the message if the consumer version is > > old, > > > > right? > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > On Wed, Nov 2, 2016 at 1:37 AM, Michael Pearce < > > > > > > michael.pea...@ig.com> > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Joel , et al. > > > > > > > > > > > > > > > > Any comments on the below idea to handle roll out / > > > > compatibility > > > > > > of > > > > > > > this > > > > > > > > feature, using a configuration? > > > > > > > > > > > > > > > > Does it make sense/clear? > > > > > > > > Does it add value? > > > > > > > > Do we want to enforce flag by default, or value by > > > > default, or > > > > > > both? > > > > > > > > > > > > > > > > Cheers > > > > > > > > Mike > > > > > > > > > > > > > > > > > > > > > > > > On 10/27/16, 4:47 PM, "Michael Pearce" < > > > > michael.pea...@ig.com> > > > > > > > wrote: > > > > > > > > > > > > > > > > Thanks, James, I think this is a really good > > addition > > > > to the > > > > > > KIP > > > > > > > > details, please feel free to amend the wiki/add the > use > > > > cases, > > > > > also > > > > > > > if any > > > > > > > > others you think of. I definitely think its > worthwhile > > > > > documenting. > > > > > > > If you > > > > > > > > can’t let me know ill add them next week (just > leaving > > > for > > > > a long > > > > > > > weekend > > > > > > > > off) > > > > > > > > > > > > > > > > Re Joel and others comments about upgrade and > > > > compatibility. > > > > > > > > > > > > > > > > Rather than trying to auto manage this. > > > > > > > > > > > > > > > > Actually maybe we make a configuration option, > both > > > at > > > > server > > > > > > > and per > > > > > > > > topic level to control the behavior of how the server > > > logic > > > > > should > > > > > > > work out > > > > > > > > if the record, is a tombstone record . > > > > > > > > > > > > > > > > e.g. > > > > > > > > > > > > > > > > key = compation.tombstone.marker > > > > > > > > > > > > > > > > value options: > > > > > > > > > > > > > > > > value (continues to use null value as tombstone > > > > marker) > > > > > > > > flag (expects to use the tombstone flag) > > > > > > > > value_or_flag (if either is true it treats the > > record > > > > as a > > > > > > > tombstone) > > > > > > > > > > > > > > > > This way on upgrade users can keep current > > behavior, > > > > and > > > > > slowly > > > > > > > > migrate to the new. Having a transition period of > using > > > > > > > value_or_flag, > > > > > > > > finally having flag only if an organization wishes to > > use > > > > null > > > > > > values > > > > > > > > without it being treated as a tombstone marker (use > > case > > > > noted > > > > > > below) > > > > > > > > > > > > > > > > Having it both global broker level and topic > > override > > > > also > > > > > > > allows some > > > > > > > > flexibility here. > > > > > > > > > > > > > > > > Cheers > > > > > > > > Mike > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 10/27/16, 8:03 AM, "James Cheng" < > > > > wushuja...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > > This KIP would definitely address a gap in > the > > > > current > > > > > > > > functionality, where you currently can't have a > > tombstone > > > > with > > > > > any > > > > > > > > associated content. > > > > > > > > > > > > > > > > That said, I'd like to talk about use cases, > to > > > > make sure > > > > > > > that > > > > > > > > this is in fact useful. The KIP should be updated > with > > > > whatever > > > > > use > > > > > > > cases > > > > > > > > we come up with. > > > > > > > > > > > > > > > > First of all, an observation: When we speak > > about > > > > log > > > > > > > compaction, > > > > > > > > we typically think of "the latest message for a key > is > > > > retained". > > > > > > In > > > > > > > that > > > > > > > > respect, a delete tombstone (i.e. a message with a > null > > > > payload) > > > > > is > > > > > > > treated > > > > > > > > the same as any other Kafka message: the latest > message > > > is > > > > > > retained. > > > > > > > It > > > > > > > > doesn't matter whether the latest message is null, or > > if > > > > the > > > > > latest > > > > > > > message > > > > > > > > has actual content. In all cases, the last message is > > > > retained. > > > > > > > > > > > > > > > > The only way a delete tombstone is treated > > > > differently > > > > > from > > > > > > > other > > > > > > > > Kafka messages is that it automatically disappears > > after > > > a > > > > while. > > > > > > > The time > > > > > > > > of deletion is specified using delete.retention.ms. > > > > > > > > > > > > > > > > So what we're really talking about is, do we > > want > > > > to > > > > > > support > > > > > > > > messages in a log-compacted topic that auto-delete > > > > themselves > > > > > after > > > > > > > a while? > > > > > > > > > > > > > > > > In a thread from 2015, there was a discussion > > on > > > > > > first-class > > > > > > > > support of headers between Roger Hoover, Felix GV, > Jun > > > > Rao, and > > > > > I. > > > > > > > See > > > > > > > > thread at https://groups.google.com/d/ > > > > msg/confluent-platform/ > > > > > > > > 8xPbjyUE_7E/yQ1AeCufL_gJ < > https://groups.google.com/d/ > > > > > > > > msg/confluent-platform/8xPbjyUE_7E/yQ1AeCufL_gJ> . > In > > > that > > > > > thread, > > > > > > > Jun > > > > > > > > raised a good question that I didn't have a good > answer > > > > for at > > > > > the > > > > > > > time: If > > > > > > > > a message is going to auto-delete itself after a > while, > > > how > > > > > > > important was > > > > > > > > the message? That is, what information did the > message > > > > contain > > > > > that > > > > > > > was > > > > > > > > important *for a while* but not so important that it > > > > needed to be > > > > > > > kept > > > > > > > > around forever? > > > > > > > > > > > > > > > > Some use cases that I can think of: > > > > > > > > > > > > > > > > 1) Tracability. I would like to know who > issued > > > > this > > > > > delete > > > > > > > > tombstone. It might include the hostname, IP of the > > > > producer of > > > > > the > > > > > > > delete. > > > > > > > > 2) Timestamps. I would like to know when this > > > > delete was > > > > > > > issued. > > > > > > > > This use case is already addressed by the > availability > > of > > > > > > per-message > > > > > > > > timestamps that came in 0.10.0 > > > > > > > > 3) Data provenance. I hope I'm using this > > phrase > > > > > correctly, > > > > > > > but > > > > > > > > what I mean is, where did this delete come from? What > > > > processing > > > > > > job > > > > > > > > emitted it? What input to the processing job caused > > this > > > > delete > > > > > to > > > > > > be > > > > > > > > produced? For example, if a record in topic A was > > > > processed and > > > > > > > caused a > > > > > > > > delete tombstone to be emitted to topic B, I might > like > > > the > > > > > offset > > > > > > > of the > > > > > > > > topic A message to be attached to the topic B > message. > > > > > > > > 4) Distributed tracing for stream topologies. > > > This > > > > might > > > > > > be a > > > > > > > > slight repeat of the above use cases. In the > > > microservices > > > > world, > > > > > > we > > > > > > > can > > > > > > > > generate call-graphs of webservices using tools like > > > > Zipkin/ > > > > > > > opentracing.io > > > > > > > > <http://opentracing.io/>, or something homegrown > like > > > > > > > > https://engineering.linkedin. > > > com/distributed-service-call- > > > > > > > > graph/real-time-distributed- > > tracing-website-performance- > > > > > > and-efficiency > > > > > > > < > > > > > > > > https://engineering.linkedin. > > > com/distributed-service-call- > > > > > > > > graph/real-time-distributed- > > tracing-website-performance- > > > > > > > and-efficiency>. > > > > > > > > I can imagine that you might want to do something > > similar > > > > for > > > > > > stream > > > > > > > > processing topologies, where stream processing jobs > > carry > > > > along > > > > > and > > > > > > > forward > > > > > > > > along a globally unique identifier, and a distributed > > > > topology > > > > > > graph > > > > > > > is > > > > > > > > generated. > > > > > > > > 5) Cases where processing a delete requires > > data > > > > that is > > > > > > not > > > > > > > > available in the message key. I'm not sure I have a > > good > > > > example > > > > > of > > > > > > > this, > > > > > > > > though. One hand-wavy example might be where I am > > > > publishing > > > > > > > documents into > > > > > > > > Kafka where the documentId is the message key, and > the > > > text > > > > > > contents > > > > > > > of the > > > > > > > > document are in the message body. And I have a > > consuming > > > > job that > > > > > > > does some > > > > > > > > analytics on the message body. If that document gets > > > > deleted, > > > > > then > > > > > > > the > > > > > > > > consuming job might need the original message body in > > > > order to > > > > > > > "delete" > > > > > > > > that message's impact from the analytics. But I'm not > > > sure > > > > that > > > > > is > > > > > > a > > > > > > > great > > > > > > > > example. If the consumer was worried about that, the > > > > consumer > > > > > would > > > > > > > > probably keep the original message around, stored by > > > > primary key. > > > > > > > And then > > > > > > > > all it would need from a delete message would be the > > > > primary key > > > > > of > > > > > > > the > > > > > > > > message. > > > > > > > > > > > > > > > > Do people think these are valid use cases? > > > > > > > > > > > > > > > > What are other use cases that people can > think > > > of? > > > > > > > > > > > > > > > > -James > > > > > > > > > > > > > > > > > On Oct 26, 2016, at 3:46 PM, Mayuresh > Gharat > > < > > > > > > > > gharatmayures...@gmail.com> wrote: > > > > > > > > > > > > > > > > > > +1 @Joel. > > > > > > > > > I think a clear migration plan of upgrading > > and > > > > > > > downgrading of > > > > > > > > server and > > > > > > > > > clients along with handling of issues that > > Joel > > > > > > mentioned, > > > > > > > on > > > > > > > > the KIP would > > > > > > > > > be really great. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > Mayuresh > > > > > > > > > > > > > > > > > > On Wed, Oct 26, 2016 at 3:31 PM, Joel > Koshy < > > > > > > > jjkosh...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > >> I'm not sure why it would be useful, but > it > > > > should be > > > > > > > > theoretically > > > > > > > > >> possible if the attribute bit alone is > > enough > > > > to mark > > > > > a > > > > > > > > tombstone. OTOH, we > > > > > > > > >> could consider that as invalid if we wish. > > > > These are > > > > > > > relevant > > > > > > > > details that > > > > > > > > >> I think should be added to the KIP. > > > > > > > > >> > > > > > > > > >> Also, in the few odd scenarios that I > > > mentioned > > > > we > > > > > > should > > > > > > > also > > > > > > > > consider > > > > > > > > >> that fetches could be coming from other > > > > > > yet-to-be-upgraded > > > > > > > > brokers in a > > > > > > > > >> cluster that is being upgraded. So we > would > > > > probably > > > > > > want > > > > > > > to > > > > > > > > continue to > > > > > > > > >> support nulls as tombstones or > down-convert > > in > > > > a way > > > > > > that > > > > > > > we > > > > > > > > are sure works > > > > > > > > >> with least surprise to fetchers. > > > > > > > > >> > > > > > > > > >> There is a slightly vague statement under > > > > > > "Compatibility, > > > > > > > > Deprecation, and > > > > > > > > >> Migration Plan" that could benefit more > > > details: > > > > > *Logic > > > > > > > would > > > > > > > > base on > > > > > > > > >> current behavior of null value or if > > tombstone > > > > flag > > > > > set > > > > > > to > > > > > > > > true, as such > > > > > > > > >> wouldn't impact any existing flows simply > > > allow > > > > new > > > > > > > producers > > > > > > > > to make use > > > > > > > > >> of the feature*. It is unclear to me based > > on > > > > that > > > > > > > whether you > > > > > > > > would > > > > > > > > >> interpret null as a tombstone if the > > tombstone > > > > > attribute > > > > > > > bit is > > > > > > > > off. > > > > > > > > >> > > > > > > > > >> On Wed, Oct 26, 2016 at 3:10 PM, Xavier > > > Léauté < > > > > > > > > xav...@confluent.io> > > > > > > > > >> wrote: > > > > > > > > >> > > > > > > > > >>> Does this mean that starting with V4 > > requests > > > > we > > > > > would > > > > > > > allow > > > > > > > > storing null > > > > > > > > >>> messages in compacted topics? The KIP > > should > > > > probably > > > > > > > clarify > > > > > > > > the > > > > > > > > >> behavior > > > > > > > > >>> for null messages where the tombstone > flag > > is > > > > not > > > > > net. > > > > > > > > >>> > > > > > > > > >>> On Wed, Oct 26, 2016 at 1:32 AM Magnus > > > > Edenhill < > > > > > > > > mag...@edenhill.se> > > > > > > > > >>> wrote: > > > > > > > > >>> > > > > > > > > >>>> 2016-10-25 21:36 GMT+02:00 Nacho Solis > > > > > > > > <nso...@linkedin.com.invalid>: > > > > > > > > >>>> > > > > > > > > >>>>> I think you probably require a > MagicByte > > > > bump if > > > > > you > > > > > > > expect > > > > > > > > correct > > > > > > > > >>>>> behavior of the system as a whole. > > > > > > > > >>>>> > > > > > > > > >>>>> From a client perspective you want to > > make > > > > sure > > > > > that > > > > > > > when you > > > > > > > > >> deliver a > > > > > > > > >>>>> message that the broker supports the > > > feature > > > > you're > > > > > > > expecting > > > > > > > > >>>>> (compaction). So, depending on the > > > behavior > > > > of the > > > > > > > broker on > > > > > > > > >>>> encountering > > > > > > > > >>>>> a previously undefined bit flag I would > > > > suggest > > > > > > making > > > > > > > some > > > > > > > > change to > > > > > > > > >>>> make > > > > > > > > >>>>> certain that flag-based compaction is > > > > supported. > > > > > I'm > > > > > > > going > > > > > > > > to guess > > > > > > > > >>> that > > > > > > > > >>>>> the MagicByte would do this. > > > > > > > > >>>>> > > > > > > > > >>>> > > > > > > > > >>>> I dont believe this is needed since it > is > > > > already > > > > > > > attributed > > > > > > > > through > > > > > > > > >> the > > > > > > > > >>>> request's API version. > > > > > > > > >>>> > > > > > > > > >>>> Producer: > > > > > > > > >>>> * if a client sends ProduceRequest V4 > then > > > > > > > attributes.bit5 > > > > > > > > indicates a > > > > > > > > >>>> tombstone > > > > > > > > >>>> * if a clients sends ProduceRequest <V4 > > then > > > > > > > attributes.bit5 > > > > > > > > is > > > > > > > > >> ignored > > > > > > > > >>>> and value==null indicates a tombstone > > > > > > > > >>>> * in both cases the on-disk messages are > > > > stored with > > > > > > > > attributes.bit5 > > > > > > > > >> (I > > > > > > > > >>>> assume?) > > > > > > > > >>>> > > > > > > > > >>>> Consumer: > > > > > > > > >>>> * if a clients sends FetchRequest V4 > > > messages > > > > are > > > > > > > > sendfile():ed > > > > > > > > >> directly > > > > > > > > >>>> from disk (with attributes.bit5) > > > > > > > > >>>> * if a client sends FetchRequest <V4 > > > messages > > > > are > > > > > > > slowpathed > > > > > > > > and > > > > > > > > >>>> translated from attributes.bit5 to > > > value=null > > > > as > > > > > > > required. > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > > >>>> That's my understanding anyway, please > > > > correct me if > > > > > > I'm > > > > > > > > wrong. > > > > > > > > >>>> > > > > > > > > >>>> /Magnus > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > > >>>>> On Tue, Oct 25, 2016 at 10:17 AM, > Magnus > > > > Edenhill < > > > > > > > > >> mag...@edenhill.se> > > > > > > > > >>>>> wrote: > > > > > > > > >>>>> > > > > > > > > >>>>>> It is safe to assume that a previously > > > > undefined > > > > > > > attributes > > > > > > > > bit > > > > > > > > >> will > > > > > > > > >>> be > > > > > > > > >>>>>> unset in protocol requests from > existing > > > > clients, > > > > > if > > > > > > > not, > > > > > > > > such a > > > > > > > > >>> client > > > > > > > > >>>>> is > > > > > > > > >>>>>> already violating the protocol and > needs > > > to > > > > be > > > > > > fixed. > > > > > > > > >>>>>> > > > > > > > > >>>>>> So I dont see a need for a MagicByte > > bump, > > > > both > > > > > > > broker and > > > > > > > > client > > > > > > > > >> has > > > > > > > > >>>> the > > > > > > > > >>>>>> information it needs to construct or > > parse > > > > the > > > > > > message > > > > > > > > according to > > > > > > > > >>>>> request > > > > > > > > >>>>>> version. > > > > > > > > >>>>>> > > > > > > > > >>>>>> > > > > > > > > >>>>>> 2016-10-25 18:48 GMT+02:00 Michael > > Pearce > > > < > > > > > > > > michael.pea...@ig.com>: > > > > > > > > >>>>>> > > > > > > > > >>>>>>> Hi Magnus, > > > > > > > > >>>>>>> > > > > > > > > >>>>>>> I was wondering if I even needed to > > > change > > > > those > > > > > > > also, as > > > > > > > > >>> technically > > > > > > > > >>>>>>> we’re just making use of a non used > > > > attribute > > > > > bit, > > > > > > > but im > > > > > > > > not > > > > > > > > >> 100% > > > > > > > > >>>> that > > > > > > > > >>>>>> it > > > > > > > > >>>>>>> be always false currently. > > > > > > > > >>>>>>> > > > > > > > > >>>>>>> If someone can say 100% it will > already > > > be > > > > set > > > > > > false > > > > > > > with > > > > > > > > current > > > > > > > > >>> and > > > > > > > > >>>>>>> historic bit wise masking techniques > > used > > > > over > > > > > the > > > > > > > time, > > > > > > > > we could > > > > > > > > >>> do > > > > > > > > >>>>> away > > > > > > > > >>>>>>> with both, and simply just start to > use > > > it. > > > > > > > Unfortunately > > > > > > > > I don’t > > > > > > > > >>>> have > > > > > > > > >>>>>> that > > > > > > > > >>>>>>> historic knowledge so was hoping it > > would > > > > be > > > > > > flagged > > > > > > > up in > > > > > > > > this > > > > > > > > >>>>>> discussion > > > > > > > > >>>>>>> thread ? > > > > > > > > >>>>>>> > > > > > > > > >>>>>>> Cheers > > > > > > > > >>>>>>> Mike > > > > > > > > >>>>>>> > > > > > > > > >>>>>>> On 10/25/16, 5:36 PM, "Magnus > > Edenhill" < > > > > > > > > mag...@edenhill.se> > > > > > > > > >>> wrote: > > > > > > > > >>>>>>> > > > > > > > > >>>>>>> Hi Michael, > > > > > > > > >>>>>>> > > > > > > > > >>>>>>> With the version bumps for Produce > > and > > > > Fetch > > > > > > > requests, > > > > > > > > do you > > > > > > > > >>>>> really > > > > > > > > >>>>>>> need > > > > > > > > >>>>>>> to bump MagicByte too? > > > > > > > > >>>>>>> > > > > > > > > >>>>>>> Regards, > > > > > > > > >>>>>>> Magnus > > > > > > > > >>>>>>> > > > > > > > > >>>>>>> > > > > > > > > >>>>>>> 2016-10-25 18:09 GMT+02:00 Michael > > > > Pearce < > > > > > > > > >>> michael.pea...@ig.com > > > > > > > > >>>>> : > > > > > > > > >>>>>>> > > > > > > > > >>>>>>>> Hi All, > > > > > > > > >>>>>>>> > > > > > > > > >>>>>>>> I would like to discuss the > following > > > KIP > > > > > > proposal: > > > > > > > > >>>>>>>> https://cwiki.apache.org/ > > > > > > > confluence/display/KAFKA/KIP- > > > > > > > > >>>>>>>> 87+-+Add+Compaction+Tombstone+Flag > > > > > > > > >>>>>>>> > > > > > > > > >>>>>>>> This is off the back of the > discussion > > > on > > > > KIP-82 > > > > > > / > > > > > > > KIP > > > > > > > > >>> meeting > > > > > > > > >>>>>>> where it > > > > > > > > >>>>>>>> was agreed to separate this issue > and > > > > feature. > > > > > > See: > > > > > > > > >>>>>>>> http://mail-archives.apache. > > > > > > > org/mod_mbox/kafka-dev/201610 > > > > > > > > . > > > > > > > > >>>>>>>> mbox/%3cCAJS3ho8OcR== > > > > EcxsJ8OP99pD2hz=iiGecWsv- > > > > > > > > >>>>>>>> EZsBsNyDcKr=g...@mail.gmail.com%3e > > > > > > > > >>>>>>>> > > > > > > > > >>>>>>>> Thanks > > > > > > > > >>>>>>>> Mike > > > > > > > > >>>>>>>> > > > > > > > > >>>>>>>> The information contained in this > > email > > > is > > > > > > strictly > > > > > > > > >>>> confidential > > > > > > > > >>>>>> and > > > > > > > > >>>>>>> for > > > > > > > > >>>>>>>> the use of the addressee only, > unless > > > > otherwise > > > > > > > indicated. > > > > > > > > >> If > > > > > > > > >>>> you > > > > > > > > >>>>>>> are not > > > > > > > > >>>>>>>> the intended recipient, please do > not > > > > read, > > > > > copy, > > > > > > > use or > > > > > > > > >>>> disclose > > > > > > > > >>>>>> to > > > > > > > > >>>>>>> others > > > > > > > > >>>>>>>> this message or any attachment. > Please > > > > also > > > > > notify > > > > > > > the > > > > > > > > >> sender > > > > > > > > >>>> by > > > > > > > > >>>>>>> replying > > > > > > > > >>>>>>>> to this email or by telephone > (+44(020 > > > > 7896 > > > > > 0011) > > > > > > > and then > > > > > > > > >>>> delete > > > > > > > > >>>>>>> the email > > > > > > > > >>>>>>>> and any copies of it. Opinions, > > > > conclusion (etc) > > > > > > > that do > > > > > > > > >> not > > > > > > > > >>>>> relate > > > > > > > > >>>>>>> to the > > > > > > > > >>>>>>>> official business of this company > > shall > > > be > > > > > > > understood as > > > > > > > > >>>> neither > > > > > > > > >>>>>>> given nor > > > > > > > > >>>>>>>> endorsed by it. IG is a trading name > > of > > > IG > > > > > Markets > > > > > > > Limited > > > > > > > > >> (a > > > > > > > > >>>>>> company > > > > > > > > >>>>>>>> registered in England and Wales, > > company > > > > number > > > > > > > 04008957) > > > > > > > > >> and > > > > > > > > >>>> IG > > > > > > > > >>>>>>> Index > > > > > > > > >>>>>>>> Limited (a company registered in > > England > > > > and > > > > > > Wales, > > > > > > > > company > > > > > > > > >>>>> number > > > > > > > > >>>>>>>> 01190902). Registered address at > > Cannon > > > > Bridge > > > > > > > House, 25 > > > > > > > > >>>> Dowgate > > > > > > > > >>>>>>> Hill, > > > > > > > > >>>>>>>> London EC4R 2YA. Both IG Markets > > Limited > > > > > (register > > > > > > > number > > > > > > > > >>>> 195355) > > > > > > > > >>>>>>> and IG > > > > > > > > >>>>>>>> Index Limited (register number > 114059) > > > are > > > > > > > authorised and > > > > > > > > >>>>> regulated > > > > > > > > >>>>>>> by the > > > > > > > > >>>>>>>> Financial Conduct Authority. > > > > > > > > >>>>>>>> > > > > > > > > >>>>>>> > > > > > > > > >>>>>>> > > > > > > > > >>>>>>> The information contained in this > email > > > is > > > > > strictly > > > > > > > > confidential > > > > > > > > >>> and > > > > > > > > >>>>> for > > > > > > > > >>>>>>> the use of the addressee only, unless > > > > otherwise > > > > > > > indicated. > > > > > > > > If you > > > > > > > > >>> are > > > > > > > > >>>>> not > > > > > > > > >>>>>>> the intended recipient, please do not > > > > read, copy, > > > > > > > use or > > > > > > > > disclose > > > > > > > > >>> to > > > > > > > > >>>>>> others > > > > > > > > >>>>>>> this message or any attachment. > Please > > > also > > > > > notify > > > > > > > the > > > > > > > > sender by > > > > > > > > >>>>> replying > > > > > > > > >>>>>>> to this email or by telephone > (+44(020 > > > > 7896 0011) > > > > > > > and then > > > > > > > > delete > > > > > > > > >>> the > > > > > > > > >>>>>> email > > > > > > > > >>>>>>> and any copies of it. Opinions, > > > conclusion > > > > (etc) > > > > > > > that do > > > > > > > > not > > > > > > > > >> relate > > > > > > > > >>>> to > > > > > > > > >>>>>> the > > > > > > > > >>>>>>> official business of this company > shall > > > be > > > > > > > understood as > > > > > > > > neither > > > > > > > > >>>> given > > > > > > > > >>>>>> nor > > > > > > > > >>>>>>> endorsed by it. IG is a trading name > of > > > IG > > > > > Markets > > > > > > > Limited > > > > > > > > (a > > > > > > > > >>> company > > > > > > > > >>>>>>> registered in England and Wales, > > company > > > > number > > > > > > > 04008957) > > > > > > > > and IG > > > > > > > > >>>> Index > > > > > > > > >>>>>>> Limited (a company registered in > > England > > > > and > > > > > Wales, > > > > > > > company > > > > > > > > >> number > > > > > > > > >>>>>>> 01190902). Registered address at > Cannon > > > > Bridge > > > > > > > House, 25 > > > > > > > > Dowgate > > > > > > > > >>>> Hill, > > > > > > > > >>>>>>> London EC4R 2YA. Both IG Markets > > Limited > > > > > (register > > > > > > > number > > > > > > > > 195355) > > > > > > > > >>> and > > > > > > > > >>>>> IG > > > > > > > > >>>>>>> Index Limited (register number > 114059) > > > are > > > > > > > authorised and > > > > > > > > >> regulated > > > > > > > > >>>> by > > > > > > > > >>>>>> the > > > > > > > > >>>>>>> Financial Conduct Authority. > > > > > > > > >>>>>>> > > > > > > > > >>>>>> > > > > > > > > >>>>> > > > > > > > > >>>>> > > > > > > > > >>>>> > > > > > > > > >>>>> -- > > > > > > > > >>>>> Nacho (Ignacio) Solis > > > > > > > > >>>>> Kafka > > > > > > > > >>>>> nso...@linkedin.com > > > > > > > > >>>>> > > > > > > > > >>>> > > > > > > > > >>> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > -Regards, > > > > > > > > > Mayuresh R. Gharat > > > > > > > > > (862) 250-7125 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The information contained in this email is > strictly > > > > > > confidential > > > > > > > and > > > > > > > > for the use of the addressee only, unless otherwise > > > > indicated. If > > > > > > > you are > > > > > > > > not the intended recipient, please do not read, copy, > > use > > > > or > > > > > > > disclose to > > > > > > > > others this message or any attachment. Please also > > notify > > > > the > > > > > > sender > > > > > > > by > > > > > > > > replying to this email or by telephone (+44(020 7896 > > > 0011) > > > > and > > > > > then > > > > > > > delete > > > > > > > > the email and any copies of it. Opinions, conclusion > > > (etc) > > > > that > > > > > do > > > > > > > not > > > > > > > > relate to the official business of this company shall > > be > > > > > understood > > > > > > > as > > > > > > > > neither given nor endorsed by it. IG is a trading > name > > of > > > > IG > > > > > > Markets > > > > > > > > Limited (a company registered in England and Wales, > > > company > > > > > number > > > > > > > > 04008957) and IG Index Limited (a company registered > in > > > > England > > > > > and > > > > > > > Wales, > > > > > > > > company number 01190902). Registered address at > Cannon > > > > Bridge > > > > > > House, > > > > > > > 25 > > > > > > > > Dowgate Hill, London EC4R 2YA. Both IG Markets > Limited > > > > (register > > > > > > > number > > > > > > > > 195355) and IG Index Limited (register number 114059) > > are > > > > > > authorised > > > > > > > and > > > > > > > > regulated by the Financial Conduct Authority. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The information contained in this email is strictly > > > confidential > > > > and > > > > > for > > > > > > > the use of the addressee only, unless otherwise indicated. > If > > > > you are > > > > > not > > > > > > > the intended recipient, please do not read, copy, use or > > > > disclose to > > > > > > others > > > > > > > this message or any attachment. Please also notify the > sender > > > by > > > > > replying > > > > > > > to this email or by telephone (+44(020 7896 0011) and then > > > > delete the > > > > > > email > > > > > > > and any copies of it. Opinions, conclusion (etc) that do > not > > > > relate to > > > > > > the > > > > > > > official business of this company shall be understood as > > > neither > > > > given > > > > > > nor > > > > > > > endorsed by it. IG is a trading name of IG Markets Limited > (a > > > > company > > > > > > > registered in England and Wales, company number 04008957) > and > > > IG > > > > Index > > > > > > > Limited (a company registered in England and Wales, company > > > > number > > > > > > > 01190902). Registered address at Cannon Bridge House, 25 > > > Dowgate > > > > Hill, > > > > > > > London EC4R 2YA. Both IG Markets Limited (register number > > > > 195355) and > > > > > IG > > > > > > > Index Limited (register number 114059) are authorised and > > > > regulated by > > > > > > the > > > > > > > Financial Conduct Authority. > > > > > > > > > > > > > > > > > > The information contained in this email is strictly > confidential > > > and > > > > for > > > > > the use of the addressee only, unless otherwise indicated. If > you > > > > are not > > > > > the intended recipient, please do not read, copy, use or > disclose > > > to > > > > others > > > > > this message or any attachment. Please also notify the sender > by > > > > replying > > > > > to this email or by telephone (+44(020 7896 0011) and then > delete > > > > the email > > > > > and any copies of it. Opinions, conclusion (etc) that do not > > relate > > > > to the > > > > > official business of this company shall be understood as > neither > > > > given nor > > > > > endorsed by it. IG is a trading name of IG Markets Limited (a > > > company > > > > > registered in England and Wales, company number 04008957) and > IG > > > > Index > > > > > Limited (a company registered in England and Wales, company > > number > > > > > 01190902). Registered address at Cannon Bridge House, 25 > Dowgate > > > > Hill, > > > > > London EC4R 2YA. Both IG Markets Limited (register number > 195355) > > > > and IG > > > > > Index Limited (register number 114059) are authorised and > > regulated > > > > by the > > > > > Financial Conduct Authority. > > > > > > > > > > > > > > > > > > > > > -- > > > > -Regards, > > > > Mayuresh R. Gharat > > > > (862) 250-7125 > > > > > > > > > > > > The information contained in this email is strictly confidential and > > for > > > > the use of the addressee only, unless otherwise indicated. If you are > > not > > > > the intended recipient, please do not read, copy, use or disclose to > > > others > > > > this message or any attachment. Please also notify the sender by > > replying > > > > to this email or by telephone (+44(020 7896 0011) and then delete the > > > email > > > > and any copies of it. Opinions, conclusion (etc) that do not relate > to > > > the > > > > official business of this company shall be understood as neither > given > > > nor > > > > endorsed by it. IG is a trading name of IG Markets Limited (a company > > > > registered in England and Wales, company number 04008957) and IG > Index > > > > Limited (a company registered in England and Wales, company number > > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate > Hill, > > > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and > > IG > > > > Index Limited (register number 114059) are authorised and regulated > by > > > the > > > > Financial Conduct Authority. > > > > > > > > > > > > > > > > -- > > > -Regards, > > > Mayuresh R. Gharat > > > (862) 250-7125 > > > > > > > > > -- > -Regards, > Mayuresh R. Gharat > (862) 250-7125 >
