Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

*Hi Jun, Soo sorry for the typo/mistake. On 02/12/2016, 11:19, "Michael Pearce" <michael.pea...@ig.com> wrote: Hi Jao Thanks for the response. Sorry for slow reply, both with personal sickness and also battling some critical issues encountered since upgra

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

would be useful to describe the down conversion path to consumers (i.e., brokers on message format 0.10.2 and consumers on older version). Thanks, Jun On Tue, Nov 29, 2016 at 3:18 AM, Michael Pearce <michael.pea...@ig.com> wrote: > Hi All, > > We

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

; > +1 (non-binding) > > > > Thanks, > > > > Mayuresh > > > > > > > On Nov 29, 2016, at 3:18 AM, Michael Pearce <michael.pea...@ig.com> > > wrote: > > > > > > Hi All, &g

Re: [DISCUSS] KIP-82 - Add Record Headers

e performing any actions on this key other than matching? Nacho On Fri, Nov 18, 2016 at 9:30 AM, Michael Pearce <michael.pea...@ig.com> wrote: > #jay #jun any concerns on 1 and 2 still? > > @all > To get this moving along a bit more I'd also like to ask

[VOTE] KIP-87 - Add Compaction Tombstone Flag

is for having a distinct compaction attribute “tombstone” flag instead of relying on null value, allowing non-null value delete messages. Many thanks, Michael On 22/11/2016, 15:52, "Michael Pearce" <michael.pea...@ig.com> wrote: Hi Mayuresh, LGTM. Ive just made one small adj

Re: [DISCUSS] 0.10.1.1 Plan

ed). Ismael On Thu, Nov 24, 2016 at 7:56 PM, Michael Pearce <michael.pea...@ig.com> wrote: > +1 it would be nice, and as is less restrive would not cause any issue. > > Saying that agree this is a fix build not a feature build. > > Sent using OWA for iPhone > ___

Re: [DISCUSS] 0.10.1.1 Plan

+1 it would be nice, and as is less restrive would not cause any issue. Saying that agree this is a fix build not a feature build. Sent using OWA for iPhone From: Rajini Sivaram Sent: Thursday, November 24, 2016 12:17:13 PM

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

t; Thanks, > > Mayuresh > > On Fri, Nov 18, 2016 at 8:58 AM, Michael Pearce <michael.pea...@ig.com> > wrote: > >> Many thanks for this Mayuresh. I don't have any objections. >> >> I assume we should state: >>

Re: [jira] [Created] (KAFKA-4424) Make serializer classes final

This is a change to api level / client code making it more restrictive as such would break code that has extended these classes. we should therefor have a KIP for this. At this time I would vote: -1 this will break any existing code by organisations for imo unproven performance improvement.

Re: [DISCUSS] KIP-82 - Add Record Headers

ion > > On Mon, Nov 14, 2016 at 9:16 AM, Michael Pearce <michael.pea...@ig.com> > wrote: > >> Hi Roger, >> >> The kip details/examples the original proposal for key spacing , not the >> new mentioned as per discussion namespace idea. >> >> W

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

Thanks, Mayuresh On Wed, Nov 16, 2016 at 11:07 PM, Michael Pearce <michael.pea...@ig.com> wrote: > Thanks guys, for discussing this offline and getting some consensus. > > So its clear for myself and others what is proposed now (i think i > understand, but want to make sure) &g

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

Thanks guys, for discussing this offline and getting some consensus. So its clear for myself and others what is proposed now (i think i understand, but want to make sure) Could i ask either directly update the kip to detail the migration strategy, or (re-)state your offline discussed and

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

seems better to be discussed in another KIP. Thanks, Jiangjie (Becket) Qin On Mon, Nov 14, 2016 at 8:52 AM, Michael Pearce <michael.pea...@ig.com> wrote: > I agree with Mayuresh. > > I don't see how having a magic byte helps here. > > What we are saying is that on day 1 afte

Re: [DISCUSS] KIP-92 - Add per partition lag metrics to KafkaConsumer

Should state I have no objections adding this client side, just more a question to why we don't look and propose to add this broker side also. Sent using OWA for iPhone From: Michael Pearce <michael.pea...@ig.com> Sent: Monday, November 14, 2016 4

Re: [DISCUSS] KIP-82 - Add Record Headers

s > > > > to either add (producer) > > > > or read (consumer) headers. > > > > This is an administrative task based on what features a client > > > > needs/provides and results in > > > > some sort of configuration to enable and config

Re: [DISCUSS] KIP-92 - Add per partition lag metrics to KafkaConsumer

Why do we not look to expose the lag broker side centrally? Eg like burrow. >From an operations point it's a lot easier to monitor lag centrally than per >application. Also then you'd be able to see lag of consumers not alive or >stalled. The information if the consumer uses Kafka based or

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

On Fri, Nov 11, 2016 at 8:42 AM, Mayuresh Gharat < > gharatmayures...@gmail.com > > wrote: > > > Sounds good Michael. > > > > Thanks, > > > > Mayuresh > > > > On Fri, Nov 11, 2016 at 12:48 AM, Michael Pearce <michael.pea...@ig.com> >

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

On Wed, Nov 9, 2016 at 10:49 AM, Becket Qin <becket@gmail.com> > > > wrote: > > > > > > > > > I am not sure if we need the second stage. Wouldn't it be enough to > > say > > > > > that a message is a tombstone if one of the following is true? > >

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

have completely miss-understood your 2 phased approach. Cheers Mike On 10/11/2016, 11:22, "Michael Pearce" <michael.pea...@ig.com> wrote: Agree on this point by Raidai, im happy having a two stage roll out a suggested by yourself Mayuresh, so for a period we have b

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

> > > > Thanks, > > > > Jiangjie (Becket) Qin > > > > > > On Wed, Nov 9, 2016 at 9:23 AM, Mayuresh Gharat < > > gharatmayures...@gmail.com> > > wrote: > > > > > I think it will be a g

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

Tue, Nov 8, 2016 at 12:06 AM, Michael Pearce <michael.pea...@ig.com> wrote: > Also we can add further guidance: > > To avoid the below caveat to organisations by promoting of upgrading all > consumers first before relying on producing tombstone messages wi

Re: [DISCUSS] KIP-82 - Add Record Headers

t;>>> detail. >> >> >>>>> >> >> >>>>> This is a single threaded benchmarks so all the measurements are >> per >> >> >>>>> thread. >> >> >>>>> >> >> >>>>> For 1M messages/s/threa

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

Also we can add further guidance: To avoid the below caveat to organisations by promoting of upgrading all consumers first before relying on producing tombstone messages with data Sent using OWA for iPhone From: Michael Pearce Sent: Tuesday, November 8

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

own convert that message to remove the value, right? In this case, we > cannot wait for the log cleaner to do the down conversion because that > message may have already been consumed before the log compaction happens. > > Thanks, > > Jiangjie (Becket) Qin > > > > On Mon,

Re: [DISCUSS] KIP-82 - Add Record Headers

g, per header) to be meaningful * doesn't really say anything how to parse the tag's data, so it is in effect useless on its own. Regards, Magnus 2016-11-07 18:32 GMT+01:00 Michael Pearce <michael.pea...@ig.com>: > Hi Roger, > > Thanks for the support. > > I think the ke

Re: [DISCUSS] KIP-82 - Add Record Headers

For me 5c and 5a are almost identical. The idea in the kip(5a) is that the core message just has a header length and then the header bytes, which are then in a pre agreed sub wire protocol as described. 5c instead of having a pre agreed wire format allows custom serialisation of a map of

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

remove the value to down convert the message if the consumer version is old, right? Thanks. Jiangjie (Becket) Qin On Wed, Nov 2, 2016 at 1:37 AM, Michael Pearce <michael.pea...@ig.com> wrote: > Hi Joel , et al. > > Any comments on the b

Re: [DISCUSS] KIP-82 - Add Record Headers

Hi Roger, Thanks for the support. I think the key thing is to have a common key space to make an ecosystem, there does have to be some level of contract for people to play nicely. Having map or as per current proposed in kip of having a numerical key space of map

Re: [DISCUSS] KIP-82 - Add Record Headers

the container format, and let you use complex types in your headers. However, this would mean people would still have to use avro for deserializing a Kafka message body. Our experience using this at TiVo: * We haven't run into any problems so far. * We are not yet running Kafka in production, so

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

Hi Joel , et al. Any comments on the below idea to handle roll out / compatibility of this feature, using a configuration? Does it make sense/clear? Does it add value? Do we want to enforce flag by default, or value by default, or both? Cheers Mike On 10/27/16, 4:47 PM, "Michael P

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

one, and what would you use it for? -James Sent from my iPhone > On Oct 27, 2016, at 9:17 PM, Michael Pearce <michael.pea...@ig.com> wrote: > > Hi Jay, > > I think use case that is the issue that Konstantin mentioned in the kip-82

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

*compatability From: Michael Pearce <michael.pea...@ig.com> Sent: Friday, October 28, 2016 5:17:07 AM To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag Hi Jay, I think use case that is the issue that Kons

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

y make sense if your topic contains only UPDATE and/or DELETE messages. "Normal" topics are pure inserts. If we are making this change how does it effect streams? What happens if I send a DELETE message to a non-compacted topic where deletes are impossible? -Jay On Tue, Oct 25, 2016 at

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

t sends FetchRequest >>> translated from attributes.bit5 to value=null as required. >>>> >>>> >>>> That's my understanding anyway, please correct me if I'm wrong. >>>> >>>> /Magnus >>>> &g

Re: [DISCUSS] KIP-82 - Add Record Headers

AM, Jun Rao <j...@confluent.io> wrote: > > > Michael, > > > > Yes, doing a separate KIP to address the null payload issue for compacted > > topics is a good idea. > > > > Thanks, > > > > Jun > > > > On Fri, Oct 21, 2016 at 12:57 AM,

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

and needs to be fixed. > > So I dont see a need for a MagicByte bump, both broker and client has the > information it needs to construct or parse the message according to request > version. > > > 2016-10-25 18:48 GMT+02:00 Michael Pearce <michael.pea...@ig.com>: > >

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

Hi Michael, With the version bumps for Produce and Fetch requests, do you really need to bump MagicByte too? Regards, Magnus 2016-10-25 18:09 GMT+02:00 Michael Pearce <michael.pea...@ig.com>: > Hi All, > > I would like to discuss the foll

[DISCUSS] KIP-87 - Add Compaction Tombstone Flag

Hi All, I would like to discuss the following KIP proposal: https://cwiki.apache.org/confluence/display/KAFKA/KIP-87+-+Add+Compaction+Tombstone+Flag This is off the back of the discussion on KIP-82 / KIP meeting where it was agreed to separate this issue and feature. See:

Re: [DISCUSS] KIP-80: Kafka REST Server

com on behalf of ism...@juma.me.uk> wrote: Hi Michael, Can you please share which Apache projects have a MMC? I couldn't find anything after a quick google. Ismael On Fri, Oct 21, 2016 at 7:28 AM, Michael Pearce <michael.pea...@ig.com> wrote: > So fr

Re: Kafka KIP meeting Oct 19 at 11:00am PST

I had noted that what ever the solution having compaction based on null payload was agreed isn't elegant. Shall we raise another kip to : as discussed propose using a attribute bit for delete/compaction flag as well/or instead of null value and updating compaction logic to look at that

Re: [DISCUSS] KIP-80: Kafka REST Server

gt; > the roadmap and welcoming us to contribute to the project. It doesn't > > gurantee what we want to add in the furture will be in your roadmap. > > > > Hence the reason having it part of Kafka community will help a lot as > other > > users can participate in the

Re: [DISCUSS] KIP-82 - Add Record Headers

ne KIP discussion meetings from time to time. How about we discuss this KIP Wed (Oct 19) at 11:00am PST? I will send out an invite (we typically do the meeting through Zoom and will post the video recording to Kafka wiki). Thanks, Jun On Wed, Oct 12, 2016 at 1:22 AM, Michael Pearce <michael.pea.

Re: [DISCUSS] KIP-82 - Add Record Headers

IMHO, so more than happy to have someone suggest a better alternative. Best, Mike On 10/8/16, 7:26 AM, "Michael Pearce" <michael.pea...@ig.com> wrote: >> I agree with the critique of compaction not having a value. I think we should consider fixing that dire

Re: [DISCUSS] KIP-80: Kafka REST Server

+1 I agree on the governance comments whole heartedly. Also i agree about the contribution comments made earlier in the thread, i personally am less likely to spend any of mine, or give project time within my internal projects to developers contributing to another commercial companies project

Re: [DISCUSS] KIP-82 - Add Record Headers

any alternatives on compaction delete semantics that could address this? The KIP wiki discussion I think mostly assumes that compaction-delete is what it is and can't be changed/fixed. -Dana On Fri, Oct 7, 2016 at 1:38 PM, Michael Pearce <michael.pea...@ig.com> wrote: > > Hi Jay, > >

Re: [DISCUSS] KIP-82 - Add Record Headers

whole point was to be able to share data, which doesn't work if every team chooses their own format. I agree with the critique of compaction not having a value. I think we should consider fixing that directly. -Jay On Thu, Sep 22, 2016 at 12:31 PM, Michael Pearce <michael.pea...@ig.com> wrot

Re: [DISCUSS] KIP-82 - Add Record Headers

Hi All, We have updated a sample version of an implementation of the KIP with Integer,Byte[] headers to aid discussion. See: https://github.com/michaelandrepearce/kafka/tree/headers On the kip there are a few outstanding bits that I would like to have discussed and get some majority consensus

[DISCUSS] KIP-82 - Add Record Headers

Hi All, I would like to discuss the following KIP proposal: https://cwiki.apache.org/confluence/display/KAFKA/KIP-82+-+Add+Record+Headers I have some initial ?drafts of roughly the changes that would be needed. This is no where finalized and look forward to the discussion especially as some

Re: ProducerRecord/Consumer MetaData/Headers

behalf of Ismael Juma <ism...@juma.me.uk> Sent: Thursday, September 22, 2016 9:42 AM To: dev@kafka.apache.org Subject: Re: ProducerRecord/Consumer MetaData/Headers Sorry for the delay, you should have access now. Ismael On Thu, Sep 22, 2016 at 8:15 AM, Michael Pearce <michael.pea...@ig.com>

Re: ProducerRecord/Consumer MetaData/Headers

Hi again, Sorry to be nudging this, but it seems I'm still unable to create a page in the KIP Proposal area. Just don't want to be forgotten about between all the emails. Cheers Mike From: Michael Pearce <michael.pea...@ig.com> Sent: Monday, Septem

Re: ProducerRecord/Consumer MetaData/Headers

Hi Again, I went to the wiki this afternoon to start writing it up, and seems I cannot create a page under the KIP area still. If someone could assist. Cheers Mike On 9/18/16, 7:07 PM, "Michael Pearce" <michael.pea...@ig.com> wrote: Hi Ismaelj Than

Re: ProducerRecord/Consumer MetaData/Headers

via schemas, for example. Gwen shared some thoughts in the following message: http://search-hadoop.com/m/uyzND1OXS8EoGCU2 Ismael On Sun, Sep 18, 2016 at 7:11 AM, Michael Pearce <michael.pea...@ig.com> wrote: > Hi All, (again) > > If it helps the discussion, and almost ready patch imple

Re: ProducerRecord/Consumer MetaData/Headers

a JIRA as fairly substantial api change but unsure whom can raise these so assistance in the process would be gratefully accepted.. Cheers Mike From: Michael Pearce <michael.pea...@ig.com> Date: Saturday, September 17, 2016 at 6:40 AM To: "dev@kafka.apache.org" <d

ProducerRecord/Consumer MetaData/Headers

Hi All, First of all apologies if this has been previously discussed I have just joined the mail list (I cannot find a JIRA or KIP related nor via good old google search) In our company we are looking to replace some of our more traditional message flows with Kafka. One thing we have found

<    1   2