Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.

I updated KIP-32 wiki with Anna's suggestion on ProduceResponse format change. Because it looks everyone also has a +1 on ProduceResponse format change, I take this update passed as well. KIP-32 passed with +6(binding). Thanks a lot for everyone's participation and all the comments. BTW, the

Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.

+1 from me Looking through this thread it seems there was some confusion on the migration discussion. This discussion in fact happened in the KIP-31 discuss thread, not so much in the KIP hangout. There is considerable overlap in discussions between KIP-3[1,2,3] so it makes sense to

Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.

Hi Becket and everyone, Could we please add the following functionality to this KIP. I think it would be very useful for the broker to return the timestamp in the ack to the producer (in response: timestamp per partition) and propagate it back to client in RecordMetadata. This way, if timestamp

Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.

Anna - Good suggestion. Sounds good to me as well On Fri, Jan 8, 2016 at 2:32 PM, Aditya Auradkar < aaurad...@linkedin.com.invalid> wrote: > Anna, > > That sounds good to me as well. > > Aditya > > On Fri, Jan 8, 2016 at 2:11 PM, Gwen Shapira wrote: > > > Sounds good to me

Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.

Anna, That sounds good to me as well. Aditya On Fri, Jan 8, 2016 at 2:11 PM, Gwen Shapira wrote: > Sounds good to me too. Seems pretty easy to add and can be useful for > producers. > > On Fri, Jan 8, 2016 at 1:22 PM, Joel Koshy wrote: > > > Hi Anna, >

Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.

Sounds good to me too. Seems pretty easy to add and can be useful for producers. On Fri, Jan 8, 2016 at 1:22 PM, Joel Koshy wrote: > Hi Anna, > > That sounds good to me - Becket/others any thoughts? > > Thanks, > > Joel > > On Fri, Jan 8, 2016 at 12:41 PM, Anna Povzner

Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.

Thanks a lot for the careful reading, Jun. Please see inline replies. > On Jan 6, 2016, at 3:24 AM, Jun Rao wrote: > > Jiangjie, > > Thanks for the updated KIP. Overall, a +1 on the proposal. A few minor > comments on the KIP. > > KIP-32: > 50. 6.c says "The log rolling

Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.

Hi, Jiangjie, 52. Replacing MessageSet with o.a.k.common.record will be ideal. Unfortunately, we use MessageSet in SimpleConsumer, which is part of the public api. Replacing MessageSet with o.a.k.common.record will be an incompatible api change. So, we probably should do this after we deprecate

Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.

Hi Adi, Please see inline comments. And thanks for correcting my typos :) > On Jan 5, 2016, at 3:37 AM, Aditya Auradkar > wrote: > > Hey Becket/Anna - > > I have a few comments about the KIP. > > 1. (Minor) Can we rename the KIP? It's currently "Add

Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.

+1, the KIP looks good to me now. Guozhang On Tue, Jan 5, 2016 at 1:47 AM, Becket Qin wrote: > Hi Adi, > > Please see inline comments. And thanks for correcting my typos :) > > > On Jan 5, 2016, at 3:37 AM, Aditya Auradkar > wrote: > > > >

Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.

+1 On Tue, Jan 5, 2016 at 11:12 AM, Guozhang Wang wrote: > +1, the KIP looks good to me now. > > Guozhang > > On Tue, Jan 5, 2016 at 1:47 AM, Becket Qin wrote: > > > Hi Adi, > > > > Please see inline comments. And thanks for correcting my typos :) > >

Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.

Jiangjie, Thanks for the updated KIP. Overall, a +1 on the proposal. A few minor comments on the KIP. KIP-32: 50. 6.c says "The log rolling has to depend on the earliest timestamp", which is inconsistent with KIP-33. 51. 8.b "If the time difference threshold is set to 0. The timestamp in the

Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.

I'm +1 on the design, but this proposal doesn't look like it is quite finished. We are trying to make the KIPs really be a record of design decisions we made and why. We should be able to give the document to advanced Kafka users and have them walk away with a good idea of what we are changing and

Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.

+1. Great work Becket. Aditya On Tue, Jan 5, 2016 at 11:47 AM, Neha Narkhede wrote: > +1 on the KIP. Thanks for the hard work, Becket! > > On Tue, Jan 5, 2016 at 11:24 AM, Jun Rao wrote: > > > Jiangjie, > > > > Thanks for the updated KIP. Overall, a +1 on

Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.

Hey Becket/Anna - I have a few comments about the KIP. 1. (Minor) Can we rename the KIP? It's currently "Add CreateTime and LogAppendTime etc..". This is actually the title of the now rejected Option 1. 2. (Minor) Can we rename the proposed option? It isn't really "option 4" anymore. 3. I'm not

Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.

Thanks Guozhang, Gwen and Neha for the comments. Sorry for late reply because I only have occasional gmail access from my phone... I just updated the wiki for KIP-32. Gwen, Yes, the migration plan is what you described. I agree with your comments on the version. I changed

Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.

Thanks Becket, Anne and Neha for responding to my concern. I had an offline discussion with Anne where she helped me understand the migration process. It isn't as bad as it looks in the KIP :) If I understand it correctly, the process (for users) will be: 1. Prepare for upgrade (set

Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.

Also I agree with Gwen that such changes may worth a 0.10 release or even 1.0, having it in 0.9.1 would be quite confusing to users. Guozhang On Wed, Dec 23, 2015 at 1:36 PM, Guozhang Wang wrote: > Becket, > > Please let us know once you have updated the wiki page regarding

Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.

Becket, Please let us know once you have updated the wiki page regarding the migration plan. Thanks! Guozhang On Wed, Dec 23, 2015 at 11:52 AM, Gwen Shapira wrote: > Thanks Becket, Anne and Neha for responding to my concern. > > I had an offline discussion with Anne where

Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.

One thing I want to do before +1 is to update the migration plan on the wiki regarding latest updates as well. Currently it states that in the second phase, the broker will "fill in the time field with current server time and re-compress the message" when it gets an older versioned produce request

[VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.

Hi, I am opening the voting thread for KIP-32: Add CreateTime and LogAppendTime to Kafka message. For reference, here's the KIP wiki: https://cwiki.apache.org/confluence/display/KAFKA/KIP-32+-+Add+CreateTime+and+LogAppendTime+to+Kafka+message And the mailing list threads: September:

Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.

Hi Anna, Have you talked with Jiangjie Qin before initiating the vote? Jiangjie Qin is on vocation now and he may not have time to answer question related to KIP-32 in detail. Dong On Tue, Dec 22, 2015 at 12:34 PM, Anna Povzner wrote: > Hi, > > I am opening the voting

Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.

Hi Anna, Thanks for initiating the voting process. I did not start the voting process because there were still some ongoing discussion with Jun about the timestamp regarding compressed messages. That is why the wiki page hasn't reflected the latest conversation as Guozhang pointed out.

Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.

Hey Anna, Thanks for initiating the vote and helping Becket while he is out! Hey Dong, My understanding from observing all the email threads is that there is a general agreement on the approach and the wiki is reasonably up-to-date with all the decisions. Is there a particular reason why the

Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.

Hi Anna, Thanks for the KIP, especially for the details on all the alternatives and how we arrived at the proposal. Its really great! Can you point me at where the migration plan was discussed? It looks overly complex and I have a bunch of questions, but if there was a discussion, I'd like to

Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.

Hey Gwen, Migration plan wasn't really discussed a ton in the previous threads. So it will be great to dive deep and see if there are gaps there. I had some questions, but the details listed on the KIP are great. It is complex, though the plan outlined in the wiki assumes a zero downtime upgrade

Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.

Hi Gwen, I just wanted to point out that I just started the vote. Becket wrote the proposal and led the discussions. What I understood from reading the discussion thread, the migration plan was discussed at the KIP meeting, and not much on the mailing list itself. My question about the

Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.

Hi Neha, Yeah, I actually think the KIP is ready for vote other than some minor issues. My question is more about who should start the vote and drive the design of this KIP. Given that Jiangjie is on vacation in China with limited access to gmail, I raised the question on his behalf and hopefully