Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.
Hi folks, During the implementation of the KIP, Guozhang raised another proposal on how to indicate the message timestamp type used by messages. The difference between current and the new proposal only differs on messages that are a) compressed, and b) using LogAppendTime For compressed messages using LogAppendTime, the timestamps in the current proposal is as below: 1. When a producer produces the messages, it tries to set timestamp to -1 for inner messages if it knows LogAppendTime is used. 2. When a broker receives the messages, it will overwrite the timestamp of inner message to -1 if needed and write server time to the wrapper message. Broker will do re-compression if inner message timestamp is overwritten. 3. When a consumer receives the messages, it will see the inner message timestamp is -1 so the wrapper message timestamp is used. Implementation wise, this proposal requires the producer to set timestamp for inner messages correctly to avoid broker side re-compression. To do that, the short term solution is to let producer infer the timestamp type returned by broker in ProduceResponse and set correct timestamp afterwards. This means the first few batches will still need re-compression on the broker. The long term solution is to have producer get topic configuration during metadata update. The proposed modification is: 1. When a producer produces the messages, it always using create time. 2. When a broker receives the messages, it ignores the inner messages timestamp, but simply set a wrapper message timestamp type attribute bit to 1 and set the timestamp of the wrapper message to server time. (The broker will also set the timesatmp type attribute bit accordingly for non-compressed messages using LogAppendTime). 3. When a consumer receives the messages, it checks timestamp type attribute bit of wrapper message. If it is set to 1, the inner message's timestamp will be ignored and the wrapper message's timestamp will be used. This approach uses an extra attribute bit. The good thing of the modified protocol is consumers will be able to know the timestamp type. And re-compression on broker side is completely avoided no matter what value is sent by the producer. In this approach the inner messages will have wrong timestamps. The changed proposal has gone through some discussion during the KIP hangout as well as the discussion thread. Based on the current consensus, I have updated the KIP wiki with changed proposal and will implement it Please let us know if you have any concern over the changed proposal. Thanks, Jiangjie (Becket) Qin On Thu, Jan 21, 2016 at 10:08 AM, Jiangjie Qin wrote: > I updated KIP-32 wiki with Anna's suggestion on ProduceResponse format > change. > > Because it looks everyone also has a +1 on ProduceResponse format change, I > take this update passed as well. > > KIP-32 passed with +6(binding). > > Thanks a lot for everyone's participation and all the comments. > > BTW, the patch for KIP-31 and KIP-32 has been submitted for review in PR > #764. > > Thanks, > > Jiangjie (Becket) Qin > > On Mon, Jan 11, 2016 at 10:48 PM, Becket Qin wrote: > > > That makes sense to me, too. I will update the KIP to reflect this. > > Thanks, Anna. > > > > > On Jan 9, 2016, at 9:37 AM, Neha Narkhede wrote: > > > > > > Anna - Good suggestion. Sounds good to me as well > > > > > > On Fri, Jan 8, 2016 at 2:32 PM, Aditya Auradkar < > > > aaurad...@linkedin.com.invalid> wrote: > > > > > >> Anna, > > >> > > >> That sounds good to me as well. > > >> > > >> Aditya > > >> > > >>> On Fri, Jan 8, 2016 at 2:11 PM, Gwen Shapira > > wrote: > > >>> > > >>> Sounds good to me too. Seems pretty easy to add and can be useful for > > >>> producers. > > >>> > > On Fri, Jan 8, 2016 at 1:22 PM, Joel Koshy > > wrote: > > > > Hi Anna, > > > > That sounds good to me - Becket/others any thoughts? > > > > Thanks, > > > > Joel > > > > On Fri, Jan 8, 2016 at 12:41 PM, Anna Povzner > > >> wrote: > > > > > Hi Becket and everyone, > > > > > > Could we please add the following functionality to this KIP. I > think > > >> it > > > would be very useful for the broker to return the timestamp in the > > >> ack > > >>> to > > > the producer (in response: timestamp per partition) and propagate > it > > >>> back > > > to client in RecordMetadata. This way, if timestamp type is > > LogAppendTime, > > > the producer client will see what timestamp was actually set -- and > > >> it > > > would match the timestamp that consumer sees. Also, returning the > > timestamp > > > in RecordMetadata is also useful for timestamp type = CreateTime, > > >> since > > > timestamp could be also set in KafkaProducer (if client set > timestamp > > >>> in > > > ProducerRecord to 0). > > > > > > Since this requires protocol change as well, it will be better to > > implement > > > this as part of KIP-32, rather than propo
Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.
I updated KIP-32 wiki with Anna's suggestion on ProduceResponse format change. Because it looks everyone also has a +1 on ProduceResponse format change, I take this update passed as well. KIP-32 passed with +6(binding). Thanks a lot for everyone's participation and all the comments. BTW, the patch for KIP-31 and KIP-32 has been submitted for review in PR #764. Thanks, Jiangjie (Becket) Qin On Mon, Jan 11, 2016 at 10:48 PM, Becket Qin wrote: > That makes sense to me, too. I will update the KIP to reflect this. > Thanks, Anna. > > > On Jan 9, 2016, at 9:37 AM, Neha Narkhede wrote: > > > > Anna - Good suggestion. Sounds good to me as well > > > > On Fri, Jan 8, 2016 at 2:32 PM, Aditya Auradkar < > > aaurad...@linkedin.com.invalid> wrote: > > > >> Anna, > >> > >> That sounds good to me as well. > >> > >> Aditya > >> > >>> On Fri, Jan 8, 2016 at 2:11 PM, Gwen Shapira > wrote: > >>> > >>> Sounds good to me too. Seems pretty easy to add and can be useful for > >>> producers. > >>> > On Fri, Jan 8, 2016 at 1:22 PM, Joel Koshy > wrote: > > Hi Anna, > > That sounds good to me - Becket/others any thoughts? > > Thanks, > > Joel > > On Fri, Jan 8, 2016 at 12:41 PM, Anna Povzner > >> wrote: > > > Hi Becket and everyone, > > > > Could we please add the following functionality to this KIP. I think > >> it > > would be very useful for the broker to return the timestamp in the > >> ack > >>> to > > the producer (in response: timestamp per partition) and propagate it > >>> back > > to client in RecordMetadata. This way, if timestamp type is > LogAppendTime, > > the producer client will see what timestamp was actually set -- and > >> it > > would match the timestamp that consumer sees. Also, returning the > timestamp > > in RecordMetadata is also useful for timestamp type = CreateTime, > >> since > > timestamp could be also set in KafkaProducer (if client set timestamp > >>> in > > ProducerRecord to 0). > > > > Since this requires protocol change as well, it will be better to > implement > > this as part of KIP-32, rather than proposing a new KIP. > > > > Thanks, > > Anna > > > > > > On Fri, Jan 8, 2016 at 12:53 PM, Joel Koshy > >>> wrote: > > > >> +1 from me > >> > >> Looking through this thread it seems there was some confusion on > >> the > >> migration discussion. This discussion in fact happened in the > >> KIP-31 > >> discuss thread, not so much in the KIP hangout. There is > >> considerable > >> overlap in discussions between KIP-3[1,2,3] so it makes sense to > >> cross-reference all of these. > >> > >> I'm finding the Apache list archive a little cumbersome to use > >> (e.g., > the > >> current link in KIP-31 points to the beginning of September > >> archives) > but > >> the emails discussing migration were in October: > >>> http://mail-archives.apache.org/mod_mbox/kafka-dev/201510.mbox/thread > >> > >> Markmail has a better interface but interestingly it has not > >> indexed > any > > of > >> the emails from August, September and early October ( > >> > http://markmail.org/search/?q=list%3Aorg.apache.incubator.kafka-dev+date%3A201509-201511+order%3Adate-backward > >> ). > >> Perhaps KIPs should include a permalink to the first message of the > > DISCUSS > >> thread. E.g., > >> > http://mail-archives.apache.org/mod_mbox/kafka-dev/201509.mbox/%3CCAHrRUm5jvL_dPeZWnfBD-vONgSZWOq1VL1Ss8OSUOCPXmtg8rQ%40mail.gmail.com%3E > >> > >> Also, just to clarify Jay's comments on the content of KIPs: I > >> think > > having > >> a pseudo-code spec/implementation guide is useful (especially for > >> client-side KIPs). While the motivation should definitely capture > >>> “why > we > >> are doing the KIP” it probably shouldn’t have to exhaustively > >> capture > > “why > >> we are doing the KIP *this way*”. i.e., some of the discussions are > >> extremely nuanced and in this case spans multiple KIPs so links to > other > >> KIPs and the discuss threads and KIP hangout recordings are perhaps > >> sufficient to fill this gap - or maybe a new section that > >> summarizes > the > >> discussions. > >> > >> Thanks, > >> > >> Joel > >> > >>> On Wed, Jan 6, 2016 at 9:29 AM, Jun Rao wrote: > >>> > >>> Hi, Jiangjie, > >>> > >>> 52. Replacing MessageSet with o.a.k.common.record will be ideal. > >>> Unfortunately, we use MessageSet in SimpleConsumer, which is part > >>> of > > the > >>> public api. Replacing MessageSet with o.a.k.common.record will be > >>> an > >>> incompatible api change. So, we probably should do this after we > >> deprecate > >>> SimpleConsumer. > >>> > >>> My original question is actually whether we just bump up magic > >> byte >
Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.
That makes sense to me, too. I will update the KIP to reflect this. Thanks, Anna. > On Jan 9, 2016, at 9:37 AM, Neha Narkhede wrote: > > Anna - Good suggestion. Sounds good to me as well > > On Fri, Jan 8, 2016 at 2:32 PM, Aditya Auradkar < > aaurad...@linkedin.com.invalid> wrote: > >> Anna, >> >> That sounds good to me as well. >> >> Aditya >> >>> On Fri, Jan 8, 2016 at 2:11 PM, Gwen Shapira wrote: >>> >>> Sounds good to me too. Seems pretty easy to add and can be useful for >>> producers. >>> On Fri, Jan 8, 2016 at 1:22 PM, Joel Koshy wrote: Hi Anna, That sounds good to me - Becket/others any thoughts? Thanks, Joel On Fri, Jan 8, 2016 at 12:41 PM, Anna Povzner >> wrote: > Hi Becket and everyone, > > Could we please add the following functionality to this KIP. I think >> it > would be very useful for the broker to return the timestamp in the >> ack >>> to > the producer (in response: timestamp per partition) and propagate it >>> back > to client in RecordMetadata. This way, if timestamp type is LogAppendTime, > the producer client will see what timestamp was actually set -- and >> it > would match the timestamp that consumer sees. Also, returning the timestamp > in RecordMetadata is also useful for timestamp type = CreateTime, >> since > timestamp could be also set in KafkaProducer (if client set timestamp >>> in > ProducerRecord to 0). > > Since this requires protocol change as well, it will be better to implement > this as part of KIP-32, rather than proposing a new KIP. > > Thanks, > Anna > > > On Fri, Jan 8, 2016 at 12:53 PM, Joel Koshy >>> wrote: > >> +1 from me >> >> Looking through this thread it seems there was some confusion on >> the >> migration discussion. This discussion in fact happened in the >> KIP-31 >> discuss thread, not so much in the KIP hangout. There is >> considerable >> overlap in discussions between KIP-3[1,2,3] so it makes sense to >> cross-reference all of these. >> >> I'm finding the Apache list archive a little cumbersome to use >> (e.g., the >> current link in KIP-31 points to the beginning of September >> archives) but >> the emails discussing migration were in October: >>> http://mail-archives.apache.org/mod_mbox/kafka-dev/201510.mbox/thread >> >> Markmail has a better interface but interestingly it has not >> indexed any > of >> the emails from August, September and early October ( >> http://markmail.org/search/?q=list%3Aorg.apache.incubator.kafka-dev+date%3A201509-201511+order%3Adate-backward >> ). >> Perhaps KIPs should include a permalink to the first message of the > DISCUSS >> thread. E.g., >> http://mail-archives.apache.org/mod_mbox/kafka-dev/201509.mbox/%3CCAHrRUm5jvL_dPeZWnfBD-vONgSZWOq1VL1Ss8OSUOCPXmtg8rQ%40mail.gmail.com%3E >> >> Also, just to clarify Jay's comments on the content of KIPs: I >> think > having >> a pseudo-code spec/implementation guide is useful (especially for >> client-side KIPs). While the motivation should definitely capture >>> “why we >> are doing the KIP” it probably shouldn’t have to exhaustively >> capture > “why >> we are doing the KIP *this way*”. i.e., some of the discussions are >> extremely nuanced and in this case spans multiple KIPs so links to other >> KIPs and the discuss threads and KIP hangout recordings are perhaps >> sufficient to fill this gap - or maybe a new section that >> summarizes the >> discussions. >> >> Thanks, >> >> Joel >> >>> On Wed, Jan 6, 2016 at 9:29 AM, Jun Rao wrote: >>> >>> Hi, Jiangjie, >>> >>> 52. Replacing MessageSet with o.a.k.common.record will be ideal. >>> Unfortunately, we use MessageSet in SimpleConsumer, which is part >>> of > the >>> public api. Replacing MessageSet with o.a.k.common.record will be >>> an >>> incompatible api change. So, we probably should do this after we >> deprecate >>> SimpleConsumer. >>> >>> My original question is actually whether we just bump up magic >> byte in >>> Message once to incorporate both the offset and the timestamp >>> change. > It >>> seems that the answer is yes. Could you reflect that in the KIP? >>> >>> Thanks, >>> >>> Jun >>> >>> >>> On Wed, Jan 6, 2016 at 7:01 AM, Becket Qin >> > wrote: >>> Thanks a lot for the careful reading, Jun. Please see inline replies. > On Jan 6, 2016, at 3:24 AM, Jun Rao >> wrote: > > Jiangjie, > > Thanks for the updated KIP. Overall, a +1 on the proposal. A >>> few >> minor > comments on the KIP. > > KIP-32: > 50. 6.c says "The log rolling
Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.
Anna - Good suggestion. Sounds good to me as well On Fri, Jan 8, 2016 at 2:32 PM, Aditya Auradkar < aaurad...@linkedin.com.invalid> wrote: > Anna, > > That sounds good to me as well. > > Aditya > > On Fri, Jan 8, 2016 at 2:11 PM, Gwen Shapira wrote: > > > Sounds good to me too. Seems pretty easy to add and can be useful for > > producers. > > > > On Fri, Jan 8, 2016 at 1:22 PM, Joel Koshy wrote: > > > > > Hi Anna, > > > > > > That sounds good to me - Becket/others any thoughts? > > > > > > Thanks, > > > > > > Joel > > > > > > On Fri, Jan 8, 2016 at 12:41 PM, Anna Povzner > wrote: > > > > > > > Hi Becket and everyone, > > > > > > > > Could we please add the following functionality to this KIP. I think > it > > > > would be very useful for the broker to return the timestamp in the > ack > > to > > > > the producer (in response: timestamp per partition) and propagate it > > back > > > > to client in RecordMetadata. This way, if timestamp type is > > > LogAppendTime, > > > > the producer client will see what timestamp was actually set -- and > it > > > > would match the timestamp that consumer sees. Also, returning the > > > timestamp > > > > in RecordMetadata is also useful for timestamp type = CreateTime, > since > > > > timestamp could be also set in KafkaProducer (if client set timestamp > > in > > > > ProducerRecord to 0). > > > > > > > > Since this requires protocol change as well, it will be better to > > > implement > > > > this as part of KIP-32, rather than proposing a new KIP. > > > > > > > > Thanks, > > > > Anna > > > > > > > > > > > > On Fri, Jan 8, 2016 at 12:53 PM, Joel Koshy > > wrote: > > > > > > > > > +1 from me > > > > > > > > > > Looking through this thread it seems there was some confusion on > the > > > > > migration discussion. This discussion in fact happened in the > KIP-31 > > > > > discuss thread, not so much in the KIP hangout. There is > considerable > > > > > overlap in discussions between KIP-3[1,2,3] so it makes sense to > > > > > cross-reference all of these. > > > > > > > > > > I'm finding the Apache list archive a little cumbersome to use > (e.g., > > > the > > > > > current link in KIP-31 points to the beginning of September > archives) > > > but > > > > > the emails discussing migration were in October: > > > > > > > http://mail-archives.apache.org/mod_mbox/kafka-dev/201510.mbox/thread > > > > > > > > > > Markmail has a better interface but interestingly it has not > indexed > > > any > > > > of > > > > > the emails from August, September and early October ( > > > > > > > > > > > > > > > > > > > > http://markmail.org/search/?q=list%3Aorg.apache.incubator.kafka-dev+date%3A201509-201511+order%3Adate-backward > > > > > ). > > > > > Perhaps KIPs should include a permalink to the first message of the > > > > DISCUSS > > > > > thread. E.g., > > > > > > > > > > > > > > > > > > > > http://mail-archives.apache.org/mod_mbox/kafka-dev/201509.mbox/%3CCAHrRUm5jvL_dPeZWnfBD-vONgSZWOq1VL1Ss8OSUOCPXmtg8rQ%40mail.gmail.com%3E > > > > > > > > > > Also, just to clarify Jay's comments on the content of KIPs: I > think > > > > having > > > > > a pseudo-code spec/implementation guide is useful (especially for > > > > > client-side KIPs). While the motivation should definitely capture > > “why > > > we > > > > > are doing the KIP” it probably shouldn’t have to exhaustively > capture > > > > “why > > > > > we are doing the KIP *this way*”. i.e., some of the discussions are > > > > > extremely nuanced and in this case spans multiple KIPs so links to > > > other > > > > > KIPs and the discuss threads and KIP hangout recordings are perhaps > > > > > sufficient to fill this gap - or maybe a new section that > summarizes > > > the > > > > > discussions. > > > > > > > > > > Thanks, > > > > > > > > > > Joel > > > > > > > > > > On Wed, Jan 6, 2016 at 9:29 AM, Jun Rao wrote: > > > > > > > > > > > Hi, Jiangjie, > > > > > > > > > > > > 52. Replacing MessageSet with o.a.k.common.record will be ideal. > > > > > > Unfortunately, we use MessageSet in SimpleConsumer, which is part > > of > > > > the > > > > > > public api. Replacing MessageSet with o.a.k.common.record will be > > an > > > > > > incompatible api change. So, we probably should do this after we > > > > > deprecate > > > > > > SimpleConsumer. > > > > > > > > > > > > My original question is actually whether we just bump up magic > byte > > > in > > > > > > Message once to incorporate both the offset and the timestamp > > change. > > > > It > > > > > > seems that the answer is yes. Could you reflect that in the KIP? > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > > > On Wed, Jan 6, 2016 at 7:01 AM, Becket Qin > > > > > wrote: > > > > > > > > > > > > > Thanks a lot for the careful reading, Jun. > > > > > > > Please see inline replies. > > > > > > > > > > > > > > > > > > > > > > On Jan 6, 2016, at 3:24 AM, Jun Rao > wrote: > > > > > > > > > > > > > > > > Jiangjie, > > > > > > > > > > > > > > > > Thanks
Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.
Anna, That sounds good to me as well. Aditya On Fri, Jan 8, 2016 at 2:11 PM, Gwen Shapira wrote: > Sounds good to me too. Seems pretty easy to add and can be useful for > producers. > > On Fri, Jan 8, 2016 at 1:22 PM, Joel Koshy wrote: > > > Hi Anna, > > > > That sounds good to me - Becket/others any thoughts? > > > > Thanks, > > > > Joel > > > > On Fri, Jan 8, 2016 at 12:41 PM, Anna Povzner wrote: > > > > > Hi Becket and everyone, > > > > > > Could we please add the following functionality to this KIP. I think it > > > would be very useful for the broker to return the timestamp in the ack > to > > > the producer (in response: timestamp per partition) and propagate it > back > > > to client in RecordMetadata. This way, if timestamp type is > > LogAppendTime, > > > the producer client will see what timestamp was actually set -- and it > > > would match the timestamp that consumer sees. Also, returning the > > timestamp > > > in RecordMetadata is also useful for timestamp type = CreateTime, since > > > timestamp could be also set in KafkaProducer (if client set timestamp > in > > > ProducerRecord to 0). > > > > > > Since this requires protocol change as well, it will be better to > > implement > > > this as part of KIP-32, rather than proposing a new KIP. > > > > > > Thanks, > > > Anna > > > > > > > > > On Fri, Jan 8, 2016 at 12:53 PM, Joel Koshy > wrote: > > > > > > > +1 from me > > > > > > > > Looking through this thread it seems there was some confusion on the > > > > migration discussion. This discussion in fact happened in the KIP-31 > > > > discuss thread, not so much in the KIP hangout. There is considerable > > > > overlap in discussions between KIP-3[1,2,3] so it makes sense to > > > > cross-reference all of these. > > > > > > > > I'm finding the Apache list archive a little cumbersome to use (e.g., > > the > > > > current link in KIP-31 points to the beginning of September archives) > > but > > > > the emails discussing migration were in October: > > > > > http://mail-archives.apache.org/mod_mbox/kafka-dev/201510.mbox/thread > > > > > > > > Markmail has a better interface but interestingly it has not indexed > > any > > > of > > > > the emails from August, September and early October ( > > > > > > > > > > > > > > http://markmail.org/search/?q=list%3Aorg.apache.incubator.kafka-dev+date%3A201509-201511+order%3Adate-backward > > > > ). > > > > Perhaps KIPs should include a permalink to the first message of the > > > DISCUSS > > > > thread. E.g., > > > > > > > > > > > > > > http://mail-archives.apache.org/mod_mbox/kafka-dev/201509.mbox/%3CCAHrRUm5jvL_dPeZWnfBD-vONgSZWOq1VL1Ss8OSUOCPXmtg8rQ%40mail.gmail.com%3E > > > > > > > > Also, just to clarify Jay's comments on the content of KIPs: I think > > > having > > > > a pseudo-code spec/implementation guide is useful (especially for > > > > client-side KIPs). While the motivation should definitely capture > “why > > we > > > > are doing the KIP” it probably shouldn’t have to exhaustively capture > > > “why > > > > we are doing the KIP *this way*”. i.e., some of the discussions are > > > > extremely nuanced and in this case spans multiple KIPs so links to > > other > > > > KIPs and the discuss threads and KIP hangout recordings are perhaps > > > > sufficient to fill this gap - or maybe a new section that summarizes > > the > > > > discussions. > > > > > > > > Thanks, > > > > > > > > Joel > > > > > > > > On Wed, Jan 6, 2016 at 9:29 AM, Jun Rao wrote: > > > > > > > > > Hi, Jiangjie, > > > > > > > > > > 52. Replacing MessageSet with o.a.k.common.record will be ideal. > > > > > Unfortunately, we use MessageSet in SimpleConsumer, which is part > of > > > the > > > > > public api. Replacing MessageSet with o.a.k.common.record will be > an > > > > > incompatible api change. So, we probably should do this after we > > > > deprecate > > > > > SimpleConsumer. > > > > > > > > > > My original question is actually whether we just bump up magic byte > > in > > > > > Message once to incorporate both the offset and the timestamp > change. > > > It > > > > > seems that the answer is yes. Could you reflect that in the KIP? > > > > > > > > > > Thanks, > > > > > > > > > > Jun > > > > > > > > > > > > > > > On Wed, Jan 6, 2016 at 7:01 AM, Becket Qin > > > wrote: > > > > > > > > > > > Thanks a lot for the careful reading, Jun. > > > > > > Please see inline replies. > > > > > > > > > > > > > > > > > > > On Jan 6, 2016, at 3:24 AM, Jun Rao wrote: > > > > > > > > > > > > > > Jiangjie, > > > > > > > > > > > > > > Thanks for the updated KIP. Overall, a +1 on the proposal. A > few > > > > minor > > > > > > > comments on the KIP. > > > > > > > > > > > > > > KIP-32: > > > > > > > 50. 6.c says "The log rolling has to depend on the earliest > > > > timestamp", > > > > > > > which is inconsistent with KIP-33. > > > > > > Corrected. > > > > > > > > > > > > > > 51. 8.b "If the time difference threshold is set to 0. The > > > timestamp > > > > in > > > > > > the > > > > > > > message
Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.
Sounds good to me too. Seems pretty easy to add and can be useful for producers. On Fri, Jan 8, 2016 at 1:22 PM, Joel Koshy wrote: > Hi Anna, > > That sounds good to me - Becket/others any thoughts? > > Thanks, > > Joel > > On Fri, Jan 8, 2016 at 12:41 PM, Anna Povzner wrote: > > > Hi Becket and everyone, > > > > Could we please add the following functionality to this KIP. I think it > > would be very useful for the broker to return the timestamp in the ack to > > the producer (in response: timestamp per partition) and propagate it back > > to client in RecordMetadata. This way, if timestamp type is > LogAppendTime, > > the producer client will see what timestamp was actually set -- and it > > would match the timestamp that consumer sees. Also, returning the > timestamp > > in RecordMetadata is also useful for timestamp type = CreateTime, since > > timestamp could be also set in KafkaProducer (if client set timestamp in > > ProducerRecord to 0). > > > > Since this requires protocol change as well, it will be better to > implement > > this as part of KIP-32, rather than proposing a new KIP. > > > > Thanks, > > Anna > > > > > > On Fri, Jan 8, 2016 at 12:53 PM, Joel Koshy wrote: > > > > > +1 from me > > > > > > Looking through this thread it seems there was some confusion on the > > > migration discussion. This discussion in fact happened in the KIP-31 > > > discuss thread, not so much in the KIP hangout. There is considerable > > > overlap in discussions between KIP-3[1,2,3] so it makes sense to > > > cross-reference all of these. > > > > > > I'm finding the Apache list archive a little cumbersome to use (e.g., > the > > > current link in KIP-31 points to the beginning of September archives) > but > > > the emails discussing migration were in October: > > > http://mail-archives.apache.org/mod_mbox/kafka-dev/201510.mbox/thread > > > > > > Markmail has a better interface but interestingly it has not indexed > any > > of > > > the emails from August, September and early October ( > > > > > > > > > http://markmail.org/search/?q=list%3Aorg.apache.incubator.kafka-dev+date%3A201509-201511+order%3Adate-backward > > > ). > > > Perhaps KIPs should include a permalink to the first message of the > > DISCUSS > > > thread. E.g., > > > > > > > > > http://mail-archives.apache.org/mod_mbox/kafka-dev/201509.mbox/%3CCAHrRUm5jvL_dPeZWnfBD-vONgSZWOq1VL1Ss8OSUOCPXmtg8rQ%40mail.gmail.com%3E > > > > > > Also, just to clarify Jay's comments on the content of KIPs: I think > > having > > > a pseudo-code spec/implementation guide is useful (especially for > > > client-side KIPs). While the motivation should definitely capture “why > we > > > are doing the KIP” it probably shouldn’t have to exhaustively capture > > “why > > > we are doing the KIP *this way*”. i.e., some of the discussions are > > > extremely nuanced and in this case spans multiple KIPs so links to > other > > > KIPs and the discuss threads and KIP hangout recordings are perhaps > > > sufficient to fill this gap - or maybe a new section that summarizes > the > > > discussions. > > > > > > Thanks, > > > > > > Joel > > > > > > On Wed, Jan 6, 2016 at 9:29 AM, Jun Rao wrote: > > > > > > > Hi, Jiangjie, > > > > > > > > 52. Replacing MessageSet with o.a.k.common.record will be ideal. > > > > Unfortunately, we use MessageSet in SimpleConsumer, which is part of > > the > > > > public api. Replacing MessageSet with o.a.k.common.record will be an > > > > incompatible api change. So, we probably should do this after we > > > deprecate > > > > SimpleConsumer. > > > > > > > > My original question is actually whether we just bump up magic byte > in > > > > Message once to incorporate both the offset and the timestamp change. > > It > > > > seems that the answer is yes. Could you reflect that in the KIP? > > > > > > > > Thanks, > > > > > > > > Jun > > > > > > > > > > > > On Wed, Jan 6, 2016 at 7:01 AM, Becket Qin > > wrote: > > > > > > > > > Thanks a lot for the careful reading, Jun. > > > > > Please see inline replies. > > > > > > > > > > > > > > > > On Jan 6, 2016, at 3:24 AM, Jun Rao wrote: > > > > > > > > > > > > Jiangjie, > > > > > > > > > > > > Thanks for the updated KIP. Overall, a +1 on the proposal. A few > > > minor > > > > > > comments on the KIP. > > > > > > > > > > > > KIP-32: > > > > > > 50. 6.c says "The log rolling has to depend on the earliest > > > timestamp", > > > > > > which is inconsistent with KIP-33. > > > > > Corrected. > > > > > > > > > > > > 51. 8.b "If the time difference threshold is set to 0. The > > timestamp > > > in > > > > > the > > > > > > message is equivalent to LogAppendTime." If the time difference > is > > 0 > > > > and > > > > > > CreateTime is used, all messages will likely be rejected in this > > > > > proposal. > > > > > > So, it's not equivalent to LogAppendTime. > > > > > Corrected. > > > > > > > > > > > > 52. Could you include the new value of magic byte in message > format > > > > > change? > > > > > > Also, do we have a single
Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.
Hi Anna, That sounds good to me - Becket/others any thoughts? Thanks, Joel On Fri, Jan 8, 2016 at 12:41 PM, Anna Povzner wrote: > Hi Becket and everyone, > > Could we please add the following functionality to this KIP. I think it > would be very useful for the broker to return the timestamp in the ack to > the producer (in response: timestamp per partition) and propagate it back > to client in RecordMetadata. This way, if timestamp type is LogAppendTime, > the producer client will see what timestamp was actually set -- and it > would match the timestamp that consumer sees. Also, returning the timestamp > in RecordMetadata is also useful for timestamp type = CreateTime, since > timestamp could be also set in KafkaProducer (if client set timestamp in > ProducerRecord to 0). > > Since this requires protocol change as well, it will be better to implement > this as part of KIP-32, rather than proposing a new KIP. > > Thanks, > Anna > > > On Fri, Jan 8, 2016 at 12:53 PM, Joel Koshy wrote: > > > +1 from me > > > > Looking through this thread it seems there was some confusion on the > > migration discussion. This discussion in fact happened in the KIP-31 > > discuss thread, not so much in the KIP hangout. There is considerable > > overlap in discussions between KIP-3[1,2,3] so it makes sense to > > cross-reference all of these. > > > > I'm finding the Apache list archive a little cumbersome to use (e.g., the > > current link in KIP-31 points to the beginning of September archives) but > > the emails discussing migration were in October: > > http://mail-archives.apache.org/mod_mbox/kafka-dev/201510.mbox/thread > > > > Markmail has a better interface but interestingly it has not indexed any > of > > the emails from August, September and early October ( > > > > > http://markmail.org/search/?q=list%3Aorg.apache.incubator.kafka-dev+date%3A201509-201511+order%3Adate-backward > > ). > > Perhaps KIPs should include a permalink to the first message of the > DISCUSS > > thread. E.g., > > > > > http://mail-archives.apache.org/mod_mbox/kafka-dev/201509.mbox/%3CCAHrRUm5jvL_dPeZWnfBD-vONgSZWOq1VL1Ss8OSUOCPXmtg8rQ%40mail.gmail.com%3E > > > > Also, just to clarify Jay's comments on the content of KIPs: I think > having > > a pseudo-code spec/implementation guide is useful (especially for > > client-side KIPs). While the motivation should definitely capture “why we > > are doing the KIP” it probably shouldn’t have to exhaustively capture > “why > > we are doing the KIP *this way*”. i.e., some of the discussions are > > extremely nuanced and in this case spans multiple KIPs so links to other > > KIPs and the discuss threads and KIP hangout recordings are perhaps > > sufficient to fill this gap - or maybe a new section that summarizes the > > discussions. > > > > Thanks, > > > > Joel > > > > On Wed, Jan 6, 2016 at 9:29 AM, Jun Rao wrote: > > > > > Hi, Jiangjie, > > > > > > 52. Replacing MessageSet with o.a.k.common.record will be ideal. > > > Unfortunately, we use MessageSet in SimpleConsumer, which is part of > the > > > public api. Replacing MessageSet with o.a.k.common.record will be an > > > incompatible api change. So, we probably should do this after we > > deprecate > > > SimpleConsumer. > > > > > > My original question is actually whether we just bump up magic byte in > > > Message once to incorporate both the offset and the timestamp change. > It > > > seems that the answer is yes. Could you reflect that in the KIP? > > > > > > Thanks, > > > > > > Jun > > > > > > > > > On Wed, Jan 6, 2016 at 7:01 AM, Becket Qin > wrote: > > > > > > > Thanks a lot for the careful reading, Jun. > > > > Please see inline replies. > > > > > > > > > > > > > On Jan 6, 2016, at 3:24 AM, Jun Rao wrote: > > > > > > > > > > Jiangjie, > > > > > > > > > > Thanks for the updated KIP. Overall, a +1 on the proposal. A few > > minor > > > > > comments on the KIP. > > > > > > > > > > KIP-32: > > > > > 50. 6.c says "The log rolling has to depend on the earliest > > timestamp", > > > > > which is inconsistent with KIP-33. > > > > Corrected. > > > > > > > > > > 51. 8.b "If the time difference threshold is set to 0. The > timestamp > > in > > > > the > > > > > message is equivalent to LogAppendTime." If the time difference is > 0 > > > and > > > > > CreateTime is used, all messages will likely be rejected in this > > > > proposal. > > > > > So, it's not equivalent to LogAppendTime. > > > > Corrected. > > > > > > > > > > 52. Could you include the new value of magic byte in message format > > > > change? > > > > > Also, do we have a single new message format that includes both the > > > > offset > > > > > change (relative offset for inner messages) and the addition of > > > > timestamp? > > > > I am actually thinking about this when I am writing the patch. > > > > The timestamp will be added to the o.a.k.common.record.Record and > > > > Kafka.message.Message. The offset change is in > > > > o.a.k.common.record.MemoryRecords and Kafka.message.MessageSet.
Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.
Hi Becket and everyone, Could we please add the following functionality to this KIP. I think it would be very useful for the broker to return the timestamp in the ack to the producer (in response: timestamp per partition) and propagate it back to client in RecordMetadata. This way, if timestamp type is LogAppendTime, the producer client will see what timestamp was actually set -- and it would match the timestamp that consumer sees. Also, returning the timestamp in RecordMetadata is also useful for timestamp type = CreateTime, since timestamp could be also set in KafkaProducer (if client set timestamp in ProducerRecord to 0). Since this requires protocol change as well, it will be better to implement this as part of KIP-32, rather than proposing a new KIP. Thanks, Anna On Fri, Jan 8, 2016 at 12:53 PM, Joel Koshy wrote: > +1 from me > > Looking through this thread it seems there was some confusion on the > migration discussion. This discussion in fact happened in the KIP-31 > discuss thread, not so much in the KIP hangout. There is considerable > overlap in discussions between KIP-3[1,2,3] so it makes sense to > cross-reference all of these. > > I'm finding the Apache list archive a little cumbersome to use (e.g., the > current link in KIP-31 points to the beginning of September archives) but > the emails discussing migration were in October: > http://mail-archives.apache.org/mod_mbox/kafka-dev/201510.mbox/thread > > Markmail has a better interface but interestingly it has not indexed any of > the emails from August, September and early October ( > > http://markmail.org/search/?q=list%3Aorg.apache.incubator.kafka-dev+date%3A201509-201511+order%3Adate-backward > ). > Perhaps KIPs should include a permalink to the first message of the DISCUSS > thread. E.g., > > http://mail-archives.apache.org/mod_mbox/kafka-dev/201509.mbox/%3CCAHrRUm5jvL_dPeZWnfBD-vONgSZWOq1VL1Ss8OSUOCPXmtg8rQ%40mail.gmail.com%3E > > Also, just to clarify Jay's comments on the content of KIPs: I think having > a pseudo-code spec/implementation guide is useful (especially for > client-side KIPs). While the motivation should definitely capture “why we > are doing the KIP” it probably shouldn’t have to exhaustively capture “why > we are doing the KIP *this way*”. i.e., some of the discussions are > extremely nuanced and in this case spans multiple KIPs so links to other > KIPs and the discuss threads and KIP hangout recordings are perhaps > sufficient to fill this gap - or maybe a new section that summarizes the > discussions. > > Thanks, > > Joel > > On Wed, Jan 6, 2016 at 9:29 AM, Jun Rao wrote: > > > Hi, Jiangjie, > > > > 52. Replacing MessageSet with o.a.k.common.record will be ideal. > > Unfortunately, we use MessageSet in SimpleConsumer, which is part of the > > public api. Replacing MessageSet with o.a.k.common.record will be an > > incompatible api change. So, we probably should do this after we > deprecate > > SimpleConsumer. > > > > My original question is actually whether we just bump up magic byte in > > Message once to incorporate both the offset and the timestamp change. It > > seems that the answer is yes. Could you reflect that in the KIP? > > > > Thanks, > > > > Jun > > > > > > On Wed, Jan 6, 2016 at 7:01 AM, Becket Qin wrote: > > > > > Thanks a lot for the careful reading, Jun. > > > Please see inline replies. > > > > > > > > > > On Jan 6, 2016, at 3:24 AM, Jun Rao wrote: > > > > > > > > Jiangjie, > > > > > > > > Thanks for the updated KIP. Overall, a +1 on the proposal. A few > minor > > > > comments on the KIP. > > > > > > > > KIP-32: > > > > 50. 6.c says "The log rolling has to depend on the earliest > timestamp", > > > > which is inconsistent with KIP-33. > > > Corrected. > > > > > > > > 51. 8.b "If the time difference threshold is set to 0. The timestamp > in > > > the > > > > message is equivalent to LogAppendTime." If the time difference is 0 > > and > > > > CreateTime is used, all messages will likely be rejected in this > > > proposal. > > > > So, it's not equivalent to LogAppendTime. > > > Corrected. > > > > > > > > 52. Could you include the new value of magic byte in message format > > > change? > > > > Also, do we have a single new message format that includes both the > > > offset > > > > change (relative offset for inner messages) and the addition of > > > timestamp? > > > I am actually thinking about this when I am writing the patch. > > > The timestamp will be added to the o.a.k.common.record.Record and > > > Kafka.message.Message. The offset change is in > > > o.a.k.common.record.MemoryRecords and Kafka.message.MessageSet. To > avoid > > > unnecessary changes, my current patch did not merge them together but > > > simply make sure the version of Record(Message) and > > > MemoryRecords(MessageSet) matches. > > > > > > Currently new clients uses classes in o.a.k.common.record, and the > broker > > > and old clients uses classes in kafka.message. > > > I am thinking about doing the followings: > > > 1.
Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.
+1 from me Looking through this thread it seems there was some confusion on the migration discussion. This discussion in fact happened in the KIP-31 discuss thread, not so much in the KIP hangout. There is considerable overlap in discussions between KIP-3[1,2,3] so it makes sense to cross-reference all of these. I'm finding the Apache list archive a little cumbersome to use (e.g., the current link in KIP-31 points to the beginning of September archives) but the emails discussing migration were in October: http://mail-archives.apache.org/mod_mbox/kafka-dev/201510.mbox/thread Markmail has a better interface but interestingly it has not indexed any of the emails from August, September and early October ( http://markmail.org/search/?q=list%3Aorg.apache.incubator.kafka-dev+date%3A201509-201511+order%3Adate-backward). Perhaps KIPs should include a permalink to the first message of the DISCUSS thread. E.g., http://mail-archives.apache.org/mod_mbox/kafka-dev/201509.mbox/%3CCAHrRUm5jvL_dPeZWnfBD-vONgSZWOq1VL1Ss8OSUOCPXmtg8rQ%40mail.gmail.com%3E Also, just to clarify Jay's comments on the content of KIPs: I think having a pseudo-code spec/implementation guide is useful (especially for client-side KIPs). While the motivation should definitely capture “why we are doing the KIP” it probably shouldn’t have to exhaustively capture “why we are doing the KIP *this way*”. i.e., some of the discussions are extremely nuanced and in this case spans multiple KIPs so links to other KIPs and the discuss threads and KIP hangout recordings are perhaps sufficient to fill this gap - or maybe a new section that summarizes the discussions. Thanks, Joel On Wed, Jan 6, 2016 at 9:29 AM, Jun Rao wrote: > Hi, Jiangjie, > > 52. Replacing MessageSet with o.a.k.common.record will be ideal. > Unfortunately, we use MessageSet in SimpleConsumer, which is part of the > public api. Replacing MessageSet with o.a.k.common.record will be an > incompatible api change. So, we probably should do this after we deprecate > SimpleConsumer. > > My original question is actually whether we just bump up magic byte in > Message once to incorporate both the offset and the timestamp change. It > seems that the answer is yes. Could you reflect that in the KIP? > > Thanks, > > Jun > > > On Wed, Jan 6, 2016 at 7:01 AM, Becket Qin wrote: > > > Thanks a lot for the careful reading, Jun. > > Please see inline replies. > > > > > > > On Jan 6, 2016, at 3:24 AM, Jun Rao wrote: > > > > > > Jiangjie, > > > > > > Thanks for the updated KIP. Overall, a +1 on the proposal. A few minor > > > comments on the KIP. > > > > > > KIP-32: > > > 50. 6.c says "The log rolling has to depend on the earliest timestamp", > > > which is inconsistent with KIP-33. > > Corrected. > > > > > > 51. 8.b "If the time difference threshold is set to 0. The timestamp in > > the > > > message is equivalent to LogAppendTime." If the time difference is 0 > and > > > CreateTime is used, all messages will likely be rejected in this > > proposal. > > > So, it's not equivalent to LogAppendTime. > > Corrected. > > > > > > 52. Could you include the new value of magic byte in message format > > change? > > > Also, do we have a single new message format that includes both the > > offset > > > change (relative offset for inner messages) and the addition of > > timestamp? > > I am actually thinking about this when I am writing the patch. > > The timestamp will be added to the o.a.k.common.record.Record and > > Kafka.message.Message. The offset change is in > > o.a.k.common.record.MemoryRecords and Kafka.message.MessageSet. To avoid > > unnecessary changes, my current patch did not merge them together but > > simply make sure the version of Record(Message) and > > MemoryRecords(MessageSet) matches. > > > > Currently new clients uses classes in o.a.k.common.record, and the broker > > and old clients uses classes in kafka.message. > > I am thinking about doing the followings: > > 1. Migrate broker to use o.a.k.common.record. > > 2. Add message format V0 and V1 to o.a.k.common.protocol.Protocols. > > Ideally we should be able to define all the wire protocols in > > o.a.k.common.protocol.Protocol. So instead of having Record class to > parse > > byte arrays by itself, we can use Schema to parse the records. > > > > Would that be better? > > > > > > 53. Could you document the changes in ProducerRequest V2 and > FetchRequest > > > V2 (and the responses)? > > Done. > > > > > > 54. In migration phase 1, step 2, does internal ApiVersion mean > > > inter.broker.protocol.version? > > Yes. > > > > > > 55. In canary step 2.b, it says "It will only see > > > ProduceRequest/FetchRequest V1 from other brokers and clietns.". But in > > > phase 2, a broker will receive FetchRequest V2 from other brokers. > > I meant when we canary a broker in phase 2, there will be only one broker > > entering phase 2, the other brokers will remain at phase 1. > > > > > > > > > KIP-33: > > > 60. The KIP still says maintaining index at "at
Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.
Hi, Jiangjie, 52. Replacing MessageSet with o.a.k.common.record will be ideal. Unfortunately, we use MessageSet in SimpleConsumer, which is part of the public api. Replacing MessageSet with o.a.k.common.record will be an incompatible api change. So, we probably should do this after we deprecate SimpleConsumer. My original question is actually whether we just bump up magic byte in Message once to incorporate both the offset and the timestamp change. It seems that the answer is yes. Could you reflect that in the KIP? Thanks, Jun On Wed, Jan 6, 2016 at 7:01 AM, Becket Qin wrote: > Thanks a lot for the careful reading, Jun. > Please see inline replies. > > > > On Jan 6, 2016, at 3:24 AM, Jun Rao wrote: > > > > Jiangjie, > > > > Thanks for the updated KIP. Overall, a +1 on the proposal. A few minor > > comments on the KIP. > > > > KIP-32: > > 50. 6.c says "The log rolling has to depend on the earliest timestamp", > > which is inconsistent with KIP-33. > Corrected. > > > > 51. 8.b "If the time difference threshold is set to 0. The timestamp in > the > > message is equivalent to LogAppendTime." If the time difference is 0 and > > CreateTime is used, all messages will likely be rejected in this > proposal. > > So, it's not equivalent to LogAppendTime. > Corrected. > > > > 52. Could you include the new value of magic byte in message format > change? > > Also, do we have a single new message format that includes both the > offset > > change (relative offset for inner messages) and the addition of > timestamp? > I am actually thinking about this when I am writing the patch. > The timestamp will be added to the o.a.k.common.record.Record and > Kafka.message.Message. The offset change is in > o.a.k.common.record.MemoryRecords and Kafka.message.MessageSet. To avoid > unnecessary changes, my current patch did not merge them together but > simply make sure the version of Record(Message) and > MemoryRecords(MessageSet) matches. > > Currently new clients uses classes in o.a.k.common.record, and the broker > and old clients uses classes in kafka.message. > I am thinking about doing the followings: > 1. Migrate broker to use o.a.k.common.record. > 2. Add message format V0 and V1 to o.a.k.common.protocol.Protocols. > Ideally we should be able to define all the wire protocols in > o.a.k.common.protocol.Protocol. So instead of having Record class to parse > byte arrays by itself, we can use Schema to parse the records. > > Would that be better? > > > > 53. Could you document the changes in ProducerRequest V2 and FetchRequest > > V2 (and the responses)? > Done. > > > > 54. In migration phase 1, step 2, does internal ApiVersion mean > > inter.broker.protocol.version? > Yes. > > > > 55. In canary step 2.b, it says "It will only see > > ProduceRequest/FetchRequest V1 from other brokers and clietns.". But in > > phase 2, a broker will receive FetchRequest V2 from other brokers. > I meant when we canary a broker in phase 2, there will be only one broker > entering phase 2, the other brokers will remain at phase 1. > > > > > > KIP-33: > > 60. The KIP still says maintaining index at "at minute granularity" even > > though the index interval is configurable now. > Corrected. > > > > 61. In this design, it's possible for a log segment to have an empty time > > index. In the worse case, we may have to scan more than the active > segment > > to recover the latest timestamp. > Corrected. > > > > Thanks, > > > > Jun > > > > On Mon, Jan 4, 2016 at 11:37 AM, Aditya Auradkar < > > aaurad...@linkedin.com.invalid> wrote: > > > >> Hey Becket/Anna - > >> > >> I have a few comments about the KIP. > >> > >> 1. (Minor) Can we rename the KIP? It's currently "Add CreateTime and > >> LogAppendTime etc..". This is actually the title of the now rejected > Option > >> 1. > >> 2. (Minor) Can we rename the proposed option? It isn't really "option 4" > >> anymore. > >> 3. I'm not clear on what exactly happens to compressed messages > >> when message.timestamp.type=LogAppendTime? Does every batch get > >> recompressed because the inner message gets rewritten with the server > >> timestamp? Or does the message set on disk have the timestamp set to > -1. In > >> that case, what do we use as timestamp for the message? > >> 4. Do message.timestamp.type and max.message.time.difference.ms need > to be > >> per-topic configs? It seems that this is really a client config i.e. a > >> client is the source of timestamps not a topic. It could also be a > >> broker-level config to keep things simple. > >> 5. The "Proposed Changes" section in the KIP tries to build a time-based > >> index for query but that is a separate proposal (KIP-33). Can we more > >> crisply identify what exactly will change when this KIP (and 31) is > >> implemented? It isn't super clear to me at this point. > >> > >> Aside from that, I think the "Rejected Alternatives" section of the KIP > is > >> excellent. Very good insight into what options were discussed and > rejected. > >> > >> Aditya
Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.
Hey Jay, I agree with your comments on the KIP page. As you said it is important to document what are the changes and why we made those decisions. I have added summaries to the page to help readers understand the decision we made more easily and also try to avoid being too verbose. I also clarified the scope of KIP-32 and KIP-33. Please let me know if you think we should make further improvements. Thanks, Jiangjie (Becket) Qin > On Jan 6, 2016, at 7:21 AM, Jay Kreps wrote: > > I'm +1 on the design, but this proposal doesn't look like it is quite > finished. We are trying to make the KIPs really be a record of design > decisions we made and why. We should be able to give the document to > advanced Kafka users and have them walk away with a good idea of what > we are changing and why. I think having this kind of record of why > changes were made is really valuable especially given the amount of > times these things come back up (I'm quite sure this saves time in the > long run). > > If people think this is a waste of time, let's revisit, but if not > could we try to bring this up to the level of quality of the other > KIPs? > > In particular, a few things: > > Overall this reads more like implementation notes I might make for > myself rather than a design doc. > > This doesn't really motivate the idea of creation vs append time or > when each would be used or why you would want both. > > The public interfaces section says "There are a few options to achieve > the goals. Each has their own pros and cons. Please see the details > below." This might have been true in the first version as we were > discussing but now we have one proposal, right? We should list out the > new public interfaces (binary format, java api, configs). This is a > way of calling out that "the user gets this new stuff". > > This still has a number of details that I think we've now moved to > KIP-33. i.e. the bits about indexing are now part of that? > > Cheers, > > -Jay > >> On Tue, Jan 5, 2016 at 11:47 AM, Neha Narkhede wrote: >> +1 on the KIP. Thanks for the hard work, Becket! >> >>> On Tue, Jan 5, 2016 at 11:24 AM, Jun Rao wrote: >>> >>> Jiangjie, >>> >>> Thanks for the updated KIP. Overall, a +1 on the proposal. A few minor >>> comments on the KIP. >>> >>> KIP-32: >>> 50. 6.c says "The log rolling has to depend on the earliest timestamp", >>> which is inconsistent with KIP-33. >>> >>> 51. 8.b "If the time difference threshold is set to 0. The timestamp in the >>> message is equivalent to LogAppendTime." If the time difference is 0 and >>> CreateTime is used, all messages will likely be rejected in this proposal. >>> So, it's not equivalent to LogAppendTime. >>> >>> 52. Could you include the new value of magic byte in message format change? >>> Also, do we have a single new message format that includes both the offset >>> change (relative offset for inner messages) and the addition of timestamp? >>> >>> 53. Could you document the changes in ProducerRequest V2 and FetchRequest >>> V2 (and the responses)? >>> >>> 54. In migration phase 1, step 2, does internal ApiVersion mean >>> inter.broker.protocol.version? >>> >>> 55. In canary step 2.b, it says "It will only see >>> ProduceRequest/FetchRequest V1 from other brokers and clietns.". But in >>> phase 2, a broker will receive FetchRequest V2 from other brokers. >>> >>> >>> KIP-33: >>> 60. The KIP still says maintaining index at "at minute granularity" even >>> though the index interval is configurable now. >>> >>> 61. In this design, it's possible for a log segment to have an empty time >>> index. In the worse case, we may have to scan more than the active segment >>> to recover the latest timestamp. >>> >>> Thanks, >>> >>> Jun >>> >>> On Mon, Jan 4, 2016 at 11:37 AM, Aditya Auradkar < >>> aaurad...@linkedin.com.invalid> wrote: >>> Hey Becket/Anna - I have a few comments about the KIP. 1. (Minor) Can we rename the KIP? It's currently "Add CreateTime and LogAppendTime etc..". This is actually the title of the now rejected >>> Option 1. 2. (Minor) Can we rename the proposed option? It isn't really "option 4" anymore. 3. I'm not clear on what exactly happens to compressed messages when message.timestamp.type=LogAppendTime? Does every batch get recompressed because the inner message gets rewritten with the server timestamp? Or does the message set on disk have the timestamp set to -1. >>> In that case, what do we use as timestamp for the message? 4. Do message.timestamp.type and max.message.time.difference.ms need to >>> be per-topic configs? It seems that this is really a client config i.e. a client is the source of timestamps not a topic. It could also be a broker-level config to keep things simple. 5. The "Proposed Changes" section in the KIP tries to build a time-based index for query but that is a separate proposal (KIP-33). Can we more crisply
Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.
Thanks a lot for the careful reading, Jun. Please see inline replies. > On Jan 6, 2016, at 3:24 AM, Jun Rao wrote: > > Jiangjie, > > Thanks for the updated KIP. Overall, a +1 on the proposal. A few minor > comments on the KIP. > > KIP-32: > 50. 6.c says "The log rolling has to depend on the earliest timestamp", > which is inconsistent with KIP-33. Corrected. > > 51. 8.b "If the time difference threshold is set to 0. The timestamp in the > message is equivalent to LogAppendTime." If the time difference is 0 and > CreateTime is used, all messages will likely be rejected in this proposal. > So, it's not equivalent to LogAppendTime. Corrected. > > 52. Could you include the new value of magic byte in message format change? > Also, do we have a single new message format that includes both the offset > change (relative offset for inner messages) and the addition of timestamp? I am actually thinking about this when I am writing the patch. The timestamp will be added to the o.a.k.common.record.Record and Kafka.message.Message. The offset change is in o.a.k.common.record.MemoryRecords and Kafka.message.MessageSet. To avoid unnecessary changes, my current patch did not merge them together but simply make sure the version of Record(Message) and MemoryRecords(MessageSet) matches. Currently new clients uses classes in o.a.k.common.record, and the broker and old clients uses classes in kafka.message. I am thinking about doing the followings: 1. Migrate broker to use o.a.k.common.record. 2. Add message format V0 and V1 to o.a.k.common.protocol.Protocols. Ideally we should be able to define all the wire protocols in o.a.k.common.protocol.Protocol. So instead of having Record class to parse byte arrays by itself, we can use Schema to parse the records. Would that be better? > > 53. Could you document the changes in ProducerRequest V2 and FetchRequest > V2 (and the responses)? Done. > > 54. In migration phase 1, step 2, does internal ApiVersion mean > inter.broker.protocol.version? Yes. > > 55. In canary step 2.b, it says "It will only see > ProduceRequest/FetchRequest V1 from other brokers and clietns.". But in > phase 2, a broker will receive FetchRequest V2 from other brokers. I meant when we canary a broker in phase 2, there will be only one broker entering phase 2, the other brokers will remain at phase 1. > > > KIP-33: > 60. The KIP still says maintaining index at "at minute granularity" even > though the index interval is configurable now. Corrected. > > 61. In this design, it's possible for a log segment to have an empty time > index. In the worse case, we may have to scan more than the active segment > to recover the latest timestamp. Corrected. > > Thanks, > > Jun > > On Mon, Jan 4, 2016 at 11:37 AM, Aditya Auradkar < > aaurad...@linkedin.com.invalid> wrote: > >> Hey Becket/Anna - >> >> I have a few comments about the KIP. >> >> 1. (Minor) Can we rename the KIP? It's currently "Add CreateTime and >> LogAppendTime etc..". This is actually the title of the now rejected Option >> 1. >> 2. (Minor) Can we rename the proposed option? It isn't really "option 4" >> anymore. >> 3. I'm not clear on what exactly happens to compressed messages >> when message.timestamp.type=LogAppendTime? Does every batch get >> recompressed because the inner message gets rewritten with the server >> timestamp? Or does the message set on disk have the timestamp set to -1. In >> that case, what do we use as timestamp for the message? >> 4. Do message.timestamp.type and max.message.time.difference.ms need to be >> per-topic configs? It seems that this is really a client config i.e. a >> client is the source of timestamps not a topic. It could also be a >> broker-level config to keep things simple. >> 5. The "Proposed Changes" section in the KIP tries to build a time-based >> index for query but that is a separate proposal (KIP-33). Can we more >> crisply identify what exactly will change when this KIP (and 31) is >> implemented? It isn't super clear to me at this point. >> >> Aside from that, I think the "Rejected Alternatives" section of the KIP is >> excellent. Very good insight into what options were discussed and rejected. >> >> Aditya >> >>> On Mon, Dec 28, 2015 at 3:57 PM, Becket Qin wrote: >>> >>> Thanks Guozhang, Gwen and Neha for the comments. Sorry for late reply >>> because I only have occasional gmail access from my phone... >>> >>> I just updated the wiki for KIP-32. >>> >>> Gwen, >>> >>> Yes, the migration plan is what you described. >>> >>> I agree with your comments on the version. >>> I changed message.format.version to use the release version. >>> I did not change the internal version, we can discuss this in a separate >>> thread. >>> >>> Thanks, >>> >>> Jiangjie (Becket) Qin >>> >>> >>> On Dec 24, 2015, at 5:38 AM, Guozhang Wang wrote: Also I agree with Gwen that such changes may worth a 0.10 release or >> even 1.0, having it in 0.9.1 would be quite confus
Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.
I'm +1 on the design, but this proposal doesn't look like it is quite finished. We are trying to make the KIPs really be a record of design decisions we made and why. We should be able to give the document to advanced Kafka users and have them walk away with a good idea of what we are changing and why. I think having this kind of record of why changes were made is really valuable especially given the amount of times these things come back up (I'm quite sure this saves time in the long run). If people think this is a waste of time, let's revisit, but if not could we try to bring this up to the level of quality of the other KIPs? In particular, a few things: Overall this reads more like implementation notes I might make for myself rather than a design doc. This doesn't really motivate the idea of creation vs append time or when each would be used or why you would want both. The public interfaces section says "There are a few options to achieve the goals. Each has their own pros and cons. Please see the details below." This might have been true in the first version as we were discussing but now we have one proposal, right? We should list out the new public interfaces (binary format, java api, configs). This is a way of calling out that "the user gets this new stuff". This still has a number of details that I think we've now moved to KIP-33. i.e. the bits about indexing are now part of that? Cheers, -Jay On Tue, Jan 5, 2016 at 11:47 AM, Neha Narkhede wrote: > +1 on the KIP. Thanks for the hard work, Becket! > > On Tue, Jan 5, 2016 at 11:24 AM, Jun Rao wrote: > >> Jiangjie, >> >> Thanks for the updated KIP. Overall, a +1 on the proposal. A few minor >> comments on the KIP. >> >> KIP-32: >> 50. 6.c says "The log rolling has to depend on the earliest timestamp", >> which is inconsistent with KIP-33. >> >> 51. 8.b "If the time difference threshold is set to 0. The timestamp in the >> message is equivalent to LogAppendTime." If the time difference is 0 and >> CreateTime is used, all messages will likely be rejected in this proposal. >> So, it's not equivalent to LogAppendTime. >> >> 52. Could you include the new value of magic byte in message format change? >> Also, do we have a single new message format that includes both the offset >> change (relative offset for inner messages) and the addition of timestamp? >> >> 53. Could you document the changes in ProducerRequest V2 and FetchRequest >> V2 (and the responses)? >> >> 54. In migration phase 1, step 2, does internal ApiVersion mean >> inter.broker.protocol.version? >> >> 55. In canary step 2.b, it says "It will only see >> ProduceRequest/FetchRequest V1 from other brokers and clietns.". But in >> phase 2, a broker will receive FetchRequest V2 from other brokers. >> >> >> KIP-33: >> 60. The KIP still says maintaining index at "at minute granularity" even >> though the index interval is configurable now. >> >> 61. In this design, it's possible for a log segment to have an empty time >> index. In the worse case, we may have to scan more than the active segment >> to recover the latest timestamp. >> >> Thanks, >> >> Jun >> >> On Mon, Jan 4, 2016 at 11:37 AM, Aditya Auradkar < >> aaurad...@linkedin.com.invalid> wrote: >> >> > Hey Becket/Anna - >> > >> > I have a few comments about the KIP. >> > >> > 1. (Minor) Can we rename the KIP? It's currently "Add CreateTime and >> > LogAppendTime etc..". This is actually the title of the now rejected >> Option >> > 1. >> > 2. (Minor) Can we rename the proposed option? It isn't really "option 4" >> > anymore. >> > 3. I'm not clear on what exactly happens to compressed messages >> > when message.timestamp.type=LogAppendTime? Does every batch get >> > recompressed because the inner message gets rewritten with the server >> > timestamp? Or does the message set on disk have the timestamp set to -1. >> In >> > that case, what do we use as timestamp for the message? >> > 4. Do message.timestamp.type and max.message.time.difference.ms need to >> be >> > per-topic configs? It seems that this is really a client config i.e. a >> > client is the source of timestamps not a topic. It could also be a >> > broker-level config to keep things simple. >> > 5. The "Proposed Changes" section in the KIP tries to build a time-based >> > index for query but that is a separate proposal (KIP-33). Can we more >> > crisply identify what exactly will change when this KIP (and 31) is >> > implemented? It isn't super clear to me at this point. >> > >> > Aside from that, I think the "Rejected Alternatives" section of the KIP >> is >> > excellent. Very good insight into what options were discussed and >> rejected. >> > >> > Aditya >> > >> > On Mon, Dec 28, 2015 at 3:57 PM, Becket Qin >> wrote: >> > >> > > Thanks Guozhang, Gwen and Neha for the comments. Sorry for late reply >> > > because I only have occasional gmail access from my phone... >> > > >> > > I just updated the wiki for KIP-32. >> > > >> > > Gwen, >> > > >> > > Yes, the migration plan is
Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.
+1. Great work Becket. Aditya On Tue, Jan 5, 2016 at 11:47 AM, Neha Narkhede wrote: > +1 on the KIP. Thanks for the hard work, Becket! > > On Tue, Jan 5, 2016 at 11:24 AM, Jun Rao wrote: > > > Jiangjie, > > > > Thanks for the updated KIP. Overall, a +1 on the proposal. A few minor > > comments on the KIP. > > > > KIP-32: > > 50. 6.c says "The log rolling has to depend on the earliest timestamp", > > which is inconsistent with KIP-33. > > > > 51. 8.b "If the time difference threshold is set to 0. The timestamp in > the > > message is equivalent to LogAppendTime." If the time difference is 0 and > > CreateTime is used, all messages will likely be rejected in this > proposal. > > So, it's not equivalent to LogAppendTime. > > > > 52. Could you include the new value of magic byte in message format > change? > > Also, do we have a single new message format that includes both the > offset > > change (relative offset for inner messages) and the addition of > timestamp? > > > > 53. Could you document the changes in ProducerRequest V2 and FetchRequest > > V2 (and the responses)? > > > > 54. In migration phase 1, step 2, does internal ApiVersion mean > > inter.broker.protocol.version? > > > > 55. In canary step 2.b, it says "It will only see > > ProduceRequest/FetchRequest V1 from other brokers and clietns.". But in > > phase 2, a broker will receive FetchRequest V2 from other brokers. > > > > > > KIP-33: > > 60. The KIP still says maintaining index at "at minute granularity" even > > though the index interval is configurable now. > > > > 61. In this design, it's possible for a log segment to have an empty time > > index. In the worse case, we may have to scan more than the active > segment > > to recover the latest timestamp. > > > > Thanks, > > > > Jun > > > > On Mon, Jan 4, 2016 at 11:37 AM, Aditya Auradkar < > > aaurad...@linkedin.com.invalid> wrote: > > > > > Hey Becket/Anna - > > > > > > I have a few comments about the KIP. > > > > > > 1. (Minor) Can we rename the KIP? It's currently "Add CreateTime and > > > LogAppendTime etc..". This is actually the title of the now rejected > > Option > > > 1. > > > 2. (Minor) Can we rename the proposed option? It isn't really "option > 4" > > > anymore. > > > 3. I'm not clear on what exactly happens to compressed messages > > > when message.timestamp.type=LogAppendTime? Does every batch get > > > recompressed because the inner message gets rewritten with the server > > > timestamp? Or does the message set on disk have the timestamp set to > -1. > > In > > > that case, what do we use as timestamp for the message? > > > 4. Do message.timestamp.type and max.message.time.difference.ms need > to > > be > > > per-topic configs? It seems that this is really a client config i.e. a > > > client is the source of timestamps not a topic. It could also be a > > > broker-level config to keep things simple. > > > 5. The "Proposed Changes" section in the KIP tries to build a > time-based > > > index for query but that is a separate proposal (KIP-33). Can we more > > > crisply identify what exactly will change when this KIP (and 31) is > > > implemented? It isn't super clear to me at this point. > > > > > > Aside from that, I think the "Rejected Alternatives" section of the KIP > > is > > > excellent. Very good insight into what options were discussed and > > rejected. > > > > > > Aditya > > > > > > On Mon, Dec 28, 2015 at 3:57 PM, Becket Qin > > wrote: > > > > > > > Thanks Guozhang, Gwen and Neha for the comments. Sorry for late reply > > > > because I only have occasional gmail access from my phone... > > > > > > > > I just updated the wiki for KIP-32. > > > > > > > > Gwen, > > > > > > > > Yes, the migration plan is what you described. > > > > > > > > I agree with your comments on the version. > > > > I changed message.format.version to use the release version. > > > > I did not change the internal version, we can discuss this in a > > separate > > > > thread. > > > > > > > > Thanks, > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > On Dec 24, 2015, at 5:38 AM, Guozhang Wang > > wrote: > > > > > > > > > > Also I agree with Gwen that such changes may worth a 0.10 release > or > > > even > > > > > 1.0, having it in 0.9.1 would be quite confusing to users. > > > > > > > > > > Guozhang > > > > > > > > > >> On Wed, Dec 23, 2015 at 1:36 PM, Guozhang Wang < > wangg...@gmail.com> > > > > wrote: > > > > >> > > > > >> Becket, > > > > >> > > > > >> Please let us know once you have updated the wiki page regarding > the > > > > >> migration plan. Thanks! > > > > >> > > > > >> Guozhang > > > > >> > > > > >>> On Wed, Dec 23, 2015 at 11:52 AM, Gwen Shapira < > g...@confluent.io> > > > > wrote: > > > > >>> > > > > >>> Thanks Becket, Anne and Neha for responding to my concern. > > > > >>> > > > > >>> I had an offline discussion with Anne where she helped me > > understand > > > > the > > > > >>> migration process. It isn't as bad as it looks in the KIP :) > > > > >>> > > > > >>
Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.
+1 on the KIP. Thanks for the hard work, Becket! On Tue, Jan 5, 2016 at 11:24 AM, Jun Rao wrote: > Jiangjie, > > Thanks for the updated KIP. Overall, a +1 on the proposal. A few minor > comments on the KIP. > > KIP-32: > 50. 6.c says "The log rolling has to depend on the earliest timestamp", > which is inconsistent with KIP-33. > > 51. 8.b "If the time difference threshold is set to 0. The timestamp in the > message is equivalent to LogAppendTime." If the time difference is 0 and > CreateTime is used, all messages will likely be rejected in this proposal. > So, it's not equivalent to LogAppendTime. > > 52. Could you include the new value of magic byte in message format change? > Also, do we have a single new message format that includes both the offset > change (relative offset for inner messages) and the addition of timestamp? > > 53. Could you document the changes in ProducerRequest V2 and FetchRequest > V2 (and the responses)? > > 54. In migration phase 1, step 2, does internal ApiVersion mean > inter.broker.protocol.version? > > 55. In canary step 2.b, it says "It will only see > ProduceRequest/FetchRequest V1 from other brokers and clietns.". But in > phase 2, a broker will receive FetchRequest V2 from other brokers. > > > KIP-33: > 60. The KIP still says maintaining index at "at minute granularity" even > though the index interval is configurable now. > > 61. In this design, it's possible for a log segment to have an empty time > index. In the worse case, we may have to scan more than the active segment > to recover the latest timestamp. > > Thanks, > > Jun > > On Mon, Jan 4, 2016 at 11:37 AM, Aditya Auradkar < > aaurad...@linkedin.com.invalid> wrote: > > > Hey Becket/Anna - > > > > I have a few comments about the KIP. > > > > 1. (Minor) Can we rename the KIP? It's currently "Add CreateTime and > > LogAppendTime etc..". This is actually the title of the now rejected > Option > > 1. > > 2. (Minor) Can we rename the proposed option? It isn't really "option 4" > > anymore. > > 3. I'm not clear on what exactly happens to compressed messages > > when message.timestamp.type=LogAppendTime? Does every batch get > > recompressed because the inner message gets rewritten with the server > > timestamp? Or does the message set on disk have the timestamp set to -1. > In > > that case, what do we use as timestamp for the message? > > 4. Do message.timestamp.type and max.message.time.difference.ms need to > be > > per-topic configs? It seems that this is really a client config i.e. a > > client is the source of timestamps not a topic. It could also be a > > broker-level config to keep things simple. > > 5. The "Proposed Changes" section in the KIP tries to build a time-based > > index for query but that is a separate proposal (KIP-33). Can we more > > crisply identify what exactly will change when this KIP (and 31) is > > implemented? It isn't super clear to me at this point. > > > > Aside from that, I think the "Rejected Alternatives" section of the KIP > is > > excellent. Very good insight into what options were discussed and > rejected. > > > > Aditya > > > > On Mon, Dec 28, 2015 at 3:57 PM, Becket Qin > wrote: > > > > > Thanks Guozhang, Gwen and Neha for the comments. Sorry for late reply > > > because I only have occasional gmail access from my phone... > > > > > > I just updated the wiki for KIP-32. > > > > > > Gwen, > > > > > > Yes, the migration plan is what you described. > > > > > > I agree with your comments on the version. > > > I changed message.format.version to use the release version. > > > I did not change the internal version, we can discuss this in a > separate > > > thread. > > > > > > Thanks, > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > On Dec 24, 2015, at 5:38 AM, Guozhang Wang > wrote: > > > > > > > > Also I agree with Gwen that such changes may worth a 0.10 release or > > even > > > > 1.0, having it in 0.9.1 would be quite confusing to users. > > > > > > > > Guozhang > > > > > > > >> On Wed, Dec 23, 2015 at 1:36 PM, Guozhang Wang > > > wrote: > > > >> > > > >> Becket, > > > >> > > > >> Please let us know once you have updated the wiki page regarding the > > > >> migration plan. Thanks! > > > >> > > > >> Guozhang > > > >> > > > >>> On Wed, Dec 23, 2015 at 11:52 AM, Gwen Shapira > > > wrote: > > > >>> > > > >>> Thanks Becket, Anne and Neha for responding to my concern. > > > >>> > > > >>> I had an offline discussion with Anne where she helped me > understand > > > the > > > >>> migration process. It isn't as bad as it looks in the KIP :) > > > >>> > > > >>> If I understand it correctly, the process (for users) will be: > > > >>> > > > >>> 1. Prepare for upgrade (set format.version = 0, ApiVersion = 0.9.0) > > > >>> 2. Rolling upgrade of brokers > > > >>> 3. Bump ApiVersion to 0.9.0-1, so fetch requests between brokers > will > > > use > > > >>> the new protocol > > > >>> 4. Start upgrading clients > > > >>> 5. When "enough" clients are upgraded, bump format.version to 1 >
Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.
Jiangjie, Thanks for the updated KIP. Overall, a +1 on the proposal. A few minor comments on the KIP. KIP-32: 50. 6.c says "The log rolling has to depend on the earliest timestamp", which is inconsistent with KIP-33. 51. 8.b "If the time difference threshold is set to 0. The timestamp in the message is equivalent to LogAppendTime." If the time difference is 0 and CreateTime is used, all messages will likely be rejected in this proposal. So, it's not equivalent to LogAppendTime. 52. Could you include the new value of magic byte in message format change? Also, do we have a single new message format that includes both the offset change (relative offset for inner messages) and the addition of timestamp? 53. Could you document the changes in ProducerRequest V2 and FetchRequest V2 (and the responses)? 54. In migration phase 1, step 2, does internal ApiVersion mean inter.broker.protocol.version? 55. In canary step 2.b, it says "It will only see ProduceRequest/FetchRequest V1 from other brokers and clietns.". But in phase 2, a broker will receive FetchRequest V2 from other brokers. KIP-33: 60. The KIP still says maintaining index at "at minute granularity" even though the index interval is configurable now. 61. In this design, it's possible for a log segment to have an empty time index. In the worse case, we may have to scan more than the active segment to recover the latest timestamp. Thanks, Jun On Mon, Jan 4, 2016 at 11:37 AM, Aditya Auradkar < aaurad...@linkedin.com.invalid> wrote: > Hey Becket/Anna - > > I have a few comments about the KIP. > > 1. (Minor) Can we rename the KIP? It's currently "Add CreateTime and > LogAppendTime etc..". This is actually the title of the now rejected Option > 1. > 2. (Minor) Can we rename the proposed option? It isn't really "option 4" > anymore. > 3. I'm not clear on what exactly happens to compressed messages > when message.timestamp.type=LogAppendTime? Does every batch get > recompressed because the inner message gets rewritten with the server > timestamp? Or does the message set on disk have the timestamp set to -1. In > that case, what do we use as timestamp for the message? > 4. Do message.timestamp.type and max.message.time.difference.ms need to be > per-topic configs? It seems that this is really a client config i.e. a > client is the source of timestamps not a topic. It could also be a > broker-level config to keep things simple. > 5. The "Proposed Changes" section in the KIP tries to build a time-based > index for query but that is a separate proposal (KIP-33). Can we more > crisply identify what exactly will change when this KIP (and 31) is > implemented? It isn't super clear to me at this point. > > Aside from that, I think the "Rejected Alternatives" section of the KIP is > excellent. Very good insight into what options were discussed and rejected. > > Aditya > > On Mon, Dec 28, 2015 at 3:57 PM, Becket Qin wrote: > > > Thanks Guozhang, Gwen and Neha for the comments. Sorry for late reply > > because I only have occasional gmail access from my phone... > > > > I just updated the wiki for KIP-32. > > > > Gwen, > > > > Yes, the migration plan is what you described. > > > > I agree with your comments on the version. > > I changed message.format.version to use the release version. > > I did not change the internal version, we can discuss this in a separate > > thread. > > > > Thanks, > > > > Jiangjie (Becket) Qin > > > > > > > > > On Dec 24, 2015, at 5:38 AM, Guozhang Wang wrote: > > > > > > Also I agree with Gwen that such changes may worth a 0.10 release or > even > > > 1.0, having it in 0.9.1 would be quite confusing to users. > > > > > > Guozhang > > > > > >> On Wed, Dec 23, 2015 at 1:36 PM, Guozhang Wang > > wrote: > > >> > > >> Becket, > > >> > > >> Please let us know once you have updated the wiki page regarding the > > >> migration plan. Thanks! > > >> > > >> Guozhang > > >> > > >>> On Wed, Dec 23, 2015 at 11:52 AM, Gwen Shapira > > wrote: > > >>> > > >>> Thanks Becket, Anne and Neha for responding to my concern. > > >>> > > >>> I had an offline discussion with Anne where she helped me understand > > the > > >>> migration process. It isn't as bad as it looks in the KIP :) > > >>> > > >>> If I understand it correctly, the process (for users) will be: > > >>> > > >>> 1. Prepare for upgrade (set format.version = 0, ApiVersion = 0.9.0) > > >>> 2. Rolling upgrade of brokers > > >>> 3. Bump ApiVersion to 0.9.0-1, so fetch requests between brokers will > > use > > >>> the new protocol > > >>> 4. Start upgrading clients > > >>> 5. When "enough" clients are upgraded, bump format.version to 1 > > (rolling). > > >>> > > >>> Becket, can you confirm? > > >>> > > >>> Assuming this is the process, I'm +1 on the change. > > >>> > > >>> Reminder to coders and reviewers that pull-requests with user-facing > > >>> changes should include documentation changes as well as code changes. > > >>> And a polite request to try to be helpful to users on when to use > > >>>
Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.
+1 On Tue, Jan 5, 2016 at 11:12 AM, Guozhang Wang wrote: > +1, the KIP looks good to me now. > > Guozhang > > On Tue, Jan 5, 2016 at 1:47 AM, Becket Qin wrote: > > > Hi Adi, > > > > Please see inline comments. And thanks for correcting my typos :) > > > > > On Jan 5, 2016, at 3:37 AM, Aditya Auradkar > > wrote: > > > > > > Hey Becket/Anna - > > > > > > I have a few comments about the KIP. > > > > > > 1. (Minor) Can we rename the KIP? It's currently "Add CreateTime and > > > LogAppendTime etc..". This is actually the title of the now rejected > > Option > > Done. Please notice the link will change as well. The previous link in > the > > thread no longer works. > > > 1. > > > 2. (Minor) Can we rename the proposed option? It isn't really "option > 4" > > > anymore. > > Done. > > > 3. I'm not clear on what exactly happens to compressed messages > > > when message.timestamp.type=LogAppendTime? Does every batch get > > > recompressed because the inner message gets rewritten with the server > > > timestamp? Or does the message set on disk have the timestamp set to > -1. > > In > > > that case, what do we use as timestamp for the message? > > The current proposal is to keep the inner message timestamp=-1. Broker > > only overwrites it if it is not -1, in that case the broker will do > > recompression. > > > 4. Do message.timestamp.type and max.message.time.difference.ms need > to > > be > > > per-topic configs? It seems that this is really a client config i.e. a > > > client is the source of timestamps not a topic. It could also be a > > > broker-level config to keep things simple. > > I feel the current proposal is more server oriented, i.e, server decides > > whether accept, overwrite or reject a timestamp. This ensures the > timestamp > > on the broker has the same type. I am not sure about the granularity. My > > gut feeling is that topic level configuration helps with multi tenant > > cluster. Eventually it is the application who'd decide how they want to > > define the timestamp and it might be hard to for all the applications > have > > same timestamp usage in the same cluster. > > > 5. The "Proposed Changes" section in the KIP tries to build a > time-based > > > index for query but that is a separate proposal (KIP-33). Can we more > > > crisply identify what exactly will change when this KIP (and 31) is > > > implemented? It isn't super clear to me at this point. > > Good point. They are not going to be implemented in this KIP. They are > > listed because they are closely related to KIP-32. > > > > Other than the time index and changes on top of it, KIP-31 and KIP-32 > will > > implement everything else needed for message format change. > > > > > > Aside from that, I think the "Rejected Alternatives" section of the KIP > > is > > > excellent. Very good insight into what options were discussed and > > rejected. > > > > > > Aditya > > > > > >> On Mon, Dec 28, 2015 at 3:57 PM, Becket Qin > > wrote: > > >> > > >> Thanks Guozhang, Gwen and Neha for the comments. Sorry for late reply > > >> because I only have occasional gmail access from my phone... > > >> > > >> I just updated the wiki for KIP-32. > > >> > > >> Gwen, > > >> > > >> Yes, the migration plan is what you described. > > >> > > >> I agree with your comments on the version. > > >> I changed message.format.version to use the release version. > > >> I did not change the internal version, we can discuss this in a > separate > > >> thread. > > >> > > >> Thanks, > > >> > > >> Jiangjie (Becket) Qin > > >> > > >> > > >> > > >>> On Dec 24, 2015, at 5:38 AM, Guozhang Wang > wrote: > > >>> > > >>> Also I agree with Gwen that such changes may worth a 0.10 release or > > even > > >>> 1.0, having it in 0.9.1 would be quite confusing to users. > > >>> > > >>> Guozhang > > >>> > > On Wed, Dec 23, 2015 at 1:36 PM, Guozhang Wang > > >> wrote: > > > > Becket, > > > > Please let us know once you have updated the wiki page regarding the > > migration plan. Thanks! > > > > Guozhang > > > > > On Wed, Dec 23, 2015 at 11:52 AM, Gwen Shapira > > >> wrote: > > > > > > Thanks Becket, Anne and Neha for responding to my concern. > > > > > > I had an offline discussion with Anne where she helped me > understand > > >> the > > > migration process. It isn't as bad as it looks in the KIP :) > > > > > > If I understand it correctly, the process (for users) will be: > > > > > > 1. Prepare for upgrade (set format.version = 0, ApiVersion = 0.9.0) > > > 2. Rolling upgrade of brokers > > > 3. Bump ApiVersion to 0.9.0-1, so fetch requests between brokers > will > > >> use > > > the new protocol > > > 4. Start upgrading clients > > > 5. When "enough" clients are upgraded, bump format.version to 1 > > >> (rolling). > > > > > > Becket, can you confirm? > > > > > > Assuming this is the process, I'm +1 on the change. > > > > > > Reminder to coders and reviewer
Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.
+1, the KIP looks good to me now. Guozhang On Tue, Jan 5, 2016 at 1:47 AM, Becket Qin wrote: > Hi Adi, > > Please see inline comments. And thanks for correcting my typos :) > > > On Jan 5, 2016, at 3:37 AM, Aditya Auradkar > wrote: > > > > Hey Becket/Anna - > > > > I have a few comments about the KIP. > > > > 1. (Minor) Can we rename the KIP? It's currently "Add CreateTime and > > LogAppendTime etc..". This is actually the title of the now rejected > Option > Done. Please notice the link will change as well. The previous link in the > thread no longer works. > > 1. > > 2. (Minor) Can we rename the proposed option? It isn't really "option 4" > > anymore. > Done. > > 3. I'm not clear on what exactly happens to compressed messages > > when message.timestamp.type=LogAppendTime? Does every batch get > > recompressed because the inner message gets rewritten with the server > > timestamp? Or does the message set on disk have the timestamp set to -1. > In > > that case, what do we use as timestamp for the message? > The current proposal is to keep the inner message timestamp=-1. Broker > only overwrites it if it is not -1, in that case the broker will do > recompression. > > 4. Do message.timestamp.type and max.message.time.difference.ms need to > be > > per-topic configs? It seems that this is really a client config i.e. a > > client is the source of timestamps not a topic. It could also be a > > broker-level config to keep things simple. > I feel the current proposal is more server oriented, i.e, server decides > whether accept, overwrite or reject a timestamp. This ensures the timestamp > on the broker has the same type. I am not sure about the granularity. My > gut feeling is that topic level configuration helps with multi tenant > cluster. Eventually it is the application who'd decide how they want to > define the timestamp and it might be hard to for all the applications have > same timestamp usage in the same cluster. > > 5. The "Proposed Changes" section in the KIP tries to build a time-based > > index for query but that is a separate proposal (KIP-33). Can we more > > crisply identify what exactly will change when this KIP (and 31) is > > implemented? It isn't super clear to me at this point. > Good point. They are not going to be implemented in this KIP. They are > listed because they are closely related to KIP-32. > > Other than the time index and changes on top of it, KIP-31 and KIP-32 will > implement everything else needed for message format change. > > > > Aside from that, I think the "Rejected Alternatives" section of the KIP > is > > excellent. Very good insight into what options were discussed and > rejected. > > > > Aditya > > > >> On Mon, Dec 28, 2015 at 3:57 PM, Becket Qin > wrote: > >> > >> Thanks Guozhang, Gwen and Neha for the comments. Sorry for late reply > >> because I only have occasional gmail access from my phone... > >> > >> I just updated the wiki for KIP-32. > >> > >> Gwen, > >> > >> Yes, the migration plan is what you described. > >> > >> I agree with your comments on the version. > >> I changed message.format.version to use the release version. > >> I did not change the internal version, we can discuss this in a separate > >> thread. > >> > >> Thanks, > >> > >> Jiangjie (Becket) Qin > >> > >> > >> > >>> On Dec 24, 2015, at 5:38 AM, Guozhang Wang wrote: > >>> > >>> Also I agree with Gwen that such changes may worth a 0.10 release or > even > >>> 1.0, having it in 0.9.1 would be quite confusing to users. > >>> > >>> Guozhang > >>> > On Wed, Dec 23, 2015 at 1:36 PM, Guozhang Wang > >> wrote: > > Becket, > > Please let us know once you have updated the wiki page regarding the > migration plan. Thanks! > > Guozhang > > > On Wed, Dec 23, 2015 at 11:52 AM, Gwen Shapira > >> wrote: > > > > Thanks Becket, Anne and Neha for responding to my concern. > > > > I had an offline discussion with Anne where she helped me understand > >> the > > migration process. It isn't as bad as it looks in the KIP :) > > > > If I understand it correctly, the process (for users) will be: > > > > 1. Prepare for upgrade (set format.version = 0, ApiVersion = 0.9.0) > > 2. Rolling upgrade of brokers > > 3. Bump ApiVersion to 0.9.0-1, so fetch requests between brokers will > >> use > > the new protocol > > 4. Start upgrading clients > > 5. When "enough" clients are upgraded, bump format.version to 1 > >> (rolling). > > > > Becket, can you confirm? > > > > Assuming this is the process, I'm +1 on the change. > > > > Reminder to coders and reviewers that pull-requests with user-facing > > changes should include documentation changes as well as code changes. > > And a polite request to try to be helpful to users on when to use > > create-time and when to use log-append-time as configuration - this > is > >> not > > a trivial decision. > > > > A separate point
Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.
Hi Adi, Please see inline comments. And thanks for correcting my typos :) > On Jan 5, 2016, at 3:37 AM, Aditya Auradkar > wrote: > > Hey Becket/Anna - > > I have a few comments about the KIP. > > 1. (Minor) Can we rename the KIP? It's currently "Add CreateTime and > LogAppendTime etc..". This is actually the title of the now rejected Option Done. Please notice the link will change as well. The previous link in the thread no longer works. > 1. > 2. (Minor) Can we rename the proposed option? It isn't really "option 4" > anymore. Done. > 3. I'm not clear on what exactly happens to compressed messages > when message.timestamp.type=LogAppendTime? Does every batch get > recompressed because the inner message gets rewritten with the server > timestamp? Or does the message set on disk have the timestamp set to -1. In > that case, what do we use as timestamp for the message? The current proposal is to keep the inner message timestamp=-1. Broker only overwrites it if it is not -1, in that case the broker will do recompression. > 4. Do message.timestamp.type and max.message.time.difference.ms need to be > per-topic configs? It seems that this is really a client config i.e. a > client is the source of timestamps not a topic. It could also be a > broker-level config to keep things simple. I feel the current proposal is more server oriented, i.e, server decides whether accept, overwrite or reject a timestamp. This ensures the timestamp on the broker has the same type. I am not sure about the granularity. My gut feeling is that topic level configuration helps with multi tenant cluster. Eventually it is the application who'd decide how they want to define the timestamp and it might be hard to for all the applications have same timestamp usage in the same cluster. > 5. The "Proposed Changes" section in the KIP tries to build a time-based > index for query but that is a separate proposal (KIP-33). Can we more > crisply identify what exactly will change when this KIP (and 31) is > implemented? It isn't super clear to me at this point. Good point. They are not going to be implemented in this KIP. They are listed because they are closely related to KIP-32. Other than the time index and changes on top of it, KIP-31 and KIP-32 will implement everything else needed for message format change. > > Aside from that, I think the "Rejected Alternatives" section of the KIP is > excellent. Very good insight into what options were discussed and rejected. > > Aditya > >> On Mon, Dec 28, 2015 at 3:57 PM, Becket Qin wrote: >> >> Thanks Guozhang, Gwen and Neha for the comments. Sorry for late reply >> because I only have occasional gmail access from my phone... >> >> I just updated the wiki for KIP-32. >> >> Gwen, >> >> Yes, the migration plan is what you described. >> >> I agree with your comments on the version. >> I changed message.format.version to use the release version. >> I did not change the internal version, we can discuss this in a separate >> thread. >> >> Thanks, >> >> Jiangjie (Becket) Qin >> >> >> >>> On Dec 24, 2015, at 5:38 AM, Guozhang Wang wrote: >>> >>> Also I agree with Gwen that such changes may worth a 0.10 release or even >>> 1.0, having it in 0.9.1 would be quite confusing to users. >>> >>> Guozhang >>> On Wed, Dec 23, 2015 at 1:36 PM, Guozhang Wang >> wrote: Becket, Please let us know once you have updated the wiki page regarding the migration plan. Thanks! Guozhang > On Wed, Dec 23, 2015 at 11:52 AM, Gwen Shapira >> wrote: > > Thanks Becket, Anne and Neha for responding to my concern. > > I had an offline discussion with Anne where she helped me understand >> the > migration process. It isn't as bad as it looks in the KIP :) > > If I understand it correctly, the process (for users) will be: > > 1. Prepare for upgrade (set format.version = 0, ApiVersion = 0.9.0) > 2. Rolling upgrade of brokers > 3. Bump ApiVersion to 0.9.0-1, so fetch requests between brokers will >> use > the new protocol > 4. Start upgrading clients > 5. When "enough" clients are upgraded, bump format.version to 1 >> (rolling). > > Becket, can you confirm? > > Assuming this is the process, I'm +1 on the change. > > Reminder to coders and reviewers that pull-requests with user-facing > changes should include documentation changes as well as code changes. > And a polite request to try to be helpful to users on when to use > create-time and when to use log-append-time as configuration - this is >> not > a trivial decision. > > A separate point I'm going to raise in a different thread is that we >> need > to streamline our versions a bit: > 1. I'm afraid that 0.9.0-1 will be confusing to users who care about > released versions (what if we forget to change it before the release? >> Is > it > meaningful enough to someone running off t
Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.
Hey Becket/Anna - I have a few comments about the KIP. 1. (Minor) Can we rename the KIP? It's currently "Add CreateTime and LogAppendTime etc..". This is actually the title of the now rejected Option 1. 2. (Minor) Can we rename the proposed option? It isn't really "option 4" anymore. 3. I'm not clear on what exactly happens to compressed messages when message.timestamp.type=LogAppendTime? Does every batch get recompressed because the inner message gets rewritten with the server timestamp? Or does the message set on disk have the timestamp set to -1. In that case, what do we use as timestamp for the message? 4. Do message.timestamp.type and max.message.time.difference.ms need to be per-topic configs? It seems that this is really a client config i.e. a client is the source of timestamps not a topic. It could also be a broker-level config to keep things simple. 5. The "Proposed Changes" section in the KIP tries to build a time-based index for query but that is a separate proposal (KIP-33). Can we more crisply identify what exactly will change when this KIP (and 31) is implemented? It isn't super clear to me at this point. Aside from that, I think the "Rejected Alternatives" section of the KIP is excellent. Very good insight into what options were discussed and rejected. Aditya On Mon, Dec 28, 2015 at 3:57 PM, Becket Qin wrote: > Thanks Guozhang, Gwen and Neha for the comments. Sorry for late reply > because I only have occasional gmail access from my phone... > > I just updated the wiki for KIP-32. > > Gwen, > > Yes, the migration plan is what you described. > > I agree with your comments on the version. > I changed message.format.version to use the release version. > I did not change the internal version, we can discuss this in a separate > thread. > > Thanks, > > Jiangjie (Becket) Qin > > > > > On Dec 24, 2015, at 5:38 AM, Guozhang Wang wrote: > > > > Also I agree with Gwen that such changes may worth a 0.10 release or even > > 1.0, having it in 0.9.1 would be quite confusing to users. > > > > Guozhang > > > >> On Wed, Dec 23, 2015 at 1:36 PM, Guozhang Wang > wrote: > >> > >> Becket, > >> > >> Please let us know once you have updated the wiki page regarding the > >> migration plan. Thanks! > >> > >> Guozhang > >> > >>> On Wed, Dec 23, 2015 at 11:52 AM, Gwen Shapira > wrote: > >>> > >>> Thanks Becket, Anne and Neha for responding to my concern. > >>> > >>> I had an offline discussion with Anne where she helped me understand > the > >>> migration process. It isn't as bad as it looks in the KIP :) > >>> > >>> If I understand it correctly, the process (for users) will be: > >>> > >>> 1. Prepare for upgrade (set format.version = 0, ApiVersion = 0.9.0) > >>> 2. Rolling upgrade of brokers > >>> 3. Bump ApiVersion to 0.9.0-1, so fetch requests between brokers will > use > >>> the new protocol > >>> 4. Start upgrading clients > >>> 5. When "enough" clients are upgraded, bump format.version to 1 > (rolling). > >>> > >>> Becket, can you confirm? > >>> > >>> Assuming this is the process, I'm +1 on the change. > >>> > >>> Reminder to coders and reviewers that pull-requests with user-facing > >>> changes should include documentation changes as well as code changes. > >>> And a polite request to try to be helpful to users on when to use > >>> create-time and when to use log-append-time as configuration - this is > not > >>> a trivial decision. > >>> > >>> A separate point I'm going to raise in a different thread is that we > need > >>> to streamline our versions a bit: > >>> 1. I'm afraid that 0.9.0-1 will be confusing to users who care about > >>> released versions (what if we forget to change it before the release? > Is > >>> it > >>> meaningful enough to someone running off trunk?), we need to come up > with > >>> something that will work for both LinkedIn and everyone else. > >>> 2. ApiVersion has real version numbers. message.format.version has > >>> sequence > >>> numbers. This makes us look pretty silly :) > >>> > >>> My version concerns can be addressed separately and should not hold > back > >>> this KIP. > >>> > >>> Gwen > >>> > >>> > >>> > >>> On Tue, Dec 22, 2015 at 11:01 PM, Becket Qin > >>> wrote: > >>> > Hi Anna, > > Thanks for initiating the voting process. I did not start the voting > process because there were still some ongoing discussion with Jun > about > >>> the > timestamp regarding compressed messages. That is why the wiki page > >>> hasn't > reflected the latest conversation as Guozhang pointed out. > > Like Neha said I think we have reached general agreement on this KIP. > So > it is probably fine to start the KIP voting. At least we draw more > attention to the KIP even if there are some new discussion to bring > up. > > Regarding the upgrade plan, given we decided to implement KIP-31 and > KIP-32 in the same patch to avoid change binary protocol twice, the > >>> upgrade > plan was mostly discussed on the discussion
Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.
Thanks Guozhang, Gwen and Neha for the comments. Sorry for late reply because I only have occasional gmail access from my phone... I just updated the wiki for KIP-32. Gwen, Yes, the migration plan is what you described. I agree with your comments on the version. I changed message.format.version to use the release version. I did not change the internal version, we can discuss this in a separate thread. Thanks, Jiangjie (Becket) Qin > On Dec 24, 2015, at 5:38 AM, Guozhang Wang wrote: > > Also I agree with Gwen that such changes may worth a 0.10 release or even > 1.0, having it in 0.9.1 would be quite confusing to users. > > Guozhang > >> On Wed, Dec 23, 2015 at 1:36 PM, Guozhang Wang wrote: >> >> Becket, >> >> Please let us know once you have updated the wiki page regarding the >> migration plan. Thanks! >> >> Guozhang >> >>> On Wed, Dec 23, 2015 at 11:52 AM, Gwen Shapira wrote: >>> >>> Thanks Becket, Anne and Neha for responding to my concern. >>> >>> I had an offline discussion with Anne where she helped me understand the >>> migration process. It isn't as bad as it looks in the KIP :) >>> >>> If I understand it correctly, the process (for users) will be: >>> >>> 1. Prepare for upgrade (set format.version = 0, ApiVersion = 0.9.0) >>> 2. Rolling upgrade of brokers >>> 3. Bump ApiVersion to 0.9.0-1, so fetch requests between brokers will use >>> the new protocol >>> 4. Start upgrading clients >>> 5. When "enough" clients are upgraded, bump format.version to 1 (rolling). >>> >>> Becket, can you confirm? >>> >>> Assuming this is the process, I'm +1 on the change. >>> >>> Reminder to coders and reviewers that pull-requests with user-facing >>> changes should include documentation changes as well as code changes. >>> And a polite request to try to be helpful to users on when to use >>> create-time and when to use log-append-time as configuration - this is not >>> a trivial decision. >>> >>> A separate point I'm going to raise in a different thread is that we need >>> to streamline our versions a bit: >>> 1. I'm afraid that 0.9.0-1 will be confusing to users who care about >>> released versions (what if we forget to change it before the release? Is >>> it >>> meaningful enough to someone running off trunk?), we need to come up with >>> something that will work for both LinkedIn and everyone else. >>> 2. ApiVersion has real version numbers. message.format.version has >>> sequence >>> numbers. This makes us look pretty silly :) >>> >>> My version concerns can be addressed separately and should not hold back >>> this KIP. >>> >>> Gwen >>> >>> >>> >>> On Tue, Dec 22, 2015 at 11:01 PM, Becket Qin >>> wrote: >>> Hi Anna, Thanks for initiating the voting process. I did not start the voting process because there were still some ongoing discussion with Jun about >>> the timestamp regarding compressed messages. That is why the wiki page >>> hasn't reflected the latest conversation as Guozhang pointed out. Like Neha said I think we have reached general agreement on this KIP. So it is probably fine to start the KIP voting. At least we draw more attention to the KIP even if there are some new discussion to bring up. Regarding the upgrade plan, given we decided to implement KIP-31 and KIP-32 in the same patch to avoid change binary protocol twice, the >>> upgrade plan was mostly discussed on the discussion thread of KIP-31. Since the voting has been initiated, I will update the wiki with latest conversation to avoid further confusion. BTW, I actually have started coding work on KIP-31 and KIP-32 and will focus on the patch before I return from vacation in mid Jan because I >>> have no LInkedIn VPN access in China anyway... Thanks, Jiangjie > On Dec 23, 2015, at 12:31 PM, Anna Povzner wrote: > > Hi Gwen, > > I just wanted to point out that I just started the vote. Becket wrote >>> the > proposal and led the discussions. > > What I understood from reading the discussion thread, the migration >>> plan > was discussed at the KIP meeting, and not much on the mailing list itself. > > My question about the migration plan was same as Guozhang wrote: The >>> case > when an upgraded broker receives an old producer request. The >>> proposal is > for the broker to fill in the timestamp field with the current time at the > broker. Cons: it goes against the definition of CreateTime type of the > timestamp (we are "over-writing" it at the broker). Pros: It looks >>> like > most of the use-cases would actually want that behavior, because otherwise > timestamp is useless and they will need to support an old, >>> pre-timestamp, > behavior. E.g., if we modify log retention policy to use the >>> timestamp, we > would need to support an old implementation (the one that does n
Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.
Also I agree with Gwen that such changes may worth a 0.10 release or even 1.0, having it in 0.9.1 would be quite confusing to users. Guozhang On Wed, Dec 23, 2015 at 1:36 PM, Guozhang Wang wrote: > Becket, > > Please let us know once you have updated the wiki page regarding the > migration plan. Thanks! > > Guozhang > > On Wed, Dec 23, 2015 at 11:52 AM, Gwen Shapira wrote: > >> Thanks Becket, Anne and Neha for responding to my concern. >> >> I had an offline discussion with Anne where she helped me understand the >> migration process. It isn't as bad as it looks in the KIP :) >> >> If I understand it correctly, the process (for users) will be: >> >> 1. Prepare for upgrade (set format.version = 0, ApiVersion = 0.9.0) >> 2. Rolling upgrade of brokers >> 3. Bump ApiVersion to 0.9.0-1, so fetch requests between brokers will use >> the new protocol >> 4. Start upgrading clients >> 5. When "enough" clients are upgraded, bump format.version to 1 (rolling). >> >> Becket, can you confirm? >> >> Assuming this is the process, I'm +1 on the change. >> >> Reminder to coders and reviewers that pull-requests with user-facing >> changes should include documentation changes as well as code changes. >> And a polite request to try to be helpful to users on when to use >> create-time and when to use log-append-time as configuration - this is not >> a trivial decision. >> >> A separate point I'm going to raise in a different thread is that we need >> to streamline our versions a bit: >> 1. I'm afraid that 0.9.0-1 will be confusing to users who care about >> released versions (what if we forget to change it before the release? Is >> it >> meaningful enough to someone running off trunk?), we need to come up with >> something that will work for both LinkedIn and everyone else. >> 2. ApiVersion has real version numbers. message.format.version has >> sequence >> numbers. This makes us look pretty silly :) >> >> My version concerns can be addressed separately and should not hold back >> this KIP. >> >> Gwen >> >> >> >> On Tue, Dec 22, 2015 at 11:01 PM, Becket Qin >> wrote: >> >> > Hi Anna, >> > >> > Thanks for initiating the voting process. I did not start the voting >> > process because there were still some ongoing discussion with Jun about >> the >> > timestamp regarding compressed messages. That is why the wiki page >> hasn't >> > reflected the latest conversation as Guozhang pointed out. >> > >> > Like Neha said I think we have reached general agreement on this KIP. So >> > it is probably fine to start the KIP voting. At least we draw more >> > attention to the KIP even if there are some new discussion to bring up. >> > >> > Regarding the upgrade plan, given we decided to implement KIP-31 and >> > KIP-32 in the same patch to avoid change binary protocol twice, the >> upgrade >> > plan was mostly discussed on the discussion thread of KIP-31. >> > >> > Since the voting has been initiated, I will update the wiki with latest >> > conversation to avoid further confusion. >> > >> > BTW, I actually have started coding work on KIP-31 and KIP-32 and will >> > focus on the patch before I return from vacation in mid Jan because I >> have >> > no LInkedIn VPN access in China anyway... >> > >> > Thanks, >> > >> > Jiangjie >> > >> > > On Dec 23, 2015, at 12:31 PM, Anna Povzner wrote: >> > > >> > > Hi Gwen, >> > > >> > > I just wanted to point out that I just started the vote. Becket wrote >> the >> > > proposal and led the discussions. >> > > >> > > What I understood from reading the discussion thread, the migration >> plan >> > > was discussed at the KIP meeting, and not much on the mailing list >> > itself. >> > > >> > > My question about the migration plan was same as Guozhang wrote: The >> case >> > > when an upgraded broker receives an old producer request. The >> proposal is >> > > for the broker to fill in the timestamp field with the current time at >> > the >> > > broker. Cons: it goes against the definition of CreateTime type of the >> > > timestamp (we are "over-writing" it at the broker). Pros: It looks >> like >> > > most of the use-cases would actually want that behavior, because >> > otherwise >> > > timestamp is useless and they will need to support an old, >> pre-timestamp, >> > > behavior. E.g., if we modify log retention policy to use the >> timestamp, >> > we >> > > would need to support an old implementation (the one that does not use >> > > timestamps in the message). So I actually have a preference for the >> > > proposed approach. >> > > >> > > Thanks, >> > > Anna >> > > >> > >> On Tue, Dec 22, 2015 at 8:02 PM, Neha Narkhede >> > wrote: >> > >> >> > >> Hey Gwen, >> > >> >> > >> Migration plan wasn't really discussed a ton in the previous threads. >> > So it >> > >> will be great to dive deep and see if there are gaps there. I had >> some >> > >> questions, but the details listed on the KIP are great. >> > >> >> > >> It is complex, though the plan outlined in the wiki assumes a zero >> > downtime >> > >> upgrad
Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.
Becket, Please let us know once you have updated the wiki page regarding the migration plan. Thanks! Guozhang On Wed, Dec 23, 2015 at 11:52 AM, Gwen Shapira wrote: > Thanks Becket, Anne and Neha for responding to my concern. > > I had an offline discussion with Anne where she helped me understand the > migration process. It isn't as bad as it looks in the KIP :) > > If I understand it correctly, the process (for users) will be: > > 1. Prepare for upgrade (set format.version = 0, ApiVersion = 0.9.0) > 2. Rolling upgrade of brokers > 3. Bump ApiVersion to 0.9.0-1, so fetch requests between brokers will use > the new protocol > 4. Start upgrading clients > 5. When "enough" clients are upgraded, bump format.version to 1 (rolling). > > Becket, can you confirm? > > Assuming this is the process, I'm +1 on the change. > > Reminder to coders and reviewers that pull-requests with user-facing > changes should include documentation changes as well as code changes. > And a polite request to try to be helpful to users on when to use > create-time and when to use log-append-time as configuration - this is not > a trivial decision. > > A separate point I'm going to raise in a different thread is that we need > to streamline our versions a bit: > 1. I'm afraid that 0.9.0-1 will be confusing to users who care about > released versions (what if we forget to change it before the release? Is it > meaningful enough to someone running off trunk?), we need to come up with > something that will work for both LinkedIn and everyone else. > 2. ApiVersion has real version numbers. message.format.version has sequence > numbers. This makes us look pretty silly :) > > My version concerns can be addressed separately and should not hold back > this KIP. > > Gwen > > > > On Tue, Dec 22, 2015 at 11:01 PM, Becket Qin wrote: > > > Hi Anna, > > > > Thanks for initiating the voting process. I did not start the voting > > process because there were still some ongoing discussion with Jun about > the > > timestamp regarding compressed messages. That is why the wiki page hasn't > > reflected the latest conversation as Guozhang pointed out. > > > > Like Neha said I think we have reached general agreement on this KIP. So > > it is probably fine to start the KIP voting. At least we draw more > > attention to the KIP even if there are some new discussion to bring up. > > > > Regarding the upgrade plan, given we decided to implement KIP-31 and > > KIP-32 in the same patch to avoid change binary protocol twice, the > upgrade > > plan was mostly discussed on the discussion thread of KIP-31. > > > > Since the voting has been initiated, I will update the wiki with latest > > conversation to avoid further confusion. > > > > BTW, I actually have started coding work on KIP-31 and KIP-32 and will > > focus on the patch before I return from vacation in mid Jan because I > have > > no LInkedIn VPN access in China anyway... > > > > Thanks, > > > > Jiangjie > > > > > On Dec 23, 2015, at 12:31 PM, Anna Povzner wrote: > > > > > > Hi Gwen, > > > > > > I just wanted to point out that I just started the vote. Becket wrote > the > > > proposal and led the discussions. > > > > > > What I understood from reading the discussion thread, the migration > plan > > > was discussed at the KIP meeting, and not much on the mailing list > > itself. > > > > > > My question about the migration plan was same as Guozhang wrote: The > case > > > when an upgraded broker receives an old producer request. The proposal > is > > > for the broker to fill in the timestamp field with the current time at > > the > > > broker. Cons: it goes against the definition of CreateTime type of the > > > timestamp (we are "over-writing" it at the broker). Pros: It looks like > > > most of the use-cases would actually want that behavior, because > > otherwise > > > timestamp is useless and they will need to support an old, > pre-timestamp, > > > behavior. E.g., if we modify log retention policy to use the timestamp, > > we > > > would need to support an old implementation (the one that does not use > > > timestamps in the message). So I actually have a preference for the > > > proposed approach. > > > > > > Thanks, > > > Anna > > > > > >> On Tue, Dec 22, 2015 at 8:02 PM, Neha Narkhede > > wrote: > > >> > > >> Hey Gwen, > > >> > > >> Migration plan wasn't really discussed a ton in the previous threads. > > So it > > >> will be great to dive deep and see if there are gaps there. I had some > > >> questions, but the details listed on the KIP are great. > > >> > > >> It is complex, though the plan outlined in the wiki assumes a zero > > downtime > > >> upgrade assuming that producers and consumers can't be upgraded in > > tandem. > > >> This is typical for companies that have a significant Kafka footprint, > > like > > >> LinkedIn. > > >> > > >> Thanks, > > >> Neha > > >> > > >>> On Tue, Dec 22, 2015 at 7:48 PM, Gwen Shapira > > wrote: > > >>> > > >>> Hi Anna, > > >>> > > >>> Thanks for the KIP, especial
Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.
Thanks Becket, Anne and Neha for responding to my concern. I had an offline discussion with Anne where she helped me understand the migration process. It isn't as bad as it looks in the KIP :) If I understand it correctly, the process (for users) will be: 1. Prepare for upgrade (set format.version = 0, ApiVersion = 0.9.0) 2. Rolling upgrade of brokers 3. Bump ApiVersion to 0.9.0-1, so fetch requests between brokers will use the new protocol 4. Start upgrading clients 5. When "enough" clients are upgraded, bump format.version to 1 (rolling). Becket, can you confirm? Assuming this is the process, I'm +1 on the change. Reminder to coders and reviewers that pull-requests with user-facing changes should include documentation changes as well as code changes. And a polite request to try to be helpful to users on when to use create-time and when to use log-append-time as configuration - this is not a trivial decision. A separate point I'm going to raise in a different thread is that we need to streamline our versions a bit: 1. I'm afraid that 0.9.0-1 will be confusing to users who care about released versions (what if we forget to change it before the release? Is it meaningful enough to someone running off trunk?), we need to come up with something that will work for both LinkedIn and everyone else. 2. ApiVersion has real version numbers. message.format.version has sequence numbers. This makes us look pretty silly :) My version concerns can be addressed separately and should not hold back this KIP. Gwen On Tue, Dec 22, 2015 at 11:01 PM, Becket Qin wrote: > Hi Anna, > > Thanks for initiating the voting process. I did not start the voting > process because there were still some ongoing discussion with Jun about the > timestamp regarding compressed messages. That is why the wiki page hasn't > reflected the latest conversation as Guozhang pointed out. > > Like Neha said I think we have reached general agreement on this KIP. So > it is probably fine to start the KIP voting. At least we draw more > attention to the KIP even if there are some new discussion to bring up. > > Regarding the upgrade plan, given we decided to implement KIP-31 and > KIP-32 in the same patch to avoid change binary protocol twice, the upgrade > plan was mostly discussed on the discussion thread of KIP-31. > > Since the voting has been initiated, I will update the wiki with latest > conversation to avoid further confusion. > > BTW, I actually have started coding work on KIP-31 and KIP-32 and will > focus on the patch before I return from vacation in mid Jan because I have > no LInkedIn VPN access in China anyway... > > Thanks, > > Jiangjie > > > On Dec 23, 2015, at 12:31 PM, Anna Povzner wrote: > > > > Hi Gwen, > > > > I just wanted to point out that I just started the vote. Becket wrote the > > proposal and led the discussions. > > > > What I understood from reading the discussion thread, the migration plan > > was discussed at the KIP meeting, and not much on the mailing list > itself. > > > > My question about the migration plan was same as Guozhang wrote: The case > > when an upgraded broker receives an old producer request. The proposal is > > for the broker to fill in the timestamp field with the current time at > the > > broker. Cons: it goes against the definition of CreateTime type of the > > timestamp (we are "over-writing" it at the broker). Pros: It looks like > > most of the use-cases would actually want that behavior, because > otherwise > > timestamp is useless and they will need to support an old, pre-timestamp, > > behavior. E.g., if we modify log retention policy to use the timestamp, > we > > would need to support an old implementation (the one that does not use > > timestamps in the message). So I actually have a preference for the > > proposed approach. > > > > Thanks, > > Anna > > > >> On Tue, Dec 22, 2015 at 8:02 PM, Neha Narkhede > wrote: > >> > >> Hey Gwen, > >> > >> Migration plan wasn't really discussed a ton in the previous threads. > So it > >> will be great to dive deep and see if there are gaps there. I had some > >> questions, but the details listed on the KIP are great. > >> > >> It is complex, though the plan outlined in the wiki assumes a zero > downtime > >> upgrade assuming that producers and consumers can't be upgraded in > tandem. > >> This is typical for companies that have a significant Kafka footprint, > like > >> LinkedIn. > >> > >> Thanks, > >> Neha > >> > >>> On Tue, Dec 22, 2015 at 7:48 PM, Gwen Shapira > wrote: > >>> > >>> Hi Anna, > >>> > >>> Thanks for the KIP, especially for the details on all the alternatives > >> and > >>> how we arrived at the proposal. Its really great! > >>> > >>> Can you point me at where the migration plan was discussed? It looks > >> overly > >>> complex and I have a bunch of questions, but if there was a discussion, > >> I'd > >>> like to read up rather than repeating it :) > >>> > >>> Gwen > >>> > On Tue, Dec 22, 2015 at 12:34 PM, Anna Povzner > >>
Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.
Hi Neha, Yeah, I actually think the KIP is ready for vote other than some minor issues. My question is more about who should start the vote and drive the design of this KIP. Given that Jiangjie is on vacation in China with limited access to gmail, I raised the question on his behalf and hopefully he would be able to join the discussion and answer questions during the vote of this process. I love the way we propose, discuss and implement the KIP for kafka. I want to do my part to join and facilitate this development. Thanks, Dong On Tue, Dec 22, 2015 at 7:46 PM, Neha Narkhede wrote: > Hey Anna, > > Thanks for initiating the vote and helping Becket while he is out! > > Hey Dong, > > My understanding from observing all the email threads is that there is a > general agreement on the approach and the wiki is reasonably up-to-date > with all the decisions. Is there a particular reason why the KIP process > should wait? > > > On Tue, Dec 22, 2015 at 2:15 PM, Dong Lin wrote: > > > Hi Anna, > > > > Have you talked with Jiangjie Qin before initiating the vote? Jiangjie > Qin > > is on vocation now and he may not have time to answer question related to > > KIP-32 in detail. > > > > Dong > > > > On Tue, Dec 22, 2015 at 12:34 PM, Anna Povzner > wrote: > > > > > Hi, > > > > > > I am opening the voting thread for KIP-32: Add CreateTime and > > > LogAppendTime to Kafka message. > > > > > > For reference, here's the KIP wiki: > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-32+-+Add+CreateTime+and+LogAppendTime+to+Kafka+message > > > > > > And the mailing list threads: > > > > > > September: > > > > > > http://mail-archives.apache.org/mod_mbox/kafka-dev/201509.mbox/%3CCAHrRUm6NMg%3DPh4HAJdxr%3DpmZhfFcD5OEV2yxj3fg%2BXnEBTW%2B3w%40mail.gmail.com%3E > > > > > > October: > > > > > > http://mail-archives.apache.org/mod_mbox/kafka-dev/201510.mbox/%3CCAHrRUm7RiBAJxwO15s1tztz%3D15oibO-QJ%2B_w8AxafTnuw3jjCw%40mail.gmail.com%3E > > > > > > December: > > > > > > http://mail-archives.apache.org/mod_mbox/kafka-dev/201512.mbox/%3CCAHrRUm4ugxDYzyy26MGRGKpK4hsjT4EKTuu18M3wztYq4PE%3DaQ%40mail.gmail.com%3E > > > > > > > > > Thanks > > > Anna > > > > > > > > > -- > Thanks, > Neha >
Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.
Hi Anna, Thanks for initiating the voting process. I did not start the voting process because there were still some ongoing discussion with Jun about the timestamp regarding compressed messages. That is why the wiki page hasn't reflected the latest conversation as Guozhang pointed out. Like Neha said I think we have reached general agreement on this KIP. So it is probably fine to start the KIP voting. At least we draw more attention to the KIP even if there are some new discussion to bring up. Regarding the upgrade plan, given we decided to implement KIP-31 and KIP-32 in the same patch to avoid change binary protocol twice, the upgrade plan was mostly discussed on the discussion thread of KIP-31. Since the voting has been initiated, I will update the wiki with latest conversation to avoid further confusion. BTW, I actually have started coding work on KIP-31 and KIP-32 and will focus on the patch before I return from vacation in mid Jan because I have no LInkedIn VPN access in China anyway... Thanks, Jiangjie > On Dec 23, 2015, at 12:31 PM, Anna Povzner wrote: > > Hi Gwen, > > I just wanted to point out that I just started the vote. Becket wrote the > proposal and led the discussions. > > What I understood from reading the discussion thread, the migration plan > was discussed at the KIP meeting, and not much on the mailing list itself. > > My question about the migration plan was same as Guozhang wrote: The case > when an upgraded broker receives an old producer request. The proposal is > for the broker to fill in the timestamp field with the current time at the > broker. Cons: it goes against the definition of CreateTime type of the > timestamp (we are "over-writing" it at the broker). Pros: It looks like > most of the use-cases would actually want that behavior, because otherwise > timestamp is useless and they will need to support an old, pre-timestamp, > behavior. E.g., if we modify log retention policy to use the timestamp, we > would need to support an old implementation (the one that does not use > timestamps in the message). So I actually have a preference for the > proposed approach. > > Thanks, > Anna > >> On Tue, Dec 22, 2015 at 8:02 PM, Neha Narkhede wrote: >> >> Hey Gwen, >> >> Migration plan wasn't really discussed a ton in the previous threads. So it >> will be great to dive deep and see if there are gaps there. I had some >> questions, but the details listed on the KIP are great. >> >> It is complex, though the plan outlined in the wiki assumes a zero downtime >> upgrade assuming that producers and consumers can't be upgraded in tandem. >> This is typical for companies that have a significant Kafka footprint, like >> LinkedIn. >> >> Thanks, >> Neha >> >>> On Tue, Dec 22, 2015 at 7:48 PM, Gwen Shapira wrote: >>> >>> Hi Anna, >>> >>> Thanks for the KIP, especially for the details on all the alternatives >> and >>> how we arrived at the proposal. Its really great! >>> >>> Can you point me at where the migration plan was discussed? It looks >> overly >>> complex and I have a bunch of questions, but if there was a discussion, >> I'd >>> like to read up rather than repeating it :) >>> >>> Gwen >>> On Tue, Dec 22, 2015 at 12:34 PM, Anna Povzner >>> wrote: >>> Hi, I am opening the voting thread for KIP-32: Add CreateTime and LogAppendTime to Kafka message. For reference, here's the KIP wiki: >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-32+-+Add+CreateTime+and+LogAppendTime+to+Kafka+message And the mailing list threads: September: >> http://mail-archives.apache.org/mod_mbox/kafka-dev/201509.mbox/%3CCAHrRUm6NMg%3DPh4HAJdxr%3DpmZhfFcD5OEV2yxj3fg%2BXnEBTW%2B3w%40mail.gmail.com%3E October: >> http://mail-archives.apache.org/mod_mbox/kafka-dev/201510.mbox/%3CCAHrRUm7RiBAJxwO15s1tztz%3D15oibO-QJ%2B_w8AxafTnuw3jjCw%40mail.gmail.com%3E December: >> http://mail-archives.apache.org/mod_mbox/kafka-dev/201512.mbox/%3CCAHrRUm4ugxDYzyy26MGRGKpK4hsjT4EKTuu18M3wztYq4PE%3DaQ%40mail.gmail.com%3E Thanks Anna >> >> >> >> -- >> Thanks, >> Neha >>
Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.
Hi Gwen, I just wanted to point out that I just started the vote. Becket wrote the proposal and led the discussions. What I understood from reading the discussion thread, the migration plan was discussed at the KIP meeting, and not much on the mailing list itself. My question about the migration plan was same as Guozhang wrote: The case when an upgraded broker receives an old producer request. The proposal is for the broker to fill in the timestamp field with the current time at the broker. Cons: it goes against the definition of CreateTime type of the timestamp (we are "over-writing" it at the broker). Pros: It looks like most of the use-cases would actually want that behavior, because otherwise timestamp is useless and they will need to support an old, pre-timestamp, behavior. E.g., if we modify log retention policy to use the timestamp, we would need to support an old implementation (the one that does not use timestamps in the message). So I actually have a preference for the proposed approach. Thanks, Anna On Tue, Dec 22, 2015 at 8:02 PM, Neha Narkhede wrote: > Hey Gwen, > > Migration plan wasn't really discussed a ton in the previous threads. So it > will be great to dive deep and see if there are gaps there. I had some > questions, but the details listed on the KIP are great. > > It is complex, though the plan outlined in the wiki assumes a zero downtime > upgrade assuming that producers and consumers can't be upgraded in tandem. > This is typical for companies that have a significant Kafka footprint, like > LinkedIn. > > Thanks, > Neha > > On Tue, Dec 22, 2015 at 7:48 PM, Gwen Shapira wrote: > > > Hi Anna, > > > > Thanks for the KIP, especially for the details on all the alternatives > and > > how we arrived at the proposal. Its really great! > > > > Can you point me at where the migration plan was discussed? It looks > overly > > complex and I have a bunch of questions, but if there was a discussion, > I'd > > like to read up rather than repeating it :) > > > > Gwen > > > > On Tue, Dec 22, 2015 at 12:34 PM, Anna Povzner > wrote: > > > > > Hi, > > > > > > I am opening the voting thread for KIP-32: Add CreateTime and > > > LogAppendTime to Kafka message. > > > > > > For reference, here's the KIP wiki: > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-32+-+Add+CreateTime+and+LogAppendTime+to+Kafka+message > > > > > > And the mailing list threads: > > > > > > September: > > > > > > http://mail-archives.apache.org/mod_mbox/kafka-dev/201509.mbox/%3CCAHrRUm6NMg%3DPh4HAJdxr%3DpmZhfFcD5OEV2yxj3fg%2BXnEBTW%2B3w%40mail.gmail.com%3E > > > > > > October: > > > > > > http://mail-archives.apache.org/mod_mbox/kafka-dev/201510.mbox/%3CCAHrRUm7RiBAJxwO15s1tztz%3D15oibO-QJ%2B_w8AxafTnuw3jjCw%40mail.gmail.com%3E > > > > > > December: > > > > > > http://mail-archives.apache.org/mod_mbox/kafka-dev/201512.mbox/%3CCAHrRUm4ugxDYzyy26MGRGKpK4hsjT4EKTuu18M3wztYq4PE%3DaQ%40mail.gmail.com%3E > > > > > > > > > Thanks > > > Anna > > > > > > > > > -- > Thanks, > Neha >
Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.
Hey Gwen, Migration plan wasn't really discussed a ton in the previous threads. So it will be great to dive deep and see if there are gaps there. I had some questions, but the details listed on the KIP are great. It is complex, though the plan outlined in the wiki assumes a zero downtime upgrade assuming that producers and consumers can't be upgraded in tandem. This is typical for companies that have a significant Kafka footprint, like LinkedIn. Thanks, Neha On Tue, Dec 22, 2015 at 7:48 PM, Gwen Shapira wrote: > Hi Anna, > > Thanks for the KIP, especially for the details on all the alternatives and > how we arrived at the proposal. Its really great! > > Can you point me at where the migration plan was discussed? It looks overly > complex and I have a bunch of questions, but if there was a discussion, I'd > like to read up rather than repeating it :) > > Gwen > > On Tue, Dec 22, 2015 at 12:34 PM, Anna Povzner wrote: > > > Hi, > > > > I am opening the voting thread for KIP-32: Add CreateTime and > > LogAppendTime to Kafka message. > > > > For reference, here's the KIP wiki: > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-32+-+Add+CreateTime+and+LogAppendTime+to+Kafka+message > > > > And the mailing list threads: > > > > September: > > > http://mail-archives.apache.org/mod_mbox/kafka-dev/201509.mbox/%3CCAHrRUm6NMg%3DPh4HAJdxr%3DpmZhfFcD5OEV2yxj3fg%2BXnEBTW%2B3w%40mail.gmail.com%3E > > > > October: > > > http://mail-archives.apache.org/mod_mbox/kafka-dev/201510.mbox/%3CCAHrRUm7RiBAJxwO15s1tztz%3D15oibO-QJ%2B_w8AxafTnuw3jjCw%40mail.gmail.com%3E > > > > December: > > > http://mail-archives.apache.org/mod_mbox/kafka-dev/201512.mbox/%3CCAHrRUm4ugxDYzyy26MGRGKpK4hsjT4EKTuu18M3wztYq4PE%3DaQ%40mail.gmail.com%3E > > > > > > Thanks > > Anna > > > -- Thanks, Neha
Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.
Hi Anna, Thanks for the KIP, especially for the details on all the alternatives and how we arrived at the proposal. Its really great! Can you point me at where the migration plan was discussed? It looks overly complex and I have a bunch of questions, but if there was a discussion, I'd like to read up rather than repeating it :) Gwen On Tue, Dec 22, 2015 at 12:34 PM, Anna Povzner wrote: > Hi, > > I am opening the voting thread for KIP-32: Add CreateTime and > LogAppendTime to Kafka message. > > For reference, here's the KIP wiki: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-32+-+Add+CreateTime+and+LogAppendTime+to+Kafka+message > > And the mailing list threads: > > September: > http://mail-archives.apache.org/mod_mbox/kafka-dev/201509.mbox/%3CCAHrRUm6NMg%3DPh4HAJdxr%3DpmZhfFcD5OEV2yxj3fg%2BXnEBTW%2B3w%40mail.gmail.com%3E > > October: > http://mail-archives.apache.org/mod_mbox/kafka-dev/201510.mbox/%3CCAHrRUm7RiBAJxwO15s1tztz%3D15oibO-QJ%2B_w8AxafTnuw3jjCw%40mail.gmail.com%3E > > December: > http://mail-archives.apache.org/mod_mbox/kafka-dev/201512.mbox/%3CCAHrRUm4ugxDYzyy26MGRGKpK4hsjT4EKTuu18M3wztYq4PE%3DaQ%40mail.gmail.com%3E > > > Thanks > Anna >
Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.
Hey Anna, Thanks for initiating the vote and helping Becket while he is out! Hey Dong, My understanding from observing all the email threads is that there is a general agreement on the approach and the wiki is reasonably up-to-date with all the decisions. Is there a particular reason why the KIP process should wait? On Tue, Dec 22, 2015 at 2:15 PM, Dong Lin wrote: > Hi Anna, > > Have you talked with Jiangjie Qin before initiating the vote? Jiangjie Qin > is on vocation now and he may not have time to answer question related to > KIP-32 in detail. > > Dong > > On Tue, Dec 22, 2015 at 12:34 PM, Anna Povzner wrote: > > > Hi, > > > > I am opening the voting thread for KIP-32: Add CreateTime and > > LogAppendTime to Kafka message. > > > > For reference, here's the KIP wiki: > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-32+-+Add+CreateTime+and+LogAppendTime+to+Kafka+message > > > > And the mailing list threads: > > > > September: > > > http://mail-archives.apache.org/mod_mbox/kafka-dev/201509.mbox/%3CCAHrRUm6NMg%3DPh4HAJdxr%3DpmZhfFcD5OEV2yxj3fg%2BXnEBTW%2B3w%40mail.gmail.com%3E > > > > October: > > > http://mail-archives.apache.org/mod_mbox/kafka-dev/201510.mbox/%3CCAHrRUm7RiBAJxwO15s1tztz%3D15oibO-QJ%2B_w8AxafTnuw3jjCw%40mail.gmail.com%3E > > > > December: > > > http://mail-archives.apache.org/mod_mbox/kafka-dev/201512.mbox/%3CCAHrRUm4ugxDYzyy26MGRGKpK4hsjT4EKTuu18M3wztYq4PE%3DaQ%40mail.gmail.com%3E > > > > > > Thanks > > Anna > > > -- Thanks, Neha
Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.
Hi Anna, Have you talked with Jiangjie Qin before initiating the vote? Jiangjie Qin is on vocation now and he may not have time to answer question related to KIP-32 in detail. Dong On Tue, Dec 22, 2015 at 12:34 PM, Anna Povzner wrote: > Hi, > > I am opening the voting thread for KIP-32: Add CreateTime and > LogAppendTime to Kafka message. > > For reference, here's the KIP wiki: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-32+-+Add+CreateTime+and+LogAppendTime+to+Kafka+message > > And the mailing list threads: > > September: > http://mail-archives.apache.org/mod_mbox/kafka-dev/201509.mbox/%3CCAHrRUm6NMg%3DPh4HAJdxr%3DpmZhfFcD5OEV2yxj3fg%2BXnEBTW%2B3w%40mail.gmail.com%3E > > October: > http://mail-archives.apache.org/mod_mbox/kafka-dev/201510.mbox/%3CCAHrRUm7RiBAJxwO15s1tztz%3D15oibO-QJ%2B_w8AxafTnuw3jjCw%40mail.gmail.com%3E > > December: > http://mail-archives.apache.org/mod_mbox/kafka-dev/201512.mbox/%3CCAHrRUm4ugxDYzyy26MGRGKpK4hsjT4EKTuu18M3wztYq4PE%3DaQ%40mail.gmail.com%3E > > > Thanks > Anna >
Re: [VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.
One thing I want to do before +1 is to update the migration plan on the wiki regarding latest updates as well. Currently it states that in the second phase, the broker will "fill in the time field with current server time and re-compress the message" when it gets an older versioned produce request and "check and maybe overwrite the time field" when it gets a newer versioned produce request. Specifically, when the timestamp type is configured as "LogCreationTime" for that topic, while a broker received an old versioned produce request, should it fill in the time field with its own "log-append-timestamp", or should it fill it in with some value like "-1" indicating the timestamp is not available? Guozhang On Tue, Dec 22, 2015 at 12:34 PM, Anna Povzner wrote: > Hi, > > I am opening the voting thread for KIP-32: Add CreateTime and > LogAppendTime to Kafka message. > > For reference, here's the KIP wiki: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-32+-+Add+CreateTime+and+LogAppendTime+to+Kafka+message > > And the mailing list threads: > > September: > http://mail-archives.apache.org/mod_mbox/kafka-dev/201509.mbox/%3CCAHrRUm6NMg%3DPh4HAJdxr%3DpmZhfFcD5OEV2yxj3fg%2BXnEBTW%2B3w%40mail.gmail.com%3E > > October: > http://mail-archives.apache.org/mod_mbox/kafka-dev/201510.mbox/%3CCAHrRUm7RiBAJxwO15s1tztz%3D15oibO-QJ%2B_w8AxafTnuw3jjCw%40mail.gmail.com%3E > > December: > http://mail-archives.apache.org/mod_mbox/kafka-dev/201512.mbox/%3CCAHrRUm4ugxDYzyy26MGRGKpK4hsjT4EKTuu18M3wztYq4PE%3DaQ%40mail.gmail.com%3E > > > Thanks > Anna > -- -- Guozhang
[VOTE] KIP-32 Add CreateTime and LogAppendTime to Kafka message.
Hi, I am opening the voting thread for KIP-32: Add CreateTime and LogAppendTime to Kafka message. For reference, here's the KIP wiki: https://cwiki.apache.org/confluence/display/KAFKA/KIP-32+-+Add+CreateTime+and+LogAppendTime+to+Kafka+message And the mailing list threads: September: http://mail-archives.apache.org/mod_mbox/kafka-dev/201509.mbox/%3CCAHrRUm6NMg%3DPh4HAJdxr%3DpmZhfFcD5OEV2yxj3fg%2BXnEBTW%2B3w%40mail.gmail.com%3E October: http://mail-archives.apache.org/mod_mbox/kafka-dev/201510.mbox/%3CCAHrRUm7RiBAJxwO15s1tztz%3D15oibO-QJ%2B_w8AxafTnuw3jjCw%40mail.gmail.com%3E December: http://mail-archives.apache.org/mod_mbox/kafka-dev/201512.mbox/%3CCAHrRUm4ugxDYzyy26MGRGKpK4hsjT4EKTuu18M3wztYq4PE%3DaQ%40mail.gmail.com%3E Thanks Anna
