Re: [DISCUSS] KIP-262 Metadata should include the number of state stores for task

[Matthias J. Sax]

This is still open.

If you agree that we don't need it any longer, please update the wiki
pages accordingly and move the KIP into table "Discarded KIPs":
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals#KafkaImprovementProposals-DiscardedKIPs


-Mattias

On 1/3/19 10:41 PM, Richard Yu wrote:
> Hi Matthias,
> 
> I guess this is no longer necessary. I am open to anything honestly.
> I suppose we should close it (if its not already).
> 
> 
> On Fri, Oct 19, 2018 at 11:06 AM Matthias J. Sax 
> wrote:
> 
>> Any thought on my last email about discarding this KIP?
>>
>>
>> -Matthias
>>
>> On 9/14/18 11:44 AM, Matthias J. Sax wrote:
>>> Hi,
>>>
>>> we recently had a discussion on a different ticket to reduce the size of
>>> the metadata we need to send:
>>> https://issues.apache.org/jira/browse/KAFKA-7149
>>>
>>> It seems, that we actually don't need to include the number of stores in
>>> the metadata, but that we can compute the number of stores locally on
>>> each instance.
>>>
>>> With this insight, we should still try to exploit this knowledge during
>>> task assignment, however, this would be an internal change that does not
>>> require a KIP. Thus, I think that we can discard this KIP.
>>>
>>> Thoughts?
>>>
>>>
>>> -Matthias
>>>
>>> On 6/10/18 5:20 PM, Matthias J. Sax wrote:
 Richard,

 KIP-268 got merged and thus this KIP is unblocked.

 I just re-read it and think it needs some updates with regard to the
 upgrade path (ie, you should mention why upgrading is covered).

 It would also be useful to discuss how the store information is used
 during assignment. Atm, the KIP only discussed that the information
 should be added, but this is only half of the story from my point of
>> view.


 -Matthias

 On 3/22/18 9:15 PM, Matthias J. Sax wrote:
> Hi Richard,
>
> with KIP-268 in place (should be accepted soon) the upgrade path is
> covered. Thus, you can update your KIP accordingly, referring to
>> KIP-268.
>
> Can you also update your KIP similar to KIP-268 to cover the old and
>> new
> metadata format?
>
> Thanks!
>
> -Matthias
>
>
> On 2/24/18 4:07 PM, Richard Yu wrote:
>> I didn't really get what "upgrade strategy" was at the time that
>> Guozhang
>> mentioned it, so I wrote the above dialogue from my first
>> understanding. I
>> changed it to "upgrade strategy must be provided". Currently,
>> however, I do
>> not have anything in mind to facilitate upgrading older Kafka
>> brokers. If
>> you have anything in mind, please let me know.
>>
>>
>>
>>
>>
>>
>> On Sat, Feb 24, 2018 at 3:58 PM, Matthias J. Sax <
>> matth...@confluent.io>
>> wrote:
>>
>>> Thanks a lot for this KIP.
>>>
>>> I am not sure what you mean by
>>>
 which could potentially break older versions of Kafka brokers
>>>
>>> The metadata that is exchange, is not interpreted by the brokers. The
>>> problem with upgrading the metadata format affect only Kafka Streams
>>> instances.
>>>
>>> If we don't provide an upgrade strategy, changing the metadata format
>>> required to stop all running application instances, before the
>> instances
>>> can be restarted with the new code. However, this implies downtime
>> for
>>> an application and is thus not acceptable.
>>>
>>>
>>> -Matthias
>>>
>>>
>>> On 2/24/18 11:11 AM, Richard Yu wrote:
 Hi all,

 I would like to discuss a KIP I've submitted :
 https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>> 262%3A+Metadata+should+include+number+of+state+stores+for+task


 Regards,
 Richard Yu

>>>
>>>
>>
>

>>>
>>
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: [DISCUSS] KIP-262 Metadata should include the number of state stores for task

[Richard Yu]

Hi Matthias,

I guess this is no longer necessary. I am open to anything honestly.
I suppose we should close it (if its not already).


On Fri, Oct 19, 2018 at 11:06 AM Matthias J. Sax 
wrote:

> Any thought on my last email about discarding this KIP?
>
>
> -Matthias
>
> On 9/14/18 11:44 AM, Matthias J. Sax wrote:
> > Hi,
> >
> > we recently had a discussion on a different ticket to reduce the size of
> > the metadata we need to send:
> > https://issues.apache.org/jira/browse/KAFKA-7149
> >
> > It seems, that we actually don't need to include the number of stores in
> > the metadata, but that we can compute the number of stores locally on
> > each instance.
> >
> > With this insight, we should still try to exploit this knowledge during
> > task assignment, however, this would be an internal change that does not
> > require a KIP. Thus, I think that we can discard this KIP.
> >
> > Thoughts?
> >
> >
> > -Matthias
> >
> > On 6/10/18 5:20 PM, Matthias J. Sax wrote:
> >> Richard,
> >>
> >> KIP-268 got merged and thus this KIP is unblocked.
> >>
> >> I just re-read it and think it needs some updates with regard to the
> >> upgrade path (ie, you should mention why upgrading is covered).
> >>
> >> It would also be useful to discuss how the store information is used
> >> during assignment. Atm, the KIP only discussed that the information
> >> should be added, but this is only half of the story from my point of
> view.
> >>
> >>
> >> -Matthias
> >>
> >> On 3/22/18 9:15 PM, Matthias J. Sax wrote:
> >>> Hi Richard,
> >>>
> >>> with KIP-268 in place (should be accepted soon) the upgrade path is
> >>> covered. Thus, you can update your KIP accordingly, referring to
> KIP-268.
> >>>
> >>> Can you also update your KIP similar to KIP-268 to cover the old and
> new
> >>> metadata format?
> >>>
> >>> Thanks!
> >>>
> >>> -Matthias
> >>>
> >>>
> >>> On 2/24/18 4:07 PM, Richard Yu wrote:
>  I didn't really get what "upgrade strategy" was at the time that
> Guozhang
>  mentioned it, so I wrote the above dialogue from my first
> understanding. I
>  changed it to "upgrade strategy must be provided". Currently,
> however, I do
>  not have anything in mind to facilitate upgrading older Kafka
> brokers. If
>  you have anything in mind, please let me know.
> 
> 
> 
> 
> 
> 
>  On Sat, Feb 24, 2018 at 3:58 PM, Matthias J. Sax <
> matth...@confluent.io>
>  wrote:
> 
> > Thanks a lot for this KIP.
> >
> > I am not sure what you mean by
> >
> >> which could potentially break older versions of Kafka brokers
> >
> > The metadata that is exchange, is not interpreted by the brokers. The
> > problem with upgrading the metadata format affect only Kafka Streams
> > instances.
> >
> > If we don't provide an upgrade strategy, changing the metadata format
> > required to stop all running application instances, before the
> instances
> > can be restarted with the new code. However, this implies downtime
> for
> > an application and is thus not acceptable.
> >
> >
> > -Matthias
> >
> >
> > On 2/24/18 11:11 AM, Richard Yu wrote:
> >> Hi all,
> >>
> >> I would like to discuss a KIP I've submitted :
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 262%3A+Metadata+should+include+number+of+state+stores+for+task
> >>
> >>
> >> Regards,
> >> Richard Yu
> >>
> >
> >
> 
> >>>
> >>
> >
>
>

Re: [DISCUSS] KIP-262 Metadata should include the number of state stores for task

[Matthias J. Sax]

Any thought on my last email about discarding this KIP?


-Matthias

On 9/14/18 11:44 AM, Matthias J. Sax wrote:
> Hi,
> 
> we recently had a discussion on a different ticket to reduce the size of
> the metadata we need to send:
> https://issues.apache.org/jira/browse/KAFKA-7149
> 
> It seems, that we actually don't need to include the number of stores in
> the metadata, but that we can compute the number of stores locally on
> each instance.
> 
> With this insight, we should still try to exploit this knowledge during
> task assignment, however, this would be an internal change that does not
> require a KIP. Thus, I think that we can discard this KIP.
> 
> Thoughts?
> 
> 
> -Matthias
> 
> On 6/10/18 5:20 PM, Matthias J. Sax wrote:
>> Richard,
>>
>> KIP-268 got merged and thus this KIP is unblocked.
>>
>> I just re-read it and think it needs some updates with regard to the
>> upgrade path (ie, you should mention why upgrading is covered).
>>
>> It would also be useful to discuss how the store information is used
>> during assignment. Atm, the KIP only discussed that the information
>> should be added, but this is only half of the story from my point of view.
>>
>>
>> -Matthias
>>
>> On 3/22/18 9:15 PM, Matthias J. Sax wrote:
>>> Hi Richard,
>>>
>>> with KIP-268 in place (should be accepted soon) the upgrade path is
>>> covered. Thus, you can update your KIP accordingly, referring to KIP-268.
>>>
>>> Can you also update your KIP similar to KIP-268 to cover the old and new
>>> metadata format?
>>>
>>> Thanks!
>>>
>>> -Matthias
>>>
>>>
>>> On 2/24/18 4:07 PM, Richard Yu wrote:
 I didn't really get what "upgrade strategy" was at the time that Guozhang
 mentioned it, so I wrote the above dialogue from my first understanding. I
 changed it to "upgrade strategy must be provided". Currently, however, I do
 not have anything in mind to facilitate upgrading older Kafka brokers. If
 you have anything in mind, please let me know.






 On Sat, Feb 24, 2018 at 3:58 PM, Matthias J. Sax 
 wrote:

> Thanks a lot for this KIP.
>
> I am not sure what you mean by
>
>> which could potentially break older versions of Kafka brokers
>
> The metadata that is exchange, is not interpreted by the brokers. The
> problem with upgrading the metadata format affect only Kafka Streams
> instances.
>
> If we don't provide an upgrade strategy, changing the metadata format
> required to stop all running application instances, before the instances
> can be restarted with the new code. However, this implies downtime for
> an application and is thus not acceptable.
>
>
> -Matthias
>
>
> On 2/24/18 11:11 AM, Richard Yu wrote:
>> Hi all,
>>
>> I would like to discuss a KIP I've submitted :
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 262%3A+Metadata+should+include+number+of+state+stores+for+task
>>
>>
>> Regards,
>> Richard Yu
>>
>
>

>>>
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: [DISCUSS] KIP-262 Metadata should include the number of state stores for task

[Matthias J. Sax]

Hi,

we recently had a discussion on a different ticket to reduce the size of
the metadata we need to send:
https://issues.apache.org/jira/browse/KAFKA-7149

It seems, that we actually don't need to include the number of stores in
the metadata, but that we can compute the number of stores locally on
each instance.

With this insight, we should still try to exploit this knowledge during
task assignment, however, this would be an internal change that does not
require a KIP. Thus, I think that we can discard this KIP.

Thoughts?


-Matthias

On 6/10/18 5:20 PM, Matthias J. Sax wrote:
> Richard,
> 
> KIP-268 got merged and thus this KIP is unblocked.
> 
> I just re-read it and think it needs some updates with regard to the
> upgrade path (ie, you should mention why upgrading is covered).
> 
> It would also be useful to discuss how the store information is used
> during assignment. Atm, the KIP only discussed that the information
> should be added, but this is only half of the story from my point of view.
> 
> 
> -Matthias
> 
> On 3/22/18 9:15 PM, Matthias J. Sax wrote:
>> Hi Richard,
>>
>> with KIP-268 in place (should be accepted soon) the upgrade path is
>> covered. Thus, you can update your KIP accordingly, referring to KIP-268.
>>
>> Can you also update your KIP similar to KIP-268 to cover the old and new
>> metadata format?
>>
>> Thanks!
>>
>> -Matthias
>>
>>
>> On 2/24/18 4:07 PM, Richard Yu wrote:
>>> I didn't really get what "upgrade strategy" was at the time that Guozhang
>>> mentioned it, so I wrote the above dialogue from my first understanding. I
>>> changed it to "upgrade strategy must be provided". Currently, however, I do
>>> not have anything in mind to facilitate upgrading older Kafka brokers. If
>>> you have anything in mind, please let me know.
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Sat, Feb 24, 2018 at 3:58 PM, Matthias J. Sax 
>>> wrote:
>>>
 Thanks a lot for this KIP.

 I am not sure what you mean by

> which could potentially break older versions of Kafka brokers

 The metadata that is exchange, is not interpreted by the brokers. The
 problem with upgrading the metadata format affect only Kafka Streams
 instances.

 If we don't provide an upgrade strategy, changing the metadata format
 required to stop all running application instances, before the instances
 can be restarted with the new code. However, this implies downtime for
 an application and is thus not acceptable.


 -Matthias


 On 2/24/18 11:11 AM, Richard Yu wrote:
> Hi all,
>
> I would like to discuss a KIP I've submitted :
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
 262%3A+Metadata+should+include+number+of+state+stores+for+task
>
>
> Regards,
> Richard Yu
>


>>>
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: [DISCUSS] KIP-262 Metadata should include the number of state stores for task

[Matthias J. Sax]

Richard,

KIP-268 got merged and thus this KIP is unblocked.

I just re-read it and think it needs some updates with regard to the
upgrade path (ie, you should mention why upgrading is covered).

It would also be useful to discuss how the store information is used
during assignment. Atm, the KIP only discussed that the information
should be added, but this is only half of the story from my point of view.


-Matthias

On 3/22/18 9:15 PM, Matthias J. Sax wrote:
> Hi Richard,
> 
> with KIP-268 in place (should be accepted soon) the upgrade path is
> covered. Thus, you can update your KIP accordingly, referring to KIP-268.
> 
> Can you also update your KIP similar to KIP-268 to cover the old and new
> metadata format?
> 
> Thanks!
> 
> -Matthias
> 
> 
> On 2/24/18 4:07 PM, Richard Yu wrote:
>> I didn't really get what "upgrade strategy" was at the time that Guozhang
>> mentioned it, so I wrote the above dialogue from my first understanding. I
>> changed it to "upgrade strategy must be provided". Currently, however, I do
>> not have anything in mind to facilitate upgrading older Kafka brokers. If
>> you have anything in mind, please let me know.
>>
>>
>>
>>
>>
>>
>> On Sat, Feb 24, 2018 at 3:58 PM, Matthias J. Sax 
>> wrote:
>>
>>> Thanks a lot for this KIP.
>>>
>>> I am not sure what you mean by
>>>
 which could potentially break older versions of Kafka brokers
>>>
>>> The metadata that is exchange, is not interpreted by the brokers. The
>>> problem with upgrading the metadata format affect only Kafka Streams
>>> instances.
>>>
>>> If we don't provide an upgrade strategy, changing the metadata format
>>> required to stop all running application instances, before the instances
>>> can be restarted with the new code. However, this implies downtime for
>>> an application and is thus not acceptable.
>>>
>>>
>>> -Matthias
>>>
>>>
>>> On 2/24/18 11:11 AM, Richard Yu wrote:
 Hi all,

 I would like to discuss a KIP I've submitted :
 https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>> 262%3A+Metadata+should+include+number+of+state+stores+for+task


 Regards,
 Richard Yu

>>>
>>>
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: [DISCUSS] KIP-262 Metadata should include the number of state stores for task

[Richard Yu]

Hi Matthias,

Sorry for taking so long to get back to you.
I will change my KIP to also include the old format.

Thanks,
Richard

On Thu, Mar 22, 2018 at 9:15 PM, Matthias J. Sax 
wrote:

> Hi Richard,
>
> with KIP-268 in place (should be accepted soon) the upgrade path is
> covered. Thus, you can update your KIP accordingly, referring to KIP-268.
>
> Can you also update your KIP similar to KIP-268 to cover the old and new
> metadata format?
>
> Thanks!
>
> -Matthias
>
>
> On 2/24/18 4:07 PM, Richard Yu wrote:
> > I didn't really get what "upgrade strategy" was at the time that Guozhang
> > mentioned it, so I wrote the above dialogue from my first understanding.
> I
> > changed it to "upgrade strategy must be provided". Currently, however, I
> do
> > not have anything in mind to facilitate upgrading older Kafka brokers. If
> > you have anything in mind, please let me know.
> >
> >
> >
> >
> >
> >
> > On Sat, Feb 24, 2018 at 3:58 PM, Matthias J. Sax 
> > wrote:
> >
> >> Thanks a lot for this KIP.
> >>
> >> I am not sure what you mean by
> >>
> >>> which could potentially break older versions of Kafka brokers
> >>
> >> The metadata that is exchange, is not interpreted by the brokers. The
> >> problem with upgrading the metadata format affect only Kafka Streams
> >> instances.
> >>
> >> If we don't provide an upgrade strategy, changing the metadata format
> >> required to stop all running application instances, before the instances
> >> can be restarted with the new code. However, this implies downtime for
> >> an application and is thus not acceptable.
> >>
> >>
> >> -Matthias
> >>
> >>
> >> On 2/24/18 11:11 AM, Richard Yu wrote:
> >>> Hi all,
> >>>
> >>> I would like to discuss a KIP I've submitted :
> >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >> 262%3A+Metadata+should+include+number+of+state+stores+for+task
> >>>
> >>>
> >>> Regards,
> >>> Richard Yu
> >>>
> >>
> >>
> >
>
>

Re: [DISCUSS] KIP-262 Metadata should include the number of state stores for task

[Matthias J. Sax]

Hi Richard,

with KIP-268 in place (should be accepted soon) the upgrade path is
covered. Thus, you can update your KIP accordingly, referring to KIP-268.

Can you also update your KIP similar to KIP-268 to cover the old and new
metadata format?

Thanks!

-Matthias


On 2/24/18 4:07 PM, Richard Yu wrote:
> I didn't really get what "upgrade strategy" was at the time that Guozhang
> mentioned it, so I wrote the above dialogue from my first understanding. I
> changed it to "upgrade strategy must be provided". Currently, however, I do
> not have anything in mind to facilitate upgrading older Kafka brokers. If
> you have anything in mind, please let me know.
> 
> 
> 
> 
> 
> 
> On Sat, Feb 24, 2018 at 3:58 PM, Matthias J. Sax 
> wrote:
> 
>> Thanks a lot for this KIP.
>>
>> I am not sure what you mean by
>>
>>> which could potentially break older versions of Kafka brokers
>>
>> The metadata that is exchange, is not interpreted by the brokers. The
>> problem with upgrading the metadata format affect only Kafka Streams
>> instances.
>>
>> If we don't provide an upgrade strategy, changing the metadata format
>> required to stop all running application instances, before the instances
>> can be restarted with the new code. However, this implies downtime for
>> an application and is thus not acceptable.
>>
>>
>> -Matthias
>>
>>
>> On 2/24/18 11:11 AM, Richard Yu wrote:
>>> Hi all,
>>>
>>> I would like to discuss a KIP I've submitted :
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> 262%3A+Metadata+should+include+number+of+state+stores+for+task
>>>
>>>
>>> Regards,
>>> Richard Yu
>>>
>>
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: [DISCUSS] KIP-262 Metadata should include the number of state stores for task

[Richard Yu]

I didn't really get what "upgrade strategy" was at the time that Guozhang
mentioned it, so I wrote the above dialogue from my first understanding. I
changed it to "upgrade strategy must be provided". Currently, however, I do
not have anything in mind to facilitate upgrading older Kafka brokers. If
you have anything in mind, please let me know.






On Sat, Feb 24, 2018 at 3:58 PM, Matthias J. Sax 
wrote:

> Thanks a lot for this KIP.
>
> I am not sure what you mean by
>
> > which could potentially break older versions of Kafka brokers
>
> The metadata that is exchange, is not interpreted by the brokers. The
> problem with upgrading the metadata format affect only Kafka Streams
> instances.
>
> If we don't provide an upgrade strategy, changing the metadata format
> required to stop all running application instances, before the instances
> can be restarted with the new code. However, this implies downtime for
> an application and is thus not acceptable.
>
>
> -Matthias
>
>
> On 2/24/18 11:11 AM, Richard Yu wrote:
> > Hi all,
> >
> > I would like to discuss a KIP I've submitted :
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 262%3A+Metadata+should+include+number+of+state+stores+for+task
> >
> >
> > Regards,
> > Richard Yu
> >
>
>

Re: [DISCUSS] KIP-262 Metadata should include the number of state stores for task

[Matthias J. Sax]

Thanks a lot for this KIP.

I am not sure what you mean by

> which could potentially break older versions of Kafka brokers 

The metadata that is exchange, is not interpreted by the brokers. The
problem with upgrading the metadata format affect only Kafka Streams
instances.

If we don't provide an upgrade strategy, changing the metadata format
required to stop all running application instances, before the instances
can be restarted with the new code. However, this implies downtime for
an application and is thus not acceptable.


-Matthias


On 2/24/18 11:11 AM, Richard Yu wrote:
> Hi all,
> 
> I would like to discuss a KIP I've submitted :
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-262%3A+Metadata+should+include+number+of+state+stores+for+task
> 
> 
> Regards,
> Richard Yu
> 



signature.asc
Description: OpenPGP digital signature