Re: [DISCUSS] KIP-462 : Use local thread id for KStreams

2019-04-30 Thread Boyang Chen 
Thanks Bill!

Get Outlook for iOS<https://aka.ms/o0ukef>

From: Bill Bejeck 
Sent: Tuesday, April 30, 2019 6:44:17 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-462 : Use local thread id for KStreams

Thanks for the KIP Boyang.  I have no additional comments from the ones
already presented.

+1(binding)

-Bill

On Tue, Apr 30, 2019 at 4:35 PM Boyang Chen  wrote:

> Thank you Guozhang!
>
> 
> From: Guozhang Wang 
> Sent: Wednesday, May 1, 2019 3:54 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-462 : Use local thread id for KStreams
>
> +1 (binding)
>
> Guozhang
>
> On 2019/04/26 07:42:12, "Matthias J. Sax"  wrote:
> > Thanks for the KIP!
> >
> > I agree that the change makes sense, and not only for the static group
> > membership case.
> >
> > For example, if a user `closes()` a `KafkaStreams` client and creates a
> > new one (for example to recover failed threads), while the JVM is still
> > running, it is more intuitive that the thread names are number from 1 to
> > X again, and not from X+1 to 2*x on restart.
> >
> > Also, the original idea about making thread names unique across
> > application is non-intuitive itself. It might make sense if there are
> > two instances of the same application within one JVM -- however, this
> > seems to be a rather rare case. Also, the only pattern for this use case
> > seems to by dynamic scaling, and I believe we should actually void this
> > pattern by adding a `stopThread()` and `addThread()` method to
> > `KafkaStreams` directly.
> >
> >
> > -Matthias
> >
> >
> > On 4/25/19 11:13 PM, Boyang Chen wrote:
> > > Hey friends,
> > >
> > > I would like to start discussion for a very small KIP:
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-462%3A+Use+local+thread+id+for+KStreams
> > >
> > > it is trying to avoid sharing thread-id increment between multiple
> stream instances configured in one JVM. This is an important fix for static
> membership<
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances>
> to be effective for KStreams in edge case like changing `group.instance.id`
> throughout restarts due to thread-id interleaving.
> > >
> > > I will open the vote thread in the main while, since this is a very
> small fix. Feel free to continue the discussion on this thread, thank you!
> > >
> > > Boyang
> > >
> >
> >
>

Re: [DISCUSS] KIP-462 : Use local thread id for KStreams

2019-04-30 Thread Bill Bejeck 
Thanks for the KIP Boyang.  I have no additional comments from the ones
already presented.

+1(binding)

-Bill

On Tue, Apr 30, 2019 at 4:35 PM Boyang Chen  wrote:

> Thank you Guozhang!
>
> 
> From: Guozhang Wang 
> Sent: Wednesday, May 1, 2019 3:54 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-462 : Use local thread id for KStreams
>
> +1 (binding)
>
> Guozhang
>
> On 2019/04/26 07:42:12, "Matthias J. Sax"  wrote:
> > Thanks for the KIP!
> >
> > I agree that the change makes sense, and not only for the static group
> > membership case.
> >
> > For example, if a user `closes()` a `KafkaStreams` client and creates a
> > new one (for example to recover failed threads), while the JVM is still
> > running, it is more intuitive that the thread names are number from 1 to
> > X again, and not from X+1 to 2*x on restart.
> >
> > Also, the original idea about making thread names unique across
> > application is non-intuitive itself. It might make sense if there are
> > two instances of the same application within one JVM -- however, this
> > seems to be a rather rare case. Also, the only pattern for this use case
> > seems to by dynamic scaling, and I believe we should actually void this
> > pattern by adding a `stopThread()` and `addThread()` method to
> > `KafkaStreams` directly.
> >
> >
> > -Matthias
> >
> >
> > On 4/25/19 11:13 PM, Boyang Chen wrote:
> > > Hey friends,
> > >
> > > I would like to start discussion for a very small KIP:
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-462%3A+Use+local+thread+id+for+KStreams
> > >
> > > it is trying to avoid sharing thread-id increment between multiple
> stream instances configured in one JVM. This is an important fix for static
> membership<
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances>
> to be effective for KStreams in edge case like changing `group.instance.id`
> throughout restarts due to thread-id interleaving.
> > >
> > > I will open the vote thread in the main while, since this is a very
> small fix. Feel free to continue the discussion on this thread, thank you!
> > >
> > > Boyang
> > >
> >
> >
>

Re: [DISCUSS] KIP-462 : Use local thread id for KStreams

2019-04-30 Thread Boyang Chen 
Thank you Guozhang!


From: Guozhang Wang 
Sent: Wednesday, May 1, 2019 3:54 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-462 : Use local thread id for KStreams

+1 (binding)

Guozhang

On 2019/04/26 07:42:12, "Matthias J. Sax"  wrote:
> Thanks for the KIP!
>
> I agree that the change makes sense, and not only for the static group
> membership case.
>
> For example, if a user `closes()` a `KafkaStreams` client and creates a
> new one (for example to recover failed threads), while the JVM is still
> running, it is more intuitive that the thread names are number from 1 to
> X again, and not from X+1 to 2*x on restart.
>
> Also, the original idea about making thread names unique across
> application is non-intuitive itself. It might make sense if there are
> two instances of the same application within one JVM -- however, this
> seems to be a rather rare case. Also, the only pattern for this use case
> seems to by dynamic scaling, and I believe we should actually void this
> pattern by adding a `stopThread()` and `addThread()` method to
> `KafkaStreams` directly.
>
>
> -Matthias
>
>
> On 4/25/19 11:13 PM, Boyang Chen wrote:
> > Hey friends,
> >
> > I would like to start discussion for a very small KIP:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-462%3A+Use+local+thread+id+for+KStreams
> >
> > it is trying to avoid sharing thread-id increment between multiple stream 
> > instances configured in one JVM. This is an important fix for static 
> > membership<https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances>
> >  to be effective for KStreams in edge case like changing 
> > `group.instance.id` throughout restarts due to thread-id interleaving.
> >
> > I will open the vote thread in the main while, since this is a very small 
> > fix. Feel free to continue the discussion on this thread, thank you!
> >
> > Boyang
> >
>
>

Re: [DISCUSS] KIP-462 : Use local thread id for KStreams

2019-04-30 Thread Guozhang Wang 
+1 (binding)

Guozhang

On 2019/04/26 07:42:12, "Matthias J. Sax"  wrote: 
> Thanks for the KIP!
> 
> I agree that the change makes sense, and not only for the static group
> membership case.
> 
> For example, if a user `closes()` a `KafkaStreams` client and creates a
> new one (for example to recover failed threads), while the JVM is still
> running, it is more intuitive that the thread names are number from 1 to
> X again, and not from X+1 to 2*x on restart.
> 
> Also, the original idea about making thread names unique across
> application is non-intuitive itself. It might make sense if there are
> two instances of the same application within one JVM -- however, this
> seems to be a rather rare case. Also, the only pattern for this use case
> seems to by dynamic scaling, and I believe we should actually void this
> pattern by adding a `stopThread()` and `addThread()` method to
> `KafkaStreams` directly.
> 
> 
> -Matthias
> 
> 
> On 4/25/19 11:13 PM, Boyang Chen wrote:
> > Hey friends,
> > 
> > I would like to start discussion for a very small KIP:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-462%3A+Use+local+thread+id+for+KStreams
> > 
> > it is trying to avoid sharing thread-id increment between multiple stream 
> > instances configured in one JVM. This is an important fix for static 
> > membership
> >  to be effective for KStreams in edge case like changing 
> > `group.instance.id` throughout restarts due to thread-id interleaving.
> > 
> > I will open the vote thread in the main while, since this is a very small 
> > fix. Feel free to continue the discussion on this thread, thank you!
> > 
> > Boyang
> > 
> 
>

Re: [DISCUSS] KIP-462 : Use local thread id for KStreams

2019-04-30 Thread Boyang Chen 
Thank you Sophie! Added the case Matthias described in the Compatibility 
session.


From: Sophie Blee-Goldman 
Sent: Wednesday, May 1, 2019 1:30 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-462 : Use local thread id for KStreams

Hey Boyang,

I think this sounds great but one thing you might want to update is the
"Compatibility, Deprecation, and Migration Plan" -- I agree having two
instances in the same JVM is probably a rare occurrence but the (presumably
less rare) situation Matthias described would also be affected in case of
exposed thread ids. Just a small note

Sophie

On Tue, Apr 30, 2019 at 8:25 AM Boyang Chen  wrote:

> Hey Bruno,
>
> "throttling purpose" means that we use `client.id` to track the request
> quota and do the throttling based on that. So in that context, it is
> expected to use same `client.id` for a certain set of consumers.
>
> Also thank you Guozhang for the comment! Merged with the KIP.
>
> Boyang
> 
> From: Bruno Cadonna 
> Sent: Tuesday, April 30, 2019 4:15 PM
> To: dev@kafka.apache.org
> Subject: Re: Fw: [DISCUSS] KIP-462 : Use local thread id for KStreams
>
> Hi Guozhang,
>
> What do you mean exactly with "throttling purposes"?
>
> @Boyang: Thank you for the KIP!
>
> Best,
> Bruno
>
> On Tue, Apr 30, 2019 at 1:15 AM Guozhang Wang  wrote:
>
> > Hi Boyang,
> >
> > Thanks for the KIP. I think it makes sense.
> >
> > Just following up on the documentation part: since we are effectively
> > removing this guard against same client.ids of instances --- and btw,
> > semantically we would not forbid users to set the same client.ids anyways
> > for throttling purposes for example --- it's worth augmenting the
> > client.id
> > config description by stating what users should expect client.id to be
> > propagated to internal embedded clients, and therefore what's the
> expected
> > outcome if they choose to set same client.ids for different Streams
> client.
> >
> >
> > Otherwise, I've no further comments.
> >
> > Guozhang
> >
> > On Mon, Apr 29, 2019 at 3:42 PM Boyang Chen  wrote:
> >
> > > FYI
> > >
> > >
> > > 
> > > From: Boyang Chen 
> > > Sent: Tuesday, April 30, 2019 4:32 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-462 : Use local thread id for KStreams
> > >
> > > Could we get more discussions on this thread?
> > >
> > > Boyang
> > >
> > > 
> > > From: Boyang Chen 
> > > Sent: Friday, April 26, 2019 10:51 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-462 : Use local thread id for KStreams
> > >
> > > Thanks for the explanation Matthias! Will enhance the KIP motivation by
> > > your example.
> > >
> > >
> > > 
> > > From: Matthias J. Sax 
> > > Sent: Friday, April 26, 2019 3:42 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-462 : Use local thread id for KStreams
> > >
> > > Thanks for the KIP!
> > >
> > > I agree that the change makes sense, and not only for the static group
> > > membership case.
> > >
> > > For example, if a user `closes()` a `KafkaStreams` client and creates a
> > > new one (for example to recover failed threads), while the JVM is still
> > > running, it is more intuitive that the thread names are number from 1
> to
> > > X again, and not from X+1 to 2*x on restart.
> > >
> > > Also, the original idea about making thread names unique across
> > > application is non-intuitive itself. It might make sense if there are
> > > two instances of the same application within one JVM -- however, this
> > > seems to be a rather rare case. Also, the only pattern for this use
> case
> > > seems to by dynamic scaling, and I believe we should actually void this
> > > pattern by adding a `stopThread()` and `addThread()` method to
> > > `KafkaStreams` directly.
> > >
> > >
> > > -Matthias
> > >
> > >
> > > On 4/25/19 11:13 PM, Boyang Chen wrote:
> > > > Hey friends,
> > > >
> > > > I would like to start discussion for a very small KIP:
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-462%3A+Use+local+thread+id+for+KStreams
> > > >
> > > > it is trying to avoid sharing thread-id increment between multiple
> > > stream instances configured in one JVM. This is an important fix for
> > static
> > > membership<
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances
> > >
> > > to be effective for KStreams in edge case like changing `
> > group.instance.id`
> > > throughout restarts due to thread-id interleaving.
> > > >
> > > > I will open the vote thread in the main while, since this is a very
> > > small fix. Feel free to continue the discussion on this thread, thank
> > you!
> > > >
> > > > Boyang
> > > >
> > >
> > >
> >
> > --
> > -- Guozhang
> >
>

Re: [DISCUSS] KIP-462 : Use local thread id for KStreams

2019-04-30 Thread Sophie Blee-Goldman 
Hey Boyang,

I think this sounds great but one thing you might want to update is the
"Compatibility, Deprecation, and Migration Plan" -- I agree having two
instances in the same JVM is probably a rare occurrence but the (presumably
less rare) situation Matthias described would also be affected in case of
exposed thread ids. Just a small note

Sophie

On Tue, Apr 30, 2019 at 8:25 AM Boyang Chen  wrote:

> Hey Bruno,
>
> "throttling purpose" means that we use `client.id` to track the request
> quota and do the throttling based on that. So in that context, it is
> expected to use same `client.id` for a certain set of consumers.
>
> Also thank you Guozhang for the comment! Merged with the KIP.
>
> Boyang
> 
> From: Bruno Cadonna 
> Sent: Tuesday, April 30, 2019 4:15 PM
> To: dev@kafka.apache.org
> Subject: Re: Fw: [DISCUSS] KIP-462 : Use local thread id for KStreams
>
> Hi Guozhang,
>
> What do you mean exactly with "throttling purposes"?
>
> @Boyang: Thank you for the KIP!
>
> Best,
> Bruno
>
> On Tue, Apr 30, 2019 at 1:15 AM Guozhang Wang  wrote:
>
> > Hi Boyang,
> >
> > Thanks for the KIP. I think it makes sense.
> >
> > Just following up on the documentation part: since we are effectively
> > removing this guard against same client.ids of instances --- and btw,
> > semantically we would not forbid users to set the same client.ids anyways
> > for throttling purposes for example --- it's worth augmenting the
> > client.id
> > config description by stating what users should expect client.id to be
> > propagated to internal embedded clients, and therefore what's the
> expected
> > outcome if they choose to set same client.ids for different Streams
> client.
> >
> >
> > Otherwise, I've no further comments.
> >
> > Guozhang
> >
> > On Mon, Apr 29, 2019 at 3:42 PM Boyang Chen  wrote:
> >
> > > FYI
> > >
> > >
> > > 
> > > From: Boyang Chen 
> > > Sent: Tuesday, April 30, 2019 4:32 AM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-462 : Use local thread id for KStreams
> > >
> > > Could we get more discussions on this thread?
> > >
> > > Boyang
> > >
> > > 
> > > From: Boyang Chen 
> > > Sent: Friday, April 26, 2019 10:51 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-462 : Use local thread id for KStreams
> > >
> > > Thanks for the explanation Matthias! Will enhance the KIP motivation by
> > > your example.
> > >
> > >
> > > 
> > > From: Matthias J. Sax 
> > > Sent: Friday, April 26, 2019 3:42 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-462 : Use local thread id for KStreams
> > >
> > > Thanks for the KIP!
> > >
> > > I agree that the change makes sense, and not only for the static group
> > > membership case.
> > >
> > > For example, if a user `closes()` a `KafkaStreams` client and creates a
> > > new one (for example to recover failed threads), while the JVM is still
> > > running, it is more intuitive that the thread names are number from 1
> to
> > > X again, and not from X+1 to 2*x on restart.
> > >
> > > Also, the original idea about making thread names unique across
> > > application is non-intuitive itself. It might make sense if there are
> > > two instances of the same application within one JVM -- however, this
> > > seems to be a rather rare case. Also, the only pattern for this use
> case
> > > seems to by dynamic scaling, and I believe we should actually void this
> > > pattern by adding a `stopThread()` and `addThread()` method to
> > > `KafkaStreams` directly.
> > >
> > >
> > > -Matthias
> > >
> > >
> > > On 4/25/19 11:13 PM, Boyang Chen wrote:
> > > > Hey friends,
> > > >
> > > > I would like to start discussion for a very small KIP:
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-462%3A+Use+local+thread+id+for+KStreams
> > > >
> > > > it is trying to avoid sharing thread-id increment between multiple
> > > stream instances configured in one JVM. This is an important fix for
> > static
> > > membership<
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances
> > >
> > > to be effective for KStreams in edge case like changing `
> > group.instance.id`
> > > throughout restarts due to thread-id interleaving.
> > > >
> > > > I will open the vote thread in the main while, since this is a very
> > > small fix. Feel free to continue the discussion on this thread, thank
> > you!
> > > >
> > > > Boyang
> > > >
> > >
> > >
> >
> > --
> > -- Guozhang
> >
>

Re: [DISCUSS] KIP-462 : Use local thread id for KStreams

2019-04-30 Thread Boyang Chen 
Hey Bruno,

"throttling purpose" means that we use `client.id` to track the request quota 
and do the throttling based on that. So in that context, it is expected to use 
same `client.id` for a certain set of consumers.

Also thank you Guozhang for the comment! Merged with the KIP.

Boyang

From: Bruno Cadonna 
Sent: Tuesday, April 30, 2019 4:15 PM
To: dev@kafka.apache.org
Subject: Re: Fw: [DISCUSS] KIP-462 : Use local thread id for KStreams

Hi Guozhang,

What do you mean exactly with "throttling purposes"?

@Boyang: Thank you for the KIP!

Best,
Bruno

On Tue, Apr 30, 2019 at 1:15 AM Guozhang Wang  wrote:

> Hi Boyang,
>
> Thanks for the KIP. I think it makes sense.
>
> Just following up on the documentation part: since we are effectively
> removing this guard against same client.ids of instances --- and btw,
> semantically we would not forbid users to set the same client.ids anyways
> for throttling purposes for example --- it's worth augmenting the
> client.id
> config description by stating what users should expect client.id to be
> propagated to internal embedded clients, and therefore what's the expected
> outcome if they choose to set same client.ids for different Streams client.
>
>
> Otherwise, I've no further comments.
>
> Guozhang
>
> On Mon, Apr 29, 2019 at 3:42 PM Boyang Chen  wrote:
>
> > FYI
> >
> >
> > 
> > From: Boyang Chen 
> > Sent: Tuesday, April 30, 2019 4:32 AM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-462 : Use local thread id for KStreams
> >
> > Could we get more discussions on this thread?
> >
> > Boyang
> >
> > ____________
> > From: Boyang Chen 
> > Sent: Friday, April 26, 2019 10:51 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-462 : Use local thread id for KStreams
> >
> > Thanks for the explanation Matthias! Will enhance the KIP motivation by
> > your example.
> >
> >
> > 
> > From: Matthias J. Sax 
> > Sent: Friday, April 26, 2019 3:42 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-462 : Use local thread id for KStreams
> >
> > Thanks for the KIP!
> >
> > I agree that the change makes sense, and not only for the static group
> > membership case.
> >
> > For example, if a user `closes()` a `KafkaStreams` client and creates a
> > new one (for example to recover failed threads), while the JVM is still
> > running, it is more intuitive that the thread names are number from 1 to
> > X again, and not from X+1 to 2*x on restart.
> >
> > Also, the original idea about making thread names unique across
> > application is non-intuitive itself. It might make sense if there are
> > two instances of the same application within one JVM -- however, this
> > seems to be a rather rare case. Also, the only pattern for this use case
> > seems to by dynamic scaling, and I believe we should actually void this
> > pattern by adding a `stopThread()` and `addThread()` method to
> > `KafkaStreams` directly.
> >
> >
> > -Matthias
> >
> >
> > On 4/25/19 11:13 PM, Boyang Chen wrote:
> > > Hey friends,
> > >
> > > I would like to start discussion for a very small KIP:
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-462%3A+Use+local+thread+id+for+KStreams
> > >
> > > it is trying to avoid sharing thread-id increment between multiple
> > stream instances configured in one JVM. This is an important fix for
> static
> > membership<
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances
> >
> > to be effective for KStreams in edge case like changing `
> group.instance.id`
> > throughout restarts due to thread-id interleaving.
> > >
> > > I will open the vote thread in the main while, since this is a very
> > small fix. Feel free to continue the discussion on this thread, thank
> you!
> > >
> > > Boyang
> > >
> >
> >
>
> --
> -- Guozhang
>

Re: [DISCUSS] KIP-462 : Use local thread id for KStreams

2019-04-29 Thread Boyang Chen 
Could we get more discussions on this thread?

Boyang


From: Boyang Chen 
Sent: Friday, April 26, 2019 10:51 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-462 : Use local thread id for KStreams

Thanks for the explanation Matthias! Will enhance the KIP motivation by your 
example.



From: Matthias J. Sax 
Sent: Friday, April 26, 2019 3:42 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-462 : Use local thread id for KStreams

Thanks for the KIP!

I agree that the change makes sense, and not only for the static group
membership case.

For example, if a user `closes()` a `KafkaStreams` client and creates a
new one (for example to recover failed threads), while the JVM is still
running, it is more intuitive that the thread names are number from 1 to
X again, and not from X+1 to 2*x on restart.

Also, the original idea about making thread names unique across
application is non-intuitive itself. It might make sense if there are
two instances of the same application within one JVM -- however, this
seems to be a rather rare case. Also, the only pattern for this use case
seems to by dynamic scaling, and I believe we should actually void this
pattern by adding a `stopThread()` and `addThread()` method to
`KafkaStreams` directly.


-Matthias


On 4/25/19 11:13 PM, Boyang Chen wrote:
> Hey friends,
>
> I would like to start discussion for a very small KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-462%3A+Use+local+thread+id+for+KStreams
>
> it is trying to avoid sharing thread-id increment between multiple stream 
> instances configured in one JVM. This is an important fix for static 
> membership<https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances>
>  to be effective for KStreams in edge case like changing `group.instance.id` 
> throughout restarts due to thread-id interleaving.
>
> I will open the vote thread in the main while, since this is a very small 
> fix. Feel free to continue the discussion on this thread, thank you!
>
> Boyang
>

Re: [DISCUSS] KIP-462 : Use local thread id for KStreams

2019-04-26 Thread Boyang Chen 
Thanks for the explanation Matthias! Will enhance the KIP motivation by your 
example.



From: Matthias J. Sax 
Sent: Friday, April 26, 2019 3:42 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-462 : Use local thread id for KStreams

Thanks for the KIP!

I agree that the change makes sense, and not only for the static group
membership case.

For example, if a user `closes()` a `KafkaStreams` client and creates a
new one (for example to recover failed threads), while the JVM is still
running, it is more intuitive that the thread names are number from 1 to
X again, and not from X+1 to 2*x on restart.

Also, the original idea about making thread names unique across
application is non-intuitive itself. It might make sense if there are
two instances of the same application within one JVM -- however, this
seems to be a rather rare case. Also, the only pattern for this use case
seems to by dynamic scaling, and I believe we should actually void this
pattern by adding a `stopThread()` and `addThread()` method to
`KafkaStreams` directly.


-Matthias


On 4/25/19 11:13 PM, Boyang Chen wrote:
> Hey friends,
>
> I would like to start discussion for a very small KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-462%3A+Use+local+thread+id+for+KStreams
>
> it is trying to avoid sharing thread-id increment between multiple stream 
> instances configured in one JVM. This is an important fix for static 
> membership<https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances>
>  to be effective for KStreams in edge case like changing `group.instance.id` 
> throughout restarts due to thread-id interleaving.
>
> I will open the vote thread in the main while, since this is a very small 
> fix. Feel free to continue the discussion on this thread, thank you!
>
> Boyang
>

Re: [DISCUSS] KIP-462 : Use local thread id for KStreams

2019-04-26 Thread Matthias J. Sax 
Thanks for the KIP!

I agree that the change makes sense, and not only for the static group
membership case.

For example, if a user `closes()` a `KafkaStreams` client and creates a
new one (for example to recover failed threads), while the JVM is still
running, it is more intuitive that the thread names are number from 1 to
X again, and not from X+1 to 2*x on restart.

Also, the original idea about making thread names unique across
application is non-intuitive itself. It might make sense if there are
two instances of the same application within one JVM -- however, this
seems to be a rather rare case. Also, the only pattern for this use case
seems to by dynamic scaling, and I believe we should actually void this
pattern by adding a `stopThread()` and `addThread()` method to
`KafkaStreams` directly.


-Matthias


On 4/25/19 11:13 PM, Boyang Chen wrote:
> Hey friends,
> 
> I would like to start discussion for a very small KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-462%3A+Use+local+thread+id+for+KStreams
> 
> it is trying to avoid sharing thread-id increment between multiple stream 
> instances configured in one JVM. This is an important fix for static 
> membership
>  to be effective for KStreams in edge case like changing `group.instance.id` 
> throughout restarts due to thread-id interleaving.
> 
> I will open the vote thread in the main while, since this is a very small 
> fix. Feel free to continue the discussion on this thread, thank you!
> 
> Boyang
> 



signature.asc
Description: OpenPGP digital signature