[jira] [Comment Edited] (KAFKA-12675) Improve sticky general assignor scalability and performance
[
https://issues.apache.org/jira/browse/KAFKA-12675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17324358#comment-17324358
]
Travis Bischel edited comment on KAFKA-12675 at 4/19/21, 11:58 PM:
---
Great insight on getting rid of partition2AllPotentialConsumers, as well as
keeping some more things sorted! I was able to translate that into my own code
and dropped the large imbalance from 9.5s to 0.5s, as well as from 8.5G memory
util to 0.5G :)
I'll take a look at the code more in depth soon.
Edit: after further improvements I was able to get the large imbalance in my
client down to 220ms and 150MB.
was (Author: twmb):
Great insight on getting rid of partition2AllPotentialConsumers, as well as
keeping some more things sorted! I was able to translate that into my own code
and dropped the large imbalance from 9.5s to 0.5s, as well as from 8.5G memory
util to 0.5G :)
I'll take a look at the code more in depth soon.
Edit: after further improvements I was able to get the large imbalance down to
220ms and 171MB,
> Improve sticky general assignor scalability and performance
> ---
>
> Key: KAFKA-12675
> URL: https://issues.apache.org/jira/browse/KAFKA-12675
> Project: Kafka
> Issue Type: Improvement
>Reporter: Luke Chen
>Assignee: Luke Chen
>Priority: Major
>
> Currently, we have "general assignor" for non-equal subscription case and
> "constrained assignor" for all equal subscription case. There's a performance
> test for constrained assignor with:
> topicCount = {color:#ff}500{color};
> partitionCount = {color:#ff}2000{color};
> consumerCount = {color:#ff}2000{color};
> in _testLargeAssignmentAndGroupWithUniformSubscription,_ total 1 million
> partitions and we can complete the assignment within 2 second in my machine.
> However, if we let 1 of the consumer subscribe to only 1 topic, it'll use
> "general assignor", and the result with the same setting as above is:
> *OutOfMemory,*
> Even we down the count to:
> topicCount = {color:#ff}50{color};
> partitionCount = 1{color:#ff}000{color};
> consumerCount = 1{color:#ff}000{color};
> We still got *OutOfMemory*.
> With this setting:
> topicCount = {color:#ff}50{color};
> partitionCount = 8{color:#ff}00{color};
> consumerCount = 8{color:#ff}00{color};
> We can complete in 10 seconds in my machine, which is still slow.
>
> Since we are going to set default assignment strategy to
> "CooperativeStickyAssignor" soon, we should improve the scalability and
> performance for sticky general assignor.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (KAFKA-12675) Improve sticky general assignor scalability and performance
[
https://issues.apache.org/jira/browse/KAFKA-12675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17324358#comment-17324358
]
Travis Bischel edited comment on KAFKA-12675 at 4/19/21, 3:33 AM:
--
Great insight on getting rid of partition2AllPotentialConsumers, as well as
keeping some more things sorted! I was able to translate that into my own code
and dropped the large imbalance from 9.5s to 0.5s, as well as from 8.5G memory
util to 0.5G :)
I'll take a look at the code more in depth soon.
Edit: after further improvements I was able to get the large imbalance down to
220ms and 171MB,
was (Author: twmb):
Great insight on getting rid of partition2AllPotentialConsumers, as well as
keeping some more things sorted! I was able to translate that into my own code
and dropped the large imbalance from 9.5s to 0.5s, as well as from 8.5G memory
util to 0.5G :)
I'll take a look at the code more in depth soon.
> Improve sticky general assignor scalability and performance
> ---
>
> Key: KAFKA-12675
> URL: https://issues.apache.org/jira/browse/KAFKA-12675
> Project: Kafka
> Issue Type: Improvement
>Reporter: Luke Chen
>Assignee: Luke Chen
>Priority: Major
>
> Currently, we have "general assignor" for non-equal subscription case and
> "constrained assignor" for all equal subscription case. There's a performance
> test for constrained assignor with:
> topicCount = {color:#ff}500{color};
> partitionCount = {color:#ff}2000{color};
> consumerCount = {color:#ff}2000{color};
> in _testLargeAssignmentAndGroupWithUniformSubscription,_ total 1 million
> partitions and we can complete the assignment within 2 second in my machine.
> However, if we let 1 of the consumer subscribe to only 1 topic, it'll use
> "general assignor", and the result with the same setting as above is:
> *OutOfMemory,*
> Even we down the count to:
> topicCount = {color:#ff}50{color};
> partitionCount = 1{color:#ff}000{color};
> consumerCount = 1{color:#ff}000{color};
> We still got *OutOfMemory*.
> With this setting:
> topicCount = {color:#ff}50{color};
> partitionCount = 8{color:#ff}00{color};
> consumerCount = 8{color:#ff}00{color};
> We can complete in 10 seconds in my machine, which is still slow.
>
> Since we are going to set default assignment strategy to
> "CooperativeStickyAssignor" soon, we should improve the scalability and
> performance for sticky general assignor.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (KAFKA-12675) Improve sticky general assignor scalability and performance
[
https://issues.apache.org/jira/browse/KAFKA-12675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17322603#comment-17322603
]
Travis Bischel edited comment on KAFKA-12675 at 4/16/21, 4:25 AM:
--
What I mean to say is that the logic powering the existing sticky algorithm is
heuristic and not truly balanced, and that the logic itself can be changed to
be more exact to the sticky goals while being much more efficient. That is,
changes can be made for the imbalanced case similar to how [~ableegoldman] made
changes to the balanced case (for KAFKA-9987), and these changes will more
exactly fulfill the goal of sticky balancing while being more efficient. This
does not change how things are balanced / it does not change the actual sticky
aspect. Basically, improving the underlying algorithm for the imbalanced case
directly fulfills the goals of this ticket to improve the scalability and
performance.
For some numbers:
{noformat}
$ go test -run nothing -bench Java -benchmem -v -benchtime 60s
goos: darwin
goarch: amd64
pkg: github.com/twmb/franz-go/pkg/kgo/internal/sticky
cpu: Intel(R) Core(TM) i7-1068NG7 CPU @ 2.30GHz
BenchmarkJava
BenchmarkJava/large
sticky_test.go:1419: avg 2.696451608s per 1 balances of 2000 members and
100 total partitions
sticky_test.go:1419: avg 2.573368814s per 26 balances of 2000 members and
100 total partitions
BenchmarkJava/large-8 262573370133 ns/op
531478500 B/op 1038983 allocs/op
BenchmarkJava/large_imbalance
sticky_test.go:1419: avg 13.798672936s per 1 balances of 2001 members and
100 total partitions
sticky_test.go:1419: avg 9.581320518s per 4 balances of 2001 members and
100 total partitions
sticky_test.go:1419: avg 9.626729812s per 7 balances of 2001 members and
100 total partitions
BenchmarkJava/large_imbalance-879626739151
ns/op8535692965 B/op 1039100 allocs/op
BenchmarkJava/medium
sticky_test.go:1419: avg 77.798053ms per 1 balances of 1000 members and
5 total partitions
sticky_test.go:1419: avg 72.271454ms per 100 balances of 1000 members and
5 total partitions
sticky_test.go:1419: avg 72.044377ms per 996 balances of 1000 members and
5 total partitions
BenchmarkJava/medium-8 996 72044411
ns/op22502623 B/op 56085 allocs/op
BenchmarkJava/medium_imbalance
sticky_test.go:1419: avg 216.340842ms per 1 balances of 1001 members and
5 total partitions
sticky_test.go:1419: avg 217.385765ms per 100 balances of 1001 members and
5 total partitions
sticky_test.go:1419: avg 218.218478ms per 331 balances of 1001 members and
5 total partitions
BenchmarkJava/medium_imbalance-8 331 218218666
ns/op222795358 B/op 56097 allocs/op
BenchmarkJava/small
sticky_test.go:1419: avg 52.22238ms per 1 balances of 800 members and 4
total partitions
sticky_test.go:1419: avg 50.190192ms per 100 balances of 800 members and
4 total partitions
sticky_test.go:1419: avg 50.252975ms per 1434 balances of 800 members and
4 total partitions
BenchmarkJava/small-8 1434 50253022
ns/op18823337 B/op 44906 allocs/op
BenchmarkJava/small_imbalance
sticky_test.go:1419: avg 149.416236ms per 1 balances of 801 members and
4 total partitions
sticky_test.go:1419: avg 149.050743ms per 100 balances of 801 members and
4 total partitions
sticky_test.go:1419: avg 149.224721ms per 482 balances of 801 members and
4 total partitions
BenchmarkJava/small_imbalance-8 482 149224854
ns/op147060761 B/op 44914 allocs/op
{noformat}
I've just pushed the code for this benchmark in [this
commit|https://github.com/twmb/franz-go/commit/e0c960e094e8f100924411f6c5fb514b79fc761a].
This is still clearly using a decent amount of memory (up to 8G in the
imbalanced case), but I spent a good amount of time already optimizing how much
memory this can consume. I'm sure I can lose some speed to drop some memory
usage. But, as it stands, 1mil partitions and 2000 members just has to take up
some memory.
What I mean by heuristic is that the general assignor does not really have much
reasoning for what it is doing, it just somewhat tries a bunch of things and
then says "good enough" at a certain point. It also tries a bunch of things,
checks to see if what it did got a better balance score, and if not, discards
what it tried. This can be seen on lines 642 and 643 of
[AbstractStickyAssignor.java|https://github.com/apache/kafka/blob/637c44c976c115b7e770a6fd9e62e8822051b45b/clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractStickyAssignor.java#L642-L643]:
{noformat}
// if we are not preserving existi
[jira] [Comment Edited] (KAFKA-12675) Improve sticky general assignor scalability and performance
[
https://issues.apache.org/jira/browse/KAFKA-12675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17322603#comment-17322603
]
Travis Bischel edited comment on KAFKA-12675 at 4/16/21, 4:24 AM:
--
What I mean to say is that the logic powering the existing cooperative-sticky
algorithm is heuristic and not truly balanced, and that the logic itself can be
changed to be more exact to the cooperative-sticky goals while being much more
efficient. That is, changes can be made for the imbalanced case similar to how
[~ableegoldman] made changes to the balanced case (for KAFKA-9987), and these
changes will more exactly fulfill the goal of cooperative sticky while being
more efficient. This does not change how things are balanced / it does not
change the actual sticky aspect. Basically, improving the underlying algorithm
for the imbalanced case directly fulfills the goals of this ticket to improve
the scalability and performance.
For some numbers:
{noformat}
$ go test -run nothing -bench Java -benchmem -v -benchtime 60s
goos: darwin
goarch: amd64
pkg: github.com/twmb/franz-go/pkg/kgo/internal/sticky
cpu: Intel(R) Core(TM) i7-1068NG7 CPU @ 2.30GHz
BenchmarkJava
BenchmarkJava/large
sticky_test.go:1419: avg 2.696451608s per 1 balances of 2000 members and
100 total partitions
sticky_test.go:1419: avg 2.573368814s per 26 balances of 2000 members and
100 total partitions
BenchmarkJava/large-8 262573370133 ns/op
531478500 B/op 1038983 allocs/op
BenchmarkJava/large_imbalance
sticky_test.go:1419: avg 13.798672936s per 1 balances of 2001 members and
100 total partitions
sticky_test.go:1419: avg 9.581320518s per 4 balances of 2001 members and
100 total partitions
sticky_test.go:1419: avg 9.626729812s per 7 balances of 2001 members and
100 total partitions
BenchmarkJava/large_imbalance-879626739151
ns/op8535692965 B/op 1039100 allocs/op
BenchmarkJava/medium
sticky_test.go:1419: avg 77.798053ms per 1 balances of 1000 members and
5 total partitions
sticky_test.go:1419: avg 72.271454ms per 100 balances of 1000 members and
5 total partitions
sticky_test.go:1419: avg 72.044377ms per 996 balances of 1000 members and
5 total partitions
BenchmarkJava/medium-8 996 72044411
ns/op22502623 B/op 56085 allocs/op
BenchmarkJava/medium_imbalance
sticky_test.go:1419: avg 216.340842ms per 1 balances of 1001 members and
5 total partitions
sticky_test.go:1419: avg 217.385765ms per 100 balances of 1001 members and
5 total partitions
sticky_test.go:1419: avg 218.218478ms per 331 balances of 1001 members and
5 total partitions
BenchmarkJava/medium_imbalance-8 331 218218666
ns/op222795358 B/op 56097 allocs/op
BenchmarkJava/small
sticky_test.go:1419: avg 52.22238ms per 1 balances of 800 members and 4
total partitions
sticky_test.go:1419: avg 50.190192ms per 100 balances of 800 members and
4 total partitions
sticky_test.go:1419: avg 50.252975ms per 1434 balances of 800 members and
4 total partitions
BenchmarkJava/small-8 1434 50253022
ns/op18823337 B/op 44906 allocs/op
BenchmarkJava/small_imbalance
sticky_test.go:1419: avg 149.416236ms per 1 balances of 801 members and
4 total partitions
sticky_test.go:1419: avg 149.050743ms per 100 balances of 801 members and
4 total partitions
sticky_test.go:1419: avg 149.224721ms per 482 balances of 801 members and
4 total partitions
BenchmarkJava/small_imbalance-8 482 149224854
ns/op147060761 B/op 44914 allocs/op
{noformat}
I've just pushed the code for this benchmark in [this
commit|https://github.com/twmb/franz-go/commit/e0c960e094e8f100924411f6c5fb514b79fc761a].
This is still clearly using a decent amount of memory (up to 8G in the
imbalanced case), but I spent a good amount of time already optimizing how much
memory this can consume. I'm sure I can lose some speed to drop some memory
usage. But, as it stands, 1mil partitions and 2000 members just has to take up
some memory.
What I mean by heuristic is that the general assignor does not really have much
reasoning for what it is doing, it just somewhat tries a bunch of things and
then says "good enough" at a certain point. It also tries a bunch of things,
checks to see if what it did got a better balance score, and if not, discards
what it tried. This can be seen on lines 642 and 643 of
[AbstractStickyAssignor.java|https://github.com/apache/kafka/blob/637c44c976c115b7e770a6fd9e62e8822051b45b/clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractStickyAssignor.java#L642-L643]:
{noformat}
// if we
[jira] [Comment Edited] (KAFKA-12675) Improve sticky general assignor scalability and performance
[
https://issues.apache.org/jira/browse/KAFKA-12675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17322603#comment-17322603
]
Travis Bischel edited comment on KAFKA-12675 at 4/16/21, 4:10 AM:
--
What I mean to say is that the logic powering the existing cooperative-sticky
algorithm is heuristic and not truly balanced, and that the logic itself can be
changed to be more exact to the cooperative-sticky goals while being much more
efficient. That is, changes can be made for the imbalanced case similar to how
[~ableegoldman] made changes to the balanced case (for KAFKA-9987), and these
changes will more exactly fulfill the goal of cooperative sticky while being
more efficient. This does not change how things are balanced / it does not
change the actual sticky aspect. Basically, improving the underlying algorithm
for the imbalanced case directly fulfills the goals of this ticket to improve
the scalability and performance. I'll edit this comment shortly with some
benchmarking numbers.
was (Author: twmb):
What I mean to say is that the logic powering the existing cooperative-sticky
algorithm is heuristic and not truly balanced, and that the logic itself can be
changed to be more exact to the cooperative-sticky goals while being much more
efficient. That is, changes can be made for the imbalanced case similar to how
[~ableegoldman] made changes to the balanced case, and these changes will more
exactly fulfill the goal of cooperative sticky while being more efficient. This
does not change how things are balanced / it does not change the actual sticky
aspect. Basically, improving the underlying algorithm for the imbalanced case
directly fulfills the goals of this ticket to improve the scalability and
performance. I'll edit this comment shortly with some benchmarking numbers.
> Improve sticky general assignor scalability and performance
> ---
>
> Key: KAFKA-12675
> URL: https://issues.apache.org/jira/browse/KAFKA-12675
> Project: Kafka
> Issue Type: Improvement
>Reporter: Luke Chen
>Assignee: Luke Chen
>Priority: Major
>
> Currently, we have "general assignor" for non-equal subscription case and
> "constrained assignor" for all equal subscription case. There's a performance
> test for constrained assignor with:
> topicCount = {color:#ff}500{color};
> partitionCount = {color:#ff}2000{color};
> consumerCount = {color:#ff}2000{color};
> in _testLargeAssignmentAndGroupWithUniformSubscription,_ total 1 million
> partitions and we can complete the assignment within 2 second in my machine.
> However, if we let 1 of the consumer subscribe to only 1 topic, it'll use
> "general assignor", and the result with the same setting as above is:
> *OutOfMemory,*
> Even we down the count to:
> topicCount = {color:#ff}50{color};
> partitionCount = 1{color:#ff}000{color};
> consumerCount = 1{color:#ff}000{color};
> We still got *OutOfMemory*.
> With this setting:
> topicCount = {color:#ff}50{color};
> partitionCount = 8{color:#ff}00{color};
> consumerCount = 8{color:#ff}00{color};
> We can complete in 10 seconds in my machine, which is still slow.
>
> Since we are going to set default assignment strategy to
> "CooperativeStickyAssignor" soon, we should improve the scalability and
> performance for sticky general assignor.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (KAFKA-12675) Improve sticky general assignor scalability and performance
[
https://issues.apache.org/jira/browse/KAFKA-12675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17322599#comment-17322599
]
Luke Chen edited comment on KAFKA-12675 at 4/16/21, 3:59 AM:
-
[~twmb], thanks for suggestion. But in this ticket, I'd like to focus on
improving the scalability and performance first. I agree the whole algorithm
can be improved, but that needs to go through KIP and have more discussion to
be able to go on. Thanks anyway, I'll think about it. :)
was (Author: showuon):
[~twmb], thanks for suggestion. But in this ticket, I'd like to improve the
scalability and performance first. I agree the whole algorithm can be improved,
too, but that needs to go through KIP and have more discussion to be able to go
on. Thanks anyway, I'll think about it. :)
> Improve sticky general assignor scalability and performance
> ---
>
> Key: KAFKA-12675
> URL: https://issues.apache.org/jira/browse/KAFKA-12675
> Project: Kafka
> Issue Type: Improvement
>Reporter: Luke Chen
>Assignee: Luke Chen
>Priority: Major
>
> Currently, we have "general assignor" for non-equal subscription case and
> "constrained assignor" for all equal subscription case. There's a performance
> test for constrained assignor with:
> topicCount = {color:#ff}500{color};
> partitionCount = {color:#ff}2000{color};
> consumerCount = {color:#ff}2000{color};
> in _testLargeAssignmentAndGroupWithUniformSubscription,_ total 1 million
> partitions and we can complete the assignment within 2 second in my machine.
> However, if we let 1 of the consumer subscribe to only 1 topic, it'll use
> "general assignor", and the result with the same setting as above is:
> *OutOfMemory,*
> Even we down the count to:
> topicCount = {color:#ff}50{color};
> partitionCount = 1{color:#ff}000{color};
> consumerCount = 1{color:#ff}000{color};
> We still got *OutOfMemory*.
> With this setting:
> topicCount = {color:#ff}50{color};
> partitionCount = 8{color:#ff}00{color};
> consumerCount = 8{color:#ff}00{color};
> We can complete in 10 seconds in my machine, which is still slow.
>
> Since we are going to set default assignment strategy to
> "CooperativeStickyAssignor" soon, we should improve the scalability and
> performance for sticky general assignor.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
