[jira] [Commented] (KAFKA-687) Rebalance algorithm should consider partitions from all topics
[ https://issues.apache.org/jira/browse/KAFKA-687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14102664#comment-14102664 ] Joel Koshy commented on KAFKA-687: -- Updated reviewboard https://reviews.apache.org/r/23655/diff/ against branch origin/trunk > Rebalance algorithm should consider partitions from all topics > -- > > Key: KAFKA-687 > URL: https://issues.apache.org/jira/browse/KAFKA-687 > Project: Kafka > Issue Type: Improvement >Affects Versions: 0.9.0 >Reporter: Pablo Barrera >Assignee: Joel Koshy > Attachments: KAFKA-687.patch, KAFKA-687_2014-07-18_15:55:15.patch, > KAFKA-687_2014-08-19_12:07:37.patch > > > The current rebalance step, as stated in the original Kafka paper [1], splits > the partitions per topic between all the consumers. So if you have 100 topics > with 2 partitions each and 10 consumers only two consumers will be used. That > is, for each topic all partitions will be listed and shared between the > consumers in the consumer group in order (not randomly). > If the consumer group is reading from several topics at the same time it > makes sense to split all the partitions from all topics between all the > consumer. Following the example, we will have 200 partitions in total, 20 per > consumer, using the 10 consumers. > The load per topic could be different and the division should consider this. > However even a random division should be better than the current algorithm > while reading from several topics and should harm reading from a few topics > with several partitions. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-687) Rebalance algorithm should consider partitions from all topics
[ https://issues.apache.org/jira/browse/KAFKA-687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14104897#comment-14104897 ] Joel Koshy commented on KAFKA-687: -- Updated reviewboard https://reviews.apache.org/r/23655/ against branch origin/trunk > Rebalance algorithm should consider partitions from all topics > -- > > Key: KAFKA-687 > URL: https://issues.apache.org/jira/browse/KAFKA-687 > Project: Kafka > Issue Type: Improvement >Affects Versions: 0.9.0 >Reporter: Pablo Barrera >Assignee: Joel Koshy > Attachments: KAFKA-687.patch, KAFKA-687_2014-07-18_15:55:15.patch, > KAFKA-687_2014-08-19_12:07:37.patch, KAFKA-687_2014-08-20_18:09:28.patch > > > The current rebalance step, as stated in the original Kafka paper [1], splits > the partitions per topic between all the consumers. So if you have 100 topics > with 2 partitions each and 10 consumers only two consumers will be used. That > is, for each topic all partitions will be listed and shared between the > consumers in the consumer group in order (not randomly). > If the consumer group is reading from several topics at the same time it > makes sense to split all the partitions from all topics between all the > consumer. Following the example, we will have 200 partitions in total, 20 per > consumer, using the 10 consumers. > The load per topic could be different and the division should consider this. > However even a random division should be better than the current algorithm > while reading from several topics and should harm reading from a few topics > with several partitions. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-687) Rebalance algorithm should consider partitions from all topics
[ https://issues.apache.org/jira/browse/KAFKA-687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14109615#comment-14109615 ] Joel Koshy commented on KAFKA-687: -- Updated reviewboard https://reviews.apache.org/r/23655/ against branch origin/trunk > Rebalance algorithm should consider partitions from all topics > -- > > Key: KAFKA-687 > URL: https://issues.apache.org/jira/browse/KAFKA-687 > Project: Kafka > Issue Type: Improvement >Affects Versions: 0.9.0 >Reporter: Pablo Barrera >Assignee: Joel Koshy > Attachments: KAFKA-687.patch, KAFKA-687_2014-07-18_15:55:15.patch, > KAFKA-687_2014-08-19_12:07:37.patch, KAFKA-687_2014-08-20_18:09:28.patch, > KAFKA-687_2014-08-25_12:36:48.patch > > > The current rebalance step, as stated in the original Kafka paper [1], splits > the partitions per topic between all the consumers. So if you have 100 topics > with 2 partitions each and 10 consumers only two consumers will be used. That > is, for each topic all partitions will be listed and shared between the > consumers in the consumer group in order (not randomly). > If the consumer group is reading from several topics at the same time it > makes sense to split all the partitions from all topics between all the > consumer. Following the example, we will have 200 partitions in total, 20 per > consumer, using the 10 consumers. > The load per topic could be different and the division should consider this. > However even a random division should be better than the current algorithm > while reading from several topics and should harm reading from a few topics > with several partitions. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-687) Rebalance algorithm should consider partitions from all topics
[ https://issues.apache.org/jira/browse/KAFKA-687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114578#comment-14114578 ] Joel Koshy commented on KAFKA-687: -- Updated reviewboard https://reviews.apache.org/r/23655/ against branch origin/trunk > Rebalance algorithm should consider partitions from all topics > -- > > Key: KAFKA-687 > URL: https://issues.apache.org/jira/browse/KAFKA-687 > Project: Kafka > Issue Type: Improvement >Affects Versions: 0.9.0 >Reporter: Pablo Barrera >Assignee: Joel Koshy > Attachments: KAFKA-687.patch, KAFKA-687_2014-07-18_15:55:15.patch, > KAFKA-687_2014-08-19_12:07:37.patch, KAFKA-687_2014-08-20_18:09:28.patch, > KAFKA-687_2014-08-25_12:36:48.patch, KAFKA-687_2014-08-28_16:20:25.patch > > > The current rebalance step, as stated in the original Kafka paper [1], splits > the partitions per topic between all the consumers. So if you have 100 topics > with 2 partitions each and 10 consumers only two consumers will be used. That > is, for each topic all partitions will be listed and shared between the > consumers in the consumer group in order (not randomly). > If the consumer group is reading from several topics at the same time it > makes sense to split all the partitions from all topics between all the > consumer. Following the example, we will have 200 partitions in total, 20 per > consumer, using the 10 consumers. > The load per topic could be different and the division should consider this. > However even a random division should be better than the current algorithm > while reading from several topics and should harm reading from a few topics > with several partitions. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-687) Rebalance algorithm should consider partitions from all topics
[ https://issues.apache.org/jira/browse/KAFKA-687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14037620#comment-14037620 ] Joel Koshy commented on KAFKA-687: -- Although we are working on the new consumer, some people were interested (offline) in getting some form of this done in the current consumer at least for wildcard consumption. It should (in theory) be simple to do, but may take a couple days because the wildcard consumption part of the code is convoluted - mainly because when it was done we did not want to modify the existing consumer too much. Anyway, I dumped some thoughts in the comments of this gist: https://gist.github.com/jjkoshy/5c3d065161153b7b1ee3 and the unit test at the end provides one possible partition layout strategy. > Rebalance algorithm should consider partitions from all topics > -- > > Key: KAFKA-687 > URL: https://issues.apache.org/jira/browse/KAFKA-687 > Project: Kafka > Issue Type: Improvement >Affects Versions: 0.9.0 >Reporter: Pablo Barrera > > The current rebalance step, as stated in the original Kafka paper [1], splits > the partitions per topic between all the consumers. So if you have 100 topics > with 2 partitions each and 10 consumers only two consumers will be used. That > is, for each topic all partitions will be listed and shared between the > consumers in the consumer group in order (not randomly). > If the consumer group is reading from several topics at the same time it > makes sense to split all the partitions from all topics between all the > consumer. Following the example, we will have 200 partitions in total, 20 per > consumer, using the 10 consumers. > The load per topic could be different and the division should consider this. > However even a random division should be better than the current algorithm > while reading from several topics and should harm reading from a few topics > with several partitions. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-687) Rebalance algorithm should consider partitions from all topics
[ https://issues.apache.org/jira/browse/KAFKA-687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14065446#comment-14065446 ] Joel Koshy commented on KAFKA-687: -- Created reviewboard https://reviews.apache.org/r/23655/diff/ against branch origin/trunk > Rebalance algorithm should consider partitions from all topics > -- > > Key: KAFKA-687 > URL: https://issues.apache.org/jira/browse/KAFKA-687 > Project: Kafka > Issue Type: Improvement >Affects Versions: 0.9.0 >Reporter: Pablo Barrera >Assignee: Sriharsha Chintalapani > Attachments: KAFKA-687.patch > > > The current rebalance step, as stated in the original Kafka paper [1], splits > the partitions per topic between all the consumers. So if you have 100 topics > with 2 partitions each and 10 consumers only two consumers will be used. That > is, for each topic all partitions will be listed and shared between the > consumers in the consumer group in order (not randomly). > If the consumer group is reading from several topics at the same time it > makes sense to split all the partitions from all topics between all the > consumer. Following the example, we will have 200 partitions in total, 20 per > consumer, using the 10 consumers. > The load per topic could be different and the division should consider this. > However even a random division should be better than the current algorithm > while reading from several topics and should harm reading from a few topics > with several partitions. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-687) Rebalance algorithm should consider partitions from all topics
[ https://issues.apache.org/jira/browse/KAFKA-687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14065457#comment-14065457 ] Joel Koshy commented on KAFKA-687: -- I ended up abandoning the earlier approach I was thinking of in the above gist and went with I think a simpler approach. The layout algorithms are the result of discussions with [~clarkhaskins] > Rebalance algorithm should consider partitions from all topics > -- > > Key: KAFKA-687 > URL: https://issues.apache.org/jira/browse/KAFKA-687 > Project: Kafka > Issue Type: Improvement >Affects Versions: 0.9.0 >Reporter: Pablo Barrera >Assignee: Sriharsha Chintalapani > Attachments: KAFKA-687.patch > > > The current rebalance step, as stated in the original Kafka paper [1], splits > the partitions per topic between all the consumers. So if you have 100 topics > with 2 partitions each and 10 consumers only two consumers will be used. That > is, for each topic all partitions will be listed and shared between the > consumers in the consumer group in order (not randomly). > If the consumer group is reading from several topics at the same time it > makes sense to split all the partitions from all topics between all the > consumer. Following the example, we will have 200 partitions in total, 20 per > consumer, using the 10 consumers. > The load per topic could be different and the division should consider this. > However even a random division should be better than the current algorithm > while reading from several topics and should harm reading from a few topics > with several partitions. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-687) Rebalance algorithm should consider partitions from all topics
[ https://issues.apache.org/jira/browse/KAFKA-687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14067051#comment-14067051 ] Joel Koshy commented on KAFKA-687: -- Updated reviewboard https://reviews.apache.org/r/23655/diff/ against branch origin/trunk > Rebalance algorithm should consider partitions from all topics > -- > > Key: KAFKA-687 > URL: https://issues.apache.org/jira/browse/KAFKA-687 > Project: Kafka > Issue Type: Improvement >Affects Versions: 0.9.0 >Reporter: Pablo Barrera >Assignee: Joel Koshy > Attachments: KAFKA-687.patch, KAFKA-687_2014-07-18_15:55:15.patch > > > The current rebalance step, as stated in the original Kafka paper [1], splits > the partitions per topic between all the consumers. So if you have 100 topics > with 2 partitions each and 10 consumers only two consumers will be used. That > is, for each topic all partitions will be listed and shared between the > consumers in the consumer group in order (not randomly). > If the consumer group is reading from several topics at the same time it > makes sense to split all the partitions from all topics between all the > consumer. Following the example, we will have 200 partitions in total, 20 per > consumer, using the 10 consumers. > The load per topic could be different and the division should consider this. > However even a random division should be better than the current algorithm > while reading from several topics and should harm reading from a few topics > with several partitions. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-687) Rebalance algorithm should consider partitions from all topics
[
https://issues.apache.org/jira/browse/KAFKA-687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14077480#comment-14077480
]
Joel Koshy commented on KAFKA-687:
--
Short update on this:
After the initial review comments, I was trying to make the allocation module
more generic so we can reuse it in the new consumer. Furthermore, I was trying
to get rid of the "symmetric" mode (which is for wildcards only and with
identical subscriptions across all consumers) and make "roundrobin" more
general. The basic approach was to sort the consumer IDs based on a hash of the
consumerID with the topic appended to it - effectively scrambling (in a
consistent order) the list of consumer streams available for a given topic -
and then doing a round-robin assignment across available partitions of the
topic. This did not actually work as well as expected. Here is the output of
some simulations:
{code}
[2014-07-25 17:00:35,559] INFO Owned count summary for 6284 partitions
across 63 consumer ids (9 consumers with 7 streams): min: 8.00; max:
200.00; avg: 99.746032; stddev: 58.871914; ideal: 99.746033
(unit.kafka.consumer.PartitionAllocatorTest:68)
[2014-07-25 17:00:36,791] INFO Owned count summary for 6118 partitions
across 42 consumer ids (7 consumers with 6 streams): min: 57.00; max:
254.00; avg: 145.67; stddev: 60.954468; ideal: 145.72
(unit.kafka.consumer.PartitionAllocatorTest:68)
[2014-07-25 17:00:38,065] INFO Owned count summary for 10652 partitions
across 88 consumer ids (11 consumers with 8 streams): min: 4.00; max:
335.00; avg: 169.079365; stddev: 101.093266; ideal: 121.045456
(unit.kafka.consumer.PartitionAllocatorTest:68)
[2014-07-25 17:02:07,198] INFO Owned count summary for 10839 partitions
across 200 consumer ids (20 consumers with 10 streams): min: 3.00; max:
330.00; avg: 172.047619; stddev: 99.267223; ideal: 54.195000
(unit.kafka.consumer.PartitionAllocatorTest:68)
[2014-07-25 17:24:35,676] INFO Owned count summary for 6439 partitions
across 12 consumer ids (2 consumers with 6 streams): min: 445.00; max:
626.00; avg: 536.58; stddev: 58.445714; ideal: 536.583313
(unit.kafka.consumer.PartitionAllocatorTest:68)
[2014-07-25 17:24:36,787] INFO Owned count summary for 11777 partitions
across 63 consumer ids (7 consumers with 9 streams): min: 5.00; max:
369.00; avg: 186.936508; stddev: 113.972531; ideal: 186.936508
(unit.kafka.consumer.PartitionAllocatorTest:68)
[2014-07-25 17:25:20,108] INFO Owned count summary for 10488 partitions
across 144 consumer ids (18 consumers with 8 streams): min: 8.00; max:
335.00; avg: 166.476190; stddev: 101.988433; ideal: 72.86
(unit.kafka.consumer.PartitionAllocatorTest:68)
[2014-07-25 17:33:52,532] INFO Owned count summary for 5783 partitions
across 25 consumer ids (5 consumers with 5 streams): min: 141.00; max:
336.00; avg: 231.32; stddev: 69.337171; ideal: 231.320007
(unit.kafka.consumer.PartitionAllocatorTest:68)
[2014-07-25 17:33:53,268] INFO Owned count summary for 6181 partitions
across 7 consumer ids (7 consumers with 1 streams): min: 801.00; max:
980.00; avg: 883.00; stddev: 59.654561; ideal: 883.00
(unit.kafka.consumer.PartitionAllocatorTest:68)
[2014-07-25 17:33:56,124] INFO Owned count summary for 6475 partitions
across 32 consumer ids (4 consumers with 8 streams): min: 105.00; max:
299.00; avg: 202.343750; stddev: 62.999544; ideal: 202.343750
(unit.kafka.consumer.PartitionAllocatorTest:68)
[2014-07-25 17:35:10,370] INFO Owned count summary for 7739 partitions
across 162 consumer ids (18 consumers with 9 streams): min: 6.00; max:
239.00; avg: 122.841270; stddev: 69.379788; ideal: 47.771606
(unit.kafka.consumer.PartitionAllocatorTest:68)
[2014-07-25 17:35:11,834] INFO Owned count summary for 9070 partitions
across 14 consumer ids (2 consumers with 7 streams): min: 520.00; max:
774.00; avg: 647.857143; stddev: 84.860843; ideal: 647.857117
(unit.kafka.consumer.PartitionAllocatorTest:68)
[2014-07-25 17:36:37,935] INFO Owned count summary for 10933 partitions
across 85 consumer ids (17 consumers with 5 streams): min: 5.00; max:
350.00; avg: 173.539683; stddev: 105.619192; ideal: 128.623535
(unit.kafka.consumer.PartitionAllocatorTest:68)
[2014-07-25 17:36:40,641] INFO Owned count summary for 8665 partitions
across 64 consumer ids (8 consumers with 8 streams): min: 4.00; max:
267.00; avg: 137.539683; stddev: 82.121434; ideal: 135.390625
(unit.kafka.consumer.PartitionAllocatorTest:68)
[2014-07-25 17:36:42,612] INFO Owned count summary for 8432 partitions
across 48 consumer ids (6 consumers with 8 streams): min: 68.00; max:
328.00; avg: 175.67; stddev: 78.829828; ideal: 175.72
(unit.kafka.consumer.PartitionAllocatorTest:68)
[2014-07-2
[jira] [Commented] (KAFKA-687) Rebalance algorithm should consider partitions from all topics
[ https://issues.apache.org/jira/browse/KAFKA-687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14099304#comment-14099304 ] Joel Koshy commented on KAFKA-687: -- [~junrao] I was thinking over this a little more and I felt it is better not to design the new consumer's partition allocator API in this jira. There are a couple of reasons: * The new consumer's allocator's interface requirements and desired implementations will be known precisely only when we get to it - i.e., when we are implementing the partition assignment in the new consumer. So we will most likely change it anyway when we implement the new consumer. * The allocation code is not very complicated anyway so I don't think it is a lot of work to rewrite it in the new consumer implementation. * With the "more general" API that we discussed, the range allocation can no longer an exact copy (unlike the original patch). I would prefer to avoid touching the range-partitioner in the existing consumer at this point since that is the default that most people use. So what I would propose is the following: keep the partition allocation interface as in the original patch and provide only one more allocation implementation: roundrobin. This allocation scheme is legal only when using wildcards on all consumer instances and all the regexes are identical (although stream counts can be different). > Rebalance algorithm should consider partitions from all topics > -- > > Key: KAFKA-687 > URL: https://issues.apache.org/jira/browse/KAFKA-687 > Project: Kafka > Issue Type: Improvement >Affects Versions: 0.9.0 >Reporter: Pablo Barrera >Assignee: Joel Koshy > Attachments: KAFKA-687.patch, KAFKA-687_2014-07-18_15:55:15.patch > > > The current rebalance step, as stated in the original Kafka paper [1], splits > the partitions per topic between all the consumers. So if you have 100 topics > with 2 partitions each and 10 consumers only two consumers will be used. That > is, for each topic all partitions will be listed and shared between the > consumers in the consumer group in order (not randomly). > If the consumer group is reading from several topics at the same time it > makes sense to split all the partitions from all topics between all the > consumer. Following the example, we will have 200 partitions in total, 20 per > consumer, using the 10 consumers. > The load per topic could be different and the division should consider this. > However even a random division should be better than the current algorithm > while reading from several topics and should harm reading from a few topics > with several partitions. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-687) Rebalance algorithm should consider partitions from all topics
[ https://issues.apache.org/jira/browse/KAFKA-687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14099316#comment-14099316 ] Jun Rao commented on KAFKA-687: --- Yes, that sounds reasonable. We can try to minimize the changes in the old consumer. > Rebalance algorithm should consider partitions from all topics > -- > > Key: KAFKA-687 > URL: https://issues.apache.org/jira/browse/KAFKA-687 > Project: Kafka > Issue Type: Improvement >Affects Versions: 0.9.0 >Reporter: Pablo Barrera >Assignee: Joel Koshy > Attachments: KAFKA-687.patch, KAFKA-687_2014-07-18_15:55:15.patch > > > The current rebalance step, as stated in the original Kafka paper [1], splits > the partitions per topic between all the consumers. So if you have 100 topics > with 2 partitions each and 10 consumers only two consumers will be used. That > is, for each topic all partitions will be listed and shared between the > consumers in the consumer group in order (not randomly). > If the consumer group is reading from several topics at the same time it > makes sense to split all the partitions from all topics between all the > consumer. Following the example, we will have 200 partitions in total, 20 per > consumer, using the 10 consumers. > The load per topic could be different and the division should consider this. > However even a random division should be better than the current algorithm > while reading from several topics and should harm reading from a few topics > with several partitions. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-687) Rebalance algorithm should consider partitions from all topics
[ https://issues.apache.org/jira/browse/KAFKA-687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13548604#comment-13548604 ] Jay Kreps commented on KAFKA-687: - This is a very good point, and not one I had considered. It is probably not a trivial change because right now I think the election is done for each topic independently. We have in mind in the next major release after 0.8 (0.9, presumably) to move this co-ordination to the server, which would be a good time to fix this. We could either do this balancing exactly or else just randomize the start index (which would be almost as good if you had many topics. > Rebalance algorithm should consider partitions from all topics > -- > > Key: KAFKA-687 > URL: https://issues.apache.org/jira/browse/KAFKA-687 > Project: Kafka > Issue Type: Improvement >Affects Versions: 0.8.1 >Reporter: Pablo Barrera > > The current rebalance step, as stated in the original Kafka paper [1], splits > the partitions per topic between all the consumers. So if you have 100 topics > with 2 partitions each and 10 consumers only two consumers will be used. That > is, for each topic all partitions will be listed and shared between the > consumers in the consumer group in order (not randomly). > If the consumer group is reading from several topics at the same time it > makes sense to split all the partitions from all topics between all the > consumer. Following the example, we will have 200 partitions in total, 20 per > consumer, using the 10 consumers. > The load per topic could be different and the division should consider this. > However even a random division should be better than the current algorithm > while reading from several topics and should harm reading from a few topics > with several partitions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
