[jira] [Updated] (KAFKA-1792) change behavior of --generate to produce assignment config with fair replica distribution and minimal number of reassignments
[ https://issues.apache.org/jira/browse/KAFKA-1792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-1792: Fix Version/s: (was: 0.8.3) > change behavior of --generate to produce assignment config with fair replica > distribution and minimal number of reassignments > - > > Key: KAFKA-1792 > URL: https://issues.apache.org/jira/browse/KAFKA-1792 > Project: Kafka > Issue Type: Sub-task > Components: tools >Reporter: Dmitry Pekar >Assignee: Dmitry Pekar > Attachments: KAFKA-1792.patch, KAFKA-1792_2014-12-03_19:24:56.patch, > KAFKA-1792_2014-12-08_13:42:43.patch, KAFKA-1792_2014-12-19_16:48:12.patch, > KAFKA-1792_2015-01-14_12:54:52.patch, KAFKA-1792_2015-01-27_19:09:27.patch, > KAFKA-1792_2015-02-13_21:07:06.patch, KAFKA-1792_2015-02-26_16:58:23.patch, > generate_alg_tests.txt, rebalance_use_cases.txt > > > Current implementation produces fair replica distribution between specified > list of brokers. Unfortunately, it doesn't take > into account current replica assignment. > So if we have, for instance, 3 brokers id=[0..2] and are going to add fourth > broker id=3, > generate will create an assignment config which will redistribute replicas > fairly across brokers [0..3] > in the same way as those partitions were created from scratch. It will not > take into consideration current replica > assignment and accordingly will not try to minimize number of replica moves > between brokers. > As proposed by [~charmalloc] this should be improved. New output of improved > --generate algorithm should suite following requirements: > - fairness of replica distribution - every broker will have R or R+1 replicas > assigned; > - minimum of reassignments - number of replica moves between brokers will be > minimal; > Example. > Consider following replica distribution per brokers [0..3] (we just added > brokers 2 and 3): > - broker - 0, 1, 2, 3 > - replicas - 7, 6, 0, 0 > The new algorithm will produce following assignment: > - broker - 0, 1, 2, 3 > - replicas - 4, 3, 3, 3 > - moves - -3, -3, +3, +3 > It will be fair and number of moves will be 6, which is minimal for specified > initial distribution. > The scope of this issue is: > - design an algorithm matching the above requirements; > - implement this algorithm and unit tests; > - test it manually using different initial assignments; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1792) change behavior of --generate to produce assignment config with fair replica distribution and minimal number of reassignments
[ https://issues.apache.org/jira/browse/KAFKA-1792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitry Pekar updated KAFKA-1792: Attachment: rebalance_use_cases.txt > change behavior of --generate to produce assignment config with fair replica > distribution and minimal number of reassignments > - > > Key: KAFKA-1792 > URL: https://issues.apache.org/jira/browse/KAFKA-1792 > Project: Kafka > Issue Type: Sub-task > Components: tools >Reporter: Dmitry Pekar >Assignee: Dmitry Pekar > Fix For: 0.8.3 > > Attachments: KAFKA-1792.patch, KAFKA-1792_2014-12-03_19:24:56.patch, > KAFKA-1792_2014-12-08_13:42:43.patch, KAFKA-1792_2014-12-19_16:48:12.patch, > KAFKA-1792_2015-01-14_12:54:52.patch, KAFKA-1792_2015-01-27_19:09:27.patch, > KAFKA-1792_2015-02-13_21:07:06.patch, KAFKA-1792_2015-02-26_16:58:23.patch, > generate_alg_tests.txt, rebalance_use_cases.txt > > > Current implementation produces fair replica distribution between specified > list of brokers. Unfortunately, it doesn't take > into account current replica assignment. > So if we have, for instance, 3 brokers id=[0..2] and are going to add fourth > broker id=3, > generate will create an assignment config which will redistribute replicas > fairly across brokers [0..3] > in the same way as those partitions were created from scratch. It will not > take into consideration current replica > assignment and accordingly will not try to minimize number of replica moves > between brokers. > As proposed by [~charmalloc] this should be improved. New output of improved > --generate algorithm should suite following requirements: > - fairness of replica distribution - every broker will have R or R+1 replicas > assigned; > - minimum of reassignments - number of replica moves between brokers will be > minimal; > Example. > Consider following replica distribution per brokers [0..3] (we just added > brokers 2 and 3): > - broker - 0, 1, 2, 3 > - replicas - 7, 6, 0, 0 > The new algorithm will produce following assignment: > - broker - 0, 1, 2, 3 > - replicas - 4, 3, 3, 3 > - moves - -3, -3, +3, +3 > It will be fair and number of moves will be 6, which is minimal for specified > initial distribution. > The scope of this issue is: > - design an algorithm matching the above requirements; > - implement this algorithm and unit tests; > - test it manually using different initial assignments; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1792) change behavior of --generate to produce assignment config with fair replica distribution and minimal number of reassignments
[ https://issues.apache.org/jira/browse/KAFKA-1792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitry Pekar updated KAFKA-1792: Attachment: (was: rebalance_use_cases.txt) > change behavior of --generate to produce assignment config with fair replica > distribution and minimal number of reassignments > - > > Key: KAFKA-1792 > URL: https://issues.apache.org/jira/browse/KAFKA-1792 > Project: Kafka > Issue Type: Sub-task > Components: tools >Reporter: Dmitry Pekar >Assignee: Dmitry Pekar > Fix For: 0.8.3 > > Attachments: KAFKA-1792.patch, KAFKA-1792_2014-12-03_19:24:56.patch, > KAFKA-1792_2014-12-08_13:42:43.patch, KAFKA-1792_2014-12-19_16:48:12.patch, > KAFKA-1792_2015-01-14_12:54:52.patch, KAFKA-1792_2015-01-27_19:09:27.patch, > KAFKA-1792_2015-02-13_21:07:06.patch, KAFKA-1792_2015-02-26_16:58:23.patch, > generate_alg_tests.txt > > > Current implementation produces fair replica distribution between specified > list of brokers. Unfortunately, it doesn't take > into account current replica assignment. > So if we have, for instance, 3 brokers id=[0..2] and are going to add fourth > broker id=3, > generate will create an assignment config which will redistribute replicas > fairly across brokers [0..3] > in the same way as those partitions were created from scratch. It will not > take into consideration current replica > assignment and accordingly will not try to minimize number of replica moves > between brokers. > As proposed by [~charmalloc] this should be improved. New output of improved > --generate algorithm should suite following requirements: > - fairness of replica distribution - every broker will have R or R+1 replicas > assigned; > - minimum of reassignments - number of replica moves between brokers will be > minimal; > Example. > Consider following replica distribution per brokers [0..3] (we just added > brokers 2 and 3): > - broker - 0, 1, 2, 3 > - replicas - 7, 6, 0, 0 > The new algorithm will produce following assignment: > - broker - 0, 1, 2, 3 > - replicas - 4, 3, 3, 3 > - moves - -3, -3, +3, +3 > It will be fair and number of moves will be 6, which is minimal for specified > initial distribution. > The scope of this issue is: > - design an algorithm matching the above requirements; > - implement this algorithm and unit tests; > - test it manually using different initial assignments; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1792) change behavior of --generate to produce assignment config with fair replica distribution and minimal number of reassignments
[ https://issues.apache.org/jira/browse/KAFKA-1792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitry Pekar updated KAFKA-1792: Attachment: KAFKA-1792_2015-02-26_16:58:23.patch > change behavior of --generate to produce assignment config with fair replica > distribution and minimal number of reassignments > - > > Key: KAFKA-1792 > URL: https://issues.apache.org/jira/browse/KAFKA-1792 > Project: Kafka > Issue Type: Sub-task > Components: tools >Reporter: Dmitry Pekar >Assignee: Dmitry Pekar > Fix For: 0.8.3 > > Attachments: KAFKA-1792.patch, KAFKA-1792_2014-12-03_19:24:56.patch, > KAFKA-1792_2014-12-08_13:42:43.patch, KAFKA-1792_2014-12-19_16:48:12.patch, > KAFKA-1792_2015-01-14_12:54:52.patch, KAFKA-1792_2015-01-27_19:09:27.patch, > KAFKA-1792_2015-02-13_21:07:06.patch, KAFKA-1792_2015-02-26_16:58:23.patch, > generate_alg_tests.txt, rebalance_use_cases.txt > > > Current implementation produces fair replica distribution between specified > list of brokers. Unfortunately, it doesn't take > into account current replica assignment. > So if we have, for instance, 3 brokers id=[0..2] and are going to add fourth > broker id=3, > generate will create an assignment config which will redistribute replicas > fairly across brokers [0..3] > in the same way as those partitions were created from scratch. It will not > take into consideration current replica > assignment and accordingly will not try to minimize number of replica moves > between brokers. > As proposed by [~charmalloc] this should be improved. New output of improved > --generate algorithm should suite following requirements: > - fairness of replica distribution - every broker will have R or R+1 replicas > assigned; > - minimum of reassignments - number of replica moves between brokers will be > minimal; > Example. > Consider following replica distribution per brokers [0..3] (we just added > brokers 2 and 3): > - broker - 0, 1, 2, 3 > - replicas - 7, 6, 0, 0 > The new algorithm will produce following assignment: > - broker - 0, 1, 2, 3 > - replicas - 4, 3, 3, 3 > - moves - -3, -3, +3, +3 > It will be fair and number of moves will be 6, which is minimal for specified > initial distribution. > The scope of this issue is: > - design an algorithm matching the above requirements; > - implement this algorithm and unit tests; > - test it manually using different initial assignments; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1792) change behavior of --generate to produce assignment config with fair replica distribution and minimal number of reassignments
[ https://issues.apache.org/jira/browse/KAFKA-1792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitry Pekar updated KAFKA-1792: Attachment: rebalance_use_cases.txt > change behavior of --generate to produce assignment config with fair replica > distribution and minimal number of reassignments > - > > Key: KAFKA-1792 > URL: https://issues.apache.org/jira/browse/KAFKA-1792 > Project: Kafka > Issue Type: Sub-task > Components: tools >Reporter: Dmitry Pekar >Assignee: Dmitry Pekar > Fix For: 0.8.3 > > Attachments: KAFKA-1792.patch, KAFKA-1792_2014-12-03_19:24:56.patch, > KAFKA-1792_2014-12-08_13:42:43.patch, KAFKA-1792_2014-12-19_16:48:12.patch, > KAFKA-1792_2015-01-14_12:54:52.patch, KAFKA-1792_2015-01-27_19:09:27.patch, > KAFKA-1792_2015-02-13_21:07:06.patch, generate_alg_tests.txt, > rebalance_use_cases.txt > > > Current implementation produces fair replica distribution between specified > list of brokers. Unfortunately, it doesn't take > into account current replica assignment. > So if we have, for instance, 3 brokers id=[0..2] and are going to add fourth > broker id=3, > generate will create an assignment config which will redistribute replicas > fairly across brokers [0..3] > in the same way as those partitions were created from scratch. It will not > take into consideration current replica > assignment and accordingly will not try to minimize number of replica moves > between brokers. > As proposed by [~charmalloc] this should be improved. New output of improved > --generate algorithm should suite following requirements: > - fairness of replica distribution - every broker will have R or R+1 replicas > assigned; > - minimum of reassignments - number of replica moves between brokers will be > minimal; > Example. > Consider following replica distribution per brokers [0..3] (we just added > brokers 2 and 3): > - broker - 0, 1, 2, 3 > - replicas - 7, 6, 0, 0 > The new algorithm will produce following assignment: > - broker - 0, 1, 2, 3 > - replicas - 4, 3, 3, 3 > - moves - -3, -3, +3, +3 > It will be fair and number of moves will be 6, which is minimal for specified > initial distribution. > The scope of this issue is: > - design an algorithm matching the above requirements; > - implement this algorithm and unit tests; > - test it manually using different initial assignments; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1792) change behavior of --generate to produce assignment config with fair replica distribution and minimal number of reassignments
[ https://issues.apache.org/jira/browse/KAFKA-1792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitry Pekar updated KAFKA-1792: Attachment: (was: rebalance_use_cases.txt) > change behavior of --generate to produce assignment config with fair replica > distribution and minimal number of reassignments > - > > Key: KAFKA-1792 > URL: https://issues.apache.org/jira/browse/KAFKA-1792 > Project: Kafka > Issue Type: Sub-task > Components: tools >Reporter: Dmitry Pekar >Assignee: Dmitry Pekar > Fix For: 0.8.3 > > Attachments: KAFKA-1792.patch, KAFKA-1792_2014-12-03_19:24:56.patch, > KAFKA-1792_2014-12-08_13:42:43.patch, KAFKA-1792_2014-12-19_16:48:12.patch, > KAFKA-1792_2015-01-14_12:54:52.patch, KAFKA-1792_2015-01-27_19:09:27.patch, > KAFKA-1792_2015-02-13_21:07:06.patch, generate_alg_tests.txt > > > Current implementation produces fair replica distribution between specified > list of brokers. Unfortunately, it doesn't take > into account current replica assignment. > So if we have, for instance, 3 brokers id=[0..2] and are going to add fourth > broker id=3, > generate will create an assignment config which will redistribute replicas > fairly across brokers [0..3] > in the same way as those partitions were created from scratch. It will not > take into consideration current replica > assignment and accordingly will not try to minimize number of replica moves > between brokers. > As proposed by [~charmalloc] this should be improved. New output of improved > --generate algorithm should suite following requirements: > - fairness of replica distribution - every broker will have R or R+1 replicas > assigned; > - minimum of reassignments - number of replica moves between brokers will be > minimal; > Example. > Consider following replica distribution per brokers [0..3] (we just added > brokers 2 and 3): > - broker - 0, 1, 2, 3 > - replicas - 7, 6, 0, 0 > The new algorithm will produce following assignment: > - broker - 0, 1, 2, 3 > - replicas - 4, 3, 3, 3 > - moves - -3, -3, +3, +3 > It will be fair and number of moves will be 6, which is minimal for specified > initial distribution. > The scope of this issue is: > - design an algorithm matching the above requirements; > - implement this algorithm and unit tests; > - test it manually using different initial assignments; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1792) change behavior of --generate to produce assignment config with fair replica distribution and minimal number of reassignments
[ https://issues.apache.org/jira/browse/KAFKA-1792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitry Pekar updated KAFKA-1792: Attachment: KAFKA-1792_2015-02-13_21:07:06.patch > change behavior of --generate to produce assignment config with fair replica > distribution and minimal number of reassignments > - > > Key: KAFKA-1792 > URL: https://issues.apache.org/jira/browse/KAFKA-1792 > Project: Kafka > Issue Type: Sub-task > Components: tools >Reporter: Dmitry Pekar >Assignee: Dmitry Pekar > Fix For: 0.8.3 > > Attachments: KAFKA-1792.patch, KAFKA-1792_2014-12-03_19:24:56.patch, > KAFKA-1792_2014-12-08_13:42:43.patch, KAFKA-1792_2014-12-19_16:48:12.patch, > KAFKA-1792_2015-01-14_12:54:52.patch, KAFKA-1792_2015-01-27_19:09:27.patch, > KAFKA-1792_2015-02-13_21:07:06.patch, generate_alg_tests.txt, > rebalance_use_cases.txt > > > Current implementation produces fair replica distribution between specified > list of brokers. Unfortunately, it doesn't take > into account current replica assignment. > So if we have, for instance, 3 brokers id=[0..2] and are going to add fourth > broker id=3, > generate will create an assignment config which will redistribute replicas > fairly across brokers [0..3] > in the same way as those partitions were created from scratch. It will not > take into consideration current replica > assignment and accordingly will not try to minimize number of replica moves > between brokers. > As proposed by [~charmalloc] this should be improved. New output of improved > --generate algorithm should suite following requirements: > - fairness of replica distribution - every broker will have R or R+1 replicas > assigned; > - minimum of reassignments - number of replica moves between brokers will be > minimal; > Example. > Consider following replica distribution per brokers [0..3] (we just added > brokers 2 and 3): > - broker - 0, 1, 2, 3 > - replicas - 7, 6, 0, 0 > The new algorithm will produce following assignment: > - broker - 0, 1, 2, 3 > - replicas - 4, 3, 3, 3 > - moves - -3, -3, +3, +3 > It will be fair and number of moves will be 6, which is minimal for specified > initial distribution. > The scope of this issue is: > - design an algorithm matching the above requirements; > - implement this algorithm and unit tests; > - test it manually using different initial assignments; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1792) change behavior of --generate to produce assignment config with fair replica distribution and minimal number of reassignments
[ https://issues.apache.org/jira/browse/KAFKA-1792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitry Pekar updated KAFKA-1792: Attachment: KAFKA-1792_2015-01-27_19:09:27.patch > change behavior of --generate to produce assignment config with fair replica > distribution and minimal number of reassignments > - > > Key: KAFKA-1792 > URL: https://issues.apache.org/jira/browse/KAFKA-1792 > Project: Kafka > Issue Type: Sub-task > Components: tools >Reporter: Dmitry Pekar >Assignee: Dmitry Pekar > Fix For: 0.8.3 > > Attachments: KAFKA-1792.patch, KAFKA-1792_2014-12-03_19:24:56.patch, > KAFKA-1792_2014-12-08_13:42:43.patch, KAFKA-1792_2014-12-19_16:48:12.patch, > KAFKA-1792_2015-01-14_12:54:52.patch, KAFKA-1792_2015-01-27_19:09:27.patch, > generate_alg_tests.txt, rebalance_use_cases.txt > > > Current implementation produces fair replica distribution between specified > list of brokers. Unfortunately, it doesn't take > into account current replica assignment. > So if we have, for instance, 3 brokers id=[0..2] and are going to add fourth > broker id=3, > generate will create an assignment config which will redistribute replicas > fairly across brokers [0..3] > in the same way as those partitions were created from scratch. It will not > take into consideration current replica > assignment and accordingly will not try to minimize number of replica moves > between brokers. > As proposed by [~charmalloc] this should be improved. New output of improved > --generate algorithm should suite following requirements: > - fairness of replica distribution - every broker will have R or R+1 replicas > assigned; > - minimum of reassignments - number of replica moves between brokers will be > minimal; > Example. > Consider following replica distribution per brokers [0..3] (we just added > brokers 2 and 3): > - broker - 0, 1, 2, 3 > - replicas - 7, 6, 0, 0 > The new algorithm will produce following assignment: > - broker - 0, 1, 2, 3 > - replicas - 4, 3, 3, 3 > - moves - -3, -3, +3, +3 > It will be fair and number of moves will be 6, which is minimal for specified > initial distribution. > The scope of this issue is: > - design an algorithm matching the above requirements; > - implement this algorithm and unit tests; > - test it manually using different initial assignments; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1792) change behavior of --generate to produce assignment config with fair replica distribution and minimal number of reassignments
[ https://issues.apache.org/jira/browse/KAFKA-1792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitry Pekar updated KAFKA-1792: Attachment: rebalance_use_cases.txt [~nehanarkhede], [~charmalloc] Attached use cases for ReassignPartititionsCommand. PS. Sorry for the delayed response. I was on vacation. > change behavior of --generate to produce assignment config with fair replica > distribution and minimal number of reassignments > - > > Key: KAFKA-1792 > URL: https://issues.apache.org/jira/browse/KAFKA-1792 > Project: Kafka > Issue Type: Sub-task > Components: tools >Reporter: Dmitry Pekar >Assignee: Dmitry Pekar > Fix For: 0.8.3 > > Attachments: KAFKA-1792.patch, KAFKA-1792_2014-12-03_19:24:56.patch, > KAFKA-1792_2014-12-08_13:42:43.patch, KAFKA-1792_2014-12-19_16:48:12.patch, > KAFKA-1792_2015-01-14_12:54:52.patch, generate_alg_tests.txt, > rebalance_use_cases.txt > > > Current implementation produces fair replica distribution between specified > list of brokers. Unfortunately, it doesn't take > into account current replica assignment. > So if we have, for instance, 3 brokers id=[0..2] and are going to add fourth > broker id=3, > generate will create an assignment config which will redistribute replicas > fairly across brokers [0..3] > in the same way as those partitions were created from scratch. It will not > take into consideration current replica > assignment and accordingly will not try to minimize number of replica moves > between brokers. > As proposed by [~charmalloc] this should be improved. New output of improved > --generate algorithm should suite following requirements: > - fairness of replica distribution - every broker will have R or R+1 replicas > assigned; > - minimum of reassignments - number of replica moves between brokers will be > minimal; > Example. > Consider following replica distribution per brokers [0..3] (we just added > brokers 2 and 3): > - broker - 0, 1, 2, 3 > - replicas - 7, 6, 0, 0 > The new algorithm will produce following assignment: > - broker - 0, 1, 2, 3 > - replicas - 4, 3, 3, 3 > - moves - -3, -3, +3, +3 > It will be fair and number of moves will be 6, which is minimal for specified > initial distribution. > The scope of this issue is: > - design an algorithm matching the above requirements; > - implement this algorithm and unit tests; > - test it manually using different initial assignments; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1792) change behavior of --generate to produce assignment config with fair replica distribution and minimal number of reassignments
[ https://issues.apache.org/jira/browse/KAFKA-1792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitry Pekar updated KAFKA-1792: Attachment: KAFKA-1792_2015-01-14_12:54:52.patch > change behavior of --generate to produce assignment config with fair replica > distribution and minimal number of reassignments > - > > Key: KAFKA-1792 > URL: https://issues.apache.org/jira/browse/KAFKA-1792 > Project: Kafka > Issue Type: Sub-task > Components: tools >Reporter: Dmitry Pekar >Assignee: Dmitry Pekar > Fix For: 0.8.3 > > Attachments: KAFKA-1792.patch, KAFKA-1792_2014-12-03_19:24:56.patch, > KAFKA-1792_2014-12-08_13:42:43.patch, KAFKA-1792_2014-12-19_16:48:12.patch, > KAFKA-1792_2015-01-14_12:54:52.patch, generate_alg_tests.txt > > > Current implementation produces fair replica distribution between specified > list of brokers. Unfortunately, it doesn't take > into account current replica assignment. > So if we have, for instance, 3 brokers id=[0..2] and are going to add fourth > broker id=3, > generate will create an assignment config which will redistribute replicas > fairly across brokers [0..3] > in the same way as those partitions were created from scratch. It will not > take into consideration current replica > assignment and accordingly will not try to minimize number of replica moves > between brokers. > As proposed by [~charmalloc] this should be improved. New output of improved > --generate algorithm should suite following requirements: > - fairness of replica distribution - every broker will have R or R+1 replicas > assigned; > - minimum of reassignments - number of replica moves between brokers will be > minimal; > Example. > Consider following replica distribution per brokers [0..3] (we just added > brokers 2 and 3): > - broker - 0, 1, 2, 3 > - replicas - 7, 6, 0, 0 > The new algorithm will produce following assignment: > - broker - 0, 1, 2, 3 > - replicas - 4, 3, 3, 3 > - moves - -3, -3, +3, +3 > It will be fair and number of moves will be 6, which is minimal for specified > initial distribution. > The scope of this issue is: > - design an algorithm matching the above requirements; > - implement this algorithm and unit tests; > - test it manually using different initial assignments; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1792) change behavior of --generate to produce assignment config with fair replica distribution and minimal number of reassignments
[ https://issues.apache.org/jira/browse/KAFKA-1792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1792: - Reviewer: Neha Narkhede > change behavior of --generate to produce assignment config with fair replica > distribution and minimal number of reassignments > - > > Key: KAFKA-1792 > URL: https://issues.apache.org/jira/browse/KAFKA-1792 > Project: Kafka > Issue Type: Sub-task > Components: tools >Reporter: Dmitry Pekar >Assignee: Dmitry Pekar > Fix For: 0.8.3 > > Attachments: KAFKA-1792.patch, KAFKA-1792_2014-12-03_19:24:56.patch, > KAFKA-1792_2014-12-08_13:42:43.patch, KAFKA-1792_2014-12-19_16:48:12.patch, > generate_alg_tests.txt > > > Current implementation produces fair replica distribution between specified > list of brokers. Unfortunately, it doesn't take > into account current replica assignment. > So if we have, for instance, 3 brokers id=[0..2] and are going to add fourth > broker id=3, > generate will create an assignment config which will redistribute replicas > fairly across brokers [0..3] > in the same way as those partitions were created from scratch. It will not > take into consideration current replica > assignment and accordingly will not try to minimize number of replica moves > between brokers. > As proposed by [~charmalloc] this should be improved. New output of improved > --generate algorithm should suite following requirements: > - fairness of replica distribution - every broker will have R or R+1 replicas > assigned; > - minimum of reassignments - number of replica moves between brokers will be > minimal; > Example. > Consider following replica distribution per brokers [0..3] (we just added > brokers 2 and 3): > - broker - 0, 1, 2, 3 > - replicas - 7, 6, 0, 0 > The new algorithm will produce following assignment: > - broker - 0, 1, 2, 3 > - replicas - 4, 3, 3, 3 > - moves - -3, -3, +3, +3 > It will be fair and number of moves will be 6, which is minimal for specified > initial distribution. > The scope of this issue is: > - design an algorithm matching the above requirements; > - implement this algorithm and unit tests; > - test it manually using different initial assignments; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1792) change behavior of --generate to produce assignment config with fair replica distribution and minimal number of reassignments
[ https://issues.apache.org/jira/browse/KAFKA-1792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitry Pekar updated KAFKA-1792: Attachment: generate_alg_tests.txt [~nehanarkhede] Please see provided test results, for test cluster of 5 brokers. Typical scenarios are handled there such as broker addition/removal/replacement. > change behavior of --generate to produce assignment config with fair replica > distribution and minimal number of reassignments > - > > Key: KAFKA-1792 > URL: https://issues.apache.org/jira/browse/KAFKA-1792 > Project: Kafka > Issue Type: Sub-task > Components: tools >Reporter: Dmitry Pekar >Assignee: Dmitry Pekar > Fix For: 0.8.3 > > Attachments: KAFKA-1792.patch, KAFKA-1792_2014-12-03_19:24:56.patch, > KAFKA-1792_2014-12-08_13:42:43.patch, KAFKA-1792_2014-12-19_16:48:12.patch, > generate_alg_tests.txt > > > Current implementation produces fair replica distribution between specified > list of brokers. Unfortunately, it doesn't take > into account current replica assignment. > So if we have, for instance, 3 brokers id=[0..2] and are going to add fourth > broker id=3, > generate will create an assignment config which will redistribute replicas > fairly across brokers [0..3] > in the same way as those partitions were created from scratch. It will not > take into consideration current replica > assignment and accordingly will not try to minimize number of replica moves > between brokers. > As proposed by [~charmalloc] this should be improved. New output of improved > --generate algorithm should suite following requirements: > - fairness of replica distribution - every broker will have R or R+1 replicas > assigned; > - minimum of reassignments - number of replica moves between brokers will be > minimal; > Example. > Consider following replica distribution per brokers [0..3] (we just added > brokers 2 and 3): > - broker - 0, 1, 2, 3 > - replicas - 7, 6, 0, 0 > The new algorithm will produce following assignment: > - broker - 0, 1, 2, 3 > - replicas - 4, 3, 3, 3 > - moves - -3, -3, +3, +3 > It will be fair and number of moves will be 6, which is minimal for specified > initial distribution. > The scope of this issue is: > - design an algorithm matching the above requirements; > - implement this algorithm and unit tests; > - test it manually using different initial assignments; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1792) change behavior of --generate to produce assignment config with fair replica distribution and minimal number of reassignments
[ https://issues.apache.org/jira/browse/KAFKA-1792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitry Pekar updated KAFKA-1792: Attachment: KAFKA-1792_2014-12-19_16:48:12.patch > change behavior of --generate to produce assignment config with fair replica > distribution and minimal number of reassignments > - > > Key: KAFKA-1792 > URL: https://issues.apache.org/jira/browse/KAFKA-1792 > Project: Kafka > Issue Type: Sub-task > Components: tools >Reporter: Dmitry Pekar >Assignee: Dmitry Pekar > Fix For: 0.8.3 > > Attachments: KAFKA-1792.patch, KAFKA-1792_2014-12-03_19:24:56.patch, > KAFKA-1792_2014-12-08_13:42:43.patch, KAFKA-1792_2014-12-19_16:48:12.patch > > > Current implementation produces fair replica distribution between specified > list of brokers. Unfortunately, it doesn't take > into account current replica assignment. > So if we have, for instance, 3 brokers id=[0..2] and are going to add fourth > broker id=3, > generate will create an assignment config which will redistribute replicas > fairly across brokers [0..3] > in the same way as those partitions were created from scratch. It will not > take into consideration current replica > assignment and accordingly will not try to minimize number of replica moves > between brokers. > As proposed by [~charmalloc] this should be improved. New output of improved > --generate algorithm should suite following requirements: > - fairness of replica distribution - every broker will have R or R+1 replicas > assigned; > - minimum of reassignments - number of replica moves between brokers will be > minimal; > Example. > Consider following replica distribution per brokers [0..3] (we just added > brokers 2 and 3): > - broker - 0, 1, 2, 3 > - replicas - 7, 6, 0, 0 > The new algorithm will produce following assignment: > - broker - 0, 1, 2, 3 > - replicas - 4, 3, 3, 3 > - moves - -3, -3, +3, +3 > It will be fair and number of moves will be 6, which is minimal for specified > initial distribution. > The scope of this issue is: > - design an algorithm matching the above requirements; > - implement this algorithm and unit tests; > - test it manually using different initial assignments; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1792) change behavior of --generate to produce assignment config with fair replica distribution and minimal number of reassignments
[ https://issues.apache.org/jira/browse/KAFKA-1792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitry Pekar updated KAFKA-1792: Attachment: KAFKA-1792_2014-12-08_13:42:43.patch > change behavior of --generate to produce assignment config with fair replica > distribution and minimal number of reassignments > - > > Key: KAFKA-1792 > URL: https://issues.apache.org/jira/browse/KAFKA-1792 > Project: Kafka > Issue Type: Sub-task > Components: tools >Reporter: Dmitry Pekar >Assignee: Dmitry Pekar > Fix For: 0.8.3 > > Attachments: KAFKA-1792.patch, KAFKA-1792_2014-12-03_19:24:56.patch, > KAFKA-1792_2014-12-08_13:42:43.patch > > > Current implementation produces fair replica distribution between specified > list of brokers. Unfortunately, it doesn't take > into account current replica assignment. > So if we have, for instance, 3 brokers id=[0..2] and are going to add fourth > broker id=3, > generate will create an assignment config which will redistribute replicas > fairly across brokers [0..3] > in the same way as those partitions were created from scratch. It will not > take into consideration current replica > assignment and accordingly will not try to minimize number of replica moves > between brokers. > As proposed by [~charmalloc] this should be improved. New output of improved > --generate algorithm should suite following requirements: > - fairness of replica distribution - every broker will have R or R+1 replicas > assigned; > - minimum of reassignments - number of replica moves between brokers will be > minimal; > Example. > Consider following replica distribution per brokers [0..3] (we just added > brokers 2 and 3): > - broker - 0, 1, 2, 3 > - replicas - 7, 6, 0, 0 > The new algorithm will produce following assignment: > - broker - 0, 1, 2, 3 > - replicas - 4, 3, 3, 3 > - moves - -3, -3, +3, +3 > It will be fair and number of moves will be 6, which is minimal for specified > initial distribution. > The scope of this issue is: > - design an algorithm matching the above requirements; > - implement this algorithm and unit tests; > - test it manually using different initial assignments; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1792) change behavior of --generate to produce assignment config with fair replica distribution and minimal number of reassignments
[ https://issues.apache.org/jira/browse/KAFKA-1792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitry Pekar updated KAFKA-1792: Attachment: KAFKA-1792_2014-12-03_19:24:56.patch > change behavior of --generate to produce assignment config with fair replica > distribution and minimal number of reassignments > - > > Key: KAFKA-1792 > URL: https://issues.apache.org/jira/browse/KAFKA-1792 > Project: Kafka > Issue Type: Sub-task > Components: tools >Reporter: Dmitry Pekar >Assignee: Dmitry Pekar > Fix For: 0.8.3 > > Attachments: KAFKA-1792.patch, KAFKA-1792_2014-12-03_19:24:56.patch > > > Current implementation produces fair replica distribution between specified > list of brokers. Unfortunately, it doesn't take > into account current replica assignment. > So if we have, for instance, 3 brokers id=[0..2] and are going to add fourth > broker id=3, > generate will create an assignment config which will redistribute replicas > fairly across brokers [0..3] > in the same way as those partitions were created from scratch. It will not > take into consideration current replica > assignment and accordingly will not try to minimize number of replica moves > between brokers. > As proposed by [~charmalloc] this should be improved. New output of improved > --generate algorithm should suite following requirements: > - fairness of replica distribution - every broker will have R or R+1 replicas > assigned; > - minimum of reassignments - number of replica moves between brokers will be > minimal; > Example. > Consider following replica distribution per brokers [0..3] (we just added > brokers 2 and 3): > - broker - 0, 1, 2, 3 > - replicas - 7, 6, 0, 0 > The new algorithm will produce following assignment: > - broker - 0, 1, 2, 3 > - replicas - 4, 3, 3, 3 > - moves - -3, -3, +3, +3 > It will be fair and number of moves will be 6, which is minimal for specified > initial distribution. > The scope of this issue is: > - design an algorithm matching the above requirements; > - implement this algorithm and unit tests; > - test it manually using different initial assignments; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1792) change behavior of --generate to produce assignment config with fair replica distribution and minimal number of reassignments
[ https://issues.apache.org/jira/browse/KAFKA-1792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitry Pekar updated KAFKA-1792: Attachment: KAFKA-1792.patch > change behavior of --generate to produce assignment config with fair replica > distribution and minimal number of reassignments > - > > Key: KAFKA-1792 > URL: https://issues.apache.org/jira/browse/KAFKA-1792 > Project: Kafka > Issue Type: Sub-task > Components: tools >Reporter: Dmitry Pekar >Assignee: Dmitry Pekar > Fix For: 0.8.3 > > Attachments: KAFKA-1792.patch > > > Current implementation produces fair replica distribution between specified > list of brokers. Unfortunately, it doesn't take > into account current replica assignment. > So if we have, for instance, 3 brokers id=[0..2] and are going to add fourth > broker id=3, > generate will create an assignment config which will redistribute replicas > fairly across brokers [0..3] > in the same way as those partitions were created from scratch. It will not > take into consideration current replica > assignment and accordingly will not try to minimize number of replica moves > between brokers. > As proposed by [~charmalloc] this should be improved. New output of improved > --generate algorithm should suite following requirements: > - fairness of replica distribution - every broker will have R or R+1 replicas > assigned; > - minimum of reassignments - number of replica moves between brokers will be > minimal; > Example. > Consider following replica distribution per brokers [0..3] (we just added > brokers 2 and 3): > - broker - 0, 1, 2, 3 > - replicas - 7, 6, 0, 0 > The new algorithm will produce following assignment: > - broker - 0, 1, 2, 3 > - replicas - 4, 3, 3, 3 > - moves - -3, -3, +3, +3 > It will be fair and number of moves will be 6, which is minimal for specified > initial distribution. > The scope of this issue is: > - design an algorithm matching the above requirements; > - implement this algorithm and unit tests; > - test it manually using different initial assignments; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1792) change behavior of --generate to produce assignment config with fair replica distribution and minimal number of reassignments
[ https://issues.apache.org/jira/browse/KAFKA-1792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitry Pekar updated KAFKA-1792: Status: Patch Available (was: In Progress) > change behavior of --generate to produce assignment config with fair replica > distribution and minimal number of reassignments > - > > Key: KAFKA-1792 > URL: https://issues.apache.org/jira/browse/KAFKA-1792 > Project: Kafka > Issue Type: Sub-task > Components: tools >Reporter: Dmitry Pekar >Assignee: Dmitry Pekar > Fix For: 0.8.3 > > Attachments: KAFKA-1792.patch > > > Current implementation produces fair replica distribution between specified > list of brokers. Unfortunately, it doesn't take > into account current replica assignment. > So if we have, for instance, 3 brokers id=[0..2] and are going to add fourth > broker id=3, > generate will create an assignment config which will redistribute replicas > fairly across brokers [0..3] > in the same way as those partitions were created from scratch. It will not > take into consideration current replica > assignment and accordingly will not try to minimize number of replica moves > between brokers. > As proposed by [~charmalloc] this should be improved. New output of improved > --generate algorithm should suite following requirements: > - fairness of replica distribution - every broker will have R or R+1 replicas > assigned; > - minimum of reassignments - number of replica moves between brokers will be > minimal; > Example. > Consider following replica distribution per brokers [0..3] (we just added > brokers 2 and 3): > - broker - 0, 1, 2, 3 > - replicas - 7, 6, 0, 0 > The new algorithm will produce following assignment: > - broker - 0, 1, 2, 3 > - replicas - 4, 3, 3, 3 > - moves - -3, -3, +3, +3 > It will be fair and number of moves will be 6, which is minimal for specified > initial distribution. > The scope of this issue is: > - design an algorithm matching the above requirements; > - implement this algorithm and unit tests; > - test it manually using different initial assignments; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1792) change behavior of --generate to produce assignment config with fair replica distribution and minimal number of reassignments
[ https://issues.apache.org/jira/browse/KAFKA-1792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitry Pekar updated KAFKA-1792: Description: Current implementation produces fair replica distribution between specified list of brokers. Unfortunately, it doesn't take into account current replica assignment. So if we have, for instance, 3 brokers id=[0..2] and are going to add fourth broker id=3, generate will create an assignment config which will redistribute replicas fairly across brokers [0..3] in the same way as those partitions were created from scratch. It will not take into consideration current replica assignment and accordingly will not try to minimize number of replica moves between brokers. As proposed by [~charmalloc] this should be improved. New output of improved --generate algorithm should suite following requirements: - fairness of replica distribution - every broker will have R or R+1 replicas assigned; - minimum of reassignments - number of replica moves between brokers will be minimal; Example. Consider following replica distribution per brokers [0..3] (we just added brokers 2 and 3): - broker - 0, 1, 2, 3 - replicas - 7, 6, 0, 0 The new algorithm will produce following assignment: - broker - 0, 1, 2, 3 - replicas - 4, 3, 3, 3 - moves - -3, -3, +3, +3 It will be fair and number of moves will be 6, which is minimal for specified initial distribution. The scope of this issue is: - design an algorithm matching the above requirements; - implement this algorithm and unit tests; - test it manually using different initial assignments; was: Current implementation produces fair replica distribution between specified list of brokers. Unfortunately, it doesn't take into account current replica assignment. So if we have, for instance, 3 brokers id=[0..2] and are going to add fourth broker id=3, generate will create an assignment config which will redistribute replicas fairly accross brokers [0..3] in the same way as those partitions were created from scratch. It will not take into consideration current replica assignment and accordingly will not try to minimize number of replica moves between brokers. As proposed by Joe Stein this should be improved. New output of improved --generate algorithm should suite following requirements: fairness of replica distribution - every broker will have R or R+1 replicas assigned; minimum of reassignments - number of replica moves between brokers will be minimal; Example. Consider following replica distribution per brokers [0..3] (we just added brokers 2 and 3): broker - 0, 1, 2, 3 replicas - 7, 6, 0, 0 The new algorithm will produce following assignment: broker - 0, 1, 2, 3 replicas - 4, 3, 3, 3 moves - -3, -3, +3, +3 It will be fair and number of moves will be 6, which is minimal for specified initial distribution. The scope of this issue is: design an algorithm matching the above requirements; implement this algorithm and unit tests; test it manually using different initial assignments; > change behavior of --generate to produce assignment config with fair replica > distribution and minimal number of reassignments > - > > Key: KAFKA-1792 > URL: https://issues.apache.org/jira/browse/KAFKA-1792 > Project: Kafka > Issue Type: Sub-task > Components: tools >Reporter: Dmitry Pekar > Fix For: 0.8.3 > > > Current implementation produces fair replica distribution between specified > list of brokers. Unfortunately, it doesn't take > into account current replica assignment. > So if we have, for instance, 3 brokers id=[0..2] and are going to add fourth > broker id=3, > generate will create an assignment config which will redistribute replicas > fairly across brokers [0..3] > in the same way as those partitions were created from scratch. It will not > take into consideration current replica > assignment and accordingly will not try to minimize number of replica moves > between brokers. > As proposed by [~charmalloc] this should be improved. New output of improved > --generate algorithm should suite following requirements: > - fairness of replica distribution - every broker will have R or R+1 replicas > assigned; > - minimum of reassignments - number of replica moves between brokers will be > minimal; > Example. > Consider following replica distribution per brokers [0..3] (we just added > brokers 2 and 3): > - broker - 0, 1, 2, 3 > - replicas - 7, 6, 0, 0 > The new algorithm will produce following assignment: > - broker - 0, 1, 2, 3 > - replicas - 4, 3, 3, 3 > - moves - -3, -3, +3, +3 > It will be fair and number of moves will be 6, which is minimal for specified > initial distribution. > The scope of this issue is: > - design an algorithm matching
