[
https://issues.apache.org/jira/browse/KAFKA-1792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14292355#comment-14292355
]
Dmitry Pekar commented on KAFKA-1792:
-------------------------------------
[~nehanarkhede] thanks for your comments.
Please see the answers below:
1. As I've understood from previous discussions we should preserve more or less
backward compatibility with old
--generate. I even think that renaming to --rebalance was a bad idea.
"generate" name was consistent
in semantics with --execute and --verify. The scenario was
--generate/--execute/--verify. If we rename --generate
to --rebalance the naming becomes inconsistent. Also if we will remove
-broker-list/-topics option from --rebalance that
would restrict usecases of the CLI, so IMHO removing those options is
destructive.
2. Currently --decomission-broker doesn't require additional options except
broker id.
3. I think, that -broker-list/-topics should still be specified to
--rebalance/--generate as described in 1. Sure, we could
provide a better description of reassignment configuration if required. Need
to negotiate what exactly should be printed
and is it possible to determine those information from broker registry,
stored in ZK.
4. Yes.
When working on this I was under assumption, that the changes to this CLI would
be minor. If required, we still can design
a completely new CLI/or heavily change existent with required set of
commands/options, but in that case, IMHO, we should
develop and negotiate a design doc first.
[~charmalloc] I think that we need to decide about further steps of
fixing/implementing CLI changes. Your expertise is required
to make those decisions.
> change behavior of --generate to produce assignment config with fair replica
> distribution and minimal number of reassignments
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-1792
> URL: https://issues.apache.org/jira/browse/KAFKA-1792
> Project: Kafka
> Issue Type: Sub-task
> Components: tools
> Reporter: Dmitry Pekar
> Assignee: Dmitry Pekar
> Fix For: 0.8.3
>
> Attachments: KAFKA-1792.patch, KAFKA-1792_2014-12-03_19:24:56.patch,
> KAFKA-1792_2014-12-08_13:42:43.patch, KAFKA-1792_2014-12-19_16:48:12.patch,
> KAFKA-1792_2015-01-14_12:54:52.patch, generate_alg_tests.txt,
> rebalance_use_cases.txt
>
>
> Current implementation produces fair replica distribution between specified
> list of brokers. Unfortunately, it doesn't take
> into account current replica assignment.
> So if we have, for instance, 3 brokers id=[0..2] and are going to add fourth
> broker id=3,
> generate will create an assignment config which will redistribute replicas
> fairly across brokers [0..3]
> in the same way as those partitions were created from scratch. It will not
> take into consideration current replica
> assignment and accordingly will not try to minimize number of replica moves
> between brokers.
> As proposed by [~charmalloc] this should be improved. New output of improved
> --generate algorithm should suite following requirements:
> - fairness of replica distribution - every broker will have R or R+1 replicas
> assigned;
> - minimum of reassignments - number of replica moves between brokers will be
> minimal;
> Example.
> Consider following replica distribution per brokers [0..3] (we just added
> brokers 2 and 3):
> - broker - 0, 1, 2, 3
> - replicas - 7, 6, 0, 0
> The new algorithm will produce following assignment:
> - broker - 0, 1, 2, 3
> - replicas - 4, 3, 3, 3
> - moves - -3, -3, +3, +3
> It will be fair and number of moves will be 6, which is minimal for specified
> initial distribution.
> The scope of this issue is:
> - design an algorithm matching the above requirements;
> - implement this algorithm and unit tests;
> - test it manually using different initial assignments;
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
