[jira] [Created] (KAFKA-7227) Kafka cluster should has a default rack id if not set
Genmao Yu created KAFKA-7227: Summary: Kafka cluster should has a default rack id if not set Key: KAFKA-7227 URL: https://issues.apache.org/jira/browse/KAFKA-7227 Project: Kafka Issue Type: Improvement Components: core Affects Versions: 2.0.0, 1.0.0 Reporter: Genmao Yu In current behavior, if partial of broker machines have "server.rack", we will failed to create topic with exception like "Not all brokers have rack information". There is no big problem in local cluster. But, this behavior may not be very suitable for cloud environment. In cloud environment, there may be different types of machine, some has rack info, while others do not has rack info. It could happen for many reasons, like understock of specific machine. We can give each broker a default rack id. * if not set any rack info, it is the same with previous behavior * if partial of brokers are set with rack info, we can think of the remaining brokers as same rack. As far as know, if not set rack id, DataNode/NodeManager has a default rack id in Hadoop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-5929) Save pre-assignment to file to avoid too long text to display when do topic partition reassign
Genmao Yu created KAFKA-5929: Summary: Save pre-assignment to file to avoid too long text to display when do topic partition reassign Key: KAFKA-5929 URL: https://issues.apache.org/jira/browse/KAFKA-5929 Project: Kafka Issue Type: Improvement Components: admin Reporter: Genmao Yu Priority: Minor -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-5928) Avoid redundant requests to zookeeper when reassign topic partition
Genmao Yu created KAFKA-5928:
Summary: Avoid redundant requests to zookeeper when reassign topic
partition
Key: KAFKA-5928
URL: https://issues.apache.org/jira/browse/KAFKA-5928
Project: Kafka
Issue Type: Improvement
Components: admin
Affects Versions: 0.11.0.0, 0.10.2.1
Reporter: Genmao Yu
We mistakenly request topic level information according to partitions config in
the assignment json file. For example
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala#L550:
{code}
val validPartitions = proposedPartitionAssignment.filter { case (p, _) =>
validatePartition(zkUtils, p.topic, p.partition) }
{code}
If reassign 1000 partitions (in 10 topics), we need to request zookeeper 1000
times here. But actually we only need to request just 10 (topics) times. We
test a large-scale assignment, about 10K partitions. It takes tens of minutes.
After optimization, it will reduce to less than 1minute.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
