[jira] [Commented] (KAFKA-347) change number of partitions of a topic online
[ https://issues.apache.org/jira/browse/KAFKA-347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13734908#comment-13734908 ] Jun Rao commented on KAFKA-347: --- You just need to run bin/kafka-add-partitions.sh. We will add the docs when 0.8 final is released. change number of partitions of a topic online - Key: KAFKA-347 URL: https://issues.apache.org/jira/browse/KAFKA-347 Project: Kafka Issue Type: Improvement Components: core Affects Versions: 0.8 Reporter: Jun Rao Assignee: Sriram Subramanian Labels: features Fix For: 0.8.1 Attachments: kafka-347.patch, kafka-347-v2.patch, KAFKA-347-v2-rebased.patch, KAFKA-347-v3.patch, KAFKA-347-v4.patch, KAFKA-347-v5.patch We will need an admin tool to change the number of partitions of a topic online. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-347) change number of partitions of a topic online
[ https://issues.apache.org/jira/browse/KAFKA-347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13730616#comment-13730616 ] Cosmin Lehene commented on KAFKA-347: - Is there some operational documentation on how to use this? change number of partitions of a topic online - Key: KAFKA-347 URL: https://issues.apache.org/jira/browse/KAFKA-347 Project: Kafka Issue Type: Improvement Components: core Affects Versions: 0.8 Reporter: Jun Rao Assignee: Sriram Subramanian Labels: features Fix For: 0.8.1 Attachments: kafka-347.patch, kafka-347-v2.patch, KAFKA-347-v2-rebased.patch, KAFKA-347-v3.patch, KAFKA-347-v4.patch, KAFKA-347-v5.patch We will need an admin tool to change the number of partitions of a topic online. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-347) change number of partitions of a topic online
[ https://issues.apache.org/jira/browse/KAFKA-347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715390#comment-13715390 ] Sriram Subramanian commented on KAFKA-347: -- v2.4 the reason to expose it is for manual replica assignment. It is more explicit to specify the rep factor and the assignments for those. Rebased without the zkconsumer connector change. change number of partitions of a topic online - Key: KAFKA-347 URL: https://issues.apache.org/jira/browse/KAFKA-347 Project: Kafka Issue Type: Improvement Components: core Affects Versions: 0.8 Reporter: Jun Rao Assignee: Sriram Subramanian Labels: features Fix For: 0.8.1 Attachments: kafka-347.patch, kafka-347-v2.patch, KAFKA-347-v2-rebased.patch We will need an admin tool to change the number of partitions of a topic online. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-347) change number of partitions of a topic online
[ https://issues.apache.org/jira/browse/KAFKA-347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715392#comment-13715392 ] Jun Rao commented on KAFKA-347: --- Thanks for patch v2. Looks good overall. Some comments. 20. AdminUtils.createOrUpdateTopicPartitionAssignmentPathInZK(): The info logging should be different, depending on whether the topic is created or updated. Also, fix the indentation in the else clause. 21. AddPartitionsCommand: 21.1 remove unused imports 21.2 Is there any value in having the replica option? It seems that it should always be the existing replication factor. 21.3 For the replica-assignment-list option, could we make it clear in the description that this is for newly added partitions. 21.4 getManualReplicaAssignment(): We need to make sure the replica factor is the same as the existing one. 22. KafkaController.onNewTopicCreation(): Could you explain why the onNewPartition statement is moved to before the watcher registration? Normally, in order not to miss any watchers, one needs to register the watcher before reading the associated nodes in ZK. 23. PartitionStateMachine.AddPartitionsListener.handleDataChange(): 23.1 In the following statement, we are actually returning all replica assignments. val addedPartitionReplicaAssignment = ZkUtils.getReplicaAssignmentForTopics(zkClient, List(topic)) 23.2 Instead of using controllerContext.partitionLeadershipInfo to filter out existing partitions, it's probably better to use controllerContext.partitionReplicaAssignment, since leaders may not always exist. 23.3 In the error logging, could we add the affected data path? 24. AddPartitionsTest: remove unused imports 25. Did we do any test to make sure that existing consumers can pick up the new partitions? 26. The patch needs to be rebased. change number of partitions of a topic online - Key: KAFKA-347 URL: https://issues.apache.org/jira/browse/KAFKA-347 Project: Kafka Issue Type: Improvement Components: core Affects Versions: 0.8 Reporter: Jun Rao Assignee: Sriram Subramanian Labels: features Fix For: 0.8.1 Attachments: kafka-347.patch, kafka-347-v2.patch, KAFKA-347-v2-rebased.patch We will need an admin tool to change the number of partitions of a topic online. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-347) change number of partitions of a topic online
[ https://issues.apache.org/jira/browse/KAFKA-347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715469#comment-13715469 ] Sriram Subramanian commented on KAFKA-347: -- 20. Indentation seems fine to me. 21.2 It is present to make manual assignment more clear. 25 Yes the test was done. I will do another sanity check after the patch is commited. change number of partitions of a topic online - Key: KAFKA-347 URL: https://issues.apache.org/jira/browse/KAFKA-347 Project: Kafka Issue Type: Improvement Components: core Affects Versions: 0.8 Reporter: Jun Rao Assignee: Sriram Subramanian Labels: features Fix For: 0.8.1 Attachments: kafka-347.patch, kafka-347-v2.patch, KAFKA-347-v2-rebased.patch We will need an admin tool to change the number of partitions of a topic online. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-347) change number of partitions of a topic online
[ https://issues.apache.org/jira/browse/KAFKA-347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13715943#comment-13715943 ] Sriram Subramanian commented on KAFKA-347: -- added a script for addpartitions change number of partitions of a topic online - Key: KAFKA-347 URL: https://issues.apache.org/jira/browse/KAFKA-347 Project: Kafka Issue Type: Improvement Components: core Affects Versions: 0.8 Reporter: Jun Rao Assignee: Sriram Subramanian Labels: features Fix For: 0.8.1 Attachments: kafka-347.patch, kafka-347-v2.patch, KAFKA-347-v2-rebased.patch, KAFKA-347-v3.patch, KAFKA-347-v4.patch, KAFKA-347-v5.patch We will need an admin tool to change the number of partitions of a topic online. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-347) change number of partitions of a topic online
[ https://issues.apache.org/jira/browse/KAFKA-347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13716043#comment-13716043 ] Jun Rao commented on KAFKA-347: --- Thanks for the patch. Committed to 0.8. Could you also provide a patch to trunk since in trunk, all topic related commands are consolidated to a kafka-topic tool. change number of partitions of a topic online - Key: KAFKA-347 URL: https://issues.apache.org/jira/browse/KAFKA-347 Project: Kafka Issue Type: Improvement Components: core Affects Versions: 0.8 Reporter: Jun Rao Assignee: Sriram Subramanian Labels: features Fix For: 0.8.1 Attachments: kafka-347.patch, kafka-347-v2.patch, KAFKA-347-v2-rebased.patch, KAFKA-347-v3.patch, KAFKA-347-v4.patch, KAFKA-347-v5.patch We will need an admin tool to change the number of partitions of a topic online. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-347) change number of partitions of a topic online
[ https://issues.apache.org/jira/browse/KAFKA-347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13713864#comment-13713864 ] Neha Narkhede commented on KAFKA-347: - Overall, this looks great and sorry for not coming to this patch earlier. Few minor comments - 1. KafkaController 1.1 We can read the list of all topics from cache instead of reading from zookeeper, since initializeControllerContext already does that 1.2 In onNewTopicCreation, what was the motivation to move onNewPartitionCreation to before the registration of the listener 2. PartitionStateMachine.AddPartitionsListener controllerContext.partitionReplicaAssignment gets populated during the NewPartition state transition. Due to this, it is best to get rid of the following in the listener, it should happen as part of the NewPartition state change controllerContext.partitionReplicaAssignment.++=(partitionsRemainingToBeAdded) 3. AdminUtils In createOrUpdateTopicPartitionAssignmentPathInZK, please change topic creation - Topic creation 4. There is a conflict in ZookeeperConsumerConnector change number of partitions of a topic online - Key: KAFKA-347 URL: https://issues.apache.org/jira/browse/KAFKA-347 Project: Kafka Issue Type: Improvement Components: core Affects Versions: 0.8 Reporter: Jun Rao Assignee: Sriram Subramanian Labels: features Fix For: 0.8.1 Attachments: kafka-347.patch We will need an admin tool to change the number of partitions of a topic online. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-347) change number of partitions of a topic online
[ https://issues.apache.org/jira/browse/KAFKA-347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13714259#comment-13714259 ] Joel Koshy commented on KAFKA-347: -- Thanks for patch v2. I'm +1 on this as is, but if you can address some of these minor comments that would be great. v2.1 - For num partitions to add vs partitions desired - all I meant was that most of the time users would think of desired number of partitions vs how many more to add. E.g., I have eight partitions for a topic, I now want 20 instead. It is more convenient to just say I want 20 partitions instead of thinking of how many to add. OTOH since we don't support reducing partitions treating it as a num partitions to add is safer. So I don't feel very strongly about it either way. v2.2 - Re: unintentional reassignment of partitions. Yes you are right. v2.3 - Your patch still has ZookeeperConsumerConnector changes in it, so it did not apply cleanly. v2.4 - On checking the replication factor: if we don't allow having a different replication factor for the new partitions we should not even expose it as an option. v2.5 - AddPartitionsListener: no need to change it now, but just a comment: we can directly parse the replica assignment from the data object (instead of reading from zookeeper again) right? v2.6 - On moving getManualReplicaAssignment to AdminUtils - I think it would be good to do that here, but either way is fine. change number of partitions of a topic online - Key: KAFKA-347 URL: https://issues.apache.org/jira/browse/KAFKA-347 Project: Kafka Issue Type: Improvement Components: core Affects Versions: 0.8 Reporter: Jun Rao Assignee: Sriram Subramanian Labels: features Fix For: 0.8.1 Attachments: kafka-347.patch, kafka-347-v2.patch We will need an admin tool to change the number of partitions of a topic online. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-347) change number of partitions of a topic online
[ https://issues.apache.org/jira/browse/KAFKA-347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13713045#comment-13713045 ] Joel Koshy commented on KAFKA-347: -- Thank you for the patch. Couple of comments, all very minor: AddPartitionsCommand: - IMO it is more intuitive for the option to be: total partitions desired as opposed to num partitions to add - It is a bit odd that we can allow some partitions with a different replication factor from what's already there. I don't see any issues with it though. I just think it's a bit odd. One potential issue is if producers explicitly want to set acks to say 3 when there are some partitions with a replication factor of 2 and some with 3 (However, producers really should be using -1 in which case it would be fine). - I think the command can currently allow an unintentional reassignment of replicas since the persistent path is always updated. (or no?) I think this can be easily checked for and avoided. - Apart from start partition id I think getManualReplicaAssignment is identical to CreateTopicCommand's - maybe that code can be moved into AdminUtils? KafkaController: - nitpick: ZkUtils.getAllTopics(zkClient).foreach(p = partitionStateMachine.registerPartitionChangeListener(p)) (can you change p to t :) - p really looks like a partition but it is a topic ) AdminUtils: - the //for testing only comment is now misplaced. - This code is pre-existing, but would prefer changing secondReplicaShift to nextReplicaShift. - Any reason why AddPartitionsTest should not be within AdminTest? - Finally, can you rebase? Sorry for not getting to this patch sooner :( change number of partitions of a topic online - Key: KAFKA-347 URL: https://issues.apache.org/jira/browse/KAFKA-347 Project: Kafka Issue Type: Improvement Components: core Affects Versions: 0.8 Reporter: Jun Rao Assignee: Sriram Subramanian Labels: features Fix For: 0.8.1 Attachments: kafka-347.patch We will need an admin tool to change the number of partitions of a topic online. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-347) change number of partitions of a topic online
[
https://issues.apache.org/jira/browse/KAFKA-347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13707434#comment-13707434
]
Guozhang Wang commented on KAFKA-347:
-
A few comments:
1. In addPartitions function, we can set partitionStartIndex as the last
partition of the partition instead of partition 0 of the topic, since by doing
so as more and more of this function gets called, the broker owning partition
0's leader will have more leader partitions assigned to it.
2. Some of the changes in ZookeeperConsumerConnector for KAFKA-969 is also
included in the patch. Better trim them out for this patch.
3. Are there two getManualReplicaAssignment in AddPartitionsCommand and
CreateTopicCommand with one parameter difference? If yes could we combine them
into one?
Some minor stuff:
1. In AdminUtils, there are two more space indents for line of
ZkUtils.updatePersistentPath(zkClient, zkPath, jsonPartitionData)
2. In createOrUpdateTopicPartitionAssignmentPathInZK, use
if {
//
} else {
//
}
change number of partitions of a topic online
-
Key: KAFKA-347
URL: https://issues.apache.org/jira/browse/KAFKA-347
Project: Kafka
Issue Type: Improvement
Components: core
Affects Versions: 0.8
Reporter: Jun Rao
Assignee: Sriram Subramanian
Labels: features
Fix For: 0.8.1
Attachments: kafka-347.patch
We will need an admin tool to change the number of partitions of a topic
online.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-347) change number of partitions of a topic online
[ https://issues.apache.org/jira/browse/KAFKA-347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13554062#comment-13554062 ] Jay Kreps commented on KAFKA-347: - Well i think there are really two mappings here: key = partition and partition = broker This is the generalization of consistent hashing that most persistent data systems use. In Kafka key=partition is user-defined (Partitioner interface) and defaults to just hash(key)%num_partitions. partition=broker is assigned at topic creation time and from then on is semi-static (changing it is an admin command). So when adding a broker we already can move just the number of partitions we need by having the tool compute the best set of partitions to migrate or choosing at random. So the idea is that you over-partition and then the partition count doesn't change and hence the key=partition assignment doesn't change. The question is, do we need to support changing the number of partitions to handle the case where you don't over-partition by enough? If you do this then the change in mapping would be large. That could be helped a bit by a consistent hash partitioner for the key=partition mapping on the client side, but even in that case you would still have lots of values that are now in the wrong partition, so any code that depended on the partitioning would be broken. Alternately you could do the hard work of actually implementing partition splitting on the broker by having the broker split a partition into two and then migrating the new partitions. The question I would ask is, is any of this worth it? Many data system don't support partition splitting they just say choose your partition count wisely or else delete it and start fresh. Arguably most messaging use cases are particularly concerned with recent data so this might be a fine answer. So an alternate strategy would just be to spend the time working on scaling the number of partitions we can handle and over-partitioning heavily. change number of partitions of a topic online - Key: KAFKA-347 URL: https://issues.apache.org/jira/browse/KAFKA-347 Project: Kafka Issue Type: Sub-task Components: core Affects Versions: 0.8 Reporter: Jun Rao Labels: features We will need an admin tool to change the number of partitions of a topic online. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
