Hi Jazz, I'm afraid the only way of rebalancing old messages is indeed to rewrite them to the topic - thus creating duplication. Once a message has been written to a partition by Kafka this assignment is final, there is no way of moving it to another partition.
Changing the partition count of topics at a later time can be a huge headache, if you depend on partitioning. For this exact reason the general recommendation is to overpartition your topics a little when creating them, so that you can add consumers as the data volume increases. In your case the best solution might be to delete and then recreate the topic with more partitions. Now you can rewrite all your data and it will result in a clean partitioning. Hope this helps a little, feel free to get back to us if you have more questions! Best regards, Sönke On Sun, Jan 27, 2019 at 1:21 PM jaaz jozz <jazzlo...@gmail.com> wrote: > > Hello, > > I have kafka cluster with certain topic that had too few partitions, so a > large backlog of messages was collected. After i added additional > partitions, only the newly messages balanced between all the new partitions. > > What is the preferred way to balance the "old" backlog of messages inside > the original partitions across all the new partitions? > > I thought of reading and writing again all the messages backlog to this > topic and update the offsets accordingly, but it will make duplication of > messages if a new consumer group will start consuming from the beginning of > this topic. > > How can i solve this? > > Thanks. -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
