Agree. Why not write a service that reads from existing topics and writes to new ones. Point existing producers to the new partitions wait for the service to finish reading. Since you have only one partition right now, I'd assume you don't have any partitioning logic per se. Think about that since with multiple partitions you'd want your messages distributed evenly.
On Fri, Sep 19, 2014 at 9:32 PM, Jonathan Weeks <jonathanbwe...@gmail.com> wrote: > I would look at writing a service that reads from your existing topic and > writes to a new topic with (e.g. four) partitions. > > You will also need to pay attention to the partitioning policy (or > implement your own), as the default hashing in the current kafka version > default can lead to poor distribution. > > Best Regards, > > -Jonathan > > > On Sep 19, 2014, at 8:57 AM, Dennis Haller <dhal...@talemetry.com> wrote: > > > Hi, > > > > We have an interesting problem to solve due to a very large traffic > volumes > > on particular topics. In our initial system configuration we had only one > > partition per topic, and in in a couple of topics we have built up huge > > backlogs of several million messages that our consumers are slowly > > processing. > > > > However, now that we have this constant backlog, we wish to repartition > > those topics into several partitions, and allow parallel consumers to run > > to handle the high message volume. > > > > If we simply repartition the topic, say from 1 to 4 partitions, the > > backlogged messages stay in partition 1, while partitions 2,3,4 only get > > newly arrived messages. To eat away the backlog, we need to redistribute > > the backlogged messages evenly among the 4 partitions. > > > > The tools I've seen do not allow me to rewrite or "replay" the existing > > backlogged messages from one partition into the same or another topic > with > > several partitions. - using kafka.tools.MirrorMaker does not allow me to > > move the data within the same zookeeper network, and > > - using kafka.tools.ReplayLogProducer does not write to multiple > > partitions. It seems that it will write only from a single partition to a > > single partition. > > > > Does anyone have any other way to solve this problem or a better way of > > using the kafka tools? > > > > Thanks > > Dennis > >
