I can do most of that I presume. It looks like to set up a separate namespace for zk, I can add /path at the end of each node:port in my zkconnect string, e.g.: zkhost1:123/newnamespace,zkhost2:123/newnamespace right?
For mirroring, there's some vague documentation here: https://cwiki.apache.org/KAFKA/kafka-mirroring-mirrormaker.html Is this the most up to date approach for 0.7.2? Set up a MirrorMaker intermediate process that consumes from the old and produces to the new? I am not able to restart producers one by one (as there are many, on a rather asynchronous update/restart cycle). But I can eventually get them migrated over, etc. Jason On Tue, Mar 19, 2013 at 7:07 PM, Neha Narkhede <neha.narkh...@gmail.com>wrote: > Can you do the following - > > 1. Start a mirror Kafka cluster with the new version on a separate > zookeeper namespace. Configure this to mirror data from the existing kafka > cluster. > 2. Move your consumers to pull data from the mirror > 3. For each producer, one at a time, change the zookeeper namespace to > point to the mirror and restart the producer. > 4. Once the producers have moved to mirror cluster, shutdown mirroring and > old cluster. > > Thanks, > Neha > > On Tuesday, March 19, 2013, Jason Rosenberg wrote: > > > I need to upgrade some kafka broker servers. So I need to seamlessly > > migrate traffic from the old brokers to the new ones, without losing > data, > > and without stopping producers. I can temporarily stop consumers, etc. > > > > Is there a strategy for this? > > > > Also, because of the way we are embedding kafka in our framework, our > > brokerId's are auto-generated (based on hostname, etc.), so I can't > simply > > copy over broker log files, etc., by transferring an old brokerId to a > new > > host. > > > > Is there a way to change the view of the cluster from the producer's > > standpoint, without doing so from the consumers standpoint? That way, > the > > producers can start writing to the new brokers, while the consumers drain > > all data from the old brokers before switching to the new brokers. > > > > I don't actually care about ordering of messages, since the consumers are > > publishing them to a store that will index them properly based on source > > timestamp, etc. > > > > We are using zk for both producers and consumers connections. > > > > This is using 0.7.2. I assume in 0.8 it will be easier, since with > > replication, you can phase in the new servers gradually, etc., no? > > > > Thanks, > > > > Jason > > >
