Makes sense. This is a non issue in 0.8, see this - https://issues.apache.org/jira/browse/KAFKA-155#comment-13607766
Thanks, Neha On Wed, Mar 20, 2013 at 8:41 AM, Jason Rosenberg <j...@squareup.com> wrote: > I think might be cool, would be to have a feature where by you can tell a > broker to stop accepting new data produced to it, but still allow consumers > to consume from it. > > That way, you can roll out new brokers to a cluster, turn off producing to > the old nodes, then wait for the log retention period, and then remove the > old nodes from the cluster. > > Does that make sense? Could it be easily done? > > Or is all this a non-issue anyway in 0.8? > > Maybe I should just wait for 0.8 to be ready, before doing my migration > anyway..... > > Jason > > On Wed, Mar 20, 2013 at 6:42 AM, Neha Narkhede <neha.narkh...@gmail.com > >wrote: > > > The zookeeper connection URL with namespace can be > > zkhost1:123,zkhost2:123/newnamespace > > > > The wiki is up to date for Kafka 0.7.2. There is no officially supported > > feature to do that sort of migration, I suggested one approach that I > could > > think of :-) > > > > Thanks, > > Neha > > > > On Tuesday, March 19, 2013, Jason Rosenberg wrote: > > > > > I can do most of that I presume. > > > > > > It looks like to set up a separate namespace for zk, I can add /path at > > the > > > end of each node:port in my zkconnect string, e.g.: > > > zkhost1:123/newnamespace,zkhost2:123/newnamespace > > > right? > > > > > > For mirroring, there's some vague documentation here: > > > https://cwiki.apache.org/KAFKA/kafka-mirroring-mirrormaker.html > > > Is this the most up to date approach for 0.7.2? Set up a MirrorMaker > > > intermediate process that consumes from the old and produces to the > new? > > > > > > I am not able to restart producers one by one (as there are many, on a > > > rather asynchronous update/restart cycle). But I can eventually get > them > > > migrated over, etc. > > > > > > Jason > > > > > > On Tue, Mar 19, 2013 at 7:07 PM, Neha Narkhede < > neha.narkh...@gmail.com > > <javascript:;> > > > >wrote: > > > > > > > Can you do the following - > > > > > > > > 1. Start a mirror Kafka cluster with the new version on a separate > > > > zookeeper namespace. Configure this to mirror data from the existing > > > kafka > > > > cluster. > > > > 2. Move your consumers to pull data from the mirror > > > > 3. For each producer, one at a time, change the zookeeper namespace > to > > > > point to the mirror and restart the producer. > > > > 4. Once the producers have moved to mirror cluster, shutdown > mirroring > > > and > > > > old cluster. > > > > > > > > Thanks, > > > > Neha > > > > > > > > On Tuesday, March 19, 2013, Jason Rosenberg wrote: > > > > > > > > > I need to upgrade some kafka broker servers. So I need to > seamlessly > > > > > migrate traffic from the old brokers to the new ones, without > losing > > > > data, > > > > > and without stopping producers. I can temporarily stop consumers, > > etc. > > > > > > > > > > Is there a strategy for this? > > > > > > > > > > Also, because of the way we are embedding kafka in our framework, > our > > > > > brokerId's are auto-generated (based on hostname, etc.), so I can't > > > > simply > > > > > copy over broker log files, etc., by transferring an old brokerId > to > > a > > > > new > > > > > host. > > > > > > > > > > Is there a way to change the view of the cluster from the > producer's > > > > > standpoint, without doing so from the consumers standpoint? That > > way, > > > > the > > > > > producers can start writing to the new brokers, while the consumers > > > drain > > > > > all data from the old brokers before switching to the new brokers. > > > > > > > > > > I don't actually care about ordering of messages, since the > consumers > > > are > > > > > publishing them to a store that will index them properly based on > > > source > > > > > timestamp, etc. > > > > > > > > > > We are using zk for both producers and consumers connections. > > > > > > > > > > This is using 0.7.2. I assume in 0.8 it will be easier, since with > > > > > replication, you can phase in the new servers gradually, etc., no? > > > > > > > > > > Thanks, > > > > > > > > > > Jason > > > > > > > > > > > > > > >
