I don't think you can mirror messages to a different topic name in the current mirror maker implementation. Mirror maker sends the message to destination topic based on the topic name it reads from source
On Mon, Mar 9, 2015 at 5:00 PM, sunil kalva <sambarc...@gmail.com> wrote: > Can i configure different topic name in destination cluster, i mean can i > have different topic names for source and destination cluster for > mirroring. If yes how can i map source topic with destination topic name ? > > SunilKalva > > On Mon, Mar 9, 2015 at 6:41 AM, tao xiao <xiaotao...@gmail.com> wrote: > > > Ctrl+c is clean shutdown. kill -9 is not > > > > On Mon, Mar 9, 2015 at 2:32 AM, Alex Melville <amelvi...@g.hmc.edu> > wrote: > > > > > What does a "clean shutdown" of the MM entail? So far I've just been > > using > > > Ctrl + C to send an interrupt to kill it. > > > > > > > > > Alex > > > > > > On Sat, Mar 7, 2015 at 10:59 PM, Jiangjie Qin > <j...@linkedin.com.invalid > > > > > > wrote: > > > > > > > If auto.offset.reset is set to smallest, it does not mean the > consumer > > > > will always consume from the smallest. It means that if no previous > > > offset > > > > commit is found for this consumer group, then it will consume from > the > > > > smallest. So for mirror maker, you probably want to always use the > same > > > > consumer group id. This could be configured in the consumer config > file > > > > you pass into mirror maker. > > > > Another thing about duplicate messages is that if mirror maker is > > > shutdown > > > > cleanly, next time when you start it again with same consumer group > id, > > > > there should be no duplicates. But if mirror maker shutdown > > > uncleanly(e.g. > > > > By a kill -9), then next time it starts up you might still have > > duplicate > > > > messages after the last committed offsets. > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > On 3/7/15, 11:45 PM, "sunil kalva" <sambarc...@gmail.com> wrote: > > > > > > > > >Qin > > > > >Partition problem is solved by passing "--new.producer true" option > in > > > > >command line, but adding auto.offset.rese=smallest config, every > > time i > > > > >restart the Mirror tool it copies from starting ends up having lot > of > > > > >duplicate messages in destination cluster. > > > > >Could you please tell me how do i configure to make sure that > > > destination > > > > >cluster is always insync with source cluster. > > > > > > > > > >SunilKalva > > > > > > > > > >On Sun, Mar 8, 2015 at 12:54 AM, Jiangjie Qin > > <j...@linkedin.com.invalid > > > > > > > > >wrote: > > > > > > > > > >> For data not showing up, you need to make sure mirror maker > consumer > > > > >> auto.offset.reset is set to smallest, otherwise when you run > mirror > > > > >>maker > > > > >> for the first time, all the pre-existing messages won¹t be > consumed. > > > > >> For partition sticking, can you verify if your messages are keyed > > > > >>messages > > > > >> or not? If they are not keyed messages, can you check if you are > > using > > > > >>old > > > > >> producer or new producer? For old producer, the default behavior > is > > > > >> sticking to one partition for 10 min and then move to the next > > > > >>partition. > > > > >> So if you wait for more than 10 min, you should see messages in > two > > > > >> different partitions. > > > > >> > > > > >> Jiangjie (Becket) Qin > > > > >> > > > > >> On 3/7/15, 8:28 AM, "sunil kalva" <sambarc...@gmail.com> wrote: > > > > >> > > > > >> >And i also observed ,all the data is moving to one partition in > > > > >> >destination > > > > >> >cluster though i have multiple partitions for that topic in > source > > > and > > > > >> >destination clusters. > > > > >> > > > > > >> >SunilKalva > > > > >> > > > > > >> >On Sat, Mar 7, 2015 at 9:54 PM, sunil kalva < > sambarc...@gmail.com> > > > > >>wrote: > > > > >> > > > > > >> >> I ran kafka mirroring tool after producing data in source > > cluster, > > > > >>and > > > > >> >> this is not copied to destination cluster. If i produce data > > after > > > > >> >>running > > > > >> >> tool those data are copied to destination cluster. Am i missing > > > > >> >>something ? > > > > >> >> > > > > >> >> -- > > > > >> >> SunilKalva > > > > >> >> > > > > >> > > > > > >> > > > > > >> > > > > > >> >-- > > > > >> >SunilKalva > > > > >> > > > > >> > > > > > > > > > > > > > > >-- > > > > >SunilKalva > > > > > > > > > > > > > > > > > > > -- > > Regards, > > Tao > > > > > > -- > SunilKalva > -- Regards, Tao
