I think it will be very usefull if we can mirror to a different topic name on destination side. We have a use case to merge data from multiple colos to one central colo.
SunilKalva On Mon, Mar 9, 2015 at 4:29 PM, tao xiao <xiaotao...@gmail.com> wrote: > I don't think you can mirror messages to a different topic name in the > current mirror maker implementation. Mirror maker sends the message to > destination topic based on the topic name it reads from source > > On Mon, Mar 9, 2015 at 5:00 PM, sunil kalva <sambarc...@gmail.com> wrote: > > > Can i configure different topic name in destination cluster, i mean can i > > have different topic names for source and destination cluster for > > mirroring. If yes how can i map source topic with destination topic name > ? > > > > SunilKalva > > > > On Mon, Mar 9, 2015 at 6:41 AM, tao xiao <xiaotao...@gmail.com> wrote: > > > > > Ctrl+c is clean shutdown. kill -9 is not > > > > > > On Mon, Mar 9, 2015 at 2:32 AM, Alex Melville <amelvi...@g.hmc.edu> > > wrote: > > > > > > > What does a "clean shutdown" of the MM entail? So far I've just been > > > using > > > > Ctrl + C to send an interrupt to kill it. > > > > > > > > > > > > Alex > > > > > > > > On Sat, Mar 7, 2015 at 10:59 PM, Jiangjie Qin > > <j...@linkedin.com.invalid > > > > > > > > wrote: > > > > > > > > > If auto.offset.reset is set to smallest, it does not mean the > > consumer > > > > > will always consume from the smallest. It means that if no previous > > > > offset > > > > > commit is found for this consumer group, then it will consume from > > the > > > > > smallest. So for mirror maker, you probably want to always use the > > same > > > > > consumer group id. This could be configured in the consumer config > > file > > > > > you pass into mirror maker. > > > > > Another thing about duplicate messages is that if mirror maker is > > > > shutdown > > > > > cleanly, next time when you start it again with same consumer group > > id, > > > > > there should be no duplicates. But if mirror maker shutdown > > > > uncleanly(e.g. > > > > > By a kill -9), then next time it starts up you might still have > > > duplicate > > > > > messages after the last committed offsets. > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > On 3/7/15, 11:45 PM, "sunil kalva" <sambarc...@gmail.com> wrote: > > > > > > > > > > >Qin > > > > > >Partition problem is solved by passing "--new.producer true" > option > > in > > > > > >command line, but adding auto.offset.rese=smallest config, every > > > time i > > > > > >restart the Mirror tool it copies from starting ends up having lot > > of > > > > > >duplicate messages in destination cluster. > > > > > >Could you please tell me how do i configure to make sure that > > > > destination > > > > > >cluster is always insync with source cluster. > > > > > > > > > > > >SunilKalva > > > > > > > > > > > >On Sun, Mar 8, 2015 at 12:54 AM, Jiangjie Qin > > > <j...@linkedin.com.invalid > > > > > > > > > > >wrote: > > > > > > > > > > > >> For data not showing up, you need to make sure mirror maker > > consumer > > > > > >> auto.offset.reset is set to smallest, otherwise when you run > > mirror > > > > > >>maker > > > > > >> for the first time, all the pre-existing messages won¹t be > > consumed. > > > > > >> For partition sticking, can you verify if your messages are > keyed > > > > > >>messages > > > > > >> or not? If they are not keyed messages, can you check if you are > > > using > > > > > >>old > > > > > >> producer or new producer? For old producer, the default behavior > > is > > > > > >> sticking to one partition for 10 min and then move to the next > > > > > >>partition. > > > > > >> So if you wait for more than 10 min, you should see messages in > > two > > > > > >> different partitions. > > > > > >> > > > > > >> Jiangjie (Becket) Qin > > > > > >> > > > > > >> On 3/7/15, 8:28 AM, "sunil kalva" <sambarc...@gmail.com> wrote: > > > > > >> > > > > > >> >And i also observed ,all the data is moving to one partition in > > > > > >> >destination > > > > > >> >cluster though i have multiple partitions for that topic in > > source > > > > and > > > > > >> >destination clusters. > > > > > >> > > > > > > >> >SunilKalva > > > > > >> > > > > > > >> >On Sat, Mar 7, 2015 at 9:54 PM, sunil kalva < > > sambarc...@gmail.com> > > > > > >>wrote: > > > > > >> > > > > > > >> >> I ran kafka mirroring tool after producing data in source > > > cluster, > > > > > >>and > > > > > >> >> this is not copied to destination cluster. If i produce data > > > after > > > > > >> >>running > > > > > >> >> tool those data are copied to destination cluster. Am i > missing > > > > > >> >>something ? > > > > > >> >> > > > > > >> >> -- > > > > > >> >> SunilKalva > > > > > >> >> > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> >-- > > > > > >> >SunilKalva > > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > >-- > > > > > >SunilKalva > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > Regards, > > > Tao > > > > > > > > > > > -- > > SunilKalva > > > > > > -- > Regards, > Tao > -- SunilKalva
