What does a "clean shutdown" of the MM entail? So far I've just been using Ctrl + C to send an interrupt to kill it.
Alex On Sat, Mar 7, 2015 at 10:59 PM, Jiangjie Qin <j...@linkedin.com.invalid> wrote: > If auto.offset.reset is set to smallest, it does not mean the consumer > will always consume from the smallest. It means that if no previous offset > commit is found for this consumer group, then it will consume from the > smallest. So for mirror maker, you probably want to always use the same > consumer group id. This could be configured in the consumer config file > you pass into mirror maker. > Another thing about duplicate messages is that if mirror maker is shutdown > cleanly, next time when you start it again with same consumer group id, > there should be no duplicates. But if mirror maker shutdown uncleanly(e.g. > By a kill -9), then next time it starts up you might still have duplicate > messages after the last committed offsets. > > Jiangjie (Becket) Qin > > On 3/7/15, 11:45 PM, "sunil kalva" <sambarc...@gmail.com> wrote: > > >Qin > >Partition problem is solved by passing "--new.producer true" option in > >command line, but adding auto.offset.rese=smallest config, every time i > >restart the Mirror tool it copies from starting ends up having lot of > >duplicate messages in destination cluster. > >Could you please tell me how do i configure to make sure that destination > >cluster is always insync with source cluster. > > > >SunilKalva > > > >On Sun, Mar 8, 2015 at 12:54 AM, Jiangjie Qin <j...@linkedin.com.invalid> > >wrote: > > > >> For data not showing up, you need to make sure mirror maker consumer > >> auto.offset.reset is set to smallest, otherwise when you run mirror > >>maker > >> for the first time, all the pre-existing messages won¹t be consumed. > >> For partition sticking, can you verify if your messages are keyed > >>messages > >> or not? If they are not keyed messages, can you check if you are using > >>old > >> producer or new producer? For old producer, the default behavior is > >> sticking to one partition for 10 min and then move to the next > >>partition. > >> So if you wait for more than 10 min, you should see messages in two > >> different partitions. > >> > >> Jiangjie (Becket) Qin > >> > >> On 3/7/15, 8:28 AM, "sunil kalva" <sambarc...@gmail.com> wrote: > >> > >> >And i also observed ,all the data is moving to one partition in > >> >destination > >> >cluster though i have multiple partitions for that topic in source and > >> >destination clusters. > >> > > >> >SunilKalva > >> > > >> >On Sat, Mar 7, 2015 at 9:54 PM, sunil kalva <sambarc...@gmail.com> > >>wrote: > >> > > >> >> I ran kafka mirroring tool after producing data in source cluster, > >>and > >> >> this is not copied to destination cluster. If i produce data after > >> >>running > >> >> tool those data are copied to destination cluster. Am i missing > >> >>something ? > >> >> > >> >> -- > >> >> SunilKalva > >> >> > >> > > >> > > >> > > >> >-- > >> >SunilKalva > >> > >> > > > > > >-- > >SunilKalva > >
