Thanks for the answers. Much clearer now :) Unrelated question: How do you test MirrorMaker (especially around data loss)? I didn't see any unit-tests or integration tests in trunk.
Gwen On Wed, Jan 21, 2015 at 9:55 AM, Jiangjie Qin <j...@linkedin.com.invalid> wrote: > Hi Gwen, > > Please see inline answers. I¹ll update them in the KIP as well. > > Thanks. > > Jiangjie (Becket) Qin > > On 1/20/15, 6:39 PM, "Gwen Shapira" <gshap...@cloudera.com> wrote: > >>Thanks for the detailed document, Jiangjie. Super helpful. >> >>Few questions: >> >>1. You mention that "A ConsumerRebalanceListener class is created and >>could be wired into ZookeeperConsumerConnector to avoid duplicate >>messages when consumer rebalance occurs in mirror maker." >> >>Is this something the user needs to do or configure? or is the wiring >>of rebalance listener into the zookeeper consumer will be part of the >>enhancement? >>In other words, will we need to do anything extra to avoid duplicates >>during rebalance in MirrorMaker? > For ZookeeperConsumerConnector in general, users need to wire in listener > by themselves in code. > For Mirror Maker, an internal rebalance listener has been wired in by > default to avoid duplicates on consumer rebalance. User could still > specify a custom listener class in command line argument, the internal > rebalance listener will call that listener after it finishes the default > logic. >> >>2. "The only source of truth for offsets in consume-then-send pattern >>is end user." - I assume you don't mean an actual person, right? So >>what does "end user" refer to? Can you clarify when will the offset >>commit thread commit offsets? And which JIRA implements this? > By end user I mean the target cluster here. The offset commit thread > commit thread periodically. It only commit the offsets that have been > acked. >> >>3. Maintaining message order - In which JIRA do we implement this part? > KAFKA-1650 >> >>Again, thanks a lot for documenting this and even more for the >>implementation - it is super important for many use cases. >> >>Gwen >> >> >>Gwen >> >>On Tue, Jan 20, 2015 at 4:31 PM, Jiangjie Qin <j...@linkedin.com.invalid> >>wrote: >>> Hi Kafka Devs, >>> >>> We are working on Kafka Mirror Maker enhancement. A KIP is posted to >>>document and discuss on the followings: >>> 1. KAFKA-1650: No Data loss mirror maker change >>> 2. KAFKA-1839: To allow partition aware mirror. >>> 3. KAFKA-1840: To allow message filtering/format conversion >>> Feedbacks are welcome. Please let us know if you have any questions or >>>concerns. >>> >>> Thanks. >>> >>> Jiangjie (Becket) Qin >
