Hey guys, A couple questions/comments:
1. The callback and user-controlled commit offset functionality is already in the new consumer which we are working on in parallel. If we accelerated that work it might help concentrate efforts. I admit this might take slightly longer in calendar time but could still probably get done this quarter. Have you guys considered that approach? 2. I think partitioning on the hash of the topic partition is not a very good idea because that will make the case of going from a cluster with fewer partitions to one with more partitions not work. I think an intuitive way to do this would be the following: a. Default behavior: Just do what the producer does. I.e. if you specify a key use it for partitioning, if not just partition in a round-robin fashion. b. Add a --preserve-partition option that will explicitly inherent the partition from the source irrespective of whether there is a key or which partition that key would hash to. 3. You don't actually give the ConsumerRebalanceListener interface. What is that going to look like? 4. What is MirrorMakerRecord? I think ideally the MirrorMakerMessageHandler interface would take a ConsumerRecord as input and return a ProducerRecord, right? That would allow you to transform the key, value, partition, or destination topic... 5. Have you guys thought about what the implementation will look like in terms of threading architecture etc with the new consumer? That will be soon so even if we aren't starting with that let's make sure we can get rid of a lot of the current mirror maker accidental complexity in terms of threads and queues when we move to that. -Jay On Tue, Jan 20, 2015 at 4:31 PM, Jiangjie Qin <j...@linkedin.com.invalid> wrote: > Hi Kafka Devs, > > We are working on Kafka Mirror Maker enhancement. A KIP is posted to > document and discuss on the followings: > 1. KAFKA-1650: No Data loss mirror maker change > 2. KAFKA-1839: To allow partition aware mirror. > 3. KAFKA-1840: To allow message filtering/format conversion > Feedbacks are welcome. Please let us know if you have any questions or > concerns. > > Thanks. > > Jiangjie (Becket) Qin >
