Sorry for my late reply. I used to watch the mailing list with Grokbase. It somehow stops to update the mailing list and I thought you guys were too busy to reply to us:) In my case, I have 6 topics ech with 6 partitions. The total data volume is about 20G.
For my question 1), is it expected to have only a single partition after it is mirrored? To solve the issue in 1), a topic is created on source and destination at the same time with the same number of partitions. Then I noticed the consumers only fetch part of data then stop. Could you weigh in on that? Thanks. Regards, Libo From: Yu, Libo [ICG-IT] Sent: Monday, June 10, 2013 11:21 AM To: '[email protected]' Subject: rebalance issues from mirrormaker Hi, I come across several critical issues when benchmarking mirrormakers. 1 If a topic has N partitions on source cluster, after mirroring, on the destination cluster it has only one partition. 2 To solve the issue in 1), a topic is created on source and destination at the same time with the same number of partitions. After the topic is created on both clusters, some data is published to source cluster under the created topic. After publishing is done, multiple (3 in this case) mirrormaker processes are brought up on destination cluster to download data from the source. But the following exception occurs: [2013-06-10 10:38:14,955] INFO [libogroup_myhost-1370875086011-c2357858], Committing all offsets after clearing the fetcher queues (kafka.consumer.ZookeeperConsumerConnector) Exception in thread "main" kafka.common.ConsumerRebalanceFailedException: libogroup_ myhost--1370875086011-c2357858 can't rebalance after 4 retries at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:397) at kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:680) at kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.<init>(ZookeeperConsumerConnector.scala:714) at kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:140) at kafka.tools.MirrorMaker$$anonfun$4.apply(MirrorMaker.scala:118) at kafka.tools.MirrorMaker$$anonfun$4.apply(MirrorMaker.scala:118) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61) at scala.collection.immutable.List.foreach(List.scala:45) at scala.collection.TraversableLike$class.map(TraversableLike.scala:206) at scala.collection.immutable.List.map(List.scala:45) at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:118) at kafka.tools.MirrorMaker.main(MirrorMaker.scala) 3) A topic is only created on source cluster. Some data is published to the source cluster under the topic. Now launch multiple (3 in this case) consumer processes at the same time, each of which has three stream. The exception also occurs: Exception in thread "main" [2013-06-10 10:48:28,673] INFO [libogroup_ myhost -1370875697107-38deafbc], begin rebalancing consumer libogroup_myhost-1370875697107-38deafbc try #0 (kafka.consumer.ZookeeperConsumerConnector) kafka.common.ConsumerRebalanceFailedException: libogroup_ myhost -1370875697107-38deafbc can't rebalance after 4 retries at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:397) at kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:680) at kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.<init>(ZookeeperConsumerConnector.scala:714) at kafka.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:140) at kafka.tools.MirrorMaker$$anonfun$4.apply(MirrorMaker.scala:118) at kafka.tools.MirrorMaker$$anonfun$4.apply(MirrorMaker.scala:118) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61) at scala.collection.immutable.List.foreach(List.scala:45) at scala.collection.TraversableLike$class.map(TraversableLike.scala:206) at scala.collection.immutable.List.map(List.scala:45) at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:118) at kafka.tools.MirrorMaker.main(MirrorMaker.scala) Regards, Libo
