Hi, We are using kafka-0.7.2 with zookeeper (3.4.5)
Our cluster configuration: 3 brokers on 3 different machines. Each broker machine has a zookeeper instance running as well. We are currently facing a problem with messages not being consumed by the kafka high level consumers. This is a dedicated topic that gets a very high number of messages. Consumer had no Full GCs. Api called def createMessageStreams(topicCountMap: Map[String,Int]): Map[String, List[KafkaStream[Array[Byte],Array[Byte]]]] We have already handled the exceptions at the consumers level and we have ensured and checked that our threads are not dying. All our consumer thread stack java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x000000065d104ee8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:662) Kafka consumer auto commit thread stack "Kafka-consumer-autocommit-0" prio=10 tid=0x00007f99ad3a2000 nid=0x330b in Object.wait() [0x00007f9989084000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1309) - locked <0x0000000786bdd6c0> (a org.apache.zookeeper.ClientCnxn$Packet) at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1264) at org.I0Itec.zkclient.ZkConnection.writeData(ZkConnection.java:111) at org.I0Itec.zkclient.ZkClient$10.call(ZkClient.java:813) at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675) at org.I0Itec.zkclient.ZkClient.writeData(ZkClient.java:809) at org.I0Itec.zkclient.ZkClient.writeData(ZkClient.java:777) at kafka.utils.ZkUtils$.updatePersistentPath(ZkUtils.scala:103) at kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$2$$anonfun$apply$4.apply(ZookeeperConsumerConnector.scala:251) at kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$2$$anonfun$apply$4.apply(ZookeeperConsumerConnector.scala:248) at scala.collection.Iterator$class.foreach(Iterator.scala:631) at scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:474) at scala.collection.IterableLike$class.foreach(IterableLike.scala:79) at scala.collection.JavaConversions$JCollectionWrapper.foreach(JavaConversions.scala:495) at kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$2.apply(ZookeeperConsumerConnector.scala:248) at kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$2.apply(ZookeeperConsumerConnector.scala:246) at scala.collection.Iterator$class.foreach(Iterator.scala:631) at kafka.utils.Pool$$anon$1.foreach(Pool.scala:53) at scala.collection.IterableLike$class.foreach(IterableLike.scala:79) at kafka.utils.Pool.foreach(Pool.scala:24) at kafka.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:246) at kafka.consumer.ZookeeperConsumerConnector.autoCommit(ZookeeperConsumerConnector.scala:232) at kafka.consumer.ZookeeperConsumerConnector$$anonfun$1.apply$mcV$sp(ZookeeperConsumerConnector.scala:126) at kafka.utils.Utils$$anon$2.run(Utils.scala:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Fetcher Runnable stack "FetchRunnable-0" prio=10 tid=0x00007f997000c000 nid=0x38b0 sleeping[0x00007f9948da2000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at kafka.consumer.FetcherRunnable.run(FetcherRunnable.scala:99) We have defined 300 partitions per broker, and the fetch requests is not processing 857 of those partitions, they are blocked, even though data is being added to those partitions. We are not seeing any unusual logging taking place in either kakfa brokers or the zookeeper clusters. Following is the detail of the fetcher runnable in trace mode. 2013-08-14 23:28:13 TRACE FetcherRunnable:35 - recevied response from fetch request: List(FetchRequest(topic:event_queue, part:290 offset:13858793 maxSize:3145728), FetchRequest(topic:event_queue, part:71 offset:1919388040 maxSize:3145728), FetchRequest(topic:event_queue, part:14 offset:1924428035 maxSize:3145728), FetchRequest(topic:event_queue, part:287 offset:13753665 maxSize:3145728), FetchRequest(topic:event_queue, part:21 offset:1924929744 maxSize:3145728), FetchRequest(topic:event_queue, part:148 offset:237226086 maxSize:3145728), FetchRequest(topic:event_queue, part:220 offset:14046778 maxSize:3145728), FetchRequest(topic:event_queue, part:66 offset:1917452262 maxSize:3145728), FetchRequest(topic:event_queue, part:248 offset:13820576 maxSize:3145728), FetchRequest(topic:event_queue, part:213 offset:14009238 maxSize:3145728), FetchRequest(topic:event_queue, part:176 offset:13889675 maxSize:3145728), FetchRequest(topic:event_queue, part:24 offset:1922108511 maxSize:3145728), FetchRequest(topic:event_queue, part:276 offset:13860230 maxSize:3145728), FetchRequest(topic:event_queue, part:57 offset:1916430767 maxSize:3145728), FetchRequest(topic:event_queue, part:73 offset:1915483219 maxSize:3145728), FetchRequest(topic:event_queue, part:85 offset:238172050 maxSize:3145728), FetchRequest(topic:event_queue, part:150 offset:13902607 maxSize:3145728), FetchRequest(topic:event_queue, part:31 offset:1923158067 maxSize:3145728), FetchRequest(topic:event_queue, part:252 offset:13880253 maxSize:3145728), FetchRequest(topic:event_queue, part:185 offset:13632601 maxSize:3145728), FetchRequest(topic:event_queue, part:241 offset:13870809 maxSize:3145728), FetchRequest(topic:event_queue, part:169 offset:13653452 maxSize:3145728), FetchRequest(topic:event_queue, part:157 offset:13698490 maxSize:3145728), FetchRequest(topic:event_queue, part:84 offset:235590608 maxSize:3145728), FetchRequest(topic:event_queue, part:259 offset:14340519 maxSize:3145728), FetchRequest(topic:event_queue, part:139 offset:235498949 maxSize:3145728), FetchRequest(topic:event_queue, part:112 offset:235697262 maxSize:3145728), FetchRequest(topic:event_queue, part:202 offset:13646959 maxSize:3145728), FetchRequest(topic:event_queue, part:192 offset:13743839 maxSize:3145728), FetchRequest(topic:event_queue, part:42 offset:1922290453 maxSize:3145728), FetchRequest(topic:event_queue, part:122 offset:235463836 maxSize:3145728), FetchRequest(topic:event_queue, part:94 offset:236971652 maxSize:3145728), FetchRequest(topic:event_queue, part:101 offset:235374917 maxSize:3145728), FetchRequest(topic:event_queue, part:280 offset:13739293 maxSize:3145728), FetchRequest(topic:event_queue, part:298 offset:14022233 maxSize:3145728), FetchRequest(topic:event_queue, part:119 offset:235026837 maxSize:3145728), FetchRequest(topic:event_queue, part:49 offset:1924256307 maxSize:3145728), FetchRequest(topic:event_queue, part:168 offset:13845729 maxSize:3145728), FetchRequest(topic:event_queue, part:23 offset:1922510880 maxSize:3145728), FetchRequest(topic:event_queue, part:195 offset:13936005 maxSize:3145728), FetchRequest(topic:event_queue, part:277 offset:13648569 maxSize:3145728), FetchRequest(topic:event_queue, part:230 offset:13905437 maxSize:3145728), FetchRequest(topic:event_queue, part:91 offset:234971350 maxSize:3145728), FetchRequest(topic:event_queue, part:41 offset:1923355011 maxSize:3145728), FetchRequest(topic:event_queue, part:279 offset:14000137 maxSize:3145728), FetchRequest(topic:event_queue, part:65 offset:1914859800 maxSize:3145728), FetchRequest(topic:event_queue, part:221 offset:13826817 maxSize:3145728), FetchRequest(topic:event_queue, part:32 offset:1920897120 maxSize:3145728), FetchRequest(topic:event_queue, part:251 offset:13587337 maxSize:3145728), FetchRequest(topic:event_queue, part:177 offset:13673153 maxSize:3145728), FetchRequest(topic:event_queue, part:288 offset:13738515 maxSize:3145728), FetchRequest(topic:event_queue, part:212 offset:13972659 maxSize:3145728), FetchRequest(topic:event_queue, part:249 offset:13687993 maxSize:3145728), FetchRequest(topic:event_queue, part:270 offset:13973886 maxSize:3145728), FetchRequest(topic:event_queue, part:158 offset:13961991 maxSize:3145728), FetchRequest(topic:event_queue, part:260 offset:13749477 maxSize:3145728), FetchRequest(topic:event_queue, part:13 offset:1922663539 maxSize:3145728), FetchRequest(topic:event_queue, part:56 offset:1919536166 maxSize:3145728), FetchRequest(topic:event_queue, part:223 offset:13634407 maxSize:3145728), FetchRequest(topic:event_queue, part:194 offset:13726728 maxSize:3145728), FetchRequest(topic:event_queue, part:63 offset:1920874310 maxSize:3145728), FetchRequest(topic:event_queue, part:129 offset:238085303 maxSize:3145728), FetchRequest(topic:event_queue, part:4 offset:1926372785 maxSize:3145728), FetchRequest(topic:event_queue, part:120 offset:235299544 maxSize:3145728), FetchRequest(topic:event_queue, part:111 offset:236673617 maxSize:3145728), FetchRequest(topic:event_queue, part:36 offset:1923256197 maxSize:3145728), FetchRequest(topic:event_queue, part:167 offset:13864323 maxSize:3145728), FetchRequest(topic:event_queue, part:278 offset:14512652 maxSize:3145728), FetchRequest(topic:event_queue, part:22 offset:1921366567 maxSize:3145728), FetchRequest(topic:event_queue, part:50 offset:1921299677 maxSize:3145728), FetchRequest(topic:event_queue, part:102 offset:235378466 maxSize:3145728), FetchRequest(topic:event_queue, part:203 offset:13909878 maxSize:3145728), FetchRequest(topic:event_queue, part:269 offset:13730961 maxSize:3145728), FetchRequest(topic:event_queue, part:43 offset:1923207679 maxSize:3145728), FetchRequest(topic:event_queue, part:29 offset:1922837141 maxSize:3145728), FetchRequest(topic:event_queue, part:93 offset:235942183 maxSize:3145728), FetchRequest(topic:event_queue, part:231 offset:13740564 maxSize:3145728), FetchRequest(topic:event_queue, part:15 offset:1924837631 maxSize:3145728), FetchRequest(topic:event_queue, part:174 offset:14167227 maxSize:3145728), FetchRequest(topic:event_queue, part:184 offset:13792167 maxSize:3145728), FetchRequest(topic:event_queue, part:138 offset:237532346 maxSize:3145728), FetchRequest(topic:event_queue, part:222 offset:14037383 maxSize:3145728), FetchRequest(topic:event_queue, part:240 offset:13699592 maxSize:3145728), FetchRequest(topic:event_queue, part:74 offset:1915585918 maxSize:3145728), FetchRequest(topic:event_queue, part:159 offset:13656050 maxSize:3145728), FetchRequest(topic:event_queue, part:145 offset:235316997 maxSize:3145728), FetchRequest(topic:event_queue, part:83 offset:239701975 maxSize:3145728), FetchRequest(topic:event_queue, part:55 offset:1922797177 maxSize:3145728), FetchRequest(topic:event_queue, part:297 offset:13892607 maxSize:3145728), FetchRequest(topic:event_queue, part:186 offset:13827509 maxSize:3145728), FetchRequest(topic:event_queue, part:152 offset:13992704 maxSize:3145728), FetchRequest(topic:event_queue, part:257 offset:13974765 maxSize:3145728), FetchRequest(topic:event_queue, part:130 offset:238644589 maxSize:3145728), FetchRequest(topic:event_queue, part:89 offset:236607948 maxSize:3145728), FetchRequest(topic:event_queue, part:264 offset:14004638 maxSize:3145728), FetchRequest(topic:event_queue, part:64 offset:1920319726 maxSize:3145728), FetchRequest(topic:event_queue, part:103 offset:235407571 maxSize:3145728), FetchRequest(topic:event_queue, part:250 offset:14072244 maxSize:3145728), FetchRequest(topic:event_queue, part:92 offset:236982568 maxSize:3145728), FetchRequest(topic:event_queue, part:166 offset:13706618 maxSize:3145728), FetchRequest(topic:event_queue, part:8 offset:1920802613 maxSize:3145728), FetchRequest(topic:event_queue, part:271 offset:14091149 maxSize:3145728), FetchRequest(topic:event_queue, part:193 offset:13919759 maxSize:3145728), FetchRequest(topic:event_queue, part:200 offset:13964577 maxSize:3145728), FetchRequest(topic:event_queue, part:1 offset:1921914028 maxSize:3145728), FetchRequest(topic:event_queue, part:179 offset:13693807 maxSize:3145728), FetchRequest(topic:event_queue, part:2 offset:1923189002 maxSize:3145728), FetchRequest(topic:event_queue, part:160 offset:13987352 maxSize:3145728), FetchRequest(topic:event_queue, part:82 offset:234913477 maxSize:3145728), FetchRequest(topic:event_queue, part:96 offset:236478691 maxSize:3145728), FetchRequest(topic:event_queue, part:175 offset:13985981 maxSize:3145728), FetchRequest(topic:event_queue, part:30 offset:1928338809 maxSize:3145728), FetchRequest(topic:event_queue, part:44 offset:1927344365 maxSize:3145728), FetchRequest(topic:event_queue, part:61 offset:1917076449 maxSize:3145728), FetchRequest(topic:event_queue, part:54 offset:1918544576 maxSize:3145728), FetchRequest(topic:event_queue, part:137 offset:235429924 maxSize:3145728), FetchRequest(topic:event_queue, part:146 offset:234024621 maxSize:3145728), FetchRequest(topic:event_queue, part:68 offset:1915824648 maxSize:3145728), FetchRequest(topic:event_queue, part:207 offset:13983596 maxSize:3145728), FetchRequest(topic:event_queue, part:214 offset:13843173 maxSize:3145728), FetchRequest(topic:event_queue, part:131 offset:239218119 maxSize:3145728), FetchRequest(topic:event_queue, part:75 offset:234730864 maxSize:3145728), FetchRequest(topic:event_queue, part:242 offset:13924920 maxSize:3145728), FetchRequest(topic:event_queue, part:272 offset:14129560 maxSize:3145728), FetchRequest(topic:event_queue, part:16 offset:1923561885 maxSize:3145728), FetchRequest(topic:event_queue, part:124 offset:236152284 maxSize:3145728), FetchRequest(topic:event_queue, part:187 offset:14249552 maxSize:3145728), FetchRequest(topic:event_queue, part:81 offset:234178502 maxSize:3145728), FetchRequest(topic:event_queue, part:228 offset:14033290 maxSize:3145728), FetchRequest(topic:event_queue, part:165 offset:14133744 maxSize:3145728), FetchRequest(topic:event_queue, part:110 offset:236192852 maxSize:3145728), FetchRequest(topic:event_queue, part:258 offset:14139932 maxSize:3145728), FetchRequest(topic:event_queue, part:151 offset:14008504 maxSize:3145728), FetchRequest(topic:event_queue, part:117 offset:237862268 maxSize:3145728), FetchRequest(topic:event_queue, part:172 offset:13823876 maxSize:3145728), FetchRequest(topic:event_queue, part:215 offset:13807059 maxSize:3145728), FetchRequest(topic:event_queue, part:243 offset:13579687 maxSize:3145728), FetchRequest(topic:event_queue, part:299 offset:14130546 maxSize:3145728), FetchRequest(topic:event_queue, part:17 offset:1924461681 maxSize:3145728), FetchRequest(topic:event_queue, part:45 offset:1924311439 maxSize:3145728), FetchRequest(topic:event_queue, part:132 offset:238453123 maxSize:3145728), FetchRequest(topic:event_queue, part:285 offset:13712821 maxSize:3145728), FetchRequest(topic:event_queue, part:154 offset:13729543 maxSize:3145728), FetchRequest(topic:event_queue, part:229 offset:14063729 maxSize:3145728), FetchRequest(topic:event_queue, part:201 offset:14202133 maxSize:3145728), FetchRequest(topic:event_queue, part:161 offset:14083985 maxSize:3145728), FetchRequest(topic:event_queue, part:62 offset:1920871581 maxSize:3145728), FetchRequest(topic:event_queue, part:95 offset:236078056 maxSize:3145728), FetchRequest(topic:event_queue, part:76 offset:235057152 maxSize:3145728), FetchRequest(topic:event_queue, part:67 offset:1918530968 maxSize:3145728), FetchRequest(topic:event_queue, part:118 offset:236009547 maxSize:3145728), FetchRequest(topic:event_queue, part:10 offset:1920294682 maxSize:3145728), FetchRequest(topic:event_queue, part:286 offset:14140007 maxSize:3145728), FetchRequest(topic:event_queue, part:90 offset:235531660 maxSize:3145728), FetchRequest(topic:event_queue, part:164 offset:13949421 maxSize:3145728), FetchRequest(topic:event_queue, part:234 offset:14068535 maxSize:3145728), FetchRequest(topic:event_queue, part:38 offset:1921910104 maxSize:3145728), FetchRequest(topic:event_queue, part:181 offset:13684468 maxSize:3145728), FetchRequest(topic:event_queue, part:7 offset:1922581694 maxSize:3145728), FetchRequest(topic:event_queue, part:35 offset:1919656737 maxSize:3145728), FetchRequest(topic:event_queue, part:109 offset:238515986 maxSize:3145728), FetchRequest(topic:event_queue, part:188 offset:14030914 maxSize:3145728), FetchRequest(topic:event_queue, part:104 offset:237177303 maxSize:3145728), FetchRequest(topic:event_queue, part:123 offset:235236776 maxSize:3145728), FetchRequest(topic:event_queue, part:0 offset:1921886214 maxSize:3145728), FetchRequest(topic:event_queue, part:245 offset:13954109 maxSize:3145728), FetchRequest(topic:event_queue, part:136 offset:236161678 maxSize:3145728), FetchRequest(topic:event_queue, part:199 offset:14224604 maxSize:3145728), FetchRequest(topic:event_queue, part:227 offset:14037136 maxSize:3145728), FetchRequest(topic:event_queue, part:143 offset:235733339 maxSize:3145728), FetchRequest(topic:event_queue, part:206 offset:14198134 maxSize:3145728), FetchRequest(topic:event_queue, part:216 offset:13736453 maxSize:3145728), FetchRequest(topic:event_queue, part:18 offset:1921683680 maxSize:3145728), FetchRequest(topic:event_queue, part:237 offset:13669627 maxSize:3145728), FetchRequest(topic:event_queue, part:126 offset:235772779 maxSize:3145728), FetchRequest(topic:event_queue, part:209 offset:13803127 maxSize:3145728), FetchRequest(topic:event_queue, part:294 offset:13930318 maxSize:3145728), FetchRequest(topic:event_queue, part:153 offset:14247805 maxSize:3145728), FetchRequest(topic:event_queue, part:59 offset:1916170361 maxSize:3145728), FetchRequest(topic:event_queue, part:80 offset:236311660 maxSize:3145728), FetchRequest(topic:event_queue, part:87 offset:235675888 maxSize:3145728), FetchRequest(topic:event_queue, part:273 offset:13894364 maxSize:3145728), FetchRequest(topic:event_queue, part:291 offset:14120121 maxSize:3145728), FetchRequest(topic:event_queue, part:37 offset:1926437324 maxSize:3145728), FetchRequest(topic:event_queue, part:28 offset:1923884304 maxSize:3145728), FetchRequest(topic:event_queue, part:244 offset:13639986 maxSize:3145728), FetchRequest(topic:event_queue, part:162 offset:13631619 maxSize:3145728), FetchRequest(topic:event_queue, part:266 offset:14276657 maxSize:3145728), FetchRequest(topic:event_queue, part:133 offset:234190625 maxSize:3145728), FetchRequest(topic:event_queue, part:173 offset:13718276 maxSize:3145728), FetchRequest(topic:event_queue, part:284 offset:13634149 maxSize:3145728), FetchRequest(topic:event_queue, part:88 offset:239251670 maxSize:3145728), FetchRequest(topic:event_queue, part:256 offset:13963549 maxSize:3145728), FetchRequest(topic:event_queue, part:235 offset:14268066 maxSize:3145728), FetchRequest(topic:event_queue, part:180 offset:13994435 maxSize:3145728), FetchRequest(topic:event_queue, part:77 offset:237671159 maxSize:3145728), FetchRequest(topic:event_queue, part:108 offset:236023907 maxSize:3145728), FetchRequest(topic:event_queue, part:263 offset:13953716 maxSize:3145728), FetchRequest(topic:event_queue, part:70 offset:1919192226 maxSize:3145728), FetchRequest(topic:event_queue, part:142 offset:234821981 maxSize:3145728), FetchRequest(topic:event_queue, part:105 offset:236803009 maxSize:3145728), FetchRequest(topic:event_queue, part:156 offset:14194305 maxSize:3145728), FetchRequest(topic:event_queue, part:144 offset:235378461 maxSize:3145728), FetchRequest(topic:event_queue, part:46 offset:1928419689 maxSize:3145728), FetchRequest(topic:event_queue, part:198 offset:13974447 maxSize:3145728), FetchRequest(topic:event_queue, part:149 offset:238015271 maxSize:3145728), FetchRequest(topic:event_queue, part:115 offset:235811382 maxSize:3145728), FetchRequest(topic:event_queue, part:98 offset:235100030 maxSize:3145728), FetchRequest(topic:event_queue, part:265 offset:13907349 maxSize:3145728), FetchRequest(topic:event_queue, part:274 offset:13864687 maxSize:3145728), FetchRequest(topic:event_queue, part:226 offset:13537827 maxSize:3145728), FetchRequest(topic:event_queue, part:9 offset:1923454590 maxSize:3145728), FetchRequest(topic:event_queue, part:163 offset:13882392 maxSize:3145728), FetchRequest(topic:event_queue, part:255 offset:13946694 maxSize:3145728), FetchRequest(topic:event_queue, part:60 offset:1922506946 maxSize:3145728), FetchRequest(topic:event_queue, part:134 offset:236241772 maxSize:3145728), FetchRequest(topic:event_queue, part:293 offset:14174207 maxSize:3145728), FetchRequest(topic:event_queue, part:69 offset:1917935964 maxSize:3145728), FetchRequest(topic:event_queue, part:53 offset:1920040686 maxSize:3145728), FetchRequest(topic:event_queue, part:189 offset:14081801 maxSize:3145728), FetchRequest(topic:event_queue, part:125 offset:235462272 maxSize:3145728), FetchRequest(topic:event_queue, part:27 offset:1922583519 maxSize:3145728), FetchRequest(topic:event_queue, part:107 offset:235043816 maxSize:3145728), FetchRequest(topic:event_queue, part:292 offset:14225910 maxSize:3145728), FetchRequest(topic:event_queue, part:217 offset:14155810 maxSize:3145728), FetchRequest(topic:event_queue, part:208 offset:13869168 maxSize:3145728), FetchRequest(topic:event_queue, part:79 offset:237019821 maxSize:3145728), FetchRequest(topic:event_queue, part:171 offset:13921149 maxSize:3145728), FetchRequest(topic:event_queue, part:218 offset:14012275 maxSize:3145728), FetchRequest(topic:event_queue, part:52 offset:1920570629 maxSize:3145728), FetchRequest(topic:event_queue, part:283 offset:13970846 maxSize:3145728), FetchRequest(topic:event_queue, part:47 offset:1924652182 maxSize:3145728), FetchRequest(topic:event_queue, part:232 offset:14499312 maxSize:3145728), FetchRequest(topic:event_queue, part:211 offset:13847224 maxSize:3145728), FetchRequest(topic:event_queue, part:141 offset:237422517 maxSize:3145728), FetchRequest(topic:event_queue, part:106 offset:237027686 maxSize:3145728), FetchRequest(topic:event_queue, part:26 offset:1921991622 maxSize:3145728), FetchRequest(topic:event_queue, part:236 offset:13700792 maxSize:3145728), FetchRequest(topic:event_queue, part:190 offset:14048595 maxSize:3145728), FetchRequest(topic:event_queue, part:225 offset:13975669 maxSize:3145728), FetchRequest(topic:event_queue, part:40 offset:1922256327 maxSize:3145728), FetchRequest(topic:event_queue, part:116 offset:235790080 maxSize:3145728), FetchRequest(topic:event_queue, part:33 offset:1922681776 maxSize:3145728), FetchRequest(topic:event_queue, part:197 offset:13786691 maxSize:3145728), FetchRequest(topic:event_queue, part:5 offset:1924902525 maxSize:3145728), FetchRequest(topic:event_queue, part:178 offset:13756870 maxSize:3145728), FetchRequest(topic:event_queue, part:204 offset:14337005 maxSize:3145728), FetchRequest(topic:event_queue, part:97 offset:236192027 maxSize:3145728), FetchRequest(topic:event_queue, part:239 offset:13759979 maxSize:3145728), FetchRequest(topic:event_queue, part:78 offset:233418728 maxSize:3145728), FetchRequest(topic:event_queue, part:121 offset:235819188 maxSize:3145728), FetchRequest(topic:event_queue, part:12 offset:1922738965 maxSize:3145728), FetchRequest(topic:event_queue, part:114 offset:235220219 maxSize:3145728), FetchRequest(topic:event_queue, part:289 offset:14041944 maxSize:3145728), FetchRequest(topic:event_queue, part:261 offset:13667852 maxSize:3145728), FetchRequest(topic:event_queue, part:183 offset:13697688 maxSize:3145728), FetchRequest(topic:event_queue, part:254 offset:13772026 maxSize:3145728), FetchRequest(topic:event_queue, part:246 offset:13545843 maxSize:3145728), FetchRequest(topic:event_queue, part:19 offset:1925964016 maxSize:3145728), FetchRequest(topic:event_queue, part:25 offset:1924976308 maxSize:3145728), FetchRequest(topic:event_queue, part:282 offset:13591865 maxSize:3145728), FetchRequest(topic:event_queue, part:275 offset:13649678 maxSize:3145728), FetchRequest(topic:event_queue, part:155 offset:13665620 maxSize:3145728), FetchRequest(topic:event_queue, part:128 offset:235160241 maxSize:3145728), FetchRequest(topic:event_queue, part:268 offset:13889869 maxSize:3145728), FetchRequest(topic:event_queue, part:135 offset:235842584 maxSize:3145728), FetchRequest(topic:event_queue, part:72 offset:1915802630 maxSize:3145728), FetchRequest(topic:event_queue, part:296 offset:13958874 maxSize:3145728), FetchRequest(topic:event_queue, part:39 offset:1923280812 maxSize:3145728), FetchRequest(topic:event_queue, part:51 offset:1919445372 maxSize:3145728), FetchRequest(topic:event_queue, part:20 offset:1926233046 maxSize:3145728), FetchRequest(topic:event_queue, part:6 offset:1923071675 maxSize:3145728), FetchRequest(topic:event_queue, part:34 offset:1925185060 maxSize:3145728), FetchRequest(topic:event_queue, part:267 offset:14059111 maxSize:3145728), FetchRequest(topic:event_queue, part:11 offset:1921467904 maxSize:3145728), FetchRequest(topic:event_queue, part:48 offset:1924981816 maxSize:3145728), FetchRequest(topic:event_queue, part:219 offset:13847303 maxSize:3145728), FetchRequest(topic:event_queue, part:295 offset:13759737 maxSize:3145728), FetchRequest(topic:event_queue, part:170 offset:13809787 maxSize:3145728), FetchRequest(topic:event_queue, part:58 offset:1920889286 maxSize:3145728), FetchRequest(topic:event_queue, part:205 offset:13929926 maxSize:3145728), FetchRequest(topic:event_queue, part:247 offset:14051637 maxSize:3145728), FetchRequest(topic:event_queue, part:224 offset:13651775 maxSize:3145728), FetchRequest(topic:event_queue, part:147 offset:235945537 maxSize:3145728), FetchRequest(topic:event_queue, part:233 offset:13648633 maxSize:3145728), FetchRequest(topic:event_queue, part:100 offset:235475945 maxSize:3145728), FetchRequest(topic:event_queue, part:86 offset:236263448 maxSize:3145728), FetchRequest(topic:event_queue, part:113 offset:234976306 maxSize:3145728), FetchRequest(topic:event_queue, part:99 offset:235273075 maxSize:3145728), FetchRequest(topic:event_queue, part:210 offset:13753055 maxSize:3145728), FetchRequest(topic:event_queue, part:127 offset:235577543 maxSize:3145728), FetchRequest(topic:event_queue, part:262 offset:14314531 maxSize:3145728), FetchRequest(topic:event_queue, part:182 offset:13947552 maxSize:3145728), FetchRequest(topic:event_queue, part:191 offset:14075203 maxSize:3145728), FetchRequest(topic:event_queue, part:238 offset:14341058 maxSize:3145728), FetchRequest(topic:event_queue, part:253 offset:13961191 maxSize:3145728), FetchRequest(topic:event_queue, part:3 offset:1921808922 maxSize:3145728), FetchRequest(topic:event_queue, part:140 offset:235461115 maxSize:3145728), FetchRequest(topic:event_queue, part:281 offset:14095956 maxSize:3145728), FetchRequest(topic:event_queue, part:196 offset:13767437 maxSize:3145728)) 2013-08-14 23:28:13 TRACE FetcherRunnable:35 - fetched bytes: 0 2013-08-14 23:28:13 DEBUG FetcherRunnable:48 - backing off 10 ms -- Best Regards, Ramakrishna Mallya
