Hi Mayuresh, Here are the logs.
Broker-4 [2015-03-13 17:49:40,514] INFO Partition [Topic22kv,5] on broker 4: Shrinking ISR for partition [Topic22kv,5] from 2,4,3 to 2,4 (kafka.cluster.Partition) [2015-03-13 17:49:40,514] INFO Partition [Topic22kv,5] on broker 4: Shrinking ISR for partition [Topic22kv,5] from 2,4,3 to 2,4 (kafka.cluster.Partition) [2015-03-13 17:49:40,514] INFO Partition [Topic22kv,5] on broker 4: Shrinking ISR for partition [Topic22kv,5] from 2,4,3 to 2,4 (kafka.cluster.Partition) [2015-03-13 17:49:40,762] INFO Partition [Topic22kv,5] on broker 4: Expanding ISR for partition [Topic22kv,5] from 2,4 to 2,4,3 (kafka.cluster.Partition) [2015-03-13 18:47:10,514] INFO Partition [Topic22kv,5] on broker 4: Shrinking ISR for partition [Topic22kv,5] from 2,4,3 to 2,4 (kafka.cluster.Partition) [2015-03-13 18:47:10,514] INFO Partition [Topic22kv,5] on broker 4: Shrinking ISR for partition [Topic22kv,5] from 2,4,3 to 2,4 (kafka.cluster.Partition) [2015-03-13 18:47:10,514] INFO Partition [Topic22kv,5] on broker 4: Shrinking ISR for partition [Topic22kv,5] from 2,4,3 to 2,4 (kafka.cluster.Partition) [2015-03-13 18:47:11,539] INFO Partition [Topic22kv,5] on broker 4: Expanding ISR for partition [Topic22kv,5] from 2,4 to 2,4,3 (kafka.cluster.Partition) [2015-03-13 20:09:10,460] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001213156892.index.deleted (kafka.log.OffsetIndex) [2015-03-13 20:23:10,377] INFO Scheduling log segment 1218520176 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 20:24:10,377] INFO Deleting segment 1218520176 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 20:24:10,444] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001218520176.index.deleted (kafka.log.OffsetIndex) [2015-03-13 20:26:28,789] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-13 20:38:10,333] INFO Scheduling log segment 1223883126 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 20:39:10,334] INFO Deleting segment 1223883126 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 20:39:10,478] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001223883126.index.deleted (kafka.log.OffsetIndex) [2015-03-13 20:46:16,924] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-13 20:53:10,370] INFO Scheduling log segment 1229245987 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 20:54:10,371] INFO Deleting segment 1229245987 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 20:54:10,444] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001229245987.index.deleted (kafka.log.OffsetIndex) [2015-03-13 21:03:10,347] INFO Scheduling log segment 1234609321 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 21:04:10,348] INFO Deleting segment 1234609321 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 21:04:10,519] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001234609321.index.deleted (kafka.log.OffsetIndex) [2015-03-13 21:06:07,810] INFO Rolled new log segment for 'Topic22kv-5' in 0 ms. (kafka.log.Log) [2015-03-13 21:18:10,355] INFO Scheduling log segment 1239972496 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 21:19:10,408] INFO Deleting segment 1239972496 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 21:19:10,516] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001239972496.index.deleted (kafka.log.OffsetIndex) [2015-03-13 21:25:49,058] INFO Rolled new log segment for 'Topic22kv-5' in 0 ms. (kafka.log.Log) [2015-03-13 21:38:10,344] INFO Scheduling log segment 1245335417 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 21:39:10,443] INFO Deleting segment 1245335417 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 21:39:10,539] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001245335417.index.deleted (kafka.log.OffsetIndex) [2015-03-13 21:45:30,609] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-13 21:53:10,340] INFO Scheduling log segment 1250698493 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 21:54:10,340] INFO Deleting segment 1250698493 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 21:54:10,456] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001250698493.index.deleted (kafka.log.OffsetIndex) [2015-03-13 22:05:31,719] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-13 22:13:10,333] INFO Scheduling log segment 1256061631 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 22:14:10,333] INFO Deleting segment 1256061631 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 22:14:10,406] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001256061631.index.deleted (kafka.log.OffsetIndex) [2015-03-13 22:25:12,659] INFO Rolled new log segment for 'Topic22kv-5' in 0 ms. (kafka.log.Log) [2015-03-13 22:33:10,390] INFO Scheduling log segment 1261424153 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 22:34:10,390] INFO Deleting segment 1261424153 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 22:34:10,453] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001261424153.index.deleted (kafka.log.OffsetIndex) [2015-03-13 22:44:50,152] INFO Rolled new log segment for 'Topic22kv-5' in 0 ms. (kafka.log.Log) [2015-03-13 22:48:10,341] INFO Scheduling log segment 1266787556 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 22:49:10,341] INFO Deleting segment 1266787556 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 22:49:10,413] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001266787556.index.deleted (kafka.log.OffsetIndex) [2015-03-13 23:01:10,514] INFO Partition [Topic22kv,5] on broker 4: Shrinking ISR for partition [Topic22kv,5] from 2,4,3 to 2,4 (kafka.cluster.Partition) [2015-03-13 23:01:10,514] INFO Partition [Topic22kv,5] on broker 4: Shrinking ISR for partition [Topic22kv,5] from 2,4,3 to 2,4 (kafka.cluster.Partition) [2015-03-13 23:01:10,514] INFO Partition [Topic22kv,5] on broker 4: Shrinking ISR for partition [Topic22kv,5] from 2,4,3 to 2,4 (kafka.cluster.Partition) [2015-03-13 23:01:10,514] INFO Partition [Topic22kv,5] on broker 4: Shrinking ISR for partition [Topic22kv,5] from 2,4,3 to 2,4 (kafka.cluster.Partition) [2015-03-13 23:01:11,572] INFO Partition [Topic22kv,5] on broker 4: Expanding ISR for partition [Topic22kv,5] from 2,4 to 2,4,3 (kafka.cluster.Partition) [2015-03-13 23:01:11,572] INFO Partition [Topic22kv,5] on broker 4: Expanding ISR for partition [Topic22kv,5] from 2,4 to 2,4,3 (kafka.cluster.Partition) [2015-03-13 23:04:27,284] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-13 23:08:10,357] INFO Scheduling log segment 1272150400 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 23:09:10,417] INFO Deleting segment 1272150400 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 23:09:10,483] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001272150400.index.deleted (kafka.log.OffsetIndex) [2015-03-13 23:23:10,356] INFO Scheduling log segment 1277513197 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 23:23:33,321] INFO Rolled new log segment for 'Topic22kv-5' in 0 ms. (kafka.log.Log) [2015-03-13 23:24:10,356] INFO Deleting segment 1277513197 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 23:24:10,479] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001277513197.index.deleted (kafka.log.OffsetIndex) [2015-03-13 23:42:20,541] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-13 23:43:10,383] INFO Scheduling log segment 1282876266 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 23:44:10,384] INFO Deleting segment 1282876266 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 23:44:10,473] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001282876266.index.deleted (kafka.log.OffsetIndex) [2015-03-14 00:00:40,369] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 00:03:10,336] INFO Scheduling log segment 1288239314 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 00:04:10,336] INFO Deleting segment 1288239314 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 00:04:10,412] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001288239314.index.deleted (kafka.log.OffsetIndex) [2015-03-14 00:19:08,517] INFO Rolled new log segment for 'Topic22kv-5' in 0 ms. (kafka.log.Log) [2015-03-14 00:28:10,334] INFO Scheduling log segment 1293602624 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 00:29:10,334] INFO Deleting segment 1293602624 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 00:29:10,398] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001293602624.index.deleted (kafka.log.OffsetIndex) [2015-03-14 00:37:25,862] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 00:53:10,358] INFO Scheduling log segment 1298965422 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 00:54:10,360] INFO Deleting segment 1298965422 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 00:54:10,534] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001298965422.index.deleted (kafka.log.OffsetIndex) [2015-03-14 00:56:01,104] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 01:13:10,335] INFO Scheduling log segment 1304328119 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 01:14:10,336] INFO Deleting segment 1304328119 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 01:14:10,417] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001304328119.index.deleted (kafka.log.OffsetIndex) [2015-03-14 01:14:35,137] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 01:33:12,497] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 01:38:10,406] INFO Scheduling log segment 1309691074 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 01:39:10,463] INFO Deleting segment 1309691074 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 01:39:10,576] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001309691074.index.deleted (kafka.log.OffsetIndex) [2015-03-14 01:51:36,954] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 02:03:10,363] INFO Scheduling log segment 1315054067 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 02:04:10,412] INFO Deleting segment 1315054067 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 02:04:10,463] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001315054067.index.deleted (kafka.log.OffsetIndex) [2015-03-14 02:09:55,931] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 02:20:10,514] INFO Partition [Topic22kv,5] on broker 4: Shrinking ISR for partition [Topic22kv,5] from 2,4,3 to 2,4 (kafka.cluster.Partition) [2015-03-14 02:20:10,514] INFO Partition [Topic22kv,5] on broker 4: Shrinking ISR for partition [Topic22kv,5] from 2,4,3 to 2,4 (kafka.cluster.Partition) [2015-03-14 02:20:10,514] INFO Partition [Topic22kv,5] on broker 4: Shrinking ISR for partition [Topic22kv,5] from 2,4,3 to 2,4 (kafka.cluster.Partition) [2015-03-14 02:20:10,514] INFO Partition [Topic22kv,5] on broker 4: Shrinking ISR for partition [Topic22kv,5] from 2,4,3 to 2,4 (kafka.cluster.Partition) [2015-03-14 02:20:10,780] INFO Partition [Topic22kv,5] on broker 4: Expanding ISR for partition [Topic22kv,5] from 2,4 to 2,4,3 (kafka.cluster.Partition) [2015-03-14 02:20:10,780] INFO Partition [Topic22kv,5] on broker 4: Expanding ISR for partition [Topic22kv,5] from 2,4 to 2,4,3 (kafka.cluster.Partition) [2015-03-14 02:23:10,377] INFO Scheduling log segment 1320417148 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 02:24:10,401] INFO Deleting segment 1320417148 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 02:24:10,475] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001320417148.index.deleted (kafka.log.OffsetIndex) [2015-03-14 02:28:02,700] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 02:46:14,738] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 02:48:10,390] INFO Scheduling log segment 1325780427 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 02:49:10,390] INFO Deleting segment 1325780427 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 02:49:10,430] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001325780427.index.deleted (kafka.log.OffsetIndex) [2015-03-14 03:04:20,628] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 03:08:10,377] INFO Scheduling log segment 1331143826 for log Topic2015-03-14 07:46:52,338] ERROR [Replica Manager on Broker 4]: Error when processing fetch request for partition [Topic22kv,5] offset 1754769769 from follower with correlation id 1645671. Possible cause: Request for offset 1754769769 but we only have log segments in the range 1400864851 to 1754769732. (kafka.server.ReplicaManager) [2015-03-14 03:08:10,377] INFO Scheduling log segment 1331143826 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 03:09:10,455] INFO Deleting segment 1331143826 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 03:09:10,455] INFO Deleting segment 1331143826 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 03:09:10,539] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001331143826.index.deleted (kafka.log.OffsetIndex) [2015-03-14 03:09:10,539] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001331143826.index.deleted (kafka.log.OffsetIndex) [2015-03-14 03:22:34,053] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 03:22:34,053] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 03:33:10,335] INFO Scheduling log segment 1336506655 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 03:33:10,335] INFO Scheduling log segment 1336506655 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 03:34:10,352] INFO Deleting segment 1336506655 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 03:34:10,352] INFO Deleting segment 1336506655 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 03:34:10,413] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001336506655.index.deleted (kafka.log.OffsetIndex) [2015-03-14 03:34:10,413] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001336506655.index.deleted (kafka.log.OffsetIndex) [2015-03-14 03:40:54,696] INFO Rolled new log segment for 'Topic22kv-5' in 14 ms. (kafka.log.Log) [2015-03-14 03:40:54,696] INFO Rolled new log segment for 'Topic22kv-5' in 14 ms. (kafka.log.Log) [2015-03-14 03:53:10,345] INFO Scheduling log segment 1341870024 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 03:53:10,345] INFO Scheduling log segment 1341870024 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 03:54:10,346] INFO Deleting segment 1341870024 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 03:54:10,346] INFO Deleting segment 1341870024 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 03:54:10,438] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001341870024.index.deleted (kafka.log.OffsetIndex) [2015-03-14 03:54:10,438] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001341870024.index.deleted (kafka.log.OffsetIndex) [2015-03-14 03:59:15,235] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 03:59:15,235] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 04:17:17,933] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 04:17:17,933] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 04:18:10,419] INFO Scheduling log segment 1347232785 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 04:18:10,419] INFO Scheduling log segment 1347232785 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 04:19:10,419] INFO Deleting segment 1347232785 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 04:19:10,419] INFO Deleting segment 1347232785 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 04:19:10,593] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001347232785.index.deleted (kafka.log.OffsetIndex) [2015-03-14 04:19:10,593] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001347232785.index.deleted (kafka.log.OffsetIndex) [2015-03-14 04:35:42,637] INFO Rolled new log segment for 'Topic22kv-5' in 0 ms. (kafka.log.Log) [2015-03-14 04:35:42,637] INFO Rolled new log segment for 'Topic22kv-5' in 0 ms. (kafka.log.Log) [2015-03-14 04:43:10,333] INFO Scheduling log segment 1352596152 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 04:43:10,333] INFO Scheduling log segment 1352596152 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 04:44:10,333] INFO Deleting segment 1352596152 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 04:44:10,333] INFO Deleting segment 1352596152 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 04:44:10,408] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001352596152.index.deleted (kafka.log.OffsetIndex) [2015-03-14 04:44:10,408] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001352596152.index.deleted (kafka.log.OffsetIndex) [2015-03-14 04:53:51,205] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 04:53:51,205] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 05:03:10,340] INFO Scheduling log segment 1357959499 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 05:03:10,340] INFO Scheduling log segment 1357959499 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 05:04:10,355] INFO Deleting segment 1357959499 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 05:04:10,355] INFO Deleting segment 1357959499 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 05:04:10,463] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001357959499.index.deleted (kafka.log.OffsetIndex) [2015-03-14 05:04:10,463] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001357959499.index.deleted (kafka.log.OffsetIndex) [2015-03-14 05:11:57,885] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 05:11:57,885] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 05:16:10,514] INFO Partition [Topic22kv,5] on broker 4: Shrinking ISR for partition [Topic22kv,5] from 2,4,3 to 2,4 (kafka.cluster.Partition) [2015-03-14 05:16:10,514] INFO Partition [Topic22kv,5] on broker 4: Shrinking ISR for partition [Topic22kv,5] from 2,4,3 to 2,4 (kafka.cluster.Partition) [2015-03-14 05:16:10,514] INFO Partition [Topic22kv,5] on broker 4: Shrinking ISR for partition [Topic22kv,5] from 2,4,3 to 2,4 (kafka.cluster.Partition) [2015-03-14 05:16:10,514] INFO Partition [Topic22kv,5] on broker 4: Shrinking ISR for partition [Topic22kv,5] from 2,4,3 to 2,4 (kafka.cluster.Partition) [2015-03-14 05:16:10,514] INFO Partition [Topic22kv,5] on broker 4: Shrinking ISR for partition [Topic22kv,5] from 2,4,3 to 2,4 (kafka.cluster.Partition) [2015-03-14 05:16:10,717] INFO Partition [Topic22kv,5] on broker 4: Expanding ISR for partition [Topic22kv,5] from 2,4 to 2,4,3 (kafka.cluster.Partition) [2015-03-14 05:16:10,717] INFO Partition [Topic22kv,5] on broker 4: Expanding ISR for partition [Topic22kv,5] from 2,4 to 2,4,3 (kafka.cluster.Partition) [2015-03-14 05:16:10,717] INFO Partition [Topic22kv,5] on broker 4: Expanding ISR for partition [Topic22kv,5] from 2,4 to 2,4,3 (kafka.cluster.Partition) [2015-03-14 05:28:10,392] INFO Scheduling log segment 1363322696 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 05:28:10,392] INFO Scheduling log segment 1363322696 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 05:29:10,447] INFO Deleting segment 1363322696 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 05:29:10,447] INFO Deleting segment 1363322696 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 05:29:10,559] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001363322696.index.deleted (kafka.log.OffsetIndex) [2015-03-14 05:29:10,559] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001363322696.index.deleted (kafka.log.OffsetIndex) [2015-03-14 05:30:09,207] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 05:30:09,207] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 05:48:10,340] INFO Scheduling log segment 1368685783 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 05:48:10,340] INFO Scheduling log segment 1368685783 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 05:48:25,185] INFO Rolled new log segment for 'Topic22kv-5' in 0 ms. (kafka.log.Log) [2015-03-14 05:48:25,185] INFO Rolled new log segment for 'Topic22kv-5' in 0 ms. (kafka.log.Log) [2015-03-14 05:49:10,340] INFO Deleting segment 1368685783 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 05:49:10,340] INFO Deleting segment 1368685783 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 05:49:10,537] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001368685783.index.deleted (kafka.log.OffsetIndex) [2015-03-14 05:49:10,537] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001368685783.index.deleted (kafka.log.OffsetIndex) [2015-03-14 06:05:17,924] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 06:05:17,924] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 06:13:10,371] INFO Scheduling log segment 1374048783 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 06:13:10,371] INFO Scheduling log segment 1374048783 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 06:14:10,371] INFO Deleting segment 1374048783 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 06:14:10,371] INFO Deleting segment 1374048783 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 06:14:10,420] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001374048783.index.deleted (kafka.log.OffsetIndex) [2015-03-14 06:14:10,420] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001374048783.index.deleted (kafka.log.OffsetIndex) [2015-03-14 06:22:19,125] INFO Rolled new log segment for 'Topic22kv-5' in 0 ms. (kafka.log.Log) [2015-03-14 06:22:19,125] INFO Rolled new log segment for 'Topic22kv-5' in 0 ms. (kafka.log.Log) [2015-03-14 06:33:10,347] INFO Scheduling log segment 1379411624 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 06:33:10,347] INFO Scheduling log segment 1379411624 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 06:34:10,348] INFO Deleting segment 1379411624 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 06:34:10,348] INFO Deleting segment 1379411624 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 06:34:10,457] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001379411624.index.deleted (kafka.log.OffsetIndex) [2015-03-14 06:34:10,457] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001379411624.index.deleted (kafka.log.OffsetIndex) [2015-03-14 06:39:15,713] INFO Rolled new log segment for 'Topic22kv-5' in 0 ms. (kafka.log.Log) [2015-03-14 06:39:15,713] INFO Rolled new log segment for 'Topic22kv-5' in 0 ms. (kafka.log.Log) [2015-03-14 06:56:17,306] INFO Rolled new log segment for 'Topic22kv-5' in 13 ms. (kafka.log.Log) [2015-03-14 06:56:17,306] INFO Rolled new log segment for 'Topic22kv-5' in 13 ms. (kafka.log.Log) [2015-03-14 06:58:10,345] INFO Scheduling log segment 1384775041 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 06:58:10,345] INFO Scheduling log segment 1384775041 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 06:59:10,345] INFO Deleting segment 1384775041 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 06:59:10,345] INFO Deleting segment 1384775041 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 06:59:10,491] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001384775041.index.deleted (kafka.log.OffsetIndex) [2015-03-14 06:59:10,491] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001384775041.index.deleted (kafka.log.OffsetIndex) [2015-03-14 07:13:17,646] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 07:13:17,646] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 07:18:10,364] INFO Scheduling log segment 1390138512 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:18:10,364] INFO Scheduling log segment 1390138512 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:19:10,511] INFO Deleting segment 1390138512 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 07:19:10,511] INFO Deleting segment 1390138512 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 07:19:10,671] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001390138512.index.deleted (kafka.log.OffsetIndex) [2015-03-14 07:19:10,671] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001390138512.index.deleted (kafka.log.OffsetIndex) [2015-03-14 07:30:02,155] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 07:30:02,155] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 07:43:10,354] INFO Scheduling log segment 1395501499 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:43:10,354] INFO Scheduling log segment 1395501499 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:44:10,354] INFO Deleting segment 1395501499 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 07:44:10,354] INFO Deleting segment 1395501499 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 07:44:10,394] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001395501499.index.deleted (kafka.log.OffsetIndex) [2015-03-14 07:44:10,394] INFO Deleting index /vol11/kafka82/Topic22kv-5/00000000001395501499.index.deleted (kafka.log.OffsetIndex) [2015-03-14 07:46:52,338] ERROR [Replica Manager on Broker 4]: Error when processing fetch request for partition [Topic22kv,5] offset 1754769769 from follower with correlation id 1645671. Possible cause: Request for offset 1754769769 but we only have log segments in the range 1400864851 to 1754769732. (kafka.server.ReplicaManager)
Broker-3 [2015-03-14 07:46:52,517] ERROR [ReplicaFetcherThread-2-4], Current offset 1754769769 for partition [Topic22kv,5] out of range; reset offset to 1400864851 (kafka.server.ReplicaFetcherThread) [2015-03-14 07:46:52,517] WARN [ReplicaFetcherThread-2-4], Replica 3 for partition [Topic22kv,5] reset its fetch offset from 1400864851 to current leader 4's start offset 1400864851 (kafka.server.ReplicaFetcherThread) [2015-03-14 07:46:52,421] INFO Compaction for partition [Topic22kv,5] is resumed (kafka.log.LogCleaner) [2015-03-14 07:46:52,420] INFO Scheduling log segment 1749446181 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,420] INFO Scheduling log segment 1744083107 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,420] INFO Scheduling log segment 1738720377 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,420] INFO Scheduling log segment 1733357931 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,420] INFO Scheduling log segment 1727994900 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,419] INFO Scheduling log segment 1722632729 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,419] INFO Scheduling log segment 1717269789 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,419] INFO Scheduling log segment 1711906913 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,419] INFO Scheduling log segment 1706544338 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,419] INFO Scheduling log segment 1701181566 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,419] INFO Scheduling log segment 1695818413 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,419] INFO Scheduling log segment 1690455586 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,418] INFO Scheduling log segment 1685092450 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,418] INFO Scheduling log segment 1679729882 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,418] INFO Scheduling log segment 1674367433 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,418] INFO Scheduling log segment 1669004451 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,418] INFO Scheduling log segment 1663641744 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,417] INFO Scheduling log segment 1658278968 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,417] INFO Scheduling log segment 1652916834 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,417] INFO Scheduling log segment 1647554066 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,417] INFO Scheduling log segment 1642191186 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,417] INFO Scheduling log segment 1636828852 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,417] INFO Scheduling log segment 1631466326 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,417] INFO Scheduling log segment 1626103585 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,416] INFO Scheduling log segment 1620740750 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,416] INFO Scheduling log segment 1615378342 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,416] INFO Scheduling log segment 1610015711 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,416] INFO Scheduling log segment 1604653392 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,416] INFO Scheduling log segment 1599290877 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,416] INFO Scheduling log segment 1593928097 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,415] INFO Scheduling log segment 1588565423 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,415] INFO Scheduling log segment 1583202882 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,415] INFO Scheduling log segment 1577840715 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,415] INFO Scheduling log segment 1572478219 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,415] INFO Scheduling log segment 1567115844 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,415] INFO Scheduling log segment 1561752847 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,414] INFO Scheduling log segment 1556390348 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,414] INFO Scheduling log segment 1551027837 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,414] INFO Scheduling log segment 1545665202 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,414] INFO Scheduling log segment 1540302166 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,414] INFO Scheduling log segment 1534939512 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,414] INFO Scheduling log segment 1529576936 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,414] INFO Scheduling log segment 1524214801 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,413] INFO Scheduling log segment 1518852045 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,413] INFO Scheduling log segment 1513489256 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,413] INFO Scheduling log segment 1508126631 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,413] INFO Scheduling log segment 1502764133 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,413] INFO Scheduling log segment 1497401497 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,412] INFO Scheduling log segment 1492038376 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,412] INFO Scheduling log segment 1486675299 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,412] INFO Scheduling log segment 1481312627 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,412] INFO Scheduling log segment 1475949659 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,411] INFO Scheduling log segment 1470586720 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,411] INFO Scheduling log segment 1465223478 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,410] INFO Scheduling log segment 1459860085 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,409] INFO Scheduling log segment 1454497169 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,409] INFO Scheduling log segment 1449133811 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,409] INFO Scheduling log segment 1443770521 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,409] INFO Scheduling log segment 1438407760 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,409] INFO Scheduling log segment 1433044302 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,409] INFO Scheduling log segment 1427680989 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,409] INFO Scheduling log segment 1422317783 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,408] INFO Scheduling log segment 1416954195 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,408] INFO Scheduling log segment 1411591123 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,408] INFO Scheduling log segment 1406227848 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,408] INFO Scheduling log segment 1400864851 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:46:52,356] INFO The cleaning for partition [Topic22kv,5] is aborted and paused (kafka.log.LogCleaner) [2015-03-14 07:42:48,490] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001395501499.index.deleted (kafka.log.OffsetIndex) [2015-03-14 07:42:48,426] INFO Deleting segment 1395501499 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 07:41:48,425] INFO Scheduling log segment 1395501499 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:30:02,201] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 07:17:48,582] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001390138512.index.deleted (kafka.log.OffsetIndex) [2015-03-14 07:17:48,467] INFO Deleting segment 1390138512 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 07:16:48,466] INFO Scheduling log segment 1390138512 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 07:13:17,663] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 06:57:48,518] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001384775041.index.deleted (kafka.log.OffsetIndex) [2015-03-14 06:57:48,426] INFO Deleting segment 1384775041 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 06:56:48,425] INFO Scheduling log segment 1384775041 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 06:56:17,317] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 06:39:15,757] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 06:32:48,498] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001379411624.index.deleted (kafka.log.OffsetIndex) [2015-03-14 06:32:48,426] INFO Deleting segment 1379411624 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 06:31:48,425] INFO Scheduling log segment 1379411624 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 06:22:19,162] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 06:12:48,555] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001374048783.index.deleted (kafka.log.OffsetIndex) [2015-03-14 06:12:48,425] INFO Deleting segment 1374048783 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 06:11:48,425] INFO Scheduling log segment 1374048783 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 06:05:17,942] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 05:48:25,194] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 05:47:48,664] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001368685783.index.deleted (kafka.log.OffsetIndex) [2015-03-14 05:47:48,425] INFO Deleting segment 1368685783 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 05:46:48,424] INFO Scheduling log segment 1368685783 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 05:30:09,222] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 05:27:48,502] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001363322696.index.deleted (kafka.log.OffsetIndex) [2015-03-14 05:27:48,443] INFO Deleting segment 1363322696 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 05:26:48,438] INFO Scheduling log segment 1363322696 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 05:11:58,197] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 05:02:48,575] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001357959499.index.deleted (kafka.log.OffsetIndex) [2015-03-14 05:02:48,458] INFO Deleting segment 1357959499 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 05:01:48,456] INFO Scheduling log segment 1357959499 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 04:53:51,467] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 04:42:48,553] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001352596152.index.deleted (kafka.log.OffsetIndex) [2015-03-14 04:42:48,445] INFO Deleting segment 1352596152 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 04:41:48,445] INFO Scheduling log segment 1352596152 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 04:35:42,648] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 04:17:48,506] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001347232785.index.deleted (kafka.log.OffsetIndex) [2015-03-14 04:17:48,428] INFO Deleting segment 1347232785 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 04:17:17,940] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 04:16:48,425] INFO Scheduling log segment 1347232785 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 03:59:15,524] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 03:57:48,567] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001341870024.index.deleted (kafka.log.OffsetIndex) [2015-03-14 03:57:48,496] INFO Deleting segment 1341870024 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 03:56:48,425] INFO Scheduling log segment 1341870024 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 03:40:54,698] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 03:32:48,496] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001336506655.index.deleted (kafka.log.OffsetIndex) [2015-03-14 03:32:48,444] INFO Deleting segment 1336506655 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 03:31:48,443] INFO Scheduling log segment 1336506655 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 03:22:34,083] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 03:12:48,479] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001331143826.index.deleted (kafka.log.OffsetIndex) [2015-03-14 03:12:48,426] INFO Deleting segment 1331143826 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 03:11:48,424] INFO Scheduling log segment 1331143826 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 03:04:20,646] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 02:47:48,519] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001325780427.index.deleted (kafka.log.OffsetIndex) [2015-03-14 02:47:48,491] INFO Deleting segment 1325780427 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 02:46:48,435] INFO Scheduling log segment 1325780427 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 02:46:15,452] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 02:28:02,735] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 02:27:48,468] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001320417148.index.deleted (kafka.log.OffsetIndex) [2015-03-14 02:27:48,425] INFO Deleting segment 1320417148 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 02:26:48,424] INFO Scheduling log segment 1320417148 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 02:09:56,001] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 02:02:48,618] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001315054067.index.deleted (kafka.log.OffsetIndex) [2015-03-14 02:02:48,528] INFO Deleting segment 1315054067 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 02:01:48,453] INFO Scheduling log segment 1315054067 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 01:51:37,075] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 01:37:48,533] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001309691074.index.deleted (kafka.log.OffsetIndex) [2015-03-14 01:37:48,459] INFO Deleting segment 1309691074 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 01:36:48,458] INFO Scheduling log segment 1309691074 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 01:33:12,524] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 01:17:48,617] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001304328119.index.deleted (kafka.log.OffsetIndex) [2015-03-14 01:17:48,519] INFO Deleting segment 1304328119 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 01:16:48,468] INFO Scheduling log segment 1304328119 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 01:14:35,423] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 00:56:01,142] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 00:52:48,542] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001298965422.index.deleted (kafka.log.OffsetIndex) [2015-03-14 00:52:48,501] INFO Deleting segment 1298965422 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 00:51:48,440] INFO Scheduling log segment 1298965422 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 00:37:26,114] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 00:27:48,597] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001293602624.index.deleted (kafka.log.OffsetIndex) [2015-03-14 00:27:48,531] INFO Deleting segment 1293602624 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 00:26:48,447] INFO Scheduling log segment 1293602624 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 00:19:08,530] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-14 00:07:48,482] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001288239314.index.deleted (kafka.log.OffsetIndex) [2015-03-14 00:07:48,425] INFO Deleting segment 1288239314 from log Topic22kv-5. (kafka.log.Log) [2015-03-14 00:06:48,425] INFO Scheduling log segment 1288239314 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-14 00:00:40,474] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-13 23:42:48,501] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001282876266.index.deleted (kafka.log.OffsetIndex) [2015-03-13 23:42:48,427] INFO Deleting segment 1282876266 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 23:42:20,565] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-13 23:41:48,425] INFO Scheduling log segment 1282876266 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 23:27:48,528] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001277513197.index.deleted (kafka.log.OffsetIndex) [2015-03-13 23:27:48,496] INFO Deleting segment 1277513197 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 23:26:48,495] INFO Scheduling log segment 1277513197 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 23:23:33,411] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-13 23:07:48,569] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001272150400.index.deleted (kafka.log.OffsetIndex) [2015-03-13 23:07:48,508] INFO Deleting segment 1272150400 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 23:06:48,448] INFO Scheduling log segment 1272150400 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 23:04:27,303] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-13 22:52:48,514] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001266787556.index.deleted (kafka.log.OffsetIndex) [2015-03-13 22:52:48,443] INFO Deleting segment 1266787556 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 22:51:48,443] INFO Scheduling log segment 1266787556 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 22:44:50,707] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-13 22:32:48,530] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001261424153.index.deleted (kafka.log.OffsetIndex) [2015-03-13 22:32:48,490] INFO Deleting segment 1261424153 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 22:31:48,490] INFO Scheduling log segment 1261424153 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 22:25:12,749] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-13 22:12:48,497] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001256061631.index.deleted (kafka.log.OffsetIndex) [2015-03-13 22:12:48,435] INFO Deleting segment 1256061631 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 22:11:48,434] INFO Scheduling log segment 1256061631 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 22:05:32,149] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-13 21:57:48,565] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001250698493.index.deleted (kafka.log.OffsetIndex) [2015-03-13 21:57:48,504] INFO Deleting segment 1250698493 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 21:56:48,472] INFO Scheduling log segment 1250698493 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 21:47:48,575] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001245335417.index.deleted (kafka.log.OffsetIndex) [2015-03-13 21:47:48,453] INFO Deleting segment 1245335417 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 21:46:48,453] INFO Scheduling log segment 1245335417 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 21:45:30,633] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-13 21:37:48,500] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001239972496.index.deleted (kafka.log.OffsetIndex) [2015-03-13 21:37:48,425] INFO Deleting segment 1239972496 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 21:36:48,425] INFO Scheduling log segment 1239972496 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 21:27:48,565] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001234609321.index.deleted (kafka.log.OffsetIndex) [2015-03-13 21:27:48,460] INFO Deleting segment 1234609321 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 21:26:48,456] INFO Scheduling log segment 1234609321 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 21:25:49,071] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-13 21:22:48,475] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001229245987.index.deleted (kafka.log.OffsetIndex) [2015-03-13 21:22:48,437] INFO Deleting segment 1229245987 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 21:21:48,437] INFO Scheduling log segment 1229245987 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 21:12:48,535] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001223883126.index.deleted (kafka.log.OffsetIndex) [2015-03-13 21:12:48,473] INFO Deleting segment 1223883126 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 21:11:48,424] INFO Scheduling log segment 1223883126 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 21:06:07,824] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-13 21:02:48,657] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001218520176.index.deleted (kafka.log.OffsetIndex) [2015-03-13 21:02:48,458] INFO Deleting segment 1218520176 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 21:01:48,458] INFO Scheduling log segment 1218520176 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 20:52:48,470] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001213156892.index.deleted (kafka.log.OffsetIndex) [2015-03-13 20:52:48,425] INFO Deleting segment 1213156892 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 20:51:48,425] INFO Scheduling log segment 1213156892 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 20:46:17,152] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-13 20:42:48,551] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001207793635.index.deleted (kafka.log.OffsetIndex) [2015-03-13 20:42:48,443] INFO Deleting segment 1207793635 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 20:41:48,441] INFO Scheduling log segment 1207793635 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 20:37:48,508] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001202431067.index.deleted (kafka.log.OffsetIndex) [2015-03-13 20:37:48,479] INFO Deleting segment 1202431067 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 20:36:48,425] INFO Scheduling log segment 1202431067 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 20:27:48,634] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001197068203.index.deleted (kafka.log.OffsetIndex) [2015-03-13 20:27:48,441] INFO Deleting segment 1197068203 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 20:26:48,441] INFO Scheduling log segment 1197068203 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 20:26:28,811] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-13 20:17:48,540] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001191705099.index.deleted (kafka.log.OffsetIndex) [2015-03-13 20:17:48,425] INFO Deleting segment 1191705099 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 20:16:48,425] INFO Scheduling log segment 1191705099 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 20:07:48,571] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001186342054.index.deleted (kafka.log.OffsetIndex) [2015-03-13 20:07:48,441] INFO Deleting segment 1186342054 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 20:06:48,424] INFO Scheduling log segment 1186342054 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 20:06:38,850] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-13 19:57:48,517] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001180979587.index.deleted (kafka.log.OffsetIndex) [2015-03-13 19:57:48,450] INFO Deleting segment 1180979587 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 19:56:48,437] INFO Scheduling log segment 1180979587 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 19:47:48,586] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001175616287.index.deleted (kafka.log.OffsetIndex) [2015-03-13 19:47:48,467] INFO Deleting segment 1175616287 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 19:46:48,449] INFO Scheduling log segment 1175616287 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 19:46:45,072] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-13 19:37:48,497] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001170253063.index.deleted (kafka.log.OffsetIndex) [2015-03-13 19:37:48,424] INFO Deleting segment 1170253063 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 19:36:48,424] INFO Scheduling log segment 1170253063 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 19:27:48,812] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001164890173.index.deleted (kafka.log.OffsetIndex) [2015-03-13 19:27:48,524] INFO Deleting segment 1164890173 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 19:26:48,474] INFO Scheduling log segment 1164890173 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 19:26:46,181] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-13 19:17:48,506] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001159526847.index.deleted (kafka.log.OffsetIndex) [2015-03-13 19:17:48,425] INFO Deleting segment 1159526847 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 19:16:48,425] INFO Scheduling log segment 1159526847 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 19:12:48,903] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001154163901.index.deleted (kafka.log.OffsetIndex) [2015-03-13 19:12:48,480] INFO Deleting segment 1154163901 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 19:11:48,456] INFO Scheduling log segment 1154163901 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 19:06:32,899] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-13 19:02:48,450] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001148801042.index.deleted (kafka.log.OffsetIndex) [2015-03-13 19:02:48,424] INFO Deleting segment 1148801042 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 19:01:48,424] INFO Scheduling log segment 1148801042 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 18:52:48,475] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001143438192.index.deleted (kafka.log.OffsetIndex) [2015-03-13 18:52:48,425] INFO Deleting segment 1143438192 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 18:51:48,424] INFO Scheduling log segment 1143438192 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 18:46:25,065] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-13 18:42:48,604] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001138075470.index.deleted (kafka.log.OffsetIndex) [2015-03-13 18:42:48,490] INFO Deleting segment 1138075470 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 18:41:48,432] INFO Scheduling log segment 1138075470 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 18:27:48,886] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001132712509.index.deleted (kafka.log.OffsetIndex) [2015-03-13 18:27:48,453] INFO Deleting segment 1132712509 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 18:26:48,452] INFO Scheduling log segment 1132712509 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 18:26:09,155] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-13 18:22:48,530] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001127349573.index.deleted (kafka.log.OffsetIndex) [2015-03-13 18:22:48,490] INFO Deleting segment 1127349573 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 18:21:48,452] INFO Scheduling log segment 1127349573 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 18:12:48,917] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001121986528.index.deleted (kafka.log.OffsetIndex) [2015-03-13 18:12:48,444] INFO Deleting segment 1121986528 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 18:11:48,444] INFO Scheduling log segment 1121986528 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 18:05:53,313] INFO Rolled new log segment for 'Topic22kv-5' in 0 ms. (kafka.log.Log) [2015-03-13 18:02:48,581] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001116624011.index.deleted (kafka.log.OffsetIndex) [2015-03-13 18:02:48,473] INFO Deleting segment 1116624011 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 18:01:48,473] INFO Scheduling log segment 1116624011 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 17:47:48,654] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001111261639.index.deleted (kafka.log.OffsetIndex) [2015-03-13 17:47:48,425] INFO Deleting segment 1111261639 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 17:46:48,425] INFO Scheduling log segment 1111261639 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 17:45:47,148] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-13 17:37:48,467] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001105898943.index.deleted (kafka.log.OffsetIndex) [2015-03-13 17:37:48,425] INFO Deleting segment 1105898943 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 17:36:48,425] INFO Scheduling log segment 1105898943 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 17:27:48,459] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001100536086.index.deleted (kafka.log.OffsetIndex) [2015-03-13 17:27:48,426] INFO Deleting segment 1100536086 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 17:26:48,426] INFO Scheduling log segment 1100536086 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 17:26:17,601] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-13 17:12:48,561] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001095173689.index.deleted (kafka.log.OffsetIndex) [2015-03-13 17:12:48,511] INFO Deleting segment 1095173689 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 17:11:48,482] INFO Scheduling log segment 1095173689 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 17:08:20,431] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-13 17:02:48,613] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001089811436.index.deleted (kafka.log.OffsetIndex) [2015-03-13 17:02:48,471] INFO Deleting segment 1089811436 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 17:01:48,425] INFO Scheduling log segment 1089811436 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 16:51:55,894] INFO Rolled new log segment for 'Topic22kv-5' in 1 ms. (kafka.log.Log) [2015-03-13 16:47:48,531] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001084448458.index.deleted (kafka.log.OffsetIndex) [2015-03-13 16:47:48,426] INFO Deleting segment 1084448458 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 16:46:48,425] INFO Scheduling log segment 1084448458 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 16:37:48,581] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001079085670.index.deleted (kafka.log.OffsetIndex) [2015-03-13 16:37:48,426] INFO Deleting segment 1079085670 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 16:36:48,424] INFO Scheduling log segment 1079085670 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 16:27:48,507] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001073723283.index.deleted (kafka.log.OffsetIndex) [2015-03-13 16:27:48,425] INFO Deleting segment 1073723283 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 16:26:48,425] INFO Scheduling log segment 1073723283 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 16:17:48,567] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001068361452.index.deleted (kafka.log.OffsetIndex) [2015-03-13 16:17:48,426] INFO Deleting segment 1068361452 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 16:16:48,425] INFO Scheduling log segment 1068361452 for log Topic22kv-5 for deletion. (kafka.log.Log) [2015-03-13 16:07:49,967] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001062998917.index.deleted (kafka.log.OffsetIndex) [2015-03-13 16:07:49,940] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001057636089.index.deleted (kafka.log.OffsetIndex) [2015-03-13 16:07:49,936] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001036185816.index.deleted (kafka.log.OffsetIndex) [2015-03-13 16:07:49,920] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001041548278.index.deleted (kafka.log.OffsetIndex) [2015-03-13 16:07:49,919] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001052273502.index.deleted (kafka.log.OffsetIndex) [2015-03-13 16:07:49,915] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001030823130.index.deleted (kafka.log.OffsetIndex) [2015-03-13 16:07:49,904] INFO Deleting segment 1062998917 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 16:07:49,900] INFO Deleting segment 1057636089 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 16:07:49,900] INFO Deleting index /vol14/kafka82/Topic22kv-5/00000000001046910906.index.deleted (kafka.log.OffsetIndex) [2015-03-13 16:07:49,876] INFO Deleting segment 1052273502 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 16:07:49,875] INFO Deleting segment 1046910906 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 16:07:49,870] INFO Deleting segment 1041548278 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 16:07:49,869] INFO Deleting segment 1036185816 from log Topic22kv-5. (kafka.log.Log) [2015-03-13 16:07:49,865] INFO Deleting segment 1030823130 from log Topic22kv-5. (kafka.log.Log)
Thanks, Kazim Zakee > On Mar 16, 2015, at 10:48 AM, Mayuresh Gharat <gharatmayures...@gmail.com> > wrote: > > Can you provide more logs (complete) on Broker 3 till time : > > *[2015-03-14 07:46:52,517*] WARN [ReplicaFetcherThread-2-4], Replica 3 for > partition [Topic22kv,5] reset its fetch offset from 1400864851 to current > leader 4's start offset 1400864851 (kafka.server.ReplicaFetcherThread) > > I would like to see logs from time much before it sent the fetch request to > Broker 4 to the time above. I want to check if in any case Broker 3 was a > leader before broker 4 took over. > > Additional logs will help. > > > Thanks, > > Mayuresh > > > > On Sat, Mar 14, 2015 at 8:35 PM, Zakee <kzak...@netzero.net> wrote: > >> log.cleanup.policy is delete not compact. >> log.cleaner.enable=true >> log.cleaner.threads=5 >> log.cleanup.policy=delete >> log.flush.scheduler.interval.ms=3000 >> log.retention.minutes=1440 >> log.segment.bytes=1073741824 (1gb) >> >> Messages are keyed but not compressed, producer async and uses kafka >> default partitioner. >> String message = msg.getString(); >> String uniqKey = ""+rnd.nextInt();// random key >> String partKey = getPartitionKey();// partition key >> KeyedMessage<String, String> data = new KeyedMessage<String, >> String>(this.topicName, uniqKey, partKey, message); >> producer.send(data); >> >> Thanks >> Zakee >> >> >> >>> On Mar 14, 2015, at 4:23 PM, gharatmayures...@gmail.com wrote: >>> >>> Is your topic log compacted? Also if it is are the messages keyed? Or >> are the messages compressed? >>> >>> Thanks, >>> >>> Mayuresh >>> >>> Sent from my iPhone >>> >>>> On Mar 14, 2015, at 2:02 PM, Zakee <kzak...@netzero.net <mailto: >> kzak...@netzero.net>> wrote: >>>> >>>> Thanks, Jiangjie for helping resolve the kafka controller migration >> driven partition leader rebalance issue. The logs are much cleaner now. >>>> >>>> There are a few incidences of Out of range offset even though there is >> no consumers running, only producers and replica fetchers. I was trying to >> relate to a cause, looks like compaction (log segment deletion) causing >> this. Not sure whether this is expected behavior. >>>> >>>> Broker-4: >>>> [2015-03-14 07:46:52,338] ERROR [Replica Manager on Broker 4]: Error >> when processing fetch request for partition [Topic22kv,5] offset 1754769769 >> from follower with correlation id 1645671. Possible cause: Request for >> offset 1754769769 but we only have log segments in the range 1400864851 to >> 1754769732. (kafka.server.ReplicaManager) >>>> >>>> Broker-3: >>>> [2015-03-14 07:46:52,356] INFO The cleaning for partition [Topic22kv,5] >> is aborted and paused (kafka.log.LogCleaner) >>>> [2015-03-14 07:46:52,408] INFO Scheduling log segment 1400864851 for >> log Topic22kv-5 for deletion. (kafka.log.Log) >>>> … >>>> [2015-03-14 07:46:52,421] INFO Compaction for partition [Topic22kv,5] >> is resumed (kafka.log.LogCleaner) >>>> [2015-03-14 07:46:52,517] ERROR [ReplicaFetcherThread-2-4], Current >> offset 1754769769 for partition [Topic22kv,5] out of range; reset offset to >> 1400864851 (kafka.server.ReplicaFetcherThread) >>>> [2015-03-14 07:46:52,517] WARN [ReplicaFetcherThread-2-4], Replica 3 >> for partition [Topic22kv,5] reset its fetch offset from 1400864851 to >> current leader 4's start offset 1400864851 >> (kafka.server.ReplicaFetcherThread) >>>> >>>> ____________________________________________________________ >>>> Old School Yearbook Pics >>>> View Class Yearbooks Online Free. Search by School & Year. Look Now! >>>> >> http://thirdpartyoffers.netzero.net/TGL3231/5504a2032e49422021991st02vuc < >> http://thirdpartyoffers.netzero.net/TGL3231/5504a2032e49422021991st02vuc> >>>> <topic22kv_746a_314_logs.txt> >>>> >>>> >>>> Thanks >>>> Zakee >>>> >>>>> On Mar 9, 2015, at 12:18 PM, Zakee <kzak...@netzero.net> wrote: >>>>> >>>>> No broker restarts. >>>>> >>>>> Created a kafka issue: >> https://issues.apache.org/jira/browse/KAFKA-2011 < >> https://issues.apache.org/jira/browse/KAFKA-2011> >>>>> >>>>>>> Logs for rebalance: >>>>>>> [2015-03-07 16:52:48,969] INFO [Controller 2]: Resuming preferred >> replica election for partitions: (kafka.controller.KafkaController) >>>>>>> [2015-03-07 16:52:48,969] INFO [Controller 2]: Partitions that >> completed preferred replica election: (kafka.controller.KafkaController) >>>>>>> … >>>>>>> [2015-03-07 12:07:06,783] INFO [Controller 4]: Resuming preferred >> replica election for partitions: (kafka.controller.KafkaController) >>>>>>> ... >>>>>>> [2015-03-07 09:10:41,850] INFO [Controller 3]: Resuming preferred >> replica election for partitions: (kafka.controller.KafkaController) >>>>>>> ... >>>>>>> [2015-03-07 08:26:56,396] INFO [Controller 1]: Starting preferred >> replica leader election for partitions (kafka.controller.KafkaController) >>>>>>> ... >>>>>>> [2015-03-06 16:52:59,506] INFO [Controller 2]: Partitions undergoing >> preferred replica election: (kafka.controller.KafkaController) >>>>>>> >>>>>>> Also, I still see lots of below errors (~69k) going on in the logs >> since the restart. Is there any other reason than rebalance for these >> errors? >>>>>>> >>>>>>> [2015-03-07 14:23:28,963] ERROR [ReplicaFetcherThread-2-5], Error >> for partition [Topic-11,7] to broker 5:class >> kafka.common.NotLeaderForPartitionException >> (kafka.server.ReplicaFetcherThread) >>>>>>> [2015-03-07 14:23:28,963] ERROR [ReplicaFetcherThread-1-5], Error >> for partition [Topic-2,25] to broker 5:class >> kafka.common.NotLeaderForPartitionException >> (kafka.server.ReplicaFetcherThread) >>>>>>> [2015-03-07 14:23:28,963] ERROR [ReplicaFetcherThread-2-5], Error >> for partition [Topic-2,21] to broker 5:class >> kafka.common.NotLeaderForPartitionException >> (kafka.server.ReplicaFetcherThread) >>>>>>> [2015-03-07 14:23:28,963] ERROR [ReplicaFetcherThread-1-5], Error >> for partition [Topic-22,9] to broker 5:class >> kafka.common.NotLeaderForPartitionException >> (kafka.server.ReplicaFetcherThread) >>>>> >>>>> >>>>>> Could you paste the related logs in controller.log? >>>>> What specifically should I search for in the logs? >>>>> >>>>> Thanks, >>>>> Zakee >>>>> >>>>> >>>>> >>>>>> On Mar 9, 2015, at 11:35 AM, Jiangjie Qin <j...@linkedin.com.INVALID >> <mailto:j...@linkedin.com.INVALID>> wrote: >>>>>> >>>>>> Is there anything wrong with brokers around that time? E.g. Broker >> restart? >>>>>> The log you pasted are actually from replica fetchers. Could you >> paste the >>>>>> related logs in controller.log? >>>>>> >>>>>> Thanks. >>>>>> >>>>>> Jiangjie (Becket) Qin >>>>>> >>>>>>> On 3/9/15, 10:32 AM, "Zakee" <kzak...@netzero.net <mailto: >> kzak...@netzero.net>> wrote: >>>>>>> >>>>>>> Correction: Actually the rebalance happened quite until 24 hours >> after >>>>>>> the start, and thats where below errors were found. Ideally rebalance >>>>>>> should not have happened at all. >>>>>>> >>>>>>> >>>>>>> Thanks >>>>>>> Zakee >>>>>>> >>>>>>> >>>>>>> >>>>>>>>> On Mar 9, 2015, at 10:28 AM, Zakee <kzak...@netzero.net <mailto: >> kzak...@netzero.net>> wrote: >>>>>>>>> >>>>>>>>> Hmm, that sounds like a bug. Can you paste the log of leader >> rebalance >>>>>>>>> here? >>>>>>>> Thanks for you suggestions. >>>>>>>> It looks like the rebalance actually happened only once soon after I >>>>>>>> started with clean cluster and data was pushed, it didn’t happen >> again >>>>>>>> so far, and I see the partitions leader counts on brokers did not >> change >>>>>>>> since then. One of the brokers was constantly showing 0 for >> partition >>>>>>>> leader count. Is that normal? >>>>>>>> >>>>>>>> Also, I still see lots of below errors (~69k) going on in the logs >>>>>>>> since the restart. Is there any other reason than rebalance for >> these >>>>>>>> errors? >>>>>>>> >>>>>>>> [2015-03-07 14:23:28,963] ERROR [ReplicaFetcherThread-2-5], Error >> for >>>>>>>> partition [Topic-11,7] to broker 5:class >>>>>>>> kafka.common.NotLeaderForPartitionException >>>>>>>> (kafka.server.ReplicaFetcherThread) >>>>>>>> [2015-03-07 14:23:28,963] ERROR [ReplicaFetcherThread-1-5], Error >> for >>>>>>>> partition [Topic-2,25] to broker 5:class >>>>>>>> kafka.common.NotLeaderForPartitionException >>>>>>>> (kafka.server.ReplicaFetcherThread) >>>>>>>> [2015-03-07 14:23:28,963] ERROR [ReplicaFetcherThread-2-5], Error >> for >>>>>>>> partition [Topic-2,21] to broker 5:class >>>>>>>> kafka.common.NotLeaderForPartitionException >>>>>>>> (kafka.server.ReplicaFetcherThread) >>>>>>>> [2015-03-07 14:23:28,963] ERROR [ReplicaFetcherThread-1-5], Error >> for >>>>>>>> partition [Topic-22,9] to broker 5:class >>>>>>>> kafka.common.NotLeaderForPartitionException >>>>>>>> (kafka.server.ReplicaFetcherThread) >>>>>>>> >>>>>>>>> Some other things to check are: >>>>>>>>> 1. The actual property name is auto.leader.rebalance.enable, not >>>>>>>>> auto.leader.rebalance. You’ve probably known this, just to double >>>>>>>>> confirm. >>>>>>>> Yes >>>>>>>> >>>>>>>>> 2. In zookeeper path, can you verify >> /admin/preferred_replica_election >>>>>>>>> does not exist? >>>>>>>> ls /admin >>>>>>>> [delete_topics] >>>>>>>> ls /admin/preferred_replica_election >>>>>>>> Node does not exist: /admin/preferred_replica_election >>>>>>>> >>>>>>>> >>>>>>>> Thanks >>>>>>>> Zakee >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> On Mar 7, 2015, at 10:49 PM, Jiangjie Qin >> <j...@linkedin.com.INVALID <mailto:j...@linkedin.com.INVALID>> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Hmm, that sounds like a bug. Can you paste the log of leader >> rebalance >>>>>>>>> here? >>>>>>>>> Some other things to check are: >>>>>>>>> 1. The actual property name is auto.leader.rebalance.enable, not >>>>>>>>> auto.leader.rebalance. You’ve probably known this, just to double >>>>>>>>> confirm. >>>>>>>>> 2. In zookeeper path, can you verify >> /admin/preferred_replica_election >>>>>>>>> does not exist? >>>>>>>>> >>>>>>>>> Jiangjie (Becket) Qin >>>>>>>>> >>>>>>>>>> On 3/7/15, 10:24 PM, "Zakee" <kzak...@netzero.net <mailto: >> kzak...@netzero.net>> wrote: >>>>>>>>>> >>>>>>>>>> I started with clean cluster and started to push data. It still >> does >>>>>>>>>> the >>>>>>>>>> rebalance at random durations even though the >> auto.leader.relabalance >>>>>>>>>> is >>>>>>>>>> set to false. >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> Zakee >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> On Mar 6, 2015, at 3:51 PM, Jiangjie Qin >> <j...@linkedin.com.INVALID <mailto:j...@linkedin.com.INVALID>> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Yes, the rebalance should not happen in that case. That is a >> little >>>>>>>>>>> bit >>>>>>>>>>> strange. Could you try to launch a clean Kafka cluster with >>>>>>>>>>> auto.leader.election disabled and try push data? >>>>>>>>>>> When leader migration occurs, NotLeaderForPartition exception is >>>>>>>>>>> expected. >>>>>>>>>>> >>>>>>>>>>> Jiangjie (Becket) Qin >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> On 3/6/15, 3:14 PM, "Zakee" <kzak...@netzero.net <mailto: >> kzak...@netzero.net>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Yes, Jiangjie, I do see lots of these errors "Starting preferred >>>>>>>>>>>> replica >>>>>>>>>>>> leader election for partitions” in logs. I also see lot of >> Produce >>>>>>>>>>>> request failure warnings in with the NotLeader Exception. >>>>>>>>>>>> >>>>>>>>>>>> I tried switching off the auto.leader.relabalance to false. I am >>>>>>>>>>>> still >>>>>>>>>>>> noticing the rebalance happening. My understanding was the >> rebalance >>>>>>>>>>>> will >>>>>>>>>>>> not happen when this is set to false. >>>>>>>>>>>> >>>>>>>>>>>> Thanks >>>>>>>>>>>> Zakee >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> On Feb 25, 2015, at 5:17 PM, Jiangjie Qin >>>>>>>>>>>>> <j...@linkedin.com.INVALID <mailto:j...@linkedin.com.INVALID>> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> I don’t think num.replica.fetchers will help in this case. >>>>>>>>>>>>> Increasing >>>>>>>>>>>>> number of fetcher threads will only help in cases where you >> have a >>>>>>>>>>>>> large >>>>>>>>>>>>> amount of data coming into a broker and more replica fetcher >>>>>>>>>>>>> threads >>>>>>>>>>>>> will >>>>>>>>>>>>> help keep up. We usually only use 1-2 for each broker. But in >> your >>>>>>>>>>>>> case, >>>>>>>>>>>>> it looks that leader migration cause issue. >>>>>>>>>>>>> Do you see anything else in the log? Like preferred leader >>>>>>>>>>>>> election? >>>>>>>>>>>>> >>>>>>>>>>>>> Jiangjie (Becket) Qin >>>>>>>>>>>>> >>>>>>>>>>>>> On 2/25/15, 5:02 PM, "Zakee" <kzak...@netzero.net <mailto: >> kzak...@netzero.net> >>>>>>>>>>>>> <mailto:kzak...@netzero.net <mailto:kzak...@netzero.net>>> >> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, Jiangjie. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Yes, I do see under partitions usually shooting every hour. >>>>>>>>>>>>>> Anythings >>>>>>>>>>>>>> that >>>>>>>>>>>>>> I could try to reduce it? >>>>>>>>>>>>>> >>>>>>>>>>>>>> How does "num.replica.fetchers" affect the replica sync? >> Currently >>>>>>>>>>>>>> have >>>>>>>>>>>>>> configured 7 each of 5 brokers. >>>>>>>>>>>>>> >>>>>>>>>>>>>> -Zakee >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Wed, Feb 25, 2015 at 4:17 PM, Jiangjie Qin >>>>>>>>>>>>>> <j...@linkedin.com.invalid <mailto:j...@linkedin.com.invalid >>>> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> These messages are usually caused by leader migration. I >> think as >>>>>>>>>>>>>>> long >>>>>>>>>>>>>>> as >>>>>>>>>>>>>>> you don¹t see this lasting for ever and got a bunch of under >>>>>>>>>>>>>>> replicated >>>>>>>>>>>>>>> partitions, it should be fine. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Jiangjie (Becket) Qin >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 2/25/15, 4:07 PM, "Zakee" <kzak...@netzero.net <mailto: >> kzak...@netzero.net>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Need to know if I should I be worried about this or ignore >> them. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I see tons of these exceptions/warnings in the broker logs, >> not >>>>>>>>>>>>>>>> sure >>>>>>>>>>>>>>> what >>>>>>>>>>>>>>>> causes them and what could be done to fix them. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> ERROR [ReplicaFetcherThread-3-5], Error for partition >>>>>>>>>>>>>>>> [TestTopic] >>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>> broker >>>>>>>>>>>>>>>> 5:class kafka.common.NotLeaderForPartitionException >>>>>>>>>>>>>>>> (kafka.server.ReplicaFetcherThread) >>>>>>>>>>>>>>>> [2015-02-25 11:01:41,785] ERROR [ReplicaFetcherThread-3-5], >>>>>>>>>>>>>>>> Error >>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>> partition [TestTopic] to broker 5:class >>>>>>>>>>>>>>>> kafka.common.NotLeaderForPartitionException >>>>>>>>>>>>>>>> (kafka.server.ReplicaFetcherThread) >>>>>>>>>>>>>>>> [2015-02-25 11:01:41,785] WARN [Replica Manager on Broker >> 2]: >>>>>>>>>>>>>>>> Fetch >>>>>>>>>>>>>>>> request >>>>>>>>>>>>>>>> with correlation id 950084 from client >> ReplicaFetcherThread-1-2 >>>>>>>>>>>>>>>> on >>>>>>>>>>>>>>>> partition [TestTopic,2] failed due to Leader not local for >>>>>>>>>>>>>>>> partition >>>>>>>>>>>>>>>> [TestTopic,2] on broker 2 (kafka.server.ReplicaManager) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Any ideas? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -Zakee >>>>>>>>>>>>>>>> ____________________________________________________________ >>>>>>>>>>>>>>>> Next Apple Sensation >>>>>>>>>>>>>>>> 1 little-known path to big profits >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >> http://thirdpartyoffers.netzero.net/TGL3231/54ee63b9e704b63b94061 < >> http://thirdpartyoffers.netzero.net/TGL3231/54ee63b9e704b63b94061> >>>>>>>>>>>>>>>> st0 >>>>>>>>>>>>>>>> 3v >>>>>>>>>>>>>>>> uc >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ____________________________________________________________ >>>>>>>>>>>>>>> Extended Stay America >>>>>>>>>>>>>>> Get Fantastic Amenities, low rates! Kitchen, Ample Workspace, >>>>>>>>>>>>>>> Free >>>>>>>>>>>>>>> WIFI >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >> http://thirdpartyoffers.netzero.net/TGL3255/54ee66f26da6f66f10ad4m < >> http://thirdpartyoffers.netzero.net/TGL3255/54ee66f26da6f66f10ad4m> >>>>>>>>>>>>>>> p02 >>>>>>>>>>>>>>> du >>>>>>>>>>>>>>> c >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> ____________________________________________________________ >>>>>>>>>>>>> Extended Stay America >>>>>>>>>>>>> Official Site. Free WIFI, Kitchens. Our best rates here, >>>>>>>>>>>>> guaranteed. >>>>>>>>>>>>> >>>>>>>>>>>>> >> http://thirdpartyoffers.netzero.net/TGL3255/54ee80744cfa7747461mp13d < >> http://thirdpartyoffers.netzero.net/TGL3255/54ee80744cfa7747461mp13d> >>>>>>>>>>>>> uc >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> < >> http://thirdpartyoffers.netzero.net/TGL3255/54ee80744cfa7747461mp13 >>>>>>>>>>>>> duc >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> ____________________________________________________________ >>>>>>>>>>> The WORST exercise for aging >>>>>>>>>>> Avoid this "healthy" exercise to look & feel 5-10 years >>>>>>>>>>> YOUNGER >>>>>>>>>>> >>>>>>>>>>> >> http://thirdpartyoffers.netzero.net/TGL3255/54fa40e98a0e640e81196mp07d < >> http://thirdpartyoffers.netzero.net/TGL3255/54fa40e98a0e640e81196mp07d> >>>>>>>>>>> uc >>>>>>>>> >>>>>>>>> >>>>>>>>> ____________________________________________________________ >>>>>>>>> Seabourn Luxury Cruises >>>>>>>>> Receive special offers from the World's Finest Small-Ship >> Cruise >>>>>>>>> Line! >>>>>>>>> >>>>>>>>> >> http://thirdpartyoffers.netzero.net/TGL3255/54fbf3b0f058073b02901mp14duc < >> http://thirdpartyoffers.netzero.net/TGL3255/54fbf3b0f058073b02901mp14duc> >>>>>> >>>>>> >>>>>> ____________________________________________________________ >>>>>> Discover Seabourn >>>>>> A journey as beautiful as the destination, request a brochure today! >>>>>> >> http://thirdpartyoffers.netzero.net/TGL3255/54fdebfe6a2a36bfb0bb3mp10duc < >> http://thirdpartyoffers.netzero.net/TGL3255/54fdebfe6a2a36bfb0bb3mp10duc> >>>>> >>>>> >>>>> Thanks >>>>> Zakee >>>>> >>>>> >>>>> >>>>> ____________________________________________________________ >>>>> Want to place your ad here? >>>>> Advertise on United Online >>>>> >> http://thirdpartyoffers.netzero.net/TGL3255/54fdf80bc575a780b0397mp05duc >>>> >>> ____________________________________________________________ >>> What's your flood risk? >>> Find flood maps, interactive tools, FAQs, and agents in your area. >>> http://thirdpartyoffers.netzero.net/TGL3255/5504cccfca43a4ccf0a56mp08duc >> <http://thirdpartyoffers.netzero.net/TGL3255/5504cccfca43a4ccf0a56mp08duc> >> > > > > -- > -Regards, > Mayuresh R. Gharat > (862) 250-7125 > ____________________________________________________________ > What's your flood risk? > Find flood maps, interactive tools, FAQs, and agents in your area. > http://thirdpartyoffers.netzero.net/TGL3255/55072125266de21244da8mp12duc