Thanks GuoZhang.
After last accident, we stop all consumers and then restart all consumers
one by one then it is ok.
2 brokers, 10 partitions / broker, 3 consumers, each create 10 stream
So consumer1 consumes 10 partitions, consumer2 consumers another 10
partitions, consumer3 is idle.
Today we find some exceptions in consumer2, after these exceptions
consumer2 doesn't work, no message is consumed.
But in zk I found the owner ship doesn't change, consumer1 own 10
partitions consumer 2 own another 10 partitions/
How can we avoid this happen again?
From the log we see the error sequence:
*exception during commitOffsets, Reconnect in multifetch due to socket
error, rebalance 2 times, error in FetcherRunnable'Seems the last error
in FetcherRunnable is deadly, after this error no message is consumed, but
zk ownership doesn't release.*Here is all the kafka related log in that time
*:*
[WARN 2014-05-13 16:19:05.020]
kafka.utils.Logging$class.warn(Logging.scala:79)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c exception
during commitOffsets]
at kafka.utils.ZkUtils$.updatePersistentPath(ZkUtils.scala:103)
at
kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3$$anonfun$apply$4.apply(ZookeeperConsumerConnector.scala:251)
at
kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3$$anonfun$apply$4.apply(ZookeeperConsumerConnector.scala:248)
at
kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3.apply(ZookeeperConsumerConnector.scala:248)
at
kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3.apply(ZookeeperConsumerConnector.scala:246)
at
kafka.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:246)
at
kafka.consumer.ZookeeperConsumerConnector.autoCommit(ZookeeperConsumerConnector.scala:232)
at
kafka.consumer.ZookeeperConsumerConnector$$anonfun$1.apply$mcV$sp(ZookeeperConsumerConnector.scala:126)
at kafka.utils.Utils$$anon$2.run(Utils.scala:58)
[INFO 2014-05-13 16:19:08.991]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c ZK expired;
release old broker parition ownership; re-register consumer
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c]
[INFO 2014-05-13 16:19:08.991]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c begin
registering consumer
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c in ZK]
[WARN 2014-05-13 16:19:09.001]
kafka.utils.Logging$class.warn(Logging.scala:79)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c exception
during commitOffsets]
at kafka.utils.ZkUtils$.updatePersistentPath(ZkUtils.scala:103)
at
kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3$$anonfun$apply$4.apply(ZookeeperConsumerConnector.scala:251)
at
kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3$$anonfun$apply$4.apply(ZookeeperConsumerConnector.scala:248)
at
kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3.apply(ZookeeperConsumerConnector.scala:248)
at
kafka.consumer.ZookeeperConsumerConnector$$anonfun$commitOffsets$3.apply(ZookeeperConsumerConnector.scala:246)
at
kafka.consumer.ZookeeperConsumerConnector.commitOffsets(ZookeeperConsumerConnector.scala:246)
at
kafka.consumer.ZookeeperConsumerConnector.autoCommit(ZookeeperConsumerConnector.scala:232)
at
kafka.consumer.ZookeeperConsumerConnector$$anonfun$1.apply$mcV$sp(ZookeeperConsumerConnector.scala:126)
at kafka.utils.Utils$$anon$2.run(Utils.scala:58)
[INFO 2014-05-13 16:19:09.002]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c end registering
consumer RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c in ZK]
[INFO 2014-05-13 16:19:09.003]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c begin
rebalancing consumer
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try #0]
[INFO 2014-05-13 16:19:09.063]
kafka.utils.Logging$class.info(Logging.scala:69)
[Reconnect in multifetch due to socket error: ]
at kafka.utils.Utils$.read(Utils.scala:538)
at
kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
at kafka.network.Receive$class.readCompletely(Transmission.scala:55)
at
kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
at kafka.consumer.SimpleConsumer.getResponse(SimpleConsumer.scala:177)
at kafka.consumer.SimpleConsumer.liftedTree2$1(SimpleConsumer.scala:117)
at kafka.consumer.SimpleConsumer.multifetch(SimpleConsumer.scala:115)
at kafka.consumer.FetcherRunnable.run(FetcherRunnable.scala:60)
[INFO 2014-05-13 16:19:09.068]
kafka.utils.Logging$class.info(Logging.scala:61)
[FecherRunnable Thread[FetchRunnable-0,5,main] interrupted]
[INFO 2014-05-13 16:19:09.069]
kafka.utils.Logging$class.info(Logging.scala:61)
[stopping fetcher FetchRunnable-0 to host 10.101.10.12]
[INFO 2014-05-13 16:19:09.071]
kafka.utils.Logging$class.info(Logging.scala:61)
[Cleared all relevant queues for this fetcher]
[INFO 2014-05-13 16:19:09.071]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.071]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.071]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.072]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.072]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.072]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.072]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.072]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.072]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.072]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.072]
kafka.utils.Logging$class.info(Logging.scala:61)
[Cleared the data chunks in all the consumer message iterators]
[INFO 2014-05-13 16:19:09.073]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Committing all
offsets after clearing the fetcher queues]
[INFO 2014-05-13 16:19:09.073]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Releasing
partition ownership]
[INFO 2014-05-13 16:19:09.073]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c rebalancing the
following partitions: List(1-0, 1-1, 1-2, 1-3, 1-4, 1-5, 1-6, 1-7, 1-8,
1-9, 2-0, 2-1, 2-2, 2-3, 2-4, 2-5, 2-6, 2-7, 2-8, 2-9) for topic sensei
with consumers:
List(RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-0,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-1,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-2,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-3,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-4,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-5,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-6,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-7,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-8,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-9,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-0,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-1,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-2,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-3,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-4,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-5,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-6,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-7,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-8,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-9)]
[INFO 2014-05-13 16:19:09.073]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8 attempting to
claim partition 1-8]
[INFO 2014-05-13 16:19:09.077]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 attempting to
claim partition 1-4]
[INFO 2014-05-13 16:19:09.081]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 attempting to
claim partition 1-3]
[INFO 2014-05-13 16:19:09.085]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2 attempting to
claim partition 1-2]
[INFO 2014-05-13 16:19:09.089]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1 attempting to
claim partition 1-1]
[INFO 2014-05-13 16:19:09.093]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5 attempting to
claim partition 1-5]
[INFO 2014-05-13 16:19:09.097]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6 attempting to
claim partition 1-6]
[INFO 2014-05-13 16:19:09.101]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9 attempting to
claim partition 1-9]
[INFO 2014-05-13 16:19:09.105]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 attempting to
claim partition 1-7]
[INFO 2014-05-13 16:19:09.109]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0 attempting to
claim partition 1-0]
[INFO 2014-05-13 16:19:09.118]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2 successfully
owned partition 1-2 for topic sensei]
[INFO 2014-05-13 16:19:09.122]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 successfully
owned partition 1-3 for topic sensei]
[INFO 2014-05-13 16:19:09.127]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5 successfully
owned partition 1-5 for topic sensei]
[INFO 2014-05-13 16:19:09.132]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0 successfully
owned partition 1-0 for topic sensei]
[INFO 2014-05-13 16:19:09.137]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9 successfully
owned partition 1-9 for topic sensei]
[INFO 2014-05-13 16:19:09.153]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6 successfully
owned partition 1-6 for topic sensei]
[INFO 2014-05-13 16:19:09.157]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 successfully
owned partition 1-7 for topic sensei]
[INFO 2014-05-13 16:19:09.161]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1 successfully
owned partition 1-1 for topic sensei]
[INFO 2014-05-13 16:19:09.166]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 successfully
owned partition 1-4 for topic sensei]
[INFO 2014-05-13 16:19:09.170]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8 successfully
owned partition 1-8 for topic sensei]
[INFO 2014-05-13 16:19:09.170]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Updating the
cache]
[INFO 2014-05-13 16:19:09.171]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c selected
partitions : sensei:1-0: fetched offset = 1555136587: consumed offset =
1555136587,sensei:1-1: fetched offset = 1550960225: consumed offset =
1550960225,sensei:1-2: fetched offset = 1551158177: consumed offset =
1551158177,sensei:1-3: fetched offset = 1552939302: consumed offset =
1552939302,sensei:1-4: fetched offset = 10316617: consumed offset =
10316617,sensei:1-5: fetched offset = 1550214323: consumed offset =
1550214323,sensei:1-6: fetched offset = 1563630746: consumed offset =
1563630746,sensei:1-7: fetched offset = 1552261483: consumed offset =
1552261483,sensei:1-8: fetched offset = 1556683052: consumed offset =
1556683052,sensei:1-9: fetched offset = 1558487487: consumed offset =
1558487487]
[INFO 2014-05-13 16:19:09.171]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c end rebalancing
consumer RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try #0]
[INFO 2014-05-13 16:19:09.172]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 6 offset: 1563630746
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.172]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 7 offset: 1552261483
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.172]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 4 offset: 10316617 from
10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.172]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 5 offset: 1550214323
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.172]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 2 offset: 1551158177
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.173]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 3 offset: 1552939302
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.173]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 1 offset: 1550960225
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.173]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 8 offset: 1556683052
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.173]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 0 offset: 1555136587
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.173]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 9 offset: 1558487487
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.188]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c begin
rebalancing consumer
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try #0]
[INFO 2014-05-13 16:19:09.240]
kafka.utils.Logging$class.info(Logging.scala:61)
[FecherRunnable Thread[FetchRunnable-0,5,main] interrupted]
[INFO 2014-05-13 16:19:09.240]
kafka.utils.Logging$class.info(Logging.scala:61)
[stopping fetcher FetchRunnable-0 to host 10.101.10.12]
[INFO 2014-05-13 16:19:09.241]
kafka.utils.Logging$class.info(Logging.scala:61)
[Cleared all relevant queues for this fetcher]
[INFO 2014-05-13 16:19:09.241]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.241]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.241]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.241]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.242]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.242]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.242]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.242]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.242]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.242]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.242]
kafka.utils.Logging$class.info(Logging.scala:61)
[Cleared the data chunks in all the consumer message iterators]
[INFO 2014-05-13 16:19:09.242]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Committing all
offsets after clearing the fetcher queues]
[INFO 2014-05-13 16:19:09.285]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Releasing
partition ownership]
[INFO 2014-05-13 16:19:09.333]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c rebalancing the
following partitions: List(1-0, 1-1, 1-2, 1-3, 1-4, 1-5, 1-6, 1-7, 1-8,
1-9, 2-0, 2-1, 2-2, 2-3, 2-4, 2-5, 2-6, 2-7, 2-8, 2-9) for topic sensei
with consumers:
List(RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-0,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-1,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-2,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-3,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-4,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-5,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-6,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-7,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-8,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-9,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-0,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-1,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-2,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-3,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-4,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-5,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-6,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-7,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-8,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-9)]
[INFO 2014-05-13 16:19:09.333]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8 attempting to
claim partition 1-8]
[INFO 2014-05-13 16:19:09.337]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 attempting to
claim partition 1-4]
[INFO 2014-05-13 16:19:09.341]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 attempting to
claim partition 1-3]
[INFO 2014-05-13 16:19:09.345]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2 attempting to
claim partition 1-2]
[INFO 2014-05-13 16:19:09.349]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1 attempting to
claim partition 1-1]
[INFO 2014-05-13 16:19:09.354]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5 attempting to
claim partition 1-5]
[INFO 2014-05-13 16:19:09.358]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6 attempting to
claim partition 1-6]
[INFO 2014-05-13 16:19:09.362]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9 attempting to
claim partition 1-9]
[INFO 2014-05-13 16:19:09.366]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 attempting to
claim partition 1-7]
[INFO 2014-05-13 16:19:09.370]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0 attempting to
claim partition 1-0]
[INFO 2014-05-13 16:19:09.379]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2 successfully
owned partition 1-2 for topic sensei]
[INFO 2014-05-13 16:19:09.390]
kafka.utils.Logging$class.info(Logging.scala:61)
[/consumers/RecommendEvent/owners/sensei/1-3 exists with value
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 during
connection loss; this is ok]
[INFO 2014-05-13 16:19:09.390]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 successfully
owned partition 1-3 for topic sensei]
[INFO 2014-05-13 16:19:09.395]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5 successfully
owned partition 1-5 for topic sensei]
[INFO 2014-05-13 16:19:09.400]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0 successfully
owned partition 1-0 for topic sensei]
[INFO 2014-05-13 16:19:09.405]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9 successfully
owned partition 1-9 for topic sensei]
[INFO 2014-05-13 16:19:09.409]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6 successfully
owned partition 1-6 for topic sensei]
[INFO 2014-05-13 16:19:09.417]
kafka.utils.Logging$class.info(Logging.scala:61)
[/consumers/RecommendEvent/owners/sensei/1-7 exists with value
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 during
connection loss; this is ok]
[INFO 2014-05-13 16:19:09.417]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 successfully
owned partition 1-7 for topic sensei]
[INFO 2014-05-13 16:19:09.422]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1 successfully
owned partition 1-1 for topic sensei]
[INFO 2014-05-13 16:19:09.430]
kafka.utils.Logging$class.info(Logging.scala:61)
[/consumers/RecommendEvent/owners/sensei/1-4 exists with value
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 during
connection loss; this is ok]
[INFO 2014-05-13 16:19:09.430]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 successfully
owned partition 1-4 for topic sensei]
[INFO 2014-05-13 16:19:09.435]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8 successfully
owned partition 1-8 for topic sensei]
[INFO 2014-05-13 16:19:09.435]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Updating the
cache]
[INFO 2014-05-13 16:19:09.435]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c selected
partitions : sensei:1-0: fetched offset = 1555136587: consumed offset =
1555136587,sensei:1-1: fetched offset = 1550960225: consumed offset =
1550960225,sensei:1-2: fetched offset = 1551158177: consumed offset =
1551158177,sensei:1-3: fetched offset = 1552939302: consumed offset =
1552939302,sensei:1-4: fetched offset = 10316617: consumed offset =
10316617,sensei:1-5: fetched offset = 1550214323: consumed offset =
1550214323,sensei:1-6: fetched offset = 1563630746: consumed offset =
1563630746,sensei:1-7: fetched offset = 1552261483: consumed offset =
1552261483,sensei:1-8: fetched offset = 1556683052: consumed offset =
1556683052,sensei:1-9: fetched offset = 1558487487: consumed offset =
1558487487]
[INFO 2014-05-13 16:19:09.436]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c end rebalancing
consumer RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try #0]
[INFO 2014-05-13 16:19:09.438]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 6 offset: 1563630746
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.438]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c begin
rebalancing consumer
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try #0]
[INFO 2014-05-13 16:19:09.438]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 7 offset: 1552261483
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.438]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 4 offset: 10316617 from
10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.438]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 5 offset: 1550214323
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.438]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 2 offset: 1551158177
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.438]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 3 offset: 1552939302
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.438]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 1 offset: 1550960225
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.439]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 8 offset: 1556683052
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.439]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 0 offset: 1555136587
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.439]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 9 offset: 1558487487
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.492]
kafka.utils.Logging$class.info(Logging.scala:61)
[FecherRunnable Thread[FetchRunnable-0,5,main] interrupted]
[INFO 2014-05-13 16:19:09.492]
kafka.utils.Logging$class.info(Logging.scala:61)
[stopping fetcher FetchRunnable-0 to host 10.101.10.12]
[INFO 2014-05-13 16:19:09.492]
kafka.utils.Logging$class.info(Logging.scala:61)
[Cleared all relevant queues for this fetcher]
[INFO 2014-05-13 16:19:09.492]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.492]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.493]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.493]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.493]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.493]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.493]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.493]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.494]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.494]
kafka.utils.Logging$class.info(Logging.scala:61)
[Clearing the current data chunk for this consumer iterator]
[INFO 2014-05-13 16:19:09.494]
kafka.utils.Logging$class.info(Logging.scala:61)
[Cleared the data chunks in all the consumer message iterators]
[INFO 2014-05-13 16:19:09.494]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Committing all
offsets after clearing the fetcher queues]
[INFO 2014-05-13 16:19:09.539]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Releasing
partition ownership]
[INFO 2014-05-13 16:19:09.582]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c rebalancing the
following partitions: List(1-0, 1-1, 1-2, 1-3, 1-4, 1-5, 1-6, 1-7, 1-8,
1-9, 2-0, 2-1, 2-2, 2-3, 2-4, 2-5, 2-6, 2-7, 2-8, 2-9) for topic sensei
with consumers:
List(RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8,
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-0,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-1,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-2,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-3,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-4,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-5,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-6,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-7,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-8,
RecommendEvent_sd-sns-relation02.bj-1399952700475-b80f0553-9,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-0,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-1,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-2,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-3,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-4,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-5,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-6,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-7,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-8,
RecommendEvent_sd-sns-relation03.bj-1399952910743-2e0ccf2e-9)]
[INFO 2014-05-13 16:19:09.582]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8 attempting to
claim partition 1-8]
[INFO 2014-05-13 16:19:09.586]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 attempting to
claim partition 1-4]
[INFO 2014-05-13 16:19:09.590]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 attempting to
claim partition 1-3]
[INFO 2014-05-13 16:19:09.594]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2 attempting to
claim partition 1-2]
[INFO 2014-05-13 16:19:09.598]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1 attempting to
claim partition 1-1]
[INFO 2014-05-13 16:19:09.602]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5 attempting to
claim partition 1-5]
[INFO 2014-05-13 16:19:09.606]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6 attempting to
claim partition 1-6]
[INFO 2014-05-13 16:19:09.610]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9 attempting to
claim partition 1-9]
[INFO 2014-05-13 16:19:09.614]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 attempting to
claim partition 1-7]
[INFO 2014-05-13 16:19:09.618]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0 attempting to
claim partition 1-0]
[INFO 2014-05-13 16:19:09.633]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-2 successfully
owned partition 1-2 for topic sensei]
[INFO 2014-05-13 16:19:09.638]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-3 successfully
owned partition 1-3 for topic sensei]
[INFO 2014-05-13 16:19:09.646]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-5 successfully
owned partition 1-5 for topic sensei]
[INFO 2014-05-13 16:19:09.656]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-0 successfully
owned partition 1-0 for topic sensei]
[INFO 2014-05-13 16:19:09.660]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-9 successfully
owned partition 1-9 for topic sensei]
[INFO 2014-05-13 16:19:09.665]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-6 successfully
owned partition 1-6 for topic sensei]
[INFO 2014-05-13 16:19:09.669]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-7 successfully
owned partition 1-7 for topic sensei]
[INFO 2014-05-13 16:19:09.674]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-1 successfully
owned partition 1-1 for topic sensei]
[INFO 2014-05-13 16:19:09.679]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-4 successfully
owned partition 1-4 for topic sensei]
[INFO 2014-05-13 16:19:09.684]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c-8 successfully
owned partition 1-8 for topic sensei]
[INFO 2014-05-13 16:19:09.684]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Updating the
cache]
[INFO 2014-05-13 16:19:09.684]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c Consumer
RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c selected
partitions : sensei:1-0: fetched offset = 1555136587: consumed offset =
1555136587,sensei:1-1: fetched offset = 1550960225: consumed offset =
1550960225,sensei:1-2: fetched offset = 1551158177: consumed offset =
1551158177,sensei:1-3: fetched offset = 1552939302: consumed offset =
1552939302,sensei:1-4: fetched offset = 10316617: consumed offset =
10316617,sensei:1-5: fetched offset = 1550214323: consumed offset =
1550214323,sensei:1-6: fetched offset = 1563630746: consumed offset =
1563630746,sensei:1-7: fetched offset = 1552261483: consumed offset =
1552261483,sensei:1-8: fetched offset = 1556683052: consumed offset =
1556683052,sensei:1-9: fetched offset = 1558487487: consumed offset =
1558487487]
[INFO 2014-05-13 16:19:09.684]
kafka.utils.Logging$class.info(Logging.scala:61)
[RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c end rebalancing
consumer RecommendEvent_sd-sns-relation01.bj-1399968348749-4bc8451c try #0]
[INFO 2014-05-13 16:19:09.685]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 6 offset: 1563630746
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.685]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 7 offset: 1552261483
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.685]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 4 offset: 10316617 from
10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.685]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 5 offset: 1550214323
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.686]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 2 offset: 1551158177
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.686]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 3 offset: 1552939302
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.686]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 1 offset: 1550960225
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.686]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 8 offset: 1556683052
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.686]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 0 offset: 1555136587
from 10.101.10.12:9092]
[INFO 2014-05-13 16:19:09.686]
kafka.utils.Logging$class.info(Logging.scala:61)
[FetchRunnable-0 start fetching topic: sensei part: 9 offset: 1558487487
from 10.101.10.12:9092]
[ERROR 2014-05-13 16:19:18.687]
kafka.utils.Logging$class.error(Logging.scala:89) [error in FetcherRunnable
]
at kafka.consumer.SimpleConsumer.connect(SimpleConsumer.scala:49)
at
kafka.consumer.SimpleConsumer.getOrMakeConnection(SimpleConsumer.scala:186)
at kafka.consumer.SimpleConsumer.multifetch(SimpleConsumer.scala:113)
at kafka.consumer.FetcherRunnable.run(FetcherRunnable.scala:60)
[INFO 2014-05-13 16:19:18.687]
kafka.utils.Logging$class.info(Logging.scala:61)
[stopping fetcher FetchRunnable-0 to host 10.101.10.12]
2014-05-12 23:22 GMT+08:00 Guozhang Wang <wangg...@gmail.com>:
> Hello Yonghui,
>
> In 0.7 the consumer rebalance logic is distributed and in some corner cases
> such as soft-failure-caused-consecutive rebalances some consumer may
> consider the rebalance as complete while others are still trying the
> rebalance process. You can check the GC logs on your consumer to verify if
> that is the case:
>
> https://issues.apache.org/jira/browse/KAFKA-242
>
> If you bounce the consumers to trigger another rebalance, this issue would
> likely to be resolved.
>
> To solve this issue in 0.9 we are moving the group management like load
> rebalance from the ZK-based distributed logic into a centralized
> coordiantor. Details can be found here:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design
>
> Guozhang
>
>
> On Mon, May 12, 2014 at 12:48 AM, Yonghui Zhao <zhaoyong...@gmail.com
> >wrote:
>
> > Hi,
> >
> > We are using kafka 0.7.
> >
> > 2 brokers, each broker has 10 partitions for one topic
> > 3 consumers in one consumer group, each consumer create 10 streams.
> >
> >
> > Today, when we want to rollout new service.
> > After we restart one consumer we find exceptions and warning.
> >
> > kafka.common.ConsumerRebalanceFailedException:
> > RecommendEvent_sd-sns-relation01.bj-1399630465426-53d3aefc can't
> rebalance
> > after 4 retries
> >
> >
> > [INFO 2014-05-12 15:17:47.364]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [conflict in /consumers/RecommendEvent/owners/sensei/1-2 data:
> > RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e-2 stored data:
> > RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1]
> > [INFO 2014-05-12 15:17:47.366]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e waiting for
> the
> > partition ownership to be deleted: 1-2]
> > [INFO 2014-05-12 15:17:47.375]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [conflict in /consumers/RecommendEvent/owners/sensei/1-3 data:
> > RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e-3 stored data:
> > RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1]
> > [INFO 2014-05-12 15:17:47.375]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e waiting for
> the
> > partition ownership to be deleted: 1-3]
> > [INFO 2014-05-12 15:17:47.385]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [conflict in /consumers/RecommendEvent/owners/sensei/1-5 data:
> > RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e-5 stored data:
> > RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2]
> > [INFO 2014-05-12 15:17:47.386]
> > kafka.utils.Logging$class.info(Logging.scala:61)
> > [RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e waiting for
> the
> > partition ownership to be deleted: 1-5]
> >
> >
> >
> > And I opened zk viewer.
> >
> > In zk, we found 2 consumers in ConsumerGroup/ids:
> >
> > RecommendEvent_sd-sns-relation02.bj-1399635256619-5d8123c6
> > RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3
> >
> >
> > And in owners/topic/ we found all partitions are assigned to
> > sd-sns-relation03.bj:
> >
> > Here is the owner info:
> > 1:0 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-0
> > 1:1 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-0
> > 1:2 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1
> > 1:3 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1
> > 1:4 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2
> > 1:5 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2
> > 1:6 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-3
> > 1:7 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-3
> > 1:8 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-4
> > 1:9 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-4
> >
> > 2:0 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-0
> > 2:1 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1
> > 2:2 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2
> > 2:3 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-3
> > 2:4 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-4
> > 2:5 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-5
> > 2:6 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-6
> > 2:7 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-7
> > 2:8 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-8
> > 2:9 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-9
> >
> >
> > So all partitions are assigned to sd-sns-relation03.bj, but from logs
> and
> > counter, we are sure sd-sns-relation02.bj has input too.
> >
> >
> > My question is:
> >
> > 1. why rebalance failed?
> > 2. why owner info is wrong? btw: zkclient is 0.2
> >
>
>
>
> --
> -- Guozhang
>
