Neha,
The same issue reoccured with just 2 consumer processes. The exception was
related to conflict in writing the ephemeral node. Below was the exception.
Topic name is
"lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin"
with 30 partitions. The 2 processes were running on 2 servers with ips
10.0.8.222 and 10.0.8.225.
*2014-12-09 13:22:11 k.u.ZkUtils$ [INFO] I wrote this conflicted ephemeral
node
[{"version":1,"subscription":{"lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin":5},"pattern":"static","timestamp":"1417964160024"}]
at
/consumers/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin/ids/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d
a while back in a different session, hence I will backoff for this node to
be deleted by Zookeeper and retry*
Attached the complete error logs. The exception occured after the rebalance
failed even after 40 retries. Rebalance failed as the process already
owning some of the partitions did not give us ownership due to conflicting
ephemeral nodes. As you suggested, we ran the wchp command on the 3
zookeeper nodes at this time and figured out that the watcher was
registered for only one of the process. I am copying the kafka consumer
watcher registered on one of the zookeeper servers. (Attached are the wchp
outputs of all 3 zk servers)
*$echo wchp | nc localhost 2181 *
*/kafka/consumers/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin/ids*
* 0x34a175e1d5d0130*
"0x34a175e1d5d0130" was the ephemeral node session Id. I went back to the
zookeeper shell and checked the consumers registered for this topic and
consumer group(same as topic name). Attaching the output in zkCommands.txt.
This clearly shows that
10.0.8.222 has ephemeralOwner = 0x34a175e1d5d0130
10.0.8.225 has ephemeralOwner = 0x34a175e1d5d0127
I think we have the issue here that both consumers have written to
different ephemeral nodes. Watchers are registered for the one of the 2
ephemeral node. The root cause seems to be the inconsistent state while
writing the ephemeral nodes in ZK.
Let me know if you need more details.
-Thanks,
Mohit
On Mon, Nov 10, 2014 at 8:46 AM, Neha Narkhede <neha.narkh...@gmail.com>
wrote:
> A rebalance should trigger on all consumers when you add a new consumer to
> the group. If you don't see the zookeeper watch fire, the consumer may have
> somehow lost the watch. We have seen this behavior on older zk versions, I
> wonder it that bug got reintroduced. To verify if this is the case, you can
> run the wchp zookeeper command on the zk leader and check if each consumer
> has a watch registered.
>
> Do you have a way to try this on zk 3.3.4? I would recommend you try the
> wchp suggestion as well.
>
> On Fri, Nov 7, 2014 at 6:07 AM, Mohit Kathuria <mkathu...@sprinklr.com>
> wrote:
>
> > Hi all,
> >
> > Can someone help here. We are getting constant rebalance failure each
> time
> > a consumer is added beyond a certain number. Did quite a lot of debugging
> > on this and still not able to figure out the pattern.
> >
> > -Thanks,
> > Mohit
> >
> > On Mon, Nov 3, 2014 at 10:53 PM, Mohit Kathuria <mkathu...@sprinklr.com>
> > wrote:
> >
> > > Neha,
> > >
> > > Looks like an issue with the consumer rebalance not able to complete
> > > successfully. We were able to reproduce the issue on topic with 30
> > > partitions, 3 consumer processes(p1,p2 and p3), properties - 40
> > > rebalance.max.retries and 10000(10s) rebalance.backoff.ms.
> > >
> > > Before the process p3 was started, partition ownership was as expected:
> > >
> > > partitions 0-14 owned by p1
> > > partitions 15-29 -> owner p2
> > >
> > > As the process p3 started, rebalance was triggered. Process p3 was
> > > successfully able to acquire partition ownership for partitions 20-29
> as
> > > expected as per the rebalance algorithm. However, process p2 while
> trying
> > > to acquire ownership of partitions 10-19 saw rebalance failure after 40
> > > retries.
> > >
> > > Attaching the logs from process p2 and process p1. It says that p2 was
> > > attempting to rebalance, it was trying to acquire ownership of
> partitions
> > > 10-14 which were owned by process p1. However, at the same time process
> > p1
> > > did not get any event for giving up the partition ownership for
> > partitions
> > > 1-14.
> > > We were expecting a rebalance to have triggered in p1 - but it didn't
> and
> > > hence not giving up ownership. Is our assumption correct/incorrect?
> > > And if the rebalance gets triggered in p1 - how to figure out apart
> from
> > > logs as the logs on p1 did not have anything.
> > >
> > > *2014-11-03 06:57:36 k.c.ZookeeperConsumerConnector [INFO]
> > > [topic_consumerIdString], waiting for the partition ownership to be
> > > deleted: 11*
> > >
> > > During and after the rebalance failed on process p2, Partition
> Ownership
> > > was as below:
> > > 0-14 -> owner p1
> > > 15-19 -> none
> > > 20-29 -> owner p3
> > >
> > > This left the consumers in inconsistent state as 5 partitions were
> never
> > > consumer from and neither was the partitions ownership balanced.
> > >
> > > However, there was no conflict in creating the ephemeral node which was
> > > the case last time. Just to note that the ephemeral node conflict which
> > we
> > > were seeing earlier also appeared after rebalance failed. My hunch is
> > that
> > > fixing the rebalance failure will fix that issue as well.
> > >
> > > -Thanks,
> > > Mohit
> > >
> > >
> > >
> > > On Mon, Oct 20, 2014 at 7:48 PM, Neha Narkhede <
> neha.narkh...@gmail.com>
> > > wrote:
> > >
> > >> Mohit,
> > >>
> > >> I wonder if it is related to
> > >> https://issues.apache.org/jira/browse/KAFKA-1585. When zookeeper
> > expires
> > >> a
> > >> session, it doesn't delete the ephemeral nodes immediately. So if you
> > end
> > >> up trying to recreate ephemeral nodes quickly, it could either be in
> the
> > >> valid latest session or from the previously expired session. If you
> hit
> > >> this problem, then waiting would resolve it. But if not, then this may
> > be
> > >> a
> > >> legitimate bug in ZK 3.4.6.
> > >>
> > >> Can you try shutting down all your consumers, waiting until session
> > >> timeout
> > >> and restarting them?
> > >>
> > >> Thanks,
> > >> Neha
> > >>
> > >> On Mon, Oct 20, 2014 at 6:15 AM, Mohit Kathuria <
> mkathu...@sprinklr.com
> > >
> > >> wrote:
> > >>
> > >> > Dear Experts,
> > >> >
> > >> > We recently updated to kafka v0.8.1.1 with zookeeper v3.4.5. I have
> of
> > >> > topic with 30 partitions and 2 replicas. We are using High level
> > >> consumer
> > >> > api.
> > >> > Each consumer process which is a storm topolofy has 5 streams which
> > >> > connects to 1 or more partitions. We are not using storm's inbuilt
> > kafka
> > >> > spout. Everything runs fine till the 5th consumer process(25
> streams)
> > is
> > >> > added for this topic.
> > >> >
> > >> > As soon as the sixth consumer process is added, the newly added
> > >> partition
> > >> > does not get the ownership of the partitions that it requests for as
> > the
> > >> > already existing owners have not yet given up the ownership.
> > >> >
> > >> > We changed certain properties on consumer :
> > >> >
> > >> > 1. Max Rebalance attempts - 20 ( rebalance.backoff.ms *
> > >> > rebalance.max.retries >> zk connection timeout)
> > >> > 2. Back off ms between rebalances - 10000 (10seconds)
> > >> > 3. ZK connection timeout - 100,000 (100 seconds)
> > >> >
> > >> > Although when I am looking in the zookeeper shell when the rebalance
> > is
> > >> > happening, the consumer is registered fine on the zookeeper. Just
> that
> > >> the
> > >> > rebalance does not happen.
> > >> > After the 20th rebalance gets completed, we get
> > >> >
> > >> >
> > >> > *2014-10-11 11:10:08 k.c.ZookeeperConsumerConnector [INFO]
> > >> > [rule-engine-feed_ip-10-0-2-170-1413025767369-4679959b], Committing
> > all
> > >> > offsets after clearing the fetcher queues*
> > >> > *2014-10-11 11:10:10 c.s.m.k.i.c.KafkaFeedStreamer [WARN] Ignoring
> > >> > exception while trying to start streamer threads:
> > >> > rule-engine-feed_ip-10-0-2-170-1413025767369-4679959b can't
> rebalance
> > >> after
> > >> > 20 retries*
> > >> > *kafka.common.ConsumerRebalanceFailedException:
> > >> > rule-engine-feed_ip-10-0-2-170-1413025767369-4679959b can't
> rebalance
> > >> after
> > >> > 20 retries*
> > >> > * at
> > >> >
> > >> >
> > >>
> >
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:432)
> > >> > ~[stormjar.jar:na]*
> > >> > * at
> > >> >
> > >> >
> > >>
> >
> kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:722)
> > >> > ~[stormjar.jar:na]*
> > >> > * at
> > >> >
> > >> >
> > >>
> >
> kafka.consumer.ZookeeperConsumerConnector.consume(ZookeeperConsumerConnector.scala:212)
> > >> > ~[stormjar.jar:na]*
> > >> > * at
> > >> >
> > >> >
> > >>
> >
> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:80)
> > >> > ~[stormjar.jar:na]*
> > >> > * at
> > >> >
> > >> >
> > >>
> >
> com.spr.messaging.kafka.impl.consumer.KafkaFeedStreamer.createAndStartThreads(KafkaFeedStreamer.java:79)
> > >> > ~[stormjar.jar:na]*
> > >> > * at
> > >> >
> > >> >
> > >>
> >
> com.spr.messaging.kafka.impl.consumer.KafkaFeedStreamer.startKafkaStreamThreadsIfNecessary(KafkaFeedStreamer.java:64)
> > >> > ~[stormjar.jar:na]*
> > >> > * at
> > >> >
> > >> >
> > >>
> >
> com.spr.messaging.kafka.impl.consumer.KafkaFeedConsumerFactoryImpl.startStreamerIfNotRunning(KafkaFeedConsumerFactoryImpl.java:71)
> > >> > [stormjar.jar:na]*
> > >> > * at
> > >> >
> > >> >
> > >>
> >
> com.spr.messaging.kafka.impl.consumer.KafkaFeedPullConsumerImpl.startStreamerIfNotRunning(KafkaFeedPullConsumerImpl.java:48)
> > >> > [stormjar.jar:na]*
> > >> > * at
> > >> >
> > >> >
> > >>
> >
> com.spr.messaging.kafka.impl.KafkaFeedServiceImpl.getKafkaFeedPullConsumer(KafkaFeedServiceImpl.java:63)
> > >> > [stormjar.jar:na]*
> > >> > * at
> > >> >
> > >> >
> > >>
> >
> com.spr.storm.topology.spout.AbstractSprKafkaSpout.nextTuple(AbstractSprKafkaSpout.java:121)
> > >> > [stormjar.jar:na]*
> > >> > * at
> > >> >
> > >> >
> > >>
> >
> backtype.storm.daemon.executor$eval3848$fn__3849$fn__3864$fn__3893.invoke(executor.clj:562)
> > >> > [na:0.9.1-incubating]*
> > >> > * at
> > backtype.storm.util$async_loop$fn__384.invoke(util.clj:433)
> > >> > [na:0.9.1-incubating]*
> > >> > * at clojure.lang.AFn.run(AFn.java:24)
> [clojure-1.4.0.jar:na]*
> > >> > * at java.lang.Thread.run(Thread.java:745) [na:1.7.0_55]*
> > >> > *2014-10-11 11:10:10 k.c.ZookeeperConsumerConnector [INFO]
> > >> > [rule-engine-feed_ip-10-0-2-170-1413025767369-4679959b], begin
> > >> registering
> > >> > consumer rule-engine-feed_ip-10-0-2-170-1413025767369-4679959b in
> ZK*
> > >> > *2014-10-11 11:10:10 k.u.ZkUtils$ [INFO] conflict in
> > >> >
> > >> >
> > >>
> >
> /consumers/rule-engine-feed/ids/rule-engine-feed_ip-10-0-2-170-1413025767369-4679959b
> > >> > data:
> > >> >
> > >> >
> > >>
> >
> {"version":1,"subscription":{"rule-engine-feed":5},"pattern":"static","timestamp":"1413025810635"}
> > >> > stored data:
> > >> >
> > >> >
> > >>
> >
> {"version":1,"subscription":{"rule-engine-feed":5},"pattern":"static","timestamp":"1413025767483"}*
> > >> > *2014-10-11 11:10:10 k.u.ZkUtils$ [INFO] I wrote this conflicted
> > >> ephemeral
> > >> > node
> > >> >
> > >> >
> > >>
> >
> [{"version":1,"subscription":{"rule-engine-feed":5},"pattern":"static","timestamp":"1413025810635"}]
> > >> > at
> > >> >
> > >> >
> > >>
> >
> /consumers/rule-engine-feed/ids/rule-engine-feed_ip-10-0-2-170-1413025767369-4679959b
> > >> > a while back in a different session, hence I will backoff for this
> > node
> > >> to
> > >> > be deleted by Zookeeper and retry*
> > >> >
> > >> > Due to this error, none of the consumer consumes from these
> partitions
> > >> in
> > >> > contention which creates a sort of skewed lag on kafka side. When
> the
> > >> 6th
> > >> > consumer was added, the existing owner process of the partitions in
> > >> > question did not get rebalanced.
> > >> >
> > >> > Any help would be highly appreciated.
> > >> >
> > >> > -Thanks,
> > >> > Mohit
> > >> >
> > >>
> > >
> > >
> >
>
[zk: localhost:2181(CONNECTED) 5] get
/kafka/consumers/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin/ids/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d
{"version":1,"subscription":{"lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin":5},"pattern":"static","timestamp":"1417963753707"}
cZxid = 0x9038b798f
ctime = Sun Dec 07 14:49:13 UTC 2014
mZxid = 0x9038b798f
mtime = Sun Dec 07 14:49:13 UTC 2014
pZxid = 0x9038b798f
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x34a175e1d5d0130
dataLength = 156
numChildren = 0
[zk: localhost:2181(CONNECTED) 6]
[zk: localhost:2181(CONNECTED) 6]
[zk: localhost:2181(CONNECTED) 6]
[zk: localhost:2181(CONNECTED) 6] get
/kafka/consumers/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin/ids/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-225-1417962308114-dab7928d
{"version":1,"subscription":{"lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin":5},"pattern":"static","timestamp":"1417962308245"}
cZxid = 0x9038b36a5
ctime = Sun Dec 07 14:25:08 UTC 2014
mZxid = 0x9038b36a5
mtime = Sun Dec 07 14:25:08 UTC 2014
pZxid = 0x9038b36a5
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x34a175e1d5d0127
dataLength = 156
numChildren = 0
[zk: localhost:2181(CONNECTED) 7]
[zk: localhost:2181(CONNECTED) 7]
[zk: localhost:2181(CONNECTED) 7] storm-kafka8-1a ip-10-0-1-50:/mnt1/zookeeper-3.4.5# echo wchp | nc localhost
2181
/storm/assignments/lst_plugin_topology--LocationPlugin-kafka7-20141112_1002-126-1415786590
0x14959e6238007bd
0x14959e6238007bf
/assignments/lst_plugin_topology--SemantriaEnrichmentPlugin-kafka7-20140910_1756-6-1410371831
0x34839e733d70134
/storm/assignments/lst_plugin_topology--SemantriaEnrichmentPlugin-kafka7-20141207_1419-341-1417961987
0x14959e6238007bd
0x14959e6238007bf
/storm/assignments/LISTENING_RULE_ENGINE_kafka6-20141126_1236-316-1417005418
0x14959e6238007bd
0x14959e6238007bf
/storm/assignments/LISTENING_RULE_ENGINE_rule-engine-feed_342_v1-kafka6-20141126_1247-328-1417006092
0x14959e6238007bd
0x14959e6238007bf
/kafka/consumers/rule-engine-feed_177_v1/ids
0x14959e623801bf5
/kafka/brokers/topics/rule-engine-feed_399_v1
0x14959e623801bfd
/storm/assignments/lst_plugin_topology--LocationPlugin-kafka7-20141126_1254-335-1417006491
0x14959e6238007bd
0x14959e6238007bf
/assignments/LISTENING_RULE_ENGINE_kafka6-20141011_1056-215-1413025023
0x34839e733d70134
/storm/assignments/LISTENING_RULE_ENGINE_rule-engine-feed_177_v1-kafka6-20141126_1256-338-1417006656
0x14959e623801bfb
0x14959e6238007bd
0x14959e6238007bf
/storm/assignments/LISTENING_RULE_ENGINE_rule-engine-feed_1_v1_rule-engine-feed_113_v1_rule-engine-feed_129_v1_rule-engine-feed_249_v1_rule-engine-feed_399_v1-kafka6-20141126_1254-336-1417006545
0x3493583eee61f08
0x14959e6238007bd
0x14959e6238007bf
/storm/assignments/LISTENING_RULE_ENGINE_rule-engine-feed_177_v1-kafka6-20141126_1238-319-1417005583
0x14959e623801be6
0x14959e6238007bd
0x14959e6238007bf
/kafka/consumers/rule-engine-feed_1_v1/ids
0x14959e623801beb
/storm/assignments/LISTENING_RULE_ENGINE_rule-engine-feed_177_v1-kafka6-20141126_1245-326-1417005976
0x248c77fb3fe2e49
0x14959e6238007bd
0x14959e6238007bf
/storm/storms/LISTENING_RULE_ENGINE_kafka6-20141126_1242-323-1417005806
0x14959e623801bec
/storm/assignments/signalStream20141110_0428-59-1415593761
0x14959e6238007bd
0x14959e6238007bf
/storm/assignments/LISTENING_RULE_ENGINE_rule-engine-feed_1_v1_rule-engine-feed_113_v1_rule-engine-feed_129_v1_rule-engine-feed_249_v1_rule-engine-feed_399_v1-kafka6-20141126_1239-320-1417005640
0x14959e6238007bd
0x14959e6238007bf
0x14959e623801be8
/storm/assignments/LISTENING_RULE_ENGINE_kafka6-20141126_1233-313-1417005250
0x14959e6238007bd
0x14959e6238007bf
/storm/assignments/lst_plugin_topology--LocationPlugin-kafka7-20141126_1244-325-1417005915
0x3493583eee61ef7
0x14959e6238007bd
0x14959e6238007bf
/storm/storms/LISTENING_RULE_ENGINE_rule-engine-feed_342_v1-kafka6-20141126_1243-324-1417005861
0x14959e623801bed
/kafka/controller
0x149357e263b0008
/kafka/brokers/topics/rule-engine-feed
0x14959e623801be5
0x14959e623801bff
/storm/storms/LISTENING_RULE_ENGINE_rule-engine-feed_177_v1-kafka6-20141126_1253-334-1417006437
0x14959e623801bf6
/storm/assignments/lst_plugin_topology--LocationPlugin_999999_v1-kafka7-20141126_1235-315-1417005362
0x14959e6238007bd
0x14959e6238007bf
/storm/assignments/lst_plugin_topology--LocationPlugin_999999_v1-kafka7-20141126_1255-337-1417006598
0x14959e6238007bd
0x14959e6238007bf
0x14959e623801bf9
/storm/assignments/lst_plugin_topology--LocationPlugin-kafka7-20141126_1237-317-1417005473
0x14959e6238007bd
0x14959e6238007bf
/storm/storms/lst_plugin_topology--LocationPlugin_999999_v1-kafka7-20141126_1255-337-1417006598
0x14959e623801bf9
/storm/assignments/lst_plugin_topology--LocationPlugin-kafka7-20141112_1347-148-1415800121
0x14959e6238007bd
0x14959e6238007bf
/storm/assignments/LISTENING_RULE_ENGINE_rule-engine-feed_177_v1-kafka6-20141126_1241-322-1417005751
0x14959e6238007bd
0x14959e6238007bf
/storm/assignments/lst_plugin_topology--FullContactEnrichmentPlugin--RepustateEnrichmentPlugin-kafka7-20141126_1238-318-1417005528
0x14959e6238007bd
0x14959e6238007bf
0x3493583eee61eee
/assignments/lst_plugin_topology--SemantriaEnrichmentPlugin-kafka7-20141029_0906-348-1414573639
0x34839e733d70134
/assignments/LISTENING_RULE_ENGINE_kafka6-20141011_1053-212-1413024864
0x34839e733d70134
/kafka/brokers/topics/rule-engine-feed_1_v1
0x14959e623801beb
/kafka/brokers/topics/lst_plugin_com.spr.listening.plugin.impl.plugins.LocationPlugin_999999_v1
0x14959e623801bfe
0x14959e623801be4
/kafka/consumers/lst_plugin_com.spr.listening.plugin.impl.plugins.LocationPlugin/ids
0x14959e623801be3
0x14959e623801bfc
/kafka/consumers/rule-engine-feed_399_v1/ids
0x14959e623801bfd
/storm/assignments/LISTENING_RULE_ENGINE_rule-engine-feed_177_v1-kafka6-20141126_1250-331-1417006263
0x14959e6238007bd
0x14959e6238007bf
0x3493583eee61eff
/storm/assignments/LISTENING_RULE_ENGINE_rule-engine-feed_342_v1-kafka6-20141126_1243-324-1417005861
0x14959e6238007bd
0x14959e6238007bf
0x14959e623801bed
/storm/storms/LISTENING_RULE_ENGINE_rule-engine-feed_1_v1_rule-engine-feed_113_v1_rule-engine-feed_129_v1_rule-engine-feed_249_v1_rule-engine-feed_399_v1-kafka6-20141126_1239-320-1417005640
0x14959e623801be8
/kafka/brokers/topics/rule-engine-feed_177_v1
0x14959e623801bf5
/kafka/consumers/lst_plugin_com.spr.listening.plugin.impl.plugins.LocationPlugin_999999_v1/ids
0x14959e623801bfe
0x14959e623801be4
/storm/storms/LISTENING_RULE_ENGINE_rule-engine-feed_1_v1_rule-engine-feed_113_v1_rule-engine-feed_129_v1_rule-engine-feed_249_v1_rule-engine-feed_399_v1-kafka6-20141126_1254-336-1417006545
0x3493583eee61f08
/storm/storms/LISTENING_RULE_ENGINE_rule-engine-feed_177_v1-kafka6-20141126_1245-326-1417005976
0x248c77fb3fe2e49
/kafka/controller_epoch
0x149357e263b0008
/storm/storms/LISTENING_RULE_ENGINE_rule-engine-feed_177_v1-kafka6-20141126_1256-338-1417006656
0x14959e623801bfb
/kafka/brokers/topics/lst_plugin_com.spr.listening.plugin.impl.plugins.LocationPlugin
0x14959e623801be3
0x14959e623801bfc
/storm/assignments/lst_plugin_topology--LocationPlugin-kafka7-20141115_1204-185-1416053142
0x14959e6238007bd
0x14959e6238007bf
/storm/assignments/LISTENING_RULE_ENGINE_rule-engine-feed_342_v1-kafka6-20141126_1240-321-1417005696
0x14959e623801f35
0x14959e6238007bd
0x14959e6238007bf
/storm/assignments/LISTENING_RULE_ENGINE_rule-engine-feed_1_v1_rule-engine-feed_113_v1_rule-engine-feed_129_v1_rule-engine-feed_249_v1_rule-engine-feed_399_v1-kafka6-20141126_1249-330-1417006204
0x14959e6238007bd
0x14959e6238007bf
/storm/assignments/lst_plugin_topology--LocationPlugin_999999_v1-kafka7-20141126_1248-329-1417006147
0x14959e6238007bd
0x14959e6238007bf
/assignments/lst_plugin_topology--LocationPlugin_999999-kafka7-20140914_1652-87-1410713558
0x34839e733d70134
/storm/assignments/lst_plugin_topology--LocationPlugin_999999_v1-kafka7-20141110_1728-73-1415640537
0x14959e6238007bd
0x14959e6238007bf
/kafka/config/changes
0x149357e263b0008
/storm/assignments/lst_plugin_topology--LocationPlugin-kafka7-20141126_1234-314-1417005306
0x14959e6238007bd
0x14959e6238007bf
/storm/assignments/LISTENING_RULE_ENGINE_kafka6-20141126_1242-323-1417005806
0x14959e623801bec
0x14959e6238007bd
0x14959e6238007bf
/storm/storms/LISTENING_RULE_ENGINE_rule-engine-feed_177_v1-kafka6-20141126_1250-331-1417006263
0x3493583eee61eff
/storm/storms/LISTENING_RULE_ENGINE_rule-engine-feed_177_v1-kafka6-20141126_1238-319-1417005583
0x14959e623801be6
/storm/storms/lst_plugin_topology--FullContactEnrichmentPlugin--RepustateEnrichmentPlugin-kafka7-20141126_1238-318-1417005528
0x3493583eee61eee
/storm/assignments/signalStream20141126_1258-340-1417006770
0x14959e6238007bd
0x14959e6238007bf
/storm/assignments/LISTENING_RULE_ENGINE_kafka6-20141126_1257-339-1417006714
0x14959e6238007bd
0x14959e6238007bf
/storm/assignments/signalStream20141112_0002-121-1415750610
0x14959e6238007bd
/storm/storms/lst_plugin_topology--LocationPlugin-kafka7-20141126_1244-325-1417005915
0x3493583eee61ef7
/kafka/consumers/rule-engine-feed/ids
0x14959e623801be5
0x14959e623801bff
/storm/assignments/lst_plugin_topology--SemantriaEnrichmentPlugin-kafka7-20141207_1420-342-1417962060
0x14959e6238007bd
0x14959e6238007bf
/storm/assignments/LISTENING_RULE_ENGINE_rule-engine-feed_177_v1-kafka6-20141126_1253-334-1417006437
0x14959e623801bf6
0x14959e6238007bd
0x14959e6238007bf
/storm/assignments/LISTENING_RULE_ENGINE_kafka6-20141122_1114-255-1416654910
0x14959e6238007bd
0x14959e6238007bf
/storm/storms/LISTENING_RULE_ENGINE_rule-engine-feed_342_v1-kafka6-20141126_1240-321-1417005696
0x14959e623801f35
storm-kafka8-1a ip-10-0-1-50:/mnt1/zookeeper-3.4.5# echo wchp | nc localhost
2181
/storm/assignments/lst_plugin_topology--LocationPlugin-kafka7-20141112_1002-126-1415786590
0x14959e6238007bd
0x14959e6238007bf
/assignments/lst_plugin_topology--SemantriaEnrichmentPlugin-kafka7-20140910_1756-6-1410371831
0x34839e733d70134
/storm/assignments/lst_plugin_topology--SemantriaEnrichmentPlugin-kafka7-20141207_1419-341-1417961987
0x14959e6238007bd
0x14959e6238007bf
/storm/assignments/LISTENING_RULE_ENGINE_kafka6-20141126_1236-316-1417005418
0x14959e6238007bd
0x14959e6238007bf
/storm/assignments/LISTENING_RULE_ENGINE_rule-engine-feed_342_v1-kafka6-20141126_1247-328-1417006092
0x14959e6238007bd
0x14959e6238007bf
/kafka/consumers/rule-engine-feed_177_v1/ids
0x14959e623801bf5
/kafka/brokers/topics/rule-engine-feed_399_v1
0x14959e623801bfd
/storm/assignments/lst_plugin_topology--LocationPlugin-kafka7-20141126_1254-335-1417006491
0x14959e6238007bd
0x14959e6238007bf
/assignments/LISTENING_RULE_ENGINE_kafka6-20141011_1056-215-1413025023
0x34839e733d70134
/storm/assignments/LISTENING_RULE_ENGINE_rule-engine-feed_177_v1-kafka6-20141126_1256-338-1417006656
0x14959e623801bfb
0x14959e6238007bd
0x14959e6238007bf
/storm/assignments/LISTENING_RULE_ENGINE_rule-engine-feed_1_v1_rule-engine-feed_113_v1_rule-engine-feed_129_v1_rule-engine-feed_249_v1_rule-engine-feed_399_v1-kafka6-20141126_1254-336-1417006545
0x3493583eee61f08
0x14959e6238007bd
0x14959e6238007bf
/storm/assignments/LISTENING_RULE_ENGINE_rule-engine-feed_177_v1-kafka6-20141126_1238-319-1417005583
0x14959e623801be6
0x14959e6238007bd
0x14959e6238007bf
/kafka/consumers/rule-engine-feed_1_v1/ids
0x14959e623801beb
/storm/assignments/LISTENING_RULE_ENGINE_rule-engine-feed_177_v1-kafka6-20141126_1245-326-1417005976
0x248c77fb3fe2e49
0x14959e6238007bd
0x14959e6238007bf
/storm/storms/LISTENING_RULE_ENGINE_kafka6-20141126_1242-323-1417005806
0x14959e623801bec
/storm/assignments/signalStream20141110_0428-59-1415593761
0x14959e6238007bd
0x14959e6238007bf
/storm/assignments/LISTENING_RULE_ENGINE_rule-engine-feed_1_v1_rule-engine-feed_113_v1_rule-engine-feed_129_v1_rule-engine-feed_249_v1_rule-engine-feed_399_v1-kafka6-20141126_1239-320-1417005640
0x14959e6238007bd
0x14959e6238007bf
0x14959e623801be8
/storm/assignments/LISTENING_RULE_ENGINE_kafka6-20141126_1233-313-1417005250
0x14959e6238007bd
0x14959e6238007bf
/storm/assignments/lst_plugin_topology--LocationPlugin-kafka7-20141126_1244-325-1417005915
0x3493583eee61ef7
0x14959e6238007bd
0x14959e6238007bf
/storm/storms/LISTENING_RULE_ENGINE_rule-engine-feed_342_v1-kafka6-20141126_1243-324-1417005861
0x14959e623801bed
/kafka/controller
0x149357e263b0008
/kafka/brokers/topics/rule-engine-feed
0x14959e623801be5
0x14959e623801bff
/storm/storms/LISTENING_RULE_ENGINE_rule-engine-feed_177_v1-kafka6-20141126_1253-334-1417006437
0x14959e623801bf6
/storm/assignments/lst_plugin_topology--LocationPlugin_999999_v1-kafka7-20141126_1235-315-1417005362
0x14959e6238007bd
0x14959e6238007bf
/storm/assignments/lst_plugin_topology--LocationPlugin_999999_v1-kafka7-20141126_1255-337-1417006598
0x14959e6238007bd
0x14959e6238007bf
0x14959e623801bf9
/storm/assignments/lst_plugin_topology--LocationPlugin-kafka7-20141126_1237-317-1417005473
0x14959e6238007bd
0x14959e6238007bf
/storm/storms/lst_plugin_topology--LocationPlugin_999999_v1-kafka7-20141126_1255-337-1417006598
0x14959e623801bf9
/storm/assignments/lst_plugin_topology--LocationPlugin-kafka7-20141112_1347-148-1415800121
0x14959e6238007bd
0x14959e6238007bf
/storm/assignments/LISTENING_RULE_ENGINE_rule-engine-feed_177_v1-kafka6-20141126_1241-322-1417005751
0x14959e6238007bd
0x14959e6238007bf
/storm/assignments/lst_plugin_topology--FullContactEnrichmentPlugin--RepustateEnrichmentPlugin-kafka7-20141126_1238-318-1417005528
0x14959e6238007bd
0x14959e6238007bf
0x3493583eee61eee
/assignments/lst_plugin_topology--SemantriaEnrichmentPlugin-kafka7-20141029_0906-348-1414573639
0x34839e733d70134
/assignments/LISTENING_RULE_ENGINE_kafka6-20141011_1053-212-1413024864
0x34839e733d70134
/kafka/brokers/topics/rule-engine-feed_1_v1
0x14959e623801beb
/kafka/brokers/topics/lst_plugin_com.spr.listening.plugin.impl.plugins.LocationPlugin_999999_v1
0x14959e623801bfe
0x14959e623801be4
/kafka/consumers/lst_plugin_com.spr.listening.plugin.impl.plugins.LocationPlugin/ids
0x14959e623801be3
0x14959e623801bfc
/kafka/consumers/rule-engine-feed_399_v1/ids
0x14959e623801bfd
/storm/assignments/LISTENING_RULE_ENGINE_rule-engine-feed_177_v1-kafka6-20141126_1250-331-1417006263
0x14959e6238007bd
0x14959e6238007bf
0x3493583eee61eff
/storm/assignments/LISTENING_RULE_ENGINE_rule-engine-feed_342_v1-kafka6-20141126_1243-324-1417005861
0x14959e6238007bd
0x14959e6238007bf
0x14959e623801bed
/storm/storms/LISTENING_RULE_ENGINE_rule-engine-feed_1_v1_rule-engine-feed_113_v1_rule-engine-feed_129_v1_rule-engine-feed_249_v1_rule-engine-feed_399_v1-kafka6-20141126_1239-320-1417005640
0x14959e623801be8
/kafka/brokers/topics/rule-engine-feed_177_v1
0x14959e623801bf5
/kafka/consumers/lst_plugin_com.spr.listening.plugin.impl.plugins.LocationPlugin_999999_v1/ids
0x14959e623801bfe
0x14959e623801be4
/storm/storms/LISTENING_RULE_ENGINE_rule-engine-feed_1_v1_rule-engine-feed_113_v1_rule-engine-feed_129_v1_rule-engine-feed_249_v1_rule-engine-feed_399_v1-kafka6-20141126_1254-336-1417006545
0x3493583eee61f08
/storm/storms/LISTENING_RULE_ENGINE_rule-engine-feed_177_v1-kafka6-20141126_1245-326-1417005976
0x248c77fb3fe2e49
/kafka/controller_epoch
0x149357e263b0008
/storm/storms/LISTENING_RULE_ENGINE_rule-engine-feed_177_v1-kafka6-20141126_1256-338-1417006656
0x14959e623801bfb
/kafka/brokers/topics/lst_plugin_com.spr.listening.plugin.impl.plugins.LocationPlugin
0x14959e623801be3
0x14959e623801bfc
/storm/assignments/lst_plugin_topology--LocationPlugin-kafka7-20141115_1204-185-1416053142
0x14959e6238007bd
0x14959e6238007bf
/storm/assignments/LISTENING_RULE_ENGINE_rule-engine-feed_342_v1-kafka6-20141126_1240-321-1417005696
0x14959e623801f35
0x14959e6238007bd
0x14959e6238007bf
/storm/assignments/LISTENING_RULE_ENGINE_rule-engine-feed_1_v1_rule-engine-feed_113_v1_rule-engine-feed_129_v1_rule-engine-feed_249_v1_rule-engine-feed_399_v1-kafka6-20141126_1249-330-1417006204
0x14959e6238007bd
0x14959e6238007bf
/storm/assignments/lst_plugin_topology--LocationPlugin_999999_v1-kafka7-20141126_1248-329-1417006147
0x14959e6238007bd
0x14959e6238007bf
/assignments/lst_plugin_topology--LocationPlugin_999999-kafka7-20140914_1652-87-1410713558
0x34839e733d70134
/storm/assignments/lst_plugin_topology--LocationPlugin_999999_v1-kafka7-20141110_1728-73-1415640537
0x14959e6238007bd
0x14959e6238007bf
/kafka/config/changes
0x149357e263b0008
/storm/assignments/lst_plugin_topology--LocationPlugin-kafka7-20141126_1234-314-1417005306
0x14959e6238007bd
0x14959e6238007bf
/storm/assignments/LISTENING_RULE_ENGINE_kafka6-20141126_1242-323-1417005806
0x14959e623801bec
0x14959e6238007bd
0x14959e6238007bf
/storm/storms/LISTENING_RULE_ENGINE_rule-engine-feed_177_v1-kafka6-20141126_1250-331-1417006263
0x3493583eee61eff
/storm/storms/LISTENING_RULE_ENGINE_rule-engine-feed_177_v1-kafka6-20141126_1238-319-1417005583
0x14959e623801be6
/storm/storms/lst_plugin_topology--FullContactEnrichmentPlugin--RepustateEnrichmentPlugin-kafka7-20141126_1238-318-1417005528
0x3493583eee61eee
/storm/assignments/signalStream20141126_1258-340-1417006770
0x14959e6238007bd
0x14959e6238007bf
/storm/assignments/LISTENING_RULE_ENGINE_kafka6-20141126_1257-339-1417006714
0x14959e6238007bd
0x14959e6238007bf
/storm/assignments/signalStream20141112_0002-121-1415750610
0x14959e6238007bd
/storm/storms/lst_plugin_topology--LocationPlugin-kafka7-20141126_1244-325-1417005915
0x3493583eee61ef7
/kafka/consumers/rule-engine-feed/ids
0x14959e623801be5
0x14959e623801bff
/storm/assignments/lst_plugin_topology--SemantriaEnrichmentPlugin-kafka7-20141207_1420-342-1417962060
0x14959e6238007bd
0x14959e6238007bf
/storm/assignments/LISTENING_RULE_ENGINE_rule-engine-feed_177_v1-kafka6-20141126_1253-334-1417006437
0x14959e623801bf6
0x14959e6238007bd
0x14959e6238007bf
/storm/assignments/LISTENING_RULE_ENGINE_kafka6-20141122_1114-255-1416654910
0x14959e6238007bd
0x14959e6238007bf
/storm/storms/LISTENING_RULE_ENGINE_rule-engine-feed_342_v1-kafka6-20141126_1240-321-1417005696
0x14959e623801f35
storm-kafka8-1a ip-10-0-1-50:/mnt1/zookeeper-3.4.5#
storm-kafka8-1b ip-10-0-2-147:~# echo wchp | nc localhost 2181
/storm/assignments/lst_plugin_topology--LocationPlugin-kafka7-20141112_1002-126-1415786590
0x248c77fb3fe1a07
0x248c77fb3fe1a32
0x248c77fb3fe1a08
0x248c77fb3fe1a0a
/kafka/consumers/rule-engine-feed_129_v1/ids
0x248c77fb3fe2e43
0x248c77fb3fe2e58
/assignments/LISTENING_RULE_ENGINE_rule-engine-feed_177_v1-kafka6-20141125_1510-304-1416928286
0x3493583eee60a5a
/kafka/consumers/rule-engine-feed_249_v1/ids
0x248c77fb3fe2e44
/storm/assignments/lst_plugin_topology--SemantriaEnrichmentPlugin-kafka7-20141207_1419-341-1417961987
0x248c77fb3fe1a08
0x248c77fb3fe3284
0x248c77fb3fe1a32
0x248c77fb3fe1a07
0x3493583eee60a5a
0x248c77fb3fe1a0a
/storm/assignments/LISTENING_RULE_ENGINE_kafka6-20141126_1236-316-1417005418
0x248c77fb3fe1a08
0x248c77fb3fe2e3e
0x248c77fb3fe1a07
0x248c77fb3fe1a32
0x3493583eee60a5a
0x248c77fb3fe1a0a
/kafka/brokers/topics/rule-engine-feed_399
0x248c77fb3fe0000
/kafka/admin/reassign_partitions
0x248c77fb3fe0000
/storm/assignments/LISTENING_RULE_ENGINE_rule-engine-feed_342_v1-kafka6-20141126_1247-328-1417006092
0x248c77fb3fe1a08
0x248c77fb3fe1a32
0x248c77fb3fe1a07
0x3493583eee60a5a
0x248c77fb3fe1a0a
0x3493583eee61efa
/kafka/consumers/rule-engine-feed_177_v1/ids
0x248c77fb3fe2e55
/kafka/brokers/topics/rule-engine-feed_399_v1
0x248c77fb3fe2e51
0x248c77fb3fe0000
/storm/storms/lst_plugin_topology--LocationPlugin_999999_v1-kafka7-20141126_1235-315-1417005362
0x248c77fb3fe2e3b
/kafka/brokers/topics/rule-engine-feed_1
0x248c77fb3fe0000
/storm/assignments/lst_plugin_topology--LocationPlugin-kafka7-20141126_1254-335-1417006491
0x248c77fb3fe1a08
0x248c77fb3fe2e52
0x248c77fb3fe1a32
0x248c77fb3fe1a07
0x3493583eee60a5a
0x248c77fb3fe1a0a
/kafka/brokers/topics/rule-engine-feed_113
0x248c77fb3fe0000
/storm/assignments/LISTENING_RULE_ENGINE_rule-engine-feed_177_v1-kafka6-20141126_1256-338-1417006656
0x248c77fb3fe1a08
0x248c77fb3fe1a32
0x248c77fb3fe1a07
0x3493583eee60a5a
0x248c77fb3fe1a0a
/storm/assignments/LISTENING_RULE_ENGINE_kafka6-20141111_2351-109-1415749941
0x248c77fb3fe1a08
/storm/assignments/LISTENING_RULE_ENGINE_rule-engine-feed_1_v1_rule-engine-feed_113_v1_rule-engine-feed_129_v1_rule-engine-feed_249_v1_rule-engine-feed_399_v1-kafka6-20141126_1254-336-1417006545
0x248c77fb3fe1a08
0x248c77fb3fe1a32
0x248c77fb3fe1a07
0x3493583eee60a5a
0x248c77fb3fe1a0a
/storm/assignments/LISTENING_RULE_ENGINE_rule-engine-feed_177_v1-kafka6-20141126_1238-319-1417005583
0x248c77fb3fe1a08
0x248c77fb3fe1a32
0x248c77fb3fe1a07
0x3493583eee60a5a
0x248c77fb3fe1a0a
/storm/assignments/LISTENING_RULE_ENGINE_rule-engine-feed_1_v1_rule-engine-feed_113_v1_rule-engine-feed_129_v1_rule-engine-feed_249_v1_rule-engine-feed_342_v1_rule-engine-feed_399_v1-kafka6-20141115_1218-199-1416053942
0x248c77fb3fe1a0a
/storm/assignments/LISTENING_RULE_ENGINE_rule-engine-feed_177_v1-kafka6-20141126_1245-326-1417005976
0x248c77fb3fe1a08
0x248c77fb3fe1a32
0x248c77fb3fe1a07
0x3493583eee60a5a
0x248c77fb3fe1a0a
/storm/assignments/signalStream20141110_0428-59-1415593761
0x248c77fb3fe1a07
0x248c77fb3fe1a32
0x248c77fb3fe1a08
0x248c77fb3fe1a0a
/kafka/brokers/topics/lst_plugin_com.spr.listening.plugin.impl.plugins.RepustateEnrichmentPlugin
0x248c77fb3fe0000
/storm/storms/LISTENING_RULE_ENGINE_kafka6-20141126_1236-316-1417005418
0x248c77fb3fe2e3e
/storm/assignments/LISTENING_RULE_ENGINE_rule-engine-feed_1_v1_rule-engine-feed_113_v1_rule-engine-feed_129_v1_rule-engine-feed_249_v1_rule-engine-feed_399_v1-kafka6-20141126_1239-320-1417005640
0x248c77fb3fe1a08
0x248c77fb3fe1a32
0x248c77fb3fe1a07
0x3493583eee60a5a
0x248c77fb3fe1a0a
/storm/assignments/LISTENING_RULE_ENGINE_kafka6-20141126_1233-313-1417005250
0x248c77fb3fe1a08
0x248c77fb3fe1a32
0x248c77fb3fe1a07
0x3493583eee60a5a
0x248c77fb3fe1a0a
0x248c77fb3fe2e38
/storm/assignments/lst_plugin_topology--LocationPlugin-kafka7-20141126_1244-325-1417005915
0x248c77fb3fe1a08
0x248c77fb3fe1a32
0x248c77fb3fe1a07
0x3493583eee60a5a
0x248c77fb3fe1a0a
/kafka/controller
0x348c77fb3cc103e
0x248c77fb3fe0000
/storm/storms/LISTENING_RULE_ENGINE_kafka6-20141126_1257-339-1417006714
0x248c77fb3fe2e57
/kafka/brokers/topics/rule-engine-feed
0x248c77fb3fe2e3c
0x248c77fb3fe2e47
0x248c77fb3fe0000
/storm/assignments/lst_plugin_topology--LocationPlugin_999999_v1-kafka7-20141126_1235-315-1417005362
0x248c77fb3fe1a08
0x248c77fb3fe1a32
0x248c77fb3fe1a07
0x3493583eee60a5a
0x248c77fb3fe2e3b
0x248c77fb3fe1a0a
/kafka/brokers/topics/rule-engine-feed_342_v1
0x248c77fb3fe0000
/storm/assignments/lst_plugin_topology--LocationPlugin_999999_v1-kafka7-20141126_1255-337-1417006598
0x248c77fb3fe1a08
0x248c77fb3fe1a32
0x248c77fb3fe1a07
0x3493583eee60a5a
0x248c77fb3fe1a0a
/storm/storms/signalStream20141126_1258-340-1417006770
0x3493583eee61f0d
/storm/assignments/lst_plugin_topology--LocationPlugin-kafka7-20141112_1351-152-1415800339
0x248c77fb3fe1a08
/kafka/brokers/ids
0x248c77fb3fe0000
/storm/assignments/LISTENING_RULE_ENGINE_kafka6-20141125_1453-286-1416927245
0x248c77fb3fe1a08
/storm/assignments/lst_plugin_topology--LocationPlugin-kafka7-20141126_1237-317-1417005473
0x248c77fb3fe2e3f
0x248c77fb3fe1a08
0x248c77fb3fe1a32
0x248c77fb3fe1a07
0x3493583eee60a5a
0x248c77fb3fe1a0a
/storm/assignments/lst_plugin_topology--LocationPlugin-kafka7-20141112_1347-148-1415800121
0x248c77fb3fe1a07
0x248c77fb3fe1a32
0x248c77fb3fe1a08
0x248c77fb3fe1a0a
/kafka/brokers/topics/rule-engine-feed_129
0x248c77fb3fe0000
/storm/storms/lst_plugin_topology--LocationPlugin_999999_v1-kafka7-20141126_1248-329-1417006147
0x248c77fb3fe2e4d
/storm/assignments/LISTENING_RULE_ENGINE_rule-engine-feed_177_v1-kafka6-20141126_1241-322-1417005751
0x248c77fb3fe1a08
0x248c77fb3fe1a32
0x248c77fb3fe1a07
0x3493583eee60a5a
0x248c77fb3fe2e41
0x248c77fb3fe1a0a
/storm/assignments/lst_plugin_topology--FullContactEnrichmentPlugin--RepustateEnrichmentPlugin-kafka7-20141126_1238-318-1417005528
0x248c77fb3fe1a08
0x248c77fb3fe1a32
0x248c77fb3fe1a07
0x3493583eee60a5a
0x248c77fb3fe1a0a
/kafka/brokers/topics/full-contact-parse
0x248c77fb3fe0000
/storm/storms/LISTENING_RULE_ENGINE_rule-engine-feed_177_v1-kafka6-20141126_1241-322-1417005751
0x248c77fb3fe2e41
/kafka/brokers/topics/rule-engine-feed_113_v1
0x248c77fb3fe0000
0x248c77fb3fe2e42
/assignments/signalStream20141110_0428-59-1415593761
0x3493583eee60a5a
/kafka/brokers/topics/rule-engine-feed_1_v1
0x248c77fb3fe0000
/kafka/brokers/topics/lst_plugin_com.spr.listening.plugin.impl.plugins.LocationPlugin_999999_v1
0x248c77fb3fe0000
0x248c77fb3fe2e50
/kafka/brokers/topics/lst_plugin_com.spr.listening.plugin.impl.plugins.FullContactEnrichmentPlugin
0x248c77fb3fe0000
/kafka/consumers/lst_plugin_com.spr.listening.plugin.impl.plugins.LocationPlugin/ids
0x248c77fb3fe2e4b
/kafka/consumers/rule-engine-feed_399_v1/ids
0x248c77fb3fe2e51
/storm/assignments/LISTENING_RULE_ENGINE_rule-engine-feed_177_v1-kafka6-20141126_1250-331-1417006263
0x248c77fb3fe1a08
0x248c77fb3fe1a32
0x248c77fb3fe1a07
0x3493583eee60a5a
0x248c77fb3fe1a0a
/storm/assignments/LISTENING_RULE_ENGINE_rule-engine-feed_342_v1-kafka6-20141126_1243-324-1417005861
0x248c77fb3fe1a08
0x248c77fb3fe1a32
0x248c77fb3fe1a07
0x3493583eee60a5a
0x248c77fb3fe1a0a
/assignments/lst_plugin_topology--LocationPlugin-kafka7-20141112_1002-126-1415786590
0x3493583eee60a5a
/storm/storms/lst_plugin_topology--LocationPlugin-kafka7-20141126_1254-335-1417006491
0x248c77fb3fe2e52
/kafka/brokers/topics/rule-engine-feed_177_v1
0x248c77fb3fe0000
0x248c77fb3fe2e55
/kafka/consumers/lst_plugin_com.spr.listening.plugin.impl.plugins.LocationPlugin_999999_v1/ids
0x248c77fb3fe2e50
/kafka/brokers/topics
0x248c77fb3fe0000
/kafka/brokers/topics/rule-engine-feed_249_v1
0x248c77fb3fe2e44
0x248c77fb3fe0000
/kafka/controller_epoch
0x348c77fb3cc103e
0x248c77fb3fe0000
/storm/storms/lst_plugin_topology--SemantriaEnrichmentPlugin-kafka7-20141207_1419-341-1417961987
0x248c77fb3fe3284
/kafka/brokers/topics/lst_plugin_com.spr.listening.plugin.impl.plugins.LocationPlugin
0x248c77fb3fe0000
0x248c77fb3fe2e4b
/storm/assignments/lst_plugin_topology--LocationPlugin-kafka7-20141115_1204-185-1416053142
0x248c77fb3fe1a07
0x248c77fb3fe1a08
/kafka/brokers/topics/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin
0x248c77fb3fe0000
/storm/assignments/LISTENING_RULE_ENGINE_rule-engine-feed_342_v1-kafka6-20141126_1240-321-1417005696
0x248c77fb3fe1a08
0x248c77fb3fe1a32
0x248c77fb3fe1a07
0x3493583eee60a5a
0x248c77fb3fe1a0a
/kafka/admin/preferred_replica_election
0x248c77fb3fe0000
/storm/assignments/LISTENING_RULE_ENGINE_rule-engine-feed_1_v1_rule-engine-feed_113_v1_rule-engine-feed_129_v1_rule-engine-feed_249_v1_rule-engine-feed_399_v1-kafka6-20141126_1249-330-1417006204
0x248c77fb3fe1a08
0x248c77fb3fe1a32
0x248c77fb3fe1a07
0x3493583eee60a5a
0x248c77fb3fe1a0a
0x3493583eee61efc
/storm/assignments/lst_plugin_topology--LocationPlugin_999999_v1-kafka7-20141126_1248-329-1417006147
0x248c77fb3fe1a08
0x248c77fb3fe2e4d
0x248c77fb3fe1a32
0x248c77fb3fe1a07
0x3493583eee60a5a
0x248c77fb3fe1a0a
/kafka/brokers/topics/rule-engine-feed_129_v1
0x248c77fb3fe2e43
0x248c77fb3fe0000
0x248c77fb3fe2e58
/kafka/consumers/rule-engine-feed_113_v1/ids
0x248c77fb3fe2e42
/storm/storms/lst_plugin_topology--LocationPlugin-kafka7-20141126_1237-317-1417005473
0x248c77fb3fe2e3f
/storm/assignments/lst_plugin_topology--LocationPlugin_999999_v1-kafka7-20141110_1728-73-1415640537
0x248c77fb3fe1a07
0x248c77fb3fe1a32
0x248c77fb3fe1a08
0x248c77fb3fe1a0a
/storm/assignments/lst_plugin_topology--LocationPlugin-kafka7-20141126_1234-314-1417005306
0x248c77fb3fe1a08
0x248c77fb3fe2e39
0x248c77fb3fe1a32
0x248c77fb3fe1a07
0x3493583eee60a5a
0x248c77fb3fe1a0a
/kafka/config/changes
0x348c77fb3cc103e
0x248c77fb3fe0000
/storm/assignments/LISTENING_RULE_ENGINE_kafka6-20141126_1242-323-1417005806
0x248c77fb3fe1a08
0x248c77fb3fe1a32
0x248c77fb3fe1a07
0x3493583eee60a5a
0x248c77fb3fe1a0a
/kafka/brokers/topics/rule-engine-feed_342
0x248c77fb3fe0000
/storm/assignments/signalStream20141126_1258-340-1417006770
0x248c77fb3fe1a08
0x3493583eee61f0d
0x248c77fb3fe1a32
0x248c77fb3fe1a07
0x3493583eee60a5a
0x248c77fb3fe1a0a
/kafka/brokers/topics/rule-engine-feed_249
0x248c77fb3fe0000
/storm/assignments/LISTENING_RULE_ENGINE_kafka6-20141126_1257-339-1417006714
0x248c77fb3fe2e57
0x248c77fb3fe1a08
0x248c77fb3fe1a32
0x248c77fb3fe1a07
0x3493583eee60a5a
0x248c77fb3fe1a0a
/storm/storms/LISTENING_RULE_ENGINE_rule-engine-feed_342_v1-kafka6-20141126_1247-328-1417006092
0x3493583eee61efa
/kafka/consumers/rule-engine-feed/ids
0x248c77fb3fe2e3c
0x248c77fb3fe2e47
/storm/storms/lst_plugin_topology--LocationPlugin-kafka7-20141126_1234-314-1417005306
0x248c77fb3fe2e39
/kafka/brokers/topics/lst_plugin_com.spr.listening.plugin.impl.plugins.LocationPlugin_999999
0x248c77fb3fe0000
/storm/assignments/lst_plugin_topology--SemantriaEnrichmentPlugin-kafka7-20141207_1420-342-1417962060
0x248c77fb3fe1a08
0x248c77fb3fe1a32
0x248c77fb3fe1a07
0x3493583eee60a5a
0x248c77fb3fe1a0a
/storm/storms/LISTENING_RULE_ENGINE_rule-engine-feed_1_v1_rule-engine-feed_113_v1_rule-engine-feed_129_v1_rule-engine-feed_249_v1_rule-engine-feed_399_v1-kafka6-20141126_1249-330-1417006204
0x3493583eee61efc
/storm/assignments/LISTENING_RULE_ENGINE_rule-engine-feed_177_v1-kafka6-20141126_1253-334-1417006437
0x248c77fb3fe1a08
0x248c77fb3fe1a32
0x248c77fb3fe1a07
0x3493583eee60a5a
0x248c77fb3fe1a0a
/storm/assignments/LISTENING_RULE_ENGINE_kafka6-20141122_1114-255-1416654910
0x248c77fb3fe1a07
0x248c77fb3fe1a08
0x248c77fb3fe1a0a
/storm/storms/LISTENING_RULE_ENGINE_kafka6-20141126_1233-313-1417005250
0x248c77fb3fe2e38
storm-kafka8-1b ip-10-0-2-147:~#
storm-kafka8-1e ip-10-0-0-45:~# echo wchp | nc localhost 2181
/kafka/brokers/topics/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin
0x34a175e1d5d0130
/kafka/consumers/rule-engine-feed_342_v1/ids
0x34a175e1d5d0013
/kafka/consumers/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin/ids
0x34a175e1d5d0130
/storm/storms/lst_plugin_topology--SemantriaEnrichmentPlugin-kafka7-20141207_1420-342-1417962060
0x34a175e1d5d0124
/kafka/brokers/topics/rule-engine-feed_342_v1
0x34a175e1d5d0013
/storm/assignments/lst_plugin_topology--SemantriaEnrichmentPlugin-kafka7-20141207_1420-342-1417962060
0x34a175e1d5d0124
storm-kafka8-1e ip-10-0-0-45:~#
2014-12-07 14:55:49 k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
begin rebalancing consumer
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d
try #392014-12-07 14:55:49 k.c.ConsumerFetcherManager [INFO]
[ConsumerFetcherManager-1417963753624] Stopping leader finder thread
2014-12-07 14:55:49 k.c.ConsumerFetcherManager [INFO]
[ConsumerFetcherManager-1417963753624] Stopping all fetchers2014-12-07 14:55:49
k.c.ConsumerFetcherManager [INFO] [ConsumerFetcherManager-1417963753624] All
connections stopped2014-12-07 14:55:49 k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
Cleared all relevant queues for this fetcher2014-12-07 14:55:49
k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
Cleared the data chunks in all the consumer message iterators2014-12-07
14:55:49 k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
Committing all offsets after clearing the fetcher queues
2014-12-07 14:55:49 k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
Releasing partition ownership
2014-12-07 14:55:49 k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
Consumer
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d
rebalancing the following partitions: ArrayBuffer(0, 1, 2, 3, 4, 5, 6, 7, 8,
9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28,
29) for topic
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin with
consumers:
List(lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d-0,
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d-1,
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d-2,
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d-3,
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d-4,
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-225-1417962308114-dab7928d-0,
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-225-1417962308114-dab7928d-1,
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-225-1417962308114-dab7928d-2,
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-225-1417962308114-dab7928d-3,
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-225-1417962308114-dab7928d-4)2014-12-07
14:55:49 k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d-1
attempting to claim partition 32014-12-07 14:55:49
k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d-1
attempting to claim partition 42014-12-07 14:55:49
k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d-1
attempting to claim partition 5
2014-12-07 14:55:49 k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
lst_p
lugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d-4
attempting to claim partition 122014-12-07 14:55:49
k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
lst_p
lugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d-4
attempting to claim partition 13
2014-12-07 14:55:49 k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d-4
attempting to claim partition 12
2014-12-07 14:55:49 k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d-4
attempting to claim partition 13
2014-12-07 14:55:49 k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d-4
attempting to claim partition 14
2014-12-07 14:55:49 k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d-2
attempting to claim partition 6
2014-12-07 14:55:49 k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d-2
attempting to claim partition 7
2014-12-07 14:55:49 k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d-2
attempting to claim partition 82014-12-07 14:55:49
k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d-3
attempting to claim partition 92014-12-07 14:55:49
k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d-3
attempting to claim partition 102014-12-07 14:55:49
k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d-3
attempting to claim partition 11
2014-12-07 14:55:49 k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d-0
attempting to claim partition 0
2014-12-07 14:55:49 k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d-0
attempting to claim partition 12014-12-07 14:55:49
k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d-0
attempting to claim partition 22014-12-07 14:55:49 k.u.ZkUtils$ [INFO]
conflict in
/consumers/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin/owners/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin/4
data:
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d-1
stored data:
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-225-1417962308114-dab7928d-02014-12-07
14:55:49 k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
waiting for the partition ownership to be deleted: 42014-12-07 14:55:49
k.u.ZkUtils$ [INFO] conflict in
/consumers/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin/owners/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin/12
data:
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d-4
stored data:
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-225-1417962308114-dab7928d-22014-12-07
14:55:49 k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
waiting for the partition ownership to be deleted: 122014-12-07 14:55:49
k.u.ZkUtils$ [INFO] conflict in
/consumers/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin/owners/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin/11
data:
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d-3
stored data:
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-225-1417962308114-dab7928d-1
2014-12-07 14:55:49 k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
waiting for the partition ownership to be deleted: 11
2014-12-07 14:55:49 k.u.ZkUtils$ [INFO] conflict in
/consumers/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin/owners/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin/11
data:
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d-3
stored data:
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-225-1417962308114-dab7928d-1
2014-12-07 14:55:49 k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
waiting for the partition ownership to be deleted: 11
2014-12-07 14:55:49 k.u.ZkUtils$ [INFO] conflict in
/consumers/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin/owners/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin/6
data:
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d-2
stored data:
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-225-1417962308114-dab7928d-1
2014-12-07 14:55:49 k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
waiting for the partition ownership to be deleted: 6
2014-12-07 14:55:49 k.u.ZkUtils$ [INFO] conflict in
/consumers/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin/owners/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin/9
data:
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d-3
stored data:
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-225-1417962308114-dab7928d-1
2014-12-07 14:55:49 k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
waiting for the partition ownership to be deleted: 9
2014-12-07 14:55:49 k.u.ZkUtils$ [INFO] conflict in
/consumers/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin/owners/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin/10
data:
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d-3
stored data:
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-225-1417962308114-dab7928d-1
2014-12-07 14:55:49 k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
waiting for the partition ownership to be deleted: 10
2014-12-07 14:55:49 k.u.ZkUtils$ [INFO] conflict in
/consumers/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin/owners/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin/13
data:
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d-4
stored data:
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-225-1417962308114-dab7928d-2
2014-12-07 14:55:49 k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
waiting for the partition ownership to be deleted: 13
2014-12-07 14:55:49 k.u.ZkUtils$ [INFO] conflict in
/consumers/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin/owners/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin/5
data:
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d-1
stored data:
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-225-1417962308114-dab7928d-0
2014-12-07 14:55:49 k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
waiting for the partition ownership to be deleted: 52014-12-07 14:55:49
k.u.ZkUtils$ [INFO] conflict in
/consumers/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin/owners/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin/2
data:
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d-0
stored data:
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-225-1417962308114-dab7928d-02014-12-07
14:55:49 k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
waiting for the partition ownership to be deleted: 22014-12-07 14:55:49
k.u.ZkUtils$ [INFO] conflict in
/consumers/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin/owners/lst_plugin_com.spr.listening.plug
in.impl.plugins.SemantriaEnrichmentPlugin/14 data:
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d-4
stored data:
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-225-1417962308114-dab7928d-22014-12-07
14:55:49 k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
waiti
ng for the partition ownership to be deleted: 14
2014-12-07 14:55:49 k.u.ZkUtils$ [INFO] conflict in
/consumers/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin/owners/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin/14
data:
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d-4
stored data:
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-225-1417962308114-dab7928d-2
2014-12-07 14:55:49 k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
waiting for the partition ownership to be deleted: 14
2014-12-07 14:55:49 k.u.ZkUtils$ [INFO] conflict in
/consumers/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin/owners/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin/1
data:
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d-0
stored data:
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-225-1417962308114-dab7928d-0
2014-12-07 14:55:49 k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
waiting for the partition ownership to be deleted: 1
2014-12-07 14:55:49 k.u.ZkUtils$ [INFO] conflict in
/consumers/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin/owners/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin/0
data:
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d-0
stored data:
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-225-1417962308114-dab7928d-0
2014-12-07 14:55:49 k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
waiting for the partition ownership to be deleted: 0
2014-12-07 14:55:49 k.u.ZkUtils$ [INFO] conflict in
/consumers/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin/owners/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin/7
data:
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d-2
stored data:
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-225-1417962308114-dab7928d-1
2014-12-07 14:55:49 k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
waiting for the partition ownership to be deleted: 7
2014-12-07 14:55:50 k.u.ZkUtils$ [INFO] conflict in
/consumers/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin/owners/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin/3
data:
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d-1
stored data:
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-225-1417962308114-dab7928d-0
2014-12-07 14:55:50 k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
waiting for the partition ownership to be deleted: 3
2014-12-07 14:55:50 k.u.ZkUtils$ [INFO] conflict in
/consumers/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin/owners/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin/8
data:
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d-2
stored data:
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-225-1417962308114-dab7928d-1
2014-12-07 14:55:50 k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
waiting for the partition ownership to be deleted: 82014-12-07 14:55:50
k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
end rebalancing consumer
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d
try #39
2014-12-07 14:55:50 k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
Rebalancing attempt failed. Clearing the cache before the next rebalancing
operation is triggered2014-12-07 14:55:50 k.c.ConsumerFetcherManager [INFO]
[ConsumerFetcherManager-1417963753624] Stopping leader finder thread2014-12-07
14:55:50 k.c.ConsumerFetcherManager [INFO]
[ConsumerFetcherManager-1417963753624] Stopping all fetchers
2014-12-07 14:55:50 k.c.ConsumerFetcherManager [INFO]
[ConsumerFetcherManager-1417963753624] All connections stopped
2014-12-07 14:55:50 k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
Cleared all relevant queues for this fetcher
2014-12-07 14:55:50 k.c.ConsumerFetcherManager [INFO]
[ConsumerFetcherManager-1417963753624] Stopping all fetchers
2014-12-07 14:55:50 k.c.ConsumerFetcherManager [INFO]
[ConsumerFetcherManager-1417963753624] All connections stopped
2014-12-07 14:55:50 k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
Cleared all relevant queues for this fetcher
2014-12-07 14:55:50 k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
Cleared the data chunks in all the consumer message iterators
2014-12-07 14:55:50 k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
Committing all offsets after clearing the fetcher queues
2014-12-07 14:56:00 c.s.m.k.i.c.KafkaFeedStreamer [WARN] Ignoring exception
while trying to start streamer threads:
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d
can't rebalance after 40 retries
kafka.common.ConsumerRebalanceFailedException:
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d
can't rebalance after 40 retries
at
kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:432)
~[stormjar.jar:na]
at
kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:722)
~[stormjar.jar:na]
at
kafka.consumer.ZookeeperConsumerConnector.consume(ZookeeperConsumerConnector.scala:212)
~[stormjar.jar:na]
at
kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:80)
~[stormjar.jar:na]
at
com.spr.messaging.kafka.impl.consumer.KafkaFeedStreamer.createAndStartThreads(KafkaFeedStreamer.java:79)
~[stormjar.jar:na]
at
com.spr.messaging.kafka.impl.consumer.KafkaFeedStreamer.startKafkaStreamThreadsIfNecessary(KafkaFeedStreamer.java:64)
~[stormjar.jar:na]
at
com.spr.messaging.kafka.impl.consumer.KafkaFeedConsumerFactoryImpl.startStreamerIfNotRunning(KafkaFeedConsumerFactoryImpl.java:72)
[stormjar.jar:na]
at
com.spr.messaging.kafka.impl.consumer.KafkaFeedPullConsumerImpl.startStreamerIfNotRunning(KafkaFeedPullConsumerImpl.java:48)
[stormjar.jar:na]
at
com.spr.messaging.kafka.impl.KafkaFeedServiceImpl.getKafkaFeedPullConsumer(KafkaFeedServiceImpl.java:64)
[stormjar.jar:na]
at
com.spr.storm.topology.spout.AbstractSprKafkaSpout.nextTuple(AbstractSprKafkaSpout.java:135)
[stormjar.jar:na]
at
backtype.storm.daemon.executor$eval3848$fn__3849$fn__3864$fn__3893.invoke(executor.clj:562)
[na:0.9.1-incubating]
at backtype.storm.util$async_loop$fn__384.invoke(util.clj:433)
[na:0.9.1-incubating]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_55]
2014-12-07 14:56:00 k.c.ZookeeperConsumerConnector [INFO]
[lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d],
begin registering consumer
lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d
in ZK
2014-12-07 14:56:00 k.u.ZkUtils$ [INFO] conflict in
/consumers/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin/ids/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d
data:
{"version":1,"subscription":{"lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin":5},"pattern":"static","timestamp":"1417964160024"}
stored data:
{"version":1,"subscription":{"lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin":5},"pattern":"static","timestamp":"1417963753707"}
2014-12-07 14:56:00 k.u.ZkUtils$ [INFO] I wrote this conflicted ephemeral node
[{"version":1,"subscription":{"lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin":5},"pattern":"static","timestamp":"1417964160024"}]
at
/consumers/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin/ids/lst_plugin_com.spr.listening.plugin.impl.plugins.SemantriaEnrichmentPlugin_ip-10-0-8-222-1417963753598-b19de58d
a while back in a different session, hence I will backoff for this node to be
deleted by Zookeeper and retry
