Found the fix. I was stuck in this problem from last 3-4 days and it was
waiting just for joining the storm mailing list to be resolved. :)
I found something in supervisor.log this time, it was dumping <jar_UUID>
still hasn't started with the actual worker java command which is failing.
However, it was not showing any error. So, I copied the command from the
logs and directly run on console and it showed me the root cause. Somehow,
localhost was get appended to hdp.ambari (which was my host name) and due
to it was not able to find the server to run the command on. :(
On Mon, Aug 25, 2014 at 5:25 PM, Vikas Agarwal <vi...@infoobjects.com>
wrote:
> Hi,
>
> I have started to explore the Storm for distributed processing for our use
> case which we were earlier fulfilling by JMS based MQ system. Topology
> worked after some efforts. It has one spout (KafkaSpout from kafka-storm
> project) and 3 bolts. First bolt sets context for other two bolts which in
> turn do some processing on the tuples and persist the analyzed results in
> some DB (Mongo, Solr, HBase etc).
>
> Recently the topology stopped working. I am able to submit the topology
> and it does not throw any error in submitting the topology, however,
> nimbus.log or worker-6701.log files are not showing any progress and
> eventually topology does not consume any message. I don't have doubt on
> KafkaSpout because if it was the culprit, at least some initialization logs
> of spout and bolts should have been there in nimbus.log or worker-xxxx.log.
> Isn't it?
>
> Here is the snippet of nimbus.log after uploading the jar to cluster
>
> Uploading file from client to
> /hadoop/storm/nimbus/inbox/stormjar-31fe068b-337b-428f-8ae2-fe13c706b2ab.jar
> 2014-08-25 07:07:49 b.s.d.nimbus [INFO] Finished uploading file from
> client:
> /hadoop/storm/nimbus/inbox/stormjar-31fe068b-337b-428f-8ae2-fe13c706b2ab.jar
> 2014-08-25 07:07:49 b.s.d.nimbus [INFO] Received topology submission for
> aleads with conf {"topology.max.task.parallelism" nil,
> "topology.acker.executors" nil, "topology.kryo.register" nil,
> "topology.kryo.decorators" (), "topology.name" "aleads", "storm.id"
> "aleads-3-1408964869", "modelId" "ut", "topology.workers" 1,
> "topology.debug" true}
> 2014-08-25 07:07:50 b.s.d.nimbus [INFO] Activating aleads:
> aleads-3-1408964869
> 2014-08-25 07:07:50 b.s.s.EvenScheduler [INFO] Available slots:
> (["e56c2cc7-d35a-4355-9906-506618ff70c5" 6701]
> ["e56c2cc7-d35a-4355-9906-506618ff70c5" 6700])
> 2014-08-25 07:07:50 b.s.d.nimbus [INFO] Setting new assignment for
> topology id aleads-3-1408964869:
> #backtype.storm.daemon.common.Assignment{:master-code-dir
> "/hadoop/storm/nimbus/stormdist/aleads-3-1408964869", :node->host
> {"e56c2cc7-d35a-4355-9906-506618ff70c5" "hdp.ambari"}, :executor->node+port
> {[2 2] ["e56c2cc7-d35a-4355-9906-506618ff70c5" 6701], [3 3]
> ["e56c2cc7-d35a-4355-9906-506618ff70c5" 6701], [4 4]
> ["e56c2cc7-d35a-4355-9906-506618ff70c5" 6701], [5 5]
> ["e56c2cc7-d35a-4355-9906-506618ff70c5" 6701], [6 6]
> ["e56c2cc7-d35a-4355-9906-506618ff70c5" 6701], [7 7]
> ["e56c2cc7-d35a-4355-9906-506618ff70c5" 6701], [8 8]
> ["e56c2cc7-d35a-4355-9906-506618ff70c5" 6701], [9 9]
> ["e56c2cc7-d35a-4355-9906-506618ff70c5" 6701], [1 1]
> ["e56c2cc7-d35a-4355-9906-506618ff70c5" 6701]}, :executor->start-time-secs
> {[1 1] 1408964870, [9 9] 1408964870, [8 8] 1408964870, [7 7] 1408964870, [6
> 6] 1408964870, [5 5] 1408964870, [4 4] 1408964870, [3 3] 1408964870, [2 2]
> 1408964870}}
>
> Can anyone guess what I have done wrong and why Storm is not giving any
> error log anywhere.
>
> Storm version is 0.9.1.2.1.3.0-563 (Installed via HortonWorks)
> Kafka version is 2.10-0.8.1.1
> Storm-Kafka version 0.9.2-incubating
>
> --
> Regards,
> Vikas Agarwal
> 91 – 9928301411
>
> InfoObjects, Inc.
> Execution Matters
> http://www.infoobjects.com
> 2041 Mission College Boulevard, #280
> Santa Clara, CA 95054
> +1 (408) 988-2000 Work
> +1 (408) 716-2726 Fax
>
>
--
Regards,
Vikas Agarwal
91 – 9928301411
InfoObjects, Inc.
Execution Matters
http://www.infoobjects.com
2041 Mission College Boulevard, #280
Santa Clara, CA 95054
+1 (408) 988-2000 Work
+1 (408) 716-2726 Fax
