Re: Topology Restart due to Executor Not Alive
Hello! I have encountered the same issue in a case of out of memory in worker process. Try increase the memory of the wokers by setting nimbus.childopts property. Also, if you are creating short living object at higher rate use +UseG1GC . Since you are saying that you hold data in your memory, I'm suspecting (as I said) an OutOfMemeory error. Don't cover this just by increasing the heap size, but also I recommend to profile your worker and see if you have a memory leak. Hope that these help. Regards, Florin On Sat, Aug 23, 2014 at 2:24 AM, Andrey Yegorov andrey.yego...@gmail.com wrote: Have you figured out the rootcause/fix for this issue? I just hit it and would really appreciate some time-saving advise. -- Andrey Yegorov On Wed, Mar 12, 2014 at 10:31 AM, Josh Walton jwalton...@gmail.com wrote: Overnight last night, it appears my Storm Trident topology restarted itself. When I checked the Storm UI, it said the topology had been running for 24 hours, and showed no error or exceptions in any of the bolts. I check the nimbus log and see the following: 2014-03-12 10:55:06 b.s.d.nimbus [INFO] Executor MITAS3-74-1394565794:[34 34] not alive 2014-03-12 10:55:06 b.s.d.nimbus [INFO] Executor MITAS3-74-1394565794:[4 4] not alive 2014-03-12 10:55:06 b.s.d.nimbus [INFO] Executor MITAS3-74-1394565794:[40 40] not alive 2014-03-12 10:55:06 b.s.d.nimbus [INFO] Executor MITAS3-74-1394565794:[10 10] not alive 2014-03-12 10:55:06 b.s.d.nimbus [INFO] Executor MITAS3-74-1394565794:[16 16] not alive 2014-03-12 10:55:06 b.s.d.nimbus [INFO] Executor MITAS3-74-1394565794:[22 22] not alive 2014-03-12 10:55:06 b.s.d.nimbus [INFO] Executor MITAS3-74-1394565794:[28 28] not alive 2014-03-12 10:55:06 b.s.s.EvenScheduler [INFO] Available slots: ([5d105f66-1add-421b-8265-e7340a95928c 6700] [32ab1745-c260-4491-ae4d-92dcc5d14a62 6700]) 2014-03-12 10:55:06 b.s.d.nimbus [INFO] Reassigning MITAS3-74-1394565794 to 6 slots 2014-03-12 10:55:06 b.s.d.nimbus [INFO] Reassign executors: [[34 34] [4 4] [40 40] [10 10] [16 16] [22 22] [28 28]] It appears that an executor was alive, and must have timed out somehow since I didn't see any exceptions or stack traces in the logs. Is there a way to change the timeout? I see several timeout settings, but I'm not sure if any of those would help prevent this type of restart. I am using a custom TridentState which holds data in memory so we lost data as a result of this restart, and would like to prevent this from happening again. Thanks Josh
java.io.FileNotFoundException: File '.../stormconf.ser' does not exist
Hi, I've got the following exception: java.io.FileNotFoundException: File 'storm-local/supervisor/stormdist/SmokeTestTopology-1-1409647900/stormconf.ser' does not exist at org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:299) ~[commons-io-2.4.jar:2.4] at org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1763) ~[commons-io-2.4.jar:2.4] at backtype.storm.config$read_supervisor_storm_conf.invoke(config.clj:212) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.daemon.worker$worker_data.invoke(worker.clj:180) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.daemon.worker$fn__5940$exec_fn__1396__auto5941.invoke(worker.clj:356) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at clojure.lang.AFn.applyToHelper(AFn.java:185) [clojure-1.5.1.jar:na] at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.5.1.jar:na] at clojure.core$apply.invoke(core.clj:617) ~[clojure-1.5.1.jar:na] at backtype.storm.daemon.worker$fn__5940$mk_worker__5996.doInvoke(worker.clj:347) [storm-core-0.9.2-incubating.jar:0.9.2-incubating] at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.5.1.jar:na] at backtype.storm.daemon.worker$_main.invoke(worker.clj:454) [storm-core-0.9.2-incubating.jar:0.9.2-incubating] at clojure.lang.AFn.applyToHelper(AFn.java:172) [clojure-1.5.1.jar:na] at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.5.1.jar:na] at backtype.storm.daemon.worker.main(Unknown Source) [storm-core-0.9.2-incubating.jar:0.9.2-incubating] Anyone ideas? Thanks, Andras
Re: java.io.FileNotFoundException: File '.../stormconf.ser' does not exist
Hello! I also encountered the same issue. Perhaps someone will explain the reason for this error (maybe an inconsistent state of the supervisor, a killed topology that didn't succeed ???) Stop your supervisor. Delete the the storm-local folder. And restart the supervisor daemon. Check if is running either by looking in the storm/logs/supervisor.log or inthe storm UI if it s running. Hope that this help. Regards, Florin On Tue, Sep 2, 2014 at 12:12 PM, Andras Hatvani andras.hatv...@andrashatvani.com wrote: Hi, I've got the following exception: java.io.FileNotFoundException: File 'storm-local/supervisor/stormdist/SmokeTestTopology-1-1409647900/stormconf.ser' does not exist at org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:299) ~[commons-io-2.4.jar:2.4] at org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1763) ~[commons-io-2.4.jar:2.4] at backtype.storm.config$read_supervisor_storm_conf.invoke(config.clj:212) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.daemon.worker$worker_data.invoke(worker.clj:180) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.daemon.worker$fn__5940$exec_fn__1396__auto5941.invoke(worker.clj:356) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at clojure.lang.AFn.applyToHelper(AFn.java:185) [clojure-1.5.1.jar:na] at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.5.1.jar:na] at clojure.core$apply.invoke(core.clj:617) ~[clojure-1.5.1.jar:na] at backtype.storm.daemon.worker$fn__5940$mk_worker__5996.doInvoke(worker.clj:347) [storm-core-0.9.2-incubating.jar:0.9.2-incubating] at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.5.1.jar:na] at backtype.storm.daemon.worker$_main.invoke(worker.clj:454) [storm-core-0.9.2-incubating.jar:0.9.2-incubating] at clojure.lang.AFn.applyToHelper(AFn.java:172) [clojure-1.5.1.jar:na] at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.5.1.jar:na] at backtype.storm.daemon.worker.main(Unknown Source) [storm-core-0.9.2-incubating.jar:0.9.2-incubating] Anyone ideas? Thanks, Andras
RE: REMOTE MODE STORM DEV
Is supervisor running or dead ? . Spouts and bolts which run fine in local mode fails to do same in remote mode. Pls help us here. less supervisor.log m-0.9.2-incubating/lib/commons-io-2.4.jar:/root/apache-storm-0.9.2-incubating/lib/httpclient-4.3.3.jar:/root/apache-storm-0.9.2-incubating/lib/jline-2.11.jar:/root/apache-storm-0.9.2-incubating/lib/tools.logging-0.2.3.jar:/root/apache-storm-0.9.2-incubating/lib/logback-classic-1.0.6.jar:/root/apache-storm-0.9.2-incubating/lib/commons-logging-1.1.3.jar:/root/apache-storm-0.9.2-incubating/lib/ring-core-1.1.5.jar:/root/apache-storm-0.9.2-incubating/lib/ring-devel-0.3.11.jar:/root/apache-storm-0.9.2-incubating/lib/curator-client-2.4.0.jar:/root/apache-storm-0.9.2-incubating/lib/clj-stacktrace-0.2.4.jar:/root/apache-storm-0.9.2-incubating/lib/clj-time-0.4.1.jar:/root/apache-storm-0.9.2-incubating/lib/commons-lang-2.5.jar:/root/apache-storm-0.9.2-incubating/lib/zookeeper-3.4.5.jar:/root/apache-storm-0.9.2-incubating/lib/compojure-1.1.3.jar:/root/apache-storm-0.9.2-incubating/lib/joda-time-2.0.jar:/root/apache-storm-0.9.2-incubating/lib/chill-java-0.3.5.jar:/root/apache-storm-0.9.2-incubating/lib/clout-1.0.1.jar:/root/apache-storm-0.9.2-incubating/lib/kryo-2.21.jar:/root/apache-storm-0.9.2-incubating/lib/snakeyaml-1.11.jar:/root/apache-storm-0.9.2-incubating/lib/minlog-1.2.jar:/root/apache-storm-0.9.2-incubating/lib/storm-core-0.9.2-incubating.jar:/root/apache-storm-0.9.2-incubating/lib/jgrapht-core-0.9.0.jar:/root/apache-storm-0.9.2-incubating/lib/slf4j-api-1.6.5.jar:/root/apache-storm-0.9.2-incubating/lib/hiccup-0.3.6.jar:/root/apache-storm-0.9.2-incubating/lib/netty-3.6.3.Final.jar:/root/apache-storm-0.9.2-incubating/lib/curator-framework-2.4.0.jar:/root/apache-storm-0.9.2-incubating/lib/guava-13.0.jar:/root/apache-storm-0.9.2-incubating/lib/log4j-over-slf4j-1.6.6.jar:/root/apache-storm-0.9.2-incubating/lib/commons-fileupload-1.2.1.jar:/root/apache-storm-0.9.2-incubating/lib/servlet-api-2.5.jar:/root/apache-storm-0.9.2-incubating/lib/reflectasm-1.07-shaded.jar:/root/apache-storm-0.9.2-incubating/lib/jetty-util-6.1.26.jar:/root/apache-storm-0.9.2-incubating/lib/objenesis-1.2.jar:/root/apache-storm-0.9.2-incubating/lib/tools.cli-0.2.4.jar:/root/apache-storm-0.9.2-incubating/lib/ring-jetty-adapter-0.3.11.jar:/root/apache-storm-0.9.2-incubating/lib/commons-codec-1.6.jar:/root/apache-storm-0.9.2-incubating/lib/clojure-1.5.1.jar:/root/apache-storm-0.9.2-incubating/lib/netty-3.2.2.Final.jar:/root/apache-storm-0.9.2-incubating/lib/math.numeric-tower-0.0.1.jar:/root/apache-storm-0.9.2-incubating/lib/carbonite-1.4.0.jar:/root/apache-storm-0.9.2-incubating/lib/disruptor-2.10.1.jar:/root/apache-storm-0.9.2-incubating/lib/commons-exec-1.1.jar:/root/apache-storm-0.9.2-incubating/lib/tools.macro-0.1.0.jar:/root/apache-storm-0.9.2-incubating/lib/jetty-6.1.26.jar:/root/apache-storm-0.9.2-incubating/lib/httpcore-4.3.2.jar:/root/apache-storm-0.9.2-incubating/lib/servlet-api-2.5-20081211.jar:/root/apache-storm-0.9.2-incubating/lib/ring-servlet-0.3.11.jar:/root/apache-storm-0.9.2-incubating/lib/logback-core-1.0.6.jar:/root/apache-storm-0.9.2-incubating/conf:storm-local/supervisor/stormdist/PRIMETopology-6-1409635517/stormjar.jar' 'backtype.storm.daemon.worker' 'PRIMETopology-6-1409635517' '0574446a-f73d-42b0-bcc7-e6dd449cb75a' '6702' 'ddf2747a-276a-4724-ba08-4f5d46b0b5ec' 2014-09-02 05:34:01 b.s.d.supervisor [INFO] 5a5f5d59-d67d-4301-b899-36e5432212e1 still hasn't started 2014-09-02 05:34:02 b.s.d.supervisor [INFO] 5a5f5d59-d67d-4301-b899-36e5432212e1 still hasn't started 2014-09-02 05:34:02 b.s.d.supervisor [INFO] 5a5f5d59-d67d-4301-b899-36e5432212e1 still hasn't started 2014-09-02 05:34:03 b.s.d.supervisor [INFO] 5a5f5d59-d67d-4301-b899-36e5432212e1 still hasn't started 2014-09-02 05:34:03 b.s.d.supervisor [INFO] 5a5f5d59-d67d-4301-b899-36e5432212e1 still hasn't started 2014-09-02 05:34:04 b.s.d.supervisor [INFO] 5a5f5d59-d67d-4301-b899-36e5432212e1 still hasn't started 2014-09-02 05:34:04 b.s.d.supervisor [INFO] 5a5f5d59-d67d-4301-b899-36e5432212e1 still hasn't started 2014-09-02 05:34:05 b.s.d.supervisor [INFO] 5a5f5d59-d67d-4301-b899-36e5432212e1 still hasn't started 2014-09-02 05:34:05 b.s.d.supervisor [INFO] 5a5f5d59-d67d-4301-b899-36e5432212e1 still hasn't started 2014-09-02 05:34:06 b.s.d.supervisor [INFO] 5a5f5d59-d67d-4301-b899-36e5432212e1 still hasn't started 2014-09-02 05:34:06 b.s.d.supervisor [INFO] 5a5f5d59-d67d-4301-b899-36e5432212e1 still hasn't started 2014-09-02 05:34:07 b.s.d.supervisor [INFO] 5a5f5d59-d67d-4301-b899-36e5432212e1 still hasn't started 2014-09-02 05:34:07 b.s.d.supervisor [INFO] 5a5f5d59-d67d-4301-b899-36e5432212e1 still hasn't started 2014-09-02 05:34:08 b.s.d.supervisor [INFO] 5a5f5d59-d67d-4301-b899-36e5432212e1 still hasn't started 2014-09-02 05:34:08 b.s.d.supervisor [INFO] 5a5f5d59-d67d-4301-b899-36e5432212e1 still hasn't started 2014-09-02 05:34:09 b.s.d.supervisor [INFO]
Re: REMOTE MODE STORM DEV
Pavan, Have you tried running the command for launching worker as logged in supervisor.log? You need to run the command with storm user and may need to kill existing one if running on same port. On Tue, Sep 2, 2014 at 3:02 PM, Pavan Jakati G pava...@microland.com wrote: Is supervisor running or dead ? . Spouts and bolts which run fine in local mode fails to do same in remote mode. Pls help us here. less supervisor.log m-0.9.2-incubating/lib/commons-io-2.4.jar:/root/apache-storm-0.9.2-incubating/lib/httpclient-4.3.3.jar:/root/apache-storm-0.9.2-incubating/lib/jline-2.11.jar:/root/apache-storm-0.9.2-incubating/lib/tools.logging-0.2.3.jar:/root/apache-storm-0.9.2-incubating/lib/logback-classic-1.0.6.jar:/root/apache-storm-0.9.2-incubating/lib/commons-logging-1.1.3.jar:/root/apache-storm-0.9.2-incubating/lib/ring-core-1.1.5.jar:/root/apache-storm-0.9.2-incubating/lib/ring-devel-0.3.11.jar:/root/apache-storm-0.9.2-incubating/lib/curator-client-2.4.0.jar:/root/apache-storm-0.9.2-incubating/lib/clj-stacktrace-0.2.4.jar:/root/apache-storm-0.9.2-incubating/lib/clj-time-0.4.1.jar:/root/apache-storm-0.9.2-incubating/lib/commons-lang-2.5.jar:/root/apache-storm-0.9.2-incubating/lib/zookeeper-3.4.5.jar:/root/apache-storm-0.9.2-incubating/lib/compojure-1.1.3.jar:/root/apache-storm-0.9.2-incubating/lib/joda-time-2.0.jar:/root/apache-storm-0.9.2-incubating/lib/chill-java-0.3.5.jar:/root/apache-storm-0.9.2-incubating/lib/clout-1.0.1.jar:/root/apache-storm-0.9.2-incubating/lib/kryo-2.21.jar:/root/apache-storm-0.9.2-incubating/lib/snakeyaml-1.11.jar:/root/apache-storm-0.9.2-incubating/lib/minlog-1.2.jar:/root/apache-storm-0.9.2-incubating/lib/storm-core-0.9.2-incubating.jar:/root/apache-storm-0.9.2-incubating/lib/jgrapht-core-0.9.0.jar:/root/apache-storm-0.9.2-incubating/lib/slf4j-api-1.6.5.jar:/root/apache-storm-0.9.2-incubating/lib/hiccup-0.3.6.jar:/root/apache-storm-0.9.2-incubating/lib/netty-3.6.3.Final.jar:/root/apache-storm-0.9.2-incubating/lib/curator-framework-2.4.0.jar:/root/apache-storm-0.9.2-incubating/lib/guava-13.0.jar:/root/apache-storm-0.9.2-incubating/lib/log4j-over-slf4j-1.6.6.jar:/root/apache-storm-0.9.2-incubating/lib/commons-fileupload-1.2.1.jar:/root/apache-storm-0.9.2-incubating/lib/servlet-api-2.5.jar:/root/apache-storm-0.9.2-incubating/lib/reflectasm-1.07-shaded.jar:/root/apache-storm-0.9.2-incubating/lib/jetty-util-6.1.26.jar:/root/apache-storm-0.9.2-incubating/lib/objenesis-1.2.jar:/root/apache-storm-0.9.2-incubating/lib/tools.cli-0.2.4.jar:/root/apache-storm-0.9.2-incubating/lib/ring-jetty-adapter-0.3.11.jar:/root/apache-storm-0.9.2-incubating/lib/commons-codec-1.6.jar:/root/apache-storm-0.9.2-incubating/lib/clojure-1.5.1.jar:/root/apache-storm-0.9.2-incubating/lib/netty-3.2.2.Final.jar:/root/apache-storm-0.9.2-incubating/lib/math.numeric-tower-0.0.1.jar:/root/apache-storm-0.9.2-incubating/lib/carbonite-1.4.0.jar:/root/apache-storm-0.9.2-incubating/lib/disruptor-2.10.1.jar:/root/apache-storm-0.9.2-incubating/lib/commons-exec-1.1.jar:/root/apache-storm-0.9.2-incubating/lib/tools.macro-0.1.0.jar:/root/apache-storm-0.9.2-incubating/lib/jetty-6.1.26.jar:/root/apache-storm-0.9.2-incubating/lib/httpcore-4.3.2.jar:/root/apache-storm-0.9.2-incubating/lib/servlet-api-2.5-20081211.jar:/root/apache-storm-0.9.2-incubating/lib/ring-servlet-0.3.11.jar:/root/apache-storm-0.9.2-incubating/lib/logback-core-1.0.6.jar:/root/apache-storm-0.9.2-incubating/conf:storm-local/supervisor/stormdist/PRIMETopology-6-1409635517/stormjar.jar' 'backtype.storm.daemon.worker' 'PRIMETopology-6-1409635517' '0574446a-f73d-42b0-bcc7-e6dd449cb75a' '6702' 'ddf2747a-276a-4724-ba08-4f5d46b0b5ec' 2014-09-02 05:34:01 b.s.d.supervisor [INFO] 5a5f5d59-d67d-4301-b899-36e5432212e1 still hasn't started 2014-09-02 05:34:02 b.s.d.supervisor [INFO] 5a5f5d59-d67d-4301-b899-36e5432212e1 still hasn't started 2014-09-02 05:34:02 b.s.d.supervisor [INFO] 5a5f5d59-d67d-4301-b899-36e5432212e1 still hasn't started 2014-09-02 05:34:03 b.s.d.supervisor [INFO] 5a5f5d59-d67d-4301-b899-36e5432212e1 still hasn't started 2014-09-02 05:34:03 b.s.d.supervisor [INFO] 5a5f5d59-d67d-4301-b899-36e5432212e1 still hasn't started 2014-09-02 05:34:04 b.s.d.supervisor [INFO] 5a5f5d59-d67d-4301-b899-36e5432212e1 still hasn't started 2014-09-02 05:34:04 b.s.d.supervisor [INFO] 5a5f5d59-d67d-4301-b899-36e5432212e1 still hasn't started 2014-09-02 05:34:05 b.s.d.supervisor [INFO] 5a5f5d59-d67d-4301-b899-36e5432212e1 still hasn't started 2014-09-02 05:34:05 b.s.d.supervisor [INFO] 5a5f5d59-d67d-4301-b899-36e5432212e1 still hasn't started 2014-09-02 05:34:06 b.s.d.supervisor [INFO] 5a5f5d59-d67d-4301-b899-36e5432212e1 still hasn't started 2014-09-02 05:34:06 b.s.d.supervisor [INFO] 5a5f5d59-d67d-4301-b899-36e5432212e1 still hasn't started 2014-09-02 05:34:07 b.s.d.supervisor [INFO] 5a5f5d59-d67d-4301-b899-36e5432212e1 still hasn't started 2014-09-02 05:34:07 b.s.d.supervisor [INFO] 5a5f5d59-d67d-4301-b899-36e5432212e1 still
Re: Topology Restart due to Executor Not Alive
Hello! The nimbus.childopts is to set up the heap for the nimbus master node. You can set up this either. For the worker use worker.childopts. Please respond to this mail if it help myou or how did you solve it. Regards, Florin On Tue, Sep 2, 2014 at 11:54 AM, Spico Florin spicoflo...@gmail.com wrote: Hello! I have encountered the same issue in a case of out of memory in worker process. Try increase the memory of the wokers by setting nimbus.childopts property. Also, if you are creating short living object at higher rate use +UseG1GC . Since you are saying that you hold data in your memory, I'm suspecting (as I said) an OutOfMemeory error. Don't cover this just by increasing the heap size, but also I recommend to profile your worker and see if you have a memory leak. Hope that these help. Regards, Florin On Sat, Aug 23, 2014 at 2:24 AM, Andrey Yegorov andrey.yego...@gmail.com wrote: Have you figured out the rootcause/fix for this issue? I just hit it and would really appreciate some time-saving advise. -- Andrey Yegorov On Wed, Mar 12, 2014 at 10:31 AM, Josh Walton jwalton...@gmail.com wrote: Overnight last night, it appears my Storm Trident topology restarted itself. When I checked the Storm UI, it said the topology had been running for 24 hours, and showed no error or exceptions in any of the bolts. I check the nimbus log and see the following: 2014-03-12 10:55:06 b.s.d.nimbus [INFO] Executor MITAS3-74-1394565794:[34 34] not alive 2014-03-12 10:55:06 b.s.d.nimbus [INFO] Executor MITAS3-74-1394565794:[4 4] not alive 2014-03-12 10:55:06 b.s.d.nimbus [INFO] Executor MITAS3-74-1394565794:[40 40] not alive 2014-03-12 10:55:06 b.s.d.nimbus [INFO] Executor MITAS3-74-1394565794:[10 10] not alive 2014-03-12 10:55:06 b.s.d.nimbus [INFO] Executor MITAS3-74-1394565794:[16 16] not alive 2014-03-12 10:55:06 b.s.d.nimbus [INFO] Executor MITAS3-74-1394565794:[22 22] not alive 2014-03-12 10:55:06 b.s.d.nimbus [INFO] Executor MITAS3-74-1394565794:[28 28] not alive 2014-03-12 10:55:06 b.s.s.EvenScheduler [INFO] Available slots: ([5d105f66-1add-421b-8265-e7340a95928c 6700] [32ab1745-c260-4491-ae4d-92dcc5d14a62 6700]) 2014-03-12 10:55:06 b.s.d.nimbus [INFO] Reassigning MITAS3-74-1394565794 to 6 slots 2014-03-12 10:55:06 b.s.d.nimbus [INFO] Reassign executors: [[34 34] [4 4] [40 40] [10 10] [16 16] [22 22] [28 28]] It appears that an executor was alive, and must have timed out somehow since I didn't see any exceptions or stack traces in the logs. Is there a way to change the timeout? I see several timeout settings, but I'm not sure if any of those would help prevent this type of restart. I am using a custom TridentState which holds data in memory so we lost data as a result of this restart, and would like to prevent this from happening again. Thanks Josh
RE: REMOTE MODE STORM DEV
sudo -u storm '/usr/java/jdk1.7.0_65/bin/java' '-server' '-Xmx768m' '-Djava.library.path=storm-local/sup ervisor/stormdist/PaVan-10-1409648607/resources/Linux-amd64:storm-local/supervisor/stormdist/PaVan-10-1409648607/resources:/usr/local/lib:/opt/local/lib:/usr/lib' '-Dlogfile.name=worker-6701.log' '-Dstorm.home=/root/apache-storm-0.9.2-incubating' '-Dlogback.configurationFile=/root/apache-storm-0.9.2-incubating/logback/cluster.xml' '-Dstorm.id=PaVan-10-1409648607' '-Dworker.id=b89ee630-7bd6-4976-8050-c43644c028ed' '-Dworker.port=6701' '-cp' '/root/apache-storm-0.9.2-incubating/lib/json-simple-1.1.jar:/root/apache-storm-0.9.2-incubating/lib/core.incubator-0.1.0.jar:/root/apache-storm-0.9.2-incubating/lib/asm-4.0.jar:/root/apache-storm-0.9.2-incubating/lib/commons-io-2.4.jar:/root/apache-storm-0.9.2-incubating/lib/httpclient-4.3.3.jar:/root/apache-storm-0.9.2-incubating/lib/jline-2.11.jar:/root/apache-storm-0.9.2-incubating/lib/tools.logging-0.2.3.jar:/root/apache-storm-0.9.2-incubating/lib/logback-classic-1.0.6.jar:/root/apache-storm-0.9.2-incubating/lib/commons-logging-1.1.3.jar:/root/apache-storm-0.9.2-incubating/lib/ring-core-1.1.5.jar:/root/apache-storm-0.9.2-incubating/lib/ring-devel-0.3.11.jar:/root/apache-storm-0.9.2-incubating/lib/curator-client-2.4.0.jar:/root/apache-storm-0.9.2-incubating/lib/clj-stacktrace-0.2.4.jar:/root/apache-storm-0.9.2-incubating/lib/clj-time-0.4.1.jar:/root/apache-storm-0.9.2-incubating/lib/commons-lang-2.5.jar:/root/apache-storm-0.9.2-incubating/lib/zookeeper-3.4.5.jar:/root/apache-storm-0.9.2-incubating/lib/compojure-1.1.3.jar:/root/apache-storm-0.9.2-incubating/lib/joda-time-2.0.jar:/root/apache-storm-0.9.2-incubating/lib/chill-java-0.3.5.jar:/root/apache-storm-0.9.2-incubating/lib/clout-1.0.1.jar:/root/apache-storm-0.9.2-incubating/lib/kryo-2.21.jar:/root/apache-storm-0.9.2-incubating/lib/snakeyaml-1.11.jar:/root/apache-storm-0.9.2-incubating/lib/minlog-1.2.jar:/root/apache-storm-0.9.2-incubating/lib/storm-core-0.9.2-incubating.jar:/root/apache-storm-0.9.2-incubating/lib/jgrapht-core-0.9.0.jar:/root/apache-storm-0.9.2-incubating/lib/slf4j-api-1.6.5.jar:/root/apache-storm-0.9.2-incubating/lib/hiccup-0.3.6.jar:/root/apache-storm-0.9.2-incubating/lib/netty-3.6.3.Final.jar:/root/apache-storm-0.9.2-incubating/lib/curator-framework-2.4.0.jar:/root/apache-storm-0.9.2-incubating/lib/guava-13.0.jar:/root/apache-storm-0.9.2-incubating/lib/log4j-over-slf4j-1.6.6.jar:/root/apache-storm-0.9.2-incubating/lib/commons-fileupload-1.2.1.jar:/root/apache-storm-0.9.2-incubating/lib/servlet-api-2.5.jar:/root/apache-storm-0.9.2-incubating/lib/reflectasm-1.07-shaded.jar:/root/apache-storm-0.9.2-incubating/lib/jetty-util-6.1.26.jar:/root/apache-storm-0.9.2-incubating/lib/objenesis-1.2.jar:/root/apache-storm-0.9.2-incubating/lib/tools.cli-0.2.4.jar:/root/apache-storm-0.9.2-incubating/lib/ring-jetty-adapter-0.3.11.jar:/root/apache-storm-0.9.2-incubating/lib/commons-codec-1.6.jar:/root/apache-storm-0.9.2-incubating/lib/clojure-1.5.1.jar:/root/apache-storm-0.9.2-incubating/lib/netty-3.2.2.Final.jar:/root/apache-storm-0.9.2-incubating/lib/math.numeric-tower-0.0.1.jar:/root/apache-storm-0.9.2-incubating/lib/carbonite-1.4.0.jar:/root/apache-storm-0.9.2-incubating/lib/disruptor-2.10.1.jar:/root/apache-storm-0.9.2-incubating/lib/commons-exec-1.1.jar:/root/apache-storm-0.9.2-incubating/lib/tools.macro-0.1.0.jar:/root/apache-storm-0.9.2-incubating/lib/jetty-6.1.26.jar:/root/apache-storm-0.9.2-incubating/lib/httpcore-4.3.2.jar:/root/apache-storm-0.9.2-incubating/lib/servlet-api-2.5-20081211.jar:/root/apache-storm-0.9.2-incubating/lib/ring-servlet-0.3.11.jar:/root/apache-storm-0.9.2-incubating/lib/logback-core-1.0.6.jar:/root/apache-storm-0.9.2-incubating/conf:storm-local/supervisor/stormdist/PaVan-10-1409648607/stormjar.jar' 'backtype.storm.daemon.worker' 'PaVan-10-1409648607' '0574446a-f73d-42b0-bcc7-e6dd449cb75a' '6701' 'b89ee630-7bd6-4976-8050-c43644c028ed' Error: Could not find or load main class backtype.storm.daemon.worker Regards, PaVan… From: Vikas Agarwal [mailto:vi...@infoobjects.com] Sent: 02 September 2014 15:05 To: user@storm.incubator.apache.org Subject: Re: REMOTE MODE STORM DEV Pavan, Have you tried running the command for launching worker as logged in supervisor.log? You need to run the command with storm user and may need to kill existing one if running on same port. On Tue, Sep 2, 2014 at 3:02 PM, Pavan Jakati G pava...@microland.com wrote: Is supervisor running or dead ? . Spouts and bolts which run fine in local mode fails to do same in remote mode. Pls help us here. less supervisor.log
KafkaSpout: java.lang.IllegalArgumentException: message does not exist
Hi,
I've got an exception when trying to use a topology with a single Kafka spout
and a single Kafka bolt.
Code:
public class SmokeTestTopology {
public static final String INPUT_TOPIC = inputTopic;
public static final String ZK_ROOT_PATH = /smokeTest;
public static final String ZK_INPUT_ID = inputId;
public static final String OUTPUT_TOPIC = outputTopic;
public static StormTopology buildTopology() {
KafkaSpout kafkaSpout = prepareKafkaSpout();
KafkaBolt kafkaBolt = prepareKafkaBolt();
TopologyBuilder topologyBuilder = new TopologyBuilder();
topologyBuilder.setSpout(spout, kafkaSpout);
topologyBuilder.setBolt(bolt, kafkaBolt).shuffleGrouping(spout);
return topologyBuilder.createTopology();
}
private static KafkaBolt prepareKafkaBolt() {
return new KafkaBolt();
}
private static KafkaSpout prepareKafkaSpout() {
SpoutConfig spoutConfig = new SpoutConfig(
new ZkHosts(192.168.59.103:2181),
INPUT_TOPIC,
ZK_ROOT_PATH,
ZK_INPUT_ID);
spoutConfig.forceFromStart = true;
return new KafkaSpout(spoutConfig);
}
}
Exception:
2014-09-02 10:29:59 b.s.d.executor [ERROR]
java.lang.RuntimeException: java.lang.IllegalArgumentException: message does
not exist
at
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.daemon.executor$fn__5641$fn__5653$fn__5700.invoke(executor.clj:746)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_20]
Caused by: java.lang.IllegalArgumentException: message does not exist
at backtype.storm.tuple.Fields.fieldIndex(Fields.java:78)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.tuple.TupleImpl.fieldIndex(TupleImpl.java:100)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.tuple.TupleImpl.getValueByField(TupleImpl.java:149)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at storm.kafka.bolt.KafkaBolt.execute(KafkaBolt.java:75) ~[stormjar.jar:na]
at
backtype.storm.daemon.executor$fn__5641$tuple_action_fn__5643.invoke(executor.clj:631)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.daemon.executor$mk_task_receiver$fn__5564.invoke(executor.clj:399)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.disruptor$clojure_handler$reify__745.onEvent(disruptor.clj:58)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
… 6 common frames omitted
Anyone with similar issue?
Thanks,
Andras
Re: Supervisor always down 3s after execution
Hi everyone, I followed your instructions for installing a zookeeper server, i downloaded it on the website, extract the tar file somewhere in a machine on my cluster, i made those modifications in my zoo.cfg : # The number of milliseconds of each tick tickTime=2000 # The number of ticks that the initial # synchronization phase can take initLimit=10 # The number of ticks that can pass between # sending a request and getting an acknowledgement syncLimit=5 # the directory where the snapshot is stored. # do not use /tmp for storage, /tmp here is just # example sakes. dataDir=/home/bsoulas/zookeeper/zookeeper-3.4.6/data/ # the port at which the clients will connect clientPort=2181 # the maximum number of client connections. # increase this if you need to handle more clients #maxClientCnxns=60 # # Be sure to read the maintenance section of the # administrator guide before turning on autopurge. # # http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance # # The number of snapshots to retain in dataDir #autopurge.snapRetainCount=3 # Purge task interval in hours # Set to 0 to disable auto purge feature #autopurge.purgeInterval=1 In the log4j.properties, i uncommented the line for the log file : # Example with rolling log file log4j.rootLogger=DEBUG, CONSOLE, ROLLINGFILE Then i went to my storm.yaml (located here in my case, because i took the source version) : /home/bsoulas/incubator-storm-master/storm-dist/binary/target/apache-storm-0.9.3-ben/apache-storm-0.9.3-ben/conf This file contain this configuration : ### These MUST be filled in for a storm configuration storm.zookeeper.servers: - paradent-4 # - paradent-47 # - paradent-48 # nimbus.host: paradent-4 # # # # These may optionally be filled in: # ## List of custom serializations # topology.kryo.register: # - org.mycompany.MyType # - org.mycompany.MyType2: org.mycompany.MyType2Serializer # ## List of custom kryo decorators # topology.kryo.decorators: # - org.mycompany.MyDecorator # ## Locations of the drpc servers # drpc.servers: # - server1 # - server2 ## Metrics Consumers # topology.metrics.consumer.register: # - class: backtype.storm.metric.LoggingMetricsConsumer # parallelism.hint: 1 # - class: org.mycompany.MyMetricsConsumer # parallelism.hint: 1 # argument: # - endpoint: metrics-collector.mycompany.org dev.zookeeper.path: paradent-4.rennes.grid5000.fr: ~/home/bsoulas/zookeeper/zookeeper-3.4.6/ storm.zookeeper.port: 2181 To launch storm on the cluster, i launch it thanks to *storm nimbus *(on a machine named paradent-4), then my zookeeper Server *sh zkServer.sh start* (on paradent-4 again)(which create a *zookeeper_server.pid* where the pid of the zookeeper is written, i know it's obvious ..._ ). After i launch my *storm ui* for having a visual of my storm app (on paradent-4). Until now, everything work fine. Now, the logical way implies i launch my supervisor, on a different machine (here *paradent-39*) thanks to *storm supervisor*, it is launched but once again, 3 or 4 seconds after it's down. So i watched the supervisor.log located : /home/bsoulas/incubator-storm-master/storm-dist/binary/target/apache-storm-0.9.3-ben/apache-storm-0.9.3-ben/logs And here appear a tricky error : 2014-09-02 09:31:37 o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting 2014-09-02 09:31:37 o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=paradent-4:2181 sessionTimeout=2 watcher=org.apache.curator.ConnectionState@220df4c8 2014-09-02 09:31:37 o.a.z.ClientCnxn [INFO] Opening socket connection to server paradent-4.rennes.grid5000.fr/172.16.97.4:2181. Will not attempt to authenticate using SASL (unknown error) 2014-09-02 09:31:37 o.a.z.ClientCnxn [INFO] Socket connection established to paradent-4.rennes.grid5000.fr/172.16.97.4:2181, initiating session 2014-09-02 09:31:37 o.a.z.ClientCnxn [INFO] Session establishment complete on server paradent-4.rennes.grid5000.fr/172.16.97.4:2181, sessionid = 0x14835a48ca90004, negotiated timeout = 2 2014-09-02 09:31:37 o.a.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED 2014-09-02 09:31:37 o.a.c.f.s.ConnectionStateManager [WARN] There are no ConnectionStateListeners registered. 2014-09-02 09:31:37 b.s.zookeeper [INFO] Zookeeper state update: :connected:none 2014-09-02 09:31:38 o.a.z.ZooKeeper [INFO] Session: 0x14835a48ca90004 closed 2014-09-02 09:31:38 o.a.z.ClientCnxn [INFO] EventThread shut down 2014-09-02 09:31:38 o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting 2014-09-02 09:31:38 o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=paradent-4:2181/storm sessionTimeout=2 watcher=org.apache.curator.ConnectionState@c6d625b 2014-09-02 09:31:38 o.a.z.ClientCnxn [INFO] Opening socket connection to server paradent-4.rennes.grid5000.fr/172.16.97.4:2181. Will not attempt to authenticate using SASL (unknown error)
RE: REMOTE MODE STORM DEV
Can anybody help us get rid of below error : sudo -u storm /usr/java/jdk1.7.0_65/bin/java -server -Xmx768m -Djava.library.path=storm-local/supervisor/stormdist/PaVan-14-1409661199/resources/Linux-amd64:storm-local/supervisor/stormdist/PaVan-14-1409661199/resources:/usr/local/lib:/opt/local/lib:/usr/lib -Dlogfile.name=worker-6703.log -Dstorm.home=/root/apache-storm-0.9.2-incubating -Dlogback.configurationFile=/root/apache-storm-0.9.2-incubating/logback/cluster.xml -Dstorm.id=PaVan-14-1409661199 -Dworker.id=156a8af9-fa3b-4772-b91c-787490fe0b34 -Dworker.port=6703 -cp /root/apache-storm-0.9.2-incubating/lib/json-simple-1.1.jar:/root/apache-storm-0.9.2-incubating/lib/core.incubator-0.1.0.jar:/root/apache-storm-0.9.2-incubating/lib/asm-4.0.jar:/root/apache-storm-0.9.2-incubating/lib/commons-io-2.4.jar:/root/apache-storm-0.9.2-incubating/lib/httpclient-4.3.3.jar:/root/apache-storm-0.9.2-incubating/lib/jline-2.11.jar:/root/apache-storm-0.9.2-incubating/lib/tools.logging-0.2.3.jar:/root/apache-storm-0.9.2-incubating/lib/logback-classic-1.0.6.jar:/root/apache-storm-0.9.2-incubating/lib/commons-logging-1.1.3.jar:/root/apache-storm-0.9.2-incubating/lib/ring-core-1.1.5.jar:/root/apache-storm-0.9.2-incubating/lib/ring-devel-0.3.11.jar:/root/apache-storm-0.9.2-incubating/lib/curator-client-2.4.0.jar:/root/apache-storm-0.9.2-incubating/lib/clj-stacktrace-0.2.4.jar:/root/apache-storm-0.9.2-incubating/lib/clj-time-0.4.1.jar:/root/apache-storm-0.9.2-incubating/lib/commons-lang-2.5.jar:/root/apache-storm-0.9.2-incubating/lib/zookeeper-3.4.5.jar:/root/apache-storm-0.9.2-incubating/lib/compojure-1.1.3.jar:/root/apache-storm-0.9.2-incubating/lib/joda-time-2.0.jar:/root/apache-storm-0.9.2-incubating/lib/chill-java-0.3.5.jar:/root/apache-storm-0.9.2-incubating/lib/clout-1.0.1.jar:/root/apache-storm-0.9.2-incubating/lib/kryo-2.21.jar:/root/apache-storm-0.9.2-incubating/lib/snakeyaml-1.11.jar:/root/apache-storm-0.9.2-incubating/lib/minlog-1.2.jar:/root/apache-storm-0.9.2-incubating/lib/storm-core-0.9.2-incubating.jar:/root/apache-storm-0.9.2-incubating/lib/jgrapht-core-0.9.0.jar:/root/apache-storm-0.9.2-incubating/lib/slf4j-api-1.6.5.jar:/root/apache-storm-0.9.2-incubating/lib/hiccup-0.3.6.jar:/root/apache-storm-0.9.2-incubating/lib/netty-3.6.3.Final.jar:/root/apache-storm-0.9.2-incubating/lib/curator-framework-2.4.0.jar:/root/apache-storm-0.9.2-incubating/lib/guava-13.0.jar:/root/apache-storm-0.9.2-incubating/lib/log4j-over-slf4j-1.6.6.jar:/root/apache-storm-0.9.2-incubating/lib/commons-fileupload-1.2.1.jar:/root/apache-storm-0.9.2-incubating/lib/servlet-api-2.5.jar:/root/apache-storm-0.9.2-incubating/lib/reflectasm-1.07-shaded.jar:/root/apache-storm-0.9.2-incubating/lib/jetty-util-6.1.26.jar:/root/apache-storm-0.9.2-incubating/lib/objenesis-1.2.jar:/root/apache-storm-0.9.2-incubating/lib/tools.cli-0.2.4.jar:/root/apache-storm-0.9.2-incubating/lib/ring-jetty-adapter-0.3.11.jar:/root/apache-storm-0.9.2-incubating/lib/commons-codec-1.6.jar:/root/apache-storm-0.9.2-incubating/lib/clojure-1.5.1.jar:/root/apache-storm-0.9.2-incubating/lib/netty-3.2.2.Final.jar:/root/apache-storm-0.9.2-incubating/lib/math.numeric-tower-0.0.1.jar:/root/apache-storm-0.9.2-incubating/lib/carbonite-1.4.0.jar:/root/apache-storm-0.9.2-incubating/lib/disruptor-2.10.1.jar:/root/apache-storm-0.9.2-incubating/lib/commons-exec-1.1.jar:/root/apache-storm-0.9.2-incubating/lib/tools.macro-0.1.0.jar:/root/apache-storm-0.9.2-incubating/lib/jetty-6.1.26.jar:/root/apache-storm-0.9.2-incubating/lib/httpcore-4.3.2.jar:/root/apache-storm-0.9.2-incubating/lib/servlet-api-2.5-20081211.jar:/root/apache-storm-0.9.2-incubating/lib/ring-servlet-0.3.11.jar:/root/apache-storm-0.9.2-incubating/lib/logback-core-1.0.6.jar:/root/apache-storm-0.9.2-incubating/conf:storm-local/supervisor/stormdist/PaVan-14-1409661199/stormjar.jar backtype.storm.daemon.worker PaVan-14-1409661199 0574446a-f73d-42b0-bcc7-e6dd449cb75a 6703 156a8af9-fa3b-4772-b91c-787490fe0b34 Error: Could not find or load main class backtype.storm.daemon.worker Regards, PaVan… From: Pavan Jakati G Sent: 02 September 2014 15:23 To: user@storm.incubator.apache.org Subject: RE: REMOTE MODE STORM DEV sudo -u storm '/usr/java/jdk1.7.0_65/bin/java' '-server' '-Xmx768m' '-Djava.library.path=storm-local/sup ervisor/stormdist/PaVan-10-1409648607/resources/Linux-amd64:storm-local/supervisor/stormdist/PaVan-10-1409648607/resources:/usr/local/lib:/opt/local/lib:/usr/lib' '-Dlogfile.name=worker-6701.log' '-Dstorm.home=/root/apache-storm-0.9.2-incubating' '-Dlogback.configurationFile=/root/apache-storm-0.9.2-incubating/logback/cluster.xml' '-Dstorm.id=PaVan-10-1409648607' '-Dworker.id=b89ee630-7bd6-4976-8050-c43644c028ed' '-Dworker.port=6701' '-cp'
Re: REMOTE MODE STORM DEV
Hi Pavan, It seems you have a permission issue. Please check weather the directories that the storm user has appropriate permissions on the directories that contain the storm jars. Thanks, Supun.. On Tue, Sep 2, 2014 at 9:13 AM, Pavan Jakati G pava...@microland.com wrote: Can anybody help us get rid of below error : sudo -u storm /usr/java/jdk1.7.0_65/bin/java -server -Xmx768m -Djava.library.path=storm-local/supervisor/stormdist/PaVan-14-1409661199/resources/Linux-amd64:storm-local/supervisor/stormdist/PaVan-14-1409661199/resources:/usr/local/lib:/opt/local/lib:/usr/lib -Dlogfile.name=worker-6703.log -Dstorm.home=/root/apache-storm-0.9.2-incubating -Dlogback.configurationFile=/root/apache-storm-0.9.2-incubating/logback/cluster.xml -Dstorm.id=PaVan-14-1409661199 -Dworker.id=156a8af9-fa3b-4772-b91c-787490fe0b34 -Dworker.port=6703 -cp /root/apache-storm-0.9.2-incubating/lib/json-simple-1.1.jar:/root/apache-storm-0.9.2-incubating/lib/core.incubator-0.1.0.jar:/root/apache-storm-0.9.2-incubating/lib/asm-4.0.jar:/root/apache-storm-0.9.2-incubating/lib/commons-io-2.4.jar:/root/apache-storm-0.9.2-incubating/lib/httpclient-4.3.3.jar:/root/apache-storm-0.9.2-incubating/lib/jline-2.11.jar:/root/apache-storm-0.9.2-incubating/lib/tools.logging-0.2.3.jar:/root/apache-storm-0.9.2-incubating/lib/logback-classic-1.0.6.jar:/root/apache-storm-0.9.2-incubating/lib/commons-logging-1.1.3.jar:/root/apache-storm-0.9.2-incubating/lib/ring-core-1.1.5.jar:/root/apache-storm-0.9.2-incubating/lib/ring-devel-0.3.11.jar:/root/apache-storm-0.9.2-incubating/lib/curator-client-2.4.0.jar:/root/apache-storm-0.9.2-incubating/lib/clj-stacktrace-0.2.4.jar:/root/apache-storm-0.9.2-incubating/lib/clj-time-0.4.1.jar:/root/apache-storm-0.9.2-incubating/lib/commons-lang-2.5.jar:/root/apache-storm-0.9.2-incubating/lib/zookeeper-3.4.5.jar:/root/apache-storm-0.9.2-incubating/lib/compojure-1.1.3.jar:/root/apache-storm-0.9.2-incubating/lib/joda-time-2.0.jar:/root/apache-storm-0.9.2-incubating/lib/chill-java-0.3.5.jar:/root/apache-storm-0.9.2-incubating/lib/clout-1.0.1.jar:/root/apache-storm-0.9.2-incubating/lib/kryo-2.21.jar:/root/apache-storm-0.9.2-incubating/lib/snakeyaml-1.11.jar:/root/apache-storm-0.9.2-incubating/lib/minlog-1.2.jar:/root/apache-storm-0.9.2-incubating/lib/storm-core-0.9.2-incubating.jar:/root/apache-storm-0.9.2-incubating/lib/jgrapht-core-0.9.0.jar:/root/apache-storm-0.9.2-incubating/lib/slf4j-api-1.6.5.jar:/root/apache-storm-0.9.2-incubating/lib/hiccup-0.3.6.jar:/root/apache-storm-0.9.2-incubating/lib/netty-3.6.3.Final.jar:/root/apache-storm-0.9.2-incubating/lib/curator-framework-2.4.0.jar:/root/apache-storm-0.9.2-incubating/lib/guava-13.0.jar:/root/apache-storm-0.9.2-incubating/lib/log4j-over-slf4j-1.6.6.jar:/root/apache-storm-0.9.2-incubating/lib/commons-fileupload-1.2.1.jar:/root/apache-storm-0.9.2-incubating/lib/servlet-api-2.5.jar:/root/apache-storm-0.9.2-incubating/lib/reflectasm-1.07-shaded.jar:/root/apache-storm-0.9.2-incubating/lib/jetty-util-6.1.26.jar:/root/apache-storm-0.9.2-incubating/lib/objenesis-1.2.jar:/root/apache-storm-0.9.2-incubating/lib/tools.cli-0.2.4.jar:/root/apache-storm-0.9.2-incubating/lib/ring-jetty-adapter-0.3.11.jar:/root/apache-storm-0.9.2-incubating/lib/commons-codec-1.6.jar:/root/apache-storm-0.9.2-incubating/lib/clojure-1.5.1.jar:/root/apache-storm-0.9.2-incubating/lib/netty-3.2.2.Final.jar:/root/apache-storm-0.9.2-incubating/lib/math.numeric-tower-0.0.1.jar:/root/apache-storm-0.9.2-incubating/lib/carbonite-1.4.0.jar:/root/apache-storm-0.9.2-incubating/lib/disruptor-2.10.1.jar:/root/apache-storm-0.9.2-incubating/lib/commons-exec-1.1.jar:/root/apache-storm-0.9.2-incubating/lib/tools.macro-0.1.0.jar:/root/apache-storm-0.9.2-incubating/lib/jetty-6.1.26.jar:/root/apache-storm-0.9.2-incubating/lib/httpcore-4.3.2.jar:/root/apache-storm-0.9.2-incubating/lib/servlet-api-2.5-20081211.jar:/root/apache-storm-0.9.2-incubating/lib/ring-servlet-0.3.11.jar:/root/apache-storm-0.9.2-incubating/lib/logback-core-1.0.6.jar:/root/apache-storm-0.9.2-incubating/conf:storm-local/supervisor/stormdist/PaVan-14-1409661199/stormjar.jar backtype.storm.daemon.worker PaVan-14-1409661199 0574446a-f73d-42b0-bcc7-e6dd449cb75a 6703 156a8af9-fa3b-4772-b91c-787490fe0b34 Error: Could not find or load main class backtype.storm.daemon.worker Regards, PaVan… *From:* Pavan Jakati G *Sent:* 02 September 2014 15:23 *To:* user@storm.incubator.apache.org *Subject:* RE: REMOTE MODE STORM DEV sudo -u storm '/usr/java/jdk1.7.0_65/bin/java' '-server' '-Xmx768m' '-Djava.library.path=storm-local/sup ervisor/stormdist/PaVan-10-1409648607/resources/Linux-amd64:storm-local/supervisor/stormdist/PaVan-10-1409648607/resources:/usr/local/lib:/opt/local/lib:/usr/lib' '-Dlogfile.name=worker-6701.log' '-Dstorm.home=/root/apache-storm-0.9.2-incubating' '-Dlogback.configurationFile=/root/apache-storm-0.9.2-incubating/logback/cluster.xml' '-Dstorm.id=PaVan-10-1409648607'
Re: Error on Supervisor start
Hi Harsha, so I ran the command by hand and no exceptions were thrown this time. There was a unable to delete file exception before, but I don't think that is preventing the worker to start. On Mon, Sep 1, 2014 at 1:41 PM, Telles Nobrega tellesnobr...@gmail.com wrote: One possible problem, just thinking. When I installed zeromq and jzmq I deleted the folders afterwards, is that a problem? Do they need to be there or just to compile and install? On Mon, Sep 1, 2014 at 1:22 PM, Telles Nobrega tellesnobr...@gmail.com wrote: Hi Harsha, /usr/local/storm belongs to storm user. I've ran into the problem before installing 0.8.2 but I can't remember how to solve it. I will try to start the supervisor manually and see what happens. On Mon, Sep 1, 2014 at 1:06 PM, Harsha st...@harsha.io wrote: Hi Telles, Can you check if the storm user has permissions for /usr/local/storm. Assuming that you installed storm under /usr/local/storm and trying to run the supervisor daemon as user storm. Storm creates a dir storm-local and logs under STORM_HOME for storing metadata and logs. Before using supervisord to start storm daemons it would helpful for you test out running them manually. -Harsha On Mon, Sep 1, 2014, at 08:01 AM, Telles Nobrega wrote: Hi, I installed a storm cluster in local vms that run ubuntu, following the tutorial http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/#configure-storm but i install storm-9.1 The supervisors were not starting and I ran the command manually and got this error. 2014-09-01 14:56:16 b.s.d.worker [ERROR] Error on initialization of server mk-worker java.io.IOException: No such file or directory at java.io.UnixFileSystem.createFileExclusively(Native Method) ~[na:1.7.0_51] at java.io.File.createNewFile(File.java:1006) ~[na:1.7.0_51] at backtype.storm.util$touch.invoke(util.clj:493) ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating] at backtype.storm.daemon.worker$eval4413$exec_fn__1102__auto4414.invoke(worker.clj:352) ~[na:0.9.1-incubating] at clojure.lang.AFn.applyToHelper(AFn.java:185) [clojure-1.4.0.jar:na] at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.4.0.jar:na] at clojure.core$apply.invoke(core.clj:601) ~[clojure-1.4.0.jar:na] at backtype.storm.daemon.worker$eval4413$mk_worker__4469.doInvoke(worker.clj:344) [na:0.9.1-incubating] at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.4.0.jar:na] at backtype.storm.daemon.worker$_main.invoke(worker.clj:454) [na:0.9.1-incubating] at clojure.lang.AFn.applyToHelper(AFn.java:172) [clojure-1.4.0.jar:na] at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.4.0.jar:na] at backtype.storm.daemon.worker.main(Unknown Source) [storm-core-0.9.1-incubating.jar:0.9.1-incubating] 2014-09-01 14:56:16 b.s.util [INFO] Halting process: (Error on initialization) Have anyone seen this? Thanks -- -- Telles Mota Vidal Nobrega M.sc. Candidate at UFCG B.sc. in Computer Science at UFCG Software Engineer at OpenStack Project - HP/LSD-UFCG -- -- Telles Mota Vidal Nobrega M.sc. Candidate at UFCG B.sc. in Computer Science at UFCG Software Engineer at OpenStack Project - HP/LSD-UFCG -- -- Telles Mota Vidal Nobrega M.sc. Candidate at UFCG B.sc. in Computer Science at UFCG Software Engineer at OpenStack Project - HP/LSD-UFCG -- -- Telles Mota Vidal Nobrega M.sc. Candidate at UFCG B.sc. in Computer Science at UFCG Software Engineer at OpenStack Project - HP/LSD-UFCG
Re: Supervisor always down 3s after execution
Usually when this happens, we remove the storm directory from ZooKeeper using zkCli.sh, remove the storm-local directories and start fresh. Thanks, Supun.. On Tue, Sep 2, 2014 at 6:39 AM, Benjamin SOULAS benjamin.soula...@gmail.com wrote: Hi everyone, I followed your instructions for installing a zookeeper server, i downloaded it on the website, extract the tar file somewhere in a machine on my cluster, i made those modifications in my zoo.cfg : # The number of milliseconds of each tick tickTime=2000 # The number of ticks that the initial # synchronization phase can take initLimit=10 # The number of ticks that can pass between # sending a request and getting an acknowledgement syncLimit=5 # the directory where the snapshot is stored. # do not use /tmp for storage, /tmp here is just # example sakes. dataDir=/home/bsoulas/zookeeper/zookeeper-3.4.6/data/ # the port at which the clients will connect clientPort=2181 # the maximum number of client connections. # increase this if you need to handle more clients #maxClientCnxns=60 # # Be sure to read the maintenance section of the # administrator guide before turning on autopurge. # # http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance # # The number of snapshots to retain in dataDir #autopurge.snapRetainCount=3 # Purge task interval in hours # Set to 0 to disable auto purge feature #autopurge.purgeInterval=1 In the log4j.properties, i uncommented the line for the log file : # Example with rolling log file log4j.rootLogger=DEBUG, CONSOLE, ROLLINGFILE Then i went to my storm.yaml (located here in my case, because i took the source version) : /home/bsoulas/incubator-storm-master/storm-dist/binary/target/apache-storm-0.9.3-ben/apache-storm-0.9.3-ben/conf This file contain this configuration : ### These MUST be filled in for a storm configuration storm.zookeeper.servers: - paradent-4 # - paradent-47 # - paradent-48 # nimbus.host: paradent-4 # # # # These may optionally be filled in: # ## List of custom serializations # topology.kryo.register: # - org.mycompany.MyType # - org.mycompany.MyType2: org.mycompany.MyType2Serializer # ## List of custom kryo decorators # topology.kryo.decorators: # - org.mycompany.MyDecorator # ## Locations of the drpc servers # drpc.servers: # - server1 # - server2 ## Metrics Consumers # topology.metrics.consumer.register: # - class: backtype.storm.metric.LoggingMetricsConsumer # parallelism.hint: 1 # - class: org.mycompany.MyMetricsConsumer # parallelism.hint: 1 # argument: # - endpoint: metrics-collector.mycompany.org dev.zookeeper.path: paradent-4.rennes.grid5000.fr: ~/home/bsoulas/zookeeper/zookeeper-3.4.6/ storm.zookeeper.port: 2181 To launch storm on the cluster, i launch it thanks to *storm nimbus *(on a machine named paradent-4), then my zookeeper Server *sh zkServer.sh start* (on paradent-4 again)(which create a *zookeeper_server.pid* where the pid of the zookeeper is written, i know it's obvious ..._ ). After i launch my *storm ui* for having a visual of my storm app (on paradent-4). Until now, everything work fine. Now, the logical way implies i launch my supervisor, on a different machine (here *paradent-39*) thanks to *storm supervisor*, it is launched but once again, 3 or 4 seconds after it's down. So i watched the supervisor.log located : /home/bsoulas/incubator-storm-master/storm-dist/binary/target/apache-storm-0.9.3-ben/apache-storm-0.9.3-ben/logs And here appear a tricky error : 2014-09-02 09:31:37 o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting 2014-09-02 09:31:37 o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=paradent-4:2181 sessionTimeout=2 watcher=org.apache.curator.ConnectionState@220df4c8 2014-09-02 09:31:37 o.a.z.ClientCnxn [INFO] Opening socket connection to server paradent-4.rennes.grid5000.fr/172.16.97.4:2181. Will not attempt to authenticate using SASL (unknown error) 2014-09-02 09:31:37 o.a.z.ClientCnxn [INFO] Socket connection established to paradent-4.rennes.grid5000.fr/172.16.97.4:2181, initiating session 2014-09-02 09:31:37 o.a.z.ClientCnxn [INFO] Session establishment complete on server paradent-4.rennes.grid5000.fr/172.16.97.4:2181, sessionid = 0x14835a48ca90004, negotiated timeout = 2 2014-09-02 09:31:37 o.a.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED 2014-09-02 09:31:37 o.a.c.f.s.ConnectionStateManager [WARN] There are no ConnectionStateListeners registered. 2014-09-02 09:31:37 b.s.zookeeper [INFO] Zookeeper state update: :connected:none 2014-09-02 09:31:38 o.a.z.ZooKeeper [INFO] Session: 0x14835a48ca90004 closed 2014-09-02 09:31:38 o.a.z.ClientCnxn [INFO] EventThread shut down 2014-09-02 09:31:38 o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting 2014-09-02 09:31:38
Re: Error on Supervisor start
Hi Telles, I haven't used zeromq or jzmq before sorry I can't help you there. so I ran the command by hand and no exceptions were thrown this time So everything looks good now? -Harsha On Tue, Sep 2, 2014, at 07:32 AM, Telles Nobrega wrote: Hi Harsha, so I ran the command by hand and no exceptions were thrown this time. There was a unable to delete file exception before, but I don't think that is preventing the worker to start. On Mon, Sep 1, 2014 at 1:41 PM, Telles Nobrega [1]tellesnobr...@gmail.com wrote: One possible problem, just thinking. When I installed zeromq and jzmq I deleted the folders afterwards, is that a problem? Do they need to be there or just to compile and install? On Mon, Sep 1, 2014 at 1:22 PM, Telles Nobrega [2]tellesnobr...@gmail.com wrote: Hi Harsha, /usr/local/storm belongs to storm user. I've ran into the problem before installing 0.8.2 but I can't remember how to solve it. I will try to start the supervisor manually and see what happens. On Mon, Sep 1, 2014 at 1:06 PM, Harsha [3]st...@harsha.io wrote: Hi Telles, Can you check if the storm user has permissions for /usr/local/storm. Assuming that you installed storm under /usr/local/storm and trying to run the supervisor daemon as user storm. Storm creates a dir storm-local and logs under STORM_HOME for storing metadata and logs. Before using supervisord to start storm daemons it would helpful for you test out running them manually. -Harsha On Mon, Sep 1, 2014, at 08:01 AM, Telles Nobrega wrote: Hi, I installed a storm cluster in local vms that run ubuntu, following the tutorial [4]http://www.michael-noll.com/tutorials/running-multi -node-storm-cluster/#configure-storm but i install storm-9.1 The supervisors were not starting and I ran the command manually and got this error. 2014-09-01 14:56:16 b.s.d.worker [ERROR] Error on initialization of server mk-worker java.io.IOException: No such file or directory at java.io.UnixFileSystem.createFileExclusively(Native Method) ~[na:1.7.0_51] at java.io.File.createNewFile(File.java:1006) ~[na:1.7.0_51] at backtype.storm.util$touch.invoke(util.clj:493) ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating] at backtype.storm.daemon.worker$eval4413$exec_fn__1102__auto44 14.invoke(worker.clj:352) ~[na:0.9.1-incubating] at clojure.lang.AFn.applyToHelper(AFn.java:185) [clojure-1.4.0.jar:na] at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.4.0.jar:na] at clojure.core$apply.invoke(core.clj:601) ~[clojure-1.4.0.jar:na] at backtype.storm.daemon.worker$eval4413$mk_worker__4469.doInvoke( worker.clj:344) [na:0.9.1-incubating] at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.4.0.jar:na] at backtype.storm.daemon.worker$_main.invoke(worker.clj:454) [na:0.9.1-incubating] at clojure.lang.AFn.applyToHelper(AFn.java:172) [clojure-1.4.0.jar:na] at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.4.0.jar:na] at backtype.storm.daemon.worker.main(Unknown Source) [storm-core-0.9.1-incubating.jar:0.9.1-incubating] 2014-09-01 14:56:16 b.s.util [INFO] Halting process: (Error on initialization) Have anyone seen this? Thanks -- -- Telles Mota Vidal Nobrega M.sc. Candidate at UFCG B.sc. in Computer Science at UFCG Software Engineer at OpenStack Project - HP/LSD-UFCG -- -- Telles Mota Vidal Nobrega M.sc. Candidate at UFCG B.sc. in Computer Science at UFCG Software Engineer at OpenStack Project - HP/LSD-UFCG -- -- Telles Mota Vidal Nobrega M.sc. Candidate at UFCG B.sc. in Computer Science at UFCG Software Engineer at OpenStack Project - HP/LSD-UFCG -- -- Telles Mota Vidal Nobrega M.sc. Candidate at UFCG B.sc. in Computer Science at UFCG Software Engineer at OpenStack Project - HP/LSD-UFCG References 1. mailto:tellesnobr...@gmail.com 2. mailto:tellesnobr...@gmail.com 3. mailto:st...@harsha.io 4. http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/#configure-storm
Re: Error on Supervisor start
No, it still doesn't start, but there is no exception thrown. On Tue, Sep 2, 2014 at 12:00 PM, Harsha st...@harsha.io wrote: Hi Telles, I haven't used zeromq or jzmq before sorry I can't help you there. so I ran the command by hand and no exceptions were thrown this time So everything looks good now? -Harsha On Tue, Sep 2, 2014, at 07:32 AM, Telles Nobrega wrote: Hi Harsha, so I ran the command by hand and no exceptions were thrown this time. There was a unable to delete file exception before, but I don't think that is preventing the worker to start. On Mon, Sep 1, 2014 at 1:41 PM, Telles Nobrega tellesnobr...@gmail.com wrote: One possible problem, just thinking. When I installed zeromq and jzmq I deleted the folders afterwards, is that a problem? Do they need to be there or just to compile and install? On Mon, Sep 1, 2014 at 1:22 PM, Telles Nobrega tellesnobr...@gmail.com wrote: Hi Harsha, /usr/local/storm belongs to storm user. I've ran into the problem before installing 0.8.2 but I can't remember how to solve it. I will try to start the supervisor manually and see what happens. On Mon, Sep 1, 2014 at 1:06 PM, Harsha st...@harsha.io wrote: Hi Telles, Can you check if the storm user has permissions for /usr/local/storm. Assuming that you installed storm under /usr/local/storm and trying to run the supervisor daemon as user storm. Storm creates a dir storm-local and logs under STORM_HOME for storing metadata and logs. Before using supervisord to start storm daemons it would helpful for you test out running them manually. -Harsha On Mon, Sep 1, 2014, at 08:01 AM, Telles Nobrega wrote: Hi, I installed a storm cluster in local vms that run ubuntu, following the tutorial http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/#configure-storm but i install storm-9.1 The supervisors were not starting and I ran the command manually and got this error. 2014-09-01 14:56:16 b.s.d.worker [ERROR] Error on initialization of server mk-worker java.io.IOException: No such file or directory at java.io.UnixFileSystem.createFileExclusively(Native Method) ~[na:1.7.0_51] at java.io.File.createNewFile(File.java:1006) ~[na:1.7.0_51] at backtype.storm.util$touch.invoke(util.clj:493) ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating] at backtype.storm.daemon.worker$eval4413$exec_fn__1102__auto4414.invoke(worker.clj:352) ~[na:0.9.1-incubating] at clojure.lang.AFn.applyToHelper(AFn.java:185) [clojure-1.4.0.jar:na] at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.4.0.jar:na] at clojure.core$apply.invoke(core.clj:601) ~[clojure-1.4.0.jar:na] at backtype.storm.daemon.worker$eval4413$mk_worker__4469.doInvoke(worker.clj:344) [na:0.9.1-incubating] at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.4.0.jar:na] at backtype.storm.daemon.worker$_main.invoke(worker.clj:454) [na:0.9.1-incubating] at clojure.lang.AFn.applyToHelper(AFn.java:172) [clojure-1.4.0.jar:na] at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.4.0.jar:na] at backtype.storm.daemon.worker.main(Unknown Source) [storm-core-0.9.1-incubating.jar:0.9.1-incubating] 2014-09-01 14:56:16 b.s.util [INFO] Halting process: (Error on initialization) Have anyone seen this? Thanks -- -- Telles Mota Vidal Nobrega M.sc. Candidate at UFCG B.sc. in Computer Science at UFCG Software Engineer at OpenStack Project - HP/LSD-UFCG -- -- Telles Mota Vidal Nobrega M.sc. Candidate at UFCG B.sc. in Computer Science at UFCG Software Engineer at OpenStack Project - HP/LSD-UFCG -- -- Telles Mota Vidal Nobrega M.sc. Candidate at UFCG B.sc. in Computer Science at UFCG Software Engineer at OpenStack Project - HP/LSD-UFCG -- -- Telles Mota Vidal Nobrega M.sc. Candidate at UFCG B.sc. in Computer Science at UFCG Software Engineer at OpenStack Project - HP/LSD-UFCG -- -- Telles Mota Vidal Nobrega M.sc. Candidate at UFCG B.sc. in Computer Science at UFCG Software Engineer at OpenStack Project - HP/LSD-UFCG
Re: Error on Supervisor start
If possible can you share your storm.yaml. Incase if you are upgrading the storm from previous installations I recommend you to delete storm-local , zookeeper data.dir and start the storm daemons again. -Harsha On Tue, Sep 2, 2014, at 08:09 AM, Telles Nobrega wrote: No, it still doesn't start, but there is no exception thrown. On Tue, Sep 2, 2014 at 12:00 PM, Harsha [1]st...@harsha.io wrote: Hi Telles, I haven't used zeromq or jzmq before sorry I can't help you there. so I ran the command by hand and no exceptions were thrown this time So everything looks good now? -Harsha On Tue, Sep 2, 2014, at 07:32 AM, Telles Nobrega wrote: Hi Harsha, so I ran the command by hand and no exceptions were thrown this time. There was a unable to delete file exception before, but I don't think that is preventing the worker to start. On Mon, Sep 1, 2014 at 1:41 PM, Telles Nobrega [2]tellesnobr...@gmail.com wrote: One possible problem, just thinking. When I installed zeromq and jzmq I deleted the folders afterwards, is that a problem? Do they need to be there or just to compile and install? On Mon, Sep 1, 2014 at 1:22 PM, Telles Nobrega [3]tellesnobr...@gmail.com wrote: Hi Harsha, /usr/local/storm belongs to storm user. I've ran into the problem before installing 0.8.2 but I can't remember how to solve it. I will try to start the supervisor manually and see what happens. On Mon, Sep 1, 2014 at 1:06 PM, Harsha [4]st...@harsha.io wrote: Hi Telles, Can you check if the storm user has permissions for /usr/local/storm. Assuming that you installed storm under /usr/local/storm and trying to run the supervisor daemon as user storm. Storm creates a dir storm-local and logs under STORM_HOME for storing metadata and logs. Before using supervisord to start storm daemons it would helpful for you test out running them manually. -Harsha On Mon, Sep 1, 2014, at 08:01 AM, Telles Nobrega wrote: Hi, I installed a storm cluster in local vms that run ubuntu, following the tutorial [5]http://www.michael-noll.com/tutorials/running-multi -node-storm-cluster/#configure-storm but i install storm-9.1 The supervisors were not starting and I ran the command manually and got this error. 2014-09-01 14:56:16 b.s.d.worker [ERROR] Error on initialization of server mk-worker java.io.IOException: No such file or directory at java.io.UnixFileSystem.createFileExclusively(Native Method) ~[na:1.7.0_51] at java.io.File.createNewFile(File.java:1006) ~[na:1.7.0_51] at backtype.storm.util$touch.invoke(util.clj:493) ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating] at backtype.storm.daemon.worker$eval4413$exec_fn__1102__auto44 14.invoke(worker.clj:352) ~[na:0.9.1-incubating] at clojure.lang.AFn.applyToHelper(AFn.java:185) [clojure-1.4.0.jar:na] at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.4.0.jar:na] at clojure.core$apply.invoke(core.clj:601) ~[clojure-1.4.0.jar:na] at backtype.storm.daemon.worker$eval4413$mk_worker__4469.doInvoke( worker.clj:344) [na:0.9.1-incubating] at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.4.0.jar:na] at backtype.storm.daemon.worker$_main.invoke(worker.clj:454) [na:0.9.1-incubating] at clojure.lang.AFn.applyToHelper(AFn.java:172) [clojure-1.4.0.jar:na] at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.4.0.jar:na] at backtype.storm.daemon.worker.main(Unknown Source) [storm-core-0.9.1-incubating.jar:0.9.1-incubating] 2014-09-01 14:56:16 b.s.util [INFO] Halting process: (Error on initialization) Have anyone seen this? Thanks -- -- Telles Mota Vidal Nobrega M.sc. Candidate at UFCG B.sc. in Computer Science at UFCG Software Engineer at OpenStack Project - HP/LSD-UFCG -- -- Telles Mota Vidal Nobrega M.sc. Candidate at UFCG B.sc. in Computer Science at UFCG Software Engineer at OpenStack Project - HP/LSD-UFCG -- -- Telles Mota Vidal Nobrega M.sc. Candidate at UFCG B.sc. in Computer Science at UFCG Software Engineer at OpenStack Project - HP/LSD-UFCG -- -- Telles Mota Vidal Nobrega M.sc. Candidate at UFCG B.sc. in Computer Science at UFCG Software Engineer at OpenStack Project - HP/LSD-UFCG -- -- Telles Mota Vidal Nobrega M.sc. Candidate at UFCG B.sc. in Computer Science at UFCG Software Engineer at OpenStack Project - HP/LSD-UFCG References 1. mailto:st...@harsha.io 2. mailto:tellesnobr...@gmail.com 3. mailto:tellesnobr...@gmail.com 4. mailto:st...@harsha.io 5. http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/#configure-storm
Re: Supervisor always down 3s after execution
Hi Supun, It works at first but then it crash, again ... 2014-09-02 16:43 GMT+02:00 Supun Kamburugamuva supu...@gmail.com: Usually when this happens, we remove the storm directory from ZooKeeper using zkCli.sh, remove the storm-local directories and start fresh. Thanks, Supun.. On Tue, Sep 2, 2014 at 6:39 AM, Benjamin SOULAS benjamin.soula...@gmail.com wrote: Hi everyone, I followed your instructions for installing a zookeeper server, i downloaded it on the website, extract the tar file somewhere in a machine on my cluster, i made those modifications in my zoo.cfg : # The number of milliseconds of each tick tickTime=2000 # The number of ticks that the initial # synchronization phase can take initLimit=10 # The number of ticks that can pass between # sending a request and getting an acknowledgement syncLimit=5 # the directory where the snapshot is stored. # do not use /tmp for storage, /tmp here is just # example sakes. dataDir=/home/bsoulas/zookeeper/zookeeper-3.4.6/data/ # the port at which the clients will connect clientPort=2181 # the maximum number of client connections. # increase this if you need to handle more clients #maxClientCnxns=60 # # Be sure to read the maintenance section of the # administrator guide before turning on autopurge. # # http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance # # The number of snapshots to retain in dataDir #autopurge.snapRetainCount=3 # Purge task interval in hours # Set to 0 to disable auto purge feature #autopurge.purgeInterval=1 In the log4j.properties, i uncommented the line for the log file : # Example with rolling log file log4j.rootLogger=DEBUG, CONSOLE, ROLLINGFILE Then i went to my storm.yaml (located here in my case, because i took the source version) : /home/bsoulas/incubator-storm-master/storm-dist/binary/target/apache-storm-0.9.3-ben/apache-storm-0.9.3-ben/conf This file contain this configuration : ### These MUST be filled in for a storm configuration storm.zookeeper.servers: - paradent-4 # - paradent-47 # - paradent-48 # nimbus.host: paradent-4 # # # # These may optionally be filled in: # ## List of custom serializations # topology.kryo.register: # - org.mycompany.MyType # - org.mycompany.MyType2: org.mycompany.MyType2Serializer # ## List of custom kryo decorators # topology.kryo.decorators: # - org.mycompany.MyDecorator # ## Locations of the drpc servers # drpc.servers: # - server1 # - server2 ## Metrics Consumers # topology.metrics.consumer.register: # - class: backtype.storm.metric.LoggingMetricsConsumer # parallelism.hint: 1 # - class: org.mycompany.MyMetricsConsumer # parallelism.hint: 1 # argument: # - endpoint: metrics-collector.mycompany.org dev.zookeeper.path: paradent-4.rennes.grid5000.fr: ~/home/bsoulas/zookeeper/zookeeper-3.4.6/ storm.zookeeper.port: 2181 To launch storm on the cluster, i launch it thanks to *storm nimbus *(on a machine named paradent-4), then my zookeeper Server *sh zkServer.sh start* (on paradent-4 again)(which create a *zookeeper_server.pid* where the pid of the zookeeper is written, i know it's obvious ..._ ). After i launch my *storm ui* for having a visual of my storm app (on paradent-4). Until now, everything work fine. Now, the logical way implies i launch my supervisor, on a different machine (here *paradent-39*) thanks to *storm supervisor*, it is launched but once again, 3 or 4 seconds after it's down. So i watched the supervisor.log located : /home/bsoulas/incubator-storm-master/storm-dist/binary/target/apache-storm-0.9.3-ben/apache-storm-0.9.3-ben/logs And here appear a tricky error : 2014-09-02 09:31:37 o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting 2014-09-02 09:31:37 o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=paradent-4:2181 sessionTimeout=2 watcher=org.apache.curator.ConnectionState@220df4c8 2014-09-02 09:31:37 o.a.z.ClientCnxn [INFO] Opening socket connection to server paradent-4.rennes.grid5000.fr/172.16.97.4:2181. Will not attempt to authenticate using SASL (unknown error) 2014-09-02 09:31:37 o.a.z.ClientCnxn [INFO] Socket connection established to paradent-4.rennes.grid5000.fr/172.16.97.4:2181, initiating session 2014-09-02 09:31:37 o.a.z.ClientCnxn [INFO] Session establishment complete on server paradent-4.rennes.grid5000.fr/172.16.97.4:2181, sessionid = 0x14835a48ca90004, negotiated timeout = 2 2014-09-02 09:31:37 o.a.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED 2014-09-02 09:31:37 o.a.c.f.s.ConnectionStateManager [WARN] There are no ConnectionStateListeners registered. 2014-09-02 09:31:37 b.s.zookeeper [INFO] Zookeeper state update: :connected:none 2014-09-02 09:31:38 o.a.z.ZooKeeper [INFO] Session: 0x14835a48ca90004 closed 2014-09-02 09:31:38
Re: Supervisor always down 3s after execution
Hi Benjamin, Correct me if I missed it , in your config I don't see storm.local.dir defined. If its not defined in config storm will create one in the storm_installation dir which seems to be /home/bsoulas/incubator-storm-master/storm-dist/binary/target/a pache-storm-0.9.3-ben/apache-storm-0.9.3-ben/ and are you running the supervisor and nimbus as user bsoulas. When you are running storm nimbus or storm supervisor command which storm command its pointing. Did you export STORM_HOME=/home/bsoulas/incubator-storm-master/storm-dist/bina ry/target/apache-storm-0.9.3-ben and also added it to PATH. I am checking to see if you had any previous installation of storm and invoking the storm command from previous installation. Can you also check zookeeper logs . -Harsha On Tue, Sep 2, 2014, at 03:39 AM, Benjamin SOULAS wrote: Hi everyone, I followed your instructions for installing a zookeeper server, i downloaded it on the website, extract the tar file somewhere in a machine on my cluster, i made those modifications in my zoo.cfg : # The number of milliseconds of each tick tickTime=2000 # The number of ticks that the initial # synchronization phase can take initLimit=10 # The number of ticks that can pass between # sending a request and getting an acknowledgement syncLimit=5 # the directory where the snapshot is stored. # do not use /tmp for storage, /tmp here is just # example sakes. dataDir=/home/bsoulas/zookeeper/zookeeper-3.4.6/data/ # the port at which the clients will connect clientPort=2181 # the maximum number of client connections. # increase this if you need to handle more clients #maxClientCnxns=60 # # Be sure to read the maintenance section of the # administrator guide before turning on autopurge. # # [1]http://zookeeper.apache.org/doc/current/zookeeperAdmin.html# sc_maintenance # # The number of snapshots to retain in dataDir #autopurge.snapRetainCount=3 # Purge task interval in hours # Set to 0 to disable auto purge feature #autopurge.purgeInterval=1 In the log4j.properties, i uncommented the line for the log file : # Example with rolling log file log4j.rootLogger=DEBUG, CONSOLE, ROLLINGFILE Then i went to my storm.yaml (located here in my case, because i took the source version) : /home/bsoulas/incubator-storm-master/storm-dist/binary/target/a pache-storm-0.9.3-ben/apache-storm-0.9.3-ben/conf This file contain this configuration : ### These MUST be filled in for a storm configuration storm.zookeeper.servers: - paradent-4 # - paradent-47 # - paradent-48 # nimbus.host: paradent-4 # # # # These may optionally be filled in: # ## List of custom serializations # topology.kryo.register: # - org.mycompany.MyType # - org.mycompany.MyType2: org.mycompany.MyType2Serializer # ## List of custom kryo decorators # topology.kryo.decorators: # - org.mycompany.MyDecorator # ## Locations of the drpc servers # drpc.servers: # - server1 # - server2 ## Metrics Consumers # topology.metrics.consumer.register: # - class: backtype.storm.metric.LoggingMetricsConsumer # parallelism.hint: 1 # - class: org.mycompany.MyMetricsConsumer # parallelism.hint: 1 # argument: # - endpoint: [2]metrics-collector.mycompany.org dev.zookeeper.path: paradent-4.rennes.grid5000.fr:~/home/bsoulas/zookeeper/zookeep er-3.4.6/ storm.zookeeper.port: 2181 To launch storm on the cluster, i launch it thanks to storm nimbus (on a machine named paradent-4), then my zookeeper Server sh zkServer.sh start (on paradent-4 again)(which create a zookeeper_server.pid where the pid of the zookeeper is written, i know it's obvious ..._ ). After i launch my storm ui for having a visual of my storm app (on paradent-4). Until now, everything work fine. Now, the logical way implies i launch my supervisor, on a different machine (here paradent-39) thanks to storm supervisor, it is launched but once again, 3 or 4 seconds after it's down. So i watched the supervisor.log located : /home/bsoulas/incubator-storm-master/storm-dist/binary/target/a pache-storm-0.9.3-ben/apache-storm-0.9.3-ben/logs And here appear a tricky error : 2014-09-02 09:31:37 o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting 2014-09-02 09:31:37 o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=paradent-4:2181 sessionTimeout=2 watcher=org.apache.curator.ConnectionState@220df4c8 2014-09-02 09:31:37 o.a.z.ClientCnxn [INFO] Opening socket connection to server [3]paradent-4.rennes.grid5000.fr/172.16.97.4:2181. Will not attempt to authenticate using SASL (unknown error) 2014-09-02 09:31:37 o.a.z.ClientCnxn [INFO] Socket connection established to [4]paradent-4.rennes.grid5000.fr/172.16.97.4:2181, initiating session 2014-09-02 09:31:37 o.a.z.ClientCnxn [INFO] Session establishment complete on server [5]paradent-4.rennes.grid5000.fr/172.16.97.4:2181, sessionid = 0x14835a48ca90004, negotiated timeout = 2 2014-09-02
Re: Error on Supervisor start
It is a fresh install, different machines. nimbus.childopts: -Xmx1024m -Djava.net.preferIPv4Stack=true nimbus.host: cluster2-master-tmp-001 storm.local.dir: /app/storm storm.zookeeper.servers: - cluster2-master-tmp-001 supervisor.childopts: -Djava.net.preferIPv4Stack=true ui.childopts: -Xmx768m -Djava.net.preferIPv4Stack=true worker.childopts: -Xmx768m -Djava.net.preferIPv4Stack=true On Tue, Sep 2, 2014 at 12:23 PM, Harsha st...@harsha.io wrote: If possible can you share your storm.yaml. Incase if you are upgrading the storm from previous installations I recommend you to delete storm-local , zookeeper data.dir and start the storm daemons again. -Harsha On Tue, Sep 2, 2014, at 08:09 AM, Telles Nobrega wrote: No, it still doesn't start, but there is no exception thrown. On Tue, Sep 2, 2014 at 12:00 PM, Harsha st...@harsha.io wrote: Hi Telles, I haven't used zeromq or jzmq before sorry I can't help you there. so I ran the command by hand and no exceptions were thrown this time So everything looks good now? -Harsha On Tue, Sep 2, 2014, at 07:32 AM, Telles Nobrega wrote: Hi Harsha, so I ran the command by hand and no exceptions were thrown this time. There was a unable to delete file exception before, but I don't think that is preventing the worker to start. On Mon, Sep 1, 2014 at 1:41 PM, Telles Nobrega tellesnobr...@gmail.com wrote: One possible problem, just thinking. When I installed zeromq and jzmq I deleted the folders afterwards, is that a problem? Do they need to be there or just to compile and install? On Mon, Sep 1, 2014 at 1:22 PM, Telles Nobrega tellesnobr...@gmail.com wrote: Hi Harsha, /usr/local/storm belongs to storm user. I've ran into the problem before installing 0.8.2 but I can't remember how to solve it. I will try to start the supervisor manually and see what happens. On Mon, Sep 1, 2014 at 1:06 PM, Harsha st...@harsha.io wrote: Hi Telles, Can you check if the storm user has permissions for /usr/local/storm. Assuming that you installed storm under /usr/local/storm and trying to run the supervisor daemon as user storm. Storm creates a dir storm-local and logs under STORM_HOME for storing metadata and logs. Before using supervisord to start storm daemons it would helpful for you test out running them manually. -Harsha On Mon, Sep 1, 2014, at 08:01 AM, Telles Nobrega wrote: Hi, I installed a storm cluster in local vms that run ubuntu, following the tutorial http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/#configure-storm but i install storm-9.1 The supervisors were not starting and I ran the command manually and got this error. 2014-09-01 14:56:16 b.s.d.worker [ERROR] Error on initialization of server mk-worker java.io.IOException: No such file or directory at java.io.UnixFileSystem.createFileExclusively(Native Method) ~[na:1.7.0_51] at java.io.File.createNewFile(File.java:1006) ~[na:1.7.0_51] at backtype.storm.util$touch.invoke(util.clj:493) ~[storm-core-0.9.1-incubating.jar:0.9.1-incubating] at backtype.storm.daemon.worker$eval4413$exec_fn__1102__auto4414.invoke(worker.clj:352) ~[na:0.9.1-incubating] at clojure.lang.AFn.applyToHelper(AFn.java:185) [clojure-1.4.0.jar:na] at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.4.0.jar:na] at clojure.core$apply.invoke(core.clj:601) ~[clojure-1.4.0.jar:na] at backtype.storm.daemon.worker$eval4413$mk_worker__4469.doInvoke(worker.clj:344) [na:0.9.1-incubating] at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.4.0.jar:na] at backtype.storm.daemon.worker$_main.invoke(worker.clj:454) [na:0.9.1-incubating] at clojure.lang.AFn.applyToHelper(AFn.java:172) [clojure-1.4.0.jar:na] at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.4.0.jar:na] at backtype.storm.daemon.worker.main(Unknown Source) [storm-core-0.9.1-incubating.jar:0.9.1-incubating] 2014-09-01 14:56:16 b.s.util [INFO] Halting process: (Error on initialization) Have anyone seen this? Thanks -- -- Telles Mota Vidal Nobrega M.sc. Candidate at UFCG B.sc. in Computer Science at UFCG Software Engineer at OpenStack Project - HP/LSD-UFCG -- -- Telles Mota Vidal Nobrega M.sc. Candidate at UFCG B.sc. in Computer Science at UFCG Software Engineer at OpenStack Project - HP/LSD-UFCG -- -- Telles Mota Vidal Nobrega M.sc. Candidate at UFCG B.sc. in Computer Science at UFCG Software Engineer at OpenStack Project - HP/LSD-UFCG -- -- Telles Mota Vidal Nobrega M.sc. Candidate at UFCG B.sc. in Computer Science at UFCG Software Engineer at OpenStack Project - HP/LSD-UFCG -- -- Telles Mota Vidal Nobrega
Re: Data validation
Thanks Harsha, I had looked into that but I just didnt know if its a good idea to add another database query to the process. But I actually ended up doing that which seem to be pretty good. Thanks. Kushan -- Kushan Maskey 817.403.7500 On Fri, Aug 29, 2014 at 5:14 PM, Harsha st...@harsha.io wrote: Kushan, Why not use cassandra counter to implement this http://www.datastax.com/documentation/cql/3.0/cql/cql_using/use_counter_t.html . you can create a counter field in a table in cassandra and let the storm bolts update it. I don't have much knowledge on internal representation of cassandra counters and how accurate they will be. -Harsha On Fri, Aug 29, 2014, at 12:15 PM, Kushan Maskey wrote: I have a batch process that runs more than 100K records of data and loads into Cassandra. I am having hard time validating the exact number of data that gets stored into C*. Now C* has more than 20Million records and when I do Select COUNT(1) FROM TABLE, I get *Request did not complete within rpc_timeout. *I tried to increate teh rpc_timeout but didnt help. The load process completes successfully without having any errors in the log. So I assumed that storm and kafka is set up correctly. I have 5 bolts and now I am at a point to add counter feature to the bolts to count how many messages successfully inserted. I tried to add a static counter field. But this will not work as its in clustered environment and static fields are not good. Can anyone suggest me a better way to validate the number of records that gets inserted into C*? This is one of the initail requirements to make sure that x number of records we processed thru batch and same number of records got inserted into C*. I also tried to set StormConfig with new property like stormConfig.put(Records_add_counter, 0); Then I wanted to increment the counter by one everytie the message comes a particularbolt but I get UnsupportedOperationException. I am thinking you cannot update the the value of a property at t his point. Any help will be appreciated. Thanks. -- Kushan Maskey 817.403.7500
Re: Supervisor always down 3s after execution
Hi Harsha, You're right, I didn't export STORM_HOME ... I will do it, maybe this is the problem. Thanks 2014-09-02 18:08 GMT+02:00 Harsha st...@harsha.io: Hi Benjamin, Correct me if I missed it , in your config I don't see storm.local.dir defined. If its not defined in config storm will create one in the storm_installation dir which seems to be /home/bsoulas/incubator-storm-master/storm-dist/binary/target/apache-storm-0.9.3-ben/apache-storm-0.9.3-ben/ and are you running the supervisor and nimbus as user bsoulas. When you are running storm nimbus or storm supervisor command which storm command its pointing. Did you export STORM_HOME=/home/bsoulas/incubator-storm-master/storm-dist/binary/target/apache-storm-0.9.3-ben and also added it to PATH. I am checking to see if you had any previous installation of storm and invoking the storm command from previous installation. Can you also check zookeeper logs . -Harsha On Tue, Sep 2, 2014, at 03:39 AM, Benjamin SOULAS wrote: Hi everyone, I followed your instructions for installing a zookeeper server, i downloaded it on the website, extract the tar file somewhere in a machine on my cluster, i made those modifications in my zoo.cfg : # The number of milliseconds of each tick tickTime=2000 # The number of ticks that the initial # synchronization phase can take initLimit=10 # The number of ticks that can pass between # sending a request and getting an acknowledgement syncLimit=5 # the directory where the snapshot is stored. # do not use /tmp for storage, /tmp here is just # example sakes. dataDir=/home/bsoulas/zookeeper/zookeeper-3.4.6/data/ # the port at which the clients will connect clientPort=2181 # the maximum number of client connections. # increase this if you need to handle more clients #maxClientCnxns=60 # # Be sure to read the maintenance section of the # administrator guide before turning on autopurge. # # http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance # # The number of snapshots to retain in dataDir #autopurge.snapRetainCount=3 # Purge task interval in hours # Set to 0 to disable auto purge feature #autopurge.purgeInterval=1 In the log4j.properties, i uncommented the line for the log file : # Example with rolling log file log4j.rootLogger=DEBUG, CONSOLE, ROLLINGFILE Then i went to my storm.yaml (located here in my case, because i took the source version) : /home/bsoulas/incubator-storm-master/storm-dist/binary/target/apache-storm-0.9.3-ben/apache-storm-0.9.3-ben/conf This file contain this configuration : ### These MUST be filled in for a storm configuration storm.zookeeper.servers: - paradent-4 # - paradent-47 # - paradent-48 # nimbus.host: paradent-4 # # # # These may optionally be filled in: # ## List of custom serializations # topology.kryo.register: # - org.mycompany.MyType # - org.mycompany.MyType2: org.mycompany.MyType2Serializer # ## List of custom kryo decorators # topology.kryo.decorators: # - org.mycompany.MyDecorator # ## Locations of the drpc servers # drpc.servers: # - server1 # - server2 ## Metrics Consumers # topology.metrics.consumer.register: # - class: backtype.storm.metric.LoggingMetricsConsumer # parallelism.hint: 1 # - class: org.mycompany.MyMetricsConsumer # parallelism.hint: 1 # argument: # - endpoint: metrics-collector.mycompany.org dev.zookeeper.path: paradent-4.rennes.grid5000.fr: ~/home/bsoulas/zookeeper/zookeeper-3.4.6/ storm.zookeeper.port: 2181 To launch storm on the cluster, i launch it thanks to *storm nimbus *(on a machine named paradent-4), then my zookeeper Server *sh zkServer.sh start* (on paradent-4 again)(which create a *zookeeper_server.pid* where the pid of the zookeeper is written, i know it's obvious ..._ ). After i launch my *storm ui* for having a visual of my storm app (on paradent-4). Until now, everything work fine. Now, the logical way implies i launch my supervisor, on a different machine (here *paradent-39*) thanks to *storm supervisor*, it is launched but once again, 3 or 4 seconds after it's down. So i watched the supervisor.log located : /home/bsoulas/incubator-storm-master/storm-dist/binary/target/apache-storm-0.9.3-ben/apache-storm-0.9.3-ben/logs And here appear a tricky error : 2014-09-02 09:31:37 o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting 2014-09-02 09:31:37 o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=paradent-4:2181 sessionTimeout=2 watcher=org.apache.curator.ConnectionState@220df4c8 2014-09-02 09:31:37 o.a.z.ClientCnxn [INFO] Opening socket connection to server paradent-4.rennes.grid5000.fr/172.16.97.4:2181. Will not attempt to authenticate using SASL (unknown error) 2014-09-02 09:31:37 o.a.z.ClientCnxn [INFO] Socket connection established to
Re: REMOTE MODE STORM DEV
Pavan, It would be helpful if you can post your storm.yaml. Make sure user storm has permissions to your storm installation dir. and you used the same storm config on all your machines. -Harsha On Tue, Sep 2, 2014, at 06:53 AM, Supun Kamburugamuva wrote: Hi Pavan, It seems you have a permission issue. Please check weather the directories that the storm user has appropriate permissions on the directories that contain the storm jars. Thanks, Supun.. On Tue, Sep 2, 2014 at 9:13 AM, Pavan Jakati G [1]pava...@microland.com wrote: Can anybody help us get rid of below error : sudo -u storm /usr/java/jdk1.7.0_65/bin/java -server -Xmx768m -Djava.library.path=storm-local/supervisor/stormdist/PaVan-14-1 409661199/resources/Linux-amd64:storm-local/supervisor/stormdis t/PaVan-14-1409661199/resources:/usr/local/lib:/opt/local/lib:/ usr/lib -Dlogfile.name=worker-6703.log -Dstorm.home=/root/apache-storm-0.9.2-incubating -Dlogback.configurationFile=/root/apache-storm-0.9.2-incubating /logback/cluster.xml -Dstorm.id=PaVan-14-1409661199 -Dworker.id=156a8af9-fa3b-4772-b91c-787490fe0b34 -Dworker.port=6703 -cp /root/apache-storm-0.9.2-incubating/lib/json-simple-1.1.jar:/ro ot/apache-storm-0.9.2-incubating/lib/core.incubator-0.1.0.jar:/ root/apache-storm-0.9.2-incubating/lib/asm-4.0.jar:/root/apache -storm-0.9.2-incubating/lib/commons-io-2.4.jar:/root/apache-sto rm-0.9.2-incubating/lib/httpclient-4.3.3.jar:/root/apache-storm -0.9.2-incubating/lib/jline-2.11.jar:/root/apache-storm-0.9.2-i ncubating/lib/tools.logging-0.2.3.jar:/root/apache-storm-0.9.2- incubating/lib/logback-classic-1.0.6.jar:/root/apache-storm-0.9 .2-incubating/lib/commons-logging-1.1.3.jar:/root/apache-storm- 0.9.2-incubating/lib/ring-core-1.1.5.jar:/root/apache-storm-0.9 .2-incubating/lib/ring-devel-0.3.11.jar:/root/apache-storm-0.9. 2-incubating/lib/curator-client-2.4.0.jar:/root/apache-storm-0. 9.2-incubating/lib/clj-stacktrace-0.2.4.jar:/root/apache-storm- 0.9.2-incubating/lib/clj-time-0.4.1.jar:/root/apache-storm-0.9. 2-incubating/lib/commons-lang-2.5.jar:/root/apache-storm-0.9.2- incubating/lib/zookeeper-3.4.5.jar:/root/apache-storm-0.9.2-inc ubating/lib/compojure-1.1.3.jar:/root/apache-storm-0.9.2-incuba ting/lib/joda-time-2.0.jar:/root/apache-storm-0.9.2-incubating/ lib/chill-java-0.3.5.jar:/root/apache-storm-0.9.2-incubating/li b/clout-1.0.1.jar:/root/apache-storm-0.9.2-incubating/lib/kryo- 2.21.jar:/root/apache-storm-0.9.2-incubating/lib/snakeyaml-1.11 .jar:/root/apache-storm-0.9.2-incubating/lib/minlog-1.2.jar:/ro ot/apache-storm-0.9.2-incubating/lib/storm-core-0.9.2-incubatin g.jar:/root/apache-storm-0.9.2-incubating/lib/jgrapht-core-0.9. 0.jar:/root/apache-storm-0.9.2-incubating/lib/slf4j-api-1.6.5.j ar:/root/apache-storm-0.9.2-incubating/lib/hiccup-0.3.6.jar:/ro ot/apache-storm-0.9.2-incubating/lib/netty-3.6.3.Final.jar:/roo t/apache-storm-0.9.2-incubating/lib/curator-framework-2.4.0.jar :/root/apache-storm-0.9.2-incubating/lib/guava-13.0.jar:/root/a pache-storm-0.9.2-incubating/lib/log4j-over-slf4j-1.6.6.jar:/ro ot/apache-storm-0.9.2-incubating/lib/commons-fileupload-1.2.1.j ar:/root/apache-storm-0.9.2-incubating/lib/servlet-api-2.5.jar: /root/apache-storm-0.9.2-incubating/lib/reflectasm-1.07-shaded. jar:/root/apache-storm-0.9.2-incubating/lib/jetty-util-6.1.26.j ar:/root/apache-storm-0.9.2-incubating/lib/objenesis-1.2.jar:/r oot/apache-storm-0.9.2-incubating/lib/tools.cli-0.2.4.jar:/root /apache-storm-0.9.2-incubating/lib/ring-jetty-adapter-0.3.11.ja r:/root/apache-storm-0.9.2-incubating/lib/commons-codec-1.6.jar :/root/apache-storm-0.9.2-incubating/lib/clojure-1.5.1.jar:/roo t/apache-storm-0.9.2-incubating/lib/netty-3.2.2.Final.jar:/root /apache-storm-0.9.2-incubating/lib/math.numeric-tower-0.0.1.jar :/root/apache-storm-0.9.2-incubating/lib/carbonite-1.4.0.jar:/r oot/apache-storm-0.9.2-incubating/lib/disruptor-2.10.1.jar:/roo t/apache-storm-0.9.2-incubating/lib/commons-exec-1.1.jar:/root/ apache-storm-0.9.2-incubating/lib/tools.macro-0.1.0.jar:/root/a pache-storm-0.9.2-incubating/lib/jetty-6.1.26.jar:/root/apache- storm-0.9.2-incubating/lib/httpcore-4.3.2.jar:/root/apache-stor m-0.9.2-incubating/lib/servlet-api-2.5-20081211.jar:/root/apach e-storm-0.9.2-incubating/lib/ring-servlet-0.3.11.jar:/root/apac he-storm-0.9.2-incubating/lib/logback-core-1.0.6.jar:/root/apac he-storm-0.9.2-incubating/conf:storm-local/supervisor/stormdist /PaVan-14-1409661199/stormjar.jar backtype.storm.daemon.worker PaVan-14-1409661199 0574446a-f73d-42b0-bcc7-e6dd449cb75a 6703 156a8af9-fa3b-4772-b91c-787490fe0b34 Error: Could not find or load main class backtype.storm.daemon.worker Regards, PaVan… From: Pavan Jakati G Sent: 02 September 2014 15:23 To: [2]user@storm.incubator.apache.org Subject: RE: REMOTE MODE STORM DEV sudo -u storm '/usr/java/jdk1.7.0_65/bin/java' '-server' '-Xmx768m' '-Djava.library.path=storm-local/sup ervisor/stormdist/PaVan-10-1409648607/resources/Linux-amd64:sto
storm use case questions
Hi guys, I have a stream of sensor data coming from rabbitmq. For each sensor message, it is of the JSON format and have the following fields: deviceId: BOT-N3 reading0: 2.25 reading1: 3.78 readingN: -1.35 each float number of readingN represents a sensor reading on a specific field location. Now for each incoming message, I want to do a query which gives me the average and standard deviation of a certain 'deviceId' 's 'readingN' over a custom time range (a year ago to now, a month ago to now, etc). So if N=28, for each incoming message I will need to do 28 queries on the historic data at almost the same time. I need the query results to be returned in near real time so the other incoming messages won't get blocked. Is STORM a good solution to this issue? I have tried Elasticsearch-Logstash-Kibana stack already, It seems that when the incoming message rates are high, The messages will be blocked since the ES server can't correspond to hundreds of query requesst at the same time. Will STORM help me in this case? What is the common use case of STORM in processing real-time sensor data (coming from sensor network specifically)? Thanks! best Yuheng
Please fix the code samples in the documentation
Hi, To the Storm-developers: Please fix the code samples in the documentation, because currently every single one is unformatted, without syntax highlighting and in one row. Thanks in advance, Andras
Re: Please fix the code samples in the documentation
I think this has been pointed out before. It is being tracked: https://issues.apache.org/jira/browse/STORM-385 -- Derek On 9/2/14, 15:34, Andras Hatvani wrote: Hi, To the Storm-developers: Please fix the code samples in the documentation, because currently every single one is unformatted, without syntax highlighting and in one row. Thanks in advance, Andras
Re: Please fix the code samples in the documentation
The Clojure DSL samples are merely a subset, but relevant, of course. Andras On 03 Sep 2014, at 00:28, Derek Dagit der...@yahoo-inc.com wrote: I think this has been pointed out before. It is being tracked: https://issues.apache.org/jira/browse/STORM-385 -- Derek On 9/2/14, 15:34, Andras Hatvani wrote: Hi, To the Storm-developers: Please fix the code samples in the documentation, because currently every single one is unformatted, without syntax highlighting and in one row. Thanks in advance, Andras
Re: Please fix the code samples in the documentation
Hi Andras, There are efforts underway to update and improve the Storm documentation. Contributions are always welcome if you'd like to get involved. -Taylor On Sep 2, 2014, at 7:22 PM, Andras Hatvani andras.hatv...@andrashatvani.com wrote: The Clojure DSL samples are merely a subset, but relevant, of course. Andras On 03 Sep 2014, at 00:28, Derek Dagit der...@yahoo-inc.com wrote: I think this has been pointed out before. It is being tracked: https://issues.apache.org/jira/browse/STORM-385 -- Derek On 9/2/14, 15:34, Andras Hatvani wrote: Hi, To the Storm-developers: Please fix the code samples in the documentation, because currently every single one is unformatted, without syntax highlighting and in one row. Thanks in advance, Andras
Re: KafkaSpout: java.lang.IllegalArgumentException: message does not exist
The solution:
The keys and values will be serialized before passing to another component. In
this case the Kafka bolt received both the key (=message) and the value
(=the message itself) as byte arrays, therefore the key message couldn't be
found. Only after deserializing with the help of a scheme works the bolt as
expected.
// …further config of spoutConfig
spoutConfig.scheme = new KeyValueSchemeAsMultiScheme(new
KafkaBoltKeyValueScheme());
KafkaSpout kafkaSpout = new KafkaSpout(spoutConfig);
public class KafkaBoltKeyValueScheme extends StringKeyValueScheme {
@Override
public Fields getOutputFields() {
return new Fields(message);
}
}
Cheers,
Andras
On 02 Sep 2014, at 12:30, Andras Hatvani andras.hatv...@andrashatvani.com
wrote:
Hi,
I've got an exception when trying to use a topology with a single Kafka spout
and a single Kafka bolt.
Code:
public class SmokeTestTopology {
public static final String INPUT_TOPIC = inputTopic;
public static final String ZK_ROOT_PATH = /smokeTest;
public static final String ZK_INPUT_ID = inputId;
public static final String OUTPUT_TOPIC = outputTopic;
public static StormTopology buildTopology() {
KafkaSpout kafkaSpout = prepareKafkaSpout();
KafkaBolt kafkaBolt = prepareKafkaBolt();
TopologyBuilder topologyBuilder = new TopologyBuilder();
topologyBuilder.setSpout(spout, kafkaSpout);
topologyBuilder.setBolt(bolt, kafkaBolt).shuffleGrouping(spout);
return topologyBuilder.createTopology();
}
private static KafkaBolt prepareKafkaBolt() {
return new KafkaBolt();
}
private static KafkaSpout prepareKafkaSpout() {
SpoutConfig spoutConfig = new SpoutConfig(
new ZkHosts(192.168.59.103:2181),
INPUT_TOPIC,
ZK_ROOT_PATH,
ZK_INPUT_ID);
spoutConfig.forceFromStart = true;
return new KafkaSpout(spoutConfig);
}
}
Exception:
2014-09-02 10:29:59 b.s.d.executor [ERROR]
java.lang.RuntimeException: java.lang.IllegalArgumentException: message does
not exist
at
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.daemon.executor$fn__5641$fn__5653$fn__5700.invoke(executor.clj:746)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_20]
Caused by: java.lang.IllegalArgumentException: message does not exist
at backtype.storm.tuple.Fields.fieldIndex(Fields.java:78)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.tuple.TupleImpl.fieldIndex(TupleImpl.java:100)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.tuple.TupleImpl.getValueByField(TupleImpl.java:149)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at storm.kafka.bolt.KafkaBolt.execute(KafkaBolt.java:75) ~[stormjar.jar:na]
at
backtype.storm.daemon.executor$fn__5641$tuple_action_fn__5643.invoke(executor.clj:631)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.daemon.executor$mk_task_receiver$fn__5564.invoke(executor.clj:399)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.disruptor$clojure_handler$reify__745.onEvent(disruptor.clj:58)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
… 6 common frames omitted
Anyone with similar issue?
Thanks,
Andras
Issues with Topology with Kafka Spout
Hi, I am not sure if this mailing list would be the correct place for this, however, I decided to ask here assuming many of storm cluster installations involve Kafka as their spout. I have set following properties for Kafka Spout: kafkaConfig.bufferSizeBytes = 1024 * 1024 * 4; kafkaConfig.fetchSizeBytes = 1024 * 1024 * 4; kafkaConfig.forceFromStart = true|false; (tried both, true and false) Kafka server is started with default properties except the log retention period being 15 minutes. And Storm configuration is as mentioned the Michael Noll's blog http://www.michael-noll.com/blog/2013/06/21/understanding-storm-internal-message-buffers/ conf.put(Config.TOPOLOGY_RECEIVER_BUFFER_SIZE, 8); conf.put(Config. TOPOLOGY_TRANSFER_BUFFER_SIZE, 32); conf.put(Config. TOPOLOGY_EXECUTOR_RECEIVE_BUFFER_SIZE, 16384); conf.put(Config. TOPOLOGY_EXECUTOR_SEND_BUFFER_SIZE, 16384); topology.max.spout.pending = 1 I am using Hortonworks distribution for installing Hadoop ecosystem. We are consuming twitter stream and pushing the tweets to a Kafka topic and then Storm topology is trying to consume those tweets using KafkaSpout with configuration described above. We are using twitter filter stream and we have many filter keywords so the input flux is quite high (not high as with firehose but still very high) and varies quite a lot depending on time of the day and any of the keywords, used as track filter, being viral on a particular day. Now I am facing 3 major issues with my topology (which contains 3 bolts after the kafka spout) *1)* Too many failed messages at spout. I assumed that initially when topology starts, because of initialization latency, there might be few thousands of messages which fail, however, it seems that this behavior is not limited to initialization and messages fails quite often and very rarely I am seeing that there is no failed message in last 10 minutes. :) *2)* After a while Kafka spout begins to throw *Got fetch request with offset out of range* error message continuously and never picks any message from the kafka topic while the stream collector is still able to push the messages to the topic. *3)* Every time I submit my topology, it takes more than 10 minutes to reach messages to the first bolt. First spout tries to accumulate message (which too many failed messages) for first few minutes (10 mins or so) and then each bolt start accumulating messages sequentially and after 15-20 min, every bolt in the topology has some messages to process. I am not able to understand why a message that has been processed by spout, is not delivered to next bolt immediately. I guess the message buffers as described in Michael Noll's blog are responsible for this but still changing the buffers didn't make any change in behavior. -- Regards, Vikas Agarwal 91 – 9928301411 InfoObjects, Inc. Execution Matters http://www.infoobjects.com 2041 Mission College Boulevard, #280 Santa Clara, CA 95054 +1 (408) 988-2000 Work +1 (408) 716-2726 Fax
