What I can see in the screen after submit the topology: SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/etc/apache-storm-0.9.2-incubating/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/stuser/kafkaprj/kafka-storm-ingress/target/kafka-storm-ingress-0.0.1-SNAPSHOT-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. Running: /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java -client -Dstorm.options= -Dstorm.home=/etc/apache-storm-0.9.2-incubating -Djava.library.path=/usr/lib/jvm/java-7-openjdk-amd64 -Dstorm.conf.file= -cp /etc/apache-storm-0.9.2-incubating/lib/log4j-over-slf4j-1.6.6.jar:/etc/apache-storm-0.9.2-incubating/lib/joda-time-2.0.jar:/etc/apache-storm-0.9.2-incubating/lib/commons-codec-1.6.jar:/etc/apache-storm-0.9.2-incubating/lib/curator-framework-2.4.0.jar:/etc/apache-storm-0.9.2-incubating/lib/servlet-api-2.5.jar:/etc/apache-storm-0.9.2-incubating/lib/core.incubator-0.1.0.jar:/etc/apache-storm-0.9.2-incubating/lib/jetty-6.1.26.jar:/etc/apache-storm-0.9.2-incubating/lib/httpcore-4.3.2.jar:/etc/apache-storm-0.9.2-incubating/lib/servlet-api-2.5-20081211.jar:/etc/apache-storm-0.9.2-incubating/lib/commons-exec-1.1.jar:/etc/apache-storm-0.9.2-incubating/lib/logback-classic-1.0.6.jar:/etc/apache-storm-0.9.2-incubating/lib/minlog-1.2.jar:/etc/apache-storm-0.9.2-incubating/lib/asm-4.0.jar:/etc/apache-storm-0.9.2-incubating/lib/clojure-1.5.1.jar:/etc/apache-storm-0.9.2-incubating/lib/jline-2.11.jar:/etc/apache-storm-0.9.2-incubating/lib/clj-stacktrace-0.2.4.jar:/etc/apache-storm-0.9.2-incubating/lib/netty-3.2.2.Final.jar:/etc/apache-storm-0.9.2-incubating/lib/commons-fileupload-1.2.1.jar:/etc/apache-storm-0.9.2-incubating/lib/clout-1.0.1.jar:/etc/apache-storm-0.9.2-incubating/lib/curator-client-2.4.0.jar:/etc/apache-storm-0.9.2-incubating/lib/ring-servlet-0.3.11.jar:/etc/apache-storm-0.9.2-incubating/lib/commons-io-2.4.jar:/etc/apache-storm-0.9.2-incubating/lib/ring-devel-0.3.11.jar:/etc/apache-storm-0.9.2-incubating/lib/snakeyaml-1.11.jar:/etc/apache-storm-0.9.2-incubating/lib/reflectasm-1.07-shaded.jar:/etc/apache-storm-0.9.2-incubating/lib/chill-java-0.3.5.jar:/etc/apache-storm-0.9.2-incubating/lib/ring-jetty-adapter-0.3.11.jar:/etc/apache-storm-0.9.2-incubating/lib/compojure-1.1.3.jar:/etc/apache-storm-0.9.2-incubating/lib/objenesis-1.2.jar:/etc/apache-storm-0.9.2-incubating/lib/tools.macro-0.1.0.jar:/etc/apache-storm-0.9.2-incubating/lib/httpclient-4.3.3.jar:/etc/apache-storm-0.9.2-incubating/lib/json-simple-1.1.jar:/etc/apache-storm-0.9.2-incubating/lib/guava-13.0.jar:/etc/apache-storm-0.9.2-incubating/lib/commons-lang-2.5.jar:/etc/apache-storm-0.9.2-incubating/lib/storm-core-0.9.2-incubating.jar:/etc/apache-storm-0.9.2-incubating/lib/ring-core-1.1.5.jar:/etc/apache-storm-0.9.2-incubating/lib/hiccup-0.3.6.jar:/etc/apache-storm-0.9.2-incubating/lib/tools.logging-0.2.3.jar:/etc/apache-storm-0.9.2-incubating/lib/carbonite-1.4.0.jar:/etc/apache-storm-0.9.2-incubating/lib/math.numeric-tower-0.0.1.jar:/etc/apache-storm-0.9.2-incubating/lib/slf4j-api-1.6.5.jar:/etc/apache-storm-0.9.2-incubating/lib/tools.cli-0.2.4.jar:/etc/apache-storm-0.9.2-incubating/lib/netty-3.6.3.Final.jar:/etc/apache-storm-0.9.2-incubating/lib/disruptor-2.10.1.jar:/etc/apache-storm-0.9.2-incubating/lib/jetty-util-6.1.26.jar:/etc/apache-storm-0.9.2-incubating/lib/commons-logging-1.1.3.jar:/etc/apache-storm-0.9.2-incubating/lib/jgrapht-core-0.9.0.jar:/etc/apache-storm-0.9.2-incubating/lib/zookeeper-3.4.5.jar:/etc/apache-storm-0.9.2-incubating/lib/logback-core-1.0.6.jar:/etc/apache-storm-0.9.2-incubating/lib/clj-time-0.4.1.jar:/etc/apache-storm-0.9.2-incubating/lib/kryo-2.21.jar:target/kafka-storm-ingress-0.0.1-SNAPSHOT-jar-with-dependencies.jar:/root/.storm:/etc/apache-storm-0.9.2-incubating/bin -Dstorm.jar=target/kafka-storm-ingress-0.0.1-SNAPSHOT-jar-with-dependencies.jar storm.ingress.KafkaIngressTopology topictest SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/etc/apache-storm-0.9.2-incubating/lib/logback-classic-1.0.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/stuser/kafkaprj/kafka-storm-ingress/target/kafka-storm-ingress-0.0.1-SNAPSHOT-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. Storm cluster.... 487 [main] INFO backtype.storm.StormSubmitter - Jar not uploaded to master yet. Submitting jar... 491 [main] INFO backtype.storm.StormSubmitter - Uploading topology jar target/kafka-storm-ingress-0.0.1-SNAPSHOT-jar-with-dependencies.jar to assigned location: /app/storm/nimbus/inbox/stormjar-0364215c-c1e4-4405-b682-8b02cfea03ca.jar 639 [main] INFO backtype.storm.StormSubmitter - Successfully uploaded topology jar to assigned location: /app/storm/nimbus/inbox/stormjar-0364215c-c1e4-4405-b682-8b02cfea03ca.jar 639 [main] INFO backtype.storm.StormSubmitter - Submitting topology topictest in distributed mode with conf {"topology.workers":10} 746 [main] INFO backtype.storm.StormSubmitter - Finished submitting topology: topictest
Seems to be OK though ....., really can't diagnose what is the problem. Thanks Alec On Thu, Oct 30, 2014 at 4:04 PM, Sa Li <[email protected]> wrote: > Thanks, Bill, I turn on the debug mode, > by > Config conf = new Config(); > conf.setDebug(true); > > This is what I get from log: > root@DO-mq-dev:/etc/storm/logs# ll > total 708 > drwxr-xr-x 2 root root 4096 Aug 11 13:40 ./ > drwxr-xr-x 10 root root 4096 Aug 11 13:33 ../ > -rw-r--r-- 1 root root 0 Aug 11 13:33 access.log > -rw-r--r-- 1 root root 0 Aug 11 13:33 metrics.log > -rw-r--r-- 1 root root 407392 Oct 28 16:08 nimbus.log > -rw-r--r-- 1 root root 270708 Oct 28 15:45 supervisor.log > -rw-r--r-- 1 root root 19965 Oct 28 16:02 ui.log > root@DO-mq-dev:/etc/storm/logs# tail nimbus.log > at > org.apache.thrift7.ProcessFunction.process(ProcessFunction.java:32) > ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] > at > org.apache.thrift7.TBaseProcessor.process(TBaseProcessor.java:34) > ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] > at > org.apache.thrift7.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:632) > ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] > at > org.apache.thrift7.server.THsHaServer$Invocation.run(THsHaServer.java:201) > [storm-core-0.9.2-incubating.jar:0.9.2-incubating] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > [na:1.7.0_55] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_55] > at java.lang.Thread.run(Thread.java:744) [na:1.7.0_55] > 2014-10-28 16:08:38 o.a.z.ZooKeeper [INFO] Session: 0x14915a47cdde1f1 > closed > 2014-10-28 16:08:38 o.a.z.ClientCnxn [INFO] EventThread shut down > 2014-10-28 16:08:38 b.s.d.nimbus [INFO] Shut down master > > > This is not correct, since I submit many times topology to cluster, seems > didn't get recorded in log files. Any clues? > > thanks > > Alec > > > > > On Thu, Oct 30, 2014 at 4:20 AM, Brunner, Bill <[email protected]> > wrote: > >> Turn on debug mode and tail the log >> >> >> >> *From:* Sa Li [mailto:[email protected]] >> *Sent:* Wednesday, October 29, 2014 8:13 PM >> *To:* [email protected] >> *Subject:* Re: not writing data into DB in storm cluster, but does in >> localcluster >> >> >> >> I compile the code as >> >> mvn clean package -P cluster >> >> >> >> and run as >> >> >> >> storm jar >> target/kafka-storm-ingress-0.0.1-SNAPSHOT-jar-with-dependencies.jar >> storm.ingress.KafkaIngressTopology >> >> >> >> it running, but nothing populated into DB, how can debug it in cluster >> mode? >> >> >> >> thanks >> >> >> >> Alec >> >> >> >> On Wed, Oct 29, 2014 at 2:24 PM, Sa Li <[email protected]> wrote: >> >> Thanks for reply Bill, here how I submit it : >> >> >> >> if (args != null && args.length > 0) { >> >> System.out.println("local mode...."); >> >> cluster.submitTopology("topictest", conf, >> buildTridentKafkaTopology()); >> >> Thread.sleep(1500); >> >> >> >> //cluster.shutdown(); >> >> //drpc.shutdown(); >> >> } >> >> else { >> >> System.out.println("Storm cluster...."); >> >> conf.setNumWorkers(10); >> >> StormSubmitter.submitTopology("topictest", >> conf, buildTridentKafkaTopology()); >> >> } >> >> >> >> >> >> So I get the number of Workers as 10. >> >> >> >> >> >> Thanks >> >> >> >> Alec >> >> >> >> On Wed, Oct 29, 2014 at 2:16 PM, Brunner, Bill <[email protected]> >> wrote: >> >> Do you have least 1 Worker defined in your topology? >> >> >> >> *From:* Sa Li [mailto:[email protected]] >> *Sent:* Wednesday, October 29, 2014 5:08 PM >> *To:* [email protected] >> *Subject:* not writing data into DB in storm cluster, but does in >> localcluster >> >> >> >> Hi, All >> >> >> >> I am running a kafkaSpout to consume data from kafka and write data into >> postgresql DB, it works in localcluster even it is slow (we need to >> diagnose what the problem is). When I sumbitted it in storm cluster, it >> doesn't show exceptions, and I see the topology is alive in StormUI, but >> just no data being written into DB, why that happen? >> >> >> >> thanks >> >> >> >> Alec >> ------------------------------ >> >> This message, and any attachments, is for the intended recipient(s) only, >> may contain information that is privileged, confidential and/or proprietary >> and subject to important terms and conditions available at >> http://www.bankofamerica.com/emaildisclaimer. If you are not the >> intended recipient, please delete this message. >> >> >> >> >> ------------------------------ >> This message, and any attachments, is for the intended recipient(s) only, >> may contain information that is privileged, confidential and/or proprietary >> and subject to important terms and conditions available at >> http://www.bankofamerica.com/emaildisclaimer. If you are not the >> intended recipient, please delete this message. >> > >
