Hi Jack, Your setups are exactly the same as mine except for the zookeeper servers and nimbus host. Not sure what other ways you have tried to get the executor stats besides storm-benchmark to reproduce this issue. If you think this is a storm-benchmark issue, please open a issue at https://github.com/intel-hadoop/storm-benchmark/issues and let's continue our investigation there such that we no longer disturb people on this list.
Manu On Wed, Jul 22, 2015 at 2:05 AM, Jack <[email protected]> wrote: > Hi Manu, > > Can you share your system configuration to see if I can get the stats using > your setup? I have copy and pasted the output from SOL to show you what I > see. > > jack@node-1:/mnt/storm-benchmark$ bin/stormbench -storm > /mnt/storm/bin/storm -jar > ./target/storm-benchmark-0.1.0-jar-with-dependencies.jar -conf > ./conf/sol.yaml -c topology.workers=2 storm.benchmark.tools.Runner > storm.benchmark.benchmarks.SOL > Running: /mnt/jdk1.7.0_51/bin/java -client > -Dstorm.options=topology.workers%3D2 -Dstorm.home=/mnt/storm > -Dstorm.log.dir=/mnt/storm/logs > -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib > -Dstorm.conf.file=sol.yaml -cp > > /mnt/storm/storm-wordcount-benchmark-0.11.0-SNAPSHOT-jar-with-dependencies.jar:/mnt/storm/lib/jline-2.11.jar:/mnt/storm/lib/log4j-over-slf4j-1.6.6.jar:/mnt/storm/lib/slf4j-api-1.7.5.jar:/mnt/storm/lib/logback-core-1.0.13.jar:/mnt/storm/lib/logback-classic-1.0.13.jar:/mnt/storm/lib/jgrapht-core-0.9.0.jar:/mnt/storm/lib/disruptor-2.10.1.jar:/mnt/storm/lib/commons-logging-1.1.3.jar:/mnt/storm/lib/snakeyaml-1.11.jar:/mnt/storm/lib/chill-java-0.3.5.jar:/mnt/storm/lib/objenesis-1.2.jar:/mnt/storm/lib/minlog-1.2.jar:/mnt/storm/lib/asm-4.0.jar:/mnt/storm/lib/reflectasm-1.07-shaded.jar:/mnt/storm/lib/kryo-2.21.jar:/mnt/storm/lib/carbonite-1.4.0.jar:/mnt/storm/lib/json-simple-1.1.jar:/mnt/storm/lib/commons-lang-2.5.jar:/mnt/storm/lib/commons-exec-1.1.jar:/mnt/storm/lib/tools.cli-0.2.4.jar:/mnt/storm/lib/math.numeric-tower-0.0.1.jar:/mnt/storm/lib/tools.logging-0.2.3.jar:/mnt/storm/lib/jetty-util-6.1.26.jar:/mnt/storm/lib/jetty-6.1.26.jar:/mnt/storm/lib/ring-servlet-0.3.11.jar:/mnt/storm/lib/ring-jetty-adapter-0.3.11.jar:/mnt/storm/lib/clj-stacktrace-0.2.2.jar:/mnt/storm/lib/ring-devel-0.3.11.jar:/mnt/storm/lib/hiccup-0.3.6.jar:/mnt/storm/lib/servlet-api-2.5.jar:/mnt/storm/lib/commons-fileupload-1.2.1.jar:/mnt/storm/lib/commons-io-2.4.jar:/mnt/storm/lib/commons-codec-1.6.jar:/mnt/storm/lib/ring-core-1.1.5.jar:/mnt/storm/lib/clout-1.0.1.jar:/mnt/storm/lib/tools.macro-0.1.0.jar:/mnt/storm/lib/core.incubator-0.1.0.jar:/mnt/storm/lib/compojure-1.1.3.jar:/mnt/storm/lib/joda-time-2.0.jar:/mnt/storm/lib/clj-time-0.4.1.jar:/mnt/storm/lib/clojure-1.5.1.jar:/mnt/storm/lib/storm-core-0.9.5.jar:./target/storm-benchmark-0.1.0-jar-with-dependencies.jar:/mnt/storm-benchmark/conf:/mnt/storm/bin > -Dlog4j.configurationFile=/mnt/storm/log4j2.xml > -Dstorm.jar=./target/storm-benchmark-0.1.0-jar-with-dependencies.jar > storm.benchmark.tools.Runner storm.benchmark.benchmarks.SOL > 204 [main] INFO storm.benchmark.tools.Runner - running benchmark > storm.benchmark.benchmarks.SOL > 460 [main] INFO backtype.storm.StormSubmitter - Jar not uploaded to > master yet. Submitting jar... > 466 [main] INFO backtype.storm.StormSubmitter - Uploading topology jar > ./target/storm-benchmark-0.1.0-jar-with-dependencies.jar to assigned > location: > /tmp/storm/nimbus/inbox/stormjar-77bc1f1f-b621-4c3b-89d2-448caf7d70a7.jar > 598 [main] INFO backtype.storm.StormSubmitter - Successfully uploaded > topology jar to assigned location: > /tmp/storm/nimbus/inbox/stormjar-77bc1f1f-b621-4c3b-89d2-448caf7d70a7.jar > 598 [main] INFO backtype.storm.StormSubmitter - Submitting topology SOL > in distributed mode with conf > > {"drpc.worker.threads":64,"topology.state.synchronization.timeout.secs":60,"topology.executor.send.buffer.size":1024,"topology.worker.childopts":null,"storm.messaging.netty.max_retries":300,"topology.receiver.buffer.size":8,"topology.max.spout.pending":200,"storm.zookeeper.retry.intervalceiling.millis":30000,"metrics.enabled":true,"zmq.hwm":0,"topology.message.timeout.secs":30,"topology.max.task.parallelism":null,"topology.spout.wait.strategy":"backtype.storm.spout.SleepSpoutWaitStrategy","supervisor.slots.ports":[6700,6701,6702,6703],"topology.level":3,"storm.zookeeper.servers":["node-1"],"transactional.zookeeper.servers":null,"supervisor.childopts":"-Xmx256m","nimbus.host":"node-1","storm.cluster.mode":"distributed","ui.childopts":"-Xmx768m","component.spout_num":4,"topology.disruptor.wait.strategy":"com.lmax.disruptor.BlockingWaitStrategy","topology.enable.message.timeouts":true," > logviewer.appender.name > > ":"A1","storm.local.mode.zmq":false,"drpc.request.timeout.secs":600,"topology.multilang.serializer":"backtype.storm.multilang.JsonSerializer","storm.zookeeper.root":"\/storm","storm.thrift.transport":"backtype.storm.security.auth.SimpleTransportPlugin","topology.debug":false,"supervisor.enable":true,"nimbus.thrift.port":6627," > topology.sleep.spout.wait.strategy.time.ms > > ":1,"nimbus.reassign":true,"storm.zookeeper.connection.timeout":15000,"supervisor.heartbeat.frequency.secs":5,"topology.worker.shared.thread.pool.size":4,"zmq.threads":1,"topology.acker.executors":4," > topology.name > > ":"SOL","dev.zookeeper.path":"\/tmp\/dev-storm-zookeeper","transactional.zookeeper.root":"\/transactional","worker.heartbeat.frequency.secs":1,"nimbus.supervisor.timeout.secs":60,"nimbus.thrift.max_buffer_size":1048576,"storm.local.dir":"storm-local","task.refresh.poll.secs":10,"supervisor.worker.timeout.secs":30,"topology.tick.tuple.freq.secs":null,"storm.messaging.netty.client_worker_threads":1,"nimbus.file.copy.expiration.secs":600,"nimbus.task.timeout.secs":30,"topology.builtin.metrics.bucket.size.secs":60,"storm.messaging.transport":"backtype.storm.messaging.netty.Context","topology.transfer.buffer.size":1024,"drpc.childopts":"-Xmx768m","topology.workers":2,"supervisor.monitor.frequency.secs":3,"transactional.zookeeper.port":null,"storm.messaging.netty.server_worker_threads":1,"topology.max.error.report.per.interval":5,"storm.messaging.netty.transfer.batch.size":262144,"metrics.time":300000,"storm.messaging.netty.buffer_size":5242880,"storm.messaging.netty.min_wait_ms":100,"nimbus.childopts":"-Xmx1024m","topology.skip.missing.kryo.registrations":false,"topology.trident.batch.emit.interval.millis":500,"message.size":100,"worker.childopts":"-Xmx768m","topology.stats.sample.rate":0.05,"topology.environment":null,"storm.zookeeper.port":2181,"supervisor.worker.start.timeout.secs":120,"nimbus.cleanup.inbox.freq.secs":600,"topology.kryo.factory":"backtype.storm.serialization.DefaultKryoFactory","task.heartbeat.frequency.secs":3,"topology.fall.back.on.java.serialization":true,"topology.worker.receiver.thread.count":1,"logviewer.port":8000,"ui.port":8080,"storm.zookeeper.retry.times":5,"topology.tuple.serializer":"backtype.storm.serialization.types.ListDelegateSerializer","storm.messaging.netty.max_wait_ms":1000,"topology.error.throttle.interval.secs":10,"nimbus.task.launch.secs":120,"drpc.port":3772,"storm.zookeeper.session.timeout":20000,"drpc.invocations.port":3773,"logviewer.childopts":"-Xmx128m","java.library.path":"\/usr\/local\/lib:\/opt\/local\/lib:\/usr\/lib","storm.meta.serialization.delegate":"backtype.storm.serialization.DefaultSerializationDelegate","metrics.path":"reports"," > storm.messaging.netty.flush.check.interval.ms > > ":10,"zmq.linger.millis":5000,"nimbus.topology.validator":"backtype.storm.nimbus.DefaultTopologyValidator","nimbus.monitor.freq.secs":10,"metrics.poll":60000,"topology.executor.receive.buffer.size":1024,"topology.tasks":null,"storm.zookeeper.retry.interval":1000,"nimbus.inbox.jar.expiration.secs":3600,"topology.classpath":null,"drpc.queue.size":128,"component.bolt_num":8} > 895 [main] INFO backtype.storm.StormSubmitter - Finished submitting > topology: SOL > 911 [main] INFO storm.benchmark.metrics.MetricsCollectorConfig - writing > out storm config into .yaml file > 915 [main] INFO storm.benchmark.metrics.BasicMetricsCollector - writing > out metrics headers > [time(s),total_slots,used_slots,workers,tasks,executors,transferred > (messages),throughput (messages/s),throughput (MB/s),spout_throughput > (MB/s),spout_executors,spout_transferred (messages),spout_acked > (messages),spout_throughput > (messages/s),spout_avg_complete_latency(ms),spout_max_complete_latency(ms)] > into .csv file > 60979 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - executor > stats not found for component: bolt1 > 60979 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - executor > stats not found for component: bolt1 > 60979 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - executor > stats not found for component: bolt1 > 60979 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - executor > stats not found for component: bolt1 > 60980 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - executor > stats not found for component: bolt1 > 60980 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - executor > stats not found for component: bolt1 > 60980 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - executor > stats not found for component: bolt1 > 60980 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - executor > stats not found for component: bolt1 > 60980 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - executor > stats not found for component: bolt2 > 60980 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - executor > stats not found for component: bolt2 > 60980 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - executor > stats not found for component: bolt2 > 60980 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - executor > stats not found for component: bolt2 > 60980 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - executor > stats not found for component: bolt2 > 60980 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - executor > stats not found for component: bolt2 > 60980 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - executor > stats not found for component: bolt2 > 60980 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - executor > stats not found for component: bolt2 > 60980 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - executor > stats not found for component: spout > 60980 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - executor > stats not found for component: spout > 60980 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - executor > stats not found for component: spout > 60980 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - executor > stats not found for component: spout > 60981 [main] INFO storm.benchmark.metrics.BasicMetricsCollector - writing > out metrics results [60,16,2,2,24,24,0,0,0.0,0.000,0,0,0,0,0.0,0.0] into > .csv file > 121014 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - > executor stats not found for component: bolt1 > 121014 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - > executor stats not found for component: bolt1 > 121014 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - > executor stats not found for component: bolt1 > 121014 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - > executor stats not found for component: bolt1 > 121014 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - > executor stats not found for component: bolt1 > 121015 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - > executor stats not found for component: bolt1 > 121015 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - > executor stats not found for component: bolt1 > 121015 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - > executor stats not found for component: bolt1 > 121015 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - > executor stats not found for component: bolt2 > 121015 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - > executor stats not found for component: bolt2 > 121015 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - > executor stats not found for component: bolt2 > 121015 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - > executor stats not found for component: bolt2 > 121015 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - > executor stats not found for component: bolt2 > 121015 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - > executor stats not found for component: bolt2 > 121015 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - > executor stats not found for component: bolt2 > 121015 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - > executor stats not found for component: bolt2 > 121015 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - > executor stats not found for component: spout > 121015 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - > executor stats not found for component: spout > 121015 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - > executor stats not found for component: spout > 121015 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - > executor stats not found for component: spout > 121016 [main] INFO storm.benchmark.metrics.BasicMetricsCollector - writing > out metrics results [120,16,2,2,24,24,0,0,0.0,0.000,0,0,0,0,0.0,0.0] into > .csv file > 181047 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - > executor stats not found for component: bolt1 > 181047 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - > executor stats not found for component: bolt1 > 181047 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - > executor stats not found for component: bolt1 > 181047 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - > executor stats not found for component: bolt1 > 181047 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - > executor stats not found for component: bolt1 > 181047 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - > executor stats not found for component: bolt1 > 181047 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - > executor stats not found for component: bolt1 > 181047 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - > executor stats not found for component: bolt1 > 181047 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - > executor stats not found for component: bolt2 > 181047 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - > executor stats not found for component: bolt2 > 181047 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - > executor stats not found for component: bolt2 > 181047 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - > executor stats not found for component: bolt2 > 181047 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - > executor stats not found for component: bolt2 > 181047 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - > executor stats not found for component: bolt2 > 181047 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - > executor stats not found for component: bolt2 > 181047 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - > executor stats not found for component: bolt2 > 181048 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - > executor stats not found for component: spout > 181048 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - > executor stats not found for component: spout > 181048 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - > executor stats not found for component: spout > 181048 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - > executor stats not found for component: spout > 181048 [main] INFO storm.benchmark.metrics.BasicMetricsCollector - writing > out metrics results [180,16,2,2,24,24,0,0,0.0,0.000,0,0,0,0,0.0,0.0] into > .csv file > > Thanks for your help. > > Jack > > On Tue, Jul 21, 2015 at 5:34 AM, Manu Zhang <[email protected]> > wrote: > > > Hi Jack, > > > > I've been successfully getting metrics data running storm-benchmark > > wordcount on storm 0.9.5. > > Maybe you can file a jira and describe how to reproduce your issue. > > > > Manu > > > > On Tue, Jul 21, 2015 at 9:33 AM, Jack <[email protected]> wrote: > > > > > Hi Manu, > > > > > > I am running version storm 0.9.5 with your wordcount benchmark with 4 > > > workers, 4 spouts, 4 splits, 4 counts and 2 ack executor bolts. I set > the > > > polling interval to 5 seconds and total running time to 3 minutes. > > > > > > Jack > > > > > > Jack > > > > > > On Mon, Jul 20, 2015 at 6:09 PM, Manu Zhang <[email protected]> > > > wrote: > > > > > > > Jack, > > > > > > > > Which Storm version are you using ? And what's the setup of your > > > > configurations and topologies ? > > > > > > > > Manu > > > > > > > > On Tue, Jul 21, 2015 at 6:45 AM, Jack <[email protected]> > > wrote: > > > > > > > > > Hi Manu, > > > > > > > > > > Yes, there are executors running and the benchmark runs > successfully. > > > The > > > > > curious thing is I can see the executors stats for each spout/bolt > > if I > > > > > query the REST API directly, but prefer not to do this. > > > > > > > > > > Jack > > > > > > > > > > Jack > > > > > > > > > > On Mon, Jul 20, 2015 at 3:35 PM, Manu Zhang < > [email protected] > > > > > > > > wrote: > > > > > > > > > > > Hi Jack, > > > > > > > > > > > > Could you check if there are any executors up ? > > > > > > > > > > > > Thanks, > > > > > > Manu Zhang > > > > > > > > > > > > On Tue, Jul 21, 2015 at 1:30 AM, Jack <[email protected] > > > > > > wrote: > > > > > > > > > > > > > I'm trying to get ExecutorStats from ExecutorSummary but it > > always > > > > > > returns > > > > > > > null. Is there some sort of configuration I need to set before > I > > > can > > > > > get > > > > > > > those values? I also tried using > > > > > > > https://github.com/intel-hadoop/storm-benchmark which also > tries > > > to > > > > > > > retrieve executor stats but this benchmark also returns null > for > > > > those > > > > > > > values. Thanks > > > > > > > > > > > > > > > > > > > > > > > > > > > >
