Github user HeartSaVioR commented on the pull request:
https://github.com/apache/storm/pull/797#issuecomment-150714706
@kishorvpatil @binhnv @revans2
Here's my storm.yaml.
```
worker.childopts: "-Dfile.encoding=UTF-8 -Xmx768m
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/daum/logs/storm/heapdump.hprof
-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=1%ID%
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false"
topology.backpressure.enable: false
topology.transfer.buffer.size: 32
topology.executor.send.buffer.size: 16384
topology.executor.receive.buffer.size: 16384
```
I used topology.backpressure.enable, topology.transfer.buffer.size,
topology.executor.send.buffer.size, topology.executor.receive.buffer.size as
default but no luck.
I changed these values as
http://www.michael-noll.com/blog/2013/06/21/understanding-storm-internal-message-buffers
but also no luck.
Failed tuples appear within 3 minutes, which is same as we were seen from
#134 (origin STORM-350).
I'm using VMs (IaaS on my office), but without Disruptor 3.3.2 it doesn't
show failed tuples.
I ran perftest on command line.
> storm jar
storm_perf_test-1.0.0-SNAPSHOT-jar-for-0.10.0-rc-with-dependencies.jar
com.yahoo.storm.perftest.Main --ack --ackers 3 --bolt 4 --name test -l 1 -n 1
--workers 3 --spout 3 --testTimeSec 900 -c topology.max.spout.pending=1092
--messageSize 10
Here's log.
```
1034 [main] INFO c.y.s.p.Main - Adding in 3 spouts
1081 [main] INFO c.y.s.p.Main - Adding in 4 bolts
1231 [main] INFO b.s.StormSubmitter - Generated ZooKeeper secret payload
for MD5-digest: -6427640641721598663:-8827787475099997913
1236 [main] INFO b.s.s.a.AuthUtils - Got AutoCreds []
1297 [main] INFO b.s.StormSubmitter - Uploading topology jar
storm_perf_test-1.0.0-SNAPSHOT-jar-for-0.10.0-rc-with-dependencies.jar to
assigned location:
/data1/storm/nimbus/inbox/stormjar-38afac0d-a0d8-49a6-8529-7705d89280e9.jar
1326 [main] INFO b.s.StormSubmitter - Successfully uploaded topology jar
to assigned location:
/data1/storm/nimbus/inbox/stormjar-38afac0d-a0d8-49a6-8529-7705d89280e9.jar
1326 [main] INFO b.s.StormSubmitter - Submitting topology test_0 in
distributed mode with conf
{"storm.zookeeper.topology.auth.scheme":"digest","storm.zookeeper.topology.auth.payload":"-6427640641721598663:-8827787475099997913","topology.workers":3,"topology.acker.executors":3,"topology.debug":false,"topology.max.spout.pending":1092}
1520 [main] INFO b.s.StormSubmitter - Finished submitting topology: test_0
status topologies totalSlots slotsUsed totalExecutors
executorsWithMetrics time time-diff ms transferred throughput
(MB/s)
WAITING 1 12 0 13 0 1445640974611 0
0 0.0
WAITING 1 12 3 13 13 1445641004611 30000
72600 0.02307891845703125
WAITING 1 12 3 13 13 1445641034612 30001
608340 0.1933796318931296
RUNNING 1 12 3 13 13 1445641064611 29999
783820 0.24917797349429877
RUNNING 1 12 3 13 13 1445641094611 30000
200520 0.06374359130859375
RUNNING 1 12 3 13 13 1445641124611 30000
17140 0.005448659261067708
RUNNING 1 12 3 13 13 1445641154612 30001
0 0.0
RUNNING 1 12 3 13 13 1445641184611 29999
17980 0.0057158785989480905
RUNNING 1 12 3 13 13 1445641214611 30000
0 0.0
RUNNING 1 12 3 13 13 1445641244611 30000
18180 0.005779266357421875
RUNNING 1 12 3 13 13 1445641274611 30000
0 0.0
RUNNING 1 12 3 13 13 1445641304611 30000
17620 0.005601247151692708
RUNNING 1 12 3 13 13 1445641334611 30000
0 0.0
RUNNING 1 12 3 13 13 1445641364611 30000
16800 0.005340576171875
RUNNING 1 12 3 13 13 1445641394611 30000
0 0.0
RUNNING 1 12 3 13 13 1445641424611 30000
17500 0.005563100179036458
RUNNING 1 12 3 13 13 1445641454611 30000
0 0.0
RUNNING 1 12 3 13 13 1445641484611 30000
17840 0.005671183268229167
RUNNING 1 12 3 13 13 1445641514611 30000
0 0.0
RUNNING 1 12 3 13 13 1445641544611 30000
18260 0.005804697672526042
RUNNING 1 12 3 13 13 1445641574611 30000
0 0.0
RUNNING 1 12 3 13 13 1445641604612 30001
17980 0.0057154975530763555
RUNNING 1 12 3 13 13 1445641634611 29999
0 0.0
RUNNING 1 12 3 13 13 1445641664611 30000
18000 0.0057220458984375
RUNNING 1 12 3 13 13 1445641694611 30000
0 0.0
RUNNING 1 12 3 13 13 1445641724611 30000
17720 0.005633036295572917
RUNNING 1 12 3 13 13 1445641754611 30000
0 0.0
RUNNING 1 12 3 13 13 1445641784611 30000
18720 0.005950927734375
RUNNING 1 12 3 13 13 1445641814611 30000
0 0.0
RUNNING 1 12 3 13 13 1445641844612 30001
18640 0.005925298909307189
RUNNING 1 12 3 13 13 1445641874611 29999
0 0.0
RUNNING 1 12 3 13 13 1445641904612 30001
18160 0.005772716104775674
RUNNING 1 12 3 13 13 1445641934611 29999
0 0.0
```
It nearly stops functioning but I couldn't find any warning / error from
worker logs.
It may be nice if we can share perftest jar / binary dist file which
applies STORM-350 (with Disruptor 3.3.2) / binary dist file which doesn't apply
STORM-350 (with Disruptor 2.10.4).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
