Github user knusbaum commented on the pull request:
https://github.com/apache/storm/pull/694#issuecomment-134412604
@mjsax, I'm not sure it's not related. Here is the output from my
benchmarks:
Pre-batching (apache master)
```
status time time-diff ms transferred throughput
(MB/s)
WAITING 1440456969132 0 0 0.0
WAITING 1440456999132 30000 99560
0.031649271647135414
WAITING 1440457029131 29999 2073460
0.6591571546037212
RUNNING 1440457059131 30000 2043680
0.6496683756510416
RUNNING 1440457089132 30001 2000600
0.6359524140536461
RUNNING 1440457119133 30001 1921980
0.6109606221947549
RUNNING 1440457149131 29998 1846040
0.5868794369819967
RUNNING 1440457179132 30001 1847640
0.5873293603429365
RUNNING 1440457209133 30001 1688440
0.5367227301733171
RUNNING 1440457239131 29998 1733740
0.5511778482986105
RUNNING 1440457269131 30000 1751700
0.5568504333496094
RUNNING 1440457299133 30002 1748280
0.5557261962158252
RUNNING 1440457329132 29999 1706500 0.54249982364321
RUNNING 1440457359133 30001 1787720
0.568281940243919
RUNNING 1440457389131 29998 1717900
0.5461421121922451
RUNNING 1440457419132 30001 1779180
0.5655672378466291
RUNNING 1440457449132 30000 1639820
0.5212847391764323
RUNNING 1440457479131 29999 1743720
0.5543321374058823
RUNNING 1440457509134 30003 1681140
0.5343665767700574
RUNNING 1440457539133 29999 1716760
0.5457614985278155
RUNNING 1440457569131 29998 1700800
0.5407058061683279
RUNNING 1440457599132 30001 1754100
0.5575947863098574
RUNNING 1440457629134 30002 1669220 0.53059537445225
RUNNING 1440457659132 29998 1757240
0.5586487951735845
RUNNING 1440457689131 29999 1781520
0.5663488343491658
RUNNING 1440457719131 30000 1772920
0.5635960896809896
RUNNING 1440457749134 30003 1656020
0.5263819422907969
RUNNING 1440457779135 30001 1557820
0.49520113449017844
RUNNING 1440457809134 29999 1750760
0.5565701677360599
RUNNING 1440457839131 29997 1773740
0.5639131519760049
RUNNING 1440457869132 30001 1656360
0.5265251127371275
RUNNING 1440457899131 29999 1770140
0.5627311091847593
RUNNING 1440457929131 30000 1767480
0.5618667602539062
```
With batching, no typehints. (This pull request as of now)
```
status time time-diff ms transferred throughput (MB/s)
WAITING 1440455622424 0 0 0.0
WAITING 1440455652425 30001 87340 0.027763712807880363
WAITING 1440455682427 30002 652540 0.2074230512724933
RUNNING 1440455712428 30001 678920 0.21581566177611788
RUNNING 1440455742425 29997 729960 0.23207124179214797
RUNNING 1440455772425 30000 667320 0.21213531494140625
RUNNING 1440455802427 30002 670380 0.2130938564870415
RUNNING 1440455832423 29996 616080 0.19587267397371733
RUNNING 1440455862423 30000 645540 0.20521163940429688
RUNNING 1440455892428 30005 635080 0.2018528528122917
RUNNING 1440455922427 29999 622760 0.1979766716507738
RUNNING 1440455952428 30001 610080 0.19393274455955634
RUNNING 1440455982424 29996 633800 0.20150646144095255
RUNNING 1440456012423 29999 573820 0.18241854603161253
RUNNING 1440456042427 30004 596680 0.18965417648089627
RUNNING 1440456072423 29996 592880 0.18849660911819494
RUNNING 1440456102425 30002 558000 0.17737159807835728
RUNNING 1440456132424 29999 614140 0.19523635610444828
RUNNING 1440456162423 29999 580520 0.18454848966970774
RUNNING 1440456192423 30000 585380 0.18608729044596353
RUNNING 1440456222425 30002 568560 0.18072830789145308
RUNNING 1440456252423 29998 598440 0.19025163607912401
RUNNING 1440456282426 30003 560560 0.1781794069941964
RUNNING 1440456312423 29997 579480 0.1842301539724285
RUNNING 1440456342424 30001 589860 0.18750519391866624
RUNNING 1440456372428 30004 569560 0.18103411000278088
RUNNING 1440456402423 29995 536220 0.17048816200812114
RUNNING 1440456432427 30004 607340 0.19304244744906407
RUNNING 1440456462425 29998 563460 0.17913105217756703
RUNNING 1440456492427 30002 588020 0.18691406290687393
RUNNING 1440456522423 29996 594640 0.18905617265895872
RUNNING 1440456552423 30000 594260 0.18891016642252603
RUNNING 1440456582426 30003 582340 0.18510239023298192
RUNNING 1440456612425 29999 554180 0.17617495005367367
```
With batching, typehints added:
```
status time time-diff ms transferred throughput
(MB/s)
WAITING 1440453725066 0 0 0.0
WAITING 1440453755065 29999 78920
0.025088828644548568
WAITING 1440453785067 30002 2003120
0.6367322500765574
RUNNING 1440453815066 29999 1940420
0.6168634677959318
RUNNING 1440453845065 29999 2073440
0.6591507965630105
RUNNING 1440453875068 30003 1952840
0.6207290444458158
RUNNING 1440453905067 29999 1751360
0.556760908957382
RUNNING 1440453935068 30001 1823440
0.5796366439478059
RUNNING 1440453965065 29997 1835720
0.5836180338411445
RUNNING 1440453995070 30005 1776660
0.5646908885140237
RUNNING 1440454025066 29996 1599600
0.5085669544350705
RUNNING 1440454055065 29999 1566060
0.4978536617724497
RUNNING 1440454085068 30003 1632100
0.5187787393949407
RUNNING 1440454115065 29997 1619780
0.5149656913119698
RUNNING 1440454145066 30001 1665920
0.5295640535940468
RUNNING 1440454175068 30002 1697340
0.5395338858106075
RUNNING 1440454205065 29997 1705360
0.542173561431664
RUNNING 1440454235067 30002 1628760
0.5177343442403319
RUNNING 1440454265066 29999 1721040
0.5471221192399122
RUNNING 1440454295070 30004 1731960
0.5505018561001762
RUNNING 1440454325067 29997 1648780
0.5241854650146004
RUNNING 1440454355065 29998 1665100
0.529356325171027
RUNNING 1440454385065 30000 1573460
0.500189463297526
RUNNING 1440454415068 30003 1700960
0.5406665550892827
RUNNING 1440454445066 29998 1744260
0.554522289197535
RUNNING 1440454475065 29999 1580540
0.5024568832470198
RUNNING 1440454505068 30003 1695720
0.5390009705084179
RUNNING 1440454535066 29998 1746140
0.5551199649475329
RUNNING 1440454565065 29999 1685440
0.5358048067748091
RUNNING 1440454595066 30001 1738360
0.5525913418445947
RUNNING 1440454625069 30003 1431160
0.4549080207539142
RUNNING 1440454655065 29996 1421760
0.45202560211153153
RUNNING 1440454685070 30005 1403440
0.4460672163363398
```
With the type-hints in place, the throughput goes up to very near
apache-master levels.
These were gathered with the command:
`storm jar storm_perf_test-1.0.0-SNAPSHOT-jar-with-dependencies.jar
com.yahoo.storm.perftest.Main --ack --bolt 4 --name test -l 1 -n 1 --workers 4
--spout 3 --testTimeSec 900 -c topology.max.spout.pending=1092 --messageSize 10`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
