Github user revans2 commented on the issue:
https://github.com/apache/storm/pull/2203
Here are my numbers, and no I didn't do it headless but I was not doing
anything on the laptop at the time it was running except looking at Activity
Monitor to be sure nothing else was using too much CPU.
The setup is
MacBook Pro (Retina, 15-inch, Mid 2015)
2.8 GHz Intel Core i7
16 GB 1600 MHz DDR3
macOS 10.12.6
java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
The versions tested are 1.2.0-SNAPSHOT
(b24f5d87cbcd4faeed651e45a8e673db52469a46)
Metrics V2 (78d3de44cf0705efd199fb64b7ef092e23c3285b) which is
1.2.0-SNAPSHOT + 99bcf68 merged in.
The Numbers:
1.2.0-SNAPSHOT
```
./bin/storm jar
../../../examples/storm-starter/target/storm-starter-1.2.0-SNAPSHOT.jar
org.apache.storm.starter.ThroughputVsLatency 50000 1 3 -c
topology.max.spout.pending=500
uptime: 30 acked: 602,660 acked/sec: 20,088.67 failed: 0 99%:
741,343,231 99.9%: 757,596,159 min: 3,639,296 max: 768,081,919
mean: 95,373,435.91 stddev: 191,784,964.85 user: 71,070 sys: 4,565
gc: 247 mem: 109.46
uptime: 60 acked: 1,504,480 acked/sec: 50,149.33 failed: 0 99%:
13,320,191 99.9%: 15,794,175 min: 3,467,264 max: 30,490,623
mean: 9,679,685.54 stddev: 1,394,068.49 user: 72,565 sys: 6,299
gc: 728 mem: 43.30
uptime: 90 acked: 1,510,500 acked/sec: 50,350.00 failed: 0 99%:
13,279,231 99.9%: 14,704,639 min: 3,352,576 max: 17,612,799
mean: 9,650,125.87 stddev: 1,358,879.30 user: 72,788 sys: 6,360
gc: 732 mem: 73.51
uptime: 120 acked: 1,503,720 acked/sec: 50,124.00 failed: 0 99%:
13,352,959 99.9%: 14,745,599 min: 3,356,672 max: 16,973,823
mean: 9,683,344.13 stddev: 1,361,582.42 user: 72,855 sys: 6,239
gc: 740 mem: 101.27
uptime: 150 acked: 1,654,820 acked/sec: 55,160.67 failed: 0 99%:
13,369,343 99.9%: 14,811,135 min: 3,473,408 max: 16,736,255
mean: 9,695,874.65 stddev: 1,363,167.65 user: 72,671 sys: 6,327
gc: 741 mem: 127.25
uptime: 180 acked: 1,503,740 acked/sec: 50,124.67 failed: 0 99%:
13,377,535 99.9%: 14,811,135 min: 3,448,832 max: 17,711,103
mean: 9,682,345.18 stddev: 1,370,041.64 user: 72,848 sys: 6,280
gc: 756 mem: 41.83
./bin/storm jar
../../../examples/storm-starter/target/storm-starter-1.2.0-SNAPSHOT.jar
org.apache.storm.starter.ThroughputVsLatency 75000 1 3 -c
topology.max.spout.pending=500
uptime: 30 acked: 882,420 acked/sec: 29,414.00 failed: 0 99%:
746,586,111 99.9%: 763,363,327 min: 3,678,208 max: 768,081,919
mean: 202,478,108.44 stddev: 266,947,569.24 user: 50,763 sys: 2,752
gc: 0 mem: 38.93
uptime: 60 acked: 2,497,720 acked/sec: 83,257.33 failed: 0 99%:
213,123,071 99.9%: 256,245,759 min: 3,672,064 max: 271,581,183
mean: 46,314,967.38 stddev: 51,559,780.69 user: 118,361 sys: 7,777
gc: 1,322 mem: 85.28
uptime: 90 acked: 2,260,200 acked/sec: 75,340.00 failed: 0 99%:
129,826,815 99.9%: 152,698,879 min: 3,659,776 max: 160,432,127
mean: 29,179,878.90 stddev: 29,586,943.44 user: 110,350 sys: 7,583
gc: 1,323 mem: 30.47
uptime: 120 acked: 2,255,680 acked/sec: 75,189.33 failed: 0 99%:
148,504,575 99.9%: 179,306,495 min: 3,700,736 max: 187,301,887
mean: 16,040,979.89 stddev: 23,589,666.52 user: 103,839 sys: 7,031
gc: 1,174 mem: 51.50
uptime: 150 acked: 2,255,180 acked/sec: 75,172.67 failed: 0 99%:
81,002,495 99.9%: 117,309,439 min: 3,571,712 max: 123,994,111
mean: 10,956,757.98 stddev: 11,465,433.32 user: 100,497 sys: 6,960
gc: 1,136 mem: 90.77
uptime: 180 acked: 2,256,380 acked/sec: 75,212.67 failed: 0 99%:
35,192,831 99.9%: 49,446,911 min: 3,608,576 max: 55,672,831
mean: 9,647,021.99 stddev: 4,081,135.34 user: 100,516 sys: 6,771
gc: 1,141 mem: 110.73
./bin/storm jar
../../../examples/storm-starter/target/storm-starter-1.2.0-SNAPSHOT.jar
org.apache.storm.starter.ThroughputVsLatency 100000 1 3 -c
topology.max.spout.pending=500
uptime: 30 acked: 1,023,620 acked/sec: 34,120.67 failed: 0 99%:
1,427,111,935 99.9%: 1,442,840,575 min: 141,033,472 max: 1,448,083,455
mean: 1,205,785,740.36 stddev: 167,013,014.66 user: 56,266 sys: 2,615
gc: 0 mem: 53.83
uptime: 60 acked: 2,820,000 acked/sec: 94,000.00 failed: 0 99%:
4,177,526,783 99.9%: 4,320,133,119 min: 1,091,567,616 max: 4,353,687,551
mean: 2,617,160,518.91 stddev: 618,922,278.84 user: 127,359 sys: 7,415
gc: 1,565 mem: 88.42
uptime: 90 acked: 2,665,280 acked/sec: 88,842.67 failed: 0 99%:
7,541,358,591 99.9%: 7,918,845,951 min: 2,944,401,408 max: 7,948,206,079
mean: 4,756,939,536.76 stddev: 1,117,314,866.53 user: 128,113 sys:
7,796 gc: 1,593 mem: 115.03
uptime: 120 acked: 2,919,360 acked/sec: 97,312.00 failed: 0 99%:
12,683,575,295 99.9%: 12,784,238,591 min: 5,440,012,288 max: 12,792,627,199
mean: 9,308,123,106.98 stddev: 1,615,564,564.30 user: 121,475 sys:
7,534 gc: 1,567 mem: 47.30
uptime: 150 acked: 2,877,200 acked/sec: 95,906.67 failed: 0 99%:
14,646,509,567 99.9%: 14,763,950,079 min: 9,302,966,272 max: 14,789,115,903
mean: 11,379,620,943.33 stddev: 1,593,183,614.74 user: 133,995 sys:
7,393 gc: 1,616 mem: 133.84
uptime: 181 acked: 2,859,340 acked/sec: 92,236.77 failed: 0 99%:
16,366,174,207 99.9%: 16,424,894,463 min: 10,116,661,248 max: 16,450,060,287
mean: 12,700,704,569.76 stddev: 2,014,006,250.70 user: 129,856 sys:
7,147 gc: 1,592 mem: 90.25
```
Metrics V2:
```
./bin/storm jar
../../../examples/storm-starter/target/storm-starter-1.2.0-SNAPSHOT.jar
org.apache.storm.starter.ThroughputVsLatency 50000 1 3 -c
topology.max.spout.pending=500
uptime: 30 acked: 751,560 acked/sec: 25,052.00 failed: 0 99%:
499,908,607 99.9%: 505,675,775 min: 4,108,288 max: 509,870,079
mean: 107,075,292.72 stddev: 173,303,090.76 user: 47,136 sys: 2,570
gc: 0 mem: 65.99
uptime: 60 acked: 1,505,040 acked/sec: 50,168.00 failed: 0 99%:
86,835,199 99.9%: 127,336,447 min: 3,575,808 max: 134,873,087
mean: 11,308,860.12 stddev: 11,073,472.42 user: 87,512 sys: 6,176
gc: 763 mem: 67.76
uptime: 90 acked: 1,503,580 acked/sec: 50,119.33 failed: 0 99%:
13,901,823 99.9%: 15,450,111 min: 3,684,352 max: 18,726,911
mean: 9,930,800.24 stddev: 1,361,960.54 user: 86,289 sys: 6,234
gc: 768 mem: 93.17
uptime: 120 acked: 1,504,200 acked/sec: 50,140.00 failed: 0 99%:
39,190,527 99.9%: 92,864,511 min: 3,645,440 max: 100,859,903
mean: 10,615,784.00 stddev: 6,838,619.84 user: 86,522 sys: 6,275
gc: 787 mem: 114.78
uptime: 150 acked: 1,503,880 acked/sec: 50,129.33 failed: 0 99%:
13,885,439 99.9%: 15,302,655 min: 3,964,928 max: 19,693,567
mean: 9,949,763.37 stddev: 1,350,149.76 user: 86,026 sys: 6,226
gc: 789 mem: 136.70
uptime: 180 acked: 1,503,840 acked/sec: 50,128.00 failed: 0 99%:
13,967,359 99.9%: 15,515,647 min: 3,670,016 max: 19,316,735
mean: 9,921,255.02 stddev: 1,380,147.00 user: 85,840 sys: 6,307
gc: 814 mem: 61.50
./bin/storm jar
../../../examples/storm-starter/target/storm-starter-1.2.0-SNAPSHOT.jar
org.apache.storm.starter.ThroughputVsLatency 75000 1 3 -c
topology.max.spout.pending=500
uptime: 30 acked: 810,700 acked/sec: 27,023.33 failed: 0 99%:
1,469,054,975 99.9%: 1,484,783,615 min: 110,428,160 max: 1,490,026,495
mean: 1,103,688,559.63 stddev: 218,353,127.14 user: 59,994 sys: 2,680
gc: 0 mem: 110.42
uptime: 60 acked: 1,580,040 acked/sec: 52,668.00 failed: 0 99%:
2,017,460,223 99.9%: 2,138,046,463 min: 305,135,616 max: 2,151,677,951
mean: 1,271,180,309.48 stddev: 349,131,056.74 user: 130,862 sys: 7,058
gc: 1,521 mem: 80.61
uptime: 90 acked: 2,279,900 acked/sec: 75,996.67 failed: 0 99%:
2,208,301,055 99.9%: 2,262,827,007 min: 13,189,120 max: 2,271,215,615
mean: 876,751,968.81 stddev: 666,332,632.22 user: 128,889 sys: 7,008
gc: 1,531 mem: 69.38
uptime: 120 acked: 2,305,340 acked/sec: 76,844.67 failed: 0 99%:
2,138,046,463 99.9%: 2,298,478,591 min: 4,882,432 max: 2,317,352,959
mean: 440,761,887.25 stddev: 592,545,096.04 user: 130,032 sys: 6,988
gc: 1,491 mem: 84.19
uptime: 151 acked: 2,277,980 acked/sec: 73,483.23 failed: 0 99%:
60,391,423 99.9%: 72,155,135 min: 3,788,800 max: 80,216,063
mean: 21,421,869.58 stddev: 13,674,045.52 user: 123,179 sys: 6,837
gc: 1,367 mem: 128.93
uptime: 181 acked: 2,254,200 acked/sec: 75,140.00 failed: 0 99%:
214,433,791 99.9%: 238,551,039 min: 4,816,896 max: 247,332,863
mean: 52,724,728.68 stddev: 43,440,715.92 user: 124,503 sys: 6,916
gc: 1,454 mem: 46.17
./bin/storm jar
../../../examples/storm-starter/target/storm-starter-1.2.0-SNAPSHOT.jar
org.apache.storm.starter.ThroughputVsLatency 100000 1 3 -c
topology.max.spout.pending=500
uptime: 30 acked: 1,123,900 acked/sec: 37,463.33 failed: 0 99%:
2,592,079,871 99.9%: 2,600,468,479 min: 73,465,856 max: 2,652,897,279
mean: 1,690,311,396.11 stddev: 480,273,985.78 user: 57,811 sys: 2,666
gc: 0 mem: 61.51
uptime: 60 acked: 2,262,680 acked/sec: 75,422.67 failed: 0 99%:
10,208,935,935 99.9%: 10,536,091,647 min: 2,415,919,104 max: 10,594,811,903
mean: 6,285,873,303.48 stddev: 2,142,972,667.99 user: 127,730 sys:
6,723 gc: 1,482 mem: 117.42
uptime: 90 acked: 2,313,860 acked/sec: 77,128.67 failed: 0 99%:
17,834,180,607 99.9%: 18,052,284,415 min: 9,596,567,552 max: 18,102,616,063
mean: 13,442,454,586.16 stddev: 1,991,956,204.06 user: 131,847 sys:
7,115 gc: 1,534 mem: 70.55
uptime: 120 acked: 2,334,940 acked/sec: 77,831.33 failed: 0 99%:
24,964,497,407 99.9%: 25,316,818,943 min: 16,315,842,560 max: 25,350,373,375
mean: 20,534,292,592.51 stddev: 2,142,432,639.60 user: 130,283 sys:
6,860 gc: 1,499 mem: 73.80
uptime: 150 acked: 2,355,160 acked/sec: 78,505.33 failed: 0 99%:
32,061,259,775 99.9%: 32,413,581,311 min: 22,749,904,896 max: 32,463,912,959
mean: 27,008,051,127.76 stddev: 2,183,882,755.64 user: 129,694 sys:
6,936 gc: 1,496 mem: 68.42
uptime: 180 acked: 2,346,800 acked/sec: 78,226.67 failed: 0 99%:
38,889,586,687 99.9%: 39,258,685,439 min: 29,024,583,680 max: 39,292,239,871
mean: 33,518,848,499.25 stddev: 2,297,910,092.43 user: 131,058 sys:
6,849 gc: 1,514 mem: 71.15
```
At 50,000 sentences per second on a single worker both can keep up, but the
mean latency appears to be about 200 to 300 microseconds slower 9.6 ms vs 9.9
ms. The bigger issue in my mind is the CPU time as it went up from about 79
seconds (max 30 second period) every 30 seconds to 92 seconds (min 30 second
period after stabilizing). That was at least a 16% increase in CPU usage.
At 75,000 sentences per second metrics v2 was just able to keep up with
that throughput while 1.2.0-SNAPSHOT could handle it easily. I will not compare
the latency because metrics v2 didn't stabilize enough for me to feel good
reporting the numbers, but the CPU time went up from 118 seconds every 30
seconds to 130 seconds. That was an increase of at least 9%.
When I tried to do 100,000 sentences per second neither of the two could
keep up, but 1.2.0-SNAPSHOT hit a maximum of 97k while metrics V2 hit a maximum
of 78k or 20% less maximum throughput. Only here did we see the CPU usage be
lower for metrics V2 with 134 seconds vs 141 for 1.2.0-SNAPSHOT, but if we take
into account the actual throughput and look at CPU seconds needed to process 1k
sentences per second it is 1.7 for metrics v2 vs 1.5 for 1.2.0-SNAPSHOT.
If you need me to run the numbers again on different hardware I can, but it
may take me some more time to get access to bare metal as I find VMs produce
less reliable numbers than my laptop does.
I am not saying that these changes are a blocker for the code going in, but
I really want to understand what is happening here
---
