Github user HeartSaVioR commented on the pull request:
https://github.com/apache/storm/pull/1324#issuecomment-218962460
I'd like to show result of some performance tests to claim why this
improvement is not optional but must.
I modified ThroughputVsLatency to attach addition metrics consumer which
sleeps between 0 and 20000 ms (by random) whenever handleDataPoints() is called.
> patch / detach slow consumer
```
uptime: 30 acked: 6,700 acked/sec: 223.33 failed: 0 99%:
19,478,347,775 99.9%: 19,730,006,015 min: 1,031,798,784 max: 20,031,995,903
mean: 9,997,166,638.68 stddev: 4,955,322,338.86 user: 132,350 sys:
6,530 gc: 2,768 mem: 156.95
uptime: 60 acked: 1,250,140 acked/sec: 41,671.33 failed: 0 99%:
30,064,771,071 99.9%: 33,000,783,871 min: 8,007,680 max: 36,339,449,855
mean: 12,109,785,636.33 stddev: 9,009,882,198.75 user: 229,640 sys:
14,950 gc: 48,235 mem: 161.00
uptime: 90 acked: 905,120 acked/sec: 30,170.67 failed: 0 99%:
417,595,391 99.9%: 5,322,571,775 min: 7,536,640 max: 9,546,235,903
mean: 43,809,038.25 stddev: 340,479,608.43 user: 146,070 sys: 21,900
gc: 1,144 mem: 161.92
uptime: 120 acked: 926,660 acked/sec: 30,888.67 failed: 0 99%:
29,409,279 99.9%: 48,463,871 min: 7,438,336 max: 82,771,967
mean: 13,172,560.18 stddev: 3,608,380.15 user: 131,760 sys: 23,500
gc: 1,008 mem: 162.60
uptime: 150 acked: 926,260 acked/sec: 30,875.33 failed: 0 99%:
29,065,215 99.9%: 48,857,087 min: 8,003,584 max: 76,152,831
mean: 13,145,472.87 stddev: 3,538,817.50 user: 128,120 sys: 24,890
gc: 915 mem: 162.90
uptime: 180 acked: 903,140 acked/sec: 30,104.67 failed: 0 99%:
25,624,575 99.9%: 40,566,783 min: 7,962,624 max: 58,785,791
mean: 12,926,076.82 stddev: 2,792,878.03 user: 127,060 sys: 25,820
gc: 869 mem: 163.08
uptime: 210 acked: 903,020 acked/sec: 30,100.67 failed: 0 99%:
30,752,767 99.9%: 45,350,911 min: 7,970,816 max: 73,269,247
mean: 13,114,515.21 stddev: 3,497,139.32 user: 126,060 sys: 26,650
gc: 1,034 mem: 163.30
uptime: 240 acked: 903,160 acked/sec: 30,105.33 failed: 0 99%:
27,541,503 99.9%: 39,976,959 min: 7,921,664 max: 68,091,903
mean: 12,996,252.52 stddev: 3,025,754.06 user: 125,040 sys: 27,630
gc: 968 mem: 163.44
uptime: 270 acked: 902,840 acked/sec: 30,094.67 failed: 0 99%:
28,721,151 99.9%: 45,055,999 min: 7,888,896 max: 71,041,023
mean: 13,062,900.28 stddev: 3,299,611.61 user: 124,240 sys: 28,080
gc: 970 mem: 163.55
uptime: 300 acked: 902,540 acked/sec: 30,084.67 failed: 0 99%:
26,198,015 99.9%: 39,354,367 min: 7,929,856 max: 56,262,655
mean: 12,902,259.50 stddev: 2,817,070.04 user: 124,650 sys: 27,890
gc: 970 mem: 163.61
```
> patch / attach slow consumer
```
uptime: 30 acked: 32,340 acked/sec: 1,078.00 failed: 0 99%:
9,705,619,455 99.9%: 9,923,723,263 min: 724,041,728 max: 9,990,832,127
mean: 8,066,563,909.78 stddev: 1,020,515,611.77 user: 109,100 sys:
4,920 gc: 0 mem: 157.75
uptime: 61 acked: 1,255,480 acked/sec: 40,499.35 failed: 0 99%:
15,879,634,943 99.9%: 17,330,864,127 min: 8,003,584 max: 18,488,492,031
mean: 6,198,566,388.60 stddev: 5,041,039,825.91 user: 191,130 sys:
11,490 gc: 26,684 mem: 162.38
uptime: 91 acked: 904,680 acked/sec: 30,156.00 failed: 0 99%:
33,980,415 99.9%: 60,293,119 min: 7,274,496 max: 84,279,295
mean: 13,399,013.36 stddev: 4,398,380.66 user: 126,980 sys: 17,260
gc: 1,075 mem: 162.93
uptime: 121 acked: 904,780 acked/sec: 30,159.33 failed: 0 99%:
33,718,271 99.9%: 66,519,039 min: 7,598,080 max: 105,644,031
mean: 13,286,131.53 stddev: 4,491,906.80 user: 124,850 sys: 20,730
gc: 971 mem: 163.51
uptime: 151 acked: 903,340 acked/sec: 30,111.33 failed: 0 99%:
26,427,391 99.9%: 48,070,655 min: 7,151,616 max: 82,575,359
mean: 12,966,891.48 stddev: 3,125,613.14 user: 122,320 sys: 22,340
gc: 987 mem: 163.98
uptime: 181 acked: 903,480 acked/sec: 30,116.00 failed: 0 99%:
24,723,455 99.9%: 39,256,063 min: 6,942,720 max: 59,637,759
mean: 12,863,849.51 stddev: 2,673,948.37 user: 121,470 sys: 22,630
gc: 992 mem: 164.32
uptime: 211 acked: 903,040 acked/sec: 30,101.33 failed: 0 99%:
22,773,759 99.9%: 36,896,767 min: 7,835,648 max: 67,764,223
mean: 12,768,220.30 stddev: 2,426,130.42 user: 119,910 sys: 24,270
gc: 933 mem: 164.56
uptime: 241 acked: 903,460 acked/sec: 30,115.33 failed: 0 99%:
25,133,055 99.9%: 57,114,623 min: 7,331,840 max: 92,930,047
mean: 12,921,105.46 stddev: 3,325,266.12 user: 118,350 sys: 24,990
gc: 1,041 mem: 164.68
uptime: 271 acked: 902,980 acked/sec: 30,099.33 failed: 0 99%:
23,822,335 99.9%: 35,454,975 min: 7,913,472 max: 56,066,047
mean: 12,774,768.69 stddev: 2,407,323.94 user: 117,500 sys: 25,880
gc: 940 mem: 164.73
uptime: 301 acked: 903,160 acked/sec: 30,105.33 failed: 0 99%:
24,199,167 99.9%: 40,042,495 min: 6,901,760 max: 76,152,831
mean: 12,811,708.30 stddev: 2,607,966.30 user: 117,830 sys: 25,650
gc: 980 mem: 164.87
```
> before patch / attach slow consumer
```
uptime: 30 acked: 61,380 acked/sec: 2,046.00 failed: 0 99%:
8,824,815,615 99.9%: 9,227,468,799 min: 1,380,974,592 max: 9,487,515,647
mean: 7,127,816,475.13 stddev: 971,681,678.61 user: 110,140 sys: 4,750
gc: 0 mem: 158.50
uptime: 60 acked: 230,940 acked/sec: 7,698.00 failed: 0 99%:
15,854,469,119 99.9%: 16,324,231,167 min: 2,675,965,952 max: 16,634,609,663
mean: 11,353,373,038.74 stddev: 2,644,428,448.90 user: 82,430 sys:
4,050 gc: 6,585 mem: 161.26
uptime: 90 acked: 0 acked/sec: 0.00 failed: 0 99%:
0 99.9%: 0 min: 0 max: 0
mean: 0.00 stddev: 0.00 user: 27,980 sys: 1,700
gc: 38 mem: 161.87
uptime: 120 acked: 0 acked/sec: 0.00 failed: 0 99%:
0 99.9%: 0 min: 0 max: 0
mean: 0.00 stddev: 0.00 user: 30,600 sys: 1,630
gc: 30 mem: 162.48
uptime: 150 acked: 0 acked/sec: 0.00 failed: 0 99%:
0 99.9%: 0 min: 0 max: 0
mean: 0.00 stddev: 0.00 user: 29,600 sys: 1,740
gc: 31 mem: 163.01
uptime: 180 acked: 0 acked/sec: 0.00 failed: 0 99%:
0 99.9%: 0 min: 0 max: 0
mean: 0.00 stddev: 0.00 user: 26,900 sys: 1,990
gc: 21 mem: 163.33
uptime: 210 acked: 0 acked/sec: 0.00 failed: 0 99%:
0 99.9%: 0 min: 0 max: 0
mean: 0.00 stddev: 0.00 user: 27,020 sys: 2,000
gc: 22 mem: 163.69
uptime: 240 acked: 0 acked/sec: 0.00 failed: 0 99%:
0 99.9%: 0 min: 0 max: 0
mean: 0.00 stddev: 0.00 user: 28,060 sys: 2,180
gc: 42 mem: 164.00
uptime: 270 acked: 0 acked/sec: 0.00 failed: 0 99%:
0 99.9%: 0 min: 0 max: 0
mean: 0.00 stddev: 0.00 user: 28,940 sys: 2,370
gc: 22 mem: 164.25
uptime: 300 acked: 0 acked/sec: 0.00 failed: 0 99%:
0 99.9%: 0 min: 0 max: 0
mean: 0.00 stddev: 0.00 user: 28,840 sys: 2,480
gc: 23 mem: 164.40
```
As you may notice, backpressure is triggered from receive queue on metrics
consumer, and **it's never back to normal** because metric messages are sent to
metrics consumer even though backpressure is on.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
