----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55284/#review161001 -----------------------------------------------------------
Ship it! src/jmh/java/org/apache/aurora/benchmark/SlaAlgorithmBenchmark.java (line 99) <https://reviews.apache.org/r/55284/#comment232255> Increase this to 150k, 200k? - Santhosh Kumar Shanmugham On Jan. 8, 2017, 5:04 p.m., Reza Motamedi wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/55284/ > ----------------------------------------------------------- > > (Updated Jan. 8, 2017, 5:04 p.m.) > > > Review request for Aurora, Joshua Cohen, Mehrdad Nurolahzade, and Santhosh > Kumar Shanmugham. > > > Repository: aurora > > > Description > ------- > > # Adding SLA benchmarks benchmarks > This RB is related to > [AURORA-1854](https://issues.apache.org/jira/browse/AURORA-1854). > The goal is to add JMH benchmarks for SlaCalculation and then other metric > calculators. > > I am not sure if the results are very resonable. For instance see that the > ops/sec is 54.701±19.743, which seems to have a high variance. > > > Diffs > ----- > > src/jmh/java/org/apache/aurora/benchmark/SlaAlgorithmBenchmark.java > PRE-CREATION > src/main/java/org/apache/aurora/scheduler/sla/SlaAlgorithm.java > 5d8d5bd8f705770979f284d26d2e932aabe707e5 > > Diff: https://reviews.apache.org/r/55284/diff/ > > > Testing > ------- > > Following is the benchmarks collectcted: > > # rmotamedi@tw-mbp-rmotamedi:~/workspace/aurora on git:jmh-sla: [14:39:56] > > ./gradlew jmh -Pbenchmarks='SlaAlgorithmBenchmark' > > > the content of <./aurora/dist/reports/jmh/human.txt> > > ``` > # JMH 1.15 (released 98 days ago) > # VM version: JDK 1.8.0_111, VM 25.111-b14 > # VM invoker: > /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java > # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US > -Duser.language=en -Duser.variant > # Warmup: 1 iterations, 10 s each > # Measurement: 5 iterations, 5 s each > # Timeout: 10 min per iteration > # Threads: 1 thread, will synchronize iterations > # Benchmark mode: Throughput, ops/time > # Benchmark: > org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA > # Parameters: (numTasks = 100) > > # Run progress: 0.00% complete, ETA 00:05:15 > # Fork: 1 of 1 > # Warmup Iteration 1: 127519.626 ops/s > Iteration 1: 152258.072 ops/s > Iteration 2: 170404.745 ops/s > Iteration 3: 138346.324 ops/s > Iteration 4: 150597.403 ops/s > Iteration 5: 169640.531 ops/s > > > Result "measureMTTA": > 156249.415 ±(99.9%) 52660.004 ops/s [Average] > (min, avg, max) = (138346.324, 156249.415, 170404.745), stdev = 13675.636 > CI (99.9%): [103589.411, 208909.419] (assumes normal distribution) > > > # JMH 1.15 (released 98 days ago) > # VM version: JDK 1.8.0_111, VM 25.111-b14 > # VM invoker: > /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java > # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US > -Duser.language=en -Duser.variant > # Warmup: 1 iterations, 10 s each > # Measurement: 5 iterations, 5 s each > # Timeout: 10 min per iteration > # Threads: 1 thread, will synchronize iterations > # Benchmark mode: Throughput, ops/time > # Benchmark: > org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA > # Parameters: (numTasks = 1000) > > # Run progress: 11.11% complete, ETA 00:04:45 > # Fork: 1 of 1 > # Warmup Iteration 1: 4329.734 ops/s > Iteration 1: 4115.153 ops/s > Iteration 2: 3873.048 ops/s > Iteration 3: 3479.972 ops/s > Iteration 4: 3225.995 ops/s > Iteration 5: 3002.072 ops/s > > > Result "measureMTTA": > 3539.248 ±(99.9%) 1757.523 ops/s [Average] > (min, avg, max) = (3002.072, 3539.248, 4115.153), stdev = 456.423 > CI (99.9%): [1781.725, 5296.771] (assumes normal distribution) > > > # JMH 1.15 (released 98 days ago) > # VM version: JDK 1.8.0_111, VM 25.111-b14 > # VM invoker: > /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java > # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US > -Duser.language=en -Duser.variant > # Warmup: 1 iterations, 10 s each > # Measurement: 5 iterations, 5 s each > # Timeout: 10 min per iteration > # Threads: 1 thread, will synchronize iterations > # Benchmark mode: Throughput, ops/time > # Benchmark: > org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA > # Parameters: (numTasks = 10000) > > # Run progress: 22.22% complete, ETA 00:04:10 > # Fork: 1 of 1 > # Warmup Iteration 1: 44.147 ops/s > Iteration 1: 46.116 ops/s > Iteration 2: 54.255 ops/s > Iteration 3: 57.967 ops/s > Iteration 4: 56.174 ops/s > Iteration 5: 58.992 ops/s > > > Result "measureMTTA": > 54.701 ±(99.9%) 19.743 ops/s [Average] > (min, avg, max) = (46.116, 54.701, 58.992), stdev = 5.127 > CI (99.9%): [34.957, 74.444] (assumes normal distribution) > > > # JMH 1.15 (released 98 days ago) > # VM version: JDK 1.8.0_111, VM 25.111-b14 > # VM invoker: > /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java > # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US > -Duser.language=en -Duser.variant > # Warmup: 1 iterations, 10 s each > # Measurement: 5 iterations, 5 s each > # Timeout: 10 min per iteration > # Threads: 1 thread, will synchronize iterations > # Benchmark mode: Throughput, ops/time > # Benchmark: > org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR > # Parameters: (numTasks = 100) > > # Run progress: 33.33% complete, ETA 00:03:38 > # Fork: 1 of 1 > # Warmup Iteration 1: 106633.191 ops/s > Iteration 1: 136861.727 ops/s > Iteration 2: 120490.231 ops/s > Iteration 3: 106383.718 ops/s > Iteration 4: 126247.774 ops/s > Iteration 5: 131648.970 ops/s > > > Result "measureMTTR": > 124326.484 ±(99.9%) 45198.021 ops/s [Average] > (min, avg, max) = (106383.718, 124326.484, 136861.727), stdev = 11737.783 > CI (99.9%): [79128.463, 169524.505] (assumes normal distribution) > > > # JMH 1.15 (released 98 days ago) > # VM version: JDK 1.8.0_111, VM 25.111-b14 > # VM invoker: > /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java > # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US > -Duser.language=en -Duser.variant > # Warmup: 1 iterations, 10 s each > # Measurement: 5 iterations, 5 s each > # Timeout: 10 min per iteration > # Threads: 1 thread, will synchronize iterations > # Benchmark mode: Throughput, ops/time > # Benchmark: > org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR > # Parameters: (numTasks = 1000) > > # Run progress: 44.44% complete, ETA 00:03:01 > # Fork: 1 of 1 > # Warmup Iteration 1: 4012.815 ops/s > Iteration 1: 4396.495 ops/s > Iteration 2: 4086.905 ops/s > Iteration 3: 4024.187 ops/s > Iteration 4: 3809.202 ops/s > Iteration 5: 3551.905 ops/s > > > Result "measureMTTR": > 3973.739 ±(99.9%) 1216.296 ops/s [Average] > (min, avg, max) = (3551.905, 3973.739, 4396.495), stdev = 315.868 > CI (99.9%): [2757.443, 5190.034] (assumes normal distribution) > > > # JMH 1.15 (released 98 days ago) > # VM version: JDK 1.8.0_111, VM 25.111-b14 > # VM invoker: > /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java > # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US > -Duser.language=en -Duser.variant > # Warmup: 1 iterations, 10 s each > # Measurement: 5 iterations, 5 s each > # Timeout: 10 min per iteration > # Threads: 1 thread, will synchronize iterations > # Benchmark mode: Throughput, ops/time > # Benchmark: > org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR > # Parameters: (numTasks = 10000) > > # Run progress: 55.56% complete, ETA 00:02:25 > # Fork: 1 of 1 > # Warmup Iteration 1: 50.933 ops/s > Iteration 1: 57.153 ops/s > Iteration 2: 58.741 ops/s > Iteration 3: 56.398 ops/s > Iteration 4: 55.078 ops/s > Iteration 5: 58.142 ops/s > > > Result "measureMTTR": > 57.102 ±(99.9%) 5.567 ops/s [Average] > (min, avg, max) = (55.078, 57.102, 58.741), stdev = 1.446 > CI (99.9%): [51.535, 62.670] (assumes normal distribution) > > > # JMH 1.15 (released 98 days ago) > # VM version: JDK 1.8.0_111, VM 25.111-b14 > # VM invoker: > /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java > # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US > -Duser.language=en -Duser.variant > # Warmup: 1 iterations, 10 s each > # Measurement: 5 iterations, 5 s each > # Timeout: 10 min per iteration > # Threads: 1 thread, will synchronize iterations > # Benchmark mode: Throughput, ops/time > # Benchmark: > org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS > # Parameters: (numTasks = 100) > > # Run progress: 66.67% complete, ETA 00:01:49 > # Fork: 1 of 1 > # Warmup Iteration 1: 107724.308 ops/s > Iteration 1: 121787.913 ops/s > Iteration 2: 119594.939 ops/s > Iteration 3: 128441.840 ops/s > Iteration 4: 134176.538 ops/s > Iteration 5: 144275.262 ops/s > > > Result "measureMTTS": > 129655.298 ±(99.9%) 38446.901 ops/s [Average] > (min, avg, max) = (119594.939, 129655.298, 144275.262), stdev = 9984.538 > CI (99.9%): [91208.397, 168102.199] (assumes normal distribution) > > > # JMH 1.15 (released 98 days ago) > # VM version: JDK 1.8.0_111, VM 25.111-b14 > # VM invoker: > /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java > # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US > -Duser.language=en -Duser.variant > # Warmup: 1 iterations, 10 s each > # Measurement: 5 iterations, 5 s each > # Timeout: 10 min per iteration > # Threads: 1 thread, will synchronize iterations > # Benchmark mode: Throughput, ops/time > # Benchmark: > org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS > # Parameters: (numTasks = 1000) > > # Run progress: 77.78% complete, ETA 00:01:12 > # Fork: 1 of 1 > # Warmup Iteration 1: 3738.823 ops/s > Iteration 1: 4026.955 ops/s > Iteration 2: 3070.155 ops/s > Iteration 3: 2706.577 ops/s > Iteration 4: 3510.103 ops/s > Iteration 5: 4063.217 ops/s > > > Result "measureMTTS": > 3475.401 ±(99.9%) 2283.158 ops/s [Average] > (min, avg, max) = (2706.577, 3475.401, 4063.217), stdev = 592.929 > CI (99.9%): [1192.244, 5758.559] (assumes normal distribution) > > > # JMH 1.15 (released 98 days ago) > # VM version: JDK 1.8.0_111, VM 25.111-b14 > # VM invoker: > /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java > # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US > -Duser.language=en -Duser.variant > # Warmup: 1 iterations, 10 s each > # Measurement: 5 iterations, 5 s each > # Timeout: 10 min per iteration > # Threads: 1 thread, will synchronize iterations > # Benchmark mode: Throughput, ops/time > # Benchmark: > org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS > # Parameters: (numTasks = 10000) > > # Run progress: 88.89% complete, ETA 00:00:36 > # Fork: 1 of 1 > # Warmup Iteration 1: 56.489 ops/s > Iteration 1: 59.288 ops/s > Iteration 2: 59.160 ops/s > Iteration 3: 56.142 ops/s > Iteration 4: 53.336 ops/s > Iteration 5: 53.147 ops/s > > > Result "measureMTTS": > 56.215 ±(99.9%) 11.524 ops/s [Average] > (min, avg, max) = (53.147, 56.215, 59.288), stdev = 2.993 > CI (99.9%): [44.691, 67.738] (assumes normal distribution) > > > # Run complete. Total time: 00:05:27 > > Benchmark (numTasks) Mode Cnt > Score Error Units > SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA 100 thrpt 5 > 156249.415 ± 52660.004 ops/s > SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA 1000 thrpt 5 > 3539.248 ± 1757.523 ops/s > SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA 10000 thrpt 5 > 54.701 ± 19.743 ops/s > SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR 100 thrpt 5 > 124326.484 ± 45198.021 ops/s > SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR 1000 thrpt 5 > 3973.739 ± 1216.296 ops/s > SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR 10000 thrpt 5 > 57.102 ± 5.567 ops/s > SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS 100 thrpt 5 > 129655.298 ± 38446.901 ops/s > SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS 1000 thrpt 5 > 3475.401 ± 2283.158 ops/s > SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS 10000 thrpt 5 > 56.215 ± 11.524 ops/s > > Benchmark result is saved to dist/reports/jmh/results.txt > > ``` > > > Thanks, > > Reza Motamedi > >
