Mark Hanson created GEODE-7269: ---------------------------------- Summary: Benchmarks failure Key: GEODE-7269 URL: https://issues.apache.org/jira/browse/GEODE-7269 Project: Geode Issue Type: Bug Components: benchmarks Reporter: Mark Hanson
Benchmarks failed in [https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-main/jobs/Benchmark/builds/607] {noformat} > Task :harness:analyzeRun Running analyzer Comparing test result at /tmp/build/2ea77113/results/benchmarks-apache-develop-main-Benchmark-607-98877/branch/benchmarks_20191002113518 to baseline at /tmp/build/2ea77113/results/benchmarks-apache-develop-main-Benchmark-607-98877/baseline/benchmarks_20191002092613 SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. org.apache.geode.benchmark.tests.PartitionedFunctionExecutionWithArgumentsBenchmark average ops/second Baseline: 165126.55 Test: 163322.19 Difference: -1.1% ops/second standard error Baseline: 481.54 Test: 394.45 Difference: -18.1% ops/second standard deviation Baseline: 8326.61 Test: 6832.03 Difference: -17.9% YS 99th percentile latency Baseline: 1906.00 Test: 1905.00 Difference: -0.1% median latency Baseline: 633855.00 Test: 630271.00 Difference: -0.6% 90th percentile latency Baseline: 1326079.00 Test: 1367039.00 Difference: +3.1% 99th percentile latency Baseline: 4591615.00 Test: 4644863.00 Difference: +1.2% 99.9th percentile latency Baseline: 15359999.00 Test: 16809983.00 Difference: +9.4% average latency Baseline: 871151.53 Test: 880810.13 Difference: +1.1% latency standard deviation Baseline: 1050482.32 Test: 1100814.13 Difference: +4.8% latency standard error Baseline: 149.28 Test: 157.30 Difference: +5.4% org.apache.geode.benchmark.tests.PartitionedFunctionExecutionWithFiltersBenchmark average ops/second Baseline: 459731.05 Test: 453392.16 Difference: -1.4% ops/second standard error Baseline: 394.19 Test: 332.85 Difference: -15.6% ops/second standard deviation Baseline: 6816.12 Test: 5765.11 Difference: -15.4% YS 99th percentile latency Baseline: 1405.00 Test: 1405.00 Difference: +0.0% median latency Baseline: 449791.00 Test: 459519.00 Difference: +2.2% 90th percentile latency Baseline: 943615.00 Test: 961023.00 Difference: +1.8% 99th percentile latency Baseline: 3616767.00 Test: 3614719.00 Difference: -0.1% 99.9th percentile latency Baseline: 10559487.00 Test: 10551295.00 Difference: -0.1% average latency Baseline: 625412.03 Test: 634121.49 Difference: +1.4% latency standard deviation Baseline: 788269.89 Test: 788318.28 Difference: +0.0% latency standard error Baseline: 67.13 Test: 67.60 Difference: +0.7% org.apache.geode.benchmark.tests.PartitionedGetBenchmark average ops/second Baseline: 841000.72 Test: 828146.58 Difference: -1.5% ops/second standard error Baseline: 558.63 Test: 507.87 Difference: -9.1% ops/second standard deviation Baseline: 9659.55 Test: 8781.92 Difference: -9.1% YS 99th percentile latency Baseline: 806.00 Test: 805.00 Difference: -0.1% median latency Baseline: 332287.00 Test: 339967.00 Difference: +2.3% 90th percentile latency Baseline: 443391.00 Test: 452351.00 Difference: +2.0% 99th percentile latency Baseline: 2672639.00 Test: 2662399.00 Difference: -0.4% 99.9th percentile latency Baseline: 8089599.00 Test: 7974911.00 Difference: -1.4% average latency Baseline: 426878.72 Test: 433532.06 Difference: +1.6% latency standard deviation Baseline: 546763.26 Test: 542115.50 Difference: -0.9% latency standard error Baseline: 34.43 Test: 34.40 Difference: -0.1% org.apache.geode.benchmark.tests.PartitionedGetLongBenchmark average ops/second Baseline: 1108020.75 Test: 1107517.03 Difference: -0.0% ops/second standard error Baseline: 508.14 Test: 306.97 Difference: -39.6% ops/second standard deviation Baseline: 8801.18 Test: 5307.94 Difference: -39.7% YS 99th percentile latency Baseline: 902.00 Test: 902.00 Difference: +0.0% median latency Baseline: 276735.00 Test: 276223.00 Difference: -0.2% 90th percentile latency Baseline: 412159.00 Test: 412159.00 Difference: +0.0% 99th percentile latency Baseline: 1497087.00 Test: 1497087.00 Difference: +0.0% 99.9th percentile latency Baseline: 4624383.00 Test: 4833279.00 Difference: +4.5% average latency Baseline: 323752.23 Test: 324056.03 Difference: +0.1% latency standard deviation Baseline: 325710.03 Test: 337221.72 Difference: +3.5% latency standard error Baseline: 17.87 Test: 18.51 Difference: +3.6% org.apache.geode.benchmark.tests.PartitionedIndexedQueryBenchmark average ops/second Baseline: 18134.76 Test: 18259.64 Difference: +0.7% ops/second standard error Baseline: 5.64 Test: 10.68 Difference: +89.4% ops/second standard deviation Baseline: 97.49 Test: 184.64 Difference: +89.4% YS 99th percentile latency Baseline: 20092.87 Test: 20094.31 Difference: +0.0% median latency Baseline: 11075583.00 Test: 13991935.00 Difference: +26.3% 90th percentile latency Baseline: 32178175.00 Test: 22986751.00 Difference: -28.6% 99th percentile latency Baseline: 85786623.00 Test: 59506687.00 Difference: -30.6% 99.9th percentile latency Baseline: 103284735.00 Test: 69271551.00 Difference: -32.9% average latency Baseline: 15878192.88 Test: 15776565.53 Difference: -0.6% latency standard deviation Baseline: 16509200.47 Test: 10274770.09 Difference: -37.8% latency standard error Baseline: 7080.93 Test: 4392.16 Difference: -38.0% org.apache.geode.benchmark.tests.PartitionedPutAllBenchmark average ops/second Baseline: 660.51 Test: 592.93 Difference: -10.2% ops/second standard error Baseline: 3.94 Test: 13.05 Difference: +230.8% ops/second standard deviation Baseline: 67.17 Test: 219.13 Difference: +226.2% YS 99th percentile latency Baseline: 20099.00 Test: 20099.00 Difference: +0.0% median latency Baseline: 65306623.00 Test: 64684031.00 Difference: -1.0% 90th percentile latency Baseline: 222298111.00 Test: 221118463.00 Difference: -0.5% 99th percentile latency Baseline: 332136447.00 Test: 371982335.00 Difference: +12.0% 99.9th percentile latency Baseline: 418381823.00 Test: 8623489023.00 Difference: +1961.2% average latency Baseline: 112946954.05 Test: 128847254.66 Difference: +14.1% latency standard deviation Baseline: 82763295.05 Test: 388106155.06 Difference: +368.9% latency standard error Baseline: 189315.85 Test: 948056.57 Difference: +400.8% org.apache.geode.benchmark.tests.PartitionedPutAllLongBenchmark average ops/second Baseline: 661.43 Test: 637.88 Difference: -3.6% ops/second standard error Baseline: 4.03 Test: 4.12 Difference: +2.2% ops/second standard deviation Baseline: 68.83 Test: 70.35 Difference: +2.2% YS 99th percentile latency Baseline: 20099.00 Test: 20099.00 Difference: +0.0% median latency Baseline: 64290815.00 Test: 65765375.00 Difference: +2.3% 90th percentile latency Baseline: 215220223.00 Test: 227934207.00 Difference: +5.9% 99th percentile latency Baseline: 333185023.00 Test: 360710143.00 Difference: +8.3% 99.9th percentile latency Baseline: 425197567.00 Test: 438042623.00 Difference: +3.0% average latency Baseline: 112212752.63 Test: 116032713.49 Difference: +3.4% latency standard deviation Baseline: 82581968.17 Test: 86699325.73 Difference: +5.0% latency standard error Baseline: 188292.70 Test: 200999.53 Difference: +6.7% org.apache.geode.benchmark.tests.PartitionedPutBenchmark average ops/second Baseline: 442387.48 Test: 438685.31 Difference: -0.8% ops/second standard error Baseline: 1156.06 Test: 1238.08 Difference: +7.1% ops/second standard deviation Baseline: 19990.22 Test: 21408.48 Difference: +7.1% YS 99th percentile latency Baseline: 1703.00 Test: 1605.00 Difference: -5.8% median latency Baseline: 533503.00 Test: 532991.00 Difference: -0.1% 90th percentile latency Baseline: 1041407.00 Test: 1031679.00 Difference: -0.9% 99th percentile latency Baseline: 3389439.00 Test: 3647487.00 Difference: +7.6% 99.9th percentile latency Baseline: 71041023.00 Test: 69730303.00 Difference: -1.8% average latency Baseline: 812619.22 Test: 819287.36 Difference: +0.8% latency standard deviation Baseline: 3162083.45 Test: 3172446.86 Difference: +0.3% latency standard error Baseline: 274.51 Test: 276.55 Difference: +0.7% org.apache.geode.benchmark.tests.PartitionedPutLongBenchmark average ops/second Baseline: 435576.83 Test: 430206.87 Difference: -1.2% ops/second standard error Baseline: 942.64 Test: 1029.65 Difference: +9.2% ops/second standard deviation Baseline: 16299.70 Test: 17804.26 Difference: +9.2% YS 99th percentile latency Baseline: 1704.00 Test: 1604.00 Difference: -5.9% median latency Baseline: 530431.00 Test: 534527.00 Difference: +0.8% 90th percentile latency Baseline: 1041919.00 Test: 1027583.00 Difference: -1.4% 99th percentile latency Baseline: 3418111.00 Test: 3627007.00 Difference: +6.1% 99.9th percentile latency Baseline: 79429631.00 Test: 78839807.00 Difference: -0.7% average latency Baseline: 825364.59 Test: 835694.56 Difference: +1.3% latency standard deviation Baseline: 3513247.60 Test: 3521391.08 Difference: +0.2% latency standard error Baseline: 307.37 Test: 310.02 Difference: +0.9% org.apache.geode.benchmark.tests.ReplicatedFunctionExecutionBenchmark average ops/second Baseline: 425796.94 Test: 426212.98 Difference: +0.1% ops/second standard error Baseline: 203.05 Test: 176.30 Difference: -13.2% ops/second standard deviation Baseline: 3511.14 Test: 3048.55 Difference: -13.2% YS 99th percentile latency Baseline: 551.00 Test: 550.00 Difference: -0.2% median latency Baseline: 237951.00 Test: 238207.00 Difference: +0.1% 90th percentile latency Baseline: 316927.00 Test: 319999.00 Difference: +1.0% 99th percentile latency Baseline: 486143.00 Test: 474367.00 Difference: -2.4% 99.9th percentile latency Baseline: 1098751.00 Test: 778239.00 Difference: -29.2% average latency Baseline: 252639.01 Test: 252393.64 Difference: -0.1% latency standard deviation Baseline: 148085.78 Test: 140822.70 Difference: -4.9% latency standard error Baseline: 13.10 Test: 12.45 Difference: -5.0% org.apache.geode.benchmark.tests.ReplicatedFunctionExecutionWithArgumentsBenchmark average ops/second Baseline: 602363.99 Test: 618818.13 Difference: +2.7% ops/second standard error Baseline: 585.52 Test: 440.33 Difference: -24.8% ops/second standard deviation Baseline: 10124.58 Test: 7614.06 Difference: -24.8% YS 99th percentile latency Baseline: 1505.00 Test: 1504.00 Difference: -0.1% median latency Baseline: 715775.00 Test: 740351.00 Difference: +3.4% 90th percentile latency Baseline: 1096703.00 Test: 1108991.00 Difference: +1.1% 99th percentile latency Baseline: 9166847.00 Test: 6455295.00 Difference: -29.6% 99.9th percentile latency Baseline: 22953983.00 Test: 24051711.00 Difference: +4.8% average latency Baseline: 954315.26 Test: 928885.19 Difference: -2.7% latency standard deviation Baseline: 1699506.42 Test: 1593965.78 Difference: -6.2% latency standard error Baseline: 126.44 Test: 117.00 Difference: -7.5% org.apache.geode.benchmark.tests.ReplicatedFunctionExecutionWithFiltersBenchmark average ops/second Baseline: 598263.31 Test: 603567.04 Difference: +0.9% ops/second standard error Baseline: 436.82 Test: 417.76 Difference: -4.4% ops/second standard deviation Baseline: 7553.34 Test: 7223.80 Difference: -4.4% YS 99th percentile latency Baseline: 1103.00 Test: 1104.00 Difference: +0.1% median latency Baseline: 470271.00 Test: 493311.00 Difference: +4.9% 90th percentile latency Baseline: 719871.00 Test: 721919.00 Difference: +0.3% 99th percentile latency Baseline: 4100095.00 Test: 3348479.00 Difference: -18.3% 99.9th percentile latency Baseline: 13074431.00 Test: 11190271.00 Difference: -14.4% average latency Baseline: 600157.39 Test: 594966.11 Difference: -0.9% latency standard deviation Baseline: 914725.70 Test: 740635.84 Difference: -19.0% latency standard error Baseline: 68.28 Test: 55.05 Difference: -19.4% org.apache.geode.benchmark.tests.ReplicatedGetBenchmark average ops/second Baseline: 870190.68 Test: 860972.41 Difference: -1.1% ops/second standard error Baseline: 541.64 Test: 493.00 Difference: -9.0% ops/second standard deviation Baseline: 9365.80 Test: 8524.83 Difference: -9.0% YS 99th percentile latency Baseline: 804.00 Test: 805.00 Difference: +0.1% median latency Baseline: 324351.00 Test: 324863.00 Difference: +0.2% 90th percentile latency Baseline: 427519.00 Test: 425471.00 Difference: -0.5% 99th percentile latency Baseline: 2635775.00 Test: 2840575.00 Difference: +7.8% 99.9th percentile latency Baseline: 7774207.00 Test: 8126463.00 Difference: +4.5% average latency Baseline: 412569.73 Test: 416977.59 Difference: +1.1% latency standard deviation Baseline: 523805.97 Test: 556261.72 Difference: +6.2% latency standard error Baseline: 32.42 Test: 34.62 Difference: +6.8% org.apache.geode.benchmark.tests.ReplicatedGetLongBenchmark average ops/second Baseline: 1115957.04 Test: 1121417.41 Difference: +0.5% ops/second standard error Baseline: 502.85 Test: 564.03 Difference: +12.2% ops/second standard deviation Baseline: 8695.02 Test: 9752.91 Difference: +12.2% YS 99th percentile latency Baseline: 902.00 Test: 901.00 Difference: -0.1% median latency Baseline: 272895.00 Test: 266239.00 Difference: -2.4% 90th percentile latency Baseline: 412671.00 Test: 411647.00 Difference: -0.2% 99th percentile latency Baseline: 1494015.00 Test: 1508351.00 Difference: +1.0% 99.9th percentile latency Baseline: 4435967.00 Test: 4808703.00 Difference: +8.4% average latency Baseline: 321439.30 Test: 319877.94 Difference: -0.5% latency standard deviation Baseline: 325471.93 Test: 348138.79 Difference: +7.0% latency standard error Baseline: 17.79 Test: 18.98 Difference: +6.7% org.apache.geode.benchmark.tests.ReplicatedIndexedQueryBenchmark average ops/second Baseline: 18228.89 Test: 18244.84 Difference: +0.1% ops/second standard error Baseline: 7.23 Test: 10.20 Difference: +41.2% ops/second standard deviation Baseline: 125.14 Test: 176.40 Difference: +41.0% YS 99th percentile latency Baseline: 20093.46 Test: 20093.42 Difference: -0.0% median latency Baseline: 14000127.00 Test: 14303231.00 Difference: +2.2% 90th percentile latency Baseline: 23330815.00 Test: 22151167.00 Difference: -5.1% 99th percentile latency Baseline: 59703295.00 Test: 55902207.00 Difference: -6.4% 99.9th percentile latency Baseline: 69468159.00 Test: 65863679.00 Difference: -5.2% average latency Baseline: 15793710.81 Test: 15788088.06 Difference: -0.0% latency standard deviation Baseline: 10392676.29 Test: 9415504.26 Difference: -9.4% latency standard error Baseline: 4445.10 Test: 4026.08 Difference: -9.4% org.apache.geode.benchmark.tests.ReplicatedPutAllBenchmark average ops/second Baseline: 669.13 Test: 670.15 Difference: +0.2% ops/second standard error Baseline: 4.77 Test: 4.52 Difference: -5.3% ops/second standard deviation Baseline: 81.85 Test: 77.50 Difference: -5.3% YS 99th percentile latency Baseline: 20099.00 Test: 20099.00 Difference: +0.0% median latency Baseline: 79429631.00 Test: 77070335.00 Difference: -3.0% 90th percentile latency Baseline: 217055231.00 Test: 217448447.00 Difference: +0.2% 99th percentile latency Baseline: 347865087.00 Test: 356515839.00 Difference: +2.5% 99.9th percentile latency Baseline: 405012479.00 Test: 445644799.00 Difference: +10.0% average latency Baseline: 109546138.69 Test: 109117201.07 Difference: -0.4% latency standard deviation Baseline: 75949075.20 Test: 77910756.92 Difference: +2.6% latency standard error Baseline: 171109.43 Test: 175165.73 Difference: +2.4% org.apache.geode.benchmark.tests.ReplicatedPutAllLongBenchmark average ops/second Baseline: 673.75 Test: 681.35 Difference: +1.1% ops/second standard error Baseline: 4.59 Test: 4.81 Difference: +4.7% ops/second standard deviation Baseline: 78.88 Test: 82.76 Difference: +4.9% YS 99th percentile latency Baseline: 20099.00 Test: 20099.00 Difference: +0.0% median latency Baseline: 73334783.00 Test: 74448895.00 Difference: +1.5% 90th percentile latency Baseline: 221380607.00 Test: 215089151.00 Difference: -2.8% 99th percentile latency Baseline: 372244479.00 Test: 355991551.00 Difference: -4.4% 99.9th percentile latency Baseline: 434372607.00 Test: 477626367.00 Difference: +10.0% average latency Baseline: 108819632.70 Test: 107443738.57 Difference: -1.3% latency standard deviation Baseline: 80782678.15 Test: 78388444.16 Difference: -3.0% latency standard error Baseline: 181377.55 Test: 174863.16 Difference: -3.6% org.apache.geode.benchmark.tests.ReplicatedPutBenchmark average ops/second Baseline: 453381.24 Test: 453594.38 Difference: +0.0% ops/second standard error Baseline: 655.88 Test: 608.38 Difference: -7.2% ops/second standard deviation Baseline: 11341.21 Test: 10519.91 Difference: -7.2% YS 99th percentile latency Baseline: 1702.00 Test: 1702.00 Difference: +0.0% median latency Baseline: 525823.00 Test: 524031.00 Difference: -0.3% 90th percentile latency Baseline: 1099775.00 Test: 1114111.00 Difference: +1.3% 99th percentile latency Baseline: 2881535.00 Test: 2961407.00 Difference: +2.8% 99.9th percentile latency Baseline: 77266943.00 Test: 73203711.00 Difference: -5.3% average latency Baseline: 792895.77 Test: 792565.55 Difference: -0.0% latency standard deviation Baseline: 3174758.31 Test: 3045130.83 Difference: -4.1% latency standard error Baseline: 272.24 Test: 261.07 Difference: -4.1% org.apache.geode.benchmark.tests.ReplicatedPutLongBenchmark average ops/second Baseline: 440979.16 Test: 444911.43 Difference: +0.9% ops/second standard error Baseline: 1076.73 Test: 724.78 Difference: -32.7% ops/second standard deviation Baseline: 18618.44 Test: 12532.59 Difference: -32.7% YS 99th percentile latency Baseline: 1702.00 Test: 1702.00 Difference: +0.0% median latency Baseline: 522751.00 Test: 525823.00 Difference: +0.6% 90th percentile latency Baseline: 1124351.00 Test: 1111039.00 Difference: -1.2% 99th percentile latency Baseline: 3012607.00 Test: 2938879.00 Difference: -2.4% 99.9th percentile latency Baseline: 85065727.00 Test: 80347135.00 Difference: -5.5% average latency Baseline: 815307.00 Test: 807777.30 Difference: -0.9% latency standard deviation Baseline: 3621053.66 Test: 3505761.25 Difference: -3.2% latency standard error Baseline: 314.86 Test: 303.47 Difference: -3.6% BENCHMARK FAILED: org.apache.geode.benchmark.tests.PartitionedPutAllBenchmark average latency is 5% worse than baseline. {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005)