[
https://issues.apache.org/jira/browse/GEODE-9340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17385834#comment-17385834
]
Kamilla Aslami commented on GEODE-9340:
---------------------------------------
Re-opening this ticket because of the recent failure in [benchmark-base
#92|https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-main/jobs/benchmark-base/builds/92].
I compared the results of runs #90-#93, and also re-ran the test multiple
times against the commit from run #92, here are the results:
run #93 (commit 198f66489d842994abbcbabd621c3ed6f4d8c730):
{noformat}
average ops/second Baseline: 378109.51 Test: 396779.27 Difference: +4.9%
average latency Baseline: 1902090.07 Test: 1812617.89 Difference: -4.7%
{noformat}
run #92 (commit 198f66489d842994abbcbabd621c3ed6f4d8c730):
{noformat}
average ops/second Baseline: 402166.74 Test: 372212.52 Difference: -7.4%
average latency Baseline: 1788323.03 Test: 1928963.36 Difference: +7.9%
average ops/second Baseline: 406847.58 Test: 381640.51 Difference: -6.2%
average latency Baseline: 1766726.63 Test: 1885246.76 Difference: +6.7%
average ops/second Baseline: 401843.61 Test: 375389.09 Difference: -6.6%
average latency Baseline: 1789773.35 Test: 1914011.63 Difference: +6.9%
average ops/second Baseline: 401843.61 Test: 375389.09 Difference: -6.6%
average latency Baseline: 1789773.35 Test: 1914011.63 Difference: +6.9%
{noformat}
run #91 (commit d77cb31c85112a1ffadb6d2a152b4627216ccc97):
{noformat}
average ops/second Baseline: 407387.94 Test: 381908.21 Difference: -6.3%
average latency Baseline: 1764300.31 Test: 1876359.31 Difference: +6.4%
average ops/second Baseline: 400480.51 Test: 375298.65 Difference: -6.3%
average latency Baseline: 1794889.41 Test: 1916963.34 Difference: +6.8%
average ops/second Baseline: 411374.65 Test: 389488.66 Difference: -5.3%
average latency Baseline: 1749978.64 Test: 1846938.84 Difference: +5.5%
average ops/second Baseline: 403485.67 Test: 392747.92 Difference: -2.7%
average latency Baseline: 1780935.98 Test: 1829789.87 Difference: +2.7%
{noformat}
run #90 (commit 6890221ddba6c60c80796f581dab164f4043b65b):
{noformat}
average ops/second Baseline: 432650.78 Test: 436356.45 Difference: +0.9%
average latency Baseline: 1659669.35 Test: 1646870.95 Difference: -0.8%
{noformat}
My runs against the commit from run #92
(198f66489d842994abbcbabd621c3ed6f4d8c730):
{noformat}
average ops/second Baseline: 470750.22 Test: 456675.50 Difference: -3.0%
average latency Baseline: 1526914.73 Test: 1575478.61 Difference: +3.2%
average ops/second Baseline: 476320.68 Test: 447205.19 Difference: -6.1%
average latency Baseline: 1509780.07 Test: 1607165.71 Difference: +6.5%
average ops/second Baseline: 470806.38 Test: 449991.12 Difference: -4.4%
average latency Baseline: 1528579.11 Test: 1598805.03 Difference: +4.6%
average ops/second Baseline: 469154.56 Test: 464860.14 Difference: -0.9%
average latency Baseline: 1532256.37 Test: 1547035.80 Difference: +1.0%
{noformat}
It looks like there is some instability in this benchmark. Also, I'm not sure
why run #91 didn't fail - 3 out of 4 runs had results >5% worse than baseline.
> Benchmark instability in PartitionedPutLongBenchmark
> ----------------------------------------------------
>
> Key: GEODE-9340
> URL: https://issues.apache.org/jira/browse/GEODE-9340
> Project: Geode
> Issue Type: Bug
> Components: benchmarks
> Affects Versions: 1.15.0
> Reporter: Sarah Abbey
> Assignee: Hale Bales
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.3, 1.13.4, 1.14.0, 1.15.0
>
>
> PartitionedPutLongBenchmark failed in CI
> (https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-main/jobs/benchmark-base/builds/6):
> {code:java}
> This is ITERATION 1 of benchmarking against baseline.
> P2pPartitionedGetBenchmark avg ops/sec
> Baseline: 825011.38 Test: 835847.67 Difference: +1.3%
> avg latency
> Baseline: 871392.31 Test: 859444.66 Difference: -1.4%
> P2pPartitionedPutBenchmark avg ops/sec
> Baseline: 123838.43 Test: 122686.30 Difference: -0.9%
> avg latency
> Baseline: 6015719.73 Test: 6119472.19 Difference: +1.7%
> P2pPartitionedPutBytesBenchmark avg ops/sec
> Baseline: 174887.77 Test: 171040.93 Difference: -2.2%
> avg latency
> Baseline: 4145337.60 Test: 4236159.60 Difference: +2.2%
> PartitionedFunctionExecutionBenchmark avg ops/sec
> Baseline: 248635.36 Test: 261498.94 Difference: +5.2%
> avg latency
> Baseline: 867122.63 Test: 824550.34 Difference: -4.9%
> PartitionedFunctionExecutionWithArgumentsBenchmark avg ops/sec
> Baseline: 280071.19 Test: 275305.31 Difference: -1.7%
> avg latency
> Baseline: 1026643.12 Test: 1044307.43 Difference: +1.7%
> PartitionedFunctionExecutionWithFiltersBenchmark avg ops/sec
> Baseline: 301416.23 Test: 304317.30 Difference: +1.0%
> avg latency
> Baseline: 1908390.88 Test: 1890040.46 Difference: -1.0%
> PartitionedGetBenchmark avg ops/sec
> Baseline: 790800.52 Test: 784514.74 Difference: -0.8%
> avg latency
> Baseline: 908357.58 Test: 915790.96 Difference: +0.8%
> PartitionedGetLongBenchmark avg ops/sec
> Baseline: 1020821.32 Test: 996529.93 Difference: -2.4%
> avg latency
> Baseline: 703761.09 Test: 720744.36 Difference: +2.4%
> PartitionedGetStringBenchmark avg ops/sec
> Baseline: 1028992.93 Test: 1010447.47 Difference: -1.8%
> avg latency
> Baseline: 698009.55 Test: 710765.29 Difference: +1.8%
> PartitionedIndexedQueryBenchmark avg ops/sec
> Baseline: 30868.78 Test: 31478.90 Difference: +2.0%
> avg latency
> Baseline: 18670093.21 Test: 18278083.16 Difference: -2.1%
> PartitionedNonIndexedQueryBenchmark avg ops/sec
> Baseline: 99.45 Test: 101.97 Difference: +2.5%
> avg latency
> Baseline: 723415530.75 Test: 705653061.86 Difference: -2.5%
> PartitionedPutAllBenchmark avg ops/sec
> Baseline: 7921.61 Test: 7816.66 Difference: -1.3%
> avg latency
> Baseline: 18172638.37 Test: 18416169.28 Difference: +1.3%
> PartitionedPutAllLongBenchmark avg ops/sec
> Baseline: 1379.53 Test: 1169.16 Difference: -15.2%
> avg latency
> Baseline: 105140260.44 Test: 123722914.94 Difference: +17.7%
> PartitionedPutBenchmark avg ops/sec
> Baseline: 474986.11 Test: 467924.19 Difference: -1.5%
> avg latency
> Baseline: 1514276.31 Test: 1536263.99 Difference: +1.5%
> PartitionedPutBytesBenchmark avg ops/sec
> Baseline: 457550.69 Test: 456011.33 Difference: -0.3%
> avg latency
> Baseline: 1570713.84 Test: 1575841.02 Difference: +0.3%
> PartitionedPutLongBenchmark avg ops/sec
> Baseline: 418221.79 Test: 389221.70 Difference: -6.9%
> avg latency
> Baseline: 1717869.66 Test: 1849602.96 Difference: +7.7%
> PartitionedPutStringBenchmark avg ops/sec
> Baseline: 410007.93 Test: 390442.31 Difference: -4.8%
> avg latency
> Baseline: 1754915.95 Test: 1841243.74 Difference: +4.9%
> ReplicatedFunctionExecutionBenchmark avg ops/sec
> Baseline: 478780.89 Test: 466855.62 Difference: -2.5%
> avg latency
> Baseline: 449616.85 Test: 461217.05 Difference: +2.6%
> ReplicatedFunctionExecutionWithArgumentsBenchmark avg ops/sec
> Baseline: 504979.76 Test: 501025.09 Difference: -0.8%
> avg latency
> Baseline: 2279046.16 Test: 2296973.89 Difference: +0.8%
> ReplicatedFunctionExecutionWithFiltersBenchmark avg ops/sec
> Baseline: 523232.66 Test: 517943.28 Difference: -1.0%
> avg latency
> Baseline: 1374146.10 Test: 1387969.18 Difference: +1.0%
> ReplicatedGetBenchmark avg ops/sec
> Baseline: 799001.77 Test: 788077.23 Difference: -1.4%
> avg latency
> Baseline: 899092.08 Test: 911932.02 Difference: +1.4%
> ReplicatedGetLongBenchmark avg ops/sec
> Baseline: 1038120.92 Test: 1025267.47 Difference: -1.2%
> avg latency
> Baseline: 691970.73 Test: 700641.58 Difference: +1.3%
> ReplicatedIndexedQueryBenchmark avg ops/sec
> Baseline: 32088.81 Test: 31641.88 Difference: -1.4%
> avg latency
> Baseline: 17960402.47 Test: 18222972.89 Difference: +1.5%
> ReplicatedNonIndexedQueryBenchmark avg ops/sec
> Baseline: 49.31 Test: 48.15 Difference: -2.4%
> avg latency
> Baseline: 1456350672.39 Test: 1490270040.10 Difference: +2.3%
> ReplicatedPutAllBenchmark avg ops/sec
> Baseline: 821.20 Test: 793.66 Difference: -3.4%
> avg latency
> Baseline: 175498155.76 Test: 181712150.69 Difference: +3.5%
> ReplicatedPutAllLongBenchmark avg ops/sec
> Baseline: 855.75 Test: 812.41 Difference: -5.1%
> avg latency
> Baseline: 168343905.55 Test: 177510321.30 Difference: +5.4%
> ReplicatedPutBenchmark avg ops/sec
> Baseline: 485135.10 Test: 481879.66 Difference: -0.7%
> avg latency
> Baseline: 1482039.07 Test: 1491885.72 Difference: +0.7%
> ReplicatedPutLongBenchmark avg ops/sec
> Baseline: 454212.92 Test: 443628.38 Difference: -2.3%
> avg latency
> Baseline: 1583988.26 Test: 1620588.33 Difference: +2.3%
> This is ITERATION 2 of benchmarking against baseline.
> PartitionedPutAllLongBenchmark avg ops/sec
> Baseline: 1359.77 Test: 1263.03 Difference: -7.1%
> avg latency
> Baseline: 106472640.40 Test: 114522255.46 Difference: +7.6%
> PartitionedPutLongBenchmark avg ops/sec
> Baseline: 426518.85 Test: 400573.57 Difference: -6.1%
> avg latency
> Baseline: 1686679.27 Test: 1793189.27 Difference: +6.3%
> ReplicatedPutAllLongBenchmark avg ops/sec
> Baseline: 847.77 Test: 802.20 Difference: -5.4%
> avg latency
> Baseline: 170110197.18 Test: 179656637.86 Difference: +5.6%
> This is ITERATION 3 of benchmarking against baseline.
> PartitionedPutAllLongBenchmark avg ops/sec
> Baseline: 1308.38 Test: 1371.30 Difference: +4.8%
> avg latency
> Baseline: 110742328.38 Test: 105657848.59 Difference: -4.6%
> PartitionedPutLongBenchmark avg ops/sec
> Baseline: 422675.38 Test: 393034.80 Difference: -7.0%
> avg latency
> Baseline: 1701984.68 Test: 1830333.62 Difference: +7.5%
> ReplicatedPutAllLongBenchmark avg ops/sec
> Baseline: 882.11 Test: 868.74 Difference: -1.5%
> avg latency
> Baseline: 163447771.46 Test: 166032265.56 Difference: +1.6%
> This is ITERATION 4 of benchmarking against baseline.
> PartitionedPutLongBenchmark avg ops/sec
> Baseline: 421497.23 Test: 394141.73 Difference: -6.5%
> avg latency
> Baseline: 1705094.48 Test: 1823187.55 Difference: +6.9%
> This is ITERATION 5 of benchmarking against baseline.
> PartitionedPutLongBenchmark avg ops/sec
> Baseline: 410979.61 Test: 387125.91 Difference: -5.8%
> avg latency
> Baseline: 1749420.68 Test: 1854964.43 Difference: +6.0%
> {code}
> Potentially, there is some instability in this benchmark since it passed in
> the runs immediately before and after.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
