mustafasrepo commented on PR #9346: URL: https://github.com/apache/arrow-datafusion/pull/9346#issuecomment-1968865400
> I was curious on how this affected planning performance. Here are my results comparing [0c46d7f](https://github.com/apache/arrow-datafusion/commit/0c46d7fa105fddc4a35a4c99e4aa2a063d967abb) (parent of first commit in this PR) with [a8fac85](https://github.com/apache/arrow-datafusion/commit/a8fac85429f519ea5ae258a2a6425eaa9ab333c8). > > Seems like there are quite big regressions in physical planning. But I may have messed up. > > Results I re-ran benchmarks in my machine. The results are below <details> <summary>Results</summary> ``` Gnuplot not found, using plotters backend logical_select_one_from_700 time: [482.54 µs 484.25 µs 487.11 µs] change: [-4.6292% -2.1797% +0.0154%] (p = 0.08 > 0.05) No change in performance detected. Found 8 outliers among 100 measurements (8.00%) 2 (2.00%) low severe 2 (2.00%) low mild 3 (3.00%) high mild 1 (1.00%) high severe physical_select_one_from_700 time: [3.0204 ms 3.0227 ms 3.0250 ms] change: [-2.3949% -2.0254% -1.7849%] (p = 0.00 < 0.05) Performance has improved. logical_trivial_join_low_numbered_columns time: [452.55 µs 453.30 µs 454.05 µs] change: [-3.4288% -1.6448% -0.6310%] (p = 0.02 < 0.05) Change within noise threshold. Found 5 outliers among 100 measurements (5.00%) 1 (1.00%) low mild 2 (2.00%) high mild 2 (2.00%) high severe logical_trivial_join_high_numbered_columns time: [495.26 µs 498.76 µs 505.52 µs] change: [-0.4885% +0.0379% +0.7595%] (p = 0.93 > 0.05) No change in performance detected. Found 4 outliers among 100 measurements (4.00%) 2 (2.00%) low severe 1 (1.00%) high mild 1 (1.00%) high severe logical_aggregate_with_join time: [769.12 µs 769.87 µs 770.62 µs] change: [-18.781% -12.487% -6.5679%] (p = 0.00 < 0.05) Performance has improved. Found 7 outliers among 100 measurements (7.00%) 3 (3.00%) low mild 3 (3.00%) high mild 1 (1.00%) high severe physical_plan_tpch_q1 time: [4.8582 ms 4.8620 ms 4.8663 ms] change: [-17.477% -11.210% -6.1042%] (p = 0.00 < 0.05) Performance has improved. Found 5 outliers among 100 measurements (5.00%) 2 (2.00%) high mild 3 (3.00%) high severe physical_plan_tpch_q2 time: [7.5602 ms 7.5707 ms 7.5822 ms] change: [-30.715% -30.328% -30.007%] (p = 0.00 < 0.05) Performance has improved. Found 8 outliers among 100 measurements (8.00%) 6 (6.00%) high mild 2 (2.00%) high severe physical_plan_tpch_q3 time: [2.5096 ms 2.5143 ms 2.5196 ms] change: [-25.111% -24.050% -23.386%] (p = 0.00 < 0.05) Performance has improved. Found 12 outliers among 100 measurements (12.00%) 9 (9.00%) high mild 3 (3.00%) high severe physical_plan_tpch_q4 time: [1.9527 ms 1.9591 ms 1.9711 ms] change: [-14.387% -14.094% -13.679%] (p = 0.00 < 0.05) Performance has improved. Found 7 outliers among 100 measurements (7.00%) 2 (2.00%) high mild 5 (5.00%) high severe physical_plan_tpch_q5 time: [3.8057 ms 3.8091 ms 3.8127 ms] change: [-57.720% -57.662% -57.601%] (p = 0.00 < 0.05) Performance has improved. Found 5 outliers among 100 measurements (5.00%) 4 (4.00%) high mild 1 (1.00%) high severe physical_plan_tpch_q6 time: [1.3063 ms 1.3080 ms 1.3099 ms] change: [-5.6737% -5.1326% -4.7813%] (p = 0.00 < 0.05) Performance has improved. Found 6 outliers among 100 measurements (6.00%) 2 (2.00%) low mild 3 (3.00%) high mild 1 (1.00%) high severe physical_plan_tpch_q7 time: [5.4017 ms 5.4307 ms 5.4840 ms] change: [-44.042% -43.729% -43.180%] (p = 0.00 < 0.05) Performance has improved. Found 8 outliers among 100 measurements (8.00%) 6 (6.00%) high mild 2 (2.00%) high severe physical_plan_tpch_q8 time: [7.9176 ms 7.9289 ms 7.9414 ms] change: [-71.699% -71.653% -71.598%] (p = 0.00 < 0.05) Performance has improved. Found 9 outliers among 100 measurements (9.00%) 1 (1.00%) low mild 5 (5.00%) high mild 3 (3.00%) high severe physical_plan_tpch_q9 time: [5.9427 ms 5.9797 ms 6.0430 ms] change: [-47.161% -46.493% -45.740%] (p = 0.00 < 0.05) Performance has improved. Found 10 outliers among 100 measurements (10.00%) 3 (3.00%) high mild 7 (7.00%) high severe physical_plan_tpch_q10 time: [3.8350 ms 3.8388 ms 3.8430 ms] change: [-31.229% -31.025% -30.862%] (p = 0.00 < 0.05) Performance has improved. Found 3 outliers among 100 measurements (3.00%) 2 (2.00%) high mild 1 (1.00%) high severe physical_plan_tpch_q11 time: [2.9252 ms 2.9415 ms 2.9715 ms] change: [-16.985% -15.900% -14.659%] (p = 0.00 < 0.05) Performance has improved. Found 4 outliers among 100 measurements (4.00%) 1 (1.00%) high mild 3 (3.00%) high severe physical_plan_tpch_q12 time: [2.6822 ms 2.6845 ms 2.6870 ms] change: [-13.733% -13.572% -13.424%] (p = 0.00 < 0.05) Performance has improved. Found 4 outliers among 100 measurements (4.00%) 4 (4.00%) high mild physical_plan_tpch_q13 time: [1.5018 ms 1.5033 ms 1.5048 ms] change: [-24.810% -24.629% -24.402%] (p = 0.00 < 0.05) Performance has improved. Found 9 outliers among 100 measurements (9.00%) 1 (1.00%) low mild 5 (5.00%) high mild 3 (3.00%) high severe physical_plan_tpch_q14 time: [2.1145 ms 2.1264 ms 2.1480 ms] change: [-4.5956% -3.8425% -2.7007%] (p = 0.00 < 0.05) Performance has improved. Found 3 outliers among 100 measurements (3.00%) 1 (1.00%) high mild 2 (2.00%) high severe physical_plan_tpch_q16 time: [3.1242 ms 3.1278 ms 3.1319 ms] change: [-17.445% -17.334% -17.217%] (p = 0.00 < 0.05) Performance has improved. Found 4 outliers among 100 measurements (4.00%) 2 (2.00%) high mild 2 (2.00%) high severe physical_plan_tpch_q17 time: [2.8500 ms 2.8705 ms 2.8990 ms] change: [-8.0314% -7.2135% -6.2413%] (p = 0.00 < 0.05) Performance has improved. Found 6 outliers among 100 measurements (6.00%) 3 (3.00%) high mild 3 (3.00%) high severe physical_plan_tpch_q18 time: [3.0984 ms 3.1064 ms 3.1175 ms] change: [-10.790% -10.508% -10.196%] (p = 0.00 < 0.05) Performance has improved. Found 5 outliers among 100 measurements (5.00%) 3 (3.00%) high mild 2 (2.00%) high severe physical_plan_tpch_q19 time: [7.8895 ms 7.9316 ms 8.0083 ms] change: [-2.4917% -1.9490% -0.7408%] (p = 0.00 < 0.05) Change within noise threshold. Found 7 outliers among 100 measurements (7.00%) 6 (6.00%) high mild 1 (1.00%) high severe physical_plan_tpch_q20 time: [3.7943 ms 3.7987 ms 3.8037 ms] change: [-19.417% -17.933% -16.538%] (p = 0.00 < 0.05) Performance has improved. Found 4 outliers among 100 measurements (4.00%) 2 (2.00%) high mild 2 (2.00%) high severe physical_plan_tpch_q21 time: [5.7239 ms 5.7279 ms 5.7320 ms] change: [-37.177% -36.725% -36.397%] (p = 0.00 < 0.05) Performance has improved. Found 4 outliers among 100 measurements (4.00%) 4 (4.00%) high mild physical_plan_tpch_q22 time: [2.6413 ms 2.6610 ms 2.6942 ms] change: [-14.662% -13.905% -12.725%] (p = 0.00 < 0.05) Performance has improved. Found 9 outliers among 100 measurements (9.00%) 5 (5.00%) high mild 4 (4.00%) high severe physical_plan_tpch_all time: [9.3429 ms 9.3899 ms 9.4571 ms] change: [-2.7310% -1.5545% -0.5941%] (p = 0.00 < 0.05) Change within noise threshold. Found 6 outliers among 100 measurements (6.00%) 2 (2.00%) low mild 2 (2.00%) high mild 2 (2.00%) high severe ``` </details> According to these results, new mechanism mostly improves planning performance (especially for complex queries). since our results are contradicting. I re-ran the same benchmark in main branch twice, and in new branch twice. Then compared these results with themselves to see how much variance benchmark run produces at different runs. Below results for the different benchmarks for the same branch can be found <details> <summary>Main Run 2 vs Main Run 1</summary> ``` Gnuplot not found, using plotters backend logical_select_one_from_700 time: [490.30 µs 505.86 µs 525.48 µs] change: [-2.4316% +0.6167% +3.2873%] (p = 0.71 > 0.05) No change in performance detected. Found 5 outliers among 100 measurements (5.00%) 5 (5.00%) high severe physical_select_one_from_700 time: [3.0782 ms 3.0852 ms 3.0962 ms] change: [+1.0064% +1.2946% +1.6436%] (p = 0.00 < 0.05) Performance has regressed. Found 2 outliers among 100 measurements (2.00%) 1 (1.00%) high mild 1 (1.00%) high severe logical_trivial_join_low_numbered_columns time: [457.67 µs 458.60 µs 459.94 µs] change: [+0.3580% +1.3982% +3.7624%] (p = 0.06 > 0.05) No change in performance detected. Found 7 outliers among 100 measurements (7.00%) 4 (4.00%) low mild 2 (2.00%) high mild 1 (1.00%) high severe logical_trivial_join_high_numbered_columns time: [496.65 µs 497.61 µs 498.80 µs] change: [-0.2179% +0.0540% +0.3247%] (p = 0.70 > 0.05) No change in performance detected. Found 3 outliers among 100 measurements (3.00%) 1 (1.00%) low severe 1 (1.00%) high mild 1 (1.00%) high severe logical_aggregate_with_join time: [826.96 µs 908.94 µs 1.0125 ms] change: [+6.3560% +14.209% +23.294%] (p = 0.00 < 0.05) Performance has regressed. Found 15 outliers among 100 measurements (15.00%) 2 (2.00%) high mild 13 (13.00%) high severe physical_plan_tpch_q1 time: [5.1788 ms 5.4759 ms 5.8885 ms] change: [+1.3260% +7.2118% +15.244%] (p = 0.02 < 0.05) Performance has regressed. Found 10 outliers among 100 measurements (10.00%) 3 (3.00%) high mild 7 (7.00%) high severe physical_plan_tpch_q2 time: [10.819 ms 10.866 ms 10.924 ms] change: [-0.1234% +1.1456% +2.1468%] (p = 0.04 < 0.05) Change within noise threshold. Found 5 outliers among 100 measurements (5.00%) 3 (3.00%) high mild 2 (2.00%) high severe physical_plan_tpch_q3 time: [3.2836 ms 3.3104 ms 3.3553 ms] change: [-2.3241% -0.9046% +0.8137%] (p = 0.29 > 0.05) No change in performance detected. Found 14 outliers among 100 measurements (14.00%) 6 (6.00%) high mild 8 (8.00%) high severe physical_plan_tpch_q4 time: [2.2800 ms 2.2830 ms 2.2862 ms] change: [-1.4797% -1.0937% -0.7281%] (p = 0.00 < 0.05) Change within noise threshold. Found 3 outliers among 100 measurements (3.00%) 1 (1.00%) high mild 2 (2.00%) high severe physical_plan_tpch_q5 time: [8.9876 ms 8.9969 ms 9.0067 ms] change: [-1.4174% -0.7961% -0.2552%] (p = 0.00 < 0.05) Change within noise threshold. Found 4 outliers among 100 measurements (4.00%) 4 (4.00%) high mild physical_plan_tpch_q6 time: [1.3748 ms 1.3814 ms 1.3940 ms] change: [-9.3756% -5.6928% -2.5053%] (p = 0.00 < 0.05) Performance has improved. Found 6 outliers among 100 measurements (6.00%) 1 (1.00%) low mild 3 (3.00%) high mild 2 (2.00%) high severe physical_plan_tpch_q7 time: [9.6410 ms 9.6510 ms 9.6617 ms] change: [-0.9565% -0.4781% -0.0663%] (p = 0.03 < 0.05) Change within noise threshold. Found 4 outliers among 100 measurements (4.00%) 3 (3.00%) high mild 1 (1.00%) high severe physical_plan_tpch_q8 time: [27.948 ms 27.971 ms 27.993 ms] change: [-0.2163% +0.0701% +0.3352%] (p = 0.62 > 0.05) No change in performance detected. Found 1 outliers among 100 measurements (1.00%) 1 (1.00%) high mild physical_plan_tpch_q9 time: [11.114 ms 11.176 ms 11.285 ms] change: [-0.2915% +0.4000% +1.4525%] (p = 0.49 > 0.05) No change in performance detected. Found 3 outliers among 100 measurements (3.00%) 3 (3.00%) high severe physical_plan_tpch_q10 time: [5.5536 ms 5.5655 ms 5.5812 ms] change: [-0.6752% -0.3443% +0.0157%] (p = 0.04 < 0.05) Change within noise threshold. Found 6 outliers among 100 measurements (6.00%) 4 (4.00%) high mild 2 (2.00%) high severe physical_plan_tpch_q11 time: [3.4787 ms 3.4976 ms 3.5313 ms] change: [-1.3654% -0.6526% +0.4886%] (p = 0.18 > 0.05) No change in performance detected. Found 9 outliers among 100 measurements (9.00%) 3 (3.00%) high mild 6 (6.00%) high severe physical_plan_tpch_q12 time: [3.1018 ms 3.1061 ms 3.1110 ms] change: [-0.7425% -0.4918% -0.2454%] (p = 0.00 < 0.05) Change within noise threshold. Found 4 outliers among 100 measurements (4.00%) 2 (2.00%) high mild 2 (2.00%) high severe physical_plan_tpch_q13 time: [1.9962 ms 1.9986 ms 2.0010 ms] change: [-0.4561% -0.2516% -0.0440%] (p = 0.02 < 0.05) Change within noise threshold. Found 3 outliers among 100 measurements (3.00%) 2 (2.00%) high mild 1 (1.00%) high severe physical_plan_tpch_q14 time: [2.2037 ms 2.2113 ms 2.2203 ms] change: [-0.4389% +0.0583% +0.5657%] (p = 0.82 > 0.05) No change in performance detected. Found 12 outliers among 100 measurements (12.00%) 5 (5.00%) high mild 7 (7.00%) high severe physical_plan_tpch_q16 time: [3.7811 ms 3.7837 ms 3.7864 ms] change: [-0.4859% -0.3382% -0.1973%] (p = 0.00 < 0.05) Change within noise threshold. Found 4 outliers among 100 measurements (4.00%) 4 (4.00%) high mild physical_plan_tpch_q17 time: [3.0798 ms 3.0936 ms 3.1076 ms] change: [+0.5410% +1.5036% +2.2740%] (p = 0.00 < 0.05) Change within noise threshold. physical_plan_tpch_q18 time: [3.4659 ms 3.4712 ms 3.4769 ms] change: [-0.1817% +0.0928% +0.3386%] (p = 0.50 > 0.05) No change in performance detected. Found 6 outliers among 100 measurements (6.00%) 3 (3.00%) high mild 3 (3.00%) high severe physical_plan_tpch_q19 time: [8.0817 ms 8.0892 ms 8.0969 ms] change: [-0.7482% -0.4924% -0.2533%] (p = 0.00 < 0.05) Change within noise threshold. Found 3 outliers among 100 measurements (3.00%) 3 (3.00%) high mild physical_plan_tpch_q20 time: [4.5514 ms 4.6288 ms 4.7138 ms] change: [+5.5386% +7.4224% +9.4528%] (p = 0.00 < 0.05) Performance has regressed. Found 9 outliers among 100 measurements (9.00%) 5 (5.00%) high mild 4 (4.00%) high severe physical_plan_tpch_q21 time: [9.0060 ms 9.0524 ms 9.1166 ms] change: [-0.0386% +0.5447% +1.3691%] (p = 0.12 > 0.05) No change in performance detected. Found 17 outliers among 100 measurements (17.00%) 11 (11.00%) high mild 6 (6.00%) high severe physical_plan_tpch_q22 time: [3.0796 ms 3.0908 ms 3.1033 ms] change: [-0.0510% +0.3826% +0.7615%] (p = 0.08 > 0.05) No change in performance detected. Found 12 outliers among 100 measurements (12.00%) 2 (2.00%) high mild 10 (10.00%) high severe physical_plan_tpch_all time: [9.4791 ms 9.5382 ms 9.6367 ms] change: [+0.6728% +1.7514% +2.8703%] (p = 0.00 < 0.05) Change within noise threshold. Found 6 outliers among 100 measurements (6.00%) 1 (1.00%) low mild 3 (3.00%) high mild 2 (2.00%) high severe ``` </details> <details> <summary>Branch Run 2 vs Branch Run 1</summary> ``` Gnuplot not found, using plotters backend logical_select_one_from_700 time: [482.54 µs 484.25 µs 487.16 µs] change: [+0.1199% +1.0534% +2.4753%] (p = 0.06 > 0.05) No change in performance detected. Found 8 outliers among 100 measurements (8.00%) 2 (2.00%) low severe 2 (2.00%) low mild 3 (3.00%) high mild 1 (1.00%) high severe physical_select_one_from_700 time: [3.0204 ms 3.0227 ms 3.0250 ms] change: [-1.9446% -1.8220% -1.6973%] (p = 0.00 < 0.05) Performance has improved. logical_trivial_join_low_numbered_columns time: [452.55 µs 453.30 µs 454.05 µs] change: [-1.7532% -1.1174% -0.6057%] (p = 0.00 < 0.05) Change within noise threshold. Found 5 outliers among 100 measurements (5.00%) 1 (1.00%) low mild 2 (2.00%) high mild 2 (2.00%) high severe logical_trivial_join_high_numbered_columns time: [495.25 µs 498.76 µs 505.57 µs] change: [-1.0315% -0.4986% +0.3472%] (p = 0.15 > 0.05) No change in performance detected. Found 4 outliers among 100 measurements (4.00%) 2 (2.00%) low severe 1 (1.00%) high mild 1 (1.00%) high severe logical_aggregate_with_join time: [769.12 µs 769.87 µs 770.63 µs] change: [-0.7240% -0.4668% -0.2220%] (p = 0.00 < 0.05) Change within noise threshold. Found 7 outliers among 100 measurements (7.00%) 3 (3.00%) low mild 3 (3.00%) high mild 1 (1.00%) high severe physical_plan_tpch_q1 time: [4.8582 ms 4.8620 ms 4.8663 ms] change: [-1.5189% -1.3731% -1.2333%] (p = 0.00 < 0.05) Performance has improved. Found 5 outliers among 100 measurements (5.00%) 2 (2.00%) high mild 3 (3.00%) high severe physical_plan_tpch_q2 time: [7.5602 ms 7.5707 ms 7.5822 ms] change: [-1.4182% -1.1973% -0.9798%] (p = 0.00 < 0.05) Change within noise threshold. Found 8 outliers among 100 measurements (8.00%) 6 (6.00%) high mild 2 (2.00%) high severe physical_plan_tpch_q3 time: [2.5095 ms 2.5143 ms 2.5196 ms] change: [-1.1784% -0.7929% -0.4292%] (p = 0.00 < 0.05) Change within noise threshold. Found 12 outliers among 100 measurements (12.00%) 9 (9.00%) high mild 3 (3.00%) high severe physical_plan_tpch_q4 time: [1.9527 ms 1.9591 ms 1.9711 ms] change: [-20.849% -15.072% -9.8504%] (p = 0.00 < 0.05) Performance has improved. Found 7 outliers among 100 measurements (7.00%) 2 (2.00%) high mild 5 (5.00%) high severe physical_plan_tpch_q5 time: [3.8057 ms 3.8091 ms 3.8127 ms] change: [-1.1569% -0.9889% -0.8191%] (p = 0.00 < 0.05) Change within noise threshold. Found 5 outliers among 100 measurements (5.00%) 4 (4.00%) high mild 1 (1.00%) high severe physical_plan_tpch_q6 time: [1.3063 ms 1.3080 ms 1.3099 ms] change: [-3.3001% -2.3030% -1.4936%] (p = 0.00 < 0.05) Performance has improved. Found 6 outliers among 100 measurements (6.00%) 2 (2.00%) low mild 3 (3.00%) high mild 1 (1.00%) high severe physical_plan_tpch_q7 time: [5.4017 ms 5.4307 ms 5.4840 ms] change: [-0.8280% -0.1612% +0.8696%] (p = 0.81 > 0.05) No change in performance detected. Found 8 outliers among 100 measurements (8.00%) 6 (6.00%) high mild 2 (2.00%) high severe physical_plan_tpch_q8 time: [7.9177 ms 7.9289 ms 7.9413 ms] change: [-1.5994% -1.2327% -0.9253%] (p = 0.00 < 0.05) Change within noise threshold. Found 9 outliers among 100 measurements (9.00%) 1 (1.00%) low mild 5 (5.00%) high mild 3 (3.00%) high severe physical_plan_tpch_q9 time: [5.9426 ms 5.9797 ms 6.0429 ms] change: [-0.3560% +0.2891% +1.2648%] (p = 0.67 > 0.05) No change in performance detected. Found 10 outliers among 100 measurements (10.00%) 3 (3.00%) high mild 7 (7.00%) high severe physical_plan_tpch_q10 time: [3.8351 ms 3.8388 ms 3.8430 ms] change: [-0.5106% -0.3235% -0.1468%] (p = 0.00 < 0.05) Change within noise threshold. Found 3 outliers among 100 measurements (3.00%) 2 (2.00%) high mild 1 (1.00%) high severe physical_plan_tpch_q11 time: [2.9252 ms 2.9415 ms 2.9717 ms] change: [-2.8743% -1.1814% +0.3357%] (p = 0.17 > 0.05) No change in performance detected. Found 4 outliers among 100 measurements (4.00%) 1 (1.00%) high mild 3 (3.00%) high severe physical_plan_tpch_q12 time: [2.6821 ms 2.6845 ms 2.6870 ms] change: [-0.4900% -0.3172% -0.1498%] (p = 0.00 < 0.05) Change within noise threshold. Found 4 outliers among 100 measurements (4.00%) 4 (4.00%) high mild physical_plan_tpch_q13 time: [1.5018 ms 1.5033 ms 1.5048 ms] change: [-2.7994% -1.2002% -0.2305%] (p = 0.07 > 0.05) No change in performance detected. Found 9 outliers among 100 measurements (9.00%) 1 (1.00%) low mild 5 (5.00%) high mild 3 (3.00%) high severe physical_plan_tpch_q14 time: [2.1146 ms 2.1264 ms 2.1480 ms] change: [-0.6216% +0.0295% +1.0359%] (p = 0.96 > 0.05) No change in performance detected. Found 3 outliers among 100 measurements (3.00%) 1 (1.00%) high mild 2 (2.00%) high severe physical_plan_tpch_q16 time: [3.1242 ms 3.1278 ms 3.1318 ms] change: [-0.7674% -0.3833% -0.0720%] (p = 0.02 < 0.05) Change within noise threshold. Found 4 outliers among 100 measurements (4.00%) 2 (2.00%) high mild 2 (2.00%) high severe physical_plan_tpch_q17 time: [2.8501 ms 2.8705 ms 2.8994 ms] change: [-0.1687% +0.6001% +1.6077%] (p = 0.22 > 0.05) No change in performance detected. Found 6 outliers among 100 measurements (6.00%) 3 (3.00%) high mild 3 (3.00%) high severe physical_plan_tpch_q18 time: [3.0984 ms 3.1064 ms 3.1174 ms] change: [-0.3335% -0.0200% +0.3375%] (p = 0.92 > 0.05) No change in performance detected. Found 5 outliers among 100 measurements (5.00%) 3 (3.00%) high mild 2 (2.00%) high severe physical_plan_tpch_q19 time: [7.8894 ms 7.9316 ms 8.0084 ms] change: [-0.1952% +0.3957% +1.3828%] (p = 0.54 > 0.05) No change in performance detected. Found 7 outliers among 100 measurements (7.00%) 6 (6.00%) high mild 1 (1.00%) high severe physical_plan_tpch_q20 time: [3.7943 ms 3.7987 ms 3.8037 ms] change: [-0.4696% -0.2517% -0.0323%] (p = 0.02 < 0.05) Change within noise threshold. Found 4 outliers among 100 measurements (4.00%) 2 (2.00%) high mild 2 (2.00%) high severe physical_plan_tpch_q21 time: [5.7238 ms 5.7279 ms 5.7321 ms] change: [-0.3685% -0.2011% -0.0428%] (p = 0.02 < 0.05) Change within noise threshold. Found 4 outliers among 100 measurements (4.00%) 4 (4.00%) high mild physical_plan_tpch_q22 time: [2.6413 ms 2.6610 ms 2.6939 ms] change: [-0.2031% +0.5955% +1.8973%] (p = 0.34 > 0.05) No change in performance detected. Found 9 outliers among 100 measurements (9.00%) 5 (5.00%) high mild 4 (4.00%) high severe physical_plan_tpch_all time: [9.3429 ms 9.3899 ms 9.4570 ms] change: [-0.5264% +0.0285% +0.7099%] (p = 0.95 > 0.05) No change in performance detected. Found 6 outliers among 100 measurements (6.00%) 2 (2.00%) low mild 2 (2.00%) high mild 2 (2.00%) high severe ``` </details> according to these results, there is definitely some noise across different runs. However, they are generally within 10%. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
