andygrove opened a new pull request, #23541:
URL: https://github.com/apache/datafusion/pull/23541
> This PR was created by an LLM as a draft PR. I will mark it as ready for
review after human review.
## Which issue does this PR close?
N/A — autonomous exploratory PR.
## Rationale for this change
Build the trimmed Utf8/LargeUtf8 output by writing slices into a single
exactly-sized value buffer and reusing the input null buffer, instead of
collecting through arrow's FromIterator, which starts its value buffer at 1024
bytes, grows it by repeated doubling, and rebuilds the null bitmap row by row.
## What changes are included in this PR?
Build the trimmed Utf8/LargeUtf8 output by writing slices into a single
exactly-sized value buffer and reusing the input null buffer, instead of
collecting through arrow's FromIterator, which starts its value buffer at 1024
bytes, grows it by repeated doubling, and rebuilds the null bitmap row by row.
## Are these changes tested?
Correctness: unit tests + seeded differential fuzz (bit-identical Arrow
output vs `main`).
Benchmark (criterion):
- ltrim string_view [size=4096, len=64, remaining=60]: -2.285% faster (base
51968ns -> cand 53155ns)
- ltrim large_string [size=4096, len=64, remaining=60]: 10.791% faster (base
70426ns -> cand 62827ns)
- ltrim string [size=4096, len=64, remaining=60]: 15.055% faster (base
71344ns -> cand 60603ns)
- btrim string_view [size=4096, len=64, remaining=60]: 0.182% faster (base
87918ns -> cand 87758ns)
- btrim large_string [size=4096, len=64, remaining=60]: 9.316% faster (base
107290ns -> cand 97295ns)
- rtrim large_string [size=4096, len=64, remaining=60]: 12.753% faster (base
72178ns -> cand 62973ns)
- btrim string [size=4096, len=64, remaining=60]: 9.067% faster (base
105930ns -> cand 96326ns)
- rtrim string_view [size=4096, len=64, remaining=60]: 2.012% faster (base
52285ns -> cand 51233ns)
- rtrim string [size=4096, len=64, remaining=60]: 11.4% faster (base 69425ns
-> cand 61511ns)
- btrim string [size=4096, len=64, remaining=8]: 4.755% faster (base
803627ns -> cand 765412ns)
- btrim large_string [size=4096, len=64, remaining=8]: 4.711% faster (base
802769ns -> cand 764954ns)
- rtrim string_view [size=4096, len=64, remaining=8]: 3.178% faster (base
416499ns -> cand 403262ns)
- ltrim string_view [size=4096, len=64, remaining=8]: -0.766% faster (base
377205ns -> cand 380095ns)
- ltrim large_string [size=4096, len=64, remaining=8]: 3.418% faster (base
385659ns -> cand 372477ns)
- btrim string_view [size=4096, len=64, remaining=8]: -1.094% faster (base
763164ns -> cand 771516ns)
- ltrim string [size=4096, len=64, remaining=8]: 2.05% faster (base 380670ns
-> cand 372867ns)
- rtrim large_string [size=4096, len=64, remaining=8]: 1.891% faster (base
412638ns -> cand 404835ns)
- rtrim string [size=4096, len=64, remaining=8]: -0.213% faster (base
407115ns -> cand 407983ns)
- ltrim string [size=4096, len=12, pad=4]: 15.663% faster (base 24752ns ->
cand 20875ns)
- rtrim string [size=4096, len=12, pad=4]: 19.084% faster (base 25496ns ->
cand 20631ns)
- rtrim string_view [size=4096, len=12, pad=4]: 0.935% faster (base 23452ns
-> cand 23232ns)
- btrim large_string [size=4096, len=16, pad=4]: 22.842% faster (base
36829ns -> cand 28417ns)
- rtrim large_string [size=4096, len=12, pad=4]: 19.476% faster (base
26289ns -> cand 21169ns)
- btrim string [size=4096, len=16, pad=4]: 21.121% faster (base 36734ns ->
cand 28976ns)
- ltrim string_view [size=4096, len=12, pad=4]: -0.663% faster (base 22858ns
-> cand 23010ns)
- ltrim large_string [size=4096, len=12, pad=4]: 18.163% faster (base
25103ns -> cand 20543ns)
- btrim string_view [size=4096, len=16, pad=4]: 1.32% faster (base 34615ns
-> cand 34158ns)
- btrim string_view [size=4096, len=68, pad=4]: 2.079% faster (base 33597ns
-> cand 32898ns)
- ltrim string_view [size=4096, len=64, pad=4]: -2.779% faster (base 21991ns
-> cand 22602ns)
- btrim string [size=4096, len=68, pad=4]: 15.112% faster (base 48987ns ->
cand 41584ns)
- ltrim large_string [size=4096, len=64, pad=4]: 16.025% faster (base
41640ns -> cand 34967ns)
- btrim large_string [size=4096, len=68, pad=4]: 17.146% faster (base
49353ns -> cand 40891ns)
- rtrim string [size=4096, len=64, pad=4]: 21.115% faster (base 42069ns ->
cand 33186ns)
- ltrim string [size=4096, len=64, pad=4]: 19.147% faster (base 41375ns ->
cand 33453ns)
- rtrim string_view [size=4096, len=64, pad=4]: -1.255% faster (base 22190ns
-> cand 22468ns)
- rtrim large_string [size=4096, len=64, pad=4]: 23.5% faster (base 44456ns
-> cand 34008ns)
- rtrim large_string [size=4096, len=12, remaining=8]: 14.652% faster (base
55987ns -> cand 47784ns)
- rtrim string [size=4096, len=12, remaining=8]: 15.777% faster (base
55511ns -> cand 46753ns)
- btrim string_view [size=4096, len=12, remaining=8]: 0.582% faster (base
90871ns -> cand 90343ns)
- ltrim string [size=4096, len=12, remaining=8]: 15.751% faster (base
55374ns -> cand 46652ns)
- ltrim string_view [size=4096, len=12, remaining=8]: 1.227% faster (base
70019ns -> cand 69161ns)
- rtrim string_view [size=4096, len=12, remaining=8]: 2.434% faster (base
54018ns -> cand 52703ns)
- ltrim large_string [size=4096, len=12, remaining=8]: 15.63% faster (base
56321ns -> cand 47518ns)
- btrim string [size=4096, len=12, remaining=8]: 29.843% faster (base
119405ns -> cand 83771ns)
- btrim large_string [size=4096, len=12, remaining=8]: 29.726% faster (base
119617ns -> cand 84060ns)
- btrim string [size=4096, len=120, pad=56]: 5.423% faster (base 134295ns ->
cand 127012ns)
- btrim large_string [size=4096, len=120, pad=56]: 4.731% faster (base
135661ns -> cand 129243ns)
- ltrim large_string [size=4096, len=64, pad=56]: 10.671% faster (base
78904ns -> cand 70483ns)
- ltrim string_view [size=4096, len=64, pad=56]: 0.467% faster (base 76297ns
-> cand 75940ns)
- rtrim large_string [size=4096, len=64, pad=56]: 10.909% faster (base
79353ns -> cand 70696ns)
- rtrim string [size=4096, len=64, pad=56]: 11.069% faster (base 78551ns ->
cand 69856ns)
- btrim string_view [size=4096, len=120, pad=56]: 0.581% faster (base
133912ns -> cand 133135ns)
- rtrim string_view [size=4096, len=64, pad=56]: -0.131% faster (base
78208ns -> cand 78310ns)
- ltrim string [size=4096, len=64, pad=56]: 9.939% faster (base 78138ns ->
cand 70372ns)
Full criterion output:
```text
short strings (len <= 12)/ltrim string_view [size=4096, len=12, remaining=8]
time: [68.772 µs 69.161 µs 69.588 µs]
change: [−2.1338% −1.2266% −0.3627%] (p = 0.02 <
0.05)
Change within noise threshold.
short strings (len <= 12)/ltrim string [size=4096, len=12, remaining=8]
time: [46.637 µs 46.653 µs 46.672 µs]
change: [−16.261% −15.751% −15.300%] (p = 0.00 <
0.05)
Performance has improved.
Found 2 outliers among 10 measurements (20.00%)
1 (10.00%) low mild
1 (10.00%) high severe
short strings (len <= 12)/ltrim large_string [size=4096, len=12, remaining=8]
time: [47.002 µs 47.519 µs 48.116 µs]
change: [−17.005% −15.630% −14.316%] (p = 0.00 <
0.05)
Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high mild
long strings, short trim/ltrim string_view [size=4096, len=64, remaining=60]
time: [52.566 µs 53.156 µs 53.782 µs]
change: [+1.2396% +2.2849% +3.6410%] (p = 0.00 <
0.05)
Performance has regressed.
long strings, short trim/ltrim string [size=4096, len=64, remaining=60]
time: [59.864 µs 60.603 µs 61.344 µs]
change: [−16.396% −15.055% −13.779%] (p = 0.00 <
0.05)
Performance has improved.
long strings, short trim/ltrim large_string [size=4096, len=64, remaining=60]
time: [62.154 µs 62.827 µs 63.567 µs]
change: [−11.916% −10.791% −9.7385%] (p = 0.00 <
0.05)
Performance has improved.
Found 2 outliers among 10 measurements (20.00%)
2 (20.00%) high mild
long strings, long trim/ltrim string_view [size=4096, len=64, remaining=8]
time: [377.82 µs 380.10 µs 382.80 µs]
change: [+0.1236% +0.7661% +1.5272%] (p = 0.02 <
0.05)
Change within noise threshold.
Found 2 outliers among 10 measurements (20.00%)
2 (20.00%) high mild
long strings, long trim/ltrim string [size=4096, len=64, remaining=8]
time: [372.10 µs 372.87 µs 373.80 µs]
change: [−2.2654% −2.0500% −1.7991%] (p = 0.00 <
0.05)
Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high mild
long strings, long trim/ltrim large_string [size=4096, len=64, remaining=8]
time: [372.15 µs 372.48 µs 373.07 µs]
change: [−4.2678% −3.4179% −2.6627%] (p = 0.00 <
0.05)
Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high severe
trim spaces, short strings (len <= 12)/ltrim string_view [size=4096, len=12,
pad=4]
time: [22.970 µs 23.010 µs 23.049 µs]
change: [+0.3046% +0.6633% +1.0373%] (p = 0.00 <
0.05)
Change within noise threshold.
trim spaces, short strings (len <= 12)/ltrim string [size=4096, len=12,
pad=4]
time: [20.827 µs 20.876 µs 20.938 µs]
change: [−15.861% −15.663% −15.422%] (p = 0.00 <
0.05)
Performance has improved.
Found 2 outliers among 10 measurements (20.00%)
1 (10.00%) high mild
1 (10.00%) high severe
trim spaces, short strings (len <= 12)/ltrim large_string [size=4096,
len=12, pad=4]
time: [20.530 µs 20.544 µs 20.561 µs]
change: [−18.501% −18.163% −17.924%] (p = 0.00 <
0.05)
Performance has improved.
Found 2 outliers among 10 measurements (20.00%)
2 (20.00%) high mild
trim spaces, long strings/ltrim string_view [size=4096, len=64, pad=4]
time: [22.593 µs 22.603 µs 22.613 µs]
change: [+2.5048% +2.7791% +3.0854%] (p = 0.00 <
0.05)
Performance has regressed.
trim spaces, long strings/ltrim string [size=4096, len=64, pad=4]
time: [33.443 µs 33.454 µs 33.471 µs]
change: [−19.949% −19.147% −18.364%] (p = 0.00 <
0.05)
Performance has improved.
Found 3 outliers among 10 measurements (30.00%)
1 (10.00%) low mild
2 (20.00%) high severe
trim spaces, long strings/ltrim large_string [size=4096, len=64, pad=4]
time: [34.630 µs 34.967 µs 35.563 µs]
change: [−18.222% −16.025% −13.920%] (p = 0.00 <
0.05)
Performance has improved.
Found 2 outliers among 10 measurements (20.00%)
2 (20.00%) high severe
trim spaces, heavy padding/ltrim string_view [size=4096, len=64, pad=56]
time: [75.906 µs 75.941 µs 76.000 µs]
change: [−1.1903% −0.4672% +0.0722%] (p = 0.19 >
0.05)
No change in performance detected.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high severe
trim spaces, heavy padding/ltrim string [size=4096, len=64, pad=56]
time: [70.315 µs 70.372 µs 70.433 µs]
change: [−10.488% −9.9391% −9.3406%] (p = 0.00 <
0.05)
Performance has improved.
trim spaces, heavy padding/ltrim large_string [size=4096, len=64, pad=56]
time: [70.440 µs 70.484 µs 70.545 µs]
change: [−11.233% −10.671% −10.167%] (p = 0.00 <
0.05)
Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high mild
short strings (len <= 12)/rtrim string_view [size=4096, len=12, remaining=8]
time: [52.475 µs 52.704 µs 52.943 µs]
change: [−3.0207% −2.4340% −1.8783%] (p = 0.00 <
0.05)
Performance has improved.
short strings (len <= 12)/rtrim string [size=4096, len=12, remaining=8]
time: [46.561 µs 46.754 µs 46.949 µs]
change: [−16.647% −15.777% −14.958%] (p = 0.00 <
0.05)
Performance has improved.
short strings (len <= 12)/rtrim large_string [size=4096, len=12, remaining=8]
time: [47.477 µs 47.784 µs 48.152 µs]
change: [−15.255% −14.652% −14.023%] (p = 0.00 <
0.05)
Performance has improved.
Found 2 outliers among 10 measurements (20.00%)
2 (20.00%) high mild
long strings, short trim/rtrim string_view [size=4096, len=64, remaining=60]
time: [51.031 µs 51.233 µs 51.459 µs]
change: [−2.4596% −2.0121% −1.5299%] (p = 0.00 <
0.05)
Performance has improved.
long strings, short trim/rtrim string [size=4096, len=64, remaining=60]
time: [60.549 µs 61.511 µs 62.451 µs]
change: [−13.110% −11.400% −9.7295%] (p = 0.00 <
0.05)
Performance has improved.
long strings, short trim/rtrim large_string [size=4096, len=64, remaining=60]
time: [62.151 µs 62.974 µs 63.706 µs]
change: [−14.250% −12.753% −11.311%] (p = 0.00 <
0.05)
Performance has improved.
long strings, long trim/rtrim string_view [size=4096, len=64, remaining=8]
time: [400.34 µs 403.26 µs 407.94 µs]
change: [−4.9423% −3.1783% −1.3341%] (p = 0.01 <
0.05)
Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high severe
long strings, long trim/rtrim string [size=4096, len=64, remaining=8]
time: [404.95 µs 407.98 µs 411.21 µs]
change: [−1.6939% +0.2132% +1.9796%] (p = 0.83 >
0.05)
No change in performance detected.
long strings, long trim/rtrim large_string [size=4096, len=64, remaining=8]
time: [401.16 µs 404.84 µs 409.18 µs]
change: [−3.2274% −1.8912% −0.5131%] (p = 0.02 <
0.05)
Change within noise threshold.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high mild
trim spaces, short strings (len <= 12)/rtrim string_view [size=4096, len=12,
pad=4]
time: [23.042 µs 23.233 µs 23.388 µs]
change: [−1.8636% −0.9354% −0.1975%] (p = 0.04 <
0.05)
Change within noise threshold.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) low mild
trim spaces, short strings (len <= 12)/rtrim string [size=4096, len=12,
pad=4]
time: [20.621 µs 20.631 µs 20.645 µs]
change: [−19.407% −19.084% −18.841%] (p = 0.00 <
0.05)
Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high mild
trim spaces, short strings (len <= 12)/rtrim large_string [size=4096,
len=12, pad=4]
time: [21.157 µs 21.170 µs 21.185 µs]
change: [−20.204% −19.476% −18.798%] (p = 0.00 <
0.05)
Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high mild
trim spaces, long strings/rtrim string_view [size=4096, len=64, pad=4]
time: [22.455 µs 22.469 µs 22.485 µs]
change: [+0.6989% +1.2553% +1.7874%] (p = 0.00 <
0.05)
Change within noise threshold.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high mild
trim spaces, long strings/rtrim string [size=4096, len=64, pad=4]
time: [33.171 µs 33.186 µs 33.211 µs]
change: [−21.690% −21.115% −20.557%] (p = 0.00 <
0.05)
Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high severe
trim spaces, long strings/rtrim large_string [size=4096, len=64, pad=4]
time: [33.869 µs 34.009 µs 34.146 µs]
change: [−24.131% −23.500% −22.869%] (p = 0.00 <
0.05)
Performance has improved.
trim spaces, heavy padding/rtrim string_view [size=4096, len=64, pad=56]
time: [78.197 µs 78.310 µs 78.440 µs]
change: [−0.7780% +0.1305% +1.0420%] (p = 0.79 >
0.05)
No change in performance detected.
trim spaces, heavy padding/rtrim string [size=4096, len=64, pad=56]
time: [69.823 µs 69.857 µs 69.907 µs]
change: [−11.397% −11.069% −10.765%] (p = 0.00 <
0.05)
Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high severe
trim spaces, heavy padding/rtrim large_string [size=4096, len=64, pad=56]
time: [70.631 µs 70.696 µs 70.773 µs]
change: [−11.374% −10.909% −10.464%] (p = 0.00 <
0.05)
Performance has improved.
short strings (len <= 12)/btrim string_view [size=4096, len=12, remaining=8]
time: [89.699 µs 90.343 µs 91.130 µs]
change: [−1.8718% −0.5816% +0.4785%] (p = 0.42 >
0.05)
No change in performance detected.
Found 2 outliers among 10 measurements (20.00%)
1 (10.00%) high mild
1 (10.00%) high severe
short strings (len <= 12)/btrim string [size=4096, len=12, remaining=8]
time: [83.398 µs 83.771 µs 84.145 µs]
change: [−30.515% −29.843% −29.167%] (p = 0.00 <
0.05)
Performance has improved.
short strings (len <= 12)/btrim large_string [size=4096, len=12, remaining=8]
time: [83.788 µs 84.060 µs 84.366 µs]
change: [−29.968% −29.726% −29.499%] (p = 0.00 <
0.05)
Performance has improved.
long strings, short trim/btrim string_view [size=4096, len=64, remaining=60]
time: [87.545 µs 87.758 µs 88.028 µs]
change: [−0.5010% −0.1823% +0.1714%] (p = 0.34 >
0.05)
No change in performance detected.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high mild
long strings, short trim/btrim string [size=4096, len=64, remaining=60]
time: [96.102 µs 96.326 µs 96.558 µs]
change: [−9.3462% −9.0668% −8.7659%] (p = 0.00 <
0.05)
Performance has improved.
long strings, short trim/btrim large_string [size=4096, len=64, remaining=60]
time: [97.185 µs 97.296 µs 97.406 µs]
change: [−9.6125% −9.3157% −9.0282%] (p = 0.00 <
0.05)
Performance has improved.
long strings, long trim/btrim string_view [size=4096, len=64, remaining=8]
time: [770.37 µs 771.52 µs 772.90 µs]
change: [+0.6693% +1.0943% +1.4767%] (p = 0.00 <
0.05)
Change within noise threshold.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high mild
long strings, long trim/btrim string [size=4096, len=64, remaining=8]
time: [763.92 µs 765.41 µs 766.99 µs]
change: [−5.0952% −4.7554% −4.4481%] (p = 0.00 <
0.05)
Performance has improved.
long strings, long trim/btrim large_string [size=4096, len=64, remaining=8]
time: [764.37 µs 764.95 µs 765.72 µs]
change: [−4.8509% −4.7107% −4.5654%] (p = 0.00 <
0.05)
Performance has improved.
Found 2 outliers among 10 measurements (20.00%)
2 (20.00%) high severe
trim spaces, short strings (len <= 12)/btrim string_view [size=4096, len=16,
pad=4]
time: [34.141 µs 34.159 µs 34.179 µs]
change: [−1.3904% −1.3204% −1.2474%] (p = 0.00 <
0.05)
Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high mild
trim spaces, short strings (len <= 12)/btrim string [size=4096, len=16,
pad=4]
time: [28.965 µs 28.976 µs 28.990 µs]
change: [−21.262% −21.121% −20.969%] (p = 0.00 <
0.05)
Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high mild
trim spaces, short strings (len <= 12)/btrim large_string [size=4096,
len=16, pad=4]
time: [28.235 µs 28.417 µs 28.634 µs]
change: [−23.842% −22.842% −21.895%] (p = 0.00 <
0.05)
Performance has improved.
Found 3 outliers among 10 measurements (30.00%)
1 (10.00%) low mild
2 (20.00%) high severe
trim spaces, long strings/btrim string_view [size=4096, len=68, pad=4]
time: [32.686 µs 32.899 µs 33.100 µs]
change: [−3.4430% −2.0786% −0.9461%] (p = 0.00 <
0.05)
Change within noise threshold.
trim spaces, long strings/btrim string [size=4096, len=68, pad=4]
time: [41.553 µs 41.585 µs 41.622 µs]
change: [−15.330% −15.112% −14.914%] (p = 0.00 <
0.05)
Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high mild
trim spaces, long strings/btrim large_string [size=4096, len=68, pad=4]
time: [40.584 µs 40.891 µs 41.236 µs]
change: [−17.802% −17.146% −16.404%] (p = 0.00 <
0.05)
Performance has improved.
trim spaces, heavy padding/btrim string_view [size=4096, len=120, pad=56]
time: [133.07 µs 133.14 µs 133.23 µs]
change: [−0.6635% −0.5806% −0.4984%] (p = 0.00 <
0.05)
Change within noise threshold.
Found 2 outliers among 10 measurements (20.00%)
2 (20.00%) high severe
trim spaces, heavy padding/btrim string [size=4096, len=120, pad=56]
time: [126.90 µs 127.01 µs 127.15 µs]
change: [−5.5277% −5.4230% −5.2985%] (p = 0.00 <
0.05)
Performance has improved.
Found 2 outliers among 10 measurements (20.00%)
1 (10.00%) high mild
1 (10.00%) high severe
trim spaces, heavy padding/btrim large_string [size=4096, len=120, pad=56]
time: [128.87 µs 129.24 µs 129.72 µs]
change: [−5.0133% −4.7314% −4.3817%] (p = 0.00 <
0.05)
Performance has improved.
Found 2 outliers among 10 measurements (20.00%)
1 (10.00%) high mild
1 (10.00%) high severe
```
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be
updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api
change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
