marcin-krystianc commented on PR #50823:
URL: https://github.com/apache/arrow/pull/50823#issuecomment-6421824924
Thanks for the review, @adamreeve.
The slowdown in the non-dense `FLBA` path was real. In the previous version,
`DecodePointers` decoded every value into buffered_data_ with `DecodeDense` and
then pointed into it, so every value was copied. This is fixed, and
`Decode(FixedLenByteArray*)` now goes through the same `GetInternal` path as
`BYTE_ARRAY`, so it only copies values with both a prefix and a suffix. Other
values point to their suffix in the page or to the previous value.
Now, also added `ArrowOutput`, so `BYTE_ARRAY` and `FLBA DecodeArrow` no
longer decode into a temporary `ByteArray` buffer. They append each value
straight to the builder.
<details>
<summary> Benchmarks, main vs this PR </summary>
```
--------------------------------------------------------------------------------------------------------------------------------------
Benchmark
main PR
--------------------------------------------------------------------------------------------------------------------------------------
BM_DeltaDecodingByteArray/max-string-length:8/batch-size:512/prefixed-percent:10
2231 ns 1878 ns
BM_DeltaDecodingByteArray/max-string-length:8/batch-size:512/prefixed-percent:90
3318 ns 2849 ns
BM_DeltaDecodingByteArray/max-string-length:8/batch-size:512/prefixed-percent:99
3399 ns 2949 ns
BM_DeltaDecodingByteArray/max-string-length:8/batch-size:2048/prefixed-percent:10
8942 ns 6974 ns
BM_DeltaDecodingByteArray/max-string-length:8/batch-size:2048/prefixed-percent:90
13059 ns 11140 ns
BM_DeltaDecodingByteArray/max-string-length:8/batch-size:2048/prefixed-percent:99
13142 ns 11584 ns
BM_DeltaDecodingByteArray/max-string-length:64/batch-size:512/prefixed-percent:10
2164 ns 1860 ns
BM_DeltaDecodingByteArray/max-string-length:64/batch-size:512/prefixed-percent:90
3697 ns 3351 ns
BM_DeltaDecodingByteArray/max-string-length:64/batch-size:512/prefixed-percent:99
3853 ns 3504 ns
BM_DeltaDecodingByteArray/max-string-length:64/batch-size:2048/prefixed-percent:10
8634 ns 7586 ns
BM_DeltaDecodingByteArray/max-string-length:64/batch-size:2048/prefixed-percent:90
14722 ns 13340 ns
BM_DeltaDecodingByteArray/max-string-length:64/batch-size:2048/prefixed-percent:99
15734 ns 14378 ns
BM_DeltaDecodingByteArray/max-string-length:1024/batch-size:512/prefixed-percent:10
2666 ns 2405 ns
BM_DeltaDecodingByteArray/max-string-length:1024/batch-size:512/prefixed-percent:90
7840 ns 7474 ns
BM_DeltaDecodingByteArray/max-string-length:1024/batch-size:512/prefixed-percent:99
8681 ns 8129 ns
BM_DeltaDecodingByteArray/max-string-length:1024/batch-size:2048/prefixed-percent:10
10635 ns 9153 ns
BM_DeltaDecodingByteArray/max-string-length:1024/batch-size:2048/prefixed-percent:90
37811 ns 38028 ns
BM_DeltaDecodingByteArray/max-string-length:1024/batch-size:2048/prefixed-percent:99
41686 ns 40220 ns
BM_DeltaDecodingArrowByteArray/max-string-length:8/batch-size:512/prefixed-percent:10/null-percent:0
8048 ns 4895 ns
BM_DeltaDecodingArrowByteArray/max-string-length:8/batch-size:512/prefixed-percent:10/null-percent:50
11770 ns 7897 ns
BM_DeltaDecodingArrowByteArray/max-string-length:8/batch-size:512/prefixed-percent:90/null-percent:0
9329 ns 5591 ns
BM_DeltaDecodingArrowByteArray/max-string-length:8/batch-size:512/prefixed-percent:90/null-percent:50
12911 ns 8854 ns
BM_DeltaDecodingArrowByteArray/max-string-length:8/batch-size:512/prefixed-percent:99/null-percent:0
9195 ns 5652 ns
BM_DeltaDecodingArrowByteArray/max-string-length:8/batch-size:512/prefixed-percent:99/null-percent:50
12895 ns 8787 ns
BM_DeltaDecodingArrowByteArray/max-string-length:8/batch-size:2048/prefixed-percent:10/null-percent:0
34104 ns 18063 ns
BM_DeltaDecodingArrowByteArray/max-string-length:8/batch-size:2048/prefixed-percent:10/null-percent:50
59148 ns 32797 ns
BM_DeltaDecodingArrowByteArray/max-string-length:8/batch-size:2048/prefixed-percent:90/null-percent:0
38993 ns 21600 ns
BM_DeltaDecodingArrowByteArray/max-string-length:8/batch-size:2048/prefixed-percent:90/null-percent:50
67769 ns 39834 ns
BM_DeltaDecodingArrowByteArray/max-string-length:8/batch-size:2048/prefixed-percent:99/null-percent:0
37679 ns 23487 ns
BM_DeltaDecodingArrowByteArray/max-string-length:8/batch-size:2048/prefixed-percent:99/null-percent:50
68561 ns 38294 ns
BM_DeltaDecodingArrowByteArray/max-string-length:64/batch-size:512/prefixed-percent:10/null-percent:0
8515 ns 5015 ns
BM_DeltaDecodingArrowByteArray/max-string-length:64/batch-size:512/prefixed-percent:10/null-percent:50
12169 ns 8036 ns
BM_DeltaDecodingArrowByteArray/max-string-length:64/batch-size:512/prefixed-percent:90/null-percent:0
10279 ns 5764 ns
BM_DeltaDecodingArrowByteArray/max-string-length:64/batch-size:512/prefixed-percent:90/null-percent:50
13850 ns 8990 ns
BM_DeltaDecodingArrowByteArray/max-string-length:64/batch-size:512/prefixed-percent:99/null-percent:0
10327 ns 5678 ns
BM_DeltaDecodingArrowByteArray/max-string-length:64/batch-size:512/prefixed-percent:99/null-percent:50
14653 ns 8895 ns
BM_DeltaDecodingArrowByteArray/max-string-length:64/batch-size:2048/prefixed-percent:10/null-percent:0
37757 ns 18967 ns
BM_DeltaDecodingArrowByteArray/max-string-length:64/batch-size:2048/prefixed-percent:10/null-percent:50
61876 ns 33398 ns
BM_DeltaDecodingArrowByteArray/max-string-length:64/batch-size:2048/prefixed-percent:90/null-percent:0
49373 ns 22349 ns
BM_DeltaDecodingArrowByteArray/max-string-length:64/batch-size:2048/prefixed-percent:90/null-percent:50
76974 ns 41473 ns
BM_DeltaDecodingArrowByteArray/max-string-length:64/batch-size:2048/prefixed-percent:99/null-percent:0
48556 ns 22389 ns
BM_DeltaDecodingArrowByteArray/max-string-length:64/batch-size:2048/prefixed-percent:99/null-percent:50
77380 ns 40130 ns
BM_DeltaDecodingArrowByteArray/max-string-length:1024/batch-size:512/prefixed-percent:10/null-percent:0
17136 ns 9301 ns
BM_DeltaDecodingArrowByteArray/max-string-length:1024/batch-size:512/prefixed-percent:10/null-percent:50
19024 ns 11516 ns
BM_DeltaDecodingArrowByteArray/max-string-length:1024/batch-size:512/prefixed-percent:90/null-percent:0
22772 ns 11436 ns
BM_DeltaDecodingArrowByteArray/max-string-length:1024/batch-size:512/prefixed-percent:90/null-percent:50
26286 ns 13283 ns
BM_DeltaDecodingArrowByteArray/max-string-length:1024/batch-size:512/prefixed-percent:99/null-percent:0
23564 ns 11990 ns
BM_DeltaDecodingArrowByteArray/max-string-length:1024/batch-size:512/prefixed-percent:99/null-percent:50
26285 ns 13936 ns
BM_DeltaDecodingArrowByteArray/max-string-length:1024/batch-size:2048/prefixed-percent:10/null-percent:0
101506 ns 49041 ns
BM_DeltaDecodingArrowByteArray/max-string-length:1024/batch-size:2048/prefixed-percent:10/null-percent:50
112890 ns 60100 ns
BM_DeltaDecodingArrowByteArray/max-string-length:1024/batch-size:2048/prefixed-percent:90/null-percent:0
127686 ns 54975 ns
BM_DeltaDecodingArrowByteArray/max-string-length:1024/batch-size:2048/prefixed-percent:90/null-percent:50
141525 ns 65763 ns
BM_DeltaDecodingArrowByteArray/max-string-length:1024/batch-size:2048/prefixed-percent:99/null-percent:0
129513 ns 55034 ns
BM_DeltaDecodingArrowByteArray/max-string-length:1024/batch-size:2048/prefixed-percent:99/null-percent:50
152311 ns 66980 ns
BM_DeltaDecodingFLBA/type-length:2/batch-size:512/prefixed-percent:10
2810 ns 1850 ns
BM_DeltaDecodingFLBA/type-length:2/batch-size:512/prefixed-percent:90
3591 ns 2856 ns
BM_DeltaDecodingFLBA/type-length:2/batch-size:512/prefixed-percent:99
3616 ns 3046 ns
BM_DeltaDecodingFLBA/type-length:2/batch-size:2048/prefixed-percent:10
11025 ns 6936 ns
BM_DeltaDecodingFLBA/type-length:2/batch-size:2048/prefixed-percent:90
15660 ns 11435 ns
BM_DeltaDecodingFLBA/type-length:2/batch-size:2048/prefixed-percent:99
15891 ns 11964 ns
BM_DeltaDecodingFLBA/type-length:16/batch-size:512/prefixed-percent:10
2823 ns 1936 ns
BM_DeltaDecodingFLBA/type-length:16/batch-size:512/prefixed-percent:90
4334 ns 3589 ns
BM_DeltaDecodingFLBA/type-length:16/batch-size:512/prefixed-percent:99
4491 ns 3764 ns
BM_DeltaDecodingFLBA/type-length:16/batch-size:2048/prefixed-percent:10
10575 ns 7494 ns
BM_DeltaDecodingFLBA/type-length:16/batch-size:2048/prefixed-percent:90
18031 ns 14942 ns
BM_DeltaDecodingFLBA/type-length:16/batch-size:2048/prefixed-percent:99
19600 ns 16668 ns
BM_DeltaDecodingDenseFLBA/type-length:2/batch-size:512/prefixed-percent:10
3144 ns 2487 ns
BM_DeltaDecodingDenseFLBA/type-length:2/batch-size:512/prefixed-percent:90
4104 ns 3000 ns
BM_DeltaDecodingDenseFLBA/type-length:2/batch-size:512/prefixed-percent:99
4256 ns 3025 ns
BM_DeltaDecodingDenseFLBA/type-length:2/batch-size:2048/prefixed-percent:10
12340 ns 9668 ns
BM_DeltaDecodingDenseFLBA/type-length:2/batch-size:2048/prefixed-percent:90
16679 ns 12125 ns
BM_DeltaDecodingDenseFLBA/type-length:2/batch-size:2048/prefixed-percent:99
17299 ns 12972 ns
BM_DeltaDecodingDenseFLBA/type-length:16/batch-size:512/prefixed-percent:10
3003 ns 2573 ns
BM_DeltaDecodingDenseFLBA/type-length:16/batch-size:512/prefixed-percent:90
4586 ns 3405 ns
BM_DeltaDecodingDenseFLBA/type-length:16/batch-size:512/prefixed-percent:99
4714 ns 3495 ns
BM_DeltaDecodingDenseFLBA/type-length:16/batch-size:2048/prefixed-percent:10
11964 ns 9921 ns
BM_DeltaDecodingDenseFLBA/type-length:16/batch-size:2048/prefixed-percent:90
19559 ns 13076 ns
BM_DeltaDecodingDenseFLBA/type-length:16/batch-size:2048/prefixed-percent:99
20968 ns 14261 ns
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
