asubiotto opened a new pull request, #9940:
URL: https://github.com/apache/arrow-rs/pull/9940
# Which issue does this PR close?
- Closes #9939
# Rationale for this change
interleave_dictionaries checks if input dictionaries should be merged/GCed,
but should_merge_dictionary_values has limited type support due to the bytes
interning it does. For other types and in the general fallback case where the
heuristic fails, interleave_fallback_dictionary concatenates all the values
slices together, resulting in a lot of bloat in cases where the interleave
selection is minimal. This happens a lot e.g. in datafusion on a
multi-partition sort on a dictionary column.
# What changes are included in this PR?
This commit improves these cases by introducing a heuristic to check
interleave index coverage assuming uniform value selection. When coverage is
less than the number of values, interleave_fallback_dictionary now performs a
`take` on the value slices in order to reduce output size bloat. On a
real-world datafusion sort that motivated this change, I saw runtime drop from
20 minutes to 7 minutes. The heuristic allows us to avoid a microbenchmark
regression.
# Are these changes tested?
Yes, by existing tests and some new tests exercising this path specifically.
# Are there any user-facing changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
