gimgit commented on PR #17504: URL: https://github.com/apache/iceberg/pull/17504#issuecomment-5363544048
@anuragmantri thanks for the dev@ reply — picking this up here as suggested. How to choose `min-overlap-depth` is the right question, and it's the reason measurement and selection ship together in this PR. The intended workflow is: run `compute_sort_order_stats` first, look at the depth distribution, then set the threshold — not pick a number blind. Two ways to reason about the value: 1. **Read amplification.** Depth has a direct operational meaning: a point lookup on the sort key reads up to `depth` files. So the threshold is "how many files am I willing to read per key" — a table serving selective queries might use 3-4, a table that mostly serves scans can tolerate more. This stays meaningful across table sizes because it's about query cost, not file counts. 2. **Relative to the observed max.** After the first `compute_sort_order_stats` run, a threshold around half the reported `max_overlap_depth` selects only the worst regions and converges over repeated runs. This is the conservative starting point when query patterns are unknown. What I deliberately didn't do is pick a default: depth scales with file count per partition, so any fixed default would be wrong for someone. Unset-by-default keeps the planner honest until the user has looked at their numbers. If there's appetite for it, a follow-up could add a percentile column (e.g. p90 depth per partition) to `compute_sort_order_stats`, which would make "half the max" less sensitive to a single hot partition — max alone is dominated by the worst spot. I kept this PR to max/avg to stay reviewable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
