stuhood commented on PR #26593: URL: https://github.com/apache/datafusion/pull/26593#issuecomment-7024310074
Sorry for the long delay here! Thanks @jayshrivastava for splitting out #27384, and thanks @mbutrovich for looping back on this. I finally got around to filing #27600 to lay out the `Statistics`-based alternative in one place, along with a quick prototype and benchmark: https://github.com/paradedb/datafusion/blob/732c2171387105f567d66a27ccc573ddc2cbe2c8/BENCHMARK_EMPTY_PARTITIONS.md Two quick notes on the discussion above: * On @avantgardnerio's point in https://github.com/apache/datafusion/pull/26593#issuecomment-6639300308 (that emptiness would stay private to the source or need a new API): with #26094's `StatisticsContext`, `num_rows == Precision::Exact(0)` is already visible to rules at any node in the plan, so rules can see which ranges a task owns directly from `split_points` + `Exact(0)` (including non-contiguous slices) without adding bounds to `RangePartitioning`. * On the `Range(1600)` runtime cost: with a small batch API on `StatisticsContext` (`~0.1 ms` across `1,600` partitions) and skipping `Exact(0)` inputs in `RepartitionExec` / `CoalescePartitionsExec`, zero empty leaf streams get executed and local coalesce/repartition run within noise of a `16`-partition plan (`0.56 ms` vs `0.70 ms` coalesce, `1.36 ms` vs `1.47 ms` `Hash(16)` repartition, and `10.6 ms` vs `833 ms` for a `1,600`-way shuffle). If folks are good with the #27600 direction, I'd be happy to clean up #27374 (it's still a draft, and will now need deprecation handling since `56.0.0` went out) to get it ready for review, and follow up with PRs for the batch stats and empty-partition skipping pieces as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
