stuhood commented on PR #26593:
URL: https://github.com/apache/datafusion/pull/26593#issuecomment-7024310074

   Sorry for the long delay here! Thanks @jayshrivastava for splitting out 
#27384, and thanks @mbutrovich for looping back on this.
   
   I finally got around to filing #27600 to lay out the `Statistics`-based 
alternative in one place, along with a quick prototype and benchmark: 
https://github.com/paradedb/datafusion/blob/732c2171387105f567d66a27ccc573ddc2cbe2c8/BENCHMARK_EMPTY_PARTITIONS.md
   
   Two quick notes on the discussion above:
   * On @avantgardnerio's point in 
https://github.com/apache/datafusion/pull/26593#issuecomment-6639300308 (that 
emptiness would stay private to the source or need a new API): with #26094's 
`StatisticsContext`, `num_rows == Precision::Exact(0)` is already visible to 
rules at any node in the plan, so rules can see which ranges a task owns 
directly from `split_points` + `Exact(0)` (including non-contiguous slices) 
without adding bounds to `RangePartitioning`.
   * On the `Range(1600)` runtime cost: with a small batch API on 
`StatisticsContext` (`~0.1 ms` across `1,600` partitions) and skipping 
`Exact(0)` inputs in `RepartitionExec` / `CoalescePartitionsExec`, zero empty 
leaf streams get executed and local coalesce/repartition run within noise of a 
`16`-partition plan (`0.56 ms` vs `0.70 ms` coalesce, `1.36 ms` vs `1.47 ms` 
`Hash(16)` repartition, and `10.6 ms` vs `833 ms` for a `1,600`-way shuffle).
   
   If folks are good with the #27600 direction, I'd be happy to clean up #27374 
(it's still a draft, and will now need deprecation handling since `56.0.0` went 
out) to get it ready for review, and follow up with PRs for the batch stats and 
empty-partition skipping pieces as well.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to