pingzh commented on issue #5775: URL: https://github.com/apache/datafusion-comet/issues/5775#issuecomment-5672549320
## Implementation plan: five smaller PRs Following the implementation and review in #5785, split the work into five focused PRs. Each PR will include its own tests and documentation, with benchmarks alongside the behavior they measure. | PR | Scope | | --- | --- | | **1. Refactor shared runtime-filter code** | Move join wiring into `dynamic_filter_join.rs`, extract the shared Parquet attachment helper, and standardize `dynamic_filter_join_*` metrics. Preserve existing filtering behavior. | | **2. Fix Parquet runtime-filter correctness** | Preserve schema-conversion and timestamp-overflow errors; handle missing null statistics conservatively; guard static predicates and supplied file statistics. Validate through existing join filtering so this is useful independently of TopK. | | **3. Fuse local TopK with native scans** | Introduce the fused Spark plan, including correct offsets, projections, fallback/reversion, and avoiding a redundant final heap for one partition. Include fusion benchmarks. **Keep fusion opt-in initially**, given the known slowdown. | | **4. Push TopK thresholds into Parquet readers** | Connect the live threshold to the reader for supported integer keys. Cover fresh state per execution, resets/cancellation, fallback, reader pruning, and attachment counters. Include native/Spark correctness tests and filtering benchmarks. | | **5. Add TopK residual-filter metrics** | Add evaluated/pruned/bypassed row counts and evaluation time. Review the additional filtering pass and payload-copying cost separately, using release-build benchmarks. **This is optional for delivering reader pruning.** | ### Dependencies - **1 → 2**. - **3** can proceed independently. - **4** requires **2 + 3**. - **5** follows **4**. This separates review of the refactoring, correctness fixes, fusion, reader filtering, and extra instrumentation. The correctness safeguards and lifecycle coverage must be in place before enabling TopK reader filtering. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
