zhuqi-lucas commented on issue #25775: URL: https://github.com/apache/datafusion/issues/25775#issuecomment-5862386873
@ion-elgreco since you already have option 1 working with `DynamicFilterPhysicalExpr`, the thing worth checking is timing rather than mechanism: your source statistics only exist after the source has been read, so the target scan will usually have opened files before the filter is populated. That case is handled, and your shape is the one it is built for. At open time the opener decides whether to wrap the file stream in an `EarlyStoppingStream`, and it does so exactly when the pruner `is_watching()` — when it still watches a dynamic filter that can change. Since your source stats cannot be ready when the target files are opened, that holds for you, and the wrapper then re-checks `FilePruner::should_prune()` after each batch and ends the file scan the moment a tightened filter proves it skippable. Files opened before the filter arrives are not read to completion. The re-check is cheap by construction: `should_prune` rebuilds the pruning predicate only on the first call and whenever a watched filter has actually moved, so the steady state is an atomic load. Files not yet opened take the cheaper path and are dropped in `prune_file` without reading a byte of the range. Worth knowing where the clean tier ends. File-level runtime pruning is the mature one. Row-group pruning can only act at row-group boundaries, and the page-level selection is computed once when the file is opened and never revisited. A copy-on-write MERGE that must read every row of the files it keeps only needs file granularity, so you are on the well-trodden path. On the streaming file list itself I agree with @xudong963 — that is a separate axis from the pruning, and a custom plan in delta-rs is the right place to find out what is actually missing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
