jayzhan211 opened a new issue, #26642:
URL: https://github.com/apache/datafusion/issues/26642
### Is your feature request related to a problem or challenge?
This answers #17060: TopK dynamic filters don't work inside recursive
queries.
`RecursiveQueryExec` calls `reset_plan_states` on the recursive term before
each iteration. `SortExec::reset_state` creates a new filter with
`create_filter()`, but the scan is reset first and keeps the original filter.
Nothing updates that filter, so it stays empty and the scan reads every row on
every iteration.
```sql
SET datafusion.execution.target_partitions = 1;
SET datafusion.execution.batch_size = 10;
SET datafusion.execution.parquet.pushdown_filters = true;
COPY (SELECT CASE WHEN value = 1 THEN 1 ELSE 102 - value END AS k FROM
generate_series(1, 100))
TO '/tmp/t2.parquet' OPTIONS ('format.max_row_group_size' '10');
CREATE EXTERNAL TABLE t STORED AS PARQUET LOCATION '/tmp/t2.parquet';
EXPLAIN ANALYZE WITH RECURSIVE w(n) AS (
SELECT 1 UNION ALL
SELECT w.n + s.k FROM w CROSS JOIN (SELECT k FROM t ORDER BY k LIMIT 1) s
WHERE w.n < 4
) SELECT * FROM w ORDER BY n;
-- scan of t: DynamicFilter [ empty ], output_rows=300 (100 per iteration)
EXPLAIN ANALYZE SELECT k FROM t ORDER BY k LIMIT 1;
-- scan of t: DynamicFilter [ k@0 < 1 ], output_rows=10
```
### Describe the solution you'd like
Reset the filter in place instead of replacing it, as #26056 does for
`AggregateExec` (see the discussion there):
- `SortExec::reset_state` calls `reset()` on its existing
`DynamicFilterPhysicalExpr` and passes it to `create_filter_with_expr`, which
already rebuilds only the TopK wrapper state.
- Add a `cte.slt` case that checks the scan prunes rows under a recursive
TopK.
### Describe alternatives you've considered
Creating a new filter and reconnecting it to the scan
(`PushedDynamicFilters` in #19905). `reset_plan_states` resets the scan before
the sort, so that needs an extra pass over the plan.
### Additional context
Depends on #26056 for `DynamicFilterPhysicalExpr::reset()`. The
`HashJoinExec` counterpart is #26641. Longer term, #21650 discusses moving this
shared state out of the plan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
