kosiew commented on code in PR #27118:
URL: https://github.com/apache/datafusion/pull/27118#discussion_r4585335191


##########
datafusion/sqllogictest/test_files/dynamic_filter_pushdown_config.slt:
##########
@@ -1163,3 +1163,93 @@ SET datafusion.execution.target_partitions = 4;
 
 statement ok
 RESET datafusion.execution.parquet.pushdown_filters;
+
+# Regression for https://github.com/apache/datafusion/issues/26029.
+# When the TopK filter is pushed into a scan, each sort key is rewritten in
+# terms of the file's columns. One row per row group lets TopK tighten its
+# filter before the answer rows, which are last in each file, are read.
+statement ok
+SET datafusion.execution.target_partitions = 1;
+
+statement ok
+SET datafusion.execution.parquet.pushdown_filters = true;
+
+statement ok
+SET datafusion.explain.physical_plan_only = true;
+
+query I
+COPY (SELECT value AS b FROM generate_series(0, 19))

Review Comment:
   Optional: could we add a UInt64-backed Parquet variant using `CAST(value AS 
BIGINT UNSIGNED)` before `COPY`, assert the expected rows, and confirm 
scan-level dynamic filtering remains enabled? Issue #26029 also reports an 
invalid-comparison error with UInt64 input, which the current signed fixtures 
do not cover.



##########
datafusion/sqllogictest/test_files/dynamic_filter_pushdown_config.slt:
##########
@@ -1163,3 +1163,93 @@ SET datafusion.execution.target_partitions = 4;
 
 statement ok
 RESET datafusion.execution.parquet.pushdown_filters;
+
+# Regression for https://github.com/apache/datafusion/issues/26029.
+# When the TopK filter is pushed into a scan, each sort key is rewritten in
+# terms of the file's columns. One row per row group lets TopK tighten its
+# filter before the answer rows, which are last in each file, are read.
+statement ok
+SET datafusion.execution.target_partitions = 1;
+
+statement ok
+SET datafusion.execution.parquet.pushdown_filters = true;
+
+statement ok
+SET datafusion.explain.physical_plan_only = true;
+
+query I
+COPY (SELECT value AS b FROM generate_series(0, 19))
+TO 'test_files/scratch/dynamic_filter_pushdown_config/issue_26029.parquet'
+STORED AS PARQUET OPTIONS ('format.max_row_group_size' '1');
+----
+20
+
+statement ok
+CREATE EXTERNAL TABLE issue_26029 STORED AS PARQUET
+LOCATION 
'test_files/scratch/dynamic_filter_pushdown_config/issue_26029.parquet';
+
+# The first sort key contains the second one, `b`. The whole CASE must be
+# rewritten, not only the `b` inside it. `p` and `q` are equal, so the
+# expected rows are easy to work out.
+#
+# The middle projection's column order is not stable between runs, so those
+# parts are ignored. The check is that TopK's filter reaches the scan.
+query TT
+EXPLAIN WITH s AS (SELECT b, 19 - b AS p, 19 - b AS q FROM issue_26029)
+SELECT p FROM s ORDER BY CASE WHEN b % 2 = 0 THEN p ELSE q END, b LIMIT 3;
+----
+physical_plan
+01)ProjectionExec: expr=[p@0 as p]
+02)--SortExec: TopK(fetch=3), expr=[<slt:ignore>], 
preserve_partitioning=[false]
+03)----ProjectionExec: expr=[<slt:ignore>]
+04)------DataSourceExec: file_groups={1 group: 
[[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/dynamic_filter_pushdown_config/issue_26029.parquet]]},
 projection=[19 - b@0 as __common_expr_1, b], output_ordering=[b@1 ASC NULLS 
LAST], file_type=parquet, predicate=DynamicFilter [ empty ], 
dynamic_rg_pruning=eligible
+
+query I
+WITH s AS (SELECT b, 19 - b AS p, 19 - b AS q FROM issue_26029)
+SELECT p FROM s ORDER BY CASE WHEN b % 2 = 0 THEN p ELSE q END, b LIMIT 3;

Review Comment:
   Optional: could we add a result query with ties in the CASE key to verify 
secondary ordering by `b`? Using integer division for `p = q = (19 - b) / 2` 
creates ties; selecting `b` with the existing ORDER BY and LIMIT 3 should 
return `18, 19, 16`. The current fixture has unique first-key values, so it 
cannot exercise tie-breaking.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to