huaxingao commented on code in PR #14984:
URL: https://github.com/apache/iceberg/pull/14984#discussion_r3624052533
##########
spark/v4.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestSystemFunctionPushDownDQL.java:
##########
@@ -291,13 +290,9 @@ private void checkExpressions(
PlanUtils.collectSparkExpressions(
optimizedPlan, expression -> expression instanceof
ApplyFunctionExpression);
- if (partitioned) {
- assertThat(applyExpressions).isEmpty();
Review Comment:
This removes the partitioned parameter across all the system-function tests
(years/months/days/hours/bucket/truncate) and now asserts a single
ApplyFunctionExpression for both the partitioned and unpartitioned cases. In
4.1 the partitioned branch asserted applyExpressions.isEmpty() — for a table
partitioned by e.g. years(ts), a predicate on system.years(ts) was fully
eliminated from the optimized plan (absorbed via the partition transform). The
...OnPartitionedTable tests still create partitioned tables, so uniformly
switching to hasSize(1) means that elimination no longer happens for any
transform in 4.2.
checkPushedFilters and the result counts are unchanged, so this is a
plan-shape change rather than a correctness one. Is the loss of that
partition-transform pushdown intended behavior in Spark 4.2, or a regression
worth tracking upstream?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
