Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24947 )
Change subject: IMPALA-15421: Fix dropped conjuncts in Iceberg predicate subsetting ...................................................................... IMPALA-15421: Fix dropped conjuncts in Iceberg predicate subsetting IcebergScanPlanner collected the residual expressions of the FileScanTasks into a TreeSet ordered by ExpressionUtil.toSanitizedString(). The sanitized string replaces literal values with placeholders, e.g. both 'id != 11' and 'id != 21' become 'id != (2-digit-int)', so the set only kept one of two different residuals. With ICEBERG_PREDICATE_PUSHDOWN_SUBSETTING (on by default) the conjunct of the lost residual was then skipped from the scan node, and the query returned rows that should have been filtered out. This happens when non-identity partition transforms (e.g. truncate or day) leave different residuals for different files. This patch keys the residuals by Expression.toString(), which prints the literals in full. It uses a TreeMap so the order of the retained conjuncts stays deterministic. Consecutive files often have the same residual object (e.g. every file of an unpartitioned table), so the string conversion is skipped for them. This also removes the cost of sanitizing the residual of every file during planning. A predicate that is part of multiple residuals is only retained once. Testing: - Added EE tests with truncate and day partition transforms. Both returned wrong results without the fix. - Ran PlannerTest and test_iceberg.py. Change-Id: Ic2b0ef09255ead578ab4328885002819dddee013 Assisted-by: Claude Opus 5.5 (1M context) <[email protected]> Reviewed-on: http://gerrit.cloudera.org:8080/24947 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java A testdata/workloads/functional-query/queries/QueryTest/iceberg-predicate-subsetting.test M tests/query_test/test_iceberg.py 3 files changed, 56 insertions(+), 9 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/24947 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic2b0ef09255ead578ab4328885002819dddee013 Gerrit-Change-Number: 24947 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Noemi Pap-Takacs <[email protected]> Gerrit-Reviewer: Peter Rozsa <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
