Csaba Ringhofer has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/22995 )
Change subject: IMPALA-14123: Allow forcing predicate push down to Iceberg ...................................................................... IMPALA-14123: Allow forcing predicate push down to Iceberg Since IMPALA-11591 Impala tries to avoid pushing down predicates to Iceberg unless it is necessary (timetravel) or is likely to be useful (at least 1 partition column is involved in predicates). While this makes planning faster, it may miss opportunities to skip files during planning. This patch adds table property impala.iceberg.push_down_hint that expects a comma separated list of column names and leads to push down to Iceberg when there is a predicate on any of these columns. Users can set this manually, while in the future Impala or other tools may be able to set it automatically, e.g. during COMPUTE STATS if there are many files with non-overlapping min/max stats for a given column. Note that in most cases when Iceberg can skip files the Parquet/ORC scanner would also skip most of the data based on stat filtering. The benefit of doing it during planning is reading less footers and a "smaller" query plan. Change-Id: I8eb4ab5204c20b3991fdf305d7317f4023904a0f Reviewed-on: http://gerrit.cloudera.org:8080/22995 Reviewed-by: Csaba Ringhofer <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M fe/src/main/java/org/apache/impala/planner/IcebergScanPlanner.java A testdata/workloads/functional-query/queries/QueryTest/iceberg-predicate-push-down-hint.test M tests/query_test/test_iceberg.py 3 files changed, 100 insertions(+), 17 deletions(-) Approvals: Csaba Ringhofer: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/22995 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I8eb4ab5204c20b3991fdf305d7317f4023904a0f Gerrit-Change-Number: 22995 Gerrit-PatchSet: 5 Gerrit-Owner: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Noemi Pap-Takacs <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
