Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24137 )
Change subject: IMPALA-13810: Generate lineage records for UPDATE and MERGE statements ...................................................................... IMPALA-13810: Generate lineage records for UPDATE and MERGE statements This patch enables lineage generation for UPDATE statements on Iceberg and Kudu tables. Lineage will not include virtual columns in case of Iceberg tables, or the auto incrementing id column in case of Kudu tables. It also enables lineage generation for MERGE statements which are currently only valid on Iceberg tables. DELETE-only MERGE statements do not produce lineage, similar to DELETE statements. Added a new check to the test framework, to be able to validate queries that shouldn't produce lineage. An empty (whitespace-only) LINEAGE section will fail the test if the query produces lineage. Testing: * added new e2e tests Change-Id: Icba79f756509438455bfbf3067733f6f29284220 Assisted-by: Claude Opus 4.7 (Claude Code) Reviewed-on: http://gerrit.cloudera.org:8080/24137 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M fe/src/main/java/org/apache/impala/analysis/ColumnLineageGraph.java M fe/src/main/java/org/apache/impala/analysis/MergeInsert.java M fe/src/main/java/org/apache/impala/analysis/MergeUpdate.java M fe/src/main/java/org/apache/impala/planner/DataSink.java M fe/src/main/java/org/apache/impala/planner/DataStreamSink.java M fe/src/main/java/org/apache/impala/planner/HBaseTableSink.java M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java M fe/src/main/java/org/apache/impala/planner/IcebergBufferedDeleteSink.java M fe/src/main/java/org/apache/impala/planner/JoinBuildSink.java M fe/src/main/java/org/apache/impala/planner/KuduTableSink.java M fe/src/main/java/org/apache/impala/planner/MultiDataSink.java M fe/src/main/java/org/apache/impala/planner/PlanRootSink.java M fe/src/main/java/org/apache/impala/planner/Planner.java A testdata/workloads/functional-query/queries/QueryTest/lineage-iceberg-update-merge.test A testdata/workloads/functional-query/queries/QueryTest/lineage-kudu-update.test M tests/common/impala_test_suite.py M tests/custom_cluster/test_lineage.py 17 files changed, 3,831 insertions(+), 113 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/24137 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Icba79f756509438455bfbf3067733f6f29284220 Gerrit-Change-Number: 24137 Gerrit-PatchSet: 19 Gerrit-Owner: Daniel Vanko <[email protected]> Gerrit-Reviewer: Daniel Vanko <[email protected]> Gerrit-Reviewer: Fang-Yu Rao <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Peter Rozsa <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
