[Impala-ASF-CR] IMPALA-9053: DDLs should generate lineage graphs.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14458 ) Change subject: IMPALA-9053: DDLs should generate lineage graphs. .. IMPALA-9053: DDLs should generate lineage graphs. DDLs like 'create table' should generate minimal lineage graphs so that consumers like Atlas can use information like 'queryText' to establish lineages. This change adds a call to the computeLineageGraph() method during analysis phase of createTable which populates the graph with basic information like queryText. If it is a CTAS, this graph is enhanced in the "insert" phase with dependencies. Testing: Add an EE test to verify lineage information and also to check it is flushed to disk properly. Change-Id: Ia6c7ed9fe3265fd777fe93590cf4eb2d9ba0dd1e Reviewed-on: http://gerrit.cloudera.org:8080/14458 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/analysis/ColumnLineageGraph.java M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java M testdata/workloads/functional-query/queries/QueryTest/lineage.test M tests/custom_cluster/test_lineage.py 4 files changed, 61 insertions(+), 2 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/14458 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ia6c7ed9fe3265fd777fe93590cf4eb2d9ba0dd1e Gerrit-Change-Number: 14458 Gerrit-PatchSet: 3 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen
[Impala-ASF-CR] IMPALA-9053: DDLs should generate lineage graphs.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14458 ) Change subject: IMPALA-9053: DDLs should generate lineage graphs. .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14458 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia6c7ed9fe3265fd777fe93590cf4eb2d9ba0dd1e Gerrit-Change-Number: 14458 Gerrit-PatchSet: 2 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Thu, 17 Oct 2019 15:48:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9053: DDLs should generate lineage graphs.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14458 ) Change subject: IMPALA-9053: DDLs should generate lineage graphs. .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5102/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14458 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia6c7ed9fe3265fd777fe93590cf4eb2d9ba0dd1e Gerrit-Change-Number: 14458 Gerrit-PatchSet: 2 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Thu, 17 Oct 2019 15:48:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9053: DDLs should generate lineage graphs.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14458 ) Change subject: IMPALA-9053: DDLs should generate lineage graphs. .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14458 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia6c7ed9fe3265fd777fe93590cf4eb2d9ba0dd1e Gerrit-Change-Number: 14458 Gerrit-PatchSet: 1 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Thu, 17 Oct 2019 15:48:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9053: DDLs should generate lineage graphs.
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14458 ) Change subject: IMPALA-9053: DDLs should generate lineage graphs. .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/14458/1/testdata/workloads/functional-query/queries/QueryTest/lineage.test File testdata/workloads/functional-query/queries/QueryTest/lineage.test: PS1: > Also add a test for CTAS as a sanity check? CTAS queries are already covered later in this .test file. -- To view, visit http://gerrit.cloudera.org:8080/14458 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia6c7ed9fe3265fd777fe93590cf4eb2d9ba0dd1e Gerrit-Change-Number: 14458 Gerrit-PatchSet: 1 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Thu, 17 Oct 2019 04:55:42 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9053: DDLs should generate lineage graphs.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14458 ) Change subject: IMPALA-9053: DDLs should generate lineage graphs. .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/14458/1/testdata/workloads/functional-query/queries/QueryTest/lineage.test File testdata/workloads/functional-query/queries/QueryTest/lineage.test: PS1: Also add a test for CTAS as a sanity check? -- To view, visit http://gerrit.cloudera.org:8080/14458 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia6c7ed9fe3265fd777fe93590cf4eb2d9ba0dd1e Gerrit-Change-Number: 14458 Gerrit-PatchSet: 1 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Wed, 16 Oct 2019 23:49:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9053: DDLs should generate lineage graphs.
radford nguyen has posted comments on this change. ( http://gerrit.cloudera.org:8080/14458 ) Change subject: IMPALA-9053: DDLs should generate lineage graphs. .. Patch Set 1: Code-Review+1 I've touched lineage-related code in the past but have to admit that I don't really have much domain knowledge. That said, the changes look good to me -- To view, visit http://gerrit.cloudera.org:8080/14458 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia6c7ed9fe3265fd777fe93590cf4eb2d9ba0dd1e Gerrit-Change-Number: 14458 Gerrit-PatchSet: 1 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Wed, 16 Oct 2019 19:48:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9053: DDLs should generate lineage graphs.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14458 ) Change subject: IMPALA-9053: DDLs should generate lineage graphs. .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4804/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14458 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia6c7ed9fe3265fd777fe93590cf4eb2d9ba0dd1e Gerrit-Change-Number: 14458 Gerrit-PatchSet: 1 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 16 Oct 2019 01:51:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9053: DDLs should generate lineage graphs.
Anurag Mantripragada has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14458 Change subject: IMPALA-9053: DDLs should generate lineage graphs. .. IMPALA-9053: DDLs should generate lineage graphs. DDLs like 'create table' should generate minimal lineage graphs so that consumers like Atlas can use information like 'queryText' to establish lineages. This change adds a call to the computeLineageGraph() method during analysis phase of createTable which populates the graph with basic information like queryText. If it is a CTAS, this graph is enhanced in the "insert" phase with dependencies. Testing: Add an EE test to verify lineage information and also to check it is flushed to disk properly. Change-Id: Ia6c7ed9fe3265fd777fe93590cf4eb2d9ba0dd1e --- M fe/src/main/java/org/apache/impala/analysis/ColumnLineageGraph.java M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java M testdata/workloads/functional-query/queries/QueryTest/lineage.test M tests/custom_cluster/test_lineage.py 4 files changed, 61 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/58/14458/1 -- To view, visit http://gerrit.cloudera.org:8080/14458 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ia6c7ed9fe3265fd777fe93590cf4eb2d9ba0dd1e Gerrit-Change-Number: 14458 Gerrit-PatchSet: 1 Gerrit-Owner: Anurag Mantripragada
