[Impala-ASF-CR] IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19052 ) Change subject: IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8648/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/19052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7f1b470f40e0aaf891c9f3f327af393b2f9c74bc Gerrit-Change-Number: 19052 Gerrit-PatchSet: 3 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Sat, 01 Oct 2022 03:40:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19052 ) Change subject: IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11510/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7f1b470f40e0aaf891c9f3f327af393b2f9c74bc Gerrit-Change-Number: 19052 Gerrit-PatchSet: 3 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Sat, 01 Oct 2022 02:10:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala
Yu-Wen Lai has posted comments on this change. ( http://gerrit.cloudera.org:8080/19052 ) Change subject: IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala .. Patch Set 3: (3 comments) > Patch Set 1: > > (3 comments) > > This is a pretty nice fix! http://gerrit.cloudera.org:8080/#/c/19052/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/19052/1//COMMIT_MSG@16 PS1, Line 16: - Run existing test_load.py > We also need tests to verify the INSERT events. Could you add some tests in I realized that replication cannot be used as a verification of insert event for external tables because hive replication for external tables relies on distcp instead of insert events. Given that LOAD DATA is only applicable to external tables, we need to use another way to verify the INSERT events. Therefore, I added a test and used number of skipped events as an implicit indicator. Let me know if you have better idea. http://gerrit.cloudera.org:8080/#/c/19052/1/be/src/service/client-request-state.cc File be/src/service/client-request-state.cc: http://gerrit.cloudera.org:8080/#/c/19052/1/be/src/service/client-request-state.cc@806 PS1, Line 806: string for unpartitione > nit: Could you add a comment mentioning that the partition_name is an empty Done http://gerrit.cloudera.org:8080/#/c/19052/1/be/src/service/client-request-state.cc@809 PS1, Line 809: catalog_update.__set_sync_ddl(exec_request_->query_options.sync_ddl); : catalog_update.__set_header(GetCatalogServiceRequestHeader()); : catalog_update.target_table = exec_request_->load_data_request.table_name.table_name; : catalog_update.db_name = exec_request_->load_data_request.table_name.db_name; : catalog_update.is_overwrite = exec_request_->load_data_request.overwrite; : : const TNetworkAddress& address = > nit: these duplicate the code in ClientRequestState::ExecLoadDataRequestImp Done -- To view, visit http://gerrit.cloudera.org:8080/19052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7f1b470f40e0aaf891c9f3f327af393b2f9c74bc Gerrit-Change-Number: 19052 Gerrit-PatchSet: 3 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Sat, 01 Oct 2022 01:59:52 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala
Yu-Wen Lai has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/19052 ) Change subject: IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala .. IMPALA-8592: Add support for insert events for 'LOAD DATA' statements from Impala In this patch, we use TUpdateCatalogRequest to refresh metadata after 'LOAD DATA' instead of TResetMetadataRequest so that we can reuse the code for 'INSERT' statements. It will fire an insert event just same as what we did for 'INSERT' statements. Testing: - Run existing test_load.py - Added test_load_data_from_impala() in test_event_processing.py Change-Id: I7f1b470f40e0aaf891c9f3f327af393b2f9c74bc --- M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M common/thrift/Frontend.thrift M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/main/java/org/apache/impala/service/Frontend.java M tests/metadata/test_event_processing.py 6 files changed, 129 insertions(+), 35 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/19052/3 -- To view, visit http://gerrit.cloudera.org:8080/19052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7f1b470f40e0aaf891c9f3f327af393b2f9c74bc Gerrit-Change-Number: 19052 Gerrit-PatchSet: 3 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yu-Wen Lai
[Impala-ASF-CR] IMPALA-11631 Fix impala crashes in impala::TopNNode::Heap::Close()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19087 ) Change subject: IMPALA-11631 Fix impala crashes in impala::TopNNode::Heap::Close() .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11509/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19087 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaf45b6ef777f68e1843c076a935e4189acc6990b Gerrit-Change-Number: 19087 Gerrit-PatchSet: 3 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Fri, 30 Sep 2022 23:04:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11631 Fix impala crashes in impala::TopNNode::Heap::Close()
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/19087 ) Change subject: IMPALA-11631 Fix impala crashes in impala::TopNNode::Heap::Close() .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/19087/1/be/src/exec/topn-node.cc File be/src/exec/topn-node.cc: http://gerrit.cloudera.org:8080/#/c/19087/1/be/src/exec/topn-node.cc@695 PS1, Line 695: > Would be good to add a comment before the second loop otherwise someone cou Done -- To view, visit http://gerrit.cloudera.org:8080/19087 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaf45b6ef777f68e1843c076a935e4189acc6990b Gerrit-Change-Number: 19087 Gerrit-PatchSet: 3 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Fri, 30 Sep 2022 22:45:04 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11631 Fix impala crashes in impala::TopNNode::Heap::Close()
Yida Wu has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/19087 ) Change subject: IMPALA-11631 Fix impala crashes in impala::TopNNode::Heap::Close() .. IMPALA-11631 Fix impala crashes in impala::TopNNode::Heap::Close() The bug is introduced by IMPALA-9979, if RematerializeTuples() fails in ReclaimTuplePool(), it returns immediately with an error, however, some Heap unique_ptr in the partition_heaps_ could be already moved to the rematerialized_heaps, while the Close() of the TopNNode doesn't know which Heap unique_ptr is released in partition_heaps_, still calls Close() on the released Heap unique_ptr, which leads to a crash. The patch changes the logic of moving each Heap object after one rematerialize process succeeds, instead, we will move all the Heap objects in the partition_heaps_ only when all the rematerialize processes succeed. Therefore, there will be no half released partition_heaps_. Because it could be difficult for a testcase to inject an error for this case to create a crash. I did some hacking in the code to inject a memory allocation failure in certain cases, reproduced the issue, and proved the patch can solve the issue manually. Tests: Ran core tests. Passed manual test. Change-Id: Iaf45b6ef777f68e1843c076a935e4189acc6990b --- M be/src/exec/topn-node.cc 1 file changed, 5 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/87/19087/3 -- To view, visit http://gerrit.cloudera.org:8080/19087 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iaf45b6ef777f68e1843c076a935e4189acc6990b Gerrit-Change-Number: 19087 Gerrit-PatchSet: 3 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/19046 ) Change subject: IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses .. Patch Set 5: Xianqing, verification failed in your new test case. -- To view, visit http://gerrit.cloudera.org:8080/19046 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1003556d3afc8e8216142cac4007a4c99046caeb Gerrit-Change-Number: 19046 Gerrit-PatchSet: 5 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Xiang Yang Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Fri, 30 Sep 2022 22:08:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11631 Fix impala crashes in impala::TopNNode::Heap::Close()
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/19087 ) Change subject: IMPALA-11631 Fix impala crashes in impala::TopNNode::Heap::Close() .. Patch Set 1: (1 comment) The fix looks good. Ideally, we could use smart pointers properly and not run into these issues. But, that would require some restructuring. For instance, having a destructor for TopNNode::Heap::~Heap() (which could basically call TopNNode::Heap::Close()). TopNNode::Close() could then simply call partition_heaps_.clear() instead of looping through them and calling TopNNode::Heap::Close() http://gerrit.cloudera.org:8080/#/c/19087/1/be/src/exec/topn-node.cc File be/src/exec/topn-node.cc: http://gerrit.cloudera.org:8080/#/c/19087/1/be/src/exec/topn-node.cc@695 PS1, Line 695: Would be good to add a comment before the second loop otherwise someone could try to optimize it in future. So something like: "Only move heaps from partition_heap_ to rematerialized_heaps once all have been rematerialized. Otherwise, in case of an error, we could double free entries in partition_heap_." -- To view, visit http://gerrit.cloudera.org:8080/19087 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaf45b6ef777f68e1843c076a935e4189acc6990b Gerrit-Change-Number: 19087 Gerrit-PatchSet: 1 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 30 Sep 2022 22:06:54 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19046 ) Change subject: IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8647/ -- To view, visit http://gerrit.cloudera.org:8080/19046 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1003556d3afc8e8216142cac4007a4c99046caeb Gerrit-Change-Number: 19046 Gerrit-PatchSet: 5 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Xiang Yang Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Fri, 30 Sep 2022 21:25:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11631 Fix impala crashes in impala::TopNNode::Heap::Close()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19087 ) Change subject: IMPALA-11631 Fix impala crashes in impala::TopNNode::Heap::Close() .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11508/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19087 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaf45b6ef777f68e1843c076a935e4189acc6990b Gerrit-Change-Number: 19087 Gerrit-PatchSet: 2 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 30 Sep 2022 20:16:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11631 Fix impala crashes in impala::TopNNode::Heap::Close()
Yida Wu has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/19087 ) Change subject: IMPALA-11631 Fix impala crashes in impala::TopNNode::Heap::Close() .. IMPALA-11631 Fix impala crashes in impala::TopNNode::Heap::Close() The bug is introduced by IMPALA-9979, if RematerializeTuples() fails in ReclaimTuplePool(), it returns immediately with an error, however, some Heap unique_ptr in the partition_heaps_ could be already moved to the rematerialized_heaps, while the Close() of the TopNNode doesn't know which Heap unique_ptr is released in partition_heaps_, still calls Close() on the released Heap unique_ptr, which leads to a crash. The patch changes the logic of moving each Heap object after one rematerialize process succeeds, instead, we will move all the Heap objects in the partition_heaps_ only when all the rematerialize processes succeed. Therefore, there will be no half released partition_heaps_. Because it could be difficult for a testcase to inject an error for this case to create a crash. I did some hacking in the code to inject a memory allocation failure in certain cases, reproduced the issue, and proved the patch can solve the issue manually. Tests: Ran core tests. Passed manual test. Change-Id: Iaf45b6ef777f68e1843c076a935e4189acc6990b --- M be/src/exec/topn-node.cc 1 file changed, 2 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/87/19087/2 -- To view, visit http://gerrit.cloudera.org:8080/19087 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iaf45b6ef777f68e1843c076a935e4189acc6990b Gerrit-Change-Number: 19087 Gerrit-PatchSet: 2 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-11631 Fix impala crashes in impala::TopNNode::Heap::Close()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19087 ) Change subject: IMPALA-11631 Fix impala crashes in impala::TopNNode::Heap::Close() .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11507/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19087 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaf45b6ef777f68e1843c076a935e4189acc6990b Gerrit-Change-Number: 19087 Gerrit-PatchSet: 1 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 30 Sep 2022 19:33:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11482: Alter Table Execute Rollback for Iceberg tables.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19002 ) Change subject: IMPALA-11482: Alter Table Execute Rollback for Iceberg tables. .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11506/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19002 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic74913d3b81103949ffb5eef7cc936303494f8b9 Gerrit-Change-Number: 19002 Gerrit-PatchSet: 7 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 30 Sep 2022 19:27:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11565: Support IF NOT EXISTS in alter table add columns for kudu table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18953 ) Change subject: IMPALA-11565: Support IF NOT EXISTS in alter table add columns for kudu table .. Patch Set 8: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8646/ -- To view, visit http://gerrit.cloudera.org:8080/18953 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I82590e5372e881f2e81d4ed3dd0d32a2d3ddb517 Gerrit-Change-Number: 18953 Gerrit-PatchSet: 8 Gerrit-Owner: Baike Xia Gerrit-Reviewer: Baike Xia Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Fri, 30 Sep 2022 18:57:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11482: Alter Table Execute Rollback for Iceberg tables.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19002 ) Change subject: IMPALA-11482: Alter Table Execute Rollback for Iceberg tables. .. Patch Set 6: (2 comments) http://gerrit.cloudera.org:8080/#/c/19002/6/fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/19002/6/fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java@73 PS6, Line 73: public static final Logger LOG = LoggerFactory.getLogger(IcebergCatalogOpExecutor.class); line too long (91 > 90) http://gerrit.cloudera.org:8080/#/c/19002/6/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java File fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java: http://gerrit.cloudera.org:8080/#/c/19002/6/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java@4188 PS6, Line 4188: "be an integer type or a timestamp, but is 'DECIMAL(3,2)': EXECUTE rollback(3.14)"); line too long (92 > 90) -- To view, visit http://gerrit.cloudera.org:8080/19002 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic74913d3b81103949ffb5eef7cc936303494f8b9 Gerrit-Change-Number: 19002 Gerrit-PatchSet: 6 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 30 Sep 2022 18:57:51 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11482: Alter Table Execute Rollback for Iceberg tables.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19002 ) Change subject: IMPALA-11482: Alter Table Execute Rollback for Iceberg tables. .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11505/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19002 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic74913d3b81103949ffb5eef7cc936303494f8b9 Gerrit-Change-Number: 19002 Gerrit-PatchSet: 6 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 30 Sep 2022 19:18:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11482: Alter Table Execute Rollback for Iceberg tables.
Andrew Sherman has posted comments on this change. (
http://gerrit.cloudera.org:8080/19002 )
Change subject: IMPALA-11482: Alter Table Execute Rollback for Iceberg tables.
..
Patch Set 6:
(7 comments)
Thanks for the review comments.
This change is blocked waiting for
https://github.com/apache/iceberg/issues/5882 which we need to include the
rollback in our iceberg transaction.
http://gerrit.cloudera.org:8080/#/c/19002/5/common/thrift/JniCatalog.thrift
File common/thrift/JniCatalog.thrift:
http://gerrit.cloudera.org:8080/#/c/19002/5/common/thrift/JniCatalog.thrift@428
PS5, Line 428: // Is rollback to a date or snapshot id.
: 1: required TRollbackType kind
:
: // If kind is TIME_ID this is the date to rollback to.
: 2: optional i64 timestamp_millis
:
: // If kind is VERSION_ID this is the id to rollback to.
: 3: optional i64 snapshot_id
: }
:
: /
> Just an idea: can we embed this under TAlterTableExecuteParams?
Thanks, I had to rework but this does make the change better and will allow
'EXECUTE' functions to be added more easily.
http://gerrit.cloudera.org:8080/#/c/19002/5/fe/src/main/cup/sql-parser.cup
File fe/src/main/cup/sql-parser.cup:
http://gerrit.cloudera.org:8080/#/c/19002/5/fe/src/main/cup/sql-parser.cup@1323
PS5, Line 1323:
> And maybe this could be 'AlterTableExecuteStmt.createExecuteStmt(table, exp
Done
http://gerrit.cloudera.org:8080/#/c/19002/5/fe/src/main/java/org/apache/impala/analysis/AlterTableExecuteRollbackStmt.java
File
fe/src/main/java/org/apache/impala/analysis/AlterTableExecuteRollbackStmt.java:
http://gerrit.cloudera.org:8080/#/c/19002/5/fe/src/main/java/org/apache/impala/analysis/AlterTableExecuteRollbackStmt.java@34
PS5, Line 34:
> nit: shouldn't we name it to 'rollbackSpec'?
yes!
http://gerrit.cloudera.org:8080/#/c/19002/5/fe/src/main/java/org/apache/impala/analysis/TableRef.java
File fe/src/main/java/org/apache/impala/analysis/TableRef.java:
http://gerrit.cloudera.org:8080/#/c/19002/5/fe/src/main/java/org/apache/impala/analysis/TableRef.java@478
PS5, Line 478:
> nit: indentation is off
Done
http://gerrit.cloudera.org:8080/#/c/19002/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/19002/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1336
PS5, Line 1336:
> This will have the same problem as https://gerrit.cloudera.org/#/c/19036/
This not done. I believe we need https://github.com/apache/iceberg/issues/5882
to do this.
http://gerrit.cloudera.org:8080/#/c/19002/5/fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/19002/5/fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java@199
PS5, Line 199: "Execute rollback "
> Do we need this MessagaFormat.format here?
Thanks, yes I have to change
LOG = Logger.getLogger to
LOG = LoggerFactory.getLogger
to get this to work (with {})
http://gerrit.cloudera.org:8080/#/c/19002/5/tests/query_test/test_iceberg.py
File tests/query_test/test_iceberg.py:
http://gerrit.cloudera.org:8080/#/c/19002/5/tests/query_test/test_iceberg.py@258
PS5, Line 258: # We rolled back, but that creates a new snapshot, so now
there are 4.
: snapshots = self.get_snapshots(self.client, tbl_name,
expected_result_size=4)
> It's interesting that non-ancestor snapshots are still in the output of DES
This is just what Iceberg does.
But it makes sense to me as it makes the rollback non-destructive
--
To view, visit http://gerrit.cloudera.org:8080/19002
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic74913d3b81103949ffb5eef7cc936303494f8b9
Gerrit-Change-Number: 19002
Gerrit-PatchSet: 6
Gerrit-Owner: Andrew Sherman
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Gergely Fürnstáhl
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tamas Mate
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Fri, 30 Sep 2022 18:57:55 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11482: Alter Table Execute Rollback for Iceberg tables.
Andrew Sherman has uploaded a new patch set (#7). (
http://gerrit.cloudera.org:8080/19002 )
Change subject: IMPALA-11482: Alter Table Execute Rollback for Iceberg tables.
..
IMPALA-11482: Alter Table Execute Rollback for Iceberg tables.
Iceberg table modifications cause new table snapshots to be created;
these snapshots represent an earlier version of the table. The Iceberg
API provides a way to rollback the table to a previous snapshot.
This change add the ability to execute a rollback on Iceberg tables
using the following statements:
- ALTER TABLE EXECUTE ROLLBACK()
- ALTER TABLE EXECUTE ROLLBACK('')
The latter form of the command rolls back to the latest snapshot
that has a creation timestamp that is older than the specified
timestamp.
Note that when a table is rolled back to a snapshot, a new snapshot is
created with the same snapshot id, but with a new creation timestamp.
Testing:
- Added analysis unit tests.
- Added e2e tests.
- Added a Snapshot class to the pytests to encapsulate the output of
DESCRIBE HISTORY.
Change-Id: Ic74913d3b81103949ffb5eef7cc936303494f8b9
---
M common/thrift/JniCatalog.thrift
M fe/src/main/cup/sql-parser.cup
A
fe/src/main/java/org/apache/impala/analysis/AlterTableExecuteExpireSnapshotsStmt.java
A fe/src/main/java/org/apache/impala/analysis/AlterTableExecuteRollbackStmt.java
M fe/src/main/java/org/apache/impala/analysis/AlterTableExecuteStmt.java
M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
M fe/src/main/java/org/apache/impala/common/TransactionKeepalive.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java
M fe/src/main/jflex/sql-scanner.flex
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
A testdata/workloads/functional-query/queries/QueryTest/iceberg-rollback.test
M tests/common/iceberg_test_suite.py
M tests/query_test/test_iceberg.py
14 files changed, 547 insertions(+), 110 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/02/19002/7
--
To view, visit http://gerrit.cloudera.org:8080/19002
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic74913d3b81103949ffb5eef7cc936303494f8b9
Gerrit-Change-Number: 19002
Gerrit-PatchSet: 7
Gerrit-Owner: Andrew Sherman
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Gergely Fürnstáhl
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tamas Mate
Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-11482: Alter Table Execute Rollback for Iceberg tables.
Andrew Sherman has uploaded a new patch set (#6). (
http://gerrit.cloudera.org:8080/19002 )
Change subject: IMPALA-11482: Alter Table Execute Rollback for Iceberg tables.
..
IMPALA-11482: Alter Table Execute Rollback for Iceberg tables.
Iceberg table modifications cause new table snapshots to be created;
these snapshots represent an earlier version of the table. The Iceberg
API provides a way to rollback the table to a previous snapshot.
This change add the ability to execute a rollback on Iceberg tables
using the following statements:
- ALTER TABLE EXECUTE ROLLBACK()
- ALTER TABLE EXECUTE ROLLBACK('')
The latter form of the command rolls back to the latest snapshot
that has a creation timestamp that is older than the specified
timestamp.
Note that when a table is rolled back to a snapshot, a new snapshot is
created with the same snapshot id, but with a new creation timestamp.
Testing:
- Added analysis unit tests.
- Added e2e tests.
- Added a Snapshot class to the pytests to encapsulate the output of
DESCRIBE HISTORY.
Change-Id: Ic74913d3b81103949ffb5eef7cc936303494f8b9
---
M common/thrift/JniCatalog.thrift
M fe/src/main/cup/sql-parser.cup
A
fe/src/main/java/org/apache/impala/analysis/AlterTableExecuteExpireSnapshotsStmt.java
A fe/src/main/java/org/apache/impala/analysis/AlterTableExecuteRollbackStmt.java
M fe/src/main/java/org/apache/impala/analysis/AlterTableExecuteStmt.java
M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
M fe/src/main/java/org/apache/impala/common/TransactionKeepalive.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java
M fe/src/main/jflex/sql-scanner.flex
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
A testdata/workloads/functional-query/queries/QueryTest/iceberg-rollback.test
M tests/common/iceberg_test_suite.py
M tests/query_test/test_iceberg.py
14 files changed, 545 insertions(+), 110 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/02/19002/6
--
To view, visit http://gerrit.cloudera.org:8080/19002
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic74913d3b81103949ffb5eef7cc936303494f8b9
Gerrit-Change-Number: 19002
Gerrit-PatchSet: 6
Gerrit-Owner: Andrew Sherman
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Gergely Fürnstáhl
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tamas Mate
Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-11631 Fix impala crashes in impala::TopNNode::Heap::Close()
Yida Wu has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19087 Change subject: IMPALA-11631 Fix impala crashes in impala::TopNNode::Heap::Close() .. IMPALA-11631 Fix impala crashes in impala::TopNNode::Heap::Close() The bug is introduced by IMPALA-11631, if RematerializeTuples() fails in ReclaimTuplePool(), it returns immediately with an error, however, some Heap unique_ptr in the partition_heaps_ could be already moved to the rematerialized_heaps, while the Close() of the TopNNode doesn't know which Heap unique_ptr is released in partition_heaps_, still calls Close() on the released Heap unique_ptr, which leads to a crash. The patch changes the logic of moving each Heap object after one rematerialize process succeeds, instead, we will move all the Heap objects in the partition_heaps_ only when all the rematerialize processes succeed. Therefore, there will be no half released partition_heaps_. Because it could be difficult for a testcase to inject an error for this case to create a crash. I did some hacking in the code to inject a memory allocation failure in certain cases, reproduced the issue, and proved the patch can solve the issue manually. Tests: Ran core tests. Passed manual test. Change-Id: Iaf45b6ef777f68e1843c076a935e4189acc6990b --- M be/src/exec/topn-node.cc 1 file changed, 2 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/87/19087/1 -- To view, visit http://gerrit.cloudera.org:8080/19087 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iaf45b6ef777f68e1843c076a935e4189acc6990b Gerrit-Change-Number: 19087 Gerrit-PatchSet: 1 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat
[Impala-ASF-CR] IMPALA-11632: Exclude log4j-1.2-api in some Ranger artifacts
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/19085 ) Change subject: IMPALA-11632: Exclude log4j-1.2-api in some Ranger artifacts .. Patch Set 1: Hi all, please let me know if you have any comment on the patch. Thanks! -- To view, visit http://gerrit.cloudera.org:8080/19085 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I72bf989a4fa04b23daf9093df8764670cc5e974d Gerrit-Change-Number: 19085 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Yida Wu Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 30 Sep 2022 18:40:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11632: Exclude log4j-1.2-api in some Ranger artifacts
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19085 ) Change subject: IMPALA-11632: Exclude log4j-1.2-api in some Ranger artifacts .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11504/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19085 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I72bf989a4fa04b23daf9093df8764670cc5e974d Gerrit-Change-Number: 19085 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 30 Sep 2022 18:36:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9448: (Addendum) Avoid generic recreate for Ozone bucket
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19082 ) Change subject: IMPALA-9448: (Addendum) Avoid generic recreate for Ozone bucket .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11503/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19082 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I31a3813fac6f2d68859ba7fed53b5c5a82502780 Gerrit-Change-Number: 19082 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 30 Sep 2022 18:18:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11632: Exclude log4j-1.2-api in some Ranger artifacts
Fang-Yu Rao has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19085 Change subject: IMPALA-11632: Exclude log4j-1.2-api in some Ranger artifacts .. IMPALA-11632: Exclude log4j-1.2-api in some Ranger artifacts After RANGER-3498, Ranger's ranger-plugins-audit, ranger-plugins-common start pulling in log4j-1.2-api, which is currently banned by Impala's frontend. To be able to compile Impala after RANGER-3498, this patch excludes log4j-1.2-api when adding those Ranger dependencies mentioned above. Change-Id: I72bf989a4fa04b23daf9093df8764670cc5e974d --- M fe/pom.xml 1 file changed, 14 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/19085/1 -- To view, visit http://gerrit.cloudera.org:8080/19085 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I72bf989a4fa04b23daf9093df8764670cc5e974d Gerrit-Change-Number: 19085 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao
[Impala-ASF-CR] IMPALA-9448: (Addendum) Avoid generic recreate for Ozone bucket
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19082 to look at the new patch set (#3). Change subject: IMPALA-9448: (Addendum) Avoid generic recreate for Ozone bucket .. IMPALA-9448: (Addendum) Avoid generic recreate for Ozone bucket Avoids using `hadoop fs` to recreate an Ozone bucket during snapshot load, as this removes the encryption key. Change-Id: I31a3813fac6f2d68859ba7fed53b5c5a82502780 --- M testdata/bin/load-test-warehouse-snapshot.sh M tests/custom_cluster/test_exchange_delays.py M tests/util/filesystem_utils.py 3 files changed, 11 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/19082/3 -- To view, visit http://gerrit.cloudera.org:8080/19082 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I31a3813fac6f2d68859ba7fed53b5c5a82502780 Gerrit-Change-Number: 19082 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Smith Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19046 ) Change subject: IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/19046 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1003556d3afc8e8216142cac4007a4c99046caeb Gerrit-Change-Number: 19046 Gerrit-PatchSet: 5 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Xiang Yang Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Fri, 30 Sep 2022 16:19:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/19046 ) Change subject: IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/19046 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1003556d3afc8e8216142cac4007a4c99046caeb Gerrit-Change-Number: 19046 Gerrit-PatchSet: 4 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Xiang Yang Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Fri, 30 Sep 2022 16:19:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19046 ) Change subject: IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8647/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/19046 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1003556d3afc8e8216142cac4007a4c99046caeb Gerrit-Change-Number: 19046 Gerrit-PatchSet: 5 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Xiang Yang Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Fri, 30 Sep 2022 16:19:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11604 Planner changes for CPU usage
Kurt Deschler has posted comments on this change. ( http://gerrit.cloudera.org:8080/19033 ) Change subject: IMPALA-11604 Planner changes for CPU usage .. Patch Set 11: (1 comment) http://gerrit.cloudera.org:8080/#/c/19033/12/fe/src/main/java/org/apache/impala/analysis/SortInfo.java File fe/src/main/java/org/apache/impala/analysis/SortInfo.java: http://gerrit.cloudera.org:8080/#/c/19033/12/fe/src/main/java/org/apache/impala/analysis/SortInfo.java@323 PS12, Line 323: / Math.max(numInstances, 1); Still not comfortable with the division (by instances/cores) here. This is better done at the fragment level where the overall contention within the fragment an be assessed and applied uniformly to operators in the fragment. As minimum, there should be some non-linear adjustment in the divisor, i.e. numInstances^(0.8). Having the division here is also misleading since the function returns a per-thread estimate, no the overall processing cost. -- To view, visit http://gerrit.cloudera.org:8080/19033 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If32dc770dfffcdd0be2ba789a7720952c68a Gerrit-Change-Number: 19033 Gerrit-PatchSet: 11 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 30 Sep 2022 15:50:52 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11504: Specializing DecimalUtil::GetScaleMultiplier().
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18861 ) Change subject: IMPALA-11504: Specializing DecimalUtil::GetScaleMultiplier(). .. Patch Set 6: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8645/ -- To view, visit http://gerrit.cloudera.org:8080/18861 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I969e2977d51313e738f72c8246db003ae43a3782 Gerrit-Change-Number: 18861 Gerrit-PatchSet: 6 Gerrit-Owner: Xiang Yang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Xiang Yang Gerrit-Comment-Date: Fri, 30 Sep 2022 15:48:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10436: Require lower privilege for external Kudu table creation
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/17640 ) Change subject: IMPALA-10436: Require lower privilege for external Kudu table creation .. Patch Set 7: (5 comments) http://gerrit.cloudera.org:8080/#/c/17640/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/17640/5//COMMIT_MSG@29 PS5, Line 29: > Thanks Csaba! Prefixes provided in your example are supported. I will try t Were these tests added somewhere? Sorry if I have just missed them. http://gerrit.cloudera.org:8080/#/c/17640/6//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/17640/6//COMMIT_MSG@7 PS6, Line 7: IMPALA-10436: Require lower privilege for external Kudu table creation To me the title suggests that creating external Kudu tables will simply need less privilege, while actually a new kind of privilege was added for this. Maybe a title like "Support storage handler privileges for external Kudu table creation" would be more descriptive. http://gerrit.cloudera.org:8080/#/c/17640/5/fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java File fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java: http://gerrit.cloudera.org:8080/#/c/17640/5/fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java@406 PS5, Line 406: // Server is used by column, function, and URI resources. Can you update the comment? http://gerrit.cloudera.org:8080/#/c/17640/5/fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java@445 PS5, Line 445: } is it valid if we don't go to the if? http://gerrit.cloudera.org:8080/#/c/17640/5/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java File fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java: http://gerrit.cloudera.org:8080/#/c/17640/5/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java@325 PS5, Line 325: // Server is used by column, function, and URI resources. stale comment -- To view, visit http://gerrit.cloudera.org:8080/17640 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7936e1d8c48696169f7ad7ad92abe44a26eea3c4 Gerrit-Change-Number: 17640 Gerrit-PatchSet: 7 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Fri, 30 Sep 2022 15:29:17 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11604 Planner changes for CPU usage
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19033 ) Change subject: IMPALA-11604 Planner changes for CPU usage .. Patch Set 12: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11502/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19033 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If32dc770dfffcdd0be2ba789a7720952c68a Gerrit-Change-Number: 19033 Gerrit-PatchSet: 12 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 30 Sep 2022 15:30:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11604 Planner changes for CPU usage
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/19033 ) Change subject: IMPALA-11604 Planner changes for CPU usage .. Patch Set 12: Fix a build error. -- To view, visit http://gerrit.cloudera.org:8080/19033 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If32dc770dfffcdd0be2ba789a7720952c68a Gerrit-Change-Number: 19033 Gerrit-PatchSet: 12 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 30 Sep 2022 15:09:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11604 Planner changes for CPU usage
Qifan Chen has uploaded a new patch set (#12). ( http://gerrit.cloudera.org:8080/19033 ) Change subject: IMPALA-11604 Planner changes for CPU usage .. IMPALA-11604 Planner changes for CPU usage This patch augments IMPALA-10992 by establishing an infrastructure to allow the weighted total amount of data to process per instance to be used as a new factor in the definition and selection of an executor group. In this patch the weight component is set to 1. The weighted amount of data processed is the sum of that in every fragment in the query to reflect the worse case of minimal parallel execution among fragments. A fragment's total is the sum of that of every node in the fragment to reflect these nodes are executed in a single thread. For each node, the weighted amount of data processed is computed with a general formula as follows. D = (I * C * W) / N where D is the weighted amount of data processed I is input cardinality C is expression evaluation cost per row, set to 1 W is average row size N is number of instances A description of the computation for each kind of plan node is given below. 1. Aggregation node: C and W are the sum of the costs and partial row widths for each AggregateInfo object. 2. AnalyticEval node: C is sum of the evaluation costs for analytic functions, partition by equal and order by equal predicate; 3. CardinalityCheck node: Both C and I are 1; 4. DataSource scan node: C is computed from a subset of the selection predicates excluding data source accepted predicates; 5. EmptySet node: I is 0; 6. Exchange node: A modification of the general formula when in broadcast mode: D = (I * C * W / N) * number of receivers; 7. Hash join node: C is sum of the evaluation cost for equi-join predicate and for other join predicate, for both probe and build side; 8. Hbase scan node: N is 1 9. Hdfs and Kudu scan node: N is mt_dop when query option mt_dop >= 1, otherwise N is number of nodes * max scan threads; 10. Nested loop join node: When the right child is not a SingularRowSrc node, C is sum of the evaluation cost for equi-join predicate and for other join predicate, for both probe and build side. When the right child is a SingularRowSrc node, the cost for build side is multiplied by the cardinality from the probe side; 11. Select node: Use the general formula; 12. SingularRowSrc node: I is 1. Since the node is involved once per input in nested loop join, the total cost of this node is computed in nested loop join; 13. Sort node: C is the evalation cost for the sort expression and W is the width of the intermediate tuple being sorted; 14. Subplan node: C is 1. I is the multiplication of the cardinality of the left and the right child; 15. Union node: C is the cost of materializing rows from all non pass-through children. W is the width of all non pass-through children; 16. Unnest node: I is the cardinality of the containing subplan node and C is 1. Testing: [TBD] Change-Id: If32dc770dfffcdd0be2ba789a7720952c68a --- M common/thrift/Frontend.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/AggregateInfo.java M fe/src/main/java/org/apache/impala/analysis/Expr.java M fe/src/main/java/org/apache/impala/analysis/SortInfo.java M fe/src/main/java/org/apache/impala/planner/AggregationNode.java M fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java M fe/src/main/java/org/apache/impala/planner/CardinalityCheckNode.java M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M fe/src/main/java/org/apache/impala/planner/EmptySetNode.java M fe/src/main/java/org/apache/impala/planner/ExchangeNode.java M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java M fe/src/main/java/org/apache/impala/planner/NestedLoopJoinNode.java M fe/src/main/java/org/apache/impala/planner/PlanFragment.java M fe/src/main/java/org/apache/impala/planner/PlanNode.java M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/planner/ResourceProfile.java M fe/src/main/java/org/apache/impala/planner/ResourceProfileBuilder.java M fe/src/main/java/org/apache/impala/planner/ScanNode.java M fe/src/main/java/org/apache/impala/planner/SelectNode.java M fe/src/main/java/org/apache/impala/planner/SingularRowSrcNode.java M fe/src/main/java/org/apache/impala/planner/SortNode.java M fe/src/main/java/org/apache/impala/planner/SubplanNode.java M fe/src/main/java/org/apache/impala/planner/UnionNode.java M fe/src/main/java/org/apache/impala/planner/UnnestNode.java M fe/src/main/java/org/apache/impala/service/Frontend
[Impala-ASF-CR] IMPALA-11604 Planner changes for CPU usage
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19033 ) Change subject: IMPALA-11604 Planner changes for CPU usage .. Patch Set 11: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/11501/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/19033 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If32dc770dfffcdd0be2ba789a7720952c68a Gerrit-Change-Number: 19033 Gerrit-PatchSet: 11 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 30 Sep 2022 14:35:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11604 Planner changes for CPU usage
Qifan Chen has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/19033 ) Change subject: IMPALA-11604 Planner changes for CPU usage .. IMPALA-11604 Planner changes for CPU usage This patch augments IMPALA-10992 by establishing an infrastructure to allow the weighted total amount of data to process per instance to be used as a new factor in the definition and selection of an executor group. In this patch the weight component is set to 1. The weighted amount of data processed is the sum of that in every fragment in the query to reflect the worse case of minimal parallel execution among fragments. A fragment's total is the sum of that of every node in the fragment to reflect these nodes are executed in a single thread. For each node, the weighted amount of data processed is computed with a general formula as follows. D = (I * C * W) / N where D is the weighted amount of data processed I is input cardinality C is expression evaluation cost per row, set to 1 W is average row size N is number of instances A description of the computation for each kind of plan node is given below. 1. Aggregation node: C and W are the sum of the costs and partial row widths for each AggregateInfo object. 2. AnalyticEval node: C is sum of the evaluation costs for analytic functions, partition by equal and order by equal predicate; 3. CardinalityCheck node: Both C and I are 1; 4. DataSource scan node: C is computed from a subset of the selection predicates excluding data source accepted predicates; 5. EmptySet node: I is 0; 6. Exchange node: A modification of the general formula when in broadcast mode: D = (I * C * W / N) * number of receivers; 7. Hash join node: C is sum of the evaluation cost for equi-join predicate and for other join predicate, for both probe and build side; 8. Hbase scan node: N is 1 9. Hdfs and Kudu scan node: N is mt_dop when query option mt_dop >= 1, otherwise N is number of nodes * max scan threads; 10. Nested loop join node: When the right child is not a SingularRowSrc node, C is sum of the evaluation cost for equi-join predicate and for other join predicate, for both probe and build side. When the right child is a SingularRowSrc node, the cost for build side is multiplied by the cardinality from the probe side; 11. Select node: Use the general formula; 12. SingularRowSrc node: I is 1. Since the node is involved once per input in nested loop join, the total cost of this node is computed in nested loop join; 13. Sort node: C is the evalation cost for the sort expression and W is the width of the intermediate tuple being sorted; 14. Subplan node: C is 1. I is the multiplication of the cardinality of the left and the right child; 15. Union node: C is the cost of materializing rows from all non pass-through children. W is the width of all non pass-through children; 16. Unnest node: I is the cardinality of the containing subplan node and C is 1. Testing: [TBD] Change-Id: If32dc770dfffcdd0be2ba789a7720952c68a --- M common/thrift/Frontend.thrift M common/thrift/Query.thrift M fe/src/main/java/org/apache/impala/analysis/AggregateInfo.java M fe/src/main/java/org/apache/impala/analysis/Expr.java M fe/src/main/java/org/apache/impala/analysis/SortInfo.java M fe/src/main/java/org/apache/impala/planner/AggregationNode.java M fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java M fe/src/main/java/org/apache/impala/planner/CardinalityCheckNode.java M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M fe/src/main/java/org/apache/impala/planner/EmptySetNode.java M fe/src/main/java/org/apache/impala/planner/ExchangeNode.java M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java M fe/src/main/java/org/apache/impala/planner/HashJoinNode.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java M fe/src/main/java/org/apache/impala/planner/NestedLoopJoinNode.java M fe/src/main/java/org/apache/impala/planner/PlanFragment.java M fe/src/main/java/org/apache/impala/planner/PlanNode.java M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/planner/ResourceProfile.java M fe/src/main/java/org/apache/impala/planner/ResourceProfileBuilder.java M fe/src/main/java/org/apache/impala/planner/ScanNode.java M fe/src/main/java/org/apache/impala/planner/SelectNode.java M fe/src/main/java/org/apache/impala/planner/SingularRowSrcNode.java M fe/src/main/java/org/apache/impala/planner/SortNode.java M fe/src/main/java/org/apache/impala/planner/SubplanNode.java M fe/src/main/java/org/apache/impala/planner/UnionNode.java M fe/src/main/java/org/apache/impala/planner/UnnestNode.java M fe/src/main/java/org/apache/impala/service/Frontend
[Impala-ASF-CR] IMPALA-11604 Planner changes for CPU usage
Qifan Chen has posted comments on this change. (
http://gerrit.cloudera.org:8080/19033 )
Change subject: IMPALA-11604 Planner changes for CPU usage
..
Patch Set 11:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/19033/10/common/thrift/Frontend.thrift
File common/thrift/Frontend.thrift:
http://gerrit.cloudera.org:8080/#/c/19033/10/common/thrift/Frontend.thrift@752
PS10, Line 752: 5: optional i64 max_processing_cost_limit
> It's probably best to stick with vcores here as what the executor group pro
This thrift structure defines an executor group set for internal use. Using
processing code limit here is better since it hides the details of translating
#vcores to processing cost.
http://gerrit.cloudera.org:8080/#/c/19033/10/common/thrift/Query.thrift
File common/thrift/Query.thrift:
http://gerrit.cloudera.org:8080/#/c/19033/10/common/thrift/Query.thrift@873
PS10, Line 873: 12: optional i64 dedicated_coord_mem_estimate;
> Vcores here also.
This thrift structure contains the relevant info for a query. Using processing
code limit here is better since it hides the details of translating #vcores to
processing cost.
http://gerrit.cloudera.org:8080/#/c/19033/10/fe/src/main/java/org/apache/impala/planner/AggregationNode.java
File fe/src/main/java/org/apache/impala/planner/AggregationNode.java:
http://gerrit.cloudera.org:8080/#/c/19033/10/fe/src/main/java/org/apache/impala/planner/AggregationNode.java@627
PS10, Line 627: .setProcessingCost(
> Use (abstract) processingCost naming instead of processedBytes.
Done
http://gerrit.cloudera.org:8080/#/c/19033/10/fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java
File fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java:
http://gerrit.cloudera.org:8080/#/c/19033/10/fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java@362
PS10, Line 362: public long computeProcessingCost() {
> rename all of these to computeProcessingCost or similar
Done
http://gerrit.cloudera.org:8080/#/c/19033/10/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java
File fe/src/main/java/org/apache/impala/planner/HashJoinNode.java:
http://gerrit.cloudera.org:8080/#/c/19033/10/fe/src/main/java/org/apache/impala/planner/HashJoinNode.java@308
PS10, Line 308: float eqJoinPredicateEvalCost =
> eqJoinPredicateEvalCost
Done
http://gerrit.cloudera.org:8080/#/c/19033/10/fe/src/main/java/org/apache/impala/util/ExprUtil.java
File fe/src/main/java/org/apache/impala/util/ExprUtil.java:
http://gerrit.cloudera.org:8080/#/c/19033/10/fe/src/main/java/org/apache/impala/util/ExprUtil.java@109
PS10, Line 109: public static float computeExprsTotalCost(List exprs) {
> Use List to share the same function.
Done
--
To view, visit http://gerrit.cloudera.org:8080/19033
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If32dc770dfffcdd0be2ba789a7720952c68a
Gerrit-Change-Number: 19033
Gerrit-PatchSet: 11
Gerrit-Owner: Qifan Chen
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Qifan Chen
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Fri, 30 Sep 2022 14:22:36 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11606: add 'untracked memory' metric.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19029 ) Change subject: IMPALA-11606: add 'untracked memory' metric. .. Patch Set 4: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8644/ -- To view, visit http://gerrit.cloudera.org:8080/19029 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib16e00109d732f759c96c7a967eb1cc32124a03f Gerrit-Change-Number: 19029 Gerrit-PatchSet: 4 Gerrit-Owner: Xiang Yang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Fri, 30 Sep 2022 13:58:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11565: Support IF NOT EXISTS in alter table add columns for kudu table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18953 ) Change subject: IMPALA-11565: Support IF NOT EXISTS in alter table add columns for kudu table .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8646/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18953 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I82590e5372e881f2e81d4ed3dd0d32a2d3ddb517 Gerrit-Change-Number: 18953 Gerrit-PatchSet: 8 Gerrit-Owner: Baike Xia Gerrit-Reviewer: Baike Xia Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Fri, 30 Sep 2022 13:58:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11565: Support IF NOT EXISTS in alter table add columns for kudu table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18953 ) Change subject: IMPALA-11565: Support IF NOT EXISTS in alter table add columns for kudu table .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11500/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18953 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I82590e5372e881f2e81d4ed3dd0d32a2d3ddb517 Gerrit-Change-Number: 18953 Gerrit-PatchSet: 8 Gerrit-Owner: Baike Xia Gerrit-Reviewer: Baike Xia Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Fri, 30 Sep 2022 11:16:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11565: Support IF NOT EXISTS in alter table add columns for kudu table
Baike Xia has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/18953 ) Change subject: IMPALA-11565: Support IF NOT EXISTS in alter table add columns for kudu table .. IMPALA-11565: Support IF NOT EXISTS in alter table add columns for kudu table Impala already supports IF NOT EXISTS in alter table add columns for general hive table in IMPALA-7832, but not for kudu table. This patch try to add such semantics for kudu table. Testing: - Updated E2E DDL tests Change-Id: I82590e5372e881f2e81d4ed3dd0d32a2d3ddb517 --- M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java M testdata/workloads/functional-query/queries/QueryTest/kudu_alter.test 3 files changed, 64 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/18953/8 -- To view, visit http://gerrit.cloudera.org:8080/18953 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I82590e5372e881f2e81d4ed3dd0d32a2d3ddb517 Gerrit-Change-Number: 18953 Gerrit-PatchSet: 8 Gerrit-Owner: Baike Xia Gerrit-Reviewer: Baike Xia Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-11504: Specializing DecimalUtil::GetScaleMultiplier().
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18861 ) Change subject: IMPALA-11504: Specializing DecimalUtil::GetScaleMultiplier(). .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8645/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18861 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I969e2977d51313e738f72c8246db003ae43a3782 Gerrit-Change-Number: 18861 Gerrit-PatchSet: 6 Gerrit-Owner: Xiang Yang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Xiang Yang Gerrit-Comment-Date: Fri, 30 Sep 2022 10:41:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11504: Specializing DecimalUtil::GetScaleMultiplier().
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18861 ) Change subject: IMPALA-11504: Specializing DecimalUtil::GetScaleMultiplier(). .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11499/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18861 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I969e2977d51313e738f72c8246db003ae43a3782 Gerrit-Change-Number: 18861 Gerrit-PatchSet: 5 Gerrit-Owner: Xiang Yang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Xiang Yang Gerrit-Comment-Date: Fri, 30 Sep 2022 10:39:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11504: Specializing DecimalUtil::GetScaleMultiplier().
Xiang Yang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18861 ) Change subject: IMPALA-11504: Specializing DecimalUtil::GetScaleMultiplier(). .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/18861/4//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/18861/4//COMMIT_MSG@7 PS4, Line 7: > nit: Lack of space Done -- To view, visit http://gerrit.cloudera.org:8080/18861 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I969e2977d51313e738f72c8246db003ae43a3782 Gerrit-Change-Number: 18861 Gerrit-PatchSet: 6 Gerrit-Owner: Xiang Yang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Xiang Yang Gerrit-Comment-Date: Fri, 30 Sep 2022 10:40:23 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [IMPALA-11625] Support create/drop materialized view syntax on IMPALA
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19050 ) Change subject: [IMPALA-11625] Support create/drop materialized view syntax on IMPALA .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11498/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19050 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I77fdd34bf04a8994a215170747249356cd40622b Gerrit-Change-Number: 19050 Gerrit-PatchSet: 6 Gerrit-Owner: pengdou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 30 Sep 2022 10:37:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11504: Specializing DecimalUtil::GetScaleMultiplier().
Hello Daniel Becker, lipeng...@sensorsdata.cn, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18861 to look at the new patch set (#6). Change subject: IMPALA-11504: Specializing DecimalUtil::GetScaleMultiplier(). .. IMPALA-11504: Specializing DecimalUtil::GetScaleMultiplier(). Currently decimal-util.h didn't specialize DecimalUtil ::GetScaleMultiplier(), causing more performance loss when calculate Decimal16Value division. Testing: - Ran existing jobs. - Add decimal-util-benchmark. Change-Id: I969e2977d51313e738f72c8246db003ae43a3782 --- M be/src/benchmarks/CMakeLists.txt A be/src/benchmarks/decimal-util-benchmark.cc M be/src/runtime/decimal-test.cc M be/src/util/decimal-util.h 4 files changed, 232 insertions(+), 30 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/61/18861/6 -- To view, visit http://gerrit.cloudera.org:8080/18861 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I969e2977d51313e738f72c8246db003ae43a3782 Gerrit-Change-Number: 18861 Gerrit-PatchSet: 6 Gerrit-Owner: Xiang Yang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Xiang Yang
[Impala-ASF-CR] IMPALA-11504:Specializing DecimalUtil::GetScaleMultiplier().
Hello Daniel Becker, lipeng...@sensorsdata.cn, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18861 to look at the new patch set (#5). Change subject: IMPALA-11504:Specializing DecimalUtil::GetScaleMultiplier(). .. IMPALA-11504:Specializing DecimalUtil::GetScaleMultiplier(). Currently decimal-util.h didn't specialize DecimalUtil ::GetScaleMultiplier(), causing more performance loss when calculate Decimal16Value division. Testing: - Ran existing jobs. - Add decimal-util-benchmark. Change-Id: I969e2977d51313e738f72c8246db003ae43a3782 --- M be/src/benchmarks/CMakeLists.txt A be/src/benchmarks/decimal-util-benchmark.cc M be/src/runtime/decimal-test.cc M be/src/util/decimal-util.h 4 files changed, 232 insertions(+), 30 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/61/18861/5 -- To view, visit http://gerrit.cloudera.org:8080/18861 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I969e2977d51313e738f72c8246db003ae43a3782 Gerrit-Change-Number: 18861 Gerrit-PatchSet: 5 Gerrit-Owner: Xiang Yang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Xiang Yang
[Impala-ASF-CR] [IMPALA-11625] Support create/drop materialized view syntax on IMPALA
pengdou has uploaded a new patch set (#6). (
http://gerrit.cloudera.org:8080/19050 )
Change subject: [IMPALA-11625] Support create/drop materialized view syntax on
IMPALA
..
[IMPALA-11625] Support create/drop materialized view syntax on IMPALA
For infrequently modified datasets, building suitable materialized
views and querying on materialized views directly can significantly
reduce query response time.
Currently IMPALA only treats materialized view built by hive as
a normal table and does not support CREATE MATERIALIZED VIEW AS
SELECTED and DROP MATERIALIZED VIEW syntax on IMPALA.
CREATE TABLE AS SELECT can save the intermediate results of the
query, but it is very inconvenient to use and maintain, for example,
the calculation logic is not visible to the user.
So I implemented supporting of create/drop materialized view syntax
on impala.
CREATE MATERIALIZED VIEW syntax:
CREATE MATERIALIZED VIEW [IF NOT EXISTS] [db_name.]table_name
[PARTITIONED BY (col_name[, ...])]
[SORT BY ([column [, column ...]])]
[COMMENT 'table_comment']
[ROW FORMAT row_format]
[WITH SERDEPROPERTIES ('key1'='value1', 'key2'='value2', ...)]
[STORED AS cmvas_file_format]
LOCATION 'hdfs_path'
[CACHED IN 'pool_name' [WITH REPLICATION = integer] | UNCACHED]
[TBLPROPERTIES ('key1'='value1', 'key2'='value2', ...)]
AS
select_statement
cmvas_file_format:
PARQUET
| TEXTFILE
DROP MATERIALIZED VIEW syntax:
DROP MATERIALIZED VIEW [IF EXISTS] [db_name.]table_name
Change-Id: I77fdd34bf04a8994a215170747249356cd40622b
---
M be/src/service/client-request-state.cc
M common/thrift/JniCatalog.thrift
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
A
fe/src/main/java/org/apache/impala/analysis/CreateMaterializedViewAsSelectStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java
M fe/src/main/java/org/apache/impala/analysis/LimitElement.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/catalog/View.java
M fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/main/java/org/apache/impala/planner/PlannerContext.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/jflex/sql-scanner.flex
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
M fe/src/test/java/org/apache/impala/analysis/ParserTest.java
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
A
testdata/workloads/functional-query/queries/QueryTest/create-materialized-view-as-select.test
M tests/metadata/test_ddl.py
23 files changed, 815 insertions(+), 41 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/19050/6
--
To view, visit http://gerrit.cloudera.org:8080/19050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I77fdd34bf04a8994a215170747249356cd40622b
Gerrit-Change-Number: 19050
Gerrit-PatchSet: 6
Gerrit-Owner: pengdou
Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-11504:Specializing DecimalUtil::GetScaleMultiplier().
Xiang Yang has posted comments on this change. ( http://gerrit.cloudera.org:8080/18861 ) Change subject: IMPALA-11504:Specializing DecimalUtil::GetScaleMultiplier(). .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/18861/4/be/src/util/decimal-util.h File be/src/util/decimal-util.h: http://gerrit.cloudera.org:8080/#/c/18861/4/be/src/util/decimal-util.h@37 PS4, Line 37: // The scale upper bound for GetScaleMultiplier() : static constexpr int INT32_SCALE_UPPER_BOUND = ColumnType::MAX_DECIMAL4_PRECISION + 1; : // The scale upper bound for GetScaleMultiplier() : static constexpr int INT64_SCALE_UPPER_BOUND = ColumnType::MAX_DECIMAL8_PRECISION + 1; : // The scale upper bound for GetScaleMultiplier() : static constexpr int INT128_SCALE_UPPER_BOUND = ColumnType::MAX_PRECISION + 1; : // The scale upper bound for GetScaleMultiplier() : static constexpr int INT256_SCALE_UPPER_BOUND = 77; > I don't think these should be public class-level constants, they should be These constants are referenced in decimal-test.cc and decimal-util-benchmark.cc, if we don't make them public, we should redefine other constants with same value in decimal-test.cc and decimal-util-benchmark.cc. Do you have any better solution? -- To view, visit http://gerrit.cloudera.org:8080/18861 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I969e2977d51313e738f72c8246db003ae43a3782 Gerrit-Change-Number: 18861 Gerrit-PatchSet: 4 Gerrit-Owner: Xiang Yang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Xiang Yang Gerrit-Comment-Date: Fri, 30 Sep 2022 09:50:49 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11606: add 'untracked memory' metric.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19029 ) Change subject: IMPALA-11606: add 'untracked memory' metric. .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8644/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/19029 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib16e00109d732f759c96c7a967eb1cc32124a03f Gerrit-Change-Number: 19029 Gerrit-PatchSet: 4 Gerrit-Owner: Xiang Yang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jian Zhang Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Fri, 30 Sep 2022 08:49:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3119: DDL support for bucketed tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19055 ) Change subject: IMPALA-3119: DDL support for bucketed tables .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8643/ -- To view, visit http://gerrit.cloudera.org:8080/19055 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I919b4d4139bc3a7784fa6fdb6f064e25666d548e Gerrit-Change-Number: 19055 Gerrit-PatchSet: 5 Gerrit-Owner: Baike Xia Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 30 Sep 2022 08:37:42 + Gerrit-HasComments: No
[Impala-ASF-CR] [IMPALA-11625] Support create/drop materialized view syntax on IMPALA
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19050 ) Change subject: [IMPALA-11625] Support create/drop materialized view syntax on IMPALA .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11497/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19050 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I77fdd34bf04a8994a215170747249356cd40622b Gerrit-Change-Number: 19050 Gerrit-PatchSet: 5 Gerrit-Owner: pengdou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 30 Sep 2022 07:56:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3119: DDL support for bucketed tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19055 ) Change subject: IMPALA-3119: DDL support for bucketed tables .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/11496/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/19055 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I919b4d4139bc3a7784fa6fdb6f064e25666d548e Gerrit-Change-Number: 19055 Gerrit-PatchSet: 7 Gerrit-Owner: Baike Xia Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 30 Sep 2022 07:54:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11612: fix ORDER BY expression rewrite bug
jhkc...@qq.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/19049 ) Change subject: IMPALA-11612: fix ORDER BY expression rewrite bug .. Patch Set 3: > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8642/ Is the build server disk space insufficient? -- To view, visit http://gerrit.cloudera.org:8080/19049 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I438c61c647e3f99665c94ea15e3a9d98b9ff9912 Gerrit-Change-Number: 19049 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Fri, 30 Sep 2022 07:47:57 + Gerrit-HasComments: No
[Impala-ASF-CR] [IMPALA-11625] Support create/drop materialized view syntax on IMPALA
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/19050 )
Change subject: [IMPALA-11625] Support create/drop materialized view syntax on
IMPALA
..
Patch Set 5:
(30 comments)
http://gerrit.cloudera.org:8080/#/c/19050/5/be/src/service/client-request-state.cc
File be/src/service/client-request-state.cc:
http://gerrit.cloudera.org:8080/#/c/19050/5/be/src/service/client-request-state.cc@708
PS5, Line 708: // portion of the operation). The exception is if the user
specified IF NOT EXISTS and the table already
line too long (106 > 90)
http://gerrit.cloudera.org:8080/#/c/19050/5/be/src/service/client-request-state.cc@1116
PS5, Line 1116: // When the coordinator is not available for CTAS/CMvAS
that requires a coordinator, check
line too long (94 > 90)
http://gerrit.cloudera.org:8080/#/c/19050/5/fe/src/main/java/org/apache/impala/analysis/CreateMaterializedViewAsSelectStmt.java
File
fe/src/main/java/org/apache/impala/analysis/CreateMaterializedViewAsSelectStmt.java:
http://gerrit.cloudera.org:8080/#/c/19050/5/fe/src/main/java/org/apache/impala/analysis/CreateMaterializedViewAsSelectStmt.java@51
PS5, Line 51: private final static EnumSet
SUPPORTED_INSERT_MATERIALIZED_VIEW_FORMATS =
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/19050/5/fe/src/main/java/org/apache/impala/analysis/CreateMaterializedViewAsSelectStmt.java@68
PS5, Line 68: public CreateMaterializedViewAsSelectStmt(CmvasParams params,
List planHints) {
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/19050/5/fe/src/main/java/org/apache/impala/analysis/CreateMaterializedViewAsSelectStmt.java@104
PS5, Line 104: if
(!SUPPORTED_INSERT_MATERIALIZED_VIEW_FORMATS.contains(createStmt_.getFileFormat()))
{
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/19050/5/fe/src/main/java/org/apache/impala/analysis/CreateMaterializedViewAsSelectStmt.java@122
PS5, Line 122: throw new AnalysisException("Query statement in
CreateMaterializedViewAsSelectStmt " +
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/19050/5/fe/src/main/java/org/apache/impala/analysis/CreateMaterializedViewAsSelectStmt.java@129
PS5, Line 129: throw new AnalysisException("Query statement in
CreateMaterializedViewAsSelectStmt " +
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/19050/5/fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java
File fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java:
http://gerrit.cloudera.org:8080/#/c/19050/5/fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java@142
PS5, Line 142: // DROP MATERIALIZED VIEW IF EXISTS 'table' succeeds,
similarly to Hive, but unlike postgres.
line too long (101 > 90)
http://gerrit.cloudera.org:8080/#/c/19050/5/fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java@145
PS5, Line 145: "DROP MATERIALIZED VIEW not allowed on a
none-materialized view: %s.%s", dbName_, getTbl()));
line too long (105 > 90)
http://gerrit.cloudera.org:8080/#/c/19050/5/fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java@151
PS5, Line 151: "DROP TABLE/VIEW not allowed on a materialized view:
%s.%s", dbName_, getTbl()));
line too long (93 > 90)
http://gerrit.cloudera.org:8080/#/c/19050/5/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
File fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java:
http://gerrit.cloudera.org:8080/#/c/19050/5/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@477
PS5, Line 477: return getCreateTableSql(dbName, tableName, tableComment,
columnsSql, partitionColumnsSql,
line too long (94 > 90)
http://gerrit.cloudera.org:8080/#/c/19050/5/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@478
PS5, Line 478: primaryKeysSql, foreignKeysSql, kuduPartitionByParams,
sortProperties, tblProperties,
line too long (93 > 90)
http://gerrit.cloudera.org:8080/#/c/19050/5/fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java
File fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java:
http://gerrit.cloudera.org:8080/#/c/19050/5/fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java@80
PS5, Line 80: && !singleNodePlan.hasLimit() ||
analysisResult.isCreateMaterializedViewAsSelectStmt()) {
line too long (97 > 90)
http://gerrit.cloudera.org:8080/#/c/19050/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/19050/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@2671
PS5, Line 2671: params.getTable_name().table_name, "Load for DROP
TABLE/VIEW/MATERIALIZED VIEW ", null);
line too long (98 > 90)
http://gerrit.cloudera.org:8080/#/c/19050/5/fe/src/main/java/org/apache/impala/servi
[Impala-ASF-CR] IMPALA-3119: DDL support for bucketed tables
Baike Xia has uploaded a new patch set (#7). (
http://gerrit.cloudera.org:8080/19055 )
Change subject: IMPALA-3119: DDL support for bucketed tables
..
IMPALA-3119: DDL support for bucketed tables
Add syntactic support for creating bucketed table.
The specific syntax is as follows:
CREATE [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name(
col_name data_type
[constraint_specification]
[COMMENT 'col_comment']
[, ...]
)
[PARTITIONED BY (col_name data_type [COMMENT 'col_comment'], ...)]
[BUCKETED BY HASH([column [, column ...]])|RANDOM INTO 24 BUCKETS
[SORT BY ([column [, column ...]])]
[COMMENT 'table_comment']
[ROW FORMAT row_format]
[WITH SERDEPROPERTIES ('key1'='value1', 'key2'='value2', ...)]
[STORED AS file_format]
[LOCATION 'hdfs_path']
[CACHED IN 'pool_name' [WITH REPLICATION = integer] | UNCACHED]
[TBLPROPERTIES ('key1'='value1', 'key2'='value2', ...)]
Instructions:
1. CLUSTERED BY of Hive is not supported, because HINT has the keyword;
2. The bucket partitioning algorithm contains HASH, RANDOM, KUDU_HASH.
The default value is HASH;
3. INTO 24 BUCKETS, specifies the number of buckets, the default value
is 16;
4. Create Bucketed Table statements that do not support Kudu and
Iceberg tables, but for a Kudu table,
the hash partition is equivalent to a bucket,
and the optimization rule applies to join query;
5. In the current version, alter operations(add/drop/change/replace
columns) on bucketed tables are not supported;
This COMMIT is the first subtask of IMPALA-3118.
Change-Id: I919b4d4139bc3a7784fa6fdb6f064e25666d548e
---
M common/thrift/CatalogObjects.thrift
M common/thrift/JniCatalog.thrift
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeFileStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/TableDef.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
A fe/src/main/java/org/apache/impala/util/BucketUtils.java
M fe/src/main/jflex/sql-scanner.flex
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
M fe/src/test/java/org/apache/impala/analysis/ParserTest.java
M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java
M testdata/workloads/functional-query/queries/QueryTest/create-table.test
15 files changed, 411 insertions(+), 21 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/19055/7
--
To view, visit http://gerrit.cloudera.org:8080/19055
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I919b4d4139bc3a7784fa6fdb6f064e25666d548e
Gerrit-Change-Number: 19055
Gerrit-PatchSet: 7
Gerrit-Owner: Baike Xia
Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] [IMPALA-11625] Support create/drop materialized view syntax on IMPALA
pengdou has uploaded this change for review. (
http://gerrit.cloudera.org:8080/19050
Change subject: [IMPALA-11625] Support create/drop materialized view syntax on
IMPALA
..
[IMPALA-11625] Support create/drop materialized view syntax on IMPALA
For infrequently modified datasets, building suitable materialized
views and querying on materialized views directly can significantly
reduce query response time.
Currently IMPALA only treats materialized view built by hive as
a normal table and does not support CREATE MATERIALIZED VIEW AS
SELECTED and DROP MATERIALIZED VIEW syntax on IMPALA.
CREATE TABLE AS SELECT can save the intermediate results of the
query, but it is very inconvenient to use and maintain, for example,
the calculation logic is not visible to the user.
So I implemented supporting of create/drop materialized view syntax
on impala.
CREATE MATERIALIZED VIEW syntax:
CREATE MATERIALIZED VIEW [IF NOT EXISTS] [db_name.]table_name
[PARTITIONED BY (col_name[, ...])]
[SORT BY ([column [, column ...]])]
[COMMENT 'table_comment']
[ROW FORMAT row_format]
[WITH SERDEPROPERTIES ('key1'='value1', 'key2'='value2', ...)]
[STORED AS cmvas_file_format]
LOCATION 'hdfs_path'
[CACHED IN 'pool_name' [WITH REPLICATION = integer] | UNCACHED]
[TBLPROPERTIES ('key1'='value1', 'key2'='value2', ...)]
AS
select_statement
cmvas_file_format:
PARQUET
| TEXTFILE
DROP MATERIALIZED VIEW syntax:
DROP MATERIALIZED VIEW [IF EXISTS] [db_name.]table_name
Change-Id: I77fdd34bf04a8994a215170747249356cd40622b
---
M be/src/service/client-request-state.cc
M common/thrift/JniCatalog.thrift
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
A
fe/src/main/java/org/apache/impala/analysis/CreateMaterializedViewAsSelectStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java
M fe/src/main/java/org/apache/impala/analysis/LimitElement.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/catalog/View.java
M fe/src/main/java/org/apache/impala/planner/DistributedPlanner.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/main/java/org/apache/impala/planner/PlannerContext.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/jflex/sql-scanner.flex
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
M fe/src/test/java/org/apache/impala/analysis/ParserTest.java
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
A
testdata/workloads/functional-query/queries/QueryTest/create-materialized-view-as-select.test
M tests/metadata/test_ddl.py
23 files changed, 793 insertions(+), 37 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/19050/5
--
To view, visit http://gerrit.cloudera.org:8080/19050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I77fdd34bf04a8994a215170747249356cd40622b
Gerrit-Change-Number: 19050
Gerrit-PatchSet: 5
Gerrit-Owner: pengdou
[Impala-ASF-CR] IMPALA-11624: Bump Impyla dependency to 0.18.0
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18456 ) Change subject: IMPALA-11624: Bump Impyla dependency to 0.18.0 .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/18456/6/bin/impala-config.sh File bin/impala-config.sh: http://gerrit.cloudera.org:8080/#/c/18456/6/bin/impala-config.sh@179 PS6, Line 179: shell/ext-py > Hi Csaba. Just wondering, what will happen with shell/ext-py/thrift-0.14.2 Oops, I need to update that to - for some reason I thought the copied eggs in ext-py were removed -- To view, visit http://gerrit.cloudera.org:8080/18456 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7265558b0e07959c606cba73cd251c3edfcb3ed5 Gerrit-Change-Number: 18456 Gerrit-PatchSet: 6 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 30 Sep 2022 07:25:55 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11612: fix ORDER BY expression rewrite bug
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19049 ) Change subject: IMPALA-11612: fix ORDER BY expression rewrite bug .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8642/ -- To view, visit http://gerrit.cloudera.org:8080/19049 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I438c61c647e3f99665c94ea15e3a9d98b9ff9912 Gerrit-Change-Number: 19049 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Fri, 30 Sep 2022 07:23:03 + Gerrit-HasComments: No
