[Impala-ASF-CR] IMPALA-10801: Check the latest compaction Id before serving ACID table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17697 ) Change subject: IMPALA-10801: Check the latest compaction Id before serving ACID table .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/9179/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17697 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I86a112a77980fef7f6238978bc9668a65262101e Gerrit-Change-Number: 17697 Gerrit-PatchSet: 7 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Tue, 27 Jul 2021 06:36:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10801: Check the latest compaction Id before serving ACID table
Yu-Wen Lai has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/17697 ) Change subject: IMPALA-10801: Check the latest compaction Id before serving ACID table .. IMPALA-10801: Check the latest compaction Id before serving ACID table Since compactions don't advance write id, we don't know if a table/partition is compacted by comparing writeIdList. A possible issue is that CatalogD provides obsolete file metadata and causes a runtime error. In order to fix this issue, we introduced a HMS API that can get the latest compaction record for a table/partition (HIVE-24828). In CatalogD, we cache compaction id while loading partitions and compare the cached id with the latest compaction id before serving. If there is a newer compaction happened, it would refresh the file metadata. Besides, this patch also change how to replace the existing table after a table full reloading. The current way is to replace the table if the catalog version is not changed. For transactional tables, things get additional complexity given that file metadata refreshing and full table reloading can happen together. We can actually use writeIdList to determine whether we should replace the table for transactional tables. As long as the updated table has more recent writeIdList than the existing one, we are safe to replace the table. For Non-transactional tables, we still keep original behavior. Testing: - Add a test in PartialCatalogInfoWriteIdTest Change-Id: I86a112a77980fef7f6238978bc9668a65262101e --- M bin/impala-config.sh M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java M fe/src/main/java/org/apache/impala/util/AcidUtils.java M fe/src/test/java/org/apache/impala/catalog/PartialCatalogInfoWriteIdTest.java M testdata/bin/create-load-data.sh R testdata/cluster/ranger/setup/policy_5_revised.json 9 files changed, 370 insertions(+), 44 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/17697/7 -- To view, visit http://gerrit.cloudera.org:8080/17697 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I86a112a77980fef7f6238978bc9668a65262101e Gerrit-Change-Number: 17697 Gerrit-PatchSet: 7 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai
[Impala-ASF-CR] Frontend changes to enable 'stored as JSONFILE' This change will allow usage of commands that do not require reading the Json File like: - Create Table stored as JSONFILE - Sh
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17727 ) Change subject: Frontend changes to enable 'stored as JSONFILE' This change will allow usage of commands that do not require reading the Json File like: - Create Table stored as JSONFILE - Show Create Table - Describe .. Patch Set 2: (2 comments) We can add FE tests in AnalyzeDDLTest#TestCreateTable, e.g. https://github.com/apache/impala/blob/59d32853ee42886ae683aac95a8be7f9c89b8eb7/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java#L2571-L2589 and e2e tests in testdata/workloads/functional-query/queries/QueryTest/show-create-table.test which is used in tests/metadata/test_show_create_table.py::TestShowCreateTable::test_show_create_table. For tests on DESCRIBE, we can wait for the patch of loading json tables, and then add them in tests/metadata/test_metadata_query_statements.py. http://gerrit.cloudera.org:8080/#/c/17727/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/17727/2//COMMIT_MSG@7 PS2, Line 7: Frontend changes to enable 'stored as JSONFILE' nit: need a blank line after the title and mention the JIRA id IMPALA-10797 at the beginning. http://gerrit.cloudera.org:8080/#/c/17727/2/fe/src/main/cup/sql-parser.cup File fe/src/main/cup/sql-parser.cup: http://gerrit.cloudera.org:8080/#/c/17727/2/fe/src/main/cup/sql-parser.cup@299 PS2, Line 299: KW_JSON To be consistent with Hive, let's use KW_JSONFILE. -- To view, visit http://gerrit.cloudera.org:8080/17727 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5b8cb2f59df3af09902b49d3bdac16c19954b305 Gerrit-Change-Number: 17727 Gerrit-PatchSet: 2 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 27 Jul 2021 01:56:34 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10813: Invalidate external table from catalog cache for truncate table HMS api
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17705 ) Change subject: IMPALA-10813: Invalidate external table from catalog cache for truncate table HMS api .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/17705 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I734c2b5f371291fef32badab9efc886b4b067e10 Gerrit-Change-Number: 17705 Gerrit-PatchSet: 4 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Tue, 27 Jul 2021 01:35:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10778: Allow impala-shell to connect directly to HS2
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17660 ) Change subject: IMPALA-10778: Allow impala-shell to connect directly to HS2 .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/9178/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17660 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I674a45640a4a7b3c9a577830dbc7b16a89865a9e Gerrit-Change-Number: 17660 Gerrit-PatchSet: 7 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Steve Carlin Gerrit-Comment-Date: Tue, 27 Jul 2021 00:18:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10778: Allow impala-shell to connect directly to HS2
Steve Carlin has posted comments on this change. (
http://gerrit.cloudera.org:8080/17660 )
Change subject: IMPALA-10778: Allow impala-shell to connect directly to HS2
..
Patch Set 5:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/17660/5/tests/shell/test_shell_interactive.py
File tests/shell/test_shell_interactive.py:
http://gerrit.cloudera.org:8080/#/c/17660/5/tests/shell/test_shell_interactive.py@352
PS5, Line 352: pytest.skip("Failed, need to investigate.")
> Please file a JIRA and reference here if you want to leave this TODO.
Done
http://gerrit.cloudera.org:8080/#/c/17660/5/tests/shell/test_shell_interactive.py@1092
PS5, Line 1092: pytest.skip("The now() function is not supported in
strict hs2 mode.")
> Heh, yeah, this is more of a frontend issue
Done
--
To view, visit http://gerrit.cloudera.org:8080/17660
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I674a45640a4a7b3c9a577830dbc7b16a89865a9e
Gerrit-Change-Number: 17660
Gerrit-PatchSet: 5
Gerrit-Owner: Steve Carlin
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Steve Carlin
Gerrit-Comment-Date: Mon, 26 Jul 2021 23:57:28 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10778: Allow impala-shell to connect directly to HS2
Hello Kurt Deschler, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17660 to look at the new patch set (#7). Change subject: IMPALA-10778: Allow impala-shell to connect directly to HS2 .. IMPALA-10778: Allow impala-shell to connect directly to HS2 Impala-shell already uses HS2 protocol to connect to Impalad. This commit allows impala-shell to connect to any server (for example, Hive) using the hs2 protocol. This will be done via the "--strict_hs2_protocol" option. When the "--strict_hs2_protocol" option is turned on, only features supported by hs2 will work. For instance, "runtime-profile" is an impalad specific feature and will be disabled. The "--strict_hs2_protocol" will only work on servers that abide by the strict definition of what is supported by HS2. So one will be able to connect to Hive in this mode, but connections to Impala will not work. Any feature supported by Hive (e.g. kerberos authentication) should work as well. Change-Id: I674a45640a4a7b3c9a577830dbc7b16a89865a9e --- M fe/src/test/resources/hive-site.xml.py M shell/impala_client.py M shell/impala_shell.py M shell/impala_shell_config_defaults.py M shell/option_parser.py M tests/common/impala_test_suite.py M tests/common/test_dimensions.py M tests/conftest.py M tests/shell/test_shell_client.py M tests/shell/test_shell_commandline.py M tests/shell/test_shell_interactive.py M tests/shell/util.py 12 files changed, 322 insertions(+), 81 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/60/17660/7 -- To view, visit http://gerrit.cloudera.org:8080/17660 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I674a45640a4a7b3c9a577830dbc7b16a89865a9e Gerrit-Change-Number: 17660 Gerrit-PatchSet: 7 Gerrit-Owner: Steve Carlin Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Steve Carlin
[Impala-ASF-CR] IMPALA-10821 Fix TestTPCHJoinQueries.test outer joins failed
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17716 ) Change subject: IMPALA-10821 Fix TestTPCHJoinQueries.test_outer_joins failed .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/17716 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7e068d9da03517f8316e7a2505ce1466523d5917 Gerrit-Change-Number: 17716 Gerrit-PatchSet: 4 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Mon, 26 Jul 2021 22:22:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10821 Fix TestTPCHJoinQueries.test outer joins failed
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17716 ) Change subject: IMPALA-10821 Fix TestTPCHJoinQueries.test_outer_joins failed .. IMPALA-10821 Fix TestTPCHJoinQueries.test_outer_joins failed A new added testcase in TestTPCHJoinQueries.test_outer_joins can't pass the s3 build because the plan generated in s3 build is not the same as the default hdfs build for the scan node. In s3 build, it is "SCAN S3", while in hdfs build, it is "SCAN HDFS". The patch changed the testcase to use $FILESYSTEM_NAME, which will be changed according to the file system the testcase is using. Tests: Reran and passed the failed testcase in s3 build. Change-Id: I7e068d9da03517f8316e7a2505ce1466523d5917 Reviewed-on: http://gerrit.cloudera.org:8080/17716 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M testdata/workloads/tpch/queries/tpch-outer-joins.test 1 file changed, 2 insertions(+), 2 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/17716 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I7e068d9da03517f8316e7a2505ce1466523d5917 Gerrit-Change-Number: 17716 Gerrit-PatchSet: 5 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 ) Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server. .. Patch Set 9: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7347/ -- To view, visit http://gerrit.cloudera.org:8080/17576 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c Gerrit-Change-Number: 17576 Gerrit-PatchSet: 9 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 26 Jul 2021 21:46:17 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-3430: Runtime filter : Extend runtime filter to support Min/Max values for HDFS scans
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17706 ) Change subject: [WIP] IMPALA-3430: Runtime filter : Extend runtime filter to support Min/Max values for HDFS scans .. Patch Set 6: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/9177/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/17706 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7c2bb5baad622051d1002c9c162c672d428e5446 Gerrit-Change-Number: 17706 Gerrit-PatchSet: 6 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Comment-Date: Mon, 26 Jul 2021 21:38:46 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-3430: Runtime filter : Extend runtime filter to support Min/Max values for HDFS scans
Qifan Chen has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/17706 ) Change subject: [WIP] IMPALA-3430: Runtime filter : Extend runtime filter to support Min/Max values for HDFS scans .. [WIP] IMPALA-3430: Runtime filter : Extend runtime filter to support Min/Max values for HDFS scans This patches enables min/max filtering for non-correlated subqueries that return one row. In this case, the filters are built from the results of the subqueries and the filtering target is the scan node to be qualified by one of the subqueries. Shown below is one such query that normally gets compiled into a nested loop join. The filtering limits the values from column store_sales.ss_sales_price within the range [-infinite, avg(ss_wholesale_cost)]. select count(*) from store_sales where ss_sales_price <= (select min(ss_wholesale_cost) from store_sales); Change-Id: I7c2bb5baad622051d1002c9c162c672d428e5446 --- M be/src/exec/nested-loop-join-builder.cc M be/src/exec/nested-loop-join-builder.h M be/src/exec/nested-loop-join-node.cc M be/src/runtime/coordinator.cc M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java M fe/src/main/java/org/apache/impala/analysis/Predicate.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java M fe/src/main/java/org/apache/impala/planner/AggregationNode.java M fe/src/main/java/org/apache/impala/planner/JoinNode.java M fe/src/main/java/org/apache/impala/planner/NestedLoopJoinNode.java M fe/src/main/java/org/apache/impala/planner/PlanNode.java M fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java 14 files changed, 354 insertions(+), 24 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/17706/6 -- To view, visit http://gerrit.cloudera.org:8080/17706 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7c2bb5baad622051d1002c9c162c672d428e5446 Gerrit-Change-Number: 17706 Gerrit-PatchSet: 6 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen
[Impala-ASF-CR] IMPALA-10813: Invalidate external table from catalog cache for truncate table HMS api
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17705 ) Change subject: IMPALA-10813: Invalidate external table from catalog cache for truncate table HMS api .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7350/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/17705 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I734c2b5f371291fef32badab9efc886b4b067e10 Gerrit-Change-Number: 17705 Gerrit-PatchSet: 4 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Mon, 26 Jul 2021 19:21:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10813: Invalidate external table from catalog cache for truncate table HMS api
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17705 ) Change subject: IMPALA-10813: Invalidate external table from catalog cache for truncate table HMS api .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/9176/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17705 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I734c2b5f371291fef32badab9efc886b4b067e10 Gerrit-Change-Number: 17705 Gerrit-PatchSet: 4 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Mon, 26 Jul 2021 19:09:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10813: Invalidate external table from catalog cache for truncate table HMS api
Hello Vihang Karajgaonkar, kis...@cloudera.com, Yu-Wen Lai, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17705 to look at the new patch set (#4). Change subject: IMPALA-10813: Invalidate external table from catalog cache for truncate table HMS api .. IMPALA-10813: Invalidate external table from catalog cache for truncate table HMS api This patch is in continuation of IMPALA-10648 in which we missed invalidating external table for truncate_table api Testing: Enhanced exiting test to include truncate_table scenario Change-Id: I734c2b5f371291fef32badab9efc886b4b067e10 --- M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java M tests/custom_cluster/test_metastore_service.py 2 files changed, 51 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/17705/4 -- To view, visit http://gerrit.cloudera.org:8080/17705 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I734c2b5f371291fef32badab9efc886b4b067e10 Gerrit-Change-Number: 17705 Gerrit-PatchSet: 4 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai
[Impala-ASF-CR] IMPALA-10801: Check the latest compaction Id before serving ACID table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17697 ) Change subject: IMPALA-10801: Check the latest compaction Id before serving ACID table .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/9175/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17697 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I86a112a77980fef7f6238978bc9668a65262101e Gerrit-Change-Number: 17697 Gerrit-PatchSet: 6 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Mon, 26 Jul 2021 18:05:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10801: Check the latest compaction Id before serving ACID table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17697 ) Change subject: IMPALA-10801: Check the latest compaction Id before serving ACID table .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/9174/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17697 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I86a112a77980fef7f6238978bc9668a65262101e Gerrit-Change-Number: 17697 Gerrit-PatchSet: 5 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Mon, 26 Jul 2021 18:03:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10801: Check the latest compaction Id before serving ACID table
Yu-Wen Lai has posted comments on this change. (
http://gerrit.cloudera.org:8080/17697 )
Change subject: IMPALA-10801: Check the latest compaction Id before serving
ACID table
..
Patch Set 6:
(11 comments)
http://gerrit.cloudera.org:8080/#/c/17697/4//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/17697/4//COMMIT_MSG@7
PS4, Line 7: ACID ta
> nit, May be change this to say "ACID table" to be more specific.
Done
http://gerrit.cloudera.org:8080/#/c/17697/4/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/17697/4/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2125
PS4, Line 2125: Preconditions.checkSta
> Can you add a Preconditions check before this line to make sure that the ta
Done
http://gerrit.cloudera.org:8080/#/c/17697/4/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2127
PS4, Line 2127: l.readLock().lock();
> nit, can we rename this variable to something like "partsToBeRefreshed" to
Done
http://gerrit.cloudera.org:8080/#/c/17697/4/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2187
PS4, Line 2187:
> change to "ACID tables" since external tables are also HdfsTables
Done
http://gerrit.cloudera.org:8080/#/c/17697/4/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@3486
PS4, Line 3486:
: if (!tryWriteLock(hdfsTable)) {
: throw new CatalogException(String.format(
: "Error during refreshing file metadata for table %s
due to lock contention",
: hdfsTable.getFullName()));
: }
: long newVersion = incrementAndGetCatalogVersion();
: v
> This logic seems to have a race condition. How do we know that the loadReq
Thanks for pointing out this. It is for optimization so I've removed it.
http://gerrit.cloudera.org:8080/#/c/17697/4/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/17697/4/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@824
PS4, Line 824: if (isPartitioned()) {
: for (CompactionInfoStruct ci : resp.getCompactions()) {
: HdfsPartition.Builder partBuilder =
nameToPartBuilder.get(ci.getPa
> If you move this to line 805 you can avoid iterating the partBuilders twice
Done
http://gerrit.cloudera.org:8080/#/c/17697/4/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@827
PS4, Line 827: Preconditions.checkNotNull(partBuilder);
: partBuilder.setLastCompactionId(ci.getId());
: }
: } else {
: CompactionInfoStruct ci =
Iterables.getOnlyElement(resp.getCompactions());
:
> I think the code readability can be improved if you handle the non-partitio
Done
http://gerrit.cloudera.org:8080/#/c/17697/4/fe/src/test/java/org/apache/impala/catalog/PartialCatalogInfoWriteIdTest.java
File
fe/src/test/java/org/apache/impala/catalog/PartialCatalogInfoWriteIdTest.java:
http://gerrit.cloudera.org:8080/#/c/17697/4/fe/src/test/java/org/apache/impala/catalog/PartialCatalogInfoWriteIdTest.java@520
PS4, Line 520: TGetPartialCatalogObjectResponse response =
> line too long (107 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/17697/4/fe/src/test/java/org/apache/impala/catalog/PartialCatalogInfoWriteIdTest.java@539
PS4, Line 539: response = sendRequest(request);
> line too long (114 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/17697/4/fe/src/test/java/org/apache/impala/catalog/PartialCatalogInfoWriteIdTest.java@569
PS4, Line 569:
Assert.assertTrue(prePartitionInfo.getFile_descriptors().size() > 1);
> line too long (110 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/17697/4/fe/src/test/java/org/apache/impala/catalog/PartialCatalogInfoWriteIdTest.java@583
PS4, Line 583: .wantFiles()
> line too long (92 > 90)
Done
--
To view, visit http://gerrit.cloudera.org:8080/17697
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I86a112a77980fef7f6238978bc9668a65262101e
Gerrit-Change-Number: 17697
Gerrit-PatchSet: 6
Gerrit-Owner: Yu-Wen Lai
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Sourabh Goyal
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Reviewer: Yu-Wen Lai
Gerrit-Comment-Date: Mon, 26 Jul 2021 17:50:13 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10801: Check the latest compaction Id before serving ACID table
Yu-Wen Lai has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/17697 ) Change subject: IMPALA-10801: Check the latest compaction Id before serving ACID table .. IMPALA-10801: Check the latest compaction Id before serving ACID table Since compactions don't advance write id, we don't know if a table/partition is compacted by comparing writeIdList. A possible issue is that CatalogD provides obsolete file metadata and causes a runtime error. In order to fix this issue, we introduced a HMS API that can get the latest compaction record for a table/partition (HIVE-24828). In CatalogD, we cache compaction id while loading partitions and compare the cached id with the latest compaction id before serving. If there is a newer compaction happened, it would refresh the file metadata. Besides, this patch also change how to replace the existing table after a table full reloading. The current way is to replace the table if the catalog version is not changed. For transactional tables, things get additional complexity given that file metadata refreshing and full table reloading can happen together. We can actually use writeIdList to determine whether we should replace the table for transactional tables. As long as the updated table has more recent writeIdList than the existing one, we are safe to replace the table. For Non-transactional tables, we still keep original behavior. Testing: - Add a test in PartialCatalogInfoWriteIdTest Change-Id: I86a112a77980fef7f6238978bc9668a65262101e --- M bin/impala-config.sh M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java M fe/src/main/java/org/apache/impala/util/AcidUtils.java M fe/src/test/java/org/apache/impala/catalog/PartialCatalogInfoWriteIdTest.java M testdata/bin/create-load-data.sh R testdata/cluster/ranger/setup/policy_5_revised.json 9 files changed, 367 insertions(+), 44 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/17697/6 -- To view, visit http://gerrit.cloudera.org:8080/17697 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I86a112a77980fef7f6238978bc9668a65262101e Gerrit-Change-Number: 17697 Gerrit-PatchSet: 6 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai
[Impala-ASF-CR] IMPALA-10801: Check the latest compaction Id before serving request
Yu-Wen Lai has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/17697 ) Change subject: IMPALA-10801: Check the latest compaction Id before serving request .. IMPALA-10801: Check the latest compaction Id before serving request Since compactions don't advance write id, we don't know if a table/partition is compacted by comparing writeIdList. A possible issue is that CatalogD provides obsolete file metadata and causes a runtime error. In order to fix this issue, we introduced a HMS API that can get the latest compaction record for a table/partition (HIVE-24828). In CatalogD, we cache compaction id while loading partitions and compare the cached id with the latest compaction id before serving. If there is a newer compaction happened, it would refresh the file metadata. Besides, this patch also change how to replace the existing table after a table full reloading. The current way is to replace the table if the catalog version is not changed. For transactional tables, things get additional complexity given that file metadata refreshing and full table reloading can happen together. We can actually use writeIdList to determine whether we should replace the table for transactional tables. As long as the updated table has more recent writeIdList than the existing one, we are safe to replace the table. For Non-transactional tables, we still keep original behavior. Testing: - Add a test in PartialCatalogInfoWriteIdTest Change-Id: I86a112a77980fef7f6238978bc9668a65262101e --- M bin/impala-config.sh M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java M fe/src/main/java/org/apache/impala/util/AcidUtils.java M fe/src/test/java/org/apache/impala/catalog/PartialCatalogInfoWriteIdTest.java M testdata/bin/create-load-data.sh R testdata/cluster/ranger/setup/policy_5_revised.json 9 files changed, 367 insertions(+), 44 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/17697/5 -- To view, visit http://gerrit.cloudera.org:8080/17697 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I86a112a77980fef7f6238978bc9668a65262101e Gerrit-Change-Number: 17697 Gerrit-PatchSet: 5 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai
[Impala-ASF-CR] [WIP] IMPALA-3430: Runtime filter : Extend runtime filter to support Min/Max values for HDFS scans
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17706 ) Change subject: [WIP] IMPALA-3430: Runtime filter : Extend runtime filter to support Min/Max values for HDFS scans .. Patch Set 5: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/9173/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/17706 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7c2bb5baad622051d1002c9c162c672d428e5446 Gerrit-Change-Number: 17706 Gerrit-PatchSet: 5 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Comment-Date: Mon, 26 Jul 2021 16:34:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10823: Output fewer information when external frontend is used
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/17718 ) Change subject: IMPALA-10823: Output fewer information when external frontend is used .. Patch Set 3: (2 comments) Looks good! http://gerrit.cloudera.org:8080/#/c/17718/3/be/src/util/thrift-debug-util.h File be/src/util/thrift-debug-util.h: http://gerrit.cloudera.org:8080/#/c/17718/3/be/src/util/thrift-debug-util.h@139 PS3, Line 139: co nit. Can this be made static so that there is only one copy of the pattern in each impalad process? http://gerrit.cloudera.org:8080/#/c/17718/1/be/src/util/thrift-debug-util.h File be/src/util/thrift-debug-util.h: http://gerrit.cloudera.org:8080/#/c/17718/1/be/src/util/thrift-debug-util.h@142 PS1, Line 142: " *** OTHER FIELDS ARE OMITTED ***\n" > I added nit. I wonder if this can be controlled by a query option instead. Using VLOG(3) will output everything at VLOG(3) level which may not be desirable. -- To view, visit http://gerrit.cloudera.org:8080/17718 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib862bfa298855943037afef53990160dcc8321ad Gerrit-Change-Number: 17718 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 26 Jul 2021 16:30:01 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 ) Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync .. Patch Set 13: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/17645 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c Gerrit-Change-Number: 17645 Gerrit-PatchSet: 13 Gerrit-Owner: liuyao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: liuyao Gerrit-Comment-Date: Mon, 26 Jul 2021 16:25:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10821 Fix TestTPCHJoinQueries.test outer joins failed
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17716 ) Change subject: IMPALA-10821 Fix TestTPCHJoinQueries.test_outer_joins failed .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/9172/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17716 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7e068d9da03517f8316e7a2505ce1466523d5917 Gerrit-Change-Number: 17716 Gerrit-PatchSet: 3 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Mon, 26 Jul 2021 16:25:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10821 Fix TestTPCHJoinQueries.test outer joins failed
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17716 ) Change subject: IMPALA-10821 Fix TestTPCHJoinQueries.test_outer_joins failed .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7349/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17716 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7e068d9da03517f8316e7a2505ce1466523d5917 Gerrit-Change-Number: 17716 Gerrit-PatchSet: 4 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Mon, 26 Jul 2021 16:14:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10821 Fix TestTPCHJoinQueries.test outer joins failed
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17716 ) Change subject: IMPALA-10821 Fix TestTPCHJoinQueries.test_outer_joins failed .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17716 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7e068d9da03517f8316e7a2505ce1466523d5917 Gerrit-Change-Number: 17716 Gerrit-PatchSet: 4 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Mon, 26 Jul 2021 16:14:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10429 Add Support for Spilling to HDFS Path Parsing
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17720 ) Change subject: IMPALA-10429 Add Support for Spilling to HDFS Path Parsing .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7348/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17720 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0882ed1e80b02724dd5cb3cdb1fa7b6c2debcbf4 Gerrit-Change-Number: 17720 Gerrit-PatchSet: 2 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 26 Jul 2021 16:13:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10821 Fix TestTPCHJoinQueries.test outer joins failed
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/17716 ) Change subject: IMPALA-10821 Fix TestTPCHJoinQueries.test_outer_joins failed .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17716 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7e068d9da03517f8316e7a2505ce1466523d5917 Gerrit-Change-Number: 17716 Gerrit-PatchSet: 3 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Mon, 26 Jul 2021 16:12:29 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-3430: Runtime filter : Extend runtime filter to support Min/Max values for HDFS scans
Qifan Chen has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/17706 ) Change subject: [WIP] IMPALA-3430: Runtime filter : Extend runtime filter to support Min/Max values for HDFS scans .. [WIP] IMPALA-3430: Runtime filter : Extend runtime filter to support Min/Max values for HDFS scans This patches enables min/max filtering for non-correlated subqueries that return one row. In this case, the filters are built from the results of the subqueries and the filtering target is the scan node to be qualified by one of the subqueries. Shown below is one such query that normally gets compiled into a nested loop join. The filtering limits the values from column store_sales.ss_sales_price within the range [-infinite, avg(ss_wholesale_cost)]. select count(*) from store_sales where ss_sales_price <= (select min(ss_wholesale_cost) from store_sales); Change-Id: I7c2bb5baad622051d1002c9c162c672d428e5446 --- M be/src/exec/nested-loop-join-builder.cc M be/src/exec/nested-loop-join-builder.h M be/src/exec/nested-loop-join-node.cc M be/src/runtime/coordinator.cc M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java M fe/src/main/java/org/apache/impala/analysis/Predicate.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java M fe/src/main/java/org/apache/impala/planner/AggregationNode.java M fe/src/main/java/org/apache/impala/planner/JoinNode.java M fe/src/main/java/org/apache/impala/planner/NestedLoopJoinNode.java M fe/src/main/java/org/apache/impala/planner/PlanNode.java M fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java 14 files changed, 315 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/17706/5 -- To view, visit http://gerrit.cloudera.org:8080/17706 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7c2bb5baad622051d1002c9c162c672d428e5446 Gerrit-Change-Number: 17706 Gerrit-PatchSet: 5 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen
[Impala-ASF-CR] IMPALA-10821 Fix TestTPCHJoinQueries.test outer joins failed
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/17716 ) Change subject: IMPALA-10821 Fix TestTPCHJoinQueries.test_outer_joins failed .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/17716/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/17716/2//COMMIT_MSG@15 PS2, Line 15: will be changed according to the file system the testcase is > Comment probably needs to be updated to reflect the latest fix. Done -- To view, visit http://gerrit.cloudera.org:8080/17716 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7e068d9da03517f8316e7a2505ce1466523d5917 Gerrit-Change-Number: 17716 Gerrit-PatchSet: 3 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Mon, 26 Jul 2021 16:08:36 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10821 Fix TestTPCHJoinQueries.test outer joins failed
Yida Wu has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/17716 ) Change subject: IMPALA-10821 Fix TestTPCHJoinQueries.test_outer_joins failed .. IMPALA-10821 Fix TestTPCHJoinQueries.test_outer_joins failed A new added testcase in TestTPCHJoinQueries.test_outer_joins can't pass the s3 build because the plan generated in s3 build is not the same as the default hdfs build for the scan node. In s3 build, it is "SCAN S3", while in hdfs build, it is "SCAN HDFS". The patch changed the testcase to use $FILESYSTEM_NAME, which will be changed according to the file system the testcase is using. Tests: Reran and passed the failed testcase in s3 build. Change-Id: I7e068d9da03517f8316e7a2505ce1466523d5917 --- M testdata/workloads/tpch/queries/tpch-outer-joins.test 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/16/17716/3 -- To view, visit http://gerrit.cloudera.org:8080/17716 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7e068d9da03517f8316e7a2505ce1466523d5917 Gerrit-Change-Number: 17716 Gerrit-PatchSet: 3 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu
[Impala-ASF-CR] IMPALA-10805: [DOCS] Document priority based scratch directory selection
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17700 ) Change subject: IMPALA-10805: [DOCS] Document priority based scratch directory selection .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/17700 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iec170fdefcde09d4ee99d06b0876a17eb0bde2f6 Gerrit-Change-Number: 17700 Gerrit-PatchSet: 4 Gerrit-Owner: Shajini Thayasingh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 26 Jul 2021 15:54:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10805: [DOCS] Document priority based scratch directory selection
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17700 ) Change subject: IMPALA-10805: [DOCS] Document priority based scratch directory selection .. IMPALA-10805: [DOCS] Document priority based scratch directory selection Made minor changes. Incorporated feedback received by providing more examples. Explained how to configure priorities for the scratch directories. Provided an example displaying priority based configuration. Change-Id: Iec170fdefcde09d4ee99d06b0876a17eb0bde2f6 Reviewed-on: http://gerrit.cloudera.org:8080/17700 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M docs/topics/impala_disk_space.xml 1 file changed, 39 insertions(+), 0 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/17700 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Iec170fdefcde09d4ee99d06b0876a17eb0bde2f6 Gerrit-Change-Number: 17700 Gerrit-PatchSet: 5 Gerrit-Owner: Shajini Thayasingh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 ) Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server. .. Patch Set 9: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7347/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/17576 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c Gerrit-Change-Number: 17576 Gerrit-PatchSet: 9 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 26 Jul 2021 15:48:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10817: Share metastoreHmsDDL lock b/w CatalogOpExecutor and Catalog metastore server
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17713 ) Change subject: IMPALA-10817: Share metastoreHmsDDL lock b/w CatalogOpExecutor and Catalog metastore server .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7345/ -- To view, visit http://gerrit.cloudera.org:8080/17713 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I60d4f3a49eb843fa8640cd21d623fd8dda770001 Gerrit-Change-Number: 17713 Gerrit-PatchSet: 2 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Mon, 26 Jul 2021 15:48:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10805: [DOCS] Document priority based scratch directory selection
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17700 ) Change subject: IMPALA-10805: [DOCS] Document priority based scratch directory selection .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-docs-submit/374/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17700 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iec170fdefcde09d4ee99d06b0876a17eb0bde2f6 Gerrit-Change-Number: 17700 Gerrit-PatchSet: 4 Gerrit-Owner: Shajini Thayasingh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 26 Jul 2021 15:47:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10805: [DOCS] Document priority based scratch directory selection
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17700 ) Change subject: IMPALA-10805: [DOCS] Document priority based scratch directory selection .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17700 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iec170fdefcde09d4ee99d06b0876a17eb0bde2f6 Gerrit-Change-Number: 17700 Gerrit-PatchSet: 4 Gerrit-Owner: Shajini Thayasingh Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 26 Jul 2021 15:47:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10821 Fix TestTPCHJoinQueries.test outer joins failed
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/17716 ) Change subject: IMPALA-10821 Fix TestTPCHJoinQueries.test_outer_joins failed .. Patch Set 2: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/17716/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/17716/2//COMMIT_MSG@15 PS2, Line 15: expression to skip the difference of HDFS and S3. Comment probably needs to be updated to reflect the latest fix. -- To view, visit http://gerrit.cloudera.org:8080/17716 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7e068d9da03517f8316e7a2505ce1466523d5917 Gerrit-Change-Number: 17716 Gerrit-PatchSet: 2 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Mon, 26 Jul 2021 15:40:26 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10806: Create single node plan slowdown when hundreds of inline views are joined
Qifan Chen has posted comments on this change. (
http://gerrit.cloudera.org:8080/17712 )
Change subject: IMPALA-10806: Create single node plan slowdown when hundreds of
inline views are joined
..
Patch Set 2:
(9 comments)
Looks good!
http://gerrit.cloudera.org:8080/#/c/17712/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/17712/2//COMMIT_MSG@9
PS2, Line 9: Create single node plan slowdown in the following form SQL
nit. "Creating a single node plan for the following SQL sometime can slowdown"
http://gerrit.cloudera.org:8080/#/c/17712/2//COMMIT_MSG@16
PS2, Line 16: The reasons for the slow generation of plans are
nit. "are as follows".
http://gerrit.cloudera.org:8080/#/c/17712/2//COMMIT_MSG@17
PS2, Line 17: 1. auxiliary predicates are added to GlobalState.conjuncts causing
nit. "Many auxiliary predicates"
http://gerrit.cloudera.org:8080/#/c/17712/2//COMMIT_MSG@19
PS2, Line 19: i
nit. In
http://gerrit.cloudera.org:8080/#/c/17712/2//COMMIT_MSG@29
PS2, Line 29: Testing:
May add some new tests to demonstrate the compilation time reduction.
http://gerrit.cloudera.org:8080/#/c/17712/2/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/17712/2/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@391
PS2, Line 391: conjunctsWithoutAuxExpr
nit. Based on how this map is populated, it may be better to rename the map as
conjunctsFromQuery.
http://gerrit.cloudera.org:8080/#/c/17712/2/fe/src/main/java/org/apache/impala/analysis/ExprSubstitutionMap.java
File fe/src/main/java/org/apache/impala/analysis/ExprSubstitutionMap.java:
http://gerrit.cloudera.org:8080/#/c/17712/2/fe/src/main/java/org/apache/impala/analysis/ExprSubstitutionMap.java@190
PS2, Line 190: remove
Do we need to handle element not exist exception?
http://gerrit.cloudera.org:8080/#/c/17712/2/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
File fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java:
http://gerrit.cloudera.org:8080/#/c/17712/2/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java@1218
PS2, Line 1218: (!analyzer.ge
nit. If all expressions on RHS are materialized, then this entire trimming
operation is a no-op and could be expensive. Is there a way to know the
trimming is beneficial in advance?
http://gerrit.cloudera.org:8080/#/c/17712/2/fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java@1209
PS2, Line 1209: if (outputSmap != null) {
: // Remove expressions from outputSmap that are not used
according to baseSmap,
: // in order to optimize the performance of
ExprSubstitutionMap#compose
: ExprSubstitutionMap baseSmap =
inlineViewRef.getBaseTblSmap();
: Preconditions.checkState(outputSmap.size() ==
baseSmap.size());
: for (int i = outputSmap.size() - 1; i >= 0; --i) {
: List slotIds = new ArrayList<>();
: baseSmap.getRhs().get(i).getIds(null, slotIds);
: for (SlotId id: slotIds) {
: if (!analyzer.getSlotDesc(id).isMaterialized()) {
: outputSmap.remove(i);
: break;
: }
: }
: }
: }
nit. Wonder if this block of code can be made a new method as
ExprSubstituteMap::trim(ExprSubstitutionMap baseTblSMap).
--
To view, visit http://gerrit.cloudera.org:8080/17712
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifb4011b6167a0e61438a73c4dba6f1cd0a4e8c6a
Gerrit-Change-Number: 17712
Gerrit-PatchSet: 2
Gerrit-Owner: Xianqing He
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Qifan Chen
Gerrit-Comment-Date: Mon, 26 Jul 2021 14:57:01 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 ) Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server. .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/9171/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17576 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c Gerrit-Change-Number: 17576 Gerrit-PatchSet: 9 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 26 Jul 2021 13:00:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17576 to look at the new patch set (#9). Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server. .. IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server. Testing: 1. Added new unit test Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java M tests/custom_cluster/test_metastore_service.py M tests/util/event_processor_utils.py 4 files changed, 171 insertions(+), 46 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/76/17576/9 -- To view, visit http://gerrit.cloudera.org:8080/17576 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c Gerrit-Change-Number: 17576 Gerrit-PatchSet: 9 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 ) Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server. .. Patch Set 8: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/9170/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/17576 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c Gerrit-Change-Number: 17576 Gerrit-PatchSet: 8 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 26 Jul 2021 12:34:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 ) Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server. .. Patch Set 7: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/9169/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/17576 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c Gerrit-Change-Number: 17576 Gerrit-PatchSet: 7 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 26 Jul 2021 12:25:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17576 to look at the new patch set (#8). Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server. .. IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server. Testing: 1. Added new unit test Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java M tests/custom_cluster/test_metastore_service.py M tests/util/event_processor_utils.py 4 files changed, 163 insertions(+), 46 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/76/17576/8 -- To view, visit http://gerrit.cloudera.org:8080/17576 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c Gerrit-Change-Number: 17576 Gerrit-PatchSet: 8 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17576 ) Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server. .. Patch Set 7: (2 comments) http://gerrit.cloudera.org:8080/#/c/17576/7/tests/custom_cluster/test_metastore_service.py File tests/custom_cluster/test_metastore_service.py: http://gerrit.cloudera.org:8080/#/c/17576/7/tests/custom_cluster/test_metastore_service.py@775 PS7, Line 775: flake8: W291 trailing whitespace http://gerrit.cloudera.org:8080/#/c/17576/7/tests/custom_cluster/test_metastore_service.py@775 PS7, Line 775: # table already exists line has trailing whitespace -- To view, visit http://gerrit.cloudera.org:8080/17576 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c Gerrit-Change-Number: 17576 Gerrit-PatchSet: 7 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 26 Jul 2021 12:15:55 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server.
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17576 to look at the new patch set (#7). Change subject: IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server. .. IMPALA-10746: Drop table/db from catalog cache when drop table/db HMS apis are accessed from catalog's metastore server. Testing: 1. Added new unit test Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java M tests/custom_cluster/test_metastore_service.py M tests/util/event_processor_utils.py 4 files changed, 163 insertions(+), 46 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/76/17576/7 -- To view, visit http://gerrit.cloudera.org:8080/17576 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic2e2ad2630e2028b8ad26a6272ee766b27e0935c Gerrit-Change-Number: 17576 Gerrit-PatchSet: 7 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 ) Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync .. Patch Set 13: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7346/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/17645 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c Gerrit-Change-Number: 17645 Gerrit-PatchSet: 13 Gerrit-Owner: liuyao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: liuyao Gerrit-Comment-Date: Mon, 26 Jul 2021 10:19:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10823: Output fewer information when external frontend is used
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17718 ) Change subject: IMPALA-10823: Output fewer information when external frontend is used .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/9168/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17718 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib862bfa298855943037afef53990160dcc8321ad Gerrit-Change-Number: 17718 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 26 Jul 2021 10:00:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10823: Output fewer information when external frontend is used
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17718 ) Change subject: IMPALA-10823: Output fewer information when external frontend is used .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/9167/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17718 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib862bfa298855943037afef53990160dcc8321ad Gerrit-Change-Number: 17718 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 26 Jul 2021 09:51:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10817: Share metastoreHmsDDL lock b/w CatalogOpExecutor and Catalog metastore server
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17713 ) Change subject: IMPALA-10817: Share metastoreHmsDDL lock b/w CatalogOpExecutor and Catalog metastore server .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/9166/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17713 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I60d4f3a49eb843fa8640cd21d623fd8dda770001 Gerrit-Change-Number: 17713 Gerrit-PatchSet: 2 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Mon, 26 Jul 2021 09:47:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10823: Output fewer information when external frontend is used
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17718 ) Change subject: IMPALA-10823: Output fewer information when external frontend is used .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/17718/3/be/src/service/impala-hs2-server.cc File be/src/service/impala-hs2-server.cc: http://gerrit.cloudera.org:8080/#/c/17718/3/be/src/service/impala-hs2-server.cc@560 PS3, Line 560: VLOG(3) << "ExecutePlannedStatement(): plan=" << ThriftDebugString(request.plan); Please make sure you don't output sensitive data with ThriftDebugString(). If so, use impala::RedactedDebugString() instead. -- To view, visit http://gerrit.cloudera.org:8080/17718 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib862bfa298855943037afef53990160dcc8321ad Gerrit-Change-Number: 17718 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 26 Jul 2021 09:38:31 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10823: Output fewer information when external frontend is used
Hello Kurt Deschler, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17718 to look at the new patch set (#3). Change subject: IMPALA-10823: Output fewer information when external frontend is used .. IMPALA-10823: Output fewer information when external frontend is used We use ThriftDebugString() to output a TExecutePlannedStatementReq. TExecutePlannedStatementReq can be quite large since it contains a Frontend.TExecRequest as well. We only need to output a redacted version of TExecutePlannedStatementReq.statementReq. At higher log level we output TExecutePlannedStatementReq.plan as well. Change-Id: Ib862bfa298855943037afef53990160dcc8321ad --- M be/src/service/impala-hs2-server.cc M be/src/util/thrift-debug-util.h 2 files changed, 15 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/17718/3 -- To view, visit http://gerrit.cloudera.org:8080/17718 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib862bfa298855943037afef53990160dcc8321ad Gerrit-Change-Number: 17718 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-10817: Share metastoreHmsDDL lock b/w CatalogOpExecutor and Catalog metastore server
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17713 ) Change subject: IMPALA-10817: Share metastoreHmsDDL lock b/w CatalogOpExecutor and Catalog metastore server .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7345/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/17713 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I60d4f3a49eb843fa8640cd21d623fd8dda770001 Gerrit-Change-Number: 17713 Gerrit-PatchSet: 2 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Mon, 26 Jul 2021 09:36:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10823: Output fewer information when external frontend is used
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/17718 ) Change subject: IMPALA-10823: Output fewer information when external frontend is used .. Patch Set 2: (1 comment) Thanks for the comment! http://gerrit.cloudera.org:8080/#/c/17718/1/be/src/util/thrift-debug-util.h File be/src/util/thrift-debug-util.h: http://gerrit.cloudera.org:8080/#/c/17718/1/be/src/util/thrift-debug-util.h@142 PS1, Line 142: " *** OTHER FIELDS ARE OMITTED ***\n" > I downloaded the patch and output looks good. However, probably better to p I added VLOG(3) << "ExecutePlannedStatement(): plan=" << ThriftDebugString(request.plan); to impala-hs2-server.cc. So we can still have all the information at a higher log level. -- To view, visit http://gerrit.cloudera.org:8080/17718 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib862bfa298855943037afef53990160dcc8321ad Gerrit-Change-Number: 17718 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 26 Jul 2021 09:31:36 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10823: Output fewer information when external frontend is used
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17718 ) Change subject: IMPALA-10823: Output fewer information when external frontend is used .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/17718/2/be/src/service/impala-hs2-server.cc File be/src/service/impala-hs2-server.cc: http://gerrit.cloudera.org:8080/#/c/17718/2/be/src/service/impala-hs2-server.cc@560 PS2, Line 560: VLOG(3) << "ExecutePlannedStatement(): plan=" << ThriftDebugString(request.plan); Please make sure you don't output sensitive data with ThriftDebugString(). If so, use impala::RedactedDebugString() instead. -- To view, visit http://gerrit.cloudera.org:8080/17718 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib862bfa298855943037afef53990160dcc8321ad Gerrit-Change-Number: 17718 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Comment-Date: Mon, 26 Jul 2021 09:29:57 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10823: Output fewer information when external frontend is used
Hello Kurt Deschler, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17718 to look at the new patch set (#2). Change subject: IMPALA-10823: Output fewer information when external frontend is used .. IMPALA-10823: Output fewer information when external frontend is used We use ThriftDebugString() to output a TExecutePlannedStatementReq. TExecutePlannedStatementReq can be quite large since it contains a Frontend.TExecRequest as well. We only need to output a redacted version of TExecutePlannedStatementReq.statementReq. At higher log level we output TExecutePlannedStatementReq.plan as well. Change-Id: Ib862bfa298855943037afef53990160dcc8321ad --- M be/src/service/impala-hs2-server.cc M be/src/util/thrift-debug-util.h 2 files changed, 15 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/17718/2 -- To view, visit http://gerrit.cloudera.org:8080/17718 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib862bfa298855943037afef53990160dcc8321ad Gerrit-Change-Number: 17718 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler
[Impala-ASF-CR] IMPALA-10817: Share metastoreHmsDDL lock b/w CatalogOpExecutor and Catalog metastore server
Sourabh Goyal has posted comments on this change. ( http://gerrit.cloudera.org:8080/17713 ) Change subject: IMPALA-10817: Share metastoreHmsDDL lock b/w CatalogOpExecutor and Catalog metastore server .. Patch Set 2: The build failure does not seem to be related to the patch > Patch Set 1: Verified-1 > > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7336/ -- To view, visit http://gerrit.cloudera.org:8080/17713 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I60d4f3a49eb843fa8640cd21d623fd8dda770001 Gerrit-Change-Number: 17713 Gerrit-PatchSet: 2 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Mon, 26 Jul 2021 09:26:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10817: Share metastoreHmsDDL lock b/w CatalogOpExecutor and Catalog metastore server
Hello Vihang Karajgaonkar, kis...@cloudera.com, Yu-Wen Lai, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17713 to look at the new patch set (#2). Change subject: IMPALA-10817: Share metastoreHmsDDL lock b/w CatalogOpExecutor and Catalog metastore server .. IMPALA-10817: Share metastoreHmsDDL lock b/w CatalogOpExecutor and Catalog metastore server Currently, when doing create/drop table/db from catalogD, catalogOpExecutor (via Impala Shell) and metastore server (via HS2) acquires lock on their own lock objects to prevent concurrent create/drop operations in HMS. But that does not prevent these concurrent operations across CatalogOpExecutor and Metastore server. For example currently a user can perform create/drop HMS operation from Impala shell and catalog metastore server concurrently which is not the desired behavior. This patch unifies the lock object which will be acquired by both catalogOpExecutor as well as metastore server. Testing: Relying on existing tests since it is a small refactoring. Change-Id: I60d4f3a49eb843fa8640cd21d623fd8dda770001 --- M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java 2 files changed, 202 insertions(+), 137 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/17713/2 -- To view, visit http://gerrit.cloudera.org:8080/17713 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I60d4f3a49eb843fa8640cd21d623fd8dda770001 Gerrit-Change-Number: 17713 Gerrit-PatchSet: 2 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai
[Impala-ASF-CR] IMPALA-10817: Share metastoreHmsDDL lock b/w CatalogOpExecutor and Catalog metastore server
Sourabh Goyal has posted comments on this change. (
http://gerrit.cloudera.org:8080/17713 )
Change subject: IMPALA-10817: Share metastoreHmsDDL lock b/w CatalogOpExecutor
and Catalog metastore server
..
Patch Set 1:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/17713/1/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
File
fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java:
http://gerrit.cloudera.org:8080/#/c/17713/1/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@359
PS1, Line 359: // Lock used to ensure that CREATE[DROP] TABLE[DATABASE]
operations performed in
: // catalog_ and the corresponding RPC to apply the change in
HMS are atomic.
: // This lock is shared b/w HMS operations performed in
CatalogOpExecutor
: // and Metastore server
> nit, can you move this to line 319 where the field is declared.
Ack
http://gerrit.cloudera.org:8080/#/c/17713/1/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@431
PS1, Line 431: metastoreHmsDdlLock_
> In my opinion it is more readable to do instead of creating a variable whic
Calling catalogOpExecutor's api is more readable. I will make the change.
http://gerrit.cloudera.org:8080/#/c/17713/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/17713/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@5643
PS1, Line 5643: } finally {
: getMetastoreDdlLock().unlock();
: }
> you can use the same try block as in line 5632.
Thanks for pointing it out. Instead I have wrapped inner try block around
catalog_.refreshFunctions so that we can release msClient back to the pool as
soon as possible.
--
To view, visit http://gerrit.cloudera.org:8080/17713
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I60d4f3a49eb843fa8640cd21d623fd8dda770001
Gerrit-Change-Number: 17713
Gerrit-PatchSet: 1
Gerrit-Owner: Sourabh Goyal
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Sourabh Goyal
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Reviewer: Yu-Wen Lai
Gerrit-Comment-Date: Mon, 26 Jul 2021 09:23:56 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10806: Create single node plan slowdown when hundreds of inline views are joined
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17712 ) Change subject: IMPALA-10806: Create single node plan slowdown when hundreds of inline views are joined .. Patch Set 2: Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7344/ -- To view, visit http://gerrit.cloudera.org:8080/17712 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifb4011b6167a0e61438a73c4dba6f1cd0a4e8c6a Gerrit-Change-Number: 17712 Gerrit-PatchSet: 2 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 26 Jul 2021 09:18:42 + Gerrit-HasComments: No
[Impala-ASF-CR] WiP: IMPALA-9495: Support struct in select list for ORC tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17638 ) Change subject: WiP: IMPALA-9495: Support struct in select list for ORC tables .. Patch Set 3: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/9165/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/17638 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0fbe56bdcd372b72e99c0195d87a818e7fa4bc3a Gerrit-Change-Number: 17638 Gerrit-PatchSet: 3 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 26 Jul 2021 08:06:08 + Gerrit-HasComments: No
[Impala-ASF-CR] WiP: IMPALA-9495: Support struct in select list for ORC tables
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/17638 )
Change subject: WiP: IMPALA-9495: Support struct in select list for ORC tables
..
Patch Set 3:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/17638/3/fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java
File fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java:
http://gerrit.cloudera.org:8080/#/c/17638/3/fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java@239
PS3, Line 239: if (parentStructSlot_ != null) toStrHelper.add("parentSlot",
parentStructSlot_.getId());
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/17638/3/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java:
http://gerrit.cloudera.org:8080/#/c/17638/3/fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java@1018
PS3, Line 1018: "Incompatible return types 'STRUCT' and
'STRUCT' of exprs " +
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/17638/3/tests/query_test/test_nested_types.py
File tests/query_test/test_nested_types.py:
http://gerrit.cloudera.org:8080/#/c/17638/3/tests/query_test/test_nested_types.py@115
PS3, Line 115: class TestNestedTypesInSelectList(ImpalaTestSuite):
flake8: E302 expected 2 blank lines, found 1
http://gerrit.cloudera.org:8080/#/c/17638/3/tests/query_test/test_nested_types.py@130
PS3, Line 130:
flake8: E203 whitespace before ':'
--
To view, visit http://gerrit.cloudera.org:8080/17638
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0fbe56bdcd372b72e99c0195d87a818e7fa4bc3a
Gerrit-Change-Number: 17638
Gerrit-PatchSet: 3
Gerrit-Owner: Gabor Kaszab
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Mon, 26 Jul 2021 07:55:16 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] WiP: IMPALA-9495: Support struct in select list for ORC tables
Gabor Kaszab has uploaded a new patch set (#3). (
http://gerrit.cloudera.org:8080/17638 )
Change subject: WiP: IMPALA-9495: Support struct in select list for ORC tables
..
WiP: IMPALA-9495: Support struct in select list for ORC tables
This patch implements the functionality to allow structs in the select
list. When displaying the value of a struct it is formatted into a JSON
value and returned as a string. An example of such a value:
SELECT struct_col FROM some_table;
'{"int_struct_member":12,"string_struct_member":"string value"}'
-- Changes related to tuple and slot descriptors:
When providing a struct in the select list there is going to be a
SlotDescriptor for the struct slot in the topmost TupleDescriptor.
Additionally, another TupleDesriptor is created to hold SlotDescriptors
for each of the struct's children. The struct SlotDescriptor points to the
newly introduced TupleDescriptor using 'itemTupleId'.
The offsets for the children of the struct is calculated from the beginning
of the topmost TupleDescriptor and not from the TupleDescriptor that
directly holds the struct's children. The null indicator bytes as well are
stored on the level of the topmost TupleDescriptor.
-- Changes related to scalar expressions:
A struct in the select list is translated into an expression tree where the
top of this tree is a SlotRef for the struct itself and its children in the
tree are SlotRefs for the members of the struct. When evaluating a struct
SlotRef after the null checks the evaluation is delegated to the children
SlotRefs.
-- Internal representation of a struct:
When scanning a struct the rowbatch will hold the values of the struct's
children as if they were queried one by one directly in the select list.
E.g. Taking the following table:
CREATE TABLE tbl (id int, s struct) STORED AS ORC
And running the following query:
SELECT id, s FROM tbl;
After scanning the row batch will hold the following values:
(note the biggest size comes first)
1: The pointer for the string in s.b
2: The length for the string in s.b
3: The int value for s.a
4: The int value of id
5: A single null byte for all the slots: id, s, s.a, s.b
When evaluating a struct as a SlotRef a newly introduced StructVal will be
used to refer to the actual values of a struct in the row batch. This
StructVal holds a vector of pointers where each pointer represents a member
of the struct. Following the above example the StructVal would keep two
pointers, one to point to an IntVal and one to point to a StringVal.
-- Restrictions:
- Codegen support is not included in this patch.
- Only ORC file format is supported by this patch.
- Only HS2 client supports returning structs. Beeswax support is not
implemented as it is going to be deprecated anyway. Currently we receive
an error when trying to query a struct through Beeswax.
Change-Id: I0fbe56bdcd372b72e99c0195d87a818e7fa4bc3a
---
M be/src/exec/hdfs-orc-scanner.cc
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/hdfs-scanner.cc
M be/src/exec/orc-column-readers.cc
M be/src/exec/orc-column-readers.h
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M be/src/exec/parquet/parquet-collection-column-reader.cc
M be/src/exprs/expr-value.h
M be/src/exprs/scalar-expr-evaluator.cc
M be/src/exprs/scalar-expr-evaluator.h
M be/src/exprs/scalar-expr.cc
M be/src/exprs/scalar-expr.h
M be/src/exprs/scalar-expr.inline.h
M be/src/exprs/slot-ref.cc
M be/src/exprs/slot-ref.h
M be/src/runtime/buffered-tuple-stream-test.cc
M be/src/runtime/buffered-tuple-stream.cc
M be/src/runtime/buffered-tuple-stream.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/runtime/raw-value.cc
M be/src/runtime/raw-value.h
M be/src/runtime/row-batch-serialize-test.cc
M be/src/runtime/sorter.cc
M be/src/runtime/tuple.cc
M be/src/runtime/tuple.h
M be/src/runtime/types.cc
M be/src/runtime/types.h
M be/src/service/hs2-util.cc
M be/src/service/impala-beeswax-server.cc
M be/src/service/query-result-set.cc
M be/src/udf/udf.cc
M be/src/udf/udf.h
M be/src/util/debug-util.cc
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
M fe/src/main/java/org/apache/impala/analysis/SlotDescriptor.java
M fe/src/main/java/org/apache/impala/analysis/SlotRef.java
M fe/src/main/java/org/apache/impala/analysis/SortInfo.java
M fe/src/main/java/org/apache/impala/analysis/Subquery.java
M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java
M fe/src/main/java/org/apache/impala/catalog/StructType.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java
M fe
[Impala-ASF-CR] Frontend changes to enable 'stored as JSONFILE' This change will allow usage of commands that do not require reading the Json File like: - Create Table stored as JSONFILE - Sh
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17727 ) Change subject: Frontend changes to enable 'stored as JSONFILE' This change will allow usage of commands that do not require reading the Json File like: - Create Table stored as JSONFILE - Show Create Table - Describe .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/9164/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17727 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5b8cb2f59df3af09902b49d3bdac16c19954b305 Gerrit-Change-Number: 17727 Gerrit-PatchSet: 2 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 26 Jul 2021 07:33:19 + Gerrit-HasComments: No
[Impala-ASF-CR] Frontend changes to enable 'stored as JSONFILE' This change will allow usage of commands that do not require reading the Json File like: - Create Table stored as JSONFILE - Sh
shikha.asran...@gmail.com has uploaded this change for review. ( http://gerrit.cloudera.org:8080/17727 Change subject: Frontend changes to enable 'stored as JSONFILE' This change will allow usage of commands that do not require reading the Json File like: - Create Table stored as JSONFILE - Show Create Table - Describe .. Frontend changes to enable 'stored as JSONFILE' This change will allow usage of commands that do not require reading the Json File like: - Create Table stored as JSONFILE - Show Create Table - Describe Changes: - Added JSON as FileFormat to thrift and HdfsFileFormat. - Allowing Sql keyword 'jsonfile' and mapping it to JSON format. - Adding JSON serDe. - JsonFiles have input format same as TextFile, so we need to use SerDe library in use to differentiate between the two formats. Overloaded the functions querying File Format based on input format to consider serDe library too. Change-Id: I5b8cb2f59df3af09902b49d3bdac16c19954b305 --- M common/thrift/CatalogObjects.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/catalog/HdfsFileFormat.java M fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java M fe/src/main/jflex/sql-scanner.flex 6 files changed, 40 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/17727/2 -- To view, visit http://gerrit.cloudera.org:8080/17727 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I5b8cb2f59df3af09902b49d3bdac16c19954b305 Gerrit-Change-Number: 17727 Gerrit-PatchSet: 2 Gerrit-Owner: Anonymous Coward
