[Impala-ASF-CR] IMPALA-10801: Check the latest compaction Id before serving request
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17697 ) Change subject: IMPALA-10801: Check the latest compaction Id before serving request .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7332/ -- To view, visit http://gerrit.cloudera.org:8080/17697 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I86a112a77980fef7f6238978bc9668a65262101e Gerrit-Change-Number: 17697 Gerrit-PatchSet: 3 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Thu, 22 Jul 2021 04:24:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8680: Docker-based tests fail to archive the minicluster component logs
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/15898 ) Change subject: IMPALA-8680: Docker-based tests fail to archive the minicluster component logs .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/15898/6/docker/entrypoint.sh File docker/entrypoint.sh: http://gerrit.cloudera.org:8080/#/c/15898/6/docker/entrypoint.sh@304 PS6, Line 304: copy_cluster_logs One thing I noticed when I ran this is that this code doesn't execute if any of the commands above fail. That must be because we use "set -e" when executing these functions. One way to fix that would be to follow the structure that test_suite() uses. In other words: ret = 0 if ! ./buildall.sh -noclean -format -testdata -notests; then ret = 1 echo "Build + dataload failed!" else echo "Build + dataload succeeded!" fi Then, below that point, use ret to decide what needs to do. If ret is 0, then do all the current steps. If ret is 1, skip building parquet-reader/impala-profile-tool and the hardlinking duplicate HDFS blocks, but otherwise do basically the same things. It looks like test_suite() uses a lot of "cmd || true" structure to avoid exiting, then it returns ret at the end. -- To view, visit http://gerrit.cloudera.org:8080/15898 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I23e25d42992cec47c593dc388bcf0bcef828c05e Gerrit-Change-Number: 15898 Gerrit-PatchSet: 6 Gerrit-Owner: Zoltan Garaguly Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Zoltan Garaguly Gerrit-Comment-Date: Thu, 22 Jul 2021 03:34:28 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10815: Ignore events on non-default hive catalogs
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17707 ) Change subject: IMPALA-10815: Ignore events on non-default hive catalogs .. Patch Set 5: Code-Review+2 LGTM. Carrying Zoltan's +1. Thanks for fixing this! -- To view, visit http://gerrit.cloudera.org:8080/17707 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ided463b2a98331d3d305bbe92fbf3a5d2e197acf Gerrit-Change-Number: 17707 Gerrit-PatchSet: 5 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 22 Jul 2021 01:44:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10815: Ignore events on non-default hive catalogs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17707 ) Change subject: IMPALA-10815: Ignore events on non-default hive catalogs .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/17707 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ided463b2a98331d3d305bbe92fbf3a5d2e197acf Gerrit-Change-Number: 17707 Gerrit-PatchSet: 5 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 22 Jul 2021 00:59:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/17645 )
Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out
of sync
..
Patch Set 7:
(5 comments)
The current solution looks good to me. Could you also add test cases for the
LocalCatalog mode? You can use anotation like this to enable it:
@CustomClusterTestSuite.with_args(
impalad_args="--use_local_catalog=true",
catalogd_args="--catalog_topic_mode=minimal")
http://gerrit.cloudera.org:8080/#/c/17645/7/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/17645/7/be/src/service/impala-server.cc@a2142
PS7, Line 2142:
Can we keep this? If we wait for the next statestore update, this won't cause
trouble now.
In my understanding, this is the only difference we can use to distinguish DDL
updates from statestore updates.
http://gerrit.cloudera.org:8080/#/c/17645/7/be/src/service/impala-server.cc@2144
PS7, Line 2144: while (cur_service_id ==
catalog_update_info_.catalog_service_id) {
Could you add a log before the while-loop? E.g.
Catalog service id mismatch. Current id: xxx. Id in response: yyy. Catalogd
may be restarted. Waiting for new catalog update from statestore.
http://gerrit.cloudera.org:8080/#/c/17645/7/be/src/service/impala-server.cc@2150
PS7, Line 2150: if (cur_service_id == catalog_service_id) {
I think this captures the case that catalogd restart again and we get another
catalog_service_id from statestore update. Could you add a warning log in the
else-clause? E.g.
Ignoring catalog update result of catalog service id: xxx. The previous
catalog service id is yyy. Current catalog service id is zzz. Catalogd may be
restarted more than once.
http://gerrit.cloudera.org:8080/#/c/17645/4/tests/custom_cluster/test_restart_services.py
File tests/custom_cluster/test_restart_services.py:
http://gerrit.cloudera.org:8080/#/c/17645/4/tests/custom_cluster/test_restart_services.py@189
PS4, Line 189: self.execute_query_expect_success(self.client, "select name
from join_aa")
: self.exec
> I prefer Solution 2, this kind of modification is smaller and safer
Yeah, agree on these.
http://gerrit.cloudera.org:8080/#/c/17645/7/tests/custom_cluster/test_restart_services.py
File tests/custom_cluster/test_restart_services.py:
http://gerrit.cloudera.org:8080/#/c/17645/7/tests/custom_cluster/test_restart_services.py@169
PS7, Line 169: def test_restart_catalogd(self):
This is already a good test case. Can we add another one to cover the case that
catalogd is restarted twice?
--
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 7
Gerrit-Owner: liuyao
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Thomas Tauber-Marshall
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Reviewer: liuyao
Gerrit-Comment-Date: Thu, 22 Jul 2021 00:54:20 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/17645 )
Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out
of sync
..
Patch Set 7:
> Patch Set 7:
>
> 16:50:03 ERROR: cannot verify jenkins.impala.io's certificate, issued by
> ‘CN=Go Daddy Secure Certificate Authority -
> G2,OU=http://certs.godaddy.com/repository/,O=GoDaddy.com\\,
> Inc.,L=Scottsdale,ST=Arizona,C=US’:
> 16:50:03 Issued certificate has expired.
> 16:50:03 To connect to jenkins.impala.io insecurely, use
> `--no-check-certificate'.
> 16:50:03 + ssh -p 29418 impala-public-jenk...@gerrit.cloudera.org gerrit
> review --verified -1 --message '"Build' failed:
> 'https://jenkins.impala.io/job/gerrit-verify-dryrun/7319/;' --project
> Impala-ASF 17645,7
> 16:50:05 + exit 1
> 16:50:05 Build step 'Execute shell' marked build as failure
These can be ignored. The actual failure is
02:28:16 parallel-all-tests #9056 started.
08:50:03 parallel-all-tests #9056 completed. Result was FAILURE
Then you can trace to the actual failed job:
https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/4470/
They are Iceberg related failures. Seems unrelated:
metadata.test_show_create_table.TestShowCreateTable.test_show_create_table[protocol:
beeswax | table_format: text/none]
query_test.test_iceberg.TestIcebergTable.test_catalogs[protocol: beeswax |
exec_option: {'batch_size': 0, 'num_nodes': 0,
'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
parquet/none]
--
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 7
Gerrit-Owner: liuyao
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Thomas Tauber-Marshall
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Reviewer: liuyao
Gerrit-Comment-Date: Wed, 21 Jul 2021 23:56:46 +
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10801: Check the latest compaction Id before serving request
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17697 ) Change subject: IMPALA-10801: Check the latest compaction Id before serving request .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/9138/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17697 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I86a112a77980fef7f6238978bc9668a65262101e Gerrit-Change-Number: 17697 Gerrit-PatchSet: 3 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Wed, 21 Jul 2021 22:34:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10801: Check the latest compaction Id before serving request
Yu-Wen Lai has posted comments on this change. ( http://gerrit.cloudera.org:8080/17697 ) Change subject: IMPALA-10801: Check the latest compaction Id before serving request .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/17697/2/fe/src/main/java/org/apache/impala/util/AcidUtils.java File fe/src/main/java/org/apache/impala/util/AcidUtils.java: http://gerrit.cloudera.org:8080/#/c/17697/2/fe/src/main/java/org/apache/impala/util/AcidUtils.java@811 PS2, Line 811: Map partNameToCompactionId = new HashMap<>(); > Thanks for the suggestion. A batch size of 1K makes sense to me. I will tes In my local, the execution time of this api are ~1 ms for 1K partitions, ~10 ms for 10K paritions and ~30 ms for 50K partitions. Although it might takes a bit longer in a production env, we can expect it still falls in the range of tens of ms and I suppose it is a tolerable latency. -- To view, visit http://gerrit.cloudera.org:8080/17697 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I86a112a77980fef7f6238978bc9668a65262101e Gerrit-Change-Number: 17697 Gerrit-PatchSet: 3 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Wed, 21 Jul 2021 22:18:40 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10801: Check the latest compaction Id before serving request
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17697 ) Change subject: IMPALA-10801: Check the latest compaction Id before serving request .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7332/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/17697 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I86a112a77980fef7f6238978bc9668a65262101e Gerrit-Change-Number: 17697 Gerrit-PatchSet: 3 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Wed, 21 Jul 2021 22:08:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10801: Check the latest compaction Id before serving request
Yu-Wen Lai has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/17697 ) Change subject: IMPALA-10801: Check the latest compaction Id before serving request .. IMPALA-10801: Check the latest compaction Id before serving request Since compactions don't advance write id, we don't know if a table/partition is compacted by comparing writeIdList. A possible issue is that CatalogD provides obsolete file metadata and causes a runtime error. In order to fix this issue, we introduced a HMS API that can get the latest compaction record for a table/partition (HIVE-24828). In CatalogD, we cache compaction id while loading partitions and compare the cached id with the latest compaction id before serving. If there is a newer compaction happened, it would refresh the file metadata. Besides, this patch also change how to replace the existing table after a table full reloading. The current way is to replace the table if the catalog version is not changed. For transactional tables, things get additional complexity given that file metadata refreshing and full table reloading can happen together. We can actually use writeIdList to determine whether we should replace the table for transactional tables. As long as the updated table has more recent writeIdList than the existing one, we are safe to replace the table. For Non-transactional tables, we still keep original behavior. Testing: - Add a test in PartialCatalogInfoWriteIdTest Change-Id: I86a112a77980fef7f6238978bc9668a65262101e --- M bin/impala-config.sh M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/TableLoadingMgr.java M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java M fe/src/main/java/org/apache/impala/util/AcidUtils.java M fe/src/test/java/org/apache/impala/catalog/PartialCatalogInfoWriteIdTest.java M testdata/bin/create-load-data.sh R testdata/cluster/ranger/setup/policy_5_revised.json 10 files changed, 370 insertions(+), 44 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/17697/3 -- To view, visit http://gerrit.cloudera.org:8080/17697 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I86a112a77980fef7f6238978bc9668a65262101e Gerrit-Change-Number: 17697 Gerrit-PatchSet: 3 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Yu-Wen Lai
[Impala-ASF-CR] IMPALA-8762: Track host level admission stats across all coordinators
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/17683 ) Change subject: IMPALA-8762: Track host level admission stats across all coordinators .. Patch Set 2: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/17683 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2946832e0a89b077d0f3bec755e4672be2088243 Gerrit-Change-Number: 17683 Gerrit-PatchSet: 2 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 21 Jul 2021 21:58:53 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-3430: Runtime filter : Extend runtime filter to support Min/Max values for HDFS scans
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17706 ) Change subject: [WIP] IMPALA-3430: Runtime filter : Extend runtime filter to support Min/Max values for HDFS scans .. Patch Set 3: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/9137/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/17706 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7c2bb5baad622051d1002c9c162c672d428e5446 Gerrit-Change-Number: 17706 Gerrit-PatchSet: 3 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Comment-Date: Wed, 21 Jul 2021 21:08:42 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-3430: Runtime filter : Extend runtime filter to support Min/Max values for HDFS scans
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/17706 )
Change subject: [WIP] IMPALA-3430: Runtime filter : Extend runtime filter to
support Min/Max values for HDFS scans
..
Patch Set 3:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/17706/3/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java
File fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java:
http://gerrit.cloudera.org:8080/#/c/17706/3/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java@741
PS3, Line 741: LOG.error("C0: " + root.getChild(0).debugString() + ",
label=" + root.getChild(0).getDisplayLabel());
line too long (107 > 90)
http://gerrit.cloudera.org:8080/#/c/17706/3/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java@742
PS3, Line 742: LOG.error("C1: " + root.getChild(1).debugString() + ",
label=" + root.getChild(1).getDisplayLabel());
line too long (107 > 90)
http://gerrit.cloudera.org:8080/#/c/17706/3/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java@743
PS3, Line 743: LOG.error("C1.0: " +
root.getChild(1).getChild(0).debugString() + ", label=" +
root.getChild(1).getChild(0).getDisplayLabel());
line too long (133 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/17706
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7c2bb5baad622051d1002c9c162c672d428e5446
Gerrit-Change-Number: 17706
Gerrit-PatchSet: 3
Gerrit-Owner: Qifan Chen
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Qifan Chen
Gerrit-Comment-Date: Wed, 21 Jul 2021 20:42:27 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP] IMPALA-3430: Runtime filter : Extend runtime filter to support Min/Max values for HDFS scans
Qifan Chen has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/17706 ) Change subject: [WIP] IMPALA-3430: Runtime filter : Extend runtime filter to support Min/Max values for HDFS scans .. [WIP] IMPALA-3430: Runtime filter : Extend runtime filter to support Min/Max values for HDFS scans This patches enables min/max filtering in which the filters are built from non-correlated subqueries that return one row and the filtering target is the scan node to be qualified by one of the subqueries. Shown below is one such query that normally gets compiled into a nested loop join. select count(*) from store_sales where ss_sales_price < (select avg(ss_wholesale_cost) from store_sales); Change-Id: I7c2bb5baad622051d1002c9c162c672d428e5446 --- M be/src/exec/nested-loop-join-builder.cc M be/src/exec/nested-loop-join-builder.h M be/src/exec/nested-loop-join-node.cc M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java M fe/src/main/java/org/apache/impala/analysis/Predicate.java M fe/src/main/java/org/apache/impala/planner/AggregationNode.java M fe/src/main/java/org/apache/impala/planner/JoinNode.java M fe/src/main/java/org/apache/impala/planner/PlanNode.java M fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java 9 files changed, 225 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/17706/3 -- To view, visit http://gerrit.cloudera.org:8080/17706 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7c2bb5baad622051d1002c9c162c672d428e5446 Gerrit-Change-Number: 17706 Gerrit-PatchSet: 3 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen
[Impala-ASF-CR] IMPALA-9338 Fix impala crashing in impala::RowDescriptor::TupleIsNullable(int)
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17610 ) Change subject: IMPALA-9338 Fix impala crashing in impala::RowDescriptor::TupleIsNullable(int) .. IMPALA-9338 Fix impala crashing in impala::RowDescriptor::TupleIsNullable(int) The patch fixes a bug in the function of orderConjunctsByCost, which could remove the wrong object in the list when the first conjunct is not the best and there is a same conjunct with different letter cases. It could end up to have duplicate objects after reordering the list because the conjunct, which has been added to the return list, is still in the remaining list, and lead to a wrong plan later where each side of the JOIN references columns from the other side due to a double flip on a same conjunct (There are two conjuncts in the list, and they are flipped as required by the analyzer, but unfortunately, the two conjuncts are the same object). The root cause of the issue is that some parts of the analyzer are case-sensitive, but some parts are not. For example, the remove() of the List considers the conjuncts with different letter cases are the same because they refer the same columns, while the compareTo() of the String considers the letter cases. This discrepancy creates some unexpected bugs. The fix uses the index instead of the Object to remove in the remaining list to solve the bug. However, there may still be somewhere else in our code that has similar issues regarding to different letter cases, it could be better that we have a consistent policy in SQL analyzing to avoid such bugs. Regression testcases has been added to queries/tpch-outer-joins and PlannerTest/join-order. Tests: Ran the Core FE_TEST and EE_TEST. Passed the regression test in tpch-outer-joins and PlannerTest/join-order. Change-Id: I2ba031d7a6eda21a77b0e53bc41772ee9e00a528 Reviewed-on: http://gerrit.cloudera.org:8080/17610 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/planner/PlanNode.java M testdata/workloads/functional-planner/queries/PlannerTest/join-order.test M testdata/workloads/tpch/queries/tpch-outer-joins.test 3 files changed, 80 insertions(+), 3 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/17610 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I2ba031d7a6eda21a77b0e53bc41772ee9e00a528 Gerrit-Change-Number: 17610 Gerrit-PatchSet: 7 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yida Wu
[Impala-ASF-CR] IMPALA-9338 Fix impala crashing in impala::RowDescriptor::TupleIsNullable(int)
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17610 ) Change subject: IMPALA-9338 Fix impala crashing in impala::RowDescriptor::TupleIsNullable(int) .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/17610 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2ba031d7a6eda21a77b0e53bc41772ee9e00a528 Gerrit-Change-Number: 17610 Gerrit-PatchSet: 6 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Wed, 21 Jul 2021 19:18:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10815: Ignore events on non-default hive catalogs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17707 ) Change subject: IMPALA-10815: Ignore events on non-default hive catalogs .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/9136/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17707 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ided463b2a98331d3d305bbe92fbf3a5d2e197acf Gerrit-Change-Number: 17707 Gerrit-PatchSet: 5 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 21 Jul 2021 19:08:09 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP]: Initial commit to acquire table/database lock in metastore server before any HMS operation
Yu-Wen Lai has posted comments on this change. (
http://gerrit.cloudera.org:8080/17703 )
Change subject: [WIP]: Initial commit to acquire table/database lock in
metastore server before any HMS operation
..
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/17703/3/fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java
File
fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java:
http://gerrit.cloudera.org:8080/#/c/17703/3/fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java@192
PS3, Line 192: LOG.debug("Successfully executed HMS API: " + apiName);
> Can you add one sample test case, where CatalogOpExecutor and MetastoreServ
Do we need to sync table/database to latest event in this class? If we don't
directly update cache here, is it possible to delay the sync up operation until
next read?
--
To view, visit http://gerrit.cloudera.org:8080/17703
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I085eab20db61282daf4549ddbcc018aaf63cc361
Gerrit-Change-Number: 17703
Gerrit-PatchSet: 4
Gerrit-Owner: Sourabh Goyal
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Sourabh Goyal
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Reviewer: Yu-Wen Lai
Gerrit-Comment-Date: Wed, 21 Jul 2021 18:59:51 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10815: Ignore events on non-default hive catalogs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17707 ) Change subject: IMPALA-10815: Ignore events on non-default hive catalogs .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7331/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/17707 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ided463b2a98331d3d305bbe92fbf3a5d2e197acf Gerrit-Change-Number: 17707 Gerrit-PatchSet: 5 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 21 Jul 2021 18:40:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10815: Ignore events on non-default hive catalogs
Vihang Karajgaonkar has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/17707 ) Change subject: IMPALA-10815: Ignore events on non-default hive catalogs .. IMPALA-10815: Ignore events on non-default hive catalogs Hive-3 supports a new type in metastore called catalogs. Even though impala does not support custom catalogs, it is still possible that some external HMS client creates objects within a non-default catalog. This can become problematic when the objects within the custom catalog match with the name of other objects in the default catalog. For example, dropping a custom catalog generates a DROP_DATABASE event on default database of that catalog. When such event is processed, the events processor can remove the default database. This patch adds logic to ignore all the events which are generated on such non-default catalog objects. The default value of catalog is defined in the hive-site.xml of the metastore client which is used by catalogd. If the value is not present it default to "hive". Additionally, it also adds the code to validate that the default catalog name defined in the hive-site.xml of the catalogd is same as in metastore server side. If the values do not match, the events processor does not come up. This patch also removes some config validations which are specific to hive-2 since we don't support hive-2 anymore in master. Testing: 1. Added a new test which creates a custom hive catalog and events on it. The test makes sure that such events do not affect objects in catalogd. Change-Id: Ided463b2a98331d3d305bbe92fbf3a5d2e197acf --- M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventProcessorConfig.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 5 files changed, 150 insertions(+), 89 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/17707/5 -- To view, visit http://gerrit.cloudera.org:8080/17707 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ided463b2a98331d3d305bbe92fbf3a5d2e197acf Gerrit-Change-Number: 17707 Gerrit-PatchSet: 5 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-10813: Invalidate external table from catalog cache for truncate table HMS api
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/17705 ) Change subject: IMPALA-10813: Invalidate external table from catalog cache for truncate table HMS api .. Patch Set 1: Code-Review+1 (2 comments) Patch looks good to me. Can you please fix the code style issues. http://gerrit.cloudera.org:8080/#/c/17705/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/17705/1//COMMIT_MSG@9 PS1, Line 9: This patch is in continuation of IMPALA-10648 in which we missed invalidating external : table for truncate_table api nit, can you please reformat the commit message to a line-width of 72 chars? Copying this from https://cwiki.apache.org/confluence/display/IMPALA/Contributing+to+Impala in case you are not familiar. "Finally, please write a good, clear commit message, with a short, descriptive title and a message that is exactly long enough to explain what the problem was, and how it was fixed. Each should have 72 or fewer characters if possible. The first line should have an empty line after it, and the first line should begin with the ticket(s) addressed, followed by a colon and a space:" http://gerrit.cloudera.org:8080/#/c/17705/1/tests/custom_cluster/test_metastore_service.py File tests/custom_cluster/test_metastore_service.py: http://gerrit.cloudera.org:8080/#/c/17705/1/tests/custom_cluster/test_metastore_service.py@24 PS1, Line 24: from hive_metastore.ttypes import GetValidWriteIdsRequest > flake8: F401 'hive_metastore.ttypes.GetValidWriteIdsRequest' imported but u Can you please remove this if not needed? -- To view, visit http://gerrit.cloudera.org:8080/17705 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I734c2b5f371291fef32badab9efc886b4b067e10 Gerrit-Change-Number: 17705 Gerrit-PatchSet: 1 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Wed, 21 Jul 2021 17:55:52 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP]: Initial commit to acquire table/database lock in metastore server before any HMS operation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17703 ) Change subject: [WIP]: Initial commit to acquire table/database lock in metastore server before any HMS operation .. Patch Set 4: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/9135/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/17703 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I085eab20db61282daf4549ddbcc018aaf63cc361 Gerrit-Change-Number: 17703 Gerrit-PatchSet: 4 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Wed, 21 Jul 2021 17:16:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10815: Ignore events on non-default hive catalogs
Zoltan Borok-Nagy has posted comments on this change. (
http://gerrit.cloudera.org:8080/17707 )
Change subject: IMPALA-10815: Ignore events on non-default hive catalogs
..
Patch Set 4: Code-Review+1
(2 comments)
LGTM!
http://gerrit.cloudera.org:8080/#/c/17707/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/17707/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@231
PS4, Line 231: && catalog_.isBlacklistedTable(eventDb, eventTbl))) {
Should we include a check for catalogName here as well? So we don't log this
message if the db/tbl belongs to a different catalog.
Or probably we should just do the catalog check first.
http://gerrit.cloudera.org:8080/#/c/17707/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
File
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java:
http://gerrit.cloudera.org:8080/#/c/17707/4/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@367
PS4, Line 367: if (MetastoreShim.getMajorVersion() >= 2) {
We dropped support for Hive 2, so it will always be true.
--
To view, visit http://gerrit.cloudera.org:8080/17707
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ided463b2a98331d3d305bbe92fbf3a5d2e197acf
Gerrit-Change-Number: 17707
Gerrit-PatchSet: 4
Gerrit-Owner: Vihang Karajgaonkar
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Wed, 21 Jul 2021 16:58:44 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP]: Initial commit to acquire table/database lock in metastore server before any HMS operation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17703 ) Change subject: [WIP]: Initial commit to acquire table/database lock in metastore server before any HMS operation .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/17703/4/fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java File fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java: http://gerrit.cloudera.org:8080/#/c/17703/4/fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java@248 PS4, Line 248: org.apache.impala.catalog.Table tbl = getTableAndAcquireWriteLock(partition.getDbName(), line too long (92 > 90) -- To view, visit http://gerrit.cloudera.org:8080/17703 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I085eab20db61282daf4549ddbcc018aaf63cc361 Gerrit-Change-Number: 17703 Gerrit-PatchSet: 4 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Wed, 21 Jul 2021 16:57:51 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP]: Initial commit to acquire table/database lock in metastore server before any HMS operation
Hello Vihang Karajgaonkar, kis...@cloudera.com, Yu-Wen Lai, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17703 to look at the new patch set (#4). Change subject: [WIP]: Initial commit to acquire table/database lock in metastore server before any HMS operation .. [WIP]: Initial commit to acquire table/database lock in metastore server before any HMS operation Change-Id: I085eab20db61282daf4549ddbcc018aaf63cc361 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java M fe/src/main/java/org/apache/impala/catalog/metastore/HmsApiNameEnum.java M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java 5 files changed, 371 insertions(+), 18 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/17703/4 -- To view, visit http://gerrit.cloudera.org:8080/17703 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I085eab20db61282daf4549ddbcc018aaf63cc361 Gerrit-Change-Number: 17703 Gerrit-PatchSet: 4 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai
[Impala-ASF-CR] [WIP]: Initial commit to acquire table/database lock in metastore server before any HMS operation
kis...@cloudera.com has posted comments on this change. (
http://gerrit.cloudera.org:8080/17703 )
Change subject: [WIP]: Initial commit to acquire table/database lock in
metastore server before any HMS operation
..
Patch Set 3:
(10 comments)
http://gerrit.cloudera.org:8080/#/c/17703/3/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/17703/3/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@444
PS3, Line 444: tableInfo.toString());
Can you wrap the lines from 435 to 444 within an if isDebugEnabled ?
http://gerrit.cloudera.org:8080/#/c/17703/3/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@457
PS3, Line 457: // except last
Why you are you releasing the write locks here ?
http://gerrit.cloudera.org:8080/#/c/17703/3/fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java
File
fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java:
http://gerrit.cloudera.org:8080/#/c/17703/3/fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java@186
PS3, Line 186: // long newCatalogVersion =
catalog_.incrementAndGetCatalogVersion();
Please remove the line, if you are not planning to use newCatalogVersion?
http://gerrit.cloudera.org:8080/#/c/17703/3/fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java@192
PS3, Line 192: catalogOpExecutor_.syncToLatestEventId(Db db, apiName);
Can you add one sample test case, where CatalogOpExecutor and
MetastoreServiceHanlder interact with the same table at the same time ?
I just want to see how it would look like.
http://gerrit.cloudera.org:8080/#/c/17703/3/fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java@219
PS3, Line 219: catalog_.getLock().writeLock().unlock();
Do we still need Catalog level lock, since we now have global table level lock ?
http://gerrit.cloudera.org:8080/#/c/17703/3/fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java@247
PS3, Line 247: org.apache.impala.catalog.Table tbl =
getTableAndAcquireWriteLock(partition.getDbName(),
> line too long (92 > 90)
Please take care of these.
http://gerrit.cloudera.org:8080/#/c/17703/3/fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java@396
PS3, Line 396: catalogOpExecutor_.UnlockWriteLockIfErronouslyLocked();
UnlockWriteLockIfErronouslyLocked what does this mean ?
http://gerrit.cloudera.org:8080/#/c/17703/3/fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java@411
PS3, Line 411: CatalogException e =
Can you not throw TException ?
How CatalogException is handled up in the chain ?
http://gerrit.cloudera.org:8080/#/c/17703/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
File
fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java:
http://gerrit.cloudera.org:8080/#/c/17703/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@1401
PS3, Line 1401: protected void throwException(Exception cause, String apiName)
May be rethrowException ?
http://gerrit.cloudera.org:8080/#/c/17703/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2871
PS3, Line 2871: private long getCurrentEventId(MetaStoreClient msClient)
throws TException {
Where are you using this ?
--
To view, visit http://gerrit.cloudera.org:8080/17703
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I085eab20db61282daf4549ddbcc018aaf63cc361
Gerrit-Change-Number: 17703
Gerrit-PatchSet: 3
Gerrit-Owner: Sourabh Goyal
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Sourabh Goyal
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Reviewer: Yu-Wen Lai
Gerrit-Comment-Date: Wed, 21 Jul 2021 16:54:57 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7635: Reducing HashTable size by packing it's buckets efficiently.
Qifan Chen has posted comments on this change. (
http://gerrit.cloudera.org:8080/17592 )
Change subject: IMPALA-7635: Reducing HashTable size by packing it's buckets
efficiently.
..
Patch Set 7:
(10 comments)
Looks pretty good.
My only concern is whether there is a significant performance degradation, and
how much.
http://gerrit.cloudera.org:8080/#/c/17592/7//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/17592/7//COMMIT_MSG@69
PS7, Line 69: As a part of this patch a new Micro Benchmark for HashTable has
: been introduced, which will help in measuring these:
: 1. Runtime for Building Hash Table and Probing the table.
: 2. Memory consumed after building the Table.
: This would help measuring the impact of changes to the HashTable's
: data structure and algorithm.
nit. Nice addition! May be useful to include some results here.
May also include some numbers from TPCDS here.
http://gerrit.cloudera.org:8080/#/c/17592/7/be/src/exec/hash-table.h
File be/src/exec/hash-table.h:
http://gerrit.cloudera.org:8080/#/c/17592/7/be/src/exec/hash-table.h@653
PS7, Line 653: 0
nit. May define and use constant for MATCHED here.
http://gerrit.cloudera.org:8080/#/c/17592/7/be/src/exec/hash-table.h@697
PS7, Line 697: 0
nit. same as above. May use constants for FILLED, MATCHED and DUPLICATED,
instead of 0, 1 and 2 directly.
http://gerrit.cloudera.org:8080/#/c/17592/7/be/src/util/tagged-ptr.h
File be/src/util/tagged-ptr.h:
http://gerrit.cloudera.org:8080/#/c/17592/7/be/src/util/tagged-ptr.h@33
PS7, Line 33: /// bits. Tag bit 0 - corresponds to bit 63, bit 1 corresponds to
62 and so on.
nit: may add: To get the address stored together with extra information in
canonical form, one must set/reset all bits from 57 to 63.
http://gerrit.cloudera.org:8080/#/c/17592/7/be/src/util/tagged-ptr.h@71
PS7, Line 71: if (i > 6 || i < 0)
nit. UNLIKELY()?
http://gerrit.cloudera.org:8080/#/c/17592/7/be/src/util/tagged-ptr.h@78
PS7, Line 78: if (i > 6 || i < 0) {
same as above
http://gerrit.cloudera.org:8080/#/c/17592/7/be/src/util/tagged-ptr.h@91
PS7, Line 91: //return (T*)((data_ & MASK_0_56_BITS) | ~((data_ & MASK_56_BIT)
- 1));
nit. delete?
http://gerrit.cloudera.org:8080/#/c/17592/7/be/src/util/tagged-ptr.h@101
PS7, Line 101: bool operator==(const TaggedPtr ) noexcept { return data_
== a.data_; }
:
: bool operator!=(const TaggedPtr ) noexcept { return data_
!= a.data_; }
nit. I found the implementation of these two operators counter intuitive as
semantically, a comparison of two pointers means to compare the pointers,
excluding any extra bits encoded within the pointers.
Maybe add two new methods
1. bool CompareTaggedPtrEqual()
2. bool compareTaggedPtrNotEqual()
http://gerrit.cloudera.org:8080/#/c/17592/7/be/src/util/tagged-ptr.h@115
PS7, Line 115: / data_ = (dat
nit. remove?
http://gerrit.cloudera.org:8080/#/c/17592/7/fe/src/main/java/org/apache/impala/planner/PlannerContext.java
File fe/src/main/java/org/apache/impala/planner/PlannerContext.java:
http://gerrit.cloudera.org:8080/#/c/17592/7/fe/src/main/java/org/apache/impala/planner/PlannerContext.java@47
PS7, Line 47: public final static double SIZE_OF_HASH = 4;
nit. I wonder if this value can be folded back into SIZE_OF_BUCKET and
SIZE_OF_DUPLICATENODE. That is to define directly
SIZE_OF_BUCKET=12
SIZE_OF_DUPLICATENODE=20
as the separation of hash from bucket is an implementation optimization.
--
To view, visit http://gerrit.cloudera.org:8080/17592
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I72912ae9353b0d567a976ca712d2d193e035df9b
Gerrit-Change-Number: 17592
Gerrit-PatchSet: 7
Gerrit-Owner: Amogh Margoor
Gerrit-Reviewer: Amogh Margoor
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Qifan Chen
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Wed, 21 Jul 2021 15:55:17 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10810: Bump json-smart from 2.3 to 2.4.7
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17702 ) Change subject: IMPALA-10810: Bump json-smart from 2.3 to 2.4.7 .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7329/ -- To view, visit http://gerrit.cloudera.org:8080/17702 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6b43f606f40e172aa267b55c564fa64d68515bd5 Gerrit-Change-Number: 17702 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Anonymous Coward (659) Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 21 Jul 2021 14:01:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9338 Fix impala crashing in impala::RowDescriptor::TupleIsNullable(int)
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17610 ) Change subject: IMPALA-9338 Fix impala crashing in impala::RowDescriptor::TupleIsNullable(int) .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7330/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17610 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2ba031d7a6eda21a77b0e53bc41772ee9e00a528 Gerrit-Change-Number: 17610 Gerrit-PatchSet: 6 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Wed, 21 Jul 2021 13:07:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9338 Fix impala crashing in impala::RowDescriptor::TupleIsNullable(int)
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17610 ) Change subject: IMPALA-9338 Fix impala crashing in impala::RowDescriptor::TupleIsNullable(int) .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17610 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2ba031d7a6eda21a77b0e53bc41772ee9e00a528 Gerrit-Change-Number: 17610 Gerrit-PatchSet: 6 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Wed, 21 Jul 2021 13:06:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9338 Fix impala crashing in impala::RowDescriptor::TupleIsNullable(int)
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/17610 ) Change subject: IMPALA-9338 Fix impala crashing in impala::RowDescriptor::TupleIsNullable(int) .. Patch Set 5: The verification run's failure is due to "ERROR: cannot verify jenkins.impala.io's certificate, issued by ‘CN=Go Daddy Secure Certificate Authority - G2,OU=http://certs.godaddy.com/repository/,O=GoDaddy.com\\, Inc.,L=Scottsdale,ST=Arizona,C=US’:". Perhaps it is transient. I will start a new one. -- To view, visit http://gerrit.cloudera.org:8080/17610 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2ba031d7a6eda21a77b0e53bc41772ee9e00a528 Gerrit-Change-Number: 17610 Gerrit-PatchSet: 5 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Wed, 21 Jul 2021 13:05:43 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP]: Initial commit to acquire table/database lock in metastore server before any HMS operation
Sourabh Goyal has posted comments on this change. ( http://gerrit.cloudera.org:8080/17703 ) Change subject: [WIP]: Initial commit to acquire table/database lock in metastore server before any HMS operation .. Patch Set 3: Requesting an early review on the approach. Sharing few details on the approach taken: All HMS APIs for which we need to sync to the latest envent id are overriden in CatalogMetastoreServiceHandler. If either enableCatalogdHMSCache or syncToLatestEventId flag is false, we fallback to pass through api in MetastoreServiceHandler. One api to look into is exchange_partition for which we need to take lock on both source and destination table atomically. Since there was no api in CatalogServiceCatalog for acquiring lock on multiple tables, I defined one in this patch. Please review it and share your feedback. Thanks! -- To view, visit http://gerrit.cloudera.org:8080/17703 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I085eab20db61282daf4549ddbcc018aaf63cc361 Gerrit-Change-Number: 17703 Gerrit-PatchSet: 3 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Wed, 21 Jul 2021 11:16:48 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP]: Initial commit to acquire table/database lock in metastore server before any HMS operation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17703 ) Change subject: [WIP]: Initial commit to acquire table/database lock in metastore server before any HMS operation .. Patch Set 3: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/9134/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/17703 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I085eab20db61282daf4549ddbcc018aaf63cc361 Gerrit-Change-Number: 17703 Gerrit-PatchSet: 3 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 21 Jul 2021 11:12:16 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP]: Initial commit to acquire table/database lock in metastore server before any HMS operation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17703 ) Change subject: [WIP]: Initial commit to acquire table/database lock in metastore server before any HMS operation .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/17703/3/fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java File fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java: http://gerrit.cloudera.org:8080/#/c/17703/3/fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java@247 PS3, Line 247: org.apache.impala.catalog.Table tbl = getTableAndAcquireWriteLock(partition.getDbName(), line too long (92 > 90) -- To view, visit http://gerrit.cloudera.org:8080/17703 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I085eab20db61282daf4549ddbcc018aaf63cc361 Gerrit-Change-Number: 17703 Gerrit-PatchSet: 3 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 21 Jul 2021 10:55:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP]: Initial commit to acquire table/database lock in metastore server before any HMS operation
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17703 to look at the new patch set (#3). Change subject: [WIP]: Initial commit to acquire table/database lock in metastore server before any HMS operation .. [WIP]: Initial commit to acquire table/database lock in metastore server before any HMS operation Change-Id: I085eab20db61282daf4549ddbcc018aaf63cc361 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java M fe/src/main/java/org/apache/impala/catalog/metastore/HmsApiNameEnum.java M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java 5 files changed, 370 insertions(+), 18 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/17703/3 -- To view, visit http://gerrit.cloudera.org:8080/17703 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I085eab20db61282daf4549ddbcc018aaf63cc361 Gerrit-Change-Number: 17703 Gerrit-PatchSet: 3 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
liuyao has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 ) Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync .. Patch Set 7: 16:50:03 ERROR: cannot verify jenkins.impala.io's certificate, issued by ‘CN=Go Daddy Secure Certificate Authority - G2,OU=http://certs.godaddy.com/repository/,O=GoDaddy.com\\, Inc.,L=Scottsdale,ST=Arizona,C=US’: 16:50:03 Issued certificate has expired. 16:50:03 To connect to jenkins.impala.io insecurely, use `--no-check-certificate'. 16:50:03 + ssh -p 29418 impala-public-jenk...@gerrit.cloudera.org gerrit review --verified -1 --message '"Build' failed: 'https://jenkins.impala.io/job/gerrit-verify-dryrun/7319/;' --project Impala-ASF 17645,7 16:50:05 + exit 1 16:50:05 Build step 'Execute shell' marked build as failure -- To view, visit http://gerrit.cloudera.org:8080/17645 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c Gerrit-Change-Number: 17645 Gerrit-PatchSet: 7 Gerrit-Owner: liuyao Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: liuyao Gerrit-Comment-Date: Wed, 21 Jul 2021 09:59:01 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP]: Initial commit to acquire table/database lock in metastore server before any HMS operation
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17703 to look at the new patch set (#2). Change subject: [WIP]: Initial commit to acquire table/database lock in metastore server before any HMS operation .. [WIP]: Initial commit to acquire table/database lock in metastore server before any HMS operation Change-Id: I085eab20db61282daf4549ddbcc018aaf63cc361 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/metastore/CatalogMetastoreServiceHandler.java M fe/src/main/java/org/apache/impala/catalog/metastore/HmsApiNameEnum.java M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java 5 files changed, 333 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/17703/2 -- To view, visit http://gerrit.cloudera.org:8080/17703 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I085eab20db61282daf4549ddbcc018aaf63cc361 Gerrit-Change-Number: 17703 Gerrit-PatchSet: 2 Gerrit-Owner: Sourabh Goyal Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-10810: Bump json-smart from 2.3 to 2.4.7
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17702 ) Change subject: IMPALA-10810: Bump json-smart from 2.3 to 2.4.7 .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17702 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6b43f606f40e172aa267b55c564fa64d68515bd5 Gerrit-Change-Number: 17702 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Anonymous Coward (659) Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 21 Jul 2021 07:57:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10810: Bump json-smart from 2.3 to 2.4.7
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17702 ) Change subject: IMPALA-10810: Bump json-smart from 2.3 to 2.4.7 .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7329/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17702 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6b43f606f40e172aa267b55c564fa64d68515bd5 Gerrit-Change-Number: 17702 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Anonymous Coward (659) Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 21 Jul 2021 07:57:16 + Gerrit-HasComments: No
