[Impala-ASF-CR] IMPALA-8504: Support CREATE TABLE statement with Kudu/HMS integration
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13318 ) Change subject: IMPALA-8504: Support CREATE TABLE statement with Kudu/HMS integration .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3435/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13318 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I465673d749221bd5f3772814b1c22c2673a53f5c Gerrit-Change-Number: 13318 Gerrit-PatchSet: 10 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Thu, 30 May 2019 07:11:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8505: Disallow setting Kudu table id in ALTER TABLE
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13465 ) Change subject: IMPALA-8505: Disallow setting Kudu table id in ALTER TABLE .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3436/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13465 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibc1f3f87054db5e28f72a5b54ea5b3c040c1bf22 Gerrit-Change-Number: 13465 Gerrit-PatchSet: 1 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Thu, 30 May 2019 07:17:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8600 : Reload partition does not work for transactional tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13467 ) Change subject: IMPALA-8600 : Reload partition does not work for transactional tables .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3437/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13467 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8fffad27cd54f722efba1758ff44a45468f9514a Gerrit-Change-Number: 13467 Gerrit-PatchSet: 1 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 30 May 2019 07:36:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8538: HS2 + HTTP(S) + BASIC/LDAP based thrift server endpoint
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13299 ) Change subject: IMPALA-8538: HS2 + HTTP(S) + BASIC/LDAP based thrift server endpoint .. IMPALA-8538: HS2 + HTTP(S) + BASIC/LDAP based thrift server endpoint This patch adds an additional hiveserver2 endpoint for clients to connect to that uses HTTP. The endpoint can be disabled by setting --hs2_http_port=0. HTTP(S) also works when external TLS is enabled using --ssl_server_certificate. Thrift's http transport is modified to support BASIC authentication via ldap. For convenience of developing and reviewing, this patch is based on another that copied THttpServer and THttpTransport into Impala's codebase. Kerberos authentication is not supported, so the http endpoint is turned off if Kerberos is enabled and LDAP isn't. TODO = - Fuzz test the http endpoint - Add tests for LDAP + HTTPS Testing === - Parameterized JdbcTest and LdapJdbcTest to work for HS2 + HTTP mode - Added LdapHS2Test, which directly calls into the Hiveserver2 interface using a thrift http client. Manual testing with Beeline client (from Apache Hive), which has builtin support to connect to HTTP(S) based HS2 compatible endpoints. Example -- HTTP mode: > start-impala-cluster.py > JDBC_URL="jdbc:hive2://localhost:/default;transportMode=http" > beeline -u "$JDBC_URL" -- HTTPS mode: > cd $IMPALA_HOME > SSL_ARGS="--ssl_client_ca_certificate=./be/src/testutil/server-cert.pem \ --ssl_server_certificate=./be/src/testutil/server-cert.pem \ --ssl_private_key=./be/src/testutil/server-key.pem --hostname=localhost" > start-impala-cluster.py --impalad_args="$SSL_ARGS" \ --catalogd_args="$SSL_ARGS" --state_store_args="$SSL_ARGS" - Create a local trust store using 'keytool' and import the certificate from server-cert.pem (./clientkeystore in the example). > JDBC_URL="jdbc:hive2://localhost:/default;ssl=true;sslTrustStore= \ ./clientkeystore;trustStorePassword=password;transportMode=http" > beeline -u "$JDBC_URL" -- BASIC Auth with LDAP: > LDAP_ARGS="--enable_ldap_auth --ldap_uri='ldap://...' \ --ldap_bind_pattern='...' --ldap_passwords_in_clear_ok" > start-impala-cluster.py --impalad_args="$LDAP_ARGS" > JDBC_URL="jdbc:hive2://localhost:28000/default;user=...;password=\ ...;transportMode=http" > beeline -u "$JDBC_URL" -- HTTPS mode with LDAP: > start-impala-cluster.py --impalad_args="$LDAP_ARGS $SSL_ARGS" \ --catalogd_args="$SSL_ARGS" --state_store_args="$SSL_ARGS" > JDBC_URL="jdbc:hive2://localhost:28000/default;user=...;password=\ ...;ssl=true;sslTrustStore=./clientkeystore;trustStorePassword=\ password;transportMode=http" > beeline -u "$JDBC_URL" Change-Id: Ic5569ac62ef3af2868b5d0581f5029dac736b2ff Reviewed-on: http://gerrit.cloudera.org:8080/13299 Reviewed-by: Thomas Marshall Tested-by: Impala Public Jenkins --- M be/src/rpc/auth-provider.h M be/src/rpc/authentication-test.cc M be/src/rpc/authentication.cc M be/src/rpc/thrift-server.cc M be/src/rpc/thrift-server.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/service/impalad-main.cc M be/src/testutil/in-process-servers.cc M be/src/testutil/in-process-servers.h M be/src/transport/THttpServer.cpp M be/src/transport/THttpServer.h M be/src/transport/THttpTransport.cpp M be/src/transport/THttpTransport.h M bin/start-impala-cluster.py M common/thrift/generate_error_codes.py M common/thrift/metrics.json A fe/src/test/java/org/apache/impala/customcluster/LdapHS2Test.java M fe/src/test/java/org/apache/impala/customcluster/LdapJdbcTest.java M fe/src/test/java/org/apache/impala/service/JdbcTest.java M fe/src/test/java/org/apache/impala/service/JdbcTestBase.java M fe/src/test/java/org/apache/impala/testutil/ImpalaJdbcClient.java M fe/src/test/resources/users.ldif M tests/common/impala_cluster.py 24 files changed, 683 insertions(+), 147 deletions(-) Approvals: Thomas Marshall: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/13299 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic5569ac62ef3af2868b5d0581f5029dac736b2ff Gerrit-Change-Number: 13299 Gerrit-PatchSet: 7 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Mike Yoder Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8538: HS2 + HTTP(S) + BASIC/LDAP based thrift server endpoint
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13299 ) Change subject: IMPALA-8538: HS2 + HTTP(S) + BASIC/LDAP based thrift server endpoint .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13299 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic5569ac62ef3af2868b5d0581f5029dac736b2ff Gerrit-Change-Number: 13299 Gerrit-PatchSet: 6 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Mike Yoder Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 30 May 2019 08:38:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8595: Support TLSv1.2 with Python < 2.7.9 in shell
Robbie Zhang has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/13457 ) Change subject: IMPALA-8595: Support TLSv1.2 with Python < 2.7.9 in shell .. IMPALA-8595: Support TLSv1.2 with Python < 2.7.9 in shell IMPALA-5690 replaced thrift 0.9.0 with 0.9.3 in which THRIFT-3505 changed transport/TSSLSocket.py. In thrift 0.9.3, if the python version is lower than 2.7.9, TSSLSocket uses PROTOCOL_TLSv1 by default and the SSL version should be passed as an argument to TSSLSocket._init_ instead of overriding self.SSL_VERSION in TSSLSocketWithWildcardSAN._init_. Although TLSv1.2 is supported by Python from 2.7.9, Red Hat/CentOS support TLSv1.2 from 2.7.5 with upgraded python-libs. We need to get impala-shell support TLSv1.2 with Python 2.7.5 on Red Hat/CentOS. TESTING: impala-py.test tests/custom_cluster/test_client_ssl.py Change-Id: I3fb6510f4b556bd8c6b1e86380379aba8be4b805 --- M shell/TSSLSocketWithWildcardSAN.py M tests/custom_cluster/test_client_ssl.py 2 files changed, 12 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/13457/4 -- To view, visit http://gerrit.cloudera.org:8080/13457 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3fb6510f4b556bd8c6b1e86380379aba8be4b805 Gerrit-Change-Number: 13457 Gerrit-PatchSet: 4 Gerrit-Owner: Robbie Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8595: Support TLSv1.2 with Python < 2.7.9 in shell
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13457 ) Change subject: IMPALA-8595: Support TLSv1.2 with Python < 2.7.9 in shell .. Patch Set 4: (4 comments) http://gerrit.cloudera.org:8080/#/c/13457/4/tests/custom_cluster/test_client_ssl.py File tests/custom_cluster/test_client_ssl.py: http://gerrit.cloudera.org:8080/#/c/13457/4/tests/custom_cluster/test_client_ssl.py@39 PS4, Line 39: , flake8: E231 missing whitespace after ',' http://gerrit.cloudera.org:8080/#/c/13457/4/tests/custom_cluster/test_client_ssl.py@39 PS4, Line 39: , flake8: E231 missing whitespace after ',' http://gerrit.cloudera.org:8080/#/c/13457/4/tests/custom_cluster/test_client_ssl.py@41 PS4, Line 41: , flake8: E231 missing whitespace after ',' http://gerrit.cloudera.org:8080/#/c/13457/4/tests/custom_cluster/test_client_ssl.py@41 PS4, Line 41: , flake8: E231 missing whitespace after ',' -- To view, visit http://gerrit.cloudera.org:8080/13457 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3fb6510f4b556bd8c6b1e86380379aba8be4b805 Gerrit-Change-Number: 13457 Gerrit-PatchSet: 4 Gerrit-Owner: Robbie Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 30 May 2019 12:10:44 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8595: Support TLSv1.2 with Python < 2.7.9 in shell
Hello Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13457 to look at the new patch set (#5). Change subject: IMPALA-8595: Support TLSv1.2 with Python < 2.7.9 in shell .. IMPALA-8595: Support TLSv1.2 with Python < 2.7.9 in shell IMPALA-5690 replaced thrift 0.9.0 with 0.9.3 in which THRIFT-3505 changed transport/TSSLSocket.py. In thrift 0.9.3, if the python version is lower than 2.7.9, TSSLSocket uses PROTOCOL_TLSv1 by default and the SSL version should be passed as an argument to TSSLSocket._init_ instead of overriding self.SSL_VERSION in TSSLSocketWithWildcardSAN._init_. Although TLSv1.2 is supported by Python from 2.7.9, Red Hat/CentOS support TLSv1.2 from 2.7.5 with upgraded python-libs. We need to get impala-shell support TLSv1.2 with Python 2.7.5 on Red Hat/CentOS. TESTING: impala-py.test tests/custom_cluster/test_client_ssl.py Change-Id: I3fb6510f4b556bd8c6b1e86380379aba8be4b805 --- M shell/TSSLSocketWithWildcardSAN.py M tests/custom_cluster/test_client_ssl.py 2 files changed, 12 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/13457/5 -- To view, visit http://gerrit.cloudera.org:8080/13457 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3fb6510f4b556bd8c6b1e86380379aba8be4b805 Gerrit-Change-Number: 13457 Gerrit-PatchSet: 5 Gerrit-Owner: Robbie Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8595: Support TLSv1.2 with Python < 2.7.9 in shell
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13457 ) Change subject: IMPALA-8595: Support TLSv1.2 with Python < 2.7.9 in shell .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3438/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13457 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3fb6510f4b556bd8c6b1e86380379aba8be4b805 Gerrit-Change-Number: 13457 Gerrit-PatchSet: 4 Gerrit-Owner: Robbie Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 30 May 2019 12:48:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8595: Support TLSv1.2 with Python < 2.7.9 in shell
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13457 ) Change subject: IMPALA-8595: Support TLSv1.2 with Python < 2.7.9 in shell .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3439/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13457 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3fb6510f4b556bd8c6b1e86380379aba8be4b805 Gerrit-Change-Number: 13457 Gerrit-PatchSet: 5 Gerrit-Owner: Robbie Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 30 May 2019 12:50:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8595: Support TLSv1.2 with Python < 2.7.9 in shell
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13457 ) Change subject: IMPALA-8595: Support TLSv1.2 with Python < 2.7.9 in shell .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4347/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13457 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3fb6510f4b556bd8c6b1e86380379aba8be4b805 Gerrit-Change-Number: 13457 Gerrit-PatchSet: 5 Gerrit-Owner: Robbie Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 30 May 2019 13:00:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7369: part 1: Implement TRUNC, DATE TRUNC, EXTRACT, DATE PART functions for DATE
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/13363 ) Change subject: IMPALA-7369: part 1: Implement TRUNC, DATE_TRUNC, EXTRACT, DATE_PART functions for DATE .. Patch Set 5: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/13363/5/be/src/runtime/date-value.cc File be/src/runtime/date-value.cc: http://gerrit.cloudera.org:8080/#/c/13363/5/be/src/runtime/date-value.cc@120 PS5, Line 120: // The goal of this logic could be also added, e.g. that it will be fast for most days of the year. -- To view, visit http://gerrit.cloudera.org:8080/13363 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I843358a45eb5faa2c134994600546fc1d0a797c8 Gerrit-Change-Number: 13363 Gerrit-PatchSet: 5 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 30 May 2019 13:07:50 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8588: Fix revoke grant option with Ranger
Hello Fredy Wijaya, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13450 to look at the new patch set (#2). Change subject: IMPALA-8588: Fix revoke grant option with Ranger .. IMPALA-8588: Fix revoke grant option with Ranger Previously, the REVOKE GRANT OPTION type statements would not only revoke the grant option, but also the privilege as well. The behavior has been updated to only revoke the grant option. In Ranger UI, this is seen as the delegate admin option. Examples: REVOKE SELECT ON DATABASE FROM USER This will revoke the SELECT privilege on the database resource, but if there are other privileges on that resource the grant option will remain for those privileges. REVOKE GRANT OPTION FOR SELECT ON DATABASE FROM USER This will revoke the grant option for all privileges on this database resource. It will not revoke the SELECT privilege on the resource. Testing: - Ran all FE tests - Ran all E2E tests - Updated test_ranger to test behavior for when REVOKE GRANT OPTION statements are submitted. Change-Id: Iddfccb442c3be3c266dbc2d8ae85c5674c534d7c --- M fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java M tests/authorization/test_ranger.py 3 files changed, 47 insertions(+), 26 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/13450/2 -- To view, visit http://gerrit.cloudera.org:8080/13450 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iddfccb442c3be3c266dbc2d8ae85c5674c534d7c Gerrit-Change-Number: 13450 Gerrit-PatchSet: 2 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8588: Fix revoke grant option with Ranger
Austin Nobis has posted comments on this change. ( http://gerrit.cloudera.org:8080/13450 ) Change subject: IMPALA-8588: Fix revoke grant option with Ranger .. Patch Set 2: (1 comment) > Patch Set 1: > > (1 comment) http://gerrit.cloudera.org:8080/#/c/13450/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13450/1//COMMIT_MSG@12 PS1, Line 12: seen as the delegate admin option. Examples: > It will be clearer to provide some examples here: Done -- To view, visit http://gerrit.cloudera.org:8080/13450 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddfccb442c3be3c266dbc2d8ae85c5674c534d7c Gerrit-Change-Number: 13450 Gerrit-PatchSet: 2 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 30 May 2019 14:29:44 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8369: Add HIVE MAJOR VERSION section to EE tests + some fixes
Csaba Ringhofer has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13472 Change subject: IMPALA-8369: Add HIVE_MAJOR_VERSION section to EE tests + some fixes .. IMPALA-8369: Add HIVE_MAJOR_VERSION section to EE tests + some fixes Fixed tests: test_scanners.py - test_scan_truncated_file_empty (exhaustive): Added REFRESH after Hive INSERT OVERWRITE. The test worked in Hive2 only because there was an empty file with the same name as before the overwrite. test_ddl.py - test_alter_table: A Hive3 regression broke some tests + caused the dropping of the test database to hang. These tests are skipped for now, see HIVE-21806. Change-Id: I4c3cff05ed7080b655b6af64ea09c0691e7dd931 --- M testdata/workloads/functional-query/queries/QueryTest/alter-table.test M tests/common/impala_test_suite.py M tests/query_test/test_scanners.py M tests/util/test_file_parser.py 4 files changed, 29 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/13472/1 -- To view, visit http://gerrit.cloudera.org:8080/13472 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I4c3cff05ed7080b655b6af64ea09c0691e7dd931 Gerrit-Change-Number: 13472 Gerrit-PatchSet: 1 Gerrit-Owner: Csaba Ringhofer
[Impala-ASF-CR] IMPALA-8369: Add HIVE MAJOR VERSION section to EE tests + some fixes
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13472 to look at the new patch set (#2). Change subject: IMPALA-8369: Add HIVE_MAJOR_VERSION section to EE tests + some fixes .. IMPALA-8369: Add HIVE_MAJOR_VERSION section to EE tests + some fixes Fixed tests with Hive3: test_scanners.py - test_scan_truncated_file_empty (exhaustive): Added REFRESH after Hive INSERT OVERWRITE. The test worked in Hive2 only because there was an empty file with the same name as before the overwrite. test_ddl.py - test_alter_table: A Hive3 regression broke some tests + caused the dropping of the test database to hang. These tests are skipped for now, the Hive side fix is tracked in HIVE-21806. Change-Id: I4c3cff05ed7080b655b6af64ea09c0691e7dd931 --- M testdata/workloads/functional-query/queries/QueryTest/alter-table.test M tests/common/impala_test_suite.py M tests/query_test/test_scanners.py M tests/util/test_file_parser.py 4 files changed, 29 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/13472/2 -- To view, visit http://gerrit.cloudera.org:8080/13472 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4c3cff05ed7080b655b6af64ea09c0691e7dd931 Gerrit-Change-Number: 13472 Gerrit-PatchSet: 2 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR](2.x) [DOCS] Fix UPDATE/UPSERT/DELETE authorization doc
Hello Fredy Wijaya, Impala Public Jenkins, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/13474 to review the following change. Change subject: [DOCS] Fix UPDATE/UPSERT/DELETE authorization doc .. [DOCS] Fix UPDATE/UPSERT/DELETE authorization doc The patch also fixes broken links in the authorization doc. Change-Id: I9bf6109636e44ca514cfe74fb565f7c506ec0708 Reviewed-on: http://gerrit.cloudera.org:8080/10975 Reviewed-by: Fredy Wijaya Tested-by: Impala Public Jenkins --- M docs/shared/impala_common.xml M docs/topics/impala_authorization.xml 2 files changed, 339 insertions(+), 1,077 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/74/13474/1 -- To view, visit http://gerrit.cloudera.org:8080/13474 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: newchange Gerrit-Change-Id: I9bf6109636e44ca514cfe74fb565f7c506ec0708 Gerrit-Change-Number: 13474 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR](2.x) IMPALA-6881: COMPUTE STATS should require SELECT privilege at analysis
Hello Impala Public Jenkins, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/13473 to review the following change. Change subject: IMPALA-6881: COMPUTE STATS should require SELECT privilege at analysis .. IMPALA-6881: COMPUTE STATS should require SELECT privilege at analysis In order to do COMPUTE STATS, Impala performs several SELECT queries. However, in the COMPUTE STATS analysis phase, we only check for the ALTER privilege. Although the SELECT privilege is eventually checked in the target table by the SELECT child queries, it is better to check the SELECT privilege in the COMPUTE STATS analysis phase and fail early if the privilege requirements are not met instead of failing later in the SELECT child queries due to insufficient privileges. Testing: - Updated the authorization tests - Ran all FE tests Change-Id: Icead43e689404a286859344e309427eb6c68646a Reviewed-on: http://gerrit.cloudera.org:8080/10918 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java M fe/src/main/java/org/apache/impala/analysis/PartitionDef.java M fe/src/main/java/org/apache/impala/analysis/PartitionSpecBase.java M fe/src/test/java/org/apache/impala/analysis/AuditingTest.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java 7 files changed, 93 insertions(+), 51 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/73/13473/1 -- To view, visit http://gerrit.cloudera.org:8080/13473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: newchange Gerrit-Change-Id: Icead43e689404a286859344e309427eb6c68646a Gerrit-Change-Number: 13473 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR](2.x) [DOCS] Fix UPDATE/UPSERT/DELETE authorization doc
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13474 ) Change subject: [DOCS] Fix UPDATE/UPSERT/DELETE authorization doc .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/346/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/13474 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I9bf6109636e44ca514cfe74fb565f7c506ec0708 Gerrit-Change-Number: 13474 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 30 May 2019 14:36:36 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-6881: COMPUTE STATS should require SELECT privilege at analysis
Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/13473 )
Change subject: IMPALA-6881: COMPUTE STATS should require SELECT privilege at
analysis
..
Patch Set 1:
Few Conflicts:
fe/src/main/java/org/apache/impala/analysis/Analyzer.java
2406 <<< HEAD
2407 public FeTable getTable(TableName tableName, Privilege privilege,
2408 boolean addAccessEvent) throws AnalysisException,
TableLoadingException {
2409 ===
2410 public FeTable getTable(TableName tableName, boolean addAccessEvent,
2411 Privilege... privilege) throws AnalysisException,
TableLoadingException {
2412 >>> c0e00ca... IMPALA-6881: COMPUTE STATS should require SELECT
privilege at analysis
2413 Preconditions.checkNotNull(tableName);
2414 Preconditions.checkNotNull(privilege);
2415 tableName = getFqTableName(tableName);
--
To view, visit http://gerrit.cloudera.org:8080/13473
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: 2.x
Gerrit-MessageType: comment
Gerrit-Change-Id: Icead43e689404a286859344e309427eb6c68646a
Gerrit-Change-Number: 13473
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Comment-Date: Thu, 30 May 2019 14:38:20 +
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369: Add HIVE MAJOR VERSION section to EE tests + some fixes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13472 ) Change subject: IMPALA-8369: Add HIVE_MAJOR_VERSION section to EE tests + some fixes .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4348/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13472 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c3cff05ed7080b655b6af64ea09c0691e7dd931 Gerrit-Change-Number: 13472 Gerrit-PatchSet: 2 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 30 May 2019 14:52:30 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) [DOCS] Fix UPDATE/UPSERT/DELETE authorization doc
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13474 ) Change subject: [DOCS] Fix UPDATE/UPSERT/DELETE authorization doc .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/346/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/13474 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I9bf6109636e44ca514cfe74fb565f7c506ec0708 Gerrit-Change-Number: 13474 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 30 May 2019 14:57:42 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-6881: COMPUTE STATS should require SELECT privilege at analysis
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13473 ) Change subject: IMPALA-6881: COMPUTE STATS should require SELECT privilege at analysis .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3441/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: Icead43e689404a286859344e309427eb6c68646a Gerrit-Change-Number: 13473 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 30 May 2019 15:07:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369: Add HIVE MAJOR VERSION section to EE tests + some fixes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13472 ) Change subject: IMPALA-8369: Add HIVE_MAJOR_VERSION section to EE tests + some fixes .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3440/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13472 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c3cff05ed7080b655b6af64ea09c0691e7dd931 Gerrit-Change-Number: 13472 Gerrit-PatchSet: 1 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 30 May 2019 15:12:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8588: Fix revoke grant option with Ranger
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13450 ) Change subject: IMPALA-8588: Fix revoke grant option with Ranger .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/13450/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13450/2//COMMIT_MSG@20 PS2, Line 20: REVOKE GRANT OPTION FOR SELECT ON DATABASE FROM USER : : This will revoke the grant option for all privileges on this database : resource. It will not revoke the SELECT privilege on the resource. i have some concern with this, especially since Ranger behaves differently than Sentry. GRANT OPTION (delegate admin) in Ranger isn't per privilege. In other words the word `SELECT` there is actually useless. In short: REVOKE GRANT OPTION FOR SELECT ON DATABASE FROM USER can mean the same thing as REVOKE GRANT OPTION FOR INSERT ON DATABASE FROM USER . I think that's very confusing. I have 2 proposals: 1. Relax the REVOKE grammar a bit, such that "REVOKE GRANT OPTION ON DATABASE " to disable the delegate admin regardless of the privilege. Sentry doesn't support this, so we need to throw an appropriate exception accordingly. 2. Update the REVOKE grammar, such as "REVOKE SELECT ON DATABASE WITH GRANT OPTION" to mean revoke SELECT on the given database as well as revoking the delegate admin. This is essentially the opposite of GRANT SELECT ON DATABASE ... WITH GRANT OPTION that grants SELECT on the given database and enables the delegate admin. Thoughts? -- To view, visit http://gerrit.cloudera.org:8080/13450 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddfccb442c3be3c266dbc2d8ae85c5674c534d7c Gerrit-Change-Number: 13450 Gerrit-PatchSet: 2 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 30 May 2019 16:08:53 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8585: Fix HMS integration tests for Hive 3
Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13475 Change subject: IMPALA-8585: Fix HMS integration tests for Hive 3 .. IMPALA-8585: Fix HMS integration tests for Hive 3 Hive 3 changed the output format of the 'describe formatted ' command. Added a new function that can parse the new output. Hive 3 also got more permissive during scanning, e.g. it can scan a Parquet column of INT type while the HMS metadata for that column is STRING. It does automatic conversion during scanning. However, it doesn't do the opposite, i.e. when the Parquet column type is STRING and the HMS column type is INT it raises an exception. Extended the test with this case. Change-Id: I51c91c7fc706175295b78abaacf47a86156714ce (cherry picked from commit 5fb83e73e83d5a6496553d88daa9a2510e5b97bd) --- M tests/metadata/test_hms_integration.py 1 file changed, 45 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/13475/1 -- To view, visit http://gerrit.cloudera.org:8080/13475 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I51c91c7fc706175295b78abaacf47a86156714ce Gerrit-Change-Number: 13475 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8369: Fix HMS integration tests for Hive 3
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13475 to look at the new patch set (#2). Change subject: IMPALA-8369: Fix HMS integration tests for Hive 3 .. IMPALA-8369: Fix HMS integration tests for Hive 3 Hive 3 changed the output format of the 'describe formatted ' command. Added a new function that can parse the new output. Hive 3 also got more permissive during scanning, e.g. it can scan a Parquet column of INT type while the HMS metadata for that column is STRING. It does automatic conversion during scanning. However, it doesn't do the opposite, i.e. when the Parquet column type is STRING and the HMS column type is INT it raises an exception. Extended the test with this case. Change-Id: I51c91c7fc706175295b78abaacf47a86156714ce (cherry picked from commit 5fb83e73e83d5a6496553d88daa9a2510e5b97bd) --- M tests/metadata/test_hms_integration.py 1 file changed, 45 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/13475/2 -- To view, visit http://gerrit.cloudera.org:8080/13475 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I51c91c7fc706175295b78abaacf47a86156714ce Gerrit-Change-Number: 13475 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8333: Remove Impala Shell warnings part 2
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13404 ) Change subject: IMPALA-8333: Remove Impala Shell warnings part 2 .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13404 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8ae7e068894da5981fc083e690051da268bfde4d Gerrit-Change-Number: 13404 Gerrit-PatchSet: 4 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Thu, 30 May 2019 17:05:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8333: Remove Impala Shell warnings part 2
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13404 ) Change subject: IMPALA-8333: Remove Impala Shell warnings part 2 .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4349/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13404 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8ae7e068894da5981fc083e690051da268bfde4d Gerrit-Change-Number: 13404 Gerrit-PatchSet: 4 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Thu, 30 May 2019 17:05:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8504: Support CREATE TABLE statement with Kudu/HMS integration
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13318 ) Change subject: IMPALA-8504: Support CREATE TABLE statement with Kudu/HMS integration .. Patch Set 11: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4350/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13318 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I465673d749221bd5f3772814b1c22c2673a53f5c Gerrit-Change-Number: 13318 Gerrit-PatchSet: 11 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Thu, 30 May 2019 17:11:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8595: Support TLSv1.2 with Python < 2.7.9 in shell
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13457 ) Change subject: IMPALA-8595: Support TLSv1.2 with Python < 2.7.9 in shell .. Patch Set 5: I'm going to do a test run on CentOS7 -- To view, visit http://gerrit.cloudera.org:8080/13457 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3fb6510f4b556bd8c6b1e86380379aba8be4b805 Gerrit-Change-Number: 13457 Gerrit-PatchSet: 5 Gerrit-Owner: Robbie Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 30 May 2019 17:16:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369: Fix HMS integration tests for Hive 3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13475 ) Change subject: IMPALA-8369: Fix HMS integration tests for Hive 3 .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3442/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13475 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I51c91c7fc706175295b78abaacf47a86156714ce Gerrit-Change-Number: 13475 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 30 May 2019 17:20:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8553,IMPALA-8552: fix checks for remote cluster
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13386 ) Change subject: IMPALA-8553,IMPALA-8552: fix checks for remote cluster .. Patch Set 3: Just an update - I am testing this against a remote cluster and running into some issues. -- To view, visit http://gerrit.cloudera.org:8080/13386 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifa6b2a1391f53121d3d7c00c5cf0a57590899ce4 Gerrit-Change-Number: 13386 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 30 May 2019 17:20:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8491: Non-root user in container
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13451 ) Change subject: IMPALA-8491: Non-root user in container .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4351/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13451 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0bea9f44a8199851ed04fbef8caf4a2350ae2c0e Gerrit-Change-Number: 13451 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Thu, 30 May 2019 17:20:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: memcpy requires two non-null arguments
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13434 ) Change subject: IMPALA-5031: memcpy requires two non-null arguments .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13434 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2e8e57c34c2848f0dc7dbf32892cc6e86df63506 Gerrit-Change-Number: 13434 Gerrit-PatchSet: 1 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 30 May 2019 17:28:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: signed overflow is undefined behavior
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13433 ) Change subject: IMPALA-5031: signed overflow is undefined behavior .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13433 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic638ff4959eaaffc79caa3453dbccaaabcbe95c9 Gerrit-Change-Number: 13433 Gerrit-PatchSet: 1 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 30 May 2019 17:27:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: NULL is undefined in memcpy and memcmp
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13436 ) Change subject: IMPALA-5031: NULL is undefined in memcpy and memcmp .. Patch Set 2: Code-Review+2 Seems good once you fix the one bot comment. -- To view, visit http://gerrit.cloudera.org:8080/13436 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9362ce6b9ba470ed90e5bd2dc313b66ebd8c6af5 Gerrit-Change-Number: 13436 Gerrit-PatchSet: 2 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 30 May 2019 17:29:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369: Fix HMS integration tests for Hive 3
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/13475 ) Change subject: IMPALA-8369: Fix HMS integration tests for Hive 3 .. Patch Set 2: Code-Review+2 (4 comments) Just a few nits. http://gerrit.cloudera.org:8080/#/c/13475/2/tests/metadata/test_hms_integration.py File tests/metadata/test_hms_integration.py: http://gerrit.cloudera.org:8080/#/c/13475/2/tests/metadata/test_hms_integration.py@217 PS2, Line 217: output = self.run_stmt_in_hive( : 'describe formatted %s %s' % : (table, column)) nit: fits to 1 line http://gerrit.cloudera.org:8080/#/c/13475/2/tests/metadata/test_hms_integration.py@632 PS2, Line 632: if HIVE_MAJOR_VERSION == 2: Can you add a comment about Hive 3 becoming more tolerant? http://gerrit.cloudera.org:8080/#/c/13475/2/tests/metadata/test_hms_integration.py@646 PS2, Line 646: 'insert into table %s values (33,\'100\')' % table_name) nit: + 2 indentation http://gerrit.cloudera.org:8080/#/c/13475/2/tests/metadata/test_hms_integration.py@650 PS2, Line 650: self.run_stmt_in_hive, 'select * from %s' % nit: + 2 indentation -- To view, visit http://gerrit.cloudera.org:8080/13475 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I51c91c7fc706175295b78abaacf47a86156714ce Gerrit-Change-Number: 13475 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 30 May 2019 17:37:33 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8460: Simplify cluster membership management
Hello Michael Ho, Thomas Marshall, Tim Armstrong, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13207 to look at the new patch set (#15). Change subject: IMPALA-8460: Simplify cluster membership management .. IMPALA-8460: Simplify cluster membership management This change adds a class to track cluster membership called ClusterMembershipMgr. It replaces the logic that was partially duplicated between the ImpalaServer and the Coordinator and makes sure that the local backend descriptor is consistent (IMPALA-8469). The ClusterMembershipMgr maintains a view of the cluster membership and incorporates incoming updates from the statestore. It also registers the local backend with the statestore after startup. Clients can obtain a consistent, immutable snapshot of the current cluster membership from the ClusterMembershipMgr. Additionally, callbacks can be registered to receive notifications of cluster membership changes. The ImpalaServer and Frontend use this mechanism. This change also unifies the naming of executor-related classes, in particular it renames "BackendConfig" to "ExecutorGroup". In anticipation of a subsequent change (IMPALA-8484), it adds maps to store multiple executor groups. This change also disables the generation of default operators from the thrift files and instead adds explicit implementations for the ones that we rely on. This forces us to explicitly specify comparators when manipulating containers of thrift structs and will help prevent accidental bugs. Testing: This change adds a backend unit test for the new cluster membership manager. The observable behavior of Impala does not change, and the existing scheduler unit test and end to end tests should make sure of that. Change-Id: Ib3cf9a8bb060d0c6e9ec8868b7b21ce01f8740a3 --- M be/src/benchmarks/scheduler-benchmark.cc M be/src/common/logging.h M be/src/common/thread-debug-info-test.cc M be/src/experiments/hash-ring-util.cc M be/src/gutil/strings/split.cc M be/src/gutil/strings/split.h M be/src/rpc/thrift-util-test.cc M be/src/rpc/thrift-util.cc M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/runtime/krpc-data-stream-mgr.cc M be/src/runtime/query-state.h M be/src/scheduling/CMakeLists.txt D be/src/scheduling/backend-config-test.cc D be/src/scheduling/backend-config.cc D be/src/scheduling/backend-config.h A be/src/scheduling/cluster-membership-mgr-test.cc A be/src/scheduling/cluster-membership-mgr.cc A be/src/scheduling/cluster-membership-mgr.h A be/src/scheduling/cluster-membership-test-util.cc A be/src/scheduling/cluster-membership-test-util.h A be/src/scheduling/executor-group-test.cc A be/src/scheduling/executor-group.cc A be/src/scheduling/executor-group.h M be/src/scheduling/query-schedule.cc M be/src/scheduling/query-schedule.h M be/src/scheduling/scheduler-test-util.cc M be/src/scheduling/scheduler-test-util.h M be/src/scheduling/scheduler-test.cc M be/src/scheduling/scheduler.cc M be/src/scheduling/scheduler.h M be/src/service/impala-http-handler.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/statestore/statestore-subscriber.cc M be/src/statestore/statestore-subscriber.h M be/src/statestore/statestore.cc M be/src/testutil/in-process-servers.cc M be/src/util/container-util.h M be/src/util/network-util.cc M be/src/util/network-util.h M be/src/util/runtime-profile-test.cc M be/src/util/uid-util-test.cc M common/thrift/CMakeLists.txt M common/thrift/Frontend.thrift M common/thrift/StatestoreService.thrift M fe/src/main/java/org/apache/impala/service/Frontend.java M tests/custom_cluster/test_coordinators.py 48 files changed, 1,920 insertions(+), 1,012 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/13207/15 -- To view, visit http://gerrit.cloudera.org:8080/13207 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib3cf9a8bb060d0c6e9ec8868b7b21ce01f8740a3 Gerrit-Change-Number: 13207 Gerrit-PatchSet: 15 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7369: part 1: Implement TRUNC, DATE TRUNC, EXTRACT, DATE PART functions for DATE
Attila Jeges has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/13363 ) Change subject: IMPALA-7369: part 1: Implement TRUNC, DATE_TRUNC, EXTRACT, DATE_PART functions for DATE .. IMPALA-7369: part 1: Implement TRUNC, DATE_TRUNC, EXTRACT, DATE_PART functions for DATE These functions are somewhat similar in that each of them takes a DATE argument and a time unit to work with. They work identically to the corresponding TIMESTAMP functions. The only difference is that the DATE functions don't accept time-of-day units. TRUNC(DATE d, STRING unit) Truncates a DATE value to the specified time unit. The 'unit' argument is case insensitive. This argument string can be one of: S, , YEAR, SYEAR, YYY, YY, Y: Year. Q: Quarter. MONTH, MON, MM, RM: Month. WW: Same day of the week as the first day of the year. W: Same day of the week as the first day of the month. DDD, DD, J: Day. DAY, DY, D: Starting day (Monday) of the week. The impelementation mirrors Impala's TRUNC(TIMESTAMP ts, STRING unit) function. Hive and Oracle SQL have a similar function too. Reference: http://docs.oracle.com/cd/B19306_01/server.102/b14200/functions201.htm . DATE_TRUNC(STRING unit, DATE d) Truncates a DATE value to the specified precision. The 'unit' argument is case insensitive. This argument string can be one of: DAY, WEEK, MONTH, YEAR, DECADE, CENTURY, MILLENNIUM. The implementation mirrors Impala's DATE_TRUNC(STRING unit, TIMESTAMP ts) function. Vertica has a similar function too. Reference: https://my.vertica.com/docs/8.1.x/HTML/index.htm#Authoring/ SQLReferenceManual/Functions/Date-Time/DATE_TRUNC.htm . EXTRACT(DATE d, STRING unit), EXTRACT(unit FROM DATE d) Returns one of the numeric date fields from a DATE value. The 'unit' string can be one of YEAR, QUARTER, MONTH, DAY. This argument value is case-insensitive. The implementation mirrors that Impala's EXTRACT(TIMESTAMP ts, STRING unit). Hive and Oracle SQL have a similar function too. Reference: http://docs.oracle.com/cd/B19306_01/server.102/b14200/functions050.htm . DATE_PART(STRING unit, DATE date) Similar to EXTRACT(), with the argument order reversed. Supports the same date units as EXTRACT(). The implementation mirrors Impala's DATE_PART(STRING unit, TIMESTAMP ts) function. Change-Id: I843358a45eb5faa2c134994600546fc1d0a797c8 --- M be/src/benchmarks/CMakeLists.txt A be/src/benchmarks/date-benchmark.cc M be/src/exprs/expr-test.cc M be/src/exprs/udf-builtins-ir.cc M be/src/exprs/udf-builtins.cc M be/src/exprs/udf-builtins.h M be/src/runtime/date-test.cc M be/src/runtime/date-value.cc M be/src/runtime/date-value.h M common/function-registry/impala_functions.py M fe/src/main/java/org/apache/impala/analysis/ExtractFromExpr.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java 12 files changed, 1,376 insertions(+), 302 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/13363/6 -- To view, visit http://gerrit.cloudera.org:8080/13363 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I843358a45eb5faa2c134994600546fc1d0a797c8 Gerrit-Change-Number: 13363 Gerrit-PatchSet: 6 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-7369: part 1: Implement TRUNC, DATE TRUNC, EXTRACT, DATE PART functions for DATE
Attila Jeges has posted comments on this change. ( http://gerrit.cloudera.org:8080/13363 ) Change subject: IMPALA-7369: part 1: Implement TRUNC, DATE_TRUNC, EXTRACT, DATE_PART functions for DATE .. Patch Set 6: Code-Review+1 (1 comment) Carry +1 http://gerrit.cloudera.org:8080/#/c/13363/5/be/src/runtime/date-value.cc File be/src/runtime/date-value.cc: http://gerrit.cloudera.org:8080/#/c/13363/5/be/src/runtime/date-value.cc@120 PS5, Line 120: // > The goal of this logic could be also added, e.g. that it will be fast for m Done -- To view, visit http://gerrit.cloudera.org:8080/13363 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I843358a45eb5faa2c134994600546fc1d0a797c8 Gerrit-Change-Number: 13363 Gerrit-PatchSet: 6 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 30 May 2019 17:54:20 + Gerrit-HasComments: Yes
[Impala-ASF-CR](2.x) [DOCS] Fix UPDATE/UPSERT/DELETE authorization doc
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/13474 ) Change subject: [DOCS] Fix UPDATE/UPSERT/DELETE authorization doc .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13474 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I9bf6109636e44ca514cfe74fb565f7c506ec0708 Gerrit-Change-Number: 13474 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 30 May 2019 18:16:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8460: Simplify cluster membership management
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13207 ) Change subject: IMPALA-8460: Simplify cluster membership management .. Patch Set 15: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3443/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13207 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib3cf9a8bb060d0c6e9ec8868b7b21ce01f8740a3 Gerrit-Change-Number: 13207 Gerrit-PatchSet: 15 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 30 May 2019 18:20:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8536: Add Scalable Pool Configuration to Admission Controller.
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/13307 ) Change subject: IMPALA-8536: Add Scalable Pool Configuration to Admission Controller. .. Patch Set 3: (12 comments) looks good, just looked at the implementation for now, will look at the tests next http://gerrit.cloudera.org:8080/#/c/13307/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13307/3//COMMIT_MSG@14 PS3, Line 14: The new configuration parameters are: mention that a value of zero means it wont be used and will default to the non-scalable param. http://gerrit.cloudera.org:8080/#/c/13307/3//COMMIT_MSG@15 PS3, Line 15: Max Running nit: use "Max Requests" to be consistent http://gerrit.cloudera.org:8080/#/c/13307/3//COMMIT_MSG@26 PS3, Line 26: the Multiple of the current total number of : running Impalads which gives the maximum memory available across the : cluster for the pool. can you elaborate on what this number is like you did for the ones above? Also mention the unit (bytes). Also update the same in the thrift definition http://gerrit.cloudera.org:8080/#/c/13307/3/be/src/scheduling/admission-controller.h File be/src/scheduling/admission-controller.h: http://gerrit.cloudera.org:8080/#/c/13307/3/be/src/scheduling/admission-controller.h@603 PS3, Line 603: cluster_size_snapshot nit: maybe use something like latest_cluster_size. the snapshot in the name threw me off a bit since i expected a struct or something instead of just the size. Feel free to ignore this comment. http://gerrit.cloudera.org:8080/#/c/13307/3/be/src/scheduling/admission-controller.cc File be/src/scheduling/admission-controller.cc: http://gerrit.cloudera.org:8080/#/c/13307/3/be/src/scheduling/admission-controller.cc@1000 PS3, Line 1000: // Use a heuristic to limit the number of requests we dequeue locally to avoid all : // impalads dequeuing too many requests at the same time. This is based on the : // max_requests_estimate limit and the current queue size. We will attempt to : // dequeue up to this number of requests until reaching the per-pool memory limit. nit: if this functionality is completely moved over to a method, maybe we can mention all this in the method comment and remove it here http://gerrit.cloudera.org:8080/#/c/13307/3/be/src/scheduling/admission-controller.cc@1007 PS3, Line 1007: if (!status.ok()) continue; // to next pool > I think the old behaviour had the advantage of logging something if nothing I agree, would prefer returning max_to_dequeue itself and handling the <= 0 case http://gerrit.cloudera.org:8080/#/c/13307/3/be/src/scheduling/admission-controller.cc@1071 PS3, Line 1071: total_available this can still be less than equal to zero, right? like in case of overadmission. If yes then we can get rid of the branching based on PoolHasFixedRunningQueriesCountLimit http://gerrit.cloudera.org:8080/#/c/13307/3/be/src/scheduling/admission-controller.cc@1121 PS3, Line 1121: "Invalid pool config: the min_query_mem_limit $0 is greater than Seems like now this case can happen if the cluster size changes (an external reason), so calling this an invalid config might not be right. Do you think it makes sense to mention both cases (invalid config and low cluster size) and offer ways to fix it, like we do in REASON_MEM_LIMIT_TOO_LOW_FOR_RESERVATION http://gerrit.cloudera.org:8080/#/c/13307/3/be/src/scheduling/admission-controller.cc@1396 PS3, Line 1396: GetKnownBackends this might be a problem if some backends are shutting down and the query wont be scheduled on them but the max memory used is still high, causing over admission. If instead we only look at the size of the active backends, that would be a problem for queries already scheduled to run on the backends shutting down. @all Do you think it makes sense to handle this transient period right now? It might be a problem if a group of executors have started shutting down at the same time. Another solution could be use to the cluster size from the QuerySchedule itself, but that has its own set of problems. Any thoughts? http://gerrit.cloudera.org:8080/#/c/13307/3/be/src/scheduling/admission-controller.cc@1403 PS3, Line 1403: max_memory_multiple just thinking out loud: should we check if this multiple is greater than the executor.mem_to_admit for each executor because in that case this would be an invalid config. Maybe not, since there is currently nothing preventing us from setting a higher pool_config.max_mem_resources as well. http://gerrit.cloudera.org:8080/#/c/13307/3/be/src/scheduling/admission-controller.cc@1444 PS3, Line 1444: && should be OR (||) http://gerrit.cloudera.org:8080/#/c/13307/3/common/thrift/ImpalaInternalService.thrift File common/thrift/ImpalaInternalService.thrift: http://gerrit.cloudera.org:8080/#/c/13307/3/common
[Impala-ASF-CR] IMPALA-8595: Support TLSv1.2 with Python < 2.7.9 in shell
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13457 ) Change subject: IMPALA-8595: Support TLSv1.2 with Python < 2.7.9 in shell .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13457 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3fb6510f4b556bd8c6b1e86380379aba8be4b805 Gerrit-Change-Number: 13457 Gerrit-PatchSet: 5 Gerrit-Owner: Robbie Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 30 May 2019 18:28:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7369: part 1: Implement TRUNC, DATE TRUNC, EXTRACT, DATE PART functions for DATE
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13363 ) Change subject: IMPALA-7369: part 1: Implement TRUNC, DATE_TRUNC, EXTRACT, DATE_PART functions for DATE .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3444/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13363 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I843358a45eb5faa2c134994600546fc1d0a797c8 Gerrit-Change-Number: 13363 Gerrit-PatchSet: 6 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 30 May 2019 18:32:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8491: Non-root user in container
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/13451 ) Change subject: IMPALA-8491: Non-root user in container .. Patch Set 2: Wrapping my head around this one. Let me double-check my understanding: If we ran the docker container as non-root and had the daemon_entrypoint.sh change, am I right in thinking that this would be sufficient just for our testing purposes? i.e. It generates directories as the appropriate user and has a username inside the container. We want the Dockerfile change, because that handles the case where someone runs the container without a user. If we ran with only the Dockerfile change, we would have a username (so wouldn't change /etc/passwd), but the directories would be created as some unknown uid (as far as the host OS is concerned). Is that right? No hurry, but do we still need the umask? -- To view, visit http://gerrit.cloudera.org:8080/13451 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0bea9f44a8199851ed04fbef8caf4a2350ae2c0e Gerrit-Change-Number: 13451 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Thu, 30 May 2019 18:52:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8491: Non-root user in container
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13451 ) Change subject: IMPALA-8491: Non-root user in container .. Patch Set 2: > If we ran the docker container as non-root and had the daemon_entrypoint.sh > change, am I right in thinking that this would be sufficient just for our > testing purposes? i.e. It generates directories as the appropriate user and > has a username inside the container. Correct > We want the Dockerfile change, because that handles the case where someone > runs the container without a user. If we ran with only the Dockerfile change, > we would have a username (so wouldn't change /etc/passwd), but the > directories would be created as some unknown uid (as far as the host OS is > concerned). Is that right? Correct. Setting USER in a Dockerfile is consider a good practice because it prevents people accidentally running the container as root. I think you're right that we don't need the umask - I don't think it's harmful, but could remove it to keep things simpler. -- To view, visit http://gerrit.cloudera.org:8080/13451 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0bea9f44a8199851ed04fbef8caf4a2350ae2c0e Gerrit-Change-Number: 13451 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 30 May 2019 18:58:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8491: Non-root user in container
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/13451 ) Change subject: IMPALA-8491: Non-root user in container .. Patch Set 2: Code-Review+2 Cool, this makes sense. The umask thing is not a blocker. -- To view, visit http://gerrit.cloudera.org:8080/13451 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0bea9f44a8199851ed04fbef8caf4a2350ae2c0e Gerrit-Change-Number: 13451 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 30 May 2019 19:03:29 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Added the section on object ownership
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13478 ) Change subject: [DOCS] Added the section on object ownership .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/347/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/13478 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iff48684f457ef19a27524adfbcc2ae5e098320a3 Gerrit-Change-Number: 13478 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 30 May 2019 19:19:33 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Added the section on object ownership
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13478 Change subject: [DOCS] Added the section on object ownership .. [DOCS] Added the section on object ownership Change-Id: Iff48684f457ef19a27524adfbcc2ae5e098320a3 --- M docs/topics/impala_authorization.xml 1 file changed, 39 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/13478/1 -- To view, visit http://gerrit.cloudera.org:8080/13478 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iff48684f457ef19a27524adfbcc2ae5e098320a3 Gerrit-Change-Number: 13478 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] IMPALA-8564: Add table/view create time in the lineage graph
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/13399 )
Change subject: IMPALA-8564: Add table/view create time in the lineage graph
..
Patch Set 14:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/13399/13/common/thrift/Results.thrift
File common/thrift/Results.thrift:
http://gerrit.cloudera.org:8080/#/c/13399/13/common/thrift/Results.thrift@49
PS13, Line 49: struct TResultSetMetadata {
> I'm surprised to see these included in this particular class, since these a
Yeah originally I thought it was aligned with the name "metadata", but the more
I think about it, I don't think it makes sense. So putting these in
TDdlExecResponse make a lot sense. Done.
http://gerrit.cloudera.org:8080/#/c/13399/13/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/13399/13/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1953
PS13, Line 1953: msClient.getHiveClient().createTable(newTable);
> this can just be defined on line 1957 right?
Done
http://gerrit.cloudera.org:8080/#/c/13399/13/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1958
PS13, Line 1958: newTable.getDbName(),
newTable.getTableName()).getCreateTime();
> Can we add a TODO here and file a jira for Hive to add a new variant of cre
Done
--
To view, visit http://gerrit.cloudera.org:8080/13399
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If4f578d7b299a76c30323b10a883ba32f8713d82
Gerrit-Change-Number: 13399
Gerrit-PatchSet: 14
Gerrit-Owner: Fredy Wijaya
Gerrit-Reviewer: Anonymous Coward (498)
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Todd Lipcon
Gerrit-Comment-Date: Thu, 30 May 2019 19:20:47 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8564: Add table/view create time in the lineage graph
Fredy Wijaya has uploaded a new patch set (#14). (
http://gerrit.cloudera.org:8080/13399 )
Change subject: IMPALA-8564: Add table/view create time in the lineage graph
..
IMPALA-8564: Add table/view create time in the lineage graph
This patch adds table/view create time in the lineage graph. This is
needed for Impala/Atlas integration. See ATLAS-3080.
Below is an example of the updated lineage graph.
{
"queryText":"create table lineage_test_tbl as select int_col, tinyint_col
from functional.alltypes",
"queryId":"0:0",
"hash":"407f23b24758ffcb2ac445b9703f5c44",
"user":"dummy_user",
"timestamp":1547867921,
"edges":[
{
"sources":[
1
],
"targets":[
0
],
"edgeType":"PROJECTION"
},
{
"sources":[
3
],
"targets":[
2
],
"edgeType":"PROJECTION"
}
],
"vertices":[
{
"id":0,
"vertexType":"COLUMN",
"vertexId":"int_col",
"metadata":{
"tableName":"default.lineage_test_tbl",
"tableCreateTime":1559151337
}
},
{
"id":1,
"vertexType":"COLUMN",
"vertexId":"functional.alltypes.int_col",
"metadata":{
"tableName":"functional.alltypes",
"tableCreateTime":1559151317
}
},
{
"id":2,
"vertexType":"COLUMN",
"vertexId":"tinyint_col",
"metadata":{
"tableName":"default.lineage_test_tbl",
"tableCreateTime":1559151337
}
},
{
"id":3,
"vertexType":"COLUMN",
"vertexId":"functional.alltypes.tinyint_col",
"metadata":{
"tableName":"functional.alltypes",
"tableCreateTime":1559151317
}
}
]
}
Testing:
- Updated lineage tests in PlannerTest
- Updated test_lineage.py
- Ran all FE tests
Change-Id: If4f578d7b299a76c30323b10a883ba32f8713d82
---
M be/src/service/client-request-state.h
M be/src/service/frontend.cc
M be/src/service/impala-server.cc
M be/src/util/lineage-util.h
M common/thrift/CatalogService.thrift
M common/thrift/LineageGraph.thrift
M fe/src/main/java/org/apache/impala/analysis/ColumnLineageGraph.java
M fe/src/main/java/org/apache/impala/analysis/CreateOrAlterViewStmtBase.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java
M testdata/workloads/functional-planner/queries/PlannerTest/lineage.test
M tests/custom_cluster/test_lineage.py
15 files changed, 2,124 insertions(+), 621 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/13399/14
--
To view, visit http://gerrit.cloudera.org:8080/13399
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If4f578d7b299a76c30323b10a883ba32f8713d82
Gerrit-Change-Number: 13399
Gerrit-PatchSet: 14
Gerrit-Owner: Fredy Wijaya
Gerrit-Reviewer: Anonymous Coward (498)
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] [DOCS] Added the section on object ownership
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13478 ) Change subject: [DOCS] Added the section on object ownership .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/347/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/13478 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iff48684f457ef19a27524adfbcc2ae5e098320a3 Gerrit-Change-Number: 13478 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 30 May 2019 19:43:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8564: Add table/view create time in the lineage graph
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13399 ) Change subject: IMPALA-8564: Add table/view create time in the lineage graph .. Patch Set 14: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3445/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13399 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If4f578d7b299a76c30323b10a883ba32f8713d82 Gerrit-Change-Number: 13399 Gerrit-PatchSet: 14 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Anonymous Coward (498) Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 30 May 2019 20:00:52 + Gerrit-HasComments: No
[native-toolchain-CR] Bump Kudu version to 84086fe
Thomas Marshall has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13479 Change subject: Bump Kudu version to 84086fe .. Bump Kudu version to 84086fe Change-Id: Ie63a8baf1564f63d6ec9c0b953e2fa4119f51190 --- M buildall.sh 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/79/13479/1 -- To view, visit http://gerrit.cloudera.org:8080/13479 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ie63a8baf1564f63d6ec9c0b953e2fa4119f51190 Gerrit-Change-Number: 13479 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Marshall
[Impala-ASF-CR] IMPALA-8491: Non-root user in container
Hello Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13451 to look at the new patch set (#3). Change subject: IMPALA-8491: Non-root user in container .. IMPALA-8491: Non-root user in container Set a default USER in the Dockerfile per best practices so that consumers of the container don't accidentally run as root. The default user is "impala" if the container is run in docker without specifying a user. Various frameworks, including kubernetes, will run the container with an arbitrary user and group ID set. This causes issues with some Hadoop libraries, which depend on the user having a name. This is generally not the case because inside the container usernames are resolved with the container's /etc/passwd. To work around this, the entrypoint script checks if the current user has a name and if not, assigns it one (either dummyuser or $HADOOP_USER_NAME). Remove the umask setting that was required to make logs modifiable by the host user - this is not needed for our tests since the host host and container users now match up. Also run apt-get clean in Dockerfile to reduce cruft in the image. Change-Id: I0bea9f44a8199851ed04fbef8caf4a2350ae2c0e --- M bin/start-impala-cluster.py M docker/daemon_entrypoint.sh M docker/impala_base/Dockerfile 3 files changed, 21 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/51/13451/3 -- To view, visit http://gerrit.cloudera.org:8080/13451 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0bea9f44a8199851ed04fbef8caf4a2350ae2c0e Gerrit-Change-Number: 13451 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8604: Improve authorization test coverage for update/upsert/delete statements
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13480 Change subject: IMPALA-8604: Improve authorization test coverage for update/upsert/delete statements .. IMPALA-8604: Improve authorization test coverage for update/upsert/delete statements This patch improves the test authorization test coverage by adding test cases for update, upsert, and delete statements at the database and tabel level. Testing: - Ran authorization FE tests Change-Id: Ic4095476945ff413fc59ec99dc3b9dfd71d95e96 --- M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java 1 file changed, 48 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/13480/1 -- To view, visit http://gerrit.cloudera.org:8080/13480 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ic4095476945ff413fc59ec99dc3b9dfd71d95e96 Gerrit-Change-Number: 13480 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-8604: Improve authorization test coverage for update/upsert/delete statements
Austin Nobis has posted comments on this change. ( http://gerrit.cloudera.org:8080/13480 ) Change subject: IMPALA-8604: Improve authorization test coverage for update/upsert/delete statements .. Patch Set 1: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/13480 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic4095476945ff413fc59ec99dc3b9dfd71d95e96 Gerrit-Change-Number: 13480 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 30 May 2019 20:22:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369: Add HIVE MAJOR VERSION section to EE tests + some fixes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13472 ) Change subject: IMPALA-8369: Add HIVE_MAJOR_VERSION section to EE tests + some fixes .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4348/ -- To view, visit http://gerrit.cloudera.org:8080/13472 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c3cff05ed7080b655b6af64ea09c0691e7dd931 Gerrit-Change-Number: 13472 Gerrit-PatchSet: 2 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 30 May 2019 20:23:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8604: Improve authorization test coverage for update/upsert/delete statements
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13480 ) Change subject: IMPALA-8604: Improve authorization test coverage for update/upsert/delete statements .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13480 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic4095476945ff413fc59ec99dc3b9dfd71d95e96 Gerrit-Change-Number: 13480 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 30 May 2019 20:25:36 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-6881: COMPUTE STATS should require SELECT privilege at analysis
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13473 ) Change subject: IMPALA-6881: COMPUTE STATS should require SELECT privilege at analysis .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: Icead43e689404a286859344e309427eb6c68646a Gerrit-Change-Number: 13473 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 30 May 2019 20:26:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8604: Improve authorization test coverage for update/upsert/delete statements
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13480 ) Change subject: IMPALA-8604: Improve authorization test coverage for update/upsert/delete statements .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13480 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic4095476945ff413fc59ec99dc3b9dfd71d95e96 Gerrit-Change-Number: 13480 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 30 May 2019 20:26:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8604: Improve authorization test coverage for update/upsert/delete statements
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13480 ) Change subject: IMPALA-8604: Improve authorization test coverage for update/upsert/delete statements .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4353/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13480 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic4095476945ff413fc59ec99dc3b9dfd71d95e96 Gerrit-Change-Number: 13480 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 30 May 2019 20:26:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8491: Non-root user in container
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13451 ) Change subject: IMPALA-8491: Non-root user in container .. Patch Set 3: I changed the umask thing and updated the commit message. Would be good if you could take a quick look -- To view, visit http://gerrit.cloudera.org:8080/13451 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0bea9f44a8199851ed04fbef8caf4a2350ae2c0e Gerrit-Change-Number: 13451 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 30 May 2019 20:28:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8491: Non-root user in container
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/13451 ) Change subject: IMPALA-8491: Non-root user in container .. Patch Set 3: Code-Review+2 Looks good -- To view, visit http://gerrit.cloudera.org:8080/13451 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0bea9f44a8199851ed04fbef8caf4a2350ae2c0e Gerrit-Change-Number: 13451 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 30 May 2019 20:30:37 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP: IMPALA-8586: Support download URLs for CDP
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13432 ) Change subject: WIP: IMPALA-8586: Support download URLs for CDP .. Patch Set 1: Code-Review+1 Seems like a fine direction to me. Definitely agree that treating these scripts like normal code with good software engineering practices is the right direction (vs a bunch of random shell scripts that we have in various places) -- To view, visit http://gerrit.cloudera.org:8080/13432 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I67824fd82b820e68e9f5c87939ec94ca6abadb8c Gerrit-Change-Number: 13432 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 30 May 2019 20:31:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8491: Non-root user in container
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13451 ) Change subject: IMPALA-8491: Non-root user in container .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4354/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13451 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0bea9f44a8199851ed04fbef8caf4a2350ae2c0e Gerrit-Change-Number: 13451 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 30 May 2019 20:35:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8491: Non-root user in container
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13451 ) Change subject: IMPALA-8491: Non-root user in container .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13451 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0bea9f44a8199851ed04fbef8caf4a2350ae2c0e Gerrit-Change-Number: 13451 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 30 May 2019 20:35:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8491: Non-root user in container
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13451 ) Change subject: IMPALA-8491: Non-root user in container .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4351/ -- To view, visit http://gerrit.cloudera.org:8080/13451 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0bea9f44a8199851ed04fbef8caf4a2350ae2c0e Gerrit-Change-Number: 13451 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 30 May 2019 20:36:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8570: Fix flakiness in test restart statestore query resilience
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13406 ) Change subject: IMPALA-8570: Fix flakiness in test_restart_statestore_query_resilience .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4355/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13406 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0a54b6e149c42b696c954b5240d6de61453bf7f9 Gerrit-Change-Number: 13406 Gerrit-PatchSet: 5 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 30 May 2019 20:42:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8570: Fix flakiness in test restart statestore query resilience
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/13406 ) Change subject: IMPALA-8570: Fix flakiness in test_restart_statestore_query_resilience .. Patch Set 5: Code-Review+2 GVO failed due to unrelated test failure, rebasing just to be sure -- To view, visit http://gerrit.cloudera.org:8080/13406 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0a54b6e149c42b696c954b5240d6de61453bf7f9 Gerrit-Change-Number: 13406 Gerrit-PatchSet: 5 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 30 May 2019 20:41:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7322: Add storage wait time to profile
Hello Bharath Vissapragada, Anurag Mantripragada, Vihang Karajgaonkar, Sahil Takiar, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12940 to look at the new patch set (#10). Change subject: IMPALA-7322: Add storage wait time to profile .. IMPALA-7322: Add storage wait time to profile Add metrics to record storage wait time for operations with metadata load in catalog for hdfs, kudu and hbase tables. Pass storage wait time from catalog to fe through thrift and log total storage load time in query profile. Storage-load-time is the amount of time spent loading metadata from the underlying storage layer (e.g. S3, HDFS, Kudu, HBase), which does not include the amount of time spending loading data from HMS. Testing: Ran queries that can trigger all of, none of or some of the related tables loading. Check query profile for each query. Check catalog metrics for each table. Add unit tests to test_observability.py Ran all core tests. Sample output: Profile:(storage-load-time is the added property): After ran a hbase query (Metadata load finished is divided into several lines because of limitation of commit message): Query Compilation: 4s401ms - Metadata load started: 661.084us (661.084us) - Metadata load finished. loaded-tables=1/1 load-requests=1 catalog-updates=3 storage-load-time=233ms: 3s819ms (3s819ms) - Analysis finished: 3s820ms (763.979us) - Value transfer graph computed: 3s820ms (63.193us) Catalog metrics(this sample is from a hdfs table): storage-metadata-load-duration: Count: 1 Mean rate: 0.0085 1 min. rate: 0.032 5 min. rate: 0.1386 15 min. rate: 0.177 Min (msec): 111 Max (msec): 111 Mean (msec): 111.1802 Median (msec): 111.1802 75th-% (msec): 111.1802 95th-% (msec): 111.1802 99th-% (msec): 111.1802 Change-Id: I6dde7e394b7c1c396d835ef6aa0a55930c0a8660 --- M common/thrift/CatalogObjects.thrift M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/HBaseTable.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M tests/query_test/test_observability.py 7 files changed, 131 insertions(+), 25 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/12940/10 -- To view, visit http://gerrit.cloudera.org:8080/12940 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6dde7e394b7c1c396d835ef6aa0a55930c0a8660 Gerrit-Change-Number: 12940 Gerrit-PatchSet: 10 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen
[Impala-ASF-CR] IMPALA-7322: Add storage wait time to profile
Yongzhi Chen has posted comments on this change. (
http://gerrit.cloudera.org:8080/12940 )
Change subject: IMPALA-7322: Add storage wait time to profile
..
Patch Set 10:
(2 comments)
Submit a new patch
http://gerrit.cloudera.org:8080/#/c/12940/9/fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java
File fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java:
http://gerrit.cloudera.org:8080/#/c/12940/9/fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java@239
PS9, Line 239: .filter(Table.class::isInstance)
: .map(Table.class::cast)
> do u need a space between ".class" and "::"
I did not do that, it was changed by the reformatting tool(recommended by a
ramp-up doc):
git diff -U0 --no-color HEAD^ |
"${IMPALA_TOOLCHAIN}/llvm-${IMPALA_LLVM_VERSION}/share/clang/clang-format-diff.py"
-binary "${IMPALA_TOOLCHAIN}/llvm-${IMPALA_LLVM_VERSION}/bin/clang-format" -p1
-i
I will remove the space.
http://gerrit.cloudera.org:8080/#/c/12940/7/fe/src/main/java/org/apache/impala/catalog/Table.java
File fe/src/main/java/org/apache/impala/catalog/Table.java:
http://gerrit.cloudera.org:8080/#/c/12940/7/fe/src/main/java/org/apache/impala/catalog/Table.java@198
PS7, Line 198: metrics_.addTimer(REFRESH_DURATION_METRIC);
> Then we should add a snippet of the updated metrics to the commit message (
Will add.
--
To view, visit http://gerrit.cloudera.org:8080/12940
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6dde7e394b7c1c396d835ef6aa0a55930c0a8660
Gerrit-Change-Number: 12940
Gerrit-PatchSet: 10
Gerrit-Owner: Yongzhi Chen
Gerrit-Reviewer: Anurag Mantripragada
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Sahil Takiar
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Reviewer: Yongzhi Chen
Gerrit-Comment-Date: Thu, 30 May 2019 20:48:31 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8564: Add table/view create time in the lineage graph
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13399 ) Change subject: IMPALA-8564: Add table/view create time in the lineage graph .. Patch Set 14: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13399 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If4f578d7b299a76c30323b10a883ba32f8713d82 Gerrit-Change-Number: 13399 Gerrit-PatchSet: 14 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Anonymous Coward (498) Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 30 May 2019 20:52:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8564: Add table/view create time in the lineage graph
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13399 ) Change subject: IMPALA-8564: Add table/view create time in the lineage graph .. Patch Set 15: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4356/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13399 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If4f578d7b299a76c30323b10a883ba32f8713d82 Gerrit-Change-Number: 13399 Gerrit-PatchSet: 15 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Anonymous Coward (498) Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 30 May 2019 20:54:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8564: Add table/view create time in the lineage graph
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13399 ) Change subject: IMPALA-8564: Add table/view create time in the lineage graph .. Patch Set 15: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13399 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If4f578d7b299a76c30323b10a883ba32f8713d82 Gerrit-Change-Number: 13399 Gerrit-PatchSet: 15 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Anonymous Coward (498) Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 30 May 2019 20:54:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8491: Non-root user in container
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13451 ) Change subject: IMPALA-8491: Non-root user in container .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3446/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13451 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0bea9f44a8199851ed04fbef8caf4a2350ae2c0e Gerrit-Change-Number: 13451 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 30 May 2019 20:59:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8604: Improve authorization test coverage for update/upsert/delete statements
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13480 ) Change subject: IMPALA-8604: Improve authorization test coverage for update/upsert/delete statements .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3447/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13480 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic4095476945ff413fc59ec99dc3b9dfd71d95e96 Gerrit-Change-Number: 13480 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 30 May 2019 21:01:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8460: Simplify cluster membership management
Hello Michael Ho, Thomas Marshall, Tim Armstrong, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13207 to look at the new patch set (#16). Change subject: IMPALA-8460: Simplify cluster membership management .. IMPALA-8460: Simplify cluster membership management This change adds a class to track cluster membership called ClusterMembershipMgr. It replaces the logic that was partially duplicated between the ImpalaServer and the Coordinator and makes sure that the local backend descriptor is consistent (IMPALA-8469). The ClusterMembershipMgr maintains a view of the cluster membership and incorporates incoming updates from the statestore. It also registers the local backend with the statestore after startup. Clients can obtain a consistent, immutable snapshot of the current cluster membership from the ClusterMembershipMgr. Additionally, callbacks can be registered to receive notifications of cluster membership changes. The ImpalaServer and Frontend use this mechanism. This change also unifies the naming of executor-related classes, in particular it renames "BackendConfig" to "ExecutorGroup". In anticipation of a subsequent change (IMPALA-8484), it adds maps to store multiple executor groups. This change also disables the generation of default operators from the thrift files and instead adds explicit implementations for the ones that we rely on. This forces us to explicitly specify comparators when manipulating containers of thrift structs and will help prevent accidental bugs. Testing: This change adds a backend unit test for the new cluster membership manager. The observable behavior of Impala does not change, and the existing scheduler unit test and end to end tests should make sure of that. Change-Id: Ib3cf9a8bb060d0c6e9ec8868b7b21ce01f8740a3 --- M be/src/benchmarks/scheduler-benchmark.cc M be/src/common/logging.h M be/src/common/thread-debug-info-test.cc M be/src/experiments/hash-ring-util.cc M be/src/gutil/strings/split.cc M be/src/gutil/strings/split.h M be/src/rpc/thrift-util-test.cc M be/src/rpc/thrift-util.cc M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/runtime/krpc-data-stream-mgr.cc M be/src/runtime/query-state.h M be/src/scheduling/CMakeLists.txt D be/src/scheduling/backend-config-test.cc D be/src/scheduling/backend-config.cc D be/src/scheduling/backend-config.h A be/src/scheduling/cluster-membership-mgr-test.cc A be/src/scheduling/cluster-membership-mgr.cc A be/src/scheduling/cluster-membership-mgr.h A be/src/scheduling/cluster-membership-test-util.cc A be/src/scheduling/cluster-membership-test-util.h A be/src/scheduling/executor-group-test.cc A be/src/scheduling/executor-group.cc A be/src/scheduling/executor-group.h M be/src/scheduling/query-schedule.cc M be/src/scheduling/query-schedule.h M be/src/scheduling/scheduler-test-util.cc M be/src/scheduling/scheduler-test-util.h M be/src/scheduling/scheduler-test.cc M be/src/scheduling/scheduler.cc M be/src/scheduling/scheduler.h M be/src/service/impala-http-handler.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/statestore/statestore-subscriber.cc M be/src/statestore/statestore-subscriber.h M be/src/statestore/statestore.cc M be/src/testutil/in-process-servers.cc M be/src/util/container-util.h M be/src/util/network-util.cc M be/src/util/network-util.h M be/src/util/runtime-profile-test.cc M be/src/util/uid-util-test.cc M common/thrift/CMakeLists.txt M common/thrift/Frontend.thrift M common/thrift/StatestoreService.thrift M fe/src/main/java/org/apache/impala/service/Frontend.java M tests/custom_cluster/test_coordinators.py 48 files changed, 1,920 insertions(+), 1,012 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/13207/16 -- To view, visit http://gerrit.cloudera.org:8080/13207 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib3cf9a8bb060d0c6e9ec8868b7b21ce01f8740a3 Gerrit-Change-Number: 13207 Gerrit-PatchSet: 16 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7322: Add storage wait time to profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12940 ) Change subject: IMPALA-7322: Add storage wait time to profile .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3448/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12940 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6dde7e394b7c1c396d835ef6aa0a55930c0a8660 Gerrit-Change-Number: 12940 Gerrit-PatchSet: 10 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Thu, 30 May 2019 21:14:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8460: Simplify cluster membership management
Hello Michael Ho, Thomas Marshall, Tim Armstrong, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13207 to look at the new patch set (#17). Change subject: IMPALA-8460: Simplify cluster membership management .. IMPALA-8460: Simplify cluster membership management This change adds a class to track cluster membership called ClusterMembershipMgr. It replaces the logic that was partially duplicated between the ImpalaServer and the Coordinator and makes sure that the local backend descriptor is consistent (IMPALA-8469). The ClusterMembershipMgr maintains a view of the cluster membership and incorporates incoming updates from the statestore. It also registers the local backend with the statestore after startup. Clients can obtain a consistent, immutable snapshot of the current cluster membership from the ClusterMembershipMgr. Additionally, callbacks can be registered to receive notifications of cluster membership changes. The ImpalaServer and Frontend use this mechanism. This change also unifies the naming of executor-related classes, in particular it renames "BackendConfig" to "ExecutorGroup". In anticipation of a subsequent change (IMPALA-8484), it adds maps to store multiple executor groups. This change also disables the generation of default operators from the thrift files and instead adds explicit implementations for the ones that we rely on. This forces us to explicitly specify comparators when manipulating containers of thrift structs and will help prevent accidental bugs. Testing: This change adds a backend unit test for the new cluster membership manager. The observable behavior of Impala does not change, and the existing scheduler unit test and end to end tests should make sure of that. Change-Id: Ib3cf9a8bb060d0c6e9ec8868b7b21ce01f8740a3 --- M be/src/benchmarks/scheduler-benchmark.cc M be/src/common/logging.h M be/src/common/thread-debug-info-test.cc M be/src/experiments/hash-ring-util.cc M be/src/gutil/strings/split.cc M be/src/gutil/strings/split.h M be/src/rpc/thrift-util-test.cc M be/src/rpc/thrift-util.cc M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/runtime/krpc-data-stream-mgr.cc M be/src/runtime/query-state.h M be/src/scheduling/CMakeLists.txt D be/src/scheduling/backend-config-test.cc D be/src/scheduling/backend-config.cc D be/src/scheduling/backend-config.h A be/src/scheduling/cluster-membership-mgr-test.cc A be/src/scheduling/cluster-membership-mgr.cc A be/src/scheduling/cluster-membership-mgr.h A be/src/scheduling/cluster-membership-test-util.cc A be/src/scheduling/cluster-membership-test-util.h A be/src/scheduling/executor-group-test.cc A be/src/scheduling/executor-group.cc A be/src/scheduling/executor-group.h M be/src/scheduling/query-schedule.cc M be/src/scheduling/query-schedule.h M be/src/scheduling/scheduler-test-util.cc M be/src/scheduling/scheduler-test-util.h M be/src/scheduling/scheduler-test.cc M be/src/scheduling/scheduler.cc M be/src/scheduling/scheduler.h M be/src/service/impala-http-handler.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/statestore/statestore-subscriber.cc M be/src/statestore/statestore-subscriber.h M be/src/statestore/statestore.cc M be/src/testutil/in-process-servers.cc M be/src/util/container-util.h M be/src/util/network-util.cc M be/src/util/network-util.h M be/src/util/runtime-profile-test.cc M be/src/util/uid-util-test.cc M common/thrift/CMakeLists.txt M common/thrift/Frontend.thrift M common/thrift/StatestoreService.thrift M fe/src/main/java/org/apache/impala/service/Frontend.java M tests/custom_cluster/test_coordinators.py M tests/custom_cluster/test_restart_services.py 49 files changed, 1,974 insertions(+), 1,024 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/13207/17 -- To view, visit http://gerrit.cloudera.org:8080/13207 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib3cf9a8bb060d0c6e9ec8868b7b21ce01f8740a3 Gerrit-Change-Number: 13207 Gerrit-PatchSet: 17 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8460: Simplify cluster membership management
Lars Volker has posted comments on this change. (
http://gerrit.cloudera.org:8080/13207 )
Change subject: IMPALA-8460: Simplify cluster membership management
..
Patch Set 17:
(1 comment)
PS 15 and 16 are the rebase, PS 17 has additional changes to address
IMPALA-7665.
http://gerrit.cloudera.org:8080/#/c/13207/8/be/src/scheduling/cluster-membership-mgr.h
File be/src/scheduling/cluster-membership-mgr.h:
http://gerrit.cloudera.org:8080/#/c/13207/8/be/src/scheduling/cluster-membership-mgr.h@67
PS8, Line 67: class ClusterMembershipMgr {
> BeDescSharedPtr sounds more natural to me.
Done
--
To view, visit http://gerrit.cloudera.org:8080/13207
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib3cf9a8bb060d0c6e9ec8868b7b21ce01f8740a3
Gerrit-Change-Number: 13207
Gerrit-PatchSet: 17
Gerrit-Owner: Lars Volker
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Thomas Marshall
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Thu, 30 May 2019 21:16:25 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8595: Support TLSv1.2 with Python < 2.7.9 in shell
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/13457 )
Change subject: IMPALA-8595: Support TLSv1.2 with Python < 2.7.9 in shell
..
Patch Set 5:
(2 comments)
It took me a while to understand why the fix worked, but now that I do I think
we can simplify it a bit.
http://gerrit.cloudera.org:8080/#/c/13457/5/shell/TSSLSocketWithWildcardSAN.py
File shell/TSSLSocketWithWildcardSAN.py:
http://gerrit.cloudera.org:8080/#/c/13457/5/shell/TSSLSocketWithWildcardSAN.py@52
PS5, Line 52: # THRIFT-3505 changes transport/TSSLSocket.py. TSSLSocket
ignores self.SSL_VERSION.
This is inaccurate - after THRIFT-3505, SSL_VERSION still takes effect. The
problem was be that __init__ was called *before* self.SSL_VERSION was set.
So I think we should remove the self.SSL_VERSION assignment and just pass in
ssl_version - we don't need to do both. Setting self.SSL_VERSION directly is
deprecated anyway.
http://gerrit.cloudera.org:8080/#/c/13457/5/tests/custom_cluster/test_client_ssl.py
File tests/custom_cluster/test_client_ssl.py:
http://gerrit.cloudera.org:8080/#/c/13457/5/tests/custom_cluster/test_client_ssl.py@38
PS5, Line 38: if dist.find('centos') or dist.find('red hat'):
Can you put this in tests/common/environ.py, e.g. with a name is
IS_REDHAT_DERIVATIVE?
It's easier to keep track of this kind of version sniffing code if it's all in
one place instead of distributed through the codebase.
--
To view, visit http://gerrit.cloudera.org:8080/13457
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3fb6510f4b556bd8c6b1e86380379aba8be4b805
Gerrit-Change-Number: 13457
Gerrit-PatchSet: 5
Gerrit-Owner: Robbie Zhang
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Thu, 30 May 2019 21:24:40 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8460: Simplify cluster membership management
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13207 ) Change subject: IMPALA-8460: Simplify cluster membership management .. Patch Set 16: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3449/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13207 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib3cf9a8bb060d0c6e9ec8868b7b21ce01f8740a3 Gerrit-Change-Number: 13207 Gerrit-PatchSet: 16 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 30 May 2019 21:32:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8460: Simplify cluster membership management
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13207 ) Change subject: IMPALA-8460: Simplify cluster membership management .. Patch Set 17: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3450/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13207 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib3cf9a8bb060d0c6e9ec8868b7b21ce01f8740a3 Gerrit-Change-Number: 13207 Gerrit-PatchSet: 17 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 30 May 2019 21:36:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8600 : Reload partition does not work for transactional tables
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13467 ) Change subject: IMPALA-8600 : Reload partition does not work for transactional tables .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/13467/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/13467/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2509 PS1, Line 2509: hdfsTable.loadValidWriteIdList(msClient.getHiveClient()); If the insert event was for a dynamic insert that touched a bunch of partitions, are we going to end up reloading the write id list N times instead of just once? -- To view, visit http://gerrit.cloudera.org:8080/13467 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8fffad27cd54f722efba1758ff44a45468f9514a Gerrit-Change-Number: 13467 Gerrit-PatchSet: 1 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 30 May 2019 21:36:29 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8460: Simplify cluster membership management
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13207 ) Change subject: IMPALA-8460: Simplify cluster membership management .. Patch Set 17: Code-Review+1 (2 comments) http://gerrit.cloudera.org:8080/#/c/13207/17/be/src/scheduling/cluster-membership-mgr.h File be/src/scheduling/cluster-membership-mgr.h: http://gerrit.cloudera.org:8080/#/c/13207/17/be/src/scheduling/cluster-membership-mgr.h@210 PS17, Line 210: /// post-recovery grace period. Not exposed to clients and not protected by any locking. This is only accessed by UpdateMembership(),right? I.e. "protected" by update_membership_lock_ http://gerrit.cloudera.org:8080/#/c/13207/17/be/src/scheduling/cluster-membership-mgr.cc File be/src/scheduling/cluster-membership-mgr.cc: http://gerrit.cloudera.org:8080/#/c/13207/17/be/src/scheduling/cluster-membership-mgr.cc@272 PS17, Line 272: if (update_local_server) NotifyLocalServerForDeletedBackend(*new_backend_map); I guess this means we call NotifyLocalServerForDeletedBackend() in more cases (i.e. no backend was necessarily removed). I think this is ok. -- To view, visit http://gerrit.cloudera.org:8080/13207 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib3cf9a8bb060d0c6e9ec8868b7b21ce01f8740a3 Gerrit-Change-Number: 13207 Gerrit-PatchSet: 17 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 30 May 2019 21:46:16 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8502: Bump CDH BUILD NUMBER and Kudu version
Hao Hao has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13481 Change subject: IMPALA-8502: Bump CDH_BUILD_NUMBER and Kudu version .. IMPALA-8502: Bump CDH_BUILD_NUMBER and Kudu version This bumps CDH_BUILD_NUMBER to 1137441 to bring in kudu-hive.jar which is recently added to impala-minicluster-tarballs for testing Kudu tables with the Hive Metastore integration enabled. It also brings in the latest Kudu side changes which adjust external Kudu table handling. Change-Id: I4c9c7d264f08454d9da8ad2d06ead643214ee23e --- M bin/impala-config.sh 1 file changed, 3 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/81/13481/1 -- To view, visit http://gerrit.cloudera.org:8080/13481 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I4c9c7d264f08454d9da8ad2d06ead643214ee23e Gerrit-Change-Number: 13481 Gerrit-PatchSet: 1 Gerrit-Owner: Hao Hao
[Impala-ASF-CR] IMPALA-8333: Remove Impala Shell warnings part 2
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13404 ) Change subject: IMPALA-8333: Remove Impala Shell warnings part 2 .. IMPALA-8333: Remove Impala Shell warnings part 2 Set IMPALA_TOOLCHAIN_BUILD_ID=40-193a30b3af to pickup "Patch Thrift to 0.9.3-p6 to eliminate ssl warnings" Set IMPALA_THRIFT_VERSION=0.9.3-p6 to pick up the new thrift build, which removes an unnecessary and confusing warning. TESTING Change the tests in test_client_ssl.py which were looking for specific deprecation warnings to instead search for any Deprecation Warning. Ran all end-to-end tests with new toolchain. Built Impala on all supported platforms. Change-Id: I8ae7e068894da5981fc083e690051da268bfde4d Reviewed-on: http://gerrit.cloudera.org:8080/13404 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M bin/impala-config.sh M tests/custom_cluster/test_client_ssl.py 2 files changed, 6 insertions(+), 11 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13404 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I8ae7e068894da5981fc083e690051da268bfde4d Gerrit-Change-Number: 13404 Gerrit-PatchSet: 5 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR] IMPALA-8333: Remove Impala Shell warnings part 2
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13404 ) Change subject: IMPALA-8333: Remove Impala Shell warnings part 2 .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13404 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8ae7e068894da5981fc083e690051da268bfde4d Gerrit-Change-Number: 13404 Gerrit-PatchSet: 4 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Thu, 30 May 2019 22:34:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8504: Support CREATE TABLE statement with Kudu/HMS integration
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13318 ) Change subject: IMPALA-8504: Support CREATE TABLE statement with Kudu/HMS integration .. Patch Set 11: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4350/ -- To view, visit http://gerrit.cloudera.org:8080/13318 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I465673d749221bd5f3772814b1c22c2673a53f5c Gerrit-Change-Number: 13318 Gerrit-PatchSet: 11 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Mike Percy Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Thu, 30 May 2019 22:51:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8502: Bump CDH BUILD NUMBER and Kudu version
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13481 ) Change subject: IMPALA-8502: Bump CDH_BUILD_NUMBER and Kudu version .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3451/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13481 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c9c7d264f08454d9da8ad2d06ead643214ee23e Gerrit-Change-Number: 13481 Gerrit-PatchSet: 1 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Thu, 30 May 2019 23:08:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8502: Bump CDH BUILD NUMBER and Kudu version
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/13481 ) Change subject: IMPALA-8502: Bump CDH_BUILD_NUMBER and Kudu version .. Patch Set 1: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/13481 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c9c7d264f08454d9da8ad2d06ead643214ee23e Gerrit-Change-Number: 13481 Gerrit-PatchSet: 1 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Thu, 30 May 2019 23:23:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8502: Bump CDH BUILD NUMBER and Kudu version
Hello Thomas Marshall, Alexey Serbin, Grant Henke, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13481 to look at the new patch set (#2). Change subject: IMPALA-8502: Bump CDH_BUILD_NUMBER and Kudu version .. IMPALA-8502: Bump CDH_BUILD_NUMBER and Kudu version This bumps CDH_BUILD_NUMBER to 1137441 to bring in kudu-hive.jar which is recently added to impala-minicluster-tarballs for testing Kudu tables with the Hive Metastore integration enabled. It also brings in the latest Kudu side changes which adjust external Kudu table handling. Change-Id: I4c9c7d264f08454d9da8ad2d06ead643214ee23e --- M bin/impala-config.sh 1 file changed, 3 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/81/13481/2 -- To view, visit http://gerrit.cloudera.org:8080/13481 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4c9c7d264f08454d9da8ad2d06ead643214ee23e Gerrit-Change-Number: 13481 Gerrit-PatchSet: 2 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR](2.x) IMPALA-6881: COMPUTE STATS should require SELECT privilege at analysis
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13473 ) Change subject: IMPALA-6881: COMPUTE STATS should require SELECT privilege at analysis .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4357/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: Icead43e689404a286859344e309427eb6c68646a Gerrit-Change-Number: 13473 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 30 May 2019 23:31:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8502: Bump CDH BUILD NUMBER and Kudu version
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/13481 ) Change subject: IMPALA-8502: Bump CDH_BUILD_NUMBER and Kudu version .. Patch Set 2: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/13481 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c9c7d264f08454d9da8ad2d06ead643214ee23e Gerrit-Change-Number: 13481 Gerrit-PatchSet: 2 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Thu, 30 May 2019 23:40:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8502: Bump CDH BUILD NUMBER and Kudu version
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/13481 ) Change subject: IMPALA-8502: Bump CDH_BUILD_NUMBER and Kudu version .. Patch Set 2: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/13481 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c9c7d264f08454d9da8ad2d06ead643214ee23e Gerrit-Change-Number: 13481 Gerrit-PatchSet: 2 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Thu, 30 May 2019 23:50:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8460: Simplify cluster membership management
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/13207 ) Change subject: IMPALA-8460: Simplify cluster membership management .. Patch Set 17: Code-Review+2 (3 comments) http://gerrit.cloudera.org:8080/#/c/13207/17/be/src/scheduling/cluster-membership-mgr.h File be/src/scheduling/cluster-membership-mgr.h: http://gerrit.cloudera.org:8080/#/c/13207/17/be/src/scheduling/cluster-membership-mgr.h@210 PS17, Line 210: /// post-recovery grace period. Not exposed to clients and not protected by any locking. > This is only accessed by UpdateMembership(),right? I.e. "protected" by upda I guess it kinda is because of the way it is used, but I would still try to avoid mentioning that it "is protected" by the lock, because that might imply that it can be modified concurrently. http://gerrit.cloudera.org:8080/#/c/13207/17/be/src/scheduling/cluster-membership-mgr.cc File be/src/scheduling/cluster-membership-mgr.cc: http://gerrit.cloudera.org:8080/#/c/13207/17/be/src/scheduling/cluster-membership-mgr.cc@86 PS17, Line 86: UpdateMembership nit: not a huge deal but I feel this method has become too long, is there anyway we can decompose it? http://gerrit.cloudera.org:8080/#/c/13207/17/be/src/statestore/statestore-subscriber.cc File be/src/statestore/statestore-subscriber.cc: http://gerrit.cloudera.org:8080/#/c/13207/17/be/src/statestore/statestore-subscriber.cc@467 PS17, Line 467: MilliSecondsSinceLastRegistration nit: I think we dont need to expose this method anymore. we can just get rid of it altogether or just make it private. -- To view, visit http://gerrit.cloudera.org:8080/13207 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib3cf9a8bb060d0c6e9ec8868b7b21ce01f8740a3 Gerrit-Change-Number: 13207 Gerrit-PatchSet: 17 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 31 May 2019 00:07:55 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8502: Bump CDH BUILD NUMBER and Kudu version
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13481 ) Change subject: IMPALA-8502: Bump CDH_BUILD_NUMBER and Kudu version .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3452/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13481 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c9c7d264f08454d9da8ad2d06ead643214ee23e Gerrit-Change-Number: 13481 Gerrit-PatchSet: 2 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Fri, 31 May 2019 00:07:44 + Gerrit-HasComments: No
