[Impala-ASF-CR] IMPALA-8072: remove junk configs from containers
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13104 ) Change subject: IMPALA-8072: remove junk configs from containers .. IMPALA-8072: remove junk configs from containers The docker containers currently have minicluster configs baked into them. This is not necessary any more since the /opt/impala/conf directory is mounted to point at the up-to-date configs, so there's no reason to include configs in the container. Testing: Confirmed that I could build containers, start up a minicluster and run queries. Change-Id: I6d77f79620514187a5c45483e9051bd8c40dfc9e Reviewed-on: http://gerrit.cloudera.org:8080/13104 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M docker/impala_base/Dockerfile M docker/setup_build_context.py 2 files changed, 3 insertions(+), 5 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I6d77f79620514187a5c45483e9051bd8c40dfc9e Gerrit-Change-Number: 13104 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-8072: remove junk configs from containers
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13104 ) Change subject: IMPALA-8072: remove junk configs from containers .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6d77f79620514187a5c45483e9051bd8c40dfc9e Gerrit-Change-Number: 13104 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Thu, 25 Apr 2019 06:59:06 + Gerrit-HasComments: No
[Impala-ASF-CR] Configure Hive 3's HS2 to execute queries using Tez local mode
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12931 ) Change subject: Configure Hive 3's HS2 to execute queries using Tez local mode .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2904/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12931 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I76e47fbd1d6ff5103d81a8de430d5465dba284cd Gerrit-Change-Number: 12931 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 25 Apr 2019 06:48:28 + Gerrit-HasComments: No
[Impala-ASF-CR] Verify existence of HIVE AUX JARS before running HS2
Hello Vihang Karajgaonkar, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/13113 to review the following change. Change subject: Verify existence of HIVE_AUX_JARS before running HS2 .. Verify existence of HIVE_AUX_JARS before running HS2 I spent an hour today trying to debug the following issue: - we pass $POSTGRES_JDBC_DRIVER as $HIVE_AUX_JARS_PATH when starting HS2 - the path to the JDBC driver is inside the fe/target/dependency/ directory - if we haven't built the frontend yet, that jar won't exist yet - at least on HS2 from Hive 3 (maybe on Hive 2 as well?) if you pass a non-existent jar in this environment variable, it will fail with a fairly uninintelligible error like: tez.DagUtils: Localizing resource because it does not exist: file:/data/1/todd/impala/fe/target/dependency/postgresql-42.2.5.jar to dest: hdfs://localhost:20500/tmp/hive/todd/_tez_session_dir/9de357d5-59bf-4faa-8973-5212a08bc41a-resources/postgresql-42.2.5.jar tez.DagUtils: Looks like another thread or process is writing the same file tez.DagUtils: Waiting for the file hdfs://localhost:20500/tmp/hive/todd/_tez_session_dir/9de357d5-59bf-4faa-8973-5212a08bc41a-resources/postgresql-42.2.5.jar (5 attempts, with 5000ms interval) tez.DagUtils: Could not find the jar that was being uploaded tez.TezTask: Failed to execute tez graph.``` I filed HIVE-21649 to improve this logging. Change-Id: I3a2458b9e92a9243fc00d1bd80791062c89496f7 --- M testdata/bin/run-hive-server.sh 1 file changed, 11 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/13113/1 -- To view, visit http://gerrit.cloudera.org:8080/13113 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I3a2458b9e92a9243fc00d1bd80791062c89496f7 Gerrit-Change-Number: 13113 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] Configure Hive 3's HS2 to execute queries using Tez local mode
Hello Vihang Karajgaonkar, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12931 to look at the new patch set (#5). Change subject: Configure Hive 3's HS2 to execute queries using Tez local mode .. Configure Hive 3's HS2 to execute queries using Tez local mode Hive 3 no longer supports MR execution, so this sets up the appropriate configuration and classpath so that HS2 can run queries using Tez. The bulk of this patch is toolchain changes to download Tez itself. The Tez tarball is slightly odd in that it has no top-level directory, so the patch changes around bootstrap_toolchain a bit to support creating its own top-level directory for a component. The remainder of the patch is some classpath setup and hive-site changes when Hive 3 is enabled. So far I tested this manually by setting up a metastore and impala-config with USE_CDP_HIVE=true, and then connecting to HS2 using hive beeline -u 'jdbc:hive2://localhost:11050' I was able to insert and query data, and was able to verify that queries like 'select count(*)' were executing via Tez local mode. NOTE: this patch relies on a custom build of Tez, based on a private branch. I've submitted a PR to Tez upstream, referenced in the commits here. Will remove this hack once the PR is accepted and makes its way into an official build. Change-Id: I76e47fbd1d6ff5103d81a8de430d5465dba284cd --- M bin/bootstrap_toolchain.py M bin/impala-config.sh M fe/pom.xml M fe/src/test/resources/hive-site.xml.py M testdata/bin/run-hive-server.sh 5 files changed, 90 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/12931/5 -- To view, visit http://gerrit.cloudera.org:8080/12931 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I76e47fbd1d6ff5103d81a8de430d5465dba284cd Gerrit-Change-Number: 12931 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained updates at partition level.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained updates at partition level. .. Patch Set 2: (11 comments) http://gerrit.cloudera.org:8080/#/c/13111/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13111/2//COMMIT_MSG@7 PS2, Line 7: IMPALA-7973: Add support for fine grained updates at partition level. nit: mention about "events" in the title? Otherwise it looks very general . http://gerrit.cloudera.org:8080/#/c/13111/2//COMMIT_MSG@13 PS2, Line 13: nit: ..affected.. http://gerrit.cloudera.org:8080/#/c/13111/2//COMMIT_MSG@14 PS2, Line 14: HMS processes add/drop partitions : in a transaction Not sure I understand this, can you please clarify? http://gerrit.cloudera.org:8080/#/c/13111/2//COMMIT_MSG@19 PS2, Line 19: insead nit:typo http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2085 PS2, Line 2085: nit: ..it.. http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2085 PS2, Line 2085: Returns true if reloadPartitition() succeeds, false :* otherwise. Instead of referring to specific methods, rephrase it to something like returns true if the reload of the partition is successful, false otherwise http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2089 PS2, Line 2089: refresh nit:call "reload" to be consistent ? http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java: http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1102 PS2, Line 1102: // HMS adds partitions in a transactional way. We try to refresh all the : // partitions that were added to HMS. If any partition refresh fails, we throw : // MetastoreNotificationNeedsInvalidateException exception. We skip refresh of the : // partitions if the table is not present in the catalog. nit: Not super clear what this is. How are HMS transactional semantics related to what we are doing here? http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1118 PS2, Line 1118: ); nit: include the partition name too http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1125 PS2, Line 1125: ; include the partition name http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1295 PS2, Line 1295: + "event.", getFullyQualifiedTblName()); There seem to be a lot of common code between the partition event processors. Can we consolidate it by having some helpers that factor out the common code and exception handling? -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 2 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 25 Apr 2019 04:43:20 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8149 : Add support for alter database events
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13049 ) Change subject: IMPALA-8149 : Add support for alter_database events .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2903/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13049 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaf020e85cae04163bf32e31363eb4119d624640b Gerrit-Change-Number: 13049 Gerrit-PatchSet: 4 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 25 Apr 2019 04:37:05 + Gerrit-HasComments: No
[Impala-ASF-CR] Remove references to the $IMPALA HOME/thirdparty directory
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13092 ) Change subject: Remove references to the $IMPALA_HOME/thirdparty directory .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13092 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2edfd499febb5a25fdcf59b5183eccf192a08be0 Gerrit-Change-Number: 13092 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Thu, 25 Apr 2019 04:24:22 + Gerrit-HasComments: No
[Impala-ASF-CR] Make infra/python compatible with both Python 2 & 3
Akshesh Doshi has posted comments on this change. ( http://gerrit.cloudera.org:8080/13070 ) Change subject: Make infra/python compatible with both Python 2 & 3 .. Patch Set 2: Tried pushing again with different network and checked the ssh keys set up on Gerrit, but no luck this time either. If anyone wants to cherry-pick the commit, they can find it at https://github.com/akki/impala/pull/1/files -- To view, visit http://gerrit.cloudera.org:8080/13070 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If4285a021bb581f88425daa52ef8a3f844017d82 Gerrit-Change-Number: 13070 Gerrit-PatchSet: 2 Gerrit-Owner: Akshesh Doshi Gerrit-Reviewer: Akshesh Doshi Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 25 Apr 2019 04:29:04 + Gerrit-HasComments: No
[Impala-ASF-CR] Remove references to the $IMPALA HOME/thirdparty directory
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13092 ) Change subject: Remove references to the $IMPALA_HOME/thirdparty directory .. Remove references to the $IMPALA_HOME/thirdparty directory The $IMPALA_HOME/thirdparty directory is a remnant from before Impala was an Apache project. It is obsolete and unused, so this removes code that references this directory. Testing: - Ran core tests Change-Id: I2edfd499febb5a25fdcf59b5183eccf192a08be0 Reviewed-on: http://gerrit.cloudera.org:8080/13092 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M README.md M bin/impala-config.sh M cmake_modules/FindAvro.cmake M cmake_modules/FindGFlags.cmake M cmake_modules/FindGLog.cmake M cmake_modules/FindGTest.cmake M cmake_modules/FindLdap.cmake M cmake_modules/FindLz4.cmake M cmake_modules/FindOrc.cmake M cmake_modules/FindPProf.cmake M cmake_modules/FindRapidJson.cmake M cmake_modules/FindRe2.cmake M cmake_modules/FindSasl.cmake M cmake_modules/FindSnappy.cmake 14 files changed, 27 insertions(+), 95 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13092 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I2edfd499febb5a25fdcf59b5183eccf192a08be0 Gerrit-Change-Number: 13092 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-8149 : Add support for alter database events
xiaom...@cloudera.com has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/13049 ) Change subject: IMPALA-8149 : Add support for alter_database events .. IMPALA-8149 : Add support for alter_database events This change adds support for alter_database events in two parts: One is adding catalogServiceId and catalogVersion in db parameters when alter database. The other is adding alter database event, check if it's self event during process, if true do nothing, if false replace caralog cached db with event db. Testing: Enabled testAlterDisableFlagFromDb in MetastoreEventsProcessorTest. Change-Id: Iaf020e85cae04163bf32e31363eb4119d624640b --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 5 files changed, 261 insertions(+), 108 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/13049/4 -- To view, visit http://gerrit.cloudera.org:8080/13049 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iaf020e85cae04163bf32e31363eb4119d624640b Gerrit-Change-Number: 13049 Gerrit-PatchSet: 4 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. Patch Set 17: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4061/ -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 17 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Thu, 25 Apr 2019 02:57:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained updates at partition level.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained updates at partition level. .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2902/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 2 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 25 Apr 2019 02:40:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained updates at partition level.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained updates at partition level. .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2901/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 1 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 25 Apr 2019 02:25:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained updates at partition level.
Vihang Karajgaonkar has posted comments on this change. (
http://gerrit.cloudera.org:8080/13111 )
Change subject: IMPALA-7973: Add support for fine grained updates at partition
level.
..
Patch Set 2:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1106
PS2, Line 1106: addedPartitions_
Would be good if you can log a useful information about the event like number
of partitions, isReplace, tablename etc.
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1107
PS2, Line 1107: Map partSpec = new HashMap<>();
:
List fsList =
: msTbl_.getPartitionKeys();
: List partVals = partition.getValues();
: Preconditions.checkNotNull(partVals);
: Preconditions.checkState(fsList.size() ==
partVals.size());
: for (int i = 0; i < fsList.size(); i++) {
: partSpec.put(fsList.get(i).getName(),
partVals.get(i));
: }
We can avoid one unnecessary conversion from Partition -> partSpecMap ->
TPartitionKeyValue if we change the signature of the refreshPartitionIfExists
to take List here.
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1194
PS2, Line 1194: Preconditions.checkNotNull(partitionAfter_);
: Map partSpec = new HashMap<>();
: List
fsList =
: msTbl_.getPartitionKeys();
: List partVals = partitionAfter_.getValues();
: Preconditions.checkNotNull(partVals);
: Preconditions.checkState(fsList.size() ==
partVals.size());
: for (int i = 0; i < fsList.size(); i++) {
: partSpec.put(fsList.get(i).getName(), partVals.get(i));
: }
same suggestion as above. Also, instead of duplicating this logic, create a
util method in the base class (or a static method)
http://gerrit.cloudera.org:8080/#/c/13111/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1266
PS2, Line 1266: droppedPartitions_
add a null check as well
--
To view, visit http://gerrit.cloudera.org:8080/13111
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5
Gerrit-Change-Number: 13111
Gerrit-PatchSet: 2
Gerrit-Owner: Anurag Mantripragada
Gerrit-Reviewer: Bharath Krishna
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Thu, 25 Apr 2019 02:14:20 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained updates at partition level.
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/13111 )
Change subject: IMPALA-7973: Add support for fine grained updates at partition
level.
..
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13111/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/13111/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1286
PS1, Line 1286: debugLog("Could not refresh partitions of table {}
after a drop_partition event"
line too long (92 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/13111
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5
Gerrit-Change-Number: 13111
Gerrit-PatchSet: 1
Gerrit-Owner: Anurag Mantripragada
Gerrit-Reviewer: Bharath Krishna
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Thu, 25 Apr 2019 01:39:10 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained updates at partition level.
Anurag Mantripragada has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/13111 ) Change subject: IMPALA-7973: Add support for fine grained updates at partition level. .. IMPALA-7973: Add support for fine grained updates at partition level. This patch adds suport for fine grained updates for add/drop/alter partition events. Currently, partition events invalidate the table. This can be very expensive for large tables. Here, we refresh partitions in case of add/drop/alter partition events. HMS processes add/drop partitions in a transaction. We throw MetastoreNotificationNeedsInvalidateException if any of the partition refreshes fails. Testing: Modified pre-existing tests for partition events to insead test if partitions are added/dropped/altered when event processing is enabled. Change-Id: I213401329f3965dd81055197792ccf8a05368af5 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 3 files changed, 183 insertions(+), 44 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/13111/2 -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 2 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8072: remove junk configs from containers
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13104 ) Change subject: IMPALA-8072: remove junk configs from containers .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4063/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6d77f79620514187a5c45483e9051bd8c40dfc9e Gerrit-Change-Number: 13104 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Thu, 25 Apr 2019 01:42:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8072: remove junk configs from containers
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13104 ) Change subject: IMPALA-8072: remove junk configs from containers .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6d77f79620514187a5c45483e9051bd8c40dfc9e Gerrit-Change-Number: 13104 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Thu, 25 Apr 2019 01:42:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7973: Add support for fine grained updates at partition level.
Anurag Mantripragada has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13111 Change subject: IMPALA-7973: Add support for fine grained updates at partition level. .. IMPALA-7973: Add support for fine grained updates at partition level. This patch adds suport for fine grained updates for add/drop/alter partition events. Currently, partition events invalidate the table. This can be very expensive for large tables. Here, we refresh partitions in case of add/drop/alter partition events. HMS processes add/drop partitions in a transaction. We throw MetastoreNotificationNeedsInvalidateException if any of the partition refreshes fails. Testing: Modified pre-existing tests for partition events to insead test if partitions are added/dropped/altered when event processing is enabled. Change-Id: I213401329f3965dd81055197792ccf8a05368af5 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 3 files changed, 182 insertions(+), 44 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/13111/1 -- To view, visit http://gerrit.cloudera.org:8080/13111 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I213401329f3965dd81055197792ccf8a05368af5 Gerrit-Change-Number: 13111 Gerrit-PatchSet: 1 Gerrit-Owner: Anurag Mantripragada
[Impala-ASF-CR] IMPALA-1856: MetadataOp.getTypeInfo() does not return all supported types.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13094 ) Change subject: IMPALA-1856: MetadataOp.getTypeInfo() does not return all supported types. .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13094 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icdccde7c32e52ed1b0c7b13a22171e8bcd7f1f2d Gerrit-Change-Number: 13094 Gerrit-PatchSet: 5 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 25 Apr 2019 01:38:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-1856: MetadataOp.getTypeInfo() does not return all supported types.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13094 ) Change subject: IMPALA-1856: MetadataOp.getTypeInfo() does not return all supported types. .. IMPALA-1856: MetadataOp.getTypeInfo() does not return all supported types. The MetadataOp.getTypeInfo() is missing all the complex types (ARRAY, MAP, and STRUCT). Several of the primitive data types such as CHAR, VARCHAR, DECIMAL, DATE (newly added) are also not returned. The Impala JDBC client should in theory call MetadataOp.getTypeInfo() but that is not happening in the latest version of the driver. This change will only ensure that on Impala side the MetadataOp.getTypeInfo() returns correct results. Updated MetadataOp::createGetTypeInfoResults to include all supported and externally visible data types, including complex types (which were missing along with some other Primitive types). Added a new function ScalarType::isInternalType() to identify internal types such as NULL_TYPE, FIXED_UDA_INTERMEDIATE which are not exposed through getTypeInfo function or in any other manner through SQL. Testing: There was a testing gap and ideally whenever a new type is added or support for a type is changed the MetadataOp.getTypeInfo() should return the correct result set representing the supported types. - Updated FrontendTest.java to ensure that the result set from MetadataOp::getTypeInfo contains all supported and externally visible types - Added new E2E test (test_get_type_info) in tests/hs2/test_hs2.py. The new test validates that the HS2 GetTypeInfo() RPC returns supported and externally visible types. Change-Id: Icdccde7c32e52ed1b0c7b13a22171e8bcd7f1f2d Reviewed-on: http://gerrit.cloudera.org:8080/13094 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/catalog/ScalarType.java M fe/src/main/java/org/apache/impala/catalog/Type.java M fe/src/main/java/org/apache/impala/service/MetadataOp.java M fe/src/test/java/org/apache/impala/service/FrontendTest.java M tests/hs2/test_hs2.py 5 files changed, 145 insertions(+), 34 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13094 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Icdccde7c32e52ed1b0c7b13a22171e8bcd7f1f2d Gerrit-Change-Number: 13094 Gerrit-PatchSet: 6 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8446: Create a unit test for Admission Controller.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13078 ) Change subject: IMPALA-8446: Create a unit test for Admission Controller. .. Patch Set 4: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4059/ -- To view, visit http://gerrit.cloudera.org:8080/13078 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8a840590b868f2df1a06f3f397b7b0fc2b29462c Gerrit-Change-Number: 13078 Gerrit-PatchSet: 4 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 25 Apr 2019 01:24:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5351: Support storing column comment of kudu table
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12977 ) Change subject: IMPALA-5351: Support storing column comment of kudu table .. Patch Set 4: Overall it looks good. Can you add E2E tests in test_ddl.py? -- To view, visit http://gerrit.cloudera.org:8080/12977 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifb3b37eed364f12bdb3c1d7ef5be128f1475936c Gerrit-Change-Number: 12977 Gerrit-PatchSet: 4 Gerrit-Owner: helifu Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: helifu Gerrit-Comment-Date: Thu, 25 Apr 2019 01:08:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5351: Support storing column comment of kudu table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12977 ) Change subject: IMPALA-5351: Support storing column comment of kudu table .. Patch Set 4: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/2900/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/12977 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifb3b37eed364f12bdb3c1d7ef5be128f1475936c Gerrit-Change-Number: 12977 Gerrit-PatchSet: 4 Gerrit-Owner: helifu Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: helifu Gerrit-Comment-Date: Thu, 25 Apr 2019 00:54:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8072: remove junk configs from containers
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/13104 ) Change subject: IMPALA-8072: remove junk configs from containers .. Patch Set 2: Code-Review+2 Makes sense -- To view, visit http://gerrit.cloudera.org:8080/13104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6d77f79620514187a5c45483e9051bd8c40dfc9e Gerrit-Change-Number: 13104 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Thu, 25 Apr 2019 00:35:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5351: Support storing column comment of kudu table
helifu has posted comments on this change. (
http://gerrit.cloudera.org:8080/12977 )
Change subject: IMPALA-5351: Support storing column comment of kudu table
..
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/12977/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/12977/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3896
PS3, Line 3896: if
(!updateColumnComment(msTbl.getPartitionKeysIterator(), columnName, comment)) {
> line too long (92 > 90)
Done
--
To view, visit http://gerrit.cloudera.org:8080/12977
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifb3b37eed364f12bdb3c1d7ef5be128f1475936c
Gerrit-Change-Number: 12977
Gerrit-PatchSet: 3
Gerrit-Owner: helifu
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Thomas Marshall
Gerrit-Reviewer: helifu
Gerrit-Comment-Date: Thu, 25 Apr 2019 00:32:31 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5351: Support storing column comment of kudu table
Hello Thomas Marshall, Fredy Wijaya, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/12977
to look at the new patch set (#4).
Change subject: IMPALA-5351: Support storing column comment of kudu table
..
IMPALA-5351: Support storing column comment of kudu table
This patch intends to support storing column comment of kudu table
on impala side.
Belows tests passed:
1) creata kudu-table with column comment;
2) alter kudu-table with (add/alter[delete] column comment);
3) show create kudu table;
4) describe kudu-table;
5) invalidate metadata;
6) comment on column is { '' | null | 'comment' }
Change-Id: Ifb3b37eed364f12bdb3c1d7ef5be128f1475936c
---
M fe/src/main/java/org/apache/impala/analysis/AlterTableAlterColStmt.java
M fe/src/main/java/org/apache/impala/catalog/KuduColumn.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
5 files changed, 39 insertions(+), 21 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/77/12977/4
--
To view, visit http://gerrit.cloudera.org:8080/12977
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifb3b37eed364f12bdb3c1d7ef5be128f1475936c
Gerrit-Change-Number: 12977
Gerrit-PatchSet: 4
Gerrit-Owner: helifu
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Thomas Marshall
Gerrit-Reviewer: helifu
[Impala-ASF-CR] IMPALA-8295: Fix flakiness in TestAdmissionControllerStress
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13103 ) Change subject: IMPALA-8295: Fix flakiness in TestAdmissionControllerStress .. IMPALA-8295: Fix flakiness in TestAdmissionControllerStress Currently the test_mem_limit in TestAdmissionControllerStress encounters flaky failures due to a timeout waiting on queries to end. Increasing the timeout from 60 to 90 seconds to accommodate this flakiness. Testing: Ran the test in a loop for more than 15 hours without any failures. Change-Id: If9ada7bdbe2461d164100badabb905cf9a8cf4cc Reviewed-on: http://gerrit.cloudera.org:8080/13103 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M tests/custom_cluster/test_admission_controller.py 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13103 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: If9ada7bdbe2461d164100badabb905cf9a8cf4cc Gerrit-Change-Number: 13103 Gerrit-PatchSet: 4 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8295: Fix flakiness in TestAdmissionControllerStress
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13103 ) Change subject: IMPALA-8295: Fix flakiness in TestAdmissionControllerStress .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13103 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If9ada7bdbe2461d164100badabb905cf9a8cf4cc Gerrit-Change-Number: 13103 Gerrit-PatchSet: 3 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 25 Apr 2019 00:05:41 + Gerrit-HasComments: No
[Impala-ASF-CR] Remove references to the $IMPALA HOME/thirdparty directory
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/13092 ) Change subject: Remove references to the $IMPALA_HOME/thirdparty directory .. Patch Set 1: Test failure is a known flaky issue. -- To view, visit http://gerrit.cloudera.org:8080/13092 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2edfd499febb5a25fdcf59b5183eccf192a08be0 Gerrit-Change-Number: 13092 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 24 Apr 2019 23:09:31 + Gerrit-HasComments: No
[Impala-ASF-CR] Remove references to the $IMPALA HOME/thirdparty directory
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13092 ) Change subject: Remove references to the $IMPALA_HOME/thirdparty directory .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4062/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13092 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2edfd499febb5a25fdcf59b5183eccf192a08be0 Gerrit-Change-Number: 13092 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 24 Apr 2019 23:09:19 + Gerrit-HasComments: No
[Impala-ASF-CR] Remove references to the $IMPALA HOME/thirdparty directory
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13092 ) Change subject: Remove references to the $IMPALA_HOME/thirdparty directory .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13092 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2edfd499febb5a25fdcf59b5183eccf192a08be0 Gerrit-Change-Number: 13092 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 24 Apr 2019 23:09:18 + Gerrit-HasComments: No
[Impala-ASF-CR] Remove references to the $IMPALA HOME/thirdparty directory
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13092 ) Change subject: Remove references to the $IMPALA_HOME/thirdparty directory .. Patch Set 1: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4057/ -- To view, visit http://gerrit.cloudera.org:8080/13092 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2edfd499febb5a25fdcf59b5183eccf192a08be0 Gerrit-Change-Number: 13092 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 24 Apr 2019 23:04:49 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-8281: Add support for show grant user/group with Ranger
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/13074 )
Change subject: [WIP] IMPALA-8281: Add support for show grant user/group with
Ranger
..
Patch Set 6:
(8 comments)
First pass on this. Will take a look at it more.
http://gerrit.cloudera.org:8080/#/c/13074/6//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13074/6//COMMIT_MSG@11
PS6, Line 11: is called from impalad.
Mention the new syntax and the syntax that's not supported by Ranger, such as
the one without the on clause.
http://gerrit.cloudera.org:8080/#/c/13074/6/fe/src/main/cup/sql-parser.cup
File fe/src/main/cup/sql-parser.cup:
http://gerrit.cloudera.org:8080/#/c/13074/6/fe/src/main/cup/sql-parser.cup@968
PS6, Line 968: col_name.getTableName(),
: Collections.singletonList(col_name.getColumnName(
nit: indent more because it's part of PrivilegeSpec.createColumnScopedPriv
http://gerrit.cloudera.org:8080/#/c/13074/6/fe/src/main/java/org/apache/impala/analysis/ShowGrantPrincipalStmt.java
File fe/src/main/java/org/apache/impala/analysis/ShowGrantPrincipalStmt.java:
http://gerrit.cloudera.org:8080/#/c/13074/6/fe/src/main/java/org/apache/impala/analysis/ShowGrantPrincipalStmt.java@63
PS6, Line 63: principal_ =
analyzer.getCatalog().getAuthPolicy().getPrincipal(name_,
: principalType_);
: if (principal_ == null) throw new
AnalysisException(String.format("%s '%s' " +
: "does not exist.",
Principal.toString(principalType_), name_));
: break;
we should move this logic to Sentry specific implementation
http://gerrit.cloudera.org:8080/#/c/13074/6/fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java
File
fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java:
http://gerrit.cloudera.org:8080/#/c/13074/6/fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java@183
PS6, Line 183: return null;
shouldn't this throw an exception instead?
http://gerrit.cloudera.org:8080/#/c/13074/6/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java
File
fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java:
http://gerrit.cloudera.org:8080/#/c/13074/6/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java@191
PS6, Line 191: }
need default case or else the compiler may give a warning
http://gerrit.cloudera.org:8080/#/c/13074/6/fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java@354
PS6, Line 354: Unsupported principal type.
mention the unsupported type name
http://gerrit.cloudera.org:8080/#/c/13074/6/fe/src/main/java/org/apache/impala/catalog/Principal.java
File fe/src/main/java/org/apache/impala/catalog/Principal.java:
http://gerrit.cloudera.org:8080/#/c/13074/6/fe/src/main/java/org/apache/impala/catalog/Principal.java@183
PS6, Line 183: principal = "";
shouldn't this be an exception?
http://gerrit.cloudera.org:8080/#/c/13074/6/tests/authorization/test_ranger.py
File tests/authorization/test_ranger.py:
http://gerrit.cloudera.org:8080/#/c/13074/6/tests/authorization/test_ranger.py@87
PS6, Line 87: result = self.execute_query("show grant {0} {1} on
database {2}"
This is good that we can now replace with this with "show grant". However, we
need to write more exhaustive tests that does "show grant" in every single
scope. We should also test "show grant" without "on" is not supported in Ranger.
--
To view, visit http://gerrit.cloudera.org:8080/13074
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic46fb9fc36c9e11ec78d5840d22eb0668150c2a4
Gerrit-Change-Number: 13074
Gerrit-PatchSet: 6
Gerrit-Owner: Austin Nobis
Gerrit-Reviewer: Austin Nobis
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Wed, 24 Apr 2019 22:56:17 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8444: Fix performance regression when building privilege name
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13095 ) Change subject: IMPALA-8444: Fix performance regression when building privilege name .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2899/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13095 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I942d9b55f07c8972f69e532567d9b7d80fceb6e5 Gerrit-Change-Number: 13095 Gerrit-PatchSet: 7 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 24 Apr 2019 22:51:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8072: remove junk configs from containers
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13104 to look at the new patch set (#2). Change subject: IMPALA-8072: remove junk configs from containers .. IMPALA-8072: remove junk configs from containers The docker containers currently have minicluster configs baked into them. This is not necessary any more since the /opt/impala/conf directory is mounted to point at the up-to-date configs, so there's no reason to include configs in the container. Testing: Confirmed that I could build containers, start up a minicluster and run queries. Change-Id: I6d77f79620514187a5c45483e9051bd8c40dfc9e --- M docker/impala_base/Dockerfile M docker/setup_build_context.py 2 files changed, 3 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/13104/2 -- To view, visit http://gerrit.cloudera.org:8080/13104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6d77f79620514187a5c45483e9051bd8c40dfc9e Gerrit-Change-Number: 13104 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8444: Fix performance regression when building privilege name
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13095 ) Change subject: IMPALA-8444: Fix performance regression when building privilege name .. Patch Set 7: Code-Review+1 (1 comment) Carry Bharath's +1. http://gerrit.cloudera.org:8080/#/c/13095/6/fe/src/main/java/org/apache/impala/catalog/Principal.java File fe/src/main/java/org/apache/impala/catalog/Principal.java: http://gerrit.cloudera.org:8080/#/c/13095/6/fe/src/main/java/org/apache/impala/catalog/Principal.java@82 PS6, Line 82:*/ > Actually thinking again, we are at a risk of exposing the underlying keySet Done. -- To view, visit http://gerrit.cloudera.org:8080/13095 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I942d9b55f07c8972f69e532567d9b7d80fceb6e5 Gerrit-Change-Number: 13095 Gerrit-PatchSet: 7 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 24 Apr 2019 22:07:29 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8444: Fix performance regression when building privilege name
Fredy Wijaya has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/13095 ) Change subject: IMPALA-8444: Fix performance regression when building privilege name .. IMPALA-8444: Fix performance regression when building privilege name This patch fixes the performance regression when building privilege name by rewriting PrincipalPrivilege.buildPrivilegeName() with a simple string concatentation instead of using a list that gets converted into a string. Below is the result of running a benchmark using JMH comparing the old and new implementations: Result "org.apache.impala.BuildPrivilegeNameBenchmark.fast": 0.344 ±(99.9%) 0.004 us/op [Average] (min, avg, max) = (0.336, 0.344, 0.355), stdev = 0.005 CI (99.9%): [0.339, 0.348] (assumes normal distribution) Result "org.apache.impala.BuildPrivilegeNameBenchmark.slow": 0.831 ±(99.9%) 0.011 us/op [Average] (min, avg, max) = (0.807, 0.831, 0.856), stdev = 0.015 CI (99.9%): [0.820, 0.842] (assumes normal distribution) Benchmark Mode Cnt Score Error Units BuildPrivilegeNameBenchmark.fast avgt 25 0.344 ± 0.004 us/op BuildPrivilegeNameBenchmark.slow avgt 25 0.831 ± 0.011 us/op This patch also updates SentryAuthorizationPolicy.listPrivileges() to reuse the privilege names that have already been built instead of building them again. While fixing this, I found a bug where Principal stores the PrincipalPrivilege in case insensitive way. This is true for all privilege scopes, except URI. This patch fixes the issue by storing URI privilege in a case sensitive manner. This patch removes incorrect synchronization in SentryAuthorizationPolicy.listPrivileges() that can cause the operation to run in serial in a highly concurrent workload. Testing: - Ran all FE tests - Ran all E2E authorization tests - Added E2E test for URI privilege bug Change-Id: I942d9b55f07c8972f69e532567d9b7d80fceb6e5 --- M fe/src/main/java/org/apache/impala/authorization/AuthorizationPolicy.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationPolicy.java M fe/src/main/java/org/apache/impala/catalog/CatalogObjectCache.java M fe/src/main/java/org/apache/impala/catalog/Principal.java M fe/src/main/java/org/apache/impala/catalog/PrincipalPrivilege.java M tests/authorization/test_grant_revoke.py 6 files changed, 157 insertions(+), 123 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/95/13095/7 -- To view, visit http://gerrit.cloudera.org:8080/13095 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I942d9b55f07c8972f69e532567d9b7d80fceb6e5 Gerrit-Change-Number: 13095 Gerrit-PatchSet: 7 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7665: Fix unwarranted query cancellation on statestore restart
Lars Volker has posted comments on this change. (
http://gerrit.cloudera.org:8080/13061 )
Change subject: IMPALA-7665: Fix unwarranted query cancellation on statestore
restart
..
Patch Set 1:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/13061/1/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/13061/1/be/src/service/impala-server.cc@235
PS1, Line 235: scheduled to run on
running on
http://gerrit.cloudera.org:8080/#/c/13061/1/be/src/service/impala-server.cc@236
PS1, Line 236: This assumes that it will
This value should be large enough to give the statestore...
http://gerrit.cloudera.org:8080/#/c/13061/1/be/src/statestore/statestore-subscriber.h
File be/src/statestore/statestore-subscriber.h:
http://gerrit.cloudera.org:8080/#/c/13061/1/be/src/statestore/statestore-subscriber.h@215
PS1, Line 215: AtomicInt64 last_registration_ts_{0};
I think last_registration_ms_ would also convey that it's a timestamp, but have
the benefit of including the unit.
http://gerrit.cloudera.org:8080/#/c/13061/1/tests/custom_cluster/test_restart_services.py
File tests/custom_cluster/test_restart_services.py:
http://gerrit.cloudera.org:8080/#/c/13061/1/tests/custom_cluster/test_restart_services.py@103
PS1, Line 103: impalaD
nit: backend or impalad (lowercase d)
http://gerrit.cloudera.org:8080/#/c/13061/1/tests/custom_cluster/test_restart_services.py@112
PS1, Line 112: wait_for_admission_control
Does this make sure that the query is actually running, or is there a small
race still? Should we use wait_for_state below instead?
http://gerrit.cloudera.org:8080/#/c/13061/1/tests/custom_cluster/test_restart_services.py@122
PS1, Line 122: self.cluster.get_different_impalad(impalad).kill()
: start_time = time()
: self.cluster.statestored.start()
I think it would be interesting to test the case where the statestore comes
back up, but within the cancellation grace period an impalad dies. That could
be further distinguished in killing it a) before or b) after it has
re-registered itself with the statestored.
Another thing would be to kill the statestore again after 20 seconds. Would
this reset the grace period correctly?
--
To view, visit http://gerrit.cloudera.org:8080/13061
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I30b68bd8bde4bf589d58d42d6f683afb166de959
Gerrit-Change-Number: 13061
Gerrit-PatchSet: 1
Gerrit-Owner: Bikramjeet Vig
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Wed, 24 Apr 2019 21:59:36 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8444: Fix performance regression when building privilege name
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13095 ) Change subject: IMPALA-8444: Fix performance regression when building privilege name .. Patch Set 6: Code-Review+1 (1 comment) I'll let Csaba take another pass. http://gerrit.cloudera.org:8080/#/c/13095/6/fe/src/main/java/org/apache/impala/catalog/Principal.java File fe/src/main/java/org/apache/impala/catalog/Principal.java: http://gerrit.cloudera.org:8080/#/c/13095/6/fe/src/main/java/org/apache/impala/catalog/Principal.java@82 PS6, Line 82: return principalPrivileges_.keySet(); Actually thinking again, we are at a risk of exposing the underlying keySet() to the callers who can change it (same above). Probably safer to to return an ImmutableSet<> (and ImmutableList<> above). Anyway, they only do a shallow copy, so that should be fine I guess. -- To view, visit http://gerrit.cloudera.org:8080/13095 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I942d9b55f07c8972f69e532567d9b7d80fceb6e5 Gerrit-Change-Number: 13095 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 24 Apr 2019 21:57:59 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. Patch Set 16: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2898/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 16 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Wed, 24 Apr 2019 21:40:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. Patch Set 17: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 17 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Wed, 24 Apr 2019 21:20:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-2990: timeout unresponsive queries in coordinator
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12299 ) Change subject: IMPALA-2990: timeout unresponsive queries in coordinator .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2894/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12299 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I196c8c6a5633b1960e2c3a3884777be9b3824987 Gerrit-Change-Number: 12299 Gerrit-PatchSet: 9 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 24 Apr 2019 21:14:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. Patch Set 15: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2897/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 15 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Wed, 24 Apr 2019 21:29:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. Patch Set 14: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2896/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 14 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Wed, 24 Apr 2019 21:38:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. Patch Set 16: Code-Review+2 Thanks Radford and Austin for the patch! -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 16 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Wed, 24 Apr 2019 21:20:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. Patch Set 17: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4061/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 17 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Wed, 24 Apr 2019 21:20:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Austin Nobis has uploaded a new patch set (#16) to the change originally created by radford nguyen. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. IMPALA-8309: add user authorization_provider flag This commit adds a `authorization_provider` user-facing flag in order to provide a more human-readable alternative to the `authorization_factory_class` for internally-provided authorization strategies. The `authorization_factory_class` flag is retained, but no longer takes a default value if not specified. The default for `authorization_provider` is "sentry" in order to retain backwards-compatibility. If specified, `authorization_factory_class` will take precedence. Testing: - Manually started minicluster with each of following flags and verified correct authorization strategy chosen: - provider='' factory='' => sentry - provider=sentry factory='' => sentry - provider=ranger factory='' => ranger - provider='' factory=sentry => sentry - provider='' factory=ranger => ranger - provider=sentry factory=sentry => sentry - provider=ranger factory=sentry => sentry - provider=sentry factory=ranger => ranger - provider=ranger factory=ranger => ranger - Wrote unit tests to capture above assertions - Ran fe unit and e2e tests - Wrote e2e test to verify new flag behavior Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 --- M be/src/service/frontend.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/authorization/AuthorizationConfig.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationProvider.java M fe/src/main/java/org/apache/impala/authorization/NoopAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationConfig.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationConfig.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java A fe/src/main/java/org/apache/impala/util/AuthorizationUtil.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java A fe/src/test/java/org/apache/impala/util/AuthorizationUtilTest.java A tests/authorization/test_provider.py M tests/authorization/test_ranger.py 16 files changed, 335 insertions(+), 62 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/12901/16 -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 16 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: radford nguyen
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. Patch Set 13: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2895/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 13 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Wed, 24 Apr 2019 21:17:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Austin Nobis has uploaded a new patch set (#15) to the change originally created by radford nguyen. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. IMPALA-8309: add user authorization_provider flag This commit adds a `authorization_provider` user-facing flag in order to provide a more human-readable alternative to the `authorization_factory_class` for internally-provided authorization strategies. The `authorization_factory_class` flag is retained, but no longer takes a default value if not specified. The default for `authorization_provider` is "sentry" in order to retain backwards-compatibility. If specified, `authorization_factory_class` will take precedence. Testing: - Manually started minicluster with each of following flags and verified correct authorization strategy chosen: - provider='' factory='' => sentry - provider=sentry factory='' => sentry - provider=ranger factory='' => ranger - provider='' factory=sentry => sentry - provider='' factory=ranger => ranger - provider=sentry factory=sentry => sentry - provider=ranger factory=sentry => sentry - provider=sentry factory=ranger => ranger - provider=ranger factory=ranger => ranger - Wrote unit tests to capture above assertions - Ran fe unit and e2e tests - Wrote e2e test to verify new flag behavior Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 --- M be/src/service/frontend.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/authorization/AuthorizationConfig.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationProvider.java M fe/src/main/java/org/apache/impala/authorization/NoopAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationConfig.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationConfig.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java A fe/src/main/java/org/apache/impala/util/AuthorizationUtil.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java A fe/src/test/java/org/apache/impala/util/AuthorizationUtilTest.java A tests/authorization/test_provider.py M tests/authorization/test_ranger.py 16 files changed, 318 insertions(+), 61 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/12901/15 -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 15 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: radford nguyen
[Impala-ASF-CR] IMPALA-1856: MetadataOp.getTypeInfo() does not return all supported types.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13094 ) Change subject: IMPALA-1856: MetadataOp.getTypeInfo() does not return all supported types. .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2893/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13094 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icdccde7c32e52ed1b0c7b13a22171e8bcd7f1f2d Gerrit-Change-Number: 13094 Gerrit-PatchSet: 4 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Apr 2019 21:02:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Austin Nobis has uploaded a new patch set (#14) to the change originally created by radford nguyen. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. IMPALA-8309: add user authorization_provider flag This commit adds a `authorization_provider` user-facing flag in order to provide a more human-readable alternative to the `authorization_factory_class` for internally-provided authorization strategies. The `authorization_factory_class` flag is retained, but no longer takes a default value if not specified. The default for `authorization_provider` is "sentry" in order to retain backwards-compatibility. If specified, `authorization_factory_class` will take precedence. Testing: - Manually started minicluster with each of following flags and verified correct authorization strategy chosen: - provider='' factory='' => sentry - provider=sentry factory='' => sentry - provider=ranger factory='' => ranger - provider='' factory=sentry => sentry - provider='' factory=ranger => ranger - provider=sentry factory=sentry => sentry - provider=ranger factory=sentry => sentry - provider=sentry factory=ranger => ranger - provider=ranger factory=ranger => ranger - Wrote unit tests to capture above assertions - Ran fe unit and e2e tests - Wrote e2e test to verify new flag behavior Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 --- M be/src/service/frontend.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/authorization/AuthorizationConfig.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationProvider.java M fe/src/main/java/org/apache/impala/authorization/NoopAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationConfig.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationConfig.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java A fe/src/main/java/org/apache/impala/util/AuthorizationUtil.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/java/org/apache/impala/service/JniFrontendTest.java A fe/src/test/java/org/apache/impala/util/AuthorizationUtilTest.java A tests/authorization/test_provider.py M tests/authorization/test_ranger.py 17 files changed, 337 insertions(+), 61 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/12901/14 -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 14 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: radford nguyen
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Austin Nobis has uploaded a new patch set (#13) to the change originally created by radford nguyen. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. IMPALA-8309: add user authorization_provider flag This commit adds a `authorization_provider` user-facing flag in order to provide a more human-readable alternative to the `authorization_factory_class` for internally-provided authorization strategies. The `authorization_factory_class` flag is retained, but no longer takes a default value if not specified. The default for `authorization_provider` is "sentry" in order to retain backwards-compatibility. If specified, `authorization_factory_class` will take precedence. Testing: - Manually started minicluster with each of following flags and verified correct authorization strategy chosen: - provider='' factory='' => sentry - provider=sentry factory='' => sentry - provider=ranger factory='' => ranger - provider='' factory=sentry => sentry - provider='' factory=ranger => ranger - provider=sentry factory=sentry => sentry - provider=ranger factory=sentry => sentry - provider=sentry factory=ranger => ranger - provider=ranger factory=ranger => ranger - Wrote unit tests to capture above assertions - Ran fe unit and e2e tests - Wrote e2e test to verify new flag behavior Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 --- M be/src/service/frontend.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/authorization/AuthorizationConfig.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationProvider.java M fe/src/main/java/org/apache/impala/authorization/NoopAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationConfig.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationConfig.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java A fe/src/main/java/org/apache/impala/util/AuthorizationUtil.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/java/org/apache/impala/service/JniFrontendTest.java A fe/src/test/java/org/apache/impala/util/AuthorizationUtilTest.java A tests/authorization/test_provider.py M tests/authorization/test_ranger.py 17 files changed, 335 insertions(+), 61 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/12901/13 -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 13 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: radford nguyen
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. Patch Set 13: (4 comments) http://gerrit.cloudera.org:8080/#/c/12901/13/tests/authorization/test_provider.py File tests/authorization/test_provider.py: http://gerrit.cloudera.org:8080/#/c/12901/13/tests/authorization/test_provider.py@53 PS13, Line 53: T flake8: F821 undefined name 'TestProviderFails' http://gerrit.cloudera.org:8080/#/c/12901/13/tests/authorization/test_provider.py@56 PS13, Line 56: T flake8: F821 undefined name 'TestProviderFails' http://gerrit.cloudera.org:8080/#/c/12901/13/tests/authorization/test_provider.py@64 PS13, Line 64: T flake8: F821 undefined name 'TestProviderFails' http://gerrit.cloudera.org:8080/#/c/12901/13/tests/authorization/test_provider.py@73 PS13, Line 73: T flake8: F821 undefined name 'TestProviderFails' -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 13 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Wed, 24 Apr 2019 20:35:37 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8444: Fix performance regression when building privilege name
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13095 ) Change subject: IMPALA-8444: Fix performance regression when building privilege name .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2892/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13095 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I942d9b55f07c8972f69e532567d9b7d80fceb6e5 Gerrit-Change-Number: 13095 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 24 Apr 2019 20:21:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-2990: timeout unresponsive queries in coordinator
Thomas Marshall has posted comments on this change. (
http://gerrit.cloudera.org:8080/12299 )
Change subject: IMPALA-2990: timeout unresponsive queries in coordinator
..
Patch Set 9:
(9 comments)
http://gerrit.cloudera.org:8080/#/c/12299/8//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/12299/8//COMMIT_MSG@34
PS8, Line 34: 'REPORT_EXEC_STATUS_DELAY:JITTER@1000'
> Stale ?
Done
http://gerrit.cloudera.org:8080/#/c/12299/8/be/src/runtime/query-state.cc
File be/src/runtime/query-state.cc:
http://gerrit.cloudera.org:8080/#/c/12299/8/be/src/runtime/query-state.cc@400
PS8, Line 400: we set backend rpc timeouts for kr
> Can you please leave a TODO on rethinking the timeout period for RPCs in ge
Done
http://gerrit.cloudera.org:8080/#/c/12299/8/be/src/runtime/query-state.cc@414
PS8, Line 414: repor
> spent
Done
http://gerrit.cloudera.org:8080/#/c/12299/8/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/12299/8/be/src/service/impala-server.cc@217
PS8, Line 217: "report is sent.");
> and only the final report is sent.
Done
http://gerrit.cloudera.org:8080/#/c/12299/8/be/src/service/impala-server.cc@218
PS8, Line 218: 600
> To avoid regression, let's be more conservative and set it to be at least 3
Done
http://gerrit.cloudera.org:8080/#/c/12299/8/be/src/service/impala-server.cc@221
PS8, Line 221: The coordinator will wait "
: "--status_report_max_retry_s * (1 +
--status_report_cancellation_padding / 100) "
: "without receiving a status report before deciding that a
backend is unresponsive
> May be clearer to include the formula you have in the commit message.
Done
http://gerrit.cloudera.org:8080/#/c/12299/8/be/src/service/impala-server.cc@421
PS8, Line 421: if (FLAGS_abort_on_config_error) {
> May wanna check if --status_report_cancellation_padding is set to something
Done
http://gerrit.cloudera.org:8080/#/c/12299/8/be/src/service/impala-server.cc@2248
PS8, Line 2248: // this thread.
> DCHECK_GT(max_lag_ms, 0);
Done
http://gerrit.cloudera.org:8080/#/c/12299/8/tests/custom_cluster/test_rpc_timeout.py
File tests/custom_cluster/test_rpc_timeout.py:
http://gerrit.cloudera.org:8080/#/c/12299/8/tests/custom_cluster/test_rpc_timeout.py@44
PS8, Line 44:
> flake8: E501 line too long (103 > 90 characters)
Done
--
To view, visit http://gerrit.cloudera.org:8080/12299
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I196c8c6a5633b1960e2c3a3884777be9b3824987
Gerrit-Change-Number: 12299
Gerrit-PatchSet: 9
Gerrit-Owner: Thomas Marshall
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Reviewer: Thomas Marshall
Gerrit-Reviewer: Todd Lipcon
Gerrit-Comment-Date: Wed, 24 Apr 2019 20:20:44 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-2990: timeout unresponsive queries in coordinator
Hello Michael Ho, Philip Zeyliger, Todd Lipcon, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12299 to look at the new patch set (#9). Change subject: IMPALA-2990: timeout unresponsive queries in coordinator .. IMPALA-2990: timeout unresponsive queries in coordinator The coordinator currently waits indefinitely if it does not receive a status report from a backend. This could cause a query to hang indefinitely in certain situations, for example if the backend decides to cancel itself as a result of failed status report rpcs. This patch adds a thread to ImpalaServer which periodically iterates over all queries for which that server is the coordinator and cancels any that haven't had a report from a backend in a certain amount of time. This patch adds two flags: --status_report_max_retry_s: the maximum number of seconds a backend will attempt to send status reports before giving up. This is used in place of --status_report_max_retries which is now deprecated. --status_report_cancellation_padding: the coordinator will wait --status_report_max_retry_s * (1 + --status_report_cancellation_padding / 100) before concluding a backend is not responding and cancelling the query. Testing: - Added a functional test that runs a query that is cancelled through the new mechanism. Ran tests on a 10 node cluster loaded with tpch 500: - Ran the stress test for 1000 queries with the debug actions: 'REPORT_EXEC_STATUS_DELAY:JITTER@1000' Prior to this patch, this setup results in hanging queries. With this patch, no hangs were observed. - Ran perf tests with 4 concurrent streams, 3 iterations per query. Found no change in performance. Change-Id: I196c8c6a5633b1960e2c3a3884777be9b3824987 --- M be/src/common/global-flags.cc M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/runtime/query-state.cc M be/src/runtime/query-state.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M common/thrift/ImpalaInternalService.thrift M common/thrift/generate_error_codes.py M tests/custom_cluster/test_rpc_timeout.py 12 files changed, 190 insertions(+), 46 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/12299/9 -- To view, visit http://gerrit.cloudera.org:8080/12299 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I196c8c6a5633b1960e2c3a3884777be9b3824987 Gerrit-Change-Number: 12299 Gerrit-PatchSet: 9 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8386: Fix incorrect equivalence conjuncts not treated as identity
Impala Public Jenkins has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/12939 )
Change subject: IMPALA-8386: Fix incorrect equivalence conjuncts not treated as
identity
..
IMPALA-8386: Fix incorrect equivalence conjuncts not treated as identity
When generating single node plans for inline views, Impala will create
some equivalence conjuncts based on slot equivalences. However, these
conjuncts may finally be substituted to identity (e.g. a = a) which may
incorrectly reject rows with nulls. We already have some logic to remove
this kind of conjuncts but the existing checks have exceptions.
For example, consider the following tables and a query:
table Atable Btable C
+--+ +--++ +--+--+
| a_id | | b_id | amount | | a_id | b_id |
+--+ +--++ +--+--+
| 1| | 1| 10 | | 1| 1|
| 2| | 1| 20 | | 2| 2|
+--+ | 2| NULL | +--+--+
+--++
select * from (select t2.a_id, t2.amount1, t2.amount2
from a
left outer join (
select c.a_id, amount as amount1, amount as amount2
from b join c on b.b_id = c.b_id
) t2
on a.a_id = t2.a_id
) t1;
They query has 11 slots. The valueTransferGraph (slot equivalences) has
3 strongly connected components:
* {slot0 (b.b_id), slot1 (c.b_id)}
* {slot2 (c.a_id), slot4 (t2.a_id), slot8 (t1.a_id)}
* {slot3 (b.amount), slot5 (t2.amount1), slot6 (t2.amount2),
slot9 (t1.amount1), slot10 (t1.amount2)}
In SingleNodePlanner#migrateConjunctsToInlineView, when dealing with
inline view t1, a predicate "t1.amount1 = t1.amount2" will first be
created by Analyzer#createEquivConjuncts, then be substituted using the
smap_ of the inline view and become "t2.amount1 = t2.amount2". It can
still pass the IdentityPredicate check. However, the substituted one
will finally be resolved to "amount = amount" and be assigned to the
left outer join node. So nulls are incorrectly rejected.
Actually, when checking IdentityPredicates, we need to check the final
resolved version of them using base table slots (baseTblSmap_). So the
predicate "t1.amount1 = t1.amount2" will be resolved to "amount = amount"
and won't pass the IdentityPredicate check.
Tests:
* Add plan tests in PlannerTest/inline-view.test
* Run all tests locally in CORE exploration strategy
Change-Id: Ia87aa9db2de85f0716e4854a88727aad593773fa
Reviewed-on: http://gerrit.cloudera.org:8080/12939
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
---
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
M fe/src/main/java/org/apache/impala/analysis/ExprSubstitutionMap.java
M fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M testdata/workloads/functional-planner/queries/PlannerTest/inline-view.test
6 files changed, 410 insertions(+), 20 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/12939
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ia87aa9db2de85f0716e4854a88727aad593773fa
Gerrit-Change-Number: 12939
Gerrit-PatchSet: 7
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-1856: MetadataOp.getTypeInfo() does not return all supported types.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13094 ) Change subject: IMPALA-1856: MetadataOp.getTypeInfo() does not return all supported types. .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13094 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icdccde7c32e52ed1b0c7b13a22171e8bcd7f1f2d Gerrit-Change-Number: 13094 Gerrit-PatchSet: 5 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Apr 2019 20:15:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-1856: MetadataOp.getTypeInfo() does not return all supported types.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13094 ) Change subject: IMPALA-1856: MetadataOp.getTypeInfo() does not return all supported types. .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4060/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13094 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icdccde7c32e52ed1b0c7b13a22171e8bcd7f1f2d Gerrit-Change-Number: 13094 Gerrit-PatchSet: 5 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Apr 2019 20:15:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-1856: MetadataOp.getTypeInfo() does not return all supported types.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13094 ) Change subject: IMPALA-1856: MetadataOp.getTypeInfo() does not return all supported types. .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13094 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icdccde7c32e52ed1b0c7b13a22171e8bcd7f1f2d Gerrit-Change-Number: 13094 Gerrit-PatchSet: 4 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Apr 2019 20:15:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8386: Fix incorrect equivalence conjuncts not treated as identity
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12939 ) Change subject: IMPALA-8386: Fix incorrect equivalence conjuncts not treated as identity .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12939 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia87aa9db2de85f0716e4854a88727aad593773fa Gerrit-Change-Number: 12939 Gerrit-PatchSet: 6 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Apr 2019 20:08:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-1856: MetadataOp.getTypeInfo() does not return all supported types.
Abhishek Rawat has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/13094 ) Change subject: IMPALA-1856: MetadataOp.getTypeInfo() does not return all supported types. .. IMPALA-1856: MetadataOp.getTypeInfo() does not return all supported types. The MetadataOp.getTypeInfo() is missing all the complex types (ARRAY, MAP, and STRUCT). Several of the primitive data types such as CHAR, VARCHAR, DECIMAL, DATE (newly added) are also not returned. The Impala JDBC client should in theory call MetadataOp.getTypeInfo() but that is not happening in the latest version of the driver. This change will only ensure that on Impala side the MetadataOp.getTypeInfo() returns correct results. Updated MetadataOp::createGetTypeInfoResults to include all supported and externally visible data types, including complex types (which were missing along with some other Primitive types). Added a new function ScalarType::isInternalType() to identify internal types such as NULL_TYPE, FIXED_UDA_INTERMEDIATE which are not exposed through getTypeInfo function or in any other manner through SQL. Testing: There was a testing gap and ideally whenever a new type is added or support for a type is changed the MetadataOp.getTypeInfo() should return the correct result set representing the supported types. - Updated FrontendTest.java to ensure that the result set from MetadataOp::getTypeInfo contains all supported and externally visible types - Added new E2E test (test_get_type_info) in tests/hs2/test_hs2.py. The new test validates that the HS2 GetTypeInfo() RPC returns supported and externally visible types. Change-Id: Icdccde7c32e52ed1b0c7b13a22171e8bcd7f1f2d --- M fe/src/main/java/org/apache/impala/catalog/ScalarType.java M fe/src/main/java/org/apache/impala/catalog/Type.java M fe/src/main/java/org/apache/impala/service/MetadataOp.java M fe/src/test/java/org/apache/impala/service/FrontendTest.java M tests/hs2/test_hs2.py 5 files changed, 145 insertions(+), 34 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/94/13094/4 -- To view, visit http://gerrit.cloudera.org:8080/13094 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icdccde7c32e52ed1b0c7b13a22171e8bcd7f1f2d Gerrit-Change-Number: 13094 Gerrit-PatchSet: 4 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8446: Create a unit test for Admission Controller.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13078 ) Change subject: IMPALA-8446: Create a unit test for Admission Controller. .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4059/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13078 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8a840590b868f2df1a06f3f397b7b0fc2b29462c Gerrit-Change-Number: 13078 Gerrit-PatchSet: 4 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Apr 2019 20:03:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8446: Create a unit test for Admission Controller.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13078 ) Change subject: IMPALA-8446: Create a unit test for Admission Controller. .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13078 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8a840590b868f2df1a06f3f397b7b0fc2b29462c Gerrit-Change-Number: 13078 Gerrit-PatchSet: 4 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Apr 2019 20:03:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8446: Create a unit test for Admission Controller.
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/13078 ) Change subject: IMPALA-8446: Create a unit test for Admission Controller. .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13078 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8a840590b868f2df1a06f3f397b7b0fc2b29462c Gerrit-Change-Number: 13078 Gerrit-PatchSet: 3 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Apr 2019 20:02:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-1856: MetadataOp.getTypeInfo() does not return all supported types.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13094 ) Change subject: IMPALA-1856: MetadataOp.getTypeInfo() does not return all supported types. .. Patch Set 3: Code-Review+2 (1 comment) I had one nit, after that I can start the merge http://gerrit.cloudera.org:8080/#/c/13094/3/fe/src/main/java/org/apache/impala/catalog/Type.java File fe/src/main/java/org/apache/impala/catalog/Type.java: http://gerrit.cloudera.org:8080/#/c/13094/3/fe/src/main/java/org/apache/impala/catalog/Type.java@129 PS3, Line 129: /** nit: space before comment -- To view, visit http://gerrit.cloudera.org:8080/13094 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icdccde7c32e52ed1b0c7b13a22171e8bcd7f1f2d Gerrit-Change-Number: 13094 Gerrit-PatchSet: 3 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Apr 2019 19:45:30 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8446: Create a unit test for Admission Controller.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13078 ) Change subject: IMPALA-8446: Create a unit test for Admission Controller. .. Patch Set 3: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/13078 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8a840590b868f2df1a06f3f397b7b0fc2b29462c Gerrit-Change-Number: 13078 Gerrit-PatchSet: 3 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Apr 2019 19:48:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8444: Fix performance regression when building privilege name
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13095 ) Change subject: IMPALA-8444: Fix performance regression when building privilege name .. Patch Set 6: (2 comments) http://gerrit.cloudera.org:8080/#/c/13095/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13095/3//COMMIT_MSG@38 PS3, Line 38: This patch removes incorrect synchronization in : SentryAuthorizationPolicy.listPrivileges() that can cause the operation : to run in serial in a highly concurrent workload. > How did you test this? I think this codepath is sensitive in the sense that I meant to say remove incorrect synchronization. Updated the comment. http://gerrit.cloudera.org:8080/#/c/13095/3/fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationPolicy.java File fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationPolicy.java: http://gerrit.cloudera.org:8080/#/c/13095/3/fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationPolicy.java@90 PS3, Line 90: > Is there a way to avoid a copy to a HashSet()? Not sure what that is buying Yeah this isn't needed since we're returning the catalog keys backed by ConcurrentHashMap, which should be thread-safe. Done. -- To view, visit http://gerrit.cloudera.org:8080/13095 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I942d9b55f07c8972f69e532567d9b7d80fceb6e5 Gerrit-Change-Number: 13095 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 24 Apr 2019 19:33:46 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8444: Fix performance regression when building privilege name
Fredy Wijaya has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/13095 ) Change subject: IMPALA-8444: Fix performance regression when building privilege name .. IMPALA-8444: Fix performance regression when building privilege name This patch fixes the performance regression when building privilege name by rewriting PrincipalPrivilege.buildPrivilegeName() with a simple string concatentation instead of using a list that gets converted into a string. Below is the result of running a benchmark using JMH comparing the old and new implementations: Result "org.apache.impala.BuildPrivilegeNameBenchmark.fast": 0.344 ±(99.9%) 0.004 us/op [Average] (min, avg, max) = (0.336, 0.344, 0.355), stdev = 0.005 CI (99.9%): [0.339, 0.348] (assumes normal distribution) Result "org.apache.impala.BuildPrivilegeNameBenchmark.slow": 0.831 ±(99.9%) 0.011 us/op [Average] (min, avg, max) = (0.807, 0.831, 0.856), stdev = 0.015 CI (99.9%): [0.820, 0.842] (assumes normal distribution) Benchmark Mode Cnt Score Error Units BuildPrivilegeNameBenchmark.fast avgt 25 0.344 ± 0.004 us/op BuildPrivilegeNameBenchmark.slow avgt 25 0.831 ± 0.011 us/op This patch also updates SentryAuthorizationPolicy.listPrivileges() to reuse the privilege names that have already been built instead of building them again. While fixing this, I found a bug where Principal stores the PrincipalPrivilege in case insensitive way. This is true for all privilege scopes, except URI. This patch fixes the issue by storing URI privilege in a case sensitive manner. This patch removes incorrect synchronization in SentryAuthorizationPolicy.listPrivileges() that can cause the operation to run in serial in a highly concurrent workload. Testing: - Ran all FE tests - Ran all E2E authorization tests - Added E2E test for URI privilege bug Change-Id: I942d9b55f07c8972f69e532567d9b7d80fceb6e5 --- M fe/src/main/java/org/apache/impala/authorization/AuthorizationPolicy.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationPolicy.java M fe/src/main/java/org/apache/impala/catalog/CatalogObjectCache.java M fe/src/main/java/org/apache/impala/catalog/Principal.java M fe/src/main/java/org/apache/impala/catalog/PrincipalPrivilege.java M tests/authorization/test_grant_revoke.py 6 files changed, 155 insertions(+), 123 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/95/13095/6 -- To view, visit http://gerrit.cloudera.org:8080/13095 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I942d9b55f07c8972f69e532567d9b7d80fceb6e5 Gerrit-Change-Number: 13095 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-1856: MetadataOp.getTypeInfo() does not return all supported types.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13094 ) Change subject: IMPALA-1856: MetadataOp.getTypeInfo() does not return all supported types. .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2891/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13094 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icdccde7c32e52ed1b0c7b13a22171e8bcd7f1f2d Gerrit-Change-Number: 13094 Gerrit-PatchSet: 3 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Apr 2019 19:19:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8149 : Add support for alter database events
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13049 ) Change subject: IMPALA-8149 : Add support for alter_database events .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2890/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13049 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaf020e85cae04163bf32e31363eb4119d624640b Gerrit-Change-Number: 13049 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 24 Apr 2019 19:10:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7290: part 2: Add HS2 support to Impala shell
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12884 ) Change subject: IMPALA-7290: part 2: Add HS2 support to Impala shell .. Patch Set 13: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2889/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12884 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6d5cc83d545aacc659523f29b1d6feed672e2a12 Gerrit-Change-Number: 12884 Gerrit-PatchSet: 13 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 24 Apr 2019 19:10:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8295: Fix flakiness in TestAdmissionControllerStress
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13103 ) Change subject: IMPALA-8295: Fix flakiness in TestAdmissionControllerStress .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13103 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If9ada7bdbe2461d164100badabb905cf9a8cf4cc Gerrit-Change-Number: 13103 Gerrit-PatchSet: 3 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Apr 2019 19:05:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8295: Fix flakiness in TestAdmissionControllerStress
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13103 ) Change subject: IMPALA-8295: Fix flakiness in TestAdmissionControllerStress .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4058/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13103 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If9ada7bdbe2461d164100badabb905cf9a8cf4cc Gerrit-Change-Number: 13103 Gerrit-PatchSet: 3 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Apr 2019 19:05:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8295: Fix flakiness in TestAdmissionControllerStress
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/13103 ) Change subject: IMPALA-8295: Fix flakiness in TestAdmissionControllerStress .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13103 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If9ada7bdbe2461d164100badabb905cf9a8cf4cc Gerrit-Change-Number: 13103 Gerrit-PatchSet: 2 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Apr 2019 19:01:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7290: part 1: clean up shell tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13083 ) Change subject: IMPALA-7290: part 1: clean up shell tests .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2888/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13083 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe5ab7f4817e690b7d3be08d71f8f14364b84412 Gerrit-Change-Number: 13083 Gerrit-PatchSet: 8 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 24 Apr 2019 19:00:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8295: Fix flakiness in TestAdmissionControllerStress
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13103 ) Change subject: IMPALA-8295: Fix flakiness in TestAdmissionControllerStress .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2887/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13103 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If9ada7bdbe2461d164100badabb905cf9a8cf4cc Gerrit-Change-Number: 13103 Gerrit-PatchSet: 1 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Apr 2019 18:55:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8295: Fix flakiness in TestAdmissionControllerStress
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/13103 ) Change subject: IMPALA-8295: Fix flakiness in TestAdmissionControllerStress .. Patch Set 2: (1 comment) Carrying over Tim's +1 http://gerrit.cloudera.org:8080/#/c/13103/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13103/1//COMMIT_MSG@9 PS1, Line 9: encounter > encounters Done -- To view, visit http://gerrit.cloudera.org:8080/13103 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If9ada7bdbe2461d164100badabb905cf9a8cf4cc Gerrit-Change-Number: 13103 Gerrit-PatchSet: 2 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Apr 2019 18:55:30 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8295: Fix flakiness in TestAdmissionControllerStress
Hello Michael Ho, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13103 to look at the new patch set (#2). Change subject: IMPALA-8295: Fix flakiness in TestAdmissionControllerStress .. IMPALA-8295: Fix flakiness in TestAdmissionControllerStress Currently the test_mem_limit in TestAdmissionControllerStress encounters flaky failures due to a timeout waiting on queries to end. Increasing the timeout from 60 to 90 seconds to accommodate this flakiness. Testing: Ran the test in a loop for more than 15 hours without any failures. Change-Id: If9ada7bdbe2461d164100badabb905cf9a8cf4cc --- M tests/custom_cluster/test_admission_controller.py 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/13103/2 -- To view, visit http://gerrit.cloudera.org:8080/13103 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If9ada7bdbe2461d164100badabb905cf9a8cf4cc Gerrit-Change-Number: 13103 Gerrit-PatchSet: 2 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] Make infra/python compatible with both Python 2 & 3
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13070 ) Change subject: Make infra/python compatible with both Python 2 & 3 .. Patch Set 2: I've been able to push to gerrit.cloudera.org this morning, so not sure what is wrong - maybe retry? -- To view, visit http://gerrit.cloudera.org:8080/13070 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If4285a021bb581f88425daa52ef8a3f844017d82 Gerrit-Change-Number: 13070 Gerrit-PatchSet: 2 Gerrit-Owner: Akshesh Doshi Gerrit-Reviewer: Akshesh Doshi Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Apr 2019 18:34:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8295: Fix flakiness in TestAdmissionControllerStress
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13103 ) Change subject: IMPALA-8295: Fix flakiness in TestAdmissionControllerStress .. Patch Set 1: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/13103 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If9ada7bdbe2461d164100badabb905cf9a8cf4cc Gerrit-Change-Number: 13103 Gerrit-PatchSet: 1 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Apr 2019 18:39:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-1856: MetadataOp.getTypeInfo() does not return all supported types.
Abhishek Rawat has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/13094 ) Change subject: IMPALA-1856: MetadataOp.getTypeInfo() does not return all supported types. .. IMPALA-1856: MetadataOp.getTypeInfo() does not return all supported types. The MetadataOp.getTypeInfo() is missing all the complex types (ARRAY, MAP, and STRUCT). Several of the primitive data types such as CHAR, VARCHAR, DECIMAL, DATE (newly added) are also not returned. The Impala JDBC client should in theory call MetadataOp.getTypeInfo() but that is not happening in the latest version of the driver. This change will only ensure that on Impala side the MetadataOp.getTypeInfo() returns correct results. Updated MetadataOp::createGetTypeInfoResults to include all supported and externally visible data types, including complex types (which were missing along with some other Primitive types). Added a new function ScalarType::isInternalType() to identify internal types such as NULL_TYPE, FIXED_UDA_INTERMEDIATE which are not exposed through getTypeInfo function or in any other manner through SQL. Testing: There was a testing gap and ideally whenever a new type is added or support for a type is changed the MetadataOp.getTypeInfo() should return the correct result set representing the supported types. - Updated FrontendTest.java to ensure that the result set from MetadataOp::getTypeInfo contains all supported and externally visible types - Added new E2E test (test_get_type_info) in tests/hs2/test_hs2.py. The new test validates that the HS2 GetTypeInfo() RPC returns supported and externally visible types. Change-Id: Icdccde7c32e52ed1b0c7b13a22171e8bcd7f1f2d --- M fe/src/main/java/org/apache/impala/catalog/ScalarType.java M fe/src/main/java/org/apache/impala/catalog/Type.java M fe/src/main/java/org/apache/impala/service/MetadataOp.java M fe/src/test/java/org/apache/impala/service/FrontendTest.java M tests/hs2/test_hs2.py 5 files changed, 144 insertions(+), 34 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/94/13094/3 -- To view, visit http://gerrit.cloudera.org:8080/13094 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icdccde7c32e52ed1b0c7b13a22171e8bcd7f1f2d Gerrit-Change-Number: 13094 Gerrit-PatchSet: 3 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8149 : Add support for alter database events
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13049 ) Change subject: IMPALA-8149 : Add support for alter_database events .. Patch Set 3: (8 comments) http://gerrit.cloudera.org:8080/#/c/13049/3/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/13049/3/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@755 PS3, Line 755:* Removes a given version number from the catalog database/table's list of versions line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/13049/3/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@756 PS3, Line 756:* for in-flight events. line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/13049/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/13049/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@369 PS3, Line 369: case COMMENT_ON: line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/13049/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@370 PS3, Line 370: alterCommentOn(ddlRequest.getComment_on_params(), response); line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/13049/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@370 PS3, Line 370: alterCommentOn(ddlRequest.getComment_on_params(), response); tab used for whitespace http://gerrit.cloudera.org:8080/#/c/13049/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@371 PS3, Line 371: break; tab used for whitespace http://gerrit.cloudera.org:8080/#/c/13049/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3800 PS3, Line 3800: private void alterDatabase(TAlterDbParams params, TDdlExecResponse response) line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/13049/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3807 PS3, Line 3807: } line has trailing whitespace -- To view, visit http://gerrit.cloudera.org:8080/13049 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaf020e85cae04163bf32e31363eb4119d624640b Gerrit-Change-Number: 13049 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 24 Apr 2019 18:28:04 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7290: part 2: Add HS2 support to Impala shell
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12884 to look at the new patch set (#13). Change subject: IMPALA-7290: part 2: Add HS2 support to Impala shell .. IMPALA-7290: part 2: Add HS2 support to Impala shell HS2 is the new default and the user-visible differences are minimal. Beeswax is still supported via --protocol=beeswax but will be deprecated. Differences are abstracted into ImpalaClient subclasses. This support requires Impala-specific extensions to the HS2 interface, similar to the existing extensions to Beeswax. Thus the HS2 shell is only forwards-compatible with newer Impala versions. I considered trying to gracefully degrade when the new extensions weren't present, but it didn't seem to be worth the ongoing testing effort. Here are the changes required to make it work: * Switch to TBinaryProtocolAccelerated to avoid perf regression. The HS2 protocol requires decoding more primitive values (because its not a string-per-row), which was slow with the pure python implementation of TBinaryProtocol. * Added bitarray module to efficiently unpack null indicators * Minimise invasiveness of changes by transposing and stringifying the columnar results into rows in impala_client.py. The transposition needs to happen before display anyway. * Add PingImpalaHS2Service() to get back version string and webserver address. * Add CloseImpalaOperation() extension to return DML row counts. * Add is_closed member to query handles to avoid shell independently tracking whether the query handle was closed or not. * Include query status in HS2 log to match beeswax. * HS2 GetLog() command now includes query status error message for consistency with beeswax. * "set"/"set all" uses the client requests options, not the session default. This captures the effective value of TIMEZONE, which was previously missing. This also requires test changes where the tests set non-default values, e.g. for ABORT_ON_ERROR. * "set all" on the server side returns REMOVED query options - the shell needs to know these so it can correctly ignore them. * Clean up self.orig_cmd/self.last_leading comment argument passing to avoid implicit parameter passing through multiple function calls. * Clean up argument handling in shell tests to consistently pass around lists of arguments instead of strings that are subject to shell tokenisation rules. Testing: * Shell tests can run with both protocols * Add tests for formatting of all types and NULL values * Added testing for floating point output formatting, which does change as a result of switching to server-side vs client-side formatting. * Verified that newly-added tests were actually going through HS2 by disabling hs2 on the minicluster and running tests. Performance: Baseline from beeswax shell for large extract is as follows: $ time impala-shell.sh -B -q 'select * from tpch_parquet.orders' > /dev/null real0m6.708s user0m5.132s sys 0m0.204s After this change it is somewhat slower, but we generally don't consider bulk extract performance through the shell to be perf-critical: real0m7.625s user0m6.436s sys 0m0.256s Change-Id: I6d5cc83d545aacc659523f29b1d6feed672e2a12 --- M LICENSE.txt M be/src/runtime/dml-exec-state.cc M be/src/runtime/dml-exec-state.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M bin/rat_exclude_files.txt M common/thrift/ImpalaService.thrift M infra/python/deps/compiled-requirements.txt A shell/ext-py/bitarray-0.9.0/PKG-INFO A shell/ext-py/bitarray-0.9.0/bitarray/__init__.py A shell/ext-py/bitarray-0.9.0/bitarray/_bitarray.c A shell/ext-py/bitarray-0.9.0/bitarray/test_bitarray.py A shell/ext-py/bitarray-0.9.0/setup.py M shell/impala_client.py M shell/impala_shell.py M shell/impala_shell_config_defaults.py M shell/make_shell_tarball.sh M shell/option_parser.py M shell/thrift_sasl.py M testdata/workloads/functional-query/queries/QueryTest/set.test M tests/custom_cluster/test_shell_interactive.py M tests/custom_cluster/test_shell_interactive_reconnect.py M tests/hs2/test_hs2.py M tests/query_test/test_observability.py M tests/shell/test_shell_commandline.py M tests/shell/test_shell_interactive.py M tests/shell/util.py 31 files changed, 7,233 insertions(+), 458 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/84/12884/13 -- To view, visit http://gerrit.cloudera.org:8080/12884 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6d5cc83d545aacc659523f29b1d6feed672e2a12 Gerrit-Change-Number: 12884 Gerrit-PatchSet: 13 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-8295: Fix flakiness in TestAdmissionControllerStress
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/13103 ) Change subject: IMPALA-8295: Fix flakiness in TestAdmissionControllerStress .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/13103/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13103/1//COMMIT_MSG@9 PS1, Line 9: ecounters encounters -- To view, visit http://gerrit.cloudera.org:8080/13103 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If9ada7bdbe2461d164100badabb905cf9a8cf4cc Gerrit-Change-Number: 13103 Gerrit-PatchSet: 1 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Apr 2019 18:32:12 + Gerrit-HasComments: Yes
[native-toolchain-CR] Add zstd to toolchain
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13079 ) Change subject: Add zstd to toolchain .. Add zstd to toolchain This adds the latest release of zstd for the toolchain. Testing: Built locally, confirmed that binary utilities work. Built on all supported platforms. Change-Id: I17f1489c7f3cc5c1585b5472f4b6e910ee10d204 Reviewed-on: http://gerrit.cloudera.org:8080/13079 Reviewed-by: Thomas Marshall Tested-by: Tim Armstrong --- M buildall.sh A source/zstd/build.sh 2 files changed, 40 insertions(+), 0 deletions(-) Approvals: Thomas Marshall: Looks good to me, approved Tim Armstrong: Verified -- To view, visit http://gerrit.cloudera.org:8080/13079 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I17f1489c7f3cc5c1585b5472f4b6e910ee10d204 Gerrit-Change-Number: 13079 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong
[native-toolchain-CR] Add zstd to toolchain
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13079 ) Change subject: Add zstd to toolchain .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13079 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I17f1489c7f3cc5c1585b5472f4b6e910ee10d204 Gerrit-Change-Number: 13079 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Apr 2019 18:29:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8149 : Add support for alter database events
xiaom...@cloudera.com has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/13049 ) Change subject: IMPALA-8149 : Add support for alter_database events .. IMPALA-8149 : Add support for alter_database events This change adds support for alter_database events in two parts: One is adding catalogServiceId and catalogVersion in db parameters when alter database. The other is adding alter database event, check if it's self event during process, if true do nothing, if false replace caralog cached db with event db. Testing: Enabled testAlterDisableFlagFromDb in MetastoreEventsProcessorTest. Change-Id: Iaf020e85cae04163bf32e31363eb4119d624640b --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 5 files changed, 265 insertions(+), 112 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/13049/3 -- To view, visit http://gerrit.cloudera.org:8080/13049 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iaf020e85cae04163bf32e31363eb4119d624640b Gerrit-Change-Number: 13049 Gerrit-PatchSet: 3 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-7290: part 1: clean up shell tests
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13083 to look at the new patch set (#8). Change subject: IMPALA-7290: part 1: clean up shell tests .. IMPALA-7290: part 1: clean up shell tests This sets up the tests to be extensible to test shell in both beeswax and HS2 modes. Testing: * Add test dimension containing only beeswax in preparation for HS2 dimension. * Factor out hardcoded ports. * Add tests for formatting of all types and NULL values. * Merge date shell test into general type tests. * Added testing for floating point output formatting, which does change as a result of switching to server-side vs client-side formatting. * Use unique_database for tests that create tables. Change-Id: Ibe5ab7f4817e690b7d3be08d71f8f14364b84412 --- M bin/load-data.py M testdata/bin/create-load-data.sh M testdata/bin/create-tpcds-testcase-files.sh M tests/beeswax/impala_beeswax.py M tests/common/impala_test_suite.py M tests/common/test_dimensions.py M tests/custom_cluster/test_client_ssl.py M tests/custom_cluster/test_shell_interactive.py M tests/custom_cluster/test_shell_interactive_reconnect.py M tests/query_test/test_date_queries.py M tests/shell/test_shell_commandline.py M tests/shell/test_shell_interactive.py M tests/shell/util.py 13 files changed, 761 insertions(+), 608 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/13083/8 -- To view, visit http://gerrit.cloudera.org:8080/13083 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibe5ab7f4817e690b7d3be08d71f8f14364b84412 Gerrit-Change-Number: 13083 Gerrit-PatchSet: 8 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8295: Fix flakiness in TestAdmissionControllerStress
Bikramjeet Vig has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13103 Change subject: IMPALA-8295: Fix flakiness in TestAdmissionControllerStress .. IMPALA-8295: Fix flakiness in TestAdmissionControllerStress Currently the test_mem_limit in TestAdmissionControllerStress ecounters flaky failures due to a timeout waiting on queries to end. Increasing the timeout from 60 to 90 seconds to accommodate this flakiness. Testing: Ran the test in a loop for more than 15 hours without any failures. Change-Id: If9ada7bdbe2461d164100badabb905cf9a8cf4cc --- M tests/custom_cluster/test_admission_controller.py 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/13103/1 -- To view, visit http://gerrit.cloudera.org:8080/13103 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: If9ada7bdbe2461d164100badabb905cf9a8cf4cc Gerrit-Change-Number: 13103 Gerrit-PatchSet: 1 Gerrit-Owner: Bikramjeet Vig
[Impala-ASF-CR] IMPALA-8444: Fix performance regression when building privilege name
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13095 ) Change subject: IMPALA-8444: Fix performance regression when building privilege name .. Patch Set 2: (2 comments) Couple of minor comments, lgtm otherwise. http://gerrit.cloudera.org:8080/#/c/13095/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13095/3//COMMIT_MSG@38 PS3, Line 38: operation to run in serial in a highly concurrent workload. This has a : trade off of stale authorization metadata, which may be acceptable to : get a better performance. How did you test this? I think this codepath is sensitive in the sense that most prod users run with sentry turned on and any ConccurentModificationException in prod workloads might be a disaster. Probably run a concurrency stress test of some sorts with sentry turned on? http://gerrit.cloudera.org:8080/#/c/13095/3/fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationPolicy.java File fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationPolicy.java: http://gerrit.cloudera.org:8080/#/c/13095/3/fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationPolicy.java@90 PS3, Line 90: getPrivilegeNames Is there a way to avoid a copy to a HashSet()? Not sure what that is buying us. -- To view, visit http://gerrit.cloudera.org:8080/13095 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I942d9b55f07c8972f69e532567d9b7d80fceb6e5 Gerrit-Change-Number: 13095 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 24 Apr 2019 18:21:04 + Gerrit-HasComments: Yes
[native-toolchain-CR] Enable reusing ccache directories.
Thomas Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/12996 ) Change subject: Enable reusing ccache directories. .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12996 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I482aa13e833d4680efe7cab98aad7f4fb998bfc0 Gerrit-Change-Number: 12996 Gerrit-PatchSet: 3 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Wed, 24 Apr 2019 18:02:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8444: Fix performance regression when building privilege name
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13095 ) Change subject: IMPALA-8444: Fix performance regression when building privilege name .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2886/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13095 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I942d9b55f07c8972f69e532567d9b7d80fceb6e5 Gerrit-Change-Number: 13095 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 24 Apr 2019 18:07:50 + Gerrit-HasComments: No
[native-toolchain-CR] Add zstd to toolchain
Thomas Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/13079 ) Change subject: Add zstd to toolchain .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13079 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I17f1489c7f3cc5c1585b5472f4b6e910ee10d204 Gerrit-Change-Number: 13079 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Wed, 24 Apr 2019 18:01:49 + Gerrit-HasComments: No
[Impala-ASF-CR] Remove references to the $IMPALA HOME/thirdparty directory
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13092 ) Change subject: Remove references to the $IMPALA_HOME/thirdparty directory .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4057/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13092 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2edfd499febb5a25fdcf59b5183eccf192a08be0 Gerrit-Change-Number: 13092 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 24 Apr 2019 17:54:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8444: Fix performance regression when building privilege name
Fredy Wijaya has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/13095 ) Change subject: IMPALA-8444: Fix performance regression when building privilege name .. IMPALA-8444: Fix performance regression when building privilege name This patch fixes the performance regression when building privilege name by rewriting PrincipalPrivilege.buildPrivilegeName() with a simple string concatentation instead of using a list that gets converted into a string. Below is the result of running a benchmark using JMH comparing the old and new implementations: Result "org.apache.impala.BuildPrivilegeNameBenchmark.fast": 0.344 ±(99.9%) 0.004 us/op [Average] (min, avg, max) = (0.336, 0.344, 0.355), stdev = 0.005 CI (99.9%): [0.339, 0.348] (assumes normal distribution) Result "org.apache.impala.BuildPrivilegeNameBenchmark.slow": 0.831 ±(99.9%) 0.011 us/op [Average] (min, avg, max) = (0.807, 0.831, 0.856), stdev = 0.015 CI (99.9%): [0.820, 0.842] (assumes normal distribution) Benchmark Mode Cnt Score Error Units BuildPrivilegeNameBenchmark.fast avgt 25 0.344 ± 0.004 us/op BuildPrivilegeNameBenchmark.slow avgt 25 0.831 ± 0.011 us/op This patch also updates SentryAuthorizationPolicy.listPrivileges() to reuse the privilege names that have already been built instead of building them again. While fixing this, I found a bug where Principal stores the PrincipalPrivilege in case insensitive way. This is true for all privilege scopes, except URI. This patch fixes the issue by storing URI privilege in a case sensitive manner. This patch removes the synchronized keyword in SentryAuthorizationPolicy.listPrivileges() in order to not cause the operation to run in serial in a highly concurrent workload. This has a trade off of stale authorization metadata, which may be acceptable to get a better performance. Testing: - Ran all FE tests - Ran all E2E authorization tests - Added E2E test for URI privilege bug Change-Id: I942d9b55f07c8972f69e532567d9b7d80fceb6e5 --- M fe/src/main/java/org/apache/impala/authorization/AuthorizationPolicy.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationPolicy.java M fe/src/main/java/org/apache/impala/catalog/CatalogObjectCache.java M fe/src/main/java/org/apache/impala/catalog/Principal.java M fe/src/main/java/org/apache/impala/catalog/PrincipalPrivilege.java M tests/authorization/test_grant_revoke.py 6 files changed, 153 insertions(+), 121 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/95/13095/4 -- To view, visit http://gerrit.cloudera.org:8080/13095 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I942d9b55f07c8972f69e532567d9b7d80fceb6e5 Gerrit-Change-Number: 13095 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8444: Fix performance regression when building privilege name
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13095 ) Change subject: IMPALA-8444: Fix performance regression when building privilege name .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2885/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13095 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I942d9b55f07c8972f69e532567d9b7d80fceb6e5 Gerrit-Change-Number: 13095 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 24 Apr 2019 17:25:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/12901 )
Change subject: IMPALA-8309: add user authorization_provider flag
..
Patch Set 10:
(16 comments)
http://gerrit.cloudera.org:8080/#/c/12901/10//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/12901/10//COMMIT_MSG@23
PS10, Line 23:
: Manually started minicluster with each of following flags
: and verified correct authorization strategy chosen:
:
: - provider='' factory='' => sentry
: - provider=sentry factory='' => sentry
: - provider=ranger factory='' => ranger
: - provider='' factory=sentry => sentry
: - provider='' factory=ranger => ranger
: - provider=sentry factory=sentry => sentry
: - provider=ranger factory=sentry => sentry
: - provider=sentry factory=ranger => ranger
: - provider=ranger factory=ranger => ranger
:
: Wrote unit tests to capture above assertions
:
: Ran fe unit and e2e tests
:
: Wrote e2e test to verify new flag behavior
nit: add bullet points for each item?
http://gerrit.cloudera.org:8080/#/c/12901/10/fe/src/main/java/org/apache/impala/authorization/AuthorizationConfig.java
File fe/src/main/java/org/apache/impala/authorization/AuthorizationConfig.java:
http://gerrit.cloudera.org:8080/#/c/12901/10/fe/src/main/java/org/apache/impala/authorization/AuthorizationConfig.java@30
PS10, Line 30: This is
:* typically used only for logging.
this comment is not necessary
http://gerrit.cloudera.org:8080/#/c/12901/10/fe/src/main/java/org/apache/impala/authorization/AuthorizationProvider.java
File
fe/src/main/java/org/apache/impala/authorization/AuthorizationProvider.java:
http://gerrit.cloudera.org:8080/#/c/12901/10/fe/src/main/java/org/apache/impala/authorization/AuthorizationProvider.java@35
PS10, Line 35: factoryClassName
nit: the naming convention is factoryClassName_
http://gerrit.cloudera.org:8080/#/c/12901/10/fe/src/main/java/org/apache/impala/authorization/AuthorizationProvider.java@37
PS10, Line 37: AuthorizationProvider(String factoryClassName) {
: this.factoryClassName = factoryClassName;
: }
:
: /**
:* Returns the canonical name of the {@link
AuthorizationFactory} implementation
:* associated with this provider.
:*
:* @return the canonical name of the {@link
AuthorizationFactory} impl for `this`
:*/
: public String getAuthorizationFactoryClassName() {
: return this.factoryClassName;
: }
nit: remove this.
http://gerrit.cloudera.org:8080/#/c/12901/10/fe/src/main/java/org/apache/impala/service/BackendConfig.java
File fe/src/main/java/org/apache/impala/service/BackendConfig.java:
http://gerrit.cloudera.org:8080/#/c/12901/10/fe/src/main/java/org/apache/impala/service/BackendConfig.java@31
PS10, Line 31:
nit: remove extra new line
http://gerrit.cloudera.org:8080/#/c/12901/10/fe/src/main/java/org/apache/impala/service/JniCatalog.java
File fe/src/main/java/org/apache/impala/service/JniCatalog.java:
http://gerrit.cloudera.org:8080/#/c/12901/10/fe/src/main/java/org/apache/impala/service/JniCatalog.java@116
PS10, Line 116: final AuthorizationFactory authzFactory
: = JniFrontend.authzFactoryFrom(BackendConfig.INSTANCE);
sharing code between JniCatalog and JniFrontend is weird. Maybe create a helper
class or helper function elsewhere?
http://gerrit.cloudera.org:8080/#/c/12901/10/fe/src/main/java/org/apache/impala/service/JniFrontend.java
File fe/src/main/java/org/apache/impala/service/JniFrontend.java:
http://gerrit.cloudera.org:8080/#/c/12901/10/fe/src/main/java/org/apache/impala/service/JniFrontend.java@791
PS10, Line 791: );
nit: move to L790
http://gerrit.cloudera.org:8080/#/c/12901/10/fe/src/main/java/org/apache/impala/service/JniFrontend.java@796
PS10, Line 796: );
nit: move to L795
http://gerrit.cloudera.org:8080/#/c/12901/10/fe/src/main/java/org/apache/impala/service/JniFrontend.java@824
PS10, Line 824: );
nit: move to L823
http://gerrit.cloudera.org:8080/#/c/12901/10/fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
File fe/src/test/java/org/apache/impala/common/FrontendTestBase.java:
http://gerrit.cloudera.org:8080/#/c/12901/10/fe/src/test/java/org/apache/impala/common/FrontendTestBase.java@326
PS10, Line 326: public String getProviderName() {
: return "noop";
: }
nit: can be put in a single line
http://gerrit.cloudera.org:8080/#/c/12901/10/fe/src/test/java/org/apache/impala/service/JniFrontendTest.java
File fe/src/test/java/org/apache/impala/serv
