[Impala-ASF-CR] IMPALA-8921: User short name for Ranger grant/revoke requests
Bharath Vissapragada has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14185 Change subject: IMPALA-8921: User short name for Ranger grant/revoke requests .. IMPALA-8921: User short name for Ranger grant/revoke requests For certain grant/revoke Ranger commmands, we ended up passing the full name which is a problem when kerberos is enabled. Ranger expects the short name during authorization. Testing: We do not have test coverage with kerberos enabled, so I inspected the code manually to make sure we are using getShortName() everywhere. Change-Id: I3dc1bf55d50dc2e5fa6e07f16644f0a2773f2d23 --- M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java 2 files changed, 6 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/14185/1 -- To view, visit http://gerrit.cloudera.org:8080/14185 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I3dc1bf55d50dc2e5fa6e07f16644f0a2773f2d23 Gerrit-Change-Number: 14185 Gerrit-PatchSet: 1 Gerrit-Owner: Bharath Vissapragada
[Impala-ASF-CR] IMPALA-8572: Log query events before unregister.
Hello radford nguyen, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14143 to look at the new patch set (#8). Change subject: IMPALA-8572: Log query events before unregister. .. IMPALA-8572: Log query events before unregister. Currently, the query events (audits and lineages) are logged as a part of query unregistration. This delays the event logging in cases where the Unregister() is delayed by client for some reason (ex: Hue does not call Unregister until the browser tab is closed) or the client goes away without calling Unregister and the query timeout kicks in. This patch moves this event logging to an earlier stage in the query lifecycle. Moved the event logging related code into ClientRequestState for easier code refactoring. The conditions under which the events are logged are slightly modified by this patch. Without the patch, events are logged for unsuccessful queries if atleast a single fetch is perfomed. This patch relaxes this guarantee to log events for any query that reaches the FINISHED state (rows are available to fetch by the client) and does not wait for a fetch to be performed. This simplifies the coordinator state machine by avoiding unnecessary synchronization. Added some test coverage for coordinator side code paths for writing lineages. fe specific lineage tests only verified the correctness of lineage created but did not test whether it was being flushed correctly to the disk. Change-Id: I639b9c1acb9806b29292cd85be2863688453ca2e --- M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-server.cc M be/src/service/impala-server.h A testdata/workloads/functional-query/queries/QueryTest/lineage.test M tests/common/impala_test_suite.py M tests/common/test_result_verifier.py M tests/custom_cluster/test_lineage.py M tests/unittests/test_file_parser.py M tests/util/test_file_parser.py 10 files changed, 6,058 insertions(+), 239 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/14143/8 -- To view, visit http://gerrit.cloudera.org:8080/14143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I639b9c1acb9806b29292cd85be2863688453ca2e Gerrit-Change-Number: 14143 Gerrit-PatchSet: 8 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen
[Impala-ASF-CR] IMPALA-8572: Log query events before unregister.
Hello radford nguyen, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14143 to look at the new patch set (#12). Change subject: IMPALA-8572: Log query events before unregister. .. IMPALA-8572: Log query events before unregister. Currently, the query events (audits and lineages) are logged as a part of query unregistration. This delays the event logging in cases where the Unregister() is delayed by client for some reason (ex: Hue does not call Unregister until the browser tab is closed) or the client goes away without calling Unregister and the query timeout kicks in. This patch moves this event logging to an earlier stage in the query lifecycle. Moved the event logging related code into ClientRequestState for easier code refactoring. The conditions under which the events are logged are slightly modified by this patch. Without the patch, events are logged for unsuccessful queries if atleast a single fetch is perfomed. This patch relaxes this guarantee to log events for any query that reaches the FINISHED state (rows are available to fetch by the client) and does not wait for a fetch to be performed. This simplifies the coordinator state machine by avoiding unnecessary synchronization. Added some test coverage for coordinator side code paths for writing lineages. fe specific lineage tests only verified the correctness of lineage created but did not test whether it was being flushed correctly to the disk. Change-Id: I639b9c1acb9806b29292cd85be2863688453ca2e --- M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-server.cc M be/src/service/impala-server.h A testdata/workloads/functional-query/queries/QueryTest/lineage.test M tests/common/impala_test_suite.py M tests/common/test_result_verifier.py M tests/custom_cluster/test_lineage.py M tests/unittests/test_file_parser.py M tests/util/test_file_parser.py 10 files changed, 6,018 insertions(+), 243 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/14143/12 -- To view, visit http://gerrit.cloudera.org:8080/14143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I639b9c1acb9806b29292cd85be2863688453ca2e Gerrit-Change-Number: 14143 Gerrit-PatchSet: 12 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen
[Impala-ASF-CR] IMPALA-8572: Log query events before unregister.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14143 ) Change subject: IMPALA-8572: Log query events before unregister. .. Patch Set 12: Code-Review+2 (1 comment) Carrying +2. I've spent some time on the patch to see what could be wrong with synchronization but I didn't see anything wrong. Also, interestingly the tests started working in the last 2 runs. I'll try to get this in for now. Will revert and dig into it if we run into any issues in the builds. http://gerrit.cloudera.org:8080/#/c/14143/11/be/src/service/client-request-state.cc File be/src/service/client-request-state.cc: http://gerrit.cloudera.org:8080/#/c/14143/11/be/src/service/client-request-state.cc@780 PS11, Line 780: // Some metadata operations like GET_COLUMNS do not rely on WaitAsync() to launch > Do we need the wait_thread_ check though? Since after the wait thread is de Like we discussed, some codepaths like metadata op requests do not launch the async wait thread but still BlockOnWait() during fetches. In such cases the condition guards them from blocking forever. We can probably refactor the code better to make the metadata ops act like regular query paths. For now, I added a comment. -- To view, visit http://gerrit.cloudera.org:8080/14143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I639b9c1acb9806b29292cd85be2863688453ca2e Gerrit-Change-Number: 14143 Gerrit-PatchSet: 12 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Wed, 11 Sep 2019 21:51:06 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8923: remove synchronized in HBaseTable.getEstimatedRowStats
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/14188 )
Change subject: IMPALA-8923: remove synchronized in
HBaseTable.getEstimatedRowStats
..
Patch Set 2: Code-Review+2
(3 comments)
nice find.
http://gerrit.cloudera.org:8080/#/c/14188/2/fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java
File fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java:
http://gerrit.cloudera.org:8080/#/c/14188/2/fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java@369
PS2, Line 369: LOG.trace(String.format("getEstimatedRowStats for %s
from %s to %s",
nit: use parameterized logging, {}...
http://gerrit.cloudera.org:8080/#/c/14188/2/fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java@438
PS2, Line 438: if (LOG.isTraceEnabled()) {
same. Probably helps to dump the timetaken too.
http://gerrit.cloudera.org:8080/#/c/14188/2/fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java@652
PS2, Line 652: LOG.trace("getRegionsInRange");
intentional?
--
To view, visit http://gerrit.cloudera.org:8080/14188
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifa23c16ee662c4f22851c700aea2ea5be847b64d
Gerrit-Change-Number: 14188
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Comment-Date: Fri, 06 Sep 2019 18:53:53 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8228: Ownership support for Ranger authz
Hello Austin Nobis, Fredy Wijaya, Todd Lipcon, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14106
to look at the new patch set (#11).
Change subject: IMPALA-8228: Ownership support for Ranger authz
..
IMPALA-8228: Ownership support for Ranger authz
Without this patch, explicit privileges are needed even
for owners of databases/tables to perform actions on them.
Example: 'user' is the owner of database 'foo'. To create
a table 't' under 'foo', 'user' needs to be granted a CREATE
privilege on 'foo'
That is unintuitive from a user POV since users expect owners
to have ALL privileges on the objects they own. This patch extends
that support to Impala's ranger authorization plugin.
Ranger natively supports the concept of ownership by letting the
callers pass the ownership context to RangerAccessResourceImpl.
This patch plumbs the owner information for the authorizables
(currently only supported for Tables / Databases) which is then
evaulated during authorization.
For the ownership based authorization to work, ranger-admin side
policy on {OWNER} user needs to be defined.
Testing: Added some unit-tests and e-e tests that cover scenarios
where ownership is used for authorization.
Caveat: Ownership is a part of HMS thrift object. Since we do not
aggressively load HMS schemas during start-up, coordinators with
cold caches can result in weird table listings due to lack of
metadata needed for verifying ownership. This should be fixed
separately to make the behavior more consistent and user friendly.
(Added comments in the code wherever necessary along with a test
to simulate this).
Change-Id: I737b7164a3e7afb9996b3402e6872effd663f7b4
---
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/CollectionTableRef.java
M fe/src/main/java/org/apache/impala/analysis/CopyTestCaseStmt.java
M fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/DropDbStmt.java
M fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java
M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
M fe/src/main/java/org/apache/impala/authorization/Authorizable.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizableColumn.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizableDb.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizableFactory.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizableTable.java
M
fe/src/main/java/org/apache/impala/authorization/DefaultAuthorizableFactory.java
M fe/src/main/java/org/apache/impala/authorization/PrivilegeRequestBuilder.java
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaResourceBuilder.java
M
fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizableFactory.java
M fe/src/main/java/org/apache/impala/catalog/BuiltinsDb.java
M fe/src/main/java/org/apache/impala/catalog/Db.java
M fe/src/main/java/org/apache/impala/catalog/FeDb.java
M fe/src/main/java/org/apache/impala/catalog/FeTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java
M tests/authorization/test_ranger.py
30 files changed, 570 insertions(+), 101 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/14106/11
--
To view, visit http://gerrit.cloudera.org:8080/14106
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I737b7164a3e7afb9996b3402e6872effd663f7b4
Gerrit-Change-Number: 14106
Gerrit-PatchSet: 11
Gerrit-Owner: Bharath Vissapragada
Gerrit-Reviewer: Austin Nobis
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8228: Ownership support for Ranger authz
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14106 ) Change subject: IMPALA-8228: Ownership support for Ranger authz .. Patch Set 11: (1 comment) http://gerrit.cloudera.org:8080/#/c/14106/10/tests/authorization/test_ranger.py File tests/authorization/test_ranger.py: http://gerrit.cloudera.org:8080/#/c/14106/10/tests/authorization/test_ranger.py@711 PS10, Line 711: > flake8: E501 line too long (93 > 90 characters) Done -- To view, visit http://gerrit.cloudera.org:8080/14106 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I737b7164a3e7afb9996b3402e6872effd663f7b4 Gerrit-Change-Number: 14106 Gerrit-PatchSet: 11 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 11 Sep 2019 23:33:39 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8228: Ownership support for Ranger authz
Hello Austin Nobis, Fredy Wijaya, Todd Lipcon, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14106
to look at the new patch set (#12).
Change subject: IMPALA-8228: Ownership support for Ranger authz
..
IMPALA-8228: Ownership support for Ranger authz
Without this patch, explicit privileges are needed even
for owners of databases/tables to perform actions on them.
Example: 'user' is the owner of database 'foo'. To create
a table 't' under 'foo', 'user' needs to be granted a CREATE
privilege on 'foo'
That is unintuitive from a user POV since users expect owners
to have ALL privileges on the objects they own. This patch extends
that support to Impala's ranger authorization plugin.
Ranger natively supports the concept of ownership by letting the
callers pass the ownership context to RangerAccessResourceImpl.
This patch plumbs the owner information for the authorizables
(currently only supported for Tables / Databases) which is then
evaulated during authorization.
For the ownership based authorization to work, ranger-admin side
policy on {OWNER} user needs to be defined.
Testing: Added some unit-tests and e-e tests that cover scenarios
where ownership is used for authorization.
Caveat: Ownership is a part of HMS thrift object. Since we do not
aggressively load HMS schemas during start-up, coordinators with
cold caches can result in weird table listings due to lack of
metadata needed for verifying ownership. This should be fixed
separately to make the behavior more consistent and user friendly.
(Added comments in the code wherever necessary along with a test
to simulate this).
Change-Id: I737b7164a3e7afb9996b3402e6872effd663f7b4
---
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/CollectionTableRef.java
M fe/src/main/java/org/apache/impala/analysis/CopyTestCaseStmt.java
M fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/DropDbStmt.java
M fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java
M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
M fe/src/main/java/org/apache/impala/authorization/Authorizable.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizableColumn.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizableDb.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizableFactory.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizableTable.java
M
fe/src/main/java/org/apache/impala/authorization/DefaultAuthorizableFactory.java
M fe/src/main/java/org/apache/impala/authorization/PrivilegeRequestBuilder.java
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaResourceBuilder.java
M
fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizableFactory.java
M fe/src/main/java/org/apache/impala/catalog/BuiltinsDb.java
M fe/src/main/java/org/apache/impala/catalog/Db.java
M fe/src/main/java/org/apache/impala/catalog/FeDb.java
M fe/src/main/java/org/apache/impala/catalog/FeTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java
M tests/authorization/test_ranger.py
30 files changed, 570 insertions(+), 101 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/14106/12
--
To view, visit http://gerrit.cloudera.org:8080/14106
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I737b7164a3e7afb9996b3402e6872effd663f7b4
Gerrit-Change-Number: 14106
Gerrit-PatchSet: 12
Gerrit-Owner: Bharath Vissapragada
Gerrit-Reviewer: Austin Nobis
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8228: Ownership support for Ranger authz
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14106 ) Change subject: IMPALA-8228: Ownership support for Ranger authz .. Patch Set 12: Code-Review+2 (2 comments) Thanks for the reviews, carrying +2. http://gerrit.cloudera.org:8080/#/c/14106/11/fe/src/main/java/org/apache/impala/authorization/Authorizable.java File fe/src/main/java/org/apache/impala/authorization/Authorizable.java: http://gerrit.cloudera.org:8080/#/c/14106/11/fe/src/main/java/org/apache/impala/authorization/Authorizable.java@65 PS11, Line 65: int ownerHash = getOwnerUser() == null ? 0 : getOwnerUser().hashCode(); : int nameHash = getName().hashCode(); : return nameHash * 31 + ownerHash; > nit: You can use Objects.hash(nameHash, ownerHash) Objects.hash is what I tried but since owner is nullable, it causes an NPE. http://gerrit.cloudera.org:8080/#/c/14106/11/fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java File fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java: http://gerrit.cloudera.org:8080/#/c/14106/11/fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java@3068 PS11, Line 3068: String>builder() > nit: I think Java 8 can infer this without explicitly specifying the type p Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.3:testCompile (default-testCompile) on project impala-frontend: Compilation failure [ERROR] /home/bharath/impala/Impala/fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java:[3110,15] incompatible types: com.google.common.collect.ImmutableMap cannot be converted to com.google.common.collect.ImmutableMap Think we don't have this https://github.com/google/guava/issues/1166 -- To view, visit http://gerrit.cloudera.org:8080/14106 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I737b7164a3e7afb9996b3402e6872effd663f7b4 Gerrit-Change-Number: 14106 Gerrit-PatchSet: 12 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 12 Sep 2019 04:30:50 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8228: Ownership support for Ranger authz
Bharath Vissapragada has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/14106 )
Change subject: IMPALA-8228: Ownership support for Ranger authz
..
IMPALA-8228: Ownership support for Ranger authz
Without this patch, explicit privileges are needed even
for owners of databases/tables to perform actions on them.
Example: 'user' is the owner of database 'foo'. To create
a table 't' under 'foo', 'user' needs to be granted a CREATE
privilege on 'foo'
That is unintuitive from a user POV since users expect owners
to have ALL privileges on the objects they own. This patch extends
that support to Impala's ranger authorization plugin.
Ranger natively supports the concept of ownership by letting the
callers pass the ownership context to RangerAccessResourceImpl.
This patch plumbs the owner information for the authorizables
(currently only supported for Tables / Databases) which is then
evaulated during authorization.
For the ownership based authorization to work, ranger-admin side
policy on {OWNER} user needs to be defined.
Testing: Added some unit-tests and e-e tests that cover scenarios
where ownership is used for authorization.
Caveat: Ownership is a part of HMS thrift object. Since we do not
aggressively load HMS schemas during start-up, coordinators with
cold caches can result in weird table listings due to lack of
metadata needed for verifying ownership. This should be fixed
separately to make the behavior more consistent and user friendly.
(Added comments in the code wherever necessary along with a test
to simulate this).
Change-Id: I737b7164a3e7afb9996b3402e6872effd663f7b4
Reviewed-on: http://gerrit.cloudera.org:8080/14106
Reviewed-by: Bharath Vissapragada
Tested-by: Bharath Vissapragada
---
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/CollectionTableRef.java
M fe/src/main/java/org/apache/impala/analysis/CopyTestCaseStmt.java
M fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/DropDbStmt.java
M fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java
M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
M fe/src/main/java/org/apache/impala/authorization/Authorizable.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizableColumn.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizableDb.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizableFactory.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizableTable.java
M
fe/src/main/java/org/apache/impala/authorization/DefaultAuthorizableFactory.java
M fe/src/main/java/org/apache/impala/authorization/PrivilegeRequestBuilder.java
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaResourceBuilder.java
M
fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizableFactory.java
M fe/src/main/java/org/apache/impala/catalog/BuiltinsDb.java
M fe/src/main/java/org/apache/impala/catalog/Db.java
M fe/src/main/java/org/apache/impala/catalog/FeDb.java
M fe/src/main/java/org/apache/impala/catalog/FeTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java
M tests/authorization/test_ranger.py
30 files changed, 570 insertions(+), 101 deletions(-)
Approvals:
Bharath Vissapragada: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/14106
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I737b7164a3e7afb9996b3402e6872effd663f7b4
Gerrit-Change-Number: 14106
Gerrit-PatchSet: 13
Gerrit-Owner: Bharath Vissapragada
Gerrit-Reviewer: Austin Nobis
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8228: Ownership support for Ranger authz
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14106 ) Change subject: IMPALA-8228: Ownership support for Ranger authz .. Patch Set 12: Verified+1 Already verified with a previous GVO -- To view, visit http://gerrit.cloudera.org:8080/14106 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I737b7164a3e7afb9996b3402e6872effd663f7b4 Gerrit-Change-Number: 14106 Gerrit-PatchSet: 12 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 12 Sep 2019 04:31:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8228: Ownership support for Ranger authz
Hello Austin Nobis, Fredy Wijaya, Todd Lipcon, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14106
to look at the new patch set (#10).
Change subject: IMPALA-8228: Ownership support for Ranger authz
..
IMPALA-8228: Ownership support for Ranger authz
Without this patch, explicit privileges are needed even
for owners of databases/tables to perform actions on them.
Example: 'user' is the owner of database 'foo'. To create
a table 't' under 'foo', 'user' needs to be granted a CREATE
privilege on 'foo'
That is unintuitive from a user POV since users expect owners
to have ALL privileges on the objects they own. This patch extends
that support to Impala's ranger authorization plugin.
Ranger natively supports the concept of ownership by letting the
callers pass the ownership context to RangerAccessResourceImpl.
This patch plumbs the owner information for the authorizables
(currently only supported for Tables / Databases) which is then
evaulated during authorization.
For the ownership based authorization to work, ranger-admin side
policy on {OWNER} user needs to be defined.
Testing: Added some unit-tests and e-e tests that cover scenarios
where ownership is used for authorization.
Caveat: Ownership is a part of HMS thrift object. Since we do not
aggressively load HMS schemas during start-up, coordinators with
cold caches can result in weird table listings due to lack of
metadata needed for verifying ownership. This should be fixed
separately to make the behavior more consistent and user friendly.
(Added comments in the code wherever necessary along with a test
to simulate this).
Change-Id: I737b7164a3e7afb9996b3402e6872effd663f7b4
---
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/CollectionTableRef.java
M fe/src/main/java/org/apache/impala/analysis/CopyTestCaseStmt.java
M fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/DropDbStmt.java
M fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java
M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
M fe/src/main/java/org/apache/impala/authorization/Authorizable.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizableColumn.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizableDb.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizableFactory.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizableTable.java
M
fe/src/main/java/org/apache/impala/authorization/DefaultAuthorizableFactory.java
M fe/src/main/java/org/apache/impala/authorization/PrivilegeRequestBuilder.java
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaResourceBuilder.java
M
fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizableFactory.java
M fe/src/main/java/org/apache/impala/catalog/BuiltinsDb.java
M fe/src/main/java/org/apache/impala/catalog/Db.java
M fe/src/main/java/org/apache/impala/catalog/FeDb.java
M fe/src/main/java/org/apache/impala/catalog/FeTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java
M tests/authorization/test_ranger.py
30 files changed, 570 insertions(+), 101 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/14106/10
--
To view, visit http://gerrit.cloudera.org:8080/14106
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I737b7164a3e7afb9996b3402e6872effd663f7b4
Gerrit-Change-Number: 14106
Gerrit-PatchSet: 10
Gerrit-Owner: Bharath Vissapragada
Gerrit-Reviewer: Austin Nobis
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8228: Ownership support for Ranger authz
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14106 ) Change subject: IMPALA-8228: Ownership support for Ranger authz .. Patch Set 10: (2 comments) http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java File fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java: http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java@134 PS6, Line 134: path_.getRawPath().get(0), path_.getRootTable().getOwnerUser()) > nit: path_.getRootTable().getOwnerUser() should be on a new line to follow don't think we have any formatting rules around this, it was mostly around not exceeding line widths. So skipping for now. http://gerrit.cloudera.org:8080/#/c/14106/8/tests/authorization/test_ranger.py File tests/authorization/test_ranger.py: http://gerrit.cloudera.org:8080/#/c/14106/8/tests/authorization/test_ranger.py@699 PS8, Line 699: result =\ > Do we need to assert anything from the result of this command? Done -- To view, visit http://gerrit.cloudera.org:8080/14106 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I737b7164a3e7afb9996b3402e6872effd663f7b4 Gerrit-Change-Number: 14106 Gerrit-PatchSet: 10 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 11 Sep 2019 23:31:29 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8228: Ownership support for Ranger authz
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14106 ) Change subject: IMPALA-8228: Ownership support for Ranger authz .. Patch Set 10: Any more reviews for this patch? -- To view, visit http://gerrit.cloudera.org:8080/14106 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I737b7164a3e7afb9996b3402e6872effd663f7b4 Gerrit-Change-Number: 14106 Gerrit-PatchSet: 10 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 11 Sep 2019 23:31:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8932: addendum - protocol var not defined
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14225 ) Change subject: IMPALA-8932: addendum - protocol var not defined .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14225 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I75c41a02bc7f1314e48bb5a39b945119264ce478 Gerrit-Change-Number: 14225 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Comment-Date: Thu, 12 Sep 2019 18:24:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8228: Ownership support for Ranger authz
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/14106 )
Change subject: IMPALA-8228: Ownership support for Ranger authz
..
Patch Set 6:
(14 comments)
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/Authorizable.java
File fe/src/main/java/org/apache/impala/authorization/Authorizable.java:
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/Authorizable.java@62
PS6, Line 62: return getName().hashCode();
> If we're comparing owner, we should update the hashCode() as well.
Done
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/Authorizable.java@62
PS6, Line 62: return getName().hashCode();
> If we're comparing owner, we should update the hashCode() as well.
Done
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/Authorizable.java@70
PS6, Line 70: temp.getOwnerUser() == this.getOwnerUser()
> Shouldn't this use equals()?
Done
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/AuthorizableColumn.java
File fe/src/main/java/org/apache/impala/authorization/AuthorizableColumn.java:
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/AuthorizableColumn.java@31
PS6, Line 31: private final String ownerUser_;
> For consistency, annotate with @Nullable.
Done
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/AuthorizableColumn.java@34
PS6, Line 34: String ownerUser
> I think it's more readable to also annotate the parameter with @Nullable.
Done
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/AuthorizableDb.java
File fe/src/main/java/org/apache/impala/authorization/AuthorizableDb.java:
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/AuthorizableDb.java@33
PS6, Line 33: String ownerUser
> Same comment about @Nullable parameter.
Done
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/AuthorizableFactory.java
File fe/src/main/java/org/apache/impala/authorization/AuthorizableFactory.java:
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/AuthorizableFactory.java@45
PS6, Line 45: /**
:* Creates a new instance of column {@link Authorizable} for a
given database name and
:* gives access to all tables and columns.
:*/
: Authorizable newColumnAllTbls(String dbName, @Nullable String
dbOwnerUser);
:
: /**
:* Creates a new instance of column {@link Authorizable} for
given database and table
:* names and gives access to all columns.
:*/
: Authorizable newColumnInTable(
: String dbName, String tableName, @Nullable String
tblOwnerUser);
:
: /**
:* Creates a new instance of column {@link Authorizable} for
given database, table, and
:* column names.
:*/
: Authorizable newColumnInTable(
: String dbName, String tableName, String columnName,
@Nullable String tblOwnerUser);
> nit: update javadoc about the new owner parameter.
Done
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/AuthorizableTable.java
File fe/src/main/java/org/apache/impala/authorization/AuthorizableTable.java:
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/AuthorizableTable.java@34
PS6, Line 34: String ownerUser
> Same comment about @Nullable parameter.
Done
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
File
fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java:
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java@94
PS6, Line 94: resources.add(new RangerImpalaResourceBuilder()
: .database("*").table("*").column("*").build());
: resources.add(new RangerImpalaResourceBuilder()
: .database("*").function("*").build());
: resources.add(new
RangerImpalaResourceBuilder().uri("*").build());
> Should we add .owner() here as well?
This patch was meant to add the ownership only for db/tbl[col]. Also curious
what does it even mean to be a owner of a server?
http://gerrit.cloudera.org:8080/#/c/14106/6/fe/src/main/java/org/apache/impala/catalog/Db.java
File fe/src/main/java/org/apache/impala/catalog/Db.java:
[Impala-ASF-CR] IMPALA-8933: Enforce ranger deny policy
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14203 ) Change subject: IMPALA-8933: Enforce ranger deny policy .. Patch Set 2: (5 comments) http://gerrit.cloudera.org:8080/#/c/14203/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/14203/2//COMMIT_MSG@9 PS2, Line 9: This patch fixes a case where access to a given column is allowed at the table nit: Limit commit message line widths to 72 chars. http://gerrit.cloudera.org:8080/#/c/14203/2//COMMIT_MSG@16 PS2, Line 16: - Manually tested with table level allow and column level deny policies in No luck with automating it in AuthorizationStmtTest? http://gerrit.cloudera.org:8080/#/c/14203/2/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java File fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java: http://gerrit.cloudera.org:8080/#/c/14203/2/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java@234 PS2, Line 234: Trailing white space http://gerrit.cloudera.org:8080/#/c/14203/2/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java@235 PS2, Line 235: if (hasTableSelectPriv && How about ALTER privilege? Alter table drop/rename col etc? In-general, shouldn't we just skip 'hasTableSelectPriv' for col privileges and let Ranger/Sentry do its check via hasAccess()? http://gerrit.cloudera.org:8080/#/c/14203/2/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java@236 PS2, Line 236: request.getPrivilege() != Privilege.SELECT && 4 spaced indents.https://cwiki.apache.org/confluence/display/IMPALA/Impala+Style+Guide -- To view, visit http://gerrit.cloudera.org:8080/14203 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic60786cd81080feeb0bfcd92aa2be646ee8cb7da Gerrit-Change-Number: 14203 Gerrit-PatchSet: 2 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 10 Sep 2019 18:50:44 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8572: Log query events before unregister.
Hello radford nguyen, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14143 to look at the new patch set (#11). Change subject: IMPALA-8572: Log query events before unregister. .. IMPALA-8572: Log query events before unregister. Currently, the query events (audits and lineages) are logged as a part of query unregistration. This delays the event logging in cases where the Unregister() is delayed by client for some reason (ex: Hue does not call Unregister until the browser tab is closed) or the client goes away without calling Unregister and the query timeout kicks in. This patch moves this event logging to an earlier stage in the query lifecycle. Moved the event logging related code into ClientRequestState for easier code refactoring. The conditions under which the events are logged are slightly modified by this patch. Without the patch, events are logged for unsuccessful queries if atleast a single fetch is perfomed. This patch relaxes this guarantee to log events for any query that reaches the FINISHED state (rows are available to fetch by the client) and does not wait for a fetch to be performed. This simplifies the coordinator state machine by avoiding unnecessary synchronization. Added some test coverage for coordinator side code paths for writing lineages. fe specific lineage tests only verified the correctness of lineage created but did not test whether it was being flushed correctly to the disk. Change-Id: I639b9c1acb9806b29292cd85be2863688453ca2e --- M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-server.cc M be/src/service/impala-server.h A testdata/workloads/functional-query/queries/QueryTest/lineage.test M tests/common/impala_test_suite.py M tests/common/test_result_verifier.py M tests/custom_cluster/test_lineage.py M tests/unittests/test_file_parser.py M tests/util/test_file_parser.py 10 files changed, 6,016 insertions(+), 243 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/14143/11 -- To view, visit http://gerrit.cloudera.org:8080/14143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I639b9c1acb9806b29292cd85be2863688453ca2e Gerrit-Change-Number: 14143 Gerrit-PatchSet: 11 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen
[Impala-ASF-CR] IMPALA-8572: Log query events before unregister.
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/14143 )
Change subject: IMPALA-8572: Log query events before unregister.
..
Patch Set 11:
(5 comments)
Still not clear on the test failures. Addressing the comments and kicking off
another run while I look into it.
http://gerrit.cloudera.org:8080/#/c/14143/10/be/src/service/client-request-state.cc
File be/src/service/client-request-state.cc:
http://gerrit.cloudera.org:8080/#/c/14143/10/be/src/service/client-request-state.cc@749
PS10, Line 749: if (wait_thread_.get() != nullptr) {
> Should we drop the lock while joining on the wait thread? I feel like there
Done
http://gerrit.cloudera.org:8080/#/c/14143/10/be/src/service/client-request-state.cc@754
PS10, Line 754: // flush log events to audit authorization errors, if any.
> We could probably also drop the lock when calling this.
Done
http://gerrit.cloudera.org:8080/#/c/14143/10/be/src/service/client-request-state.cc@780
PS10, Line 780: if (wait_thread_.get() == nullptr) return;
> I think we can actually just delete this line, the next line is sufficient.
Done
http://gerrit.cloudera.org:8080/#/c/14143/10/be/src/service/client-request-state.cc@1395
PS10, Line 1395: Status status;
> Accessing query_status without acquiring a lock might not actually be safe
Done
http://gerrit.cloudera.org:8080/#/c/14143/10/be/src/service/client-request-state.cc@1426
PS10, Line 1426: }
> Should be a const TExecRequest& to avoid copying.
nice find.
--
To view, visit http://gerrit.cloudera.org:8080/14143
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I639b9c1acb9806b29292cd85be2863688453ca2e
Gerrit-Change-Number: 14143
Gerrit-PatchSet: 11
Gerrit-Owner: Bharath Vissapragada
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: radford nguyen
Gerrit-Comment-Date: Wed, 11 Sep 2019 00:51:25 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8933: Enforce ranger deny policy
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/14203 )
Change subject: IMPALA-8933: Enforce ranger deny policy
..
Patch Set 9:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14203/9/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java
File
fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java:
http://gerrit.cloudera.org:8080/#/c/14203/9/fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java@237
PS9, Line 237: request.getPrivilege() != Privilege.INSERT) {
So we don't need to worry about ALTER?
--
To view, visit http://gerrit.cloudera.org:8080/14203
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic60786cd81080feeb0bfcd92aa2be646ee8cb7da
Gerrit-Change-Number: 14203
Gerrit-PatchSet: 9
Gerrit-Owner: Kurt Deschler
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Wed, 11 Sep 2019 00:36:05 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8228: Ownership support for Ranger authz
Hello Austin Nobis, Fredy Wijaya, Todd Lipcon, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14106
to look at the new patch set (#7).
Change subject: IMPALA-8228: Ownership support for Ranger authz
..
IMPALA-8228: Ownership support for Ranger authz
Without this patch, explicit privileges are needed even
for owners of databases/tables to perform actions on them.
Example: 'user' is the owner of database 'foo'. To create
a table 't' under 'foo', 'user' needs to be granted a CREATE
privilege on 'foo'
That is unintuitive from a user POV since users expect owners
to have ALL privileges on the objects they own. This patch extends
that support to Impala's ranger authorization plugin.
Ranger natively supports the concept of ownership by letting the
callers pass the ownership context to RangerAccessResourceImpl.
This patch plumbs the owner information for the authorizables
(currently only supported for Tables / Databases) which is then
evaulated during authorization.
For the ownership based authorization to work, ranger-admin side
policy on {OWNER} user needs to be defined.
Testing: Added some unit-tests and e-e tests that cover scenarios
where ownership is used for authorization.
Caveat: Ownership is a part of HMS thrift object. Since we do not
aggressively load HMS schemas during start-up, coordinators with
cold caches can result in weird table listings due to lack of
metadata needed for verifying ownership. This should be fixed
separately to make the behavior more consistent and user friendly.
(Added comments in the code wherever necessary along with a test
to simulate this).
Change-Id: I737b7164a3e7afb9996b3402e6872effd663f7b4
---
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/CollectionTableRef.java
M fe/src/main/java/org/apache/impala/analysis/CopyTestCaseStmt.java
M fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/DropDbStmt.java
M fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java
M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
M fe/src/main/java/org/apache/impala/authorization/Authorizable.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizableColumn.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizableDb.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizableFactory.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizableTable.java
M
fe/src/main/java/org/apache/impala/authorization/DefaultAuthorizableFactory.java
M fe/src/main/java/org/apache/impala/authorization/PrivilegeRequestBuilder.java
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaResourceBuilder.java
M
fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizableFactory.java
M fe/src/main/java/org/apache/impala/catalog/BuiltinsDb.java
M fe/src/main/java/org/apache/impala/catalog/Db.java
M fe/src/main/java/org/apache/impala/catalog/FeDb.java
M fe/src/main/java/org/apache/impala/catalog/FeTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java
M tests/authorization/test_ranger.py
30 files changed, 541 insertions(+), 101 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/14106/7
--
To view, visit http://gerrit.cloudera.org:8080/14106
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I737b7164a3e7afb9996b3402e6872effd663f7b4
Gerrit-Change-Number: 14106
Gerrit-PatchSet: 7
Gerrit-Owner: Bharath Vissapragada
Gerrit-Reviewer: Austin Nobis
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8228: Ownership support for Ranger authz
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/14106 )
Change subject: IMPALA-8228: Ownership support for Ranger authz
..
Patch Set 7:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/14106/7/fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java
File fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java:
http://gerrit.cloudera.org:8080/#/c/14106/7/fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java@176
PS7, Line 176: builder.onTableUnknownOwner(dbName,
tableName_.getTbl()).allOf(Privilege.REFRESH).build());
> line too long (107 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/14106/7/fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java@179
PS7, Line 179: dbName, tableName_.getTbl(),
tbl.getOwnerUser()).allOf(Privilege.REFRESH).build());
> line too long (99 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/14106/7/fe/src/main/java/org/apache/impala/authorization/Authorizable.java
File fe/src/main/java/org/apache/impala/authorization/Authorizable.java:
http://gerrit.cloudera.org:8080/#/c/14106/7/fe/src/main/java/org/apache/impala/authorization/Authorizable.java@64
PS7, Line 64: public int hashCode() {
> line has trailing whitespace
Done
--
To view, visit http://gerrit.cloudera.org:8080/14106
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I737b7164a3e7afb9996b3402e6872effd663f7b4
Gerrit-Change-Number: 14106
Gerrit-PatchSet: 7
Gerrit-Owner: Bharath Vissapragada
Gerrit-Reviewer: Austin Nobis
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Todd Lipcon
Gerrit-Comment-Date: Wed, 11 Sep 2019 08:47:30 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8228: Ownership support for Ranger authz
Hello Austin Nobis, Fredy Wijaya, Todd Lipcon, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14106
to look at the new patch set (#8).
Change subject: IMPALA-8228: Ownership support for Ranger authz
..
IMPALA-8228: Ownership support for Ranger authz
Without this patch, explicit privileges are needed even
for owners of databases/tables to perform actions on them.
Example: 'user' is the owner of database 'foo'. To create
a table 't' under 'foo', 'user' needs to be granted a CREATE
privilege on 'foo'
That is unintuitive from a user POV since users expect owners
to have ALL privileges on the objects they own. This patch extends
that support to Impala's ranger authorization plugin.
Ranger natively supports the concept of ownership by letting the
callers pass the ownership context to RangerAccessResourceImpl.
This patch plumbs the owner information for the authorizables
(currently only supported for Tables / Databases) which is then
evaulated during authorization.
For the ownership based authorization to work, ranger-admin side
policy on {OWNER} user needs to be defined.
Testing: Added some unit-tests and e-e tests that cover scenarios
where ownership is used for authorization.
Caveat: Ownership is a part of HMS thrift object. Since we do not
aggressively load HMS schemas during start-up, coordinators with
cold caches can result in weird table listings due to lack of
metadata needed for verifying ownership. This should be fixed
separately to make the behavior more consistent and user friendly.
(Added comments in the code wherever necessary along with a test
to simulate this).
Change-Id: I737b7164a3e7afb9996b3402e6872effd663f7b4
---
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/CollectionTableRef.java
M fe/src/main/java/org/apache/impala/analysis/CopyTestCaseStmt.java
M fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/DropDbStmt.java
M fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java
M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
M fe/src/main/java/org/apache/impala/authorization/Authorizable.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizableColumn.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizableDb.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizableFactory.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizableTable.java
M
fe/src/main/java/org/apache/impala/authorization/DefaultAuthorizableFactory.java
M fe/src/main/java/org/apache/impala/authorization/PrivilegeRequestBuilder.java
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaResourceBuilder.java
M
fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizableFactory.java
M fe/src/main/java/org/apache/impala/catalog/BuiltinsDb.java
M fe/src/main/java/org/apache/impala/catalog/Db.java
M fe/src/main/java/org/apache/impala/catalog/FeDb.java
M fe/src/main/java/org/apache/impala/catalog/FeTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java
M tests/authorization/test_ranger.py
30 files changed, 543 insertions(+), 101 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/14106/8
--
To view, visit http://gerrit.cloudera.org:8080/14106
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I737b7164a3e7afb9996b3402e6872effd663f7b4
Gerrit-Change-Number: 14106
Gerrit-PatchSet: 8
Gerrit-Owner: Bharath Vissapragada
Gerrit-Reviewer: Austin Nobis
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8572: Log query events before unregister.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14143 ) Change subject: IMPALA-8572: Log query events before unregister. .. Patch Set 9: Yes, those mismatched results are definitely weird. -- To view, visit http://gerrit.cloudera.org:8080/14143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I639b9c1acb9806b29292cd85be2863688453ca2e Gerrit-Change-Number: 14143 Gerrit-PatchSet: 9 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Tue, 10 Sep 2019 03:34:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8572: Log query events before unregister.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14143 ) Change subject: IMPALA-8572: Log query events before unregister. .. Patch Set 9: The issue is that the lineage test does queries like following. It is changing the snapshot state as the test runs, facepalm. 04:01:31.431 insert into table functional.alltypessmall (smallint_col, int_col) 04:01:31.431 partition (year=2009, month=04) 04:01:31.431 select smallint_col, int_col 04:01:31.431 from functional.alltypes 04:01:31.431 where year=2009 and month=05; -- To view, visit http://gerrit.cloudera.org:8080/14143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I639b9c1acb9806b29292cd85be2863688453ca2e Gerrit-Change-Number: 14143 Gerrit-PatchSet: 9 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Tue, 10 Sep 2019 04:26:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8572: Skip test lineage output on S3
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14230 ) Change subject: IMPALA-8572: Skip test_lineage_output on S3 .. Patch Set 1: Thanks for the review. This test does not add any coverage specific to S3. The lineages are still written to a local directory and not S3. In a way it is redundant to run this with S3. -- To view, visit http://gerrit.cloudera.org:8080/14230 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I781c2dc42c042747eed6134cea4f3f0879a40294 Gerrit-Change-Number: 14230 Gerrit-PatchSet: 1 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Mon, 16 Sep 2019 17:04:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7322: Add storage wait time to profile
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13786 ) Change subject: IMPALA-7322: Add storage wait time to profile .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13786 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7447f8c8e7e50eb71d18643859d2e3de865368d2 Gerrit-Change-Number: 13786 Gerrit-PatchSet: 4 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Mon, 16 Sep 2019 17:39:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8572: Skip test lineage output on S3
Bharath Vissapragada has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14230 ) Change subject: IMPALA-8572: Skip test_lineage_output on S3 .. IMPALA-8572: Skip test_lineage_output on S3 The test has a hbase dependency that does not run in S3. Change-Id: I781c2dc42c042747eed6134cea4f3f0879a40294 Reviewed-on: http://gerrit.cloudera.org:8080/14230 Tested-by: Impala Public Jenkins Reviewed-by: Sahil Takiar --- M tests/custom_cluster/test_lineage.py 1 file changed, 2 insertions(+), 0 deletions(-) Approvals: Impala Public Jenkins: Verified Sahil Takiar: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/14230 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I781c2dc42c042747eed6134cea4f3f0879a40294 Gerrit-Change-Number: 14230 Gerrit-PatchSet: 2 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar
[Impala-ASF-CR] IMPALA-8228: Fix test ownership on CentOS6
Bharath Vissapragada has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14231 ) Change subject: IMPALA-8228: Fix _test_ownership on CentOS6 .. IMPALA-8228: Fix _test_ownership on CentOS6 String format() with positional formatting does not work without position argument specifiers in Python 2.6 that ships with CentOs6. Change-Id: I88cbfd18c963e1bd49b6af01a61c09fdf5f69157 Reviewed-on: http://gerrit.cloudera.org:8080/14231 Tested-by: Impala Public Jenkins Reviewed-by: Sahil Takiar --- M tests/authorization/test_ranger.py 1 file changed, 18 insertions(+), 18 deletions(-) Approvals: Impala Public Jenkins: Verified Sahil Takiar: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/14231 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I88cbfd18c963e1bd49b6af01a61c09fdf5f69157 Gerrit-Change-Number: 14231 Gerrit-PatchSet: 3 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar
[Impala-ASF-CR] IMPALA-8930: [DOCS] Object ownership support when integrated with Ranger
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/14229 )
Change subject: IMPALA-8930: [DOCS] Object ownership support when integrated
with Ranger
..
Patch Set 1:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/14229/1/docs/shared/impala_common.xml
File docs/shared/impala_common.xml:
http://gerrit.cloudera.org:8080/#/c/14229/1/docs/shared/impala_common.xml@123
PS1, Line 123:
This should be in Ranger integration section?
http://gerrit.cloudera.org:8080/#/c/14229/1/docs/shared/impala_common.xml@128
PS1, Line 128: includes
implies? (nit: extra space)
http://gerrit.cloudera.org:8080/#/c/14229/1/docs/topics/impala_authorization.xml
File docs/topics/impala_authorization.xml:
http://gerrit.cloudera.org:8080/#/c/14229/1/docs/topics/impala_authorization.xml@156
PS1, Line 156: ownership is enabled by default in Impala,
I think this should be rephrased.
Object ownership for tables, views and databases is enabled by default in
Impala. To define owner specific privileges, go to ranger UI and define
appropriate policies on {OWNER} user
http://gerrit.cloudera.org:8080/#/c/14229/1/docs/topics/impala_authorization.xml@162
PS1, Line 162: An owner has the OWNER privilege if enabled in
: Sentry.
remove.
http://gerrit.cloudera.org:8080/#/c/14229/1/docs/topics/impala_authorization.xml@173
PS1, Line 173:
We should also mention the caveat with Ranger ownership integration that "SHOW
TABLES" may not work as expected. I added it in the commit message [1]. TL;DR
"show tables" command cannot infer ownership information because the current
design of metadata lacks aggressive caching of ownership information. The user
behavior ends up like, show tables does not list the table even though a user
owns it unless it is fully loaded in the coordinator catalog cache IMPALA-8937.
Users can still do queries on the table (like select * from foo) because these
kinds of statements preload the table before performing the authz checks.
[1]
https://github.com/apache/impala/commit/ced6e98fb4c361efa4bcc7e5441ccdb8debba8e9
--
To view, visit http://gerrit.cloudera.org:8080/14229
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie4fdaf05953373c8d1870b7eface257830c7c6e5
Gerrit-Change-Number: 14229
Gerrit-PatchSet: 1
Gerrit-Owner: Alex Rodoni
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Mon, 16 Sep 2019 17:30:58 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8930: [DOCS] Object ownership support when integrated with Ranger
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14229 ) Change subject: IMPALA-8930: [DOCS] Object ownership support when integrated with Ranger .. Patch Set 3: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/14229/3/docs/shared/impala_common.xml File docs/shared/impala_common.xml: http://gerrit.cloudera.org:8080/#/c/14229/3/docs/shared/impala_common.xml@118 PS3, Line 118: Sentry update? -- To view, visit http://gerrit.cloudera.org:8080/14229 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie4fdaf05953373c8d1870b7eface257830c7c6e5 Gerrit-Change-Number: 14229 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 16 Sep 2019 20:09:16 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8930: [DOCS] Object ownership support when integrated with Ranger
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14229 ) Change subject: IMPALA-8930: [DOCS] Object ownership support when integrated with Ranger .. Patch Set 2: (2 comments) Couple of nits, lgtm otherwise. http://gerrit.cloudera.org:8080/#/c/14229/2/docs/shared/impala_common.xml File docs/shared/impala_common.xml: http://gerrit.cloudera.org:8080/#/c/14229/2/docs/shared/impala_common.xml@113 PS2, Line 113: : : Sentry-Related Content What about these? http://gerrit.cloudera.org:8080/#/c/14229/2/docs/topics/impala_authorization.xml File docs/topics/impala_authorization.xml: http://gerrit.cloudera.org:8080/#/c/14229/2/docs/topics/impala_authorization.xml@180 PS2, Line 180: . Refer the jira? -- To view, visit http://gerrit.cloudera.org:8080/14229 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie4fdaf05953373c8d1870b7eface257830c7c6e5 Gerrit-Change-Number: 14229 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 16 Sep 2019 18:34:31 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8930: [DOCS] Object ownership support when integrated with Ranger
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14229 ) Change subject: IMPALA-8930: [DOCS] Object ownership support when integrated with Ranger .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14229 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie4fdaf05953373c8d1870b7eface257830c7c6e5 Gerrit-Change-Number: 14229 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 16 Sep 2019 20:19:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8571: harden QueryEventHook execution
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13748 ) Change subject: IMPALA-8571: harden QueryEventHook execution .. Patch Set 25: I'll take a deeper look at this today. Thanks for your patience. -- To view, visit http://gerrit.cloudera.org:8080/13748 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643 Gerrit-Change-Number: 13748 Gerrit-PatchSet: 25 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Mon, 16 Sep 2019 18:40:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8921: Use short name for Ranger grant/revoke requests
Hello Quanlong Huang, Austin Nobis, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14185 to look at the new patch set (#2). Change subject: IMPALA-8921: Use short name for Ranger grant/revoke requests .. IMPALA-8921: Use short name for Ranger grant/revoke requests For certain grant/revoke Ranger commmands, we ended up passing the full name which is a problem when kerberos is enabled. Ranger expects the short name during authorization. Testing: We do not have test coverage with kerberos enabled, so I inspected the code manually to make sure we are using getShortName() everywhere. Change-Id: I3dc1bf55d50dc2e5fa6e07f16644f0a2773f2d23 --- M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java 2 files changed, 10 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/14185/2 -- To view, visit http://gerrit.cloudera.org:8080/14185 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3dc1bf55d50dc2e5fa6e07f16644f0a2773f2d23 Gerrit-Change-Number: 14185 Gerrit-PatchSet: 2 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-8921: Use short name for Ranger grant/revoke requests
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14185 ) Change subject: IMPALA-8921: Use short name for Ranger grant/revoke requests .. Patch Set 2: Code-Review+2 Carrying +2. -- To view, visit http://gerrit.cloudera.org:8080/14185 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3dc1bf55d50dc2e5fa6e07f16644f0a2773f2d23 Gerrit-Change-Number: 14185 Gerrit-PatchSet: 2 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Sat, 07 Sep 2019 19:17:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8921: Use short name for Ranger grant/revoke requests
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14185 ) Change subject: IMPALA-8921: Use short name for Ranger grant/revoke requests .. Patch Set 1: (3 comments) http://gerrit.cloudera.org:8080/#/c/14185/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/14185/1//COMMIT_MSG@7 PS1, Line 7: User > typo? Use? Done http://gerrit.cloudera.org:8080/#/c/14185/1/fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java File fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java: http://gerrit.cloudera.org:8080/#/c/14185/1/fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java@126 PS1, Line 126: new User(header.getRequesting_user()).getShortName(), true, params.getPrincipal_name(), > line too long (95 > 90) Done http://gerrit.cloudera.org:8080/#/c/14185/1/fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java@138 PS1, Line 138: new User(header.getRequesting_user()).getShortName(), false, params.getPrincipal_name(), > line too long (96 > 90) Done -- To view, visit http://gerrit.cloudera.org:8080/14185 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3dc1bf55d50dc2e5fa6e07f16644f0a2773f2d23 Gerrit-Change-Number: 14185 Gerrit-PatchSet: 1 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Sat, 07 Sep 2019 19:17:28 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8572: Log query events before unregister.
Hello radford nguyen, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14143 to look at the new patch set (#9). Change subject: IMPALA-8572: Log query events before unregister. .. IMPALA-8572: Log query events before unregister. Currently, the query events (audits and lineages) are logged as a part of query unregistration. This delays the event logging in cases where the Unregister() is delayed by client for some reason (ex: Hue does not call Unregister until the browser tab is closed) or the client goes away without calling Unregister and the query timeout kicks in. This patch moves this event logging to an earlier stage in the query lifecycle. Moved the event logging related code into ClientRequestState for easier code refactoring. The conditions under which the events are logged are slightly modified by this patch. Without the patch, events are logged for unsuccessful queries if atleast a single fetch is perfomed. This patch relaxes this guarantee to log events for any query that reaches the FINISHED state (rows are available to fetch by the client) and does not wait for a fetch to be performed. This simplifies the coordinator state machine by avoiding unnecessary synchronization. Added some test coverage for coordinator side code paths for writing lineages. fe specific lineage tests only verified the correctness of lineage created but did not test whether it was being flushed correctly to the disk. Change-Id: I639b9c1acb9806b29292cd85be2863688453ca2e --- M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-server.cc M be/src/service/impala-server.h A testdata/workloads/functional-query/queries/QueryTest/lineage.test M tests/common/impala_test_suite.py M tests/common/test_result_verifier.py M tests/custom_cluster/test_lineage.py M tests/unittests/test_file_parser.py M tests/util/test_file_parser.py 10 files changed, 6,003 insertions(+), 243 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/14143/9 -- To view, visit http://gerrit.cloudera.org:8080/14143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I639b9c1acb9806b29292cd85be2863688453ca2e Gerrit-Change-Number: 14143 Gerrit-PatchSet: 9 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen
[Impala-ASF-CR] IMPALA-8572: Log query events before unregister.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14143 ) Change subject: IMPALA-8572: Log query events before unregister. .. Patch Set 9: (7 comments) http://gerrit.cloudera.org:8080/#/c/14143/8/be/src/service/client-request-state.h File be/src/service/client-request-state.h: http://gerrit.cloudera.org:8080/#/c/14143/8/be/src/service/client-request-state.h@91 PS8, Line 91: /// Since this is a blocking operation, it is invoked from a separate thread > Mention that this log query events. Done http://gerrit.cloudera.org:8080/#/c/14143/8/be/src/service/client-request-state.h@96 PS8, Line 96: /// on Wait() are signalled *before* logging the events so that they can resume their > This comment needs to be updated, since we're waiting for wait_thread_ to s Done http://gerrit.cloudera.org:8080/#/c/14143/8/be/src/service/client-request-state.h@333 PS8, Line 333: > typo: signal Done http://gerrit.cloudera.org:8080/#/c/14143/8/be/src/service/client-request-state.h@335 PS8, Line 335: /// TODO: remove and use ExecEnv::GetInstance() instead > We could probably use a CountingBarrier instead of this pair of variables a Personally I feel like this is more readable but let me know if you feel strongly about using a barrier, I can switch. http://gerrit.cloudera.org:8080/#/c/14143/8/be/src/service/client-request-state.cc File be/src/service/client-request-state.cc: http://gerrit.cloudera.org:8080/#/c/14143/8/be/src/service/client-request-state.cc@811 PS8, Line 811: } > nit: the best practice is to signal the CV after dropping the log (otherwis Nice catch, done. http://gerrit.cloudera.org:8080/#/c/14143/8/be/src/service/impala-server.cc File be/src/service/impala-server.cc: http://gerrit.cloudera.org:8080/#/c/14143/8/be/src/service/impala-server.cc@a642 PS8, Line 642: > I ended up pulling out the before and after code into before.cc and after.c Thanks. http://gerrit.cloudera.org:8080/#/c/14143/8/testdata/workloads/functional-query/queries/QueryTest/lineage.test File testdata/workloads/functional-query/queries/QueryTest/lineage.test: http://gerrit.cloudera.org:8080/#/c/14143/8/testdata/workloads/functional-query/queries/QueryTest/lineage.test@268 PS8, Line 268: > Do we know that these unique database names are stable across different sys Yes, this is the reason the precommit failed. Fixed it. -- To view, visit http://gerrit.cloudera.org:8080/14143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I639b9c1acb9806b29292cd85be2863688453ca2e Gerrit-Change-Number: 14143 Gerrit-PatchSet: 9 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Sat, 07 Sep 2019 21:13:09 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8572: Log query events before unregister.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14143 ) Change subject: IMPALA-8572: Log query events before unregister. .. Patch Set 8: (1 comment) http://gerrit.cloudera.org:8080/#/c/14143/8/testdata/workloads/functional-query/queries/QueryTest/lineage.test File testdata/workloads/functional-query/queries/QueryTest/lineage.test: http://gerrit.cloudera.org:8080/#/c/14143/8/testdata/workloads/functional-query/queries/QueryTest/lineage.test@268 PS8, Line 268: test_haha_e4f3f13c > Yes, this is the reason the precommit failed. Fixed it. Sorry, I meant you caught the issue correctly. I fixed it. -- To view, visit http://gerrit.cloudera.org:8080/14143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I639b9c1acb9806b29292cd85be2863688453ca2e Gerrit-Change-Number: 14143 Gerrit-PatchSet: 8 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Sat, 07 Sep 2019 21:15:56 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8931: Fix fe trigger for lineage events
Bharath Vissapragada has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14194 Change subject: IMPALA-8931: Fix fe trigger for lineage events .. IMPALA-8931: Fix fe trigger for lineage events Currently, fe generates lineage objects only when --lineage_event_log_dir is configured. This is a legacy startup param. Lineages should also be generated when event hooks are configured since they can potentially consume them. Added a test that confirms that the hook is invoked when a lineage is created. Change-Id: I2d8deb05883cc3ecab27fe4afd031a1e7ccb0829 --- M fe/src/main/java/org/apache/impala/service/BackendConfig.java M tests/custom_cluster/test_query_event_hooks.py 2 files changed, 13 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/94/14194/1 -- To view, visit http://gerrit.cloudera.org:8080/14194 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I2d8deb05883cc3ecab27fe4afd031a1e7ccb0829 Gerrit-Change-Number: 14194 Gerrit-PatchSet: 1 Gerrit-Owner: Bharath Vissapragada
[Impala-ASF-CR] IMPALA-8931: Fix fe trigger for lineage events
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14194 to look at the new patch set (#2). Change subject: IMPALA-8931: Fix fe trigger for lineage events .. IMPALA-8931: Fix fe trigger for lineage events Currently, fe generates lineage objects only when --lineage_event_log_dir is configured. This is a legacy startup param. Lineages should also be generated when event hooks are configured since they can potentially consume them. Added a test that confirms that the hook is invoked when a lineage is created. Change-Id: I2d8deb05883cc3ecab27fe4afd031a1e7ccb0829 --- M fe/src/main/java/org/apache/impala/service/BackendConfig.java M tests/custom_cluster/test_query_event_hooks.py 2 files changed, 13 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/94/14194/2 -- To view, visit http://gerrit.cloudera.org:8080/14194 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2d8deb05883cc3ecab27fe4afd031a1e7ccb0829 Gerrit-Change-Number: 14194 Gerrit-PatchSet: 2 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8923: remove synchronized in HBaseTable.getEstimatedRowStats
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14188 ) Change subject: IMPALA-8923: remove synchronized in HBaseTable.getEstimatedRowStats .. Patch Set 3: Code-Review+2 (2 comments) Feel free to carry +2 http://gerrit.cloudera.org:8080/#/c/14188/3/fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java File fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java: http://gerrit.cloudera.org:8080/#/c/14188/3/fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java@439 PS3, Line 439: timeUsed nit: timeElapsed? http://gerrit.cloudera.org:8080/#/c/14188/3/fe/src/main/java/org/apache/impala/catalog/FeHBaseTable.java@672 PS3, Line 672: timeUsed same. -- To view, visit http://gerrit.cloudera.org:8080/14188 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifa23c16ee662c4f22851c700aea2ea5be847b64d Gerrit-Change-Number: 14188 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Sat, 07 Sep 2019 01:02:40 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8228: Ownership support for Ranger authz
Hello Austin Nobis, Todd Lipcon, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14106
to look at the new patch set (#6).
Change subject: IMPALA-8228: Ownership support for Ranger authz
..
IMPALA-8228: Ownership support for Ranger authz
Without this patch, explicit privileges are needed even
for owners of databases/tables to perform actions on them.
Example: 'user' is the owner of database 'foo'. To create
a table 't' under 'foo', 'user' needs to be granted a CREATE
privilege on 'foo'
That is unintuitive from a user POV since users expect owners
to have ALL privileges on the objects they own. This patch extends
that support to Impala's ranger authorization plugin.
Ranger natively supports the concept of ownership by letting the
callers pass the ownership context to RangerAccessResourceImpl.
This patch plumbs the owner information for the authorizables
(currently only supported for Tables / Databases) which is then
evaulated during authorization.
For the ownership based authorization to work, ranger-admin side
policy on {OWNER} user needs to be defined.
Testing: Added some unit-tests and e-e tests that cover scenarios
where ownership is used for authorization.
Caveat: Ownership is a part of HMS thrift object. Since we do not
aggressively load HMS schemas during start-up, coordinators with
cold caches can result in weird table listings due to lack of
metadata needed for verifying ownership. This should be fixed
separately to make the behavior more consistent and user friendly.
(Added comments in the code wherever necessary along with a test
to simulate this).
Change-Id: I737b7164a3e7afb9996b3402e6872effd663f7b4
---
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/CollectionTableRef.java
M fe/src/main/java/org/apache/impala/analysis/CopyTestCaseStmt.java
M fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/DropDbStmt.java
M fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java
M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
M fe/src/main/java/org/apache/impala/authorization/Authorizable.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizableColumn.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizableDb.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizableFactory.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizableTable.java
M
fe/src/main/java/org/apache/impala/authorization/DefaultAuthorizableFactory.java
M fe/src/main/java/org/apache/impala/authorization/PrivilegeRequestBuilder.java
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaResourceBuilder.java
M
fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizableFactory.java
M fe/src/main/java/org/apache/impala/catalog/BuiltinsDb.java
M fe/src/main/java/org/apache/impala/catalog/Db.java
M fe/src/main/java/org/apache/impala/catalog/FeDb.java
M fe/src/main/java/org/apache/impala/catalog/FeTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java
M tests/authorization/test_ranger.py
30 files changed, 523 insertions(+), 96 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/14106/6
--
To view, visit http://gerrit.cloudera.org:8080/14106
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I737b7164a3e7afb9996b3402e6872effd663f7b4
Gerrit-Change-Number: 14106
Gerrit-PatchSet: 6
Gerrit-Owner: Bharath Vissapragada
Gerrit-Reviewer: Austin Nobis
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8228: Ownership support for Ranger authz
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14106 ) Change subject: IMPALA-8228: Ownership support for Ranger authz .. Patch Set 6: Todd, this is ready for review. Added tests. I think the code can be refactored better to make it easy to understand and i'm looking into it. Meanwhile pushing it out for review. -- To view, visit http://gerrit.cloudera.org:8080/14106 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I737b7164a3e7afb9996b3402e6872effd663f7b4 Gerrit-Change-Number: 14106 Gerrit-PatchSet: 6 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 09 Sep 2019 15:40:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7975(continued): Improve supportability of the automatic invalidate feature.
Bharath Vissapragada has abandoned this change. ( http://gerrit.cloudera.org:8080/13947 ) Change subject: IMPALA-7975(continued): Improve supportability of the automatic invalidate feature. .. Abandoned This was submitted as a draft and the original author is not active anymore. Abandoning. -- To view, visit http://gerrit.cloudera.org:8080/13947 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: Ib113a5b9458dcf483b183e927544a6c6d46e1af3 Gerrit-Change-Number: 13947 Gerrit-PatchSet: 4 Gerrit-Owner: Sharanitha Harish Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8932: shell shouldn't retry kerberos over http
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14201 ) Change subject: IMPALA-8932: shell shouldn't retry kerberos over http .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14201 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5dde277a6a0ddbe5a919bcf376bbc19f0b48e95e Gerrit-Change-Number: 14201 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Mon, 09 Sep 2019 23:57:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8572: Log query events before unregister.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14143 ) Change subject: IMPALA-8572: Log query events before unregister. .. Patch Set 9: I didn't get a chance to debug it but my theory at this point is that my local data (and schema) snapshot with which I generated the test file could be stale. I'll take a look later today. Thanks for the reviews. -- To view, visit http://gerrit.cloudera.org:8080/14143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I639b9c1acb9806b29292cd85be2863688453ca2e Gerrit-Change-Number: 14143 Gerrit-PatchSet: 9 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Tue, 10 Sep 2019 00:18:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8761: Improve events processor configuration validation.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14240 ) Change subject: IMPALA-8761: Improve events processor configuration validation. .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14240 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I73480872ef93215d05c1fd922e64eb68a8a63a42 Gerrit-Change-Number: 14240 Gerrit-PatchSet: 3 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 18 Sep 2019 21:36:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8761: Improve events processor configuration validation.
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/14240 )
Change subject: IMPALA-8761: Improve events processor configuration validation.
..
Patch Set 2: Code-Review+2
(4 comments)
http://gerrit.cloudera.org:8080/#/c/14240/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/14240/2//COMMIT_MSG@9
PS2, Line 9: This patch aims to improve the validation of configuration keys
from the HMS.
nit: line wrap
http://gerrit.cloudera.org:8080/#/c/14240/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
File
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java:
http://gerrit.cloudera.org:8080/#/c/14240/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@267
PS2, Line 267: failedValidationResults
nit: validationErrors?
http://gerrit.cloudera.org:8080/#/c/14240/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@273
PS2, Line 273: if (!result.isValid()) {
nit: single line.
http://gerrit.cloudera.org:8080/#/c/14240/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@282
PS2, Line 282: failedValidationResults.size() > 0
nit: !validationErrors.isEmpty() (more concise)
--
To view, visit http://gerrit.cloudera.org:8080/14240
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I73480872ef93215d05c1fd922e64eb68a8a63a42
Gerrit-Change-Number: 14240
Gerrit-PatchSet: 2
Gerrit-Owner: Anurag Mantripragada
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Wed, 18 Sep 2019 18:50:25 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8944: Update and re-enable S3PlannerTest
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14248 ) Change subject: IMPALA-8944: Update and re-enable S3PlannerTest .. Patch Set 1: Code-Review+1 (1 comment) I'll let Joe take another look. http://gerrit.cloudera.org:8080/#/c/14248/1/fe/src/test/java/org/apache/impala/planner/PlannerTest.java File fe/src/test/java/org/apache/impala/planner/PlannerTest.java: http://gerrit.cloudera.org:8080/#/c/14248/1/fe/src/test/java/org/apache/impala/planner/PlannerTest.java@57 PS1, Line 57: HBaseTestDataRegionAssignment assignment = new HBaseTestDataRegionAssignment(); nit: Add a quick commit why this is needed? Kinda not obvious. -- To view, visit http://gerrit.cloudera.org:8080/14248 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1690b6c5346376cfd4845c72062cc237e0f9 Gerrit-Change-Number: 14248 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 18 Sep 2019 18:42:33 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8944: Update and re-enable S3PlannerTest
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14248 ) Change subject: IMPALA-8944: Update and re-enable S3PlannerTest .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14248 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1690b6c5346376cfd4845c72062cc237e0f9 Gerrit-Change-Number: 14248 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 18 Sep 2019 19:01:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8228: Fix test ownership for CentOS6
Bharath Vissapragada has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14231 Change subject: IMPALA-8228: Fix _test_ownership for CentOS6 .. IMPALA-8228: Fix _test_ownership for CentOS6 String format() with positional formatting does not work without position argument specifiers in Python 2.6 that ships with CentOs6. Change-Id: I88cbfd18c963e1bd49b6af01a61c09fdf5f69157 --- M tests/authorization/test_ranger.py 1 file changed, 17 insertions(+), 17 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/14231/1 -- To view, visit http://gerrit.cloudera.org:8080/14231 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I88cbfd18c963e1bd49b6af01a61c09fdf5f69157 Gerrit-Change-Number: 14231 Gerrit-PatchSet: 1 Gerrit-Owner: Bharath Vissapragada
[Impala-ASF-CR] IMPALA-8572: Skip test lineage output on S3
Bharath Vissapragada has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14230 Change subject: IMPALA-8572: Skip test_lineage_output on S3 .. IMPALA-8572: Skip test_lineage_output on S3 The test has a hbase dependency that does not run in S3. Change-Id: I781c2dc42c042747eed6134cea4f3f0879a40294 --- M tests/custom_cluster/test_lineage.py 1 file changed, 2 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/30/14230/1 -- To view, visit http://gerrit.cloudera.org:8080/14230 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I781c2dc42c042747eed6134cea4f3f0879a40294 Gerrit-Change-Number: 14230 Gerrit-PatchSet: 1 Gerrit-Owner: Bharath Vissapragada
[Impala-ASF-CR] IMPALA-8228: Fix test ownership on CentOS6
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14231 to look at the new patch set (#2). Change subject: IMPALA-8228: Fix _test_ownership on CentOS6 .. IMPALA-8228: Fix _test_ownership on CentOS6 String format() with positional formatting does not work without position argument specifiers in Python 2.6 that ships with CentOs6. Change-Id: I88cbfd18c963e1bd49b6af01a61c09fdf5f69157 --- M tests/authorization/test_ranger.py 1 file changed, 18 insertions(+), 18 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/14231/2 -- To view, visit http://gerrit.cloudera.org:8080/14231 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I88cbfd18c963e1bd49b6af01a61c09fdf5f69157 Gerrit-Change-Number: 14231 Gerrit-PatchSet: 2 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7975(continued): Improve supportability of the automatic invalidate feature.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14200 ) Change subject: IMPALA-7975(continued): Improve supportability of the automatic invalidate feature. .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14200 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iadab4a6cd3ad3c6c1b7931747ac55cd58ce3bb3d Gerrit-Change-Number: 14200 Gerrit-PatchSet: 1 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 13 Sep 2019 22:08:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8572: Log query events before unregister.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14143 ) Change subject: IMPALA-8572: Log query events before unregister. .. Patch Set 8: There was a test failure with mismatched lineage output that I'm looking into, but that does not seem to be related to the core life cycle changes. So, this is ready for review. -- To view, visit http://gerrit.cloudera.org:8080/14143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I639b9c1acb9806b29292cd85be2863688453ca2e Gerrit-Change-Number: 14143 Gerrit-PatchSet: 8 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Fri, 06 Sep 2019 15:12:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8228: Ownership support for Ranger authz
Hello Austin Nobis, Fredy Wijaya, Todd Lipcon, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14106
to look at the new patch set (#9).
Change subject: IMPALA-8228: Ownership support for Ranger authz
..
IMPALA-8228: Ownership support for Ranger authz
Without this patch, explicit privileges are needed even
for owners of databases/tables to perform actions on them.
Example: 'user' is the owner of database 'foo'. To create
a table 't' under 'foo', 'user' needs to be granted a CREATE
privilege on 'foo'
That is unintuitive from a user POV since users expect owners
to have ALL privileges on the objects they own. This patch extends
that support to Impala's ranger authorization plugin.
Ranger natively supports the concept of ownership by letting the
callers pass the ownership context to RangerAccessResourceImpl.
This patch plumbs the owner information for the authorizables
(currently only supported for Tables / Databases) which is then
evaulated during authorization.
For the ownership based authorization to work, ranger-admin side
policy on {OWNER} user needs to be defined.
Testing: Added some unit-tests and e-e tests that cover scenarios
where ownership is used for authorization.
Caveat: Ownership is a part of HMS thrift object. Since we do not
aggressively load HMS schemas during start-up, coordinators with
cold caches can result in weird table listings due to lack of
metadata needed for verifying ownership. This should be fixed
separately to make the behavior more consistent and user friendly.
(Added comments in the code wherever necessary along with a test
to simulate this).
Change-Id: I737b7164a3e7afb9996b3402e6872effd663f7b4
---
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/CollectionTableRef.java
M fe/src/main/java/org/apache/impala/analysis/CopyTestCaseStmt.java
M fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/DropDbStmt.java
M fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java
M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
M fe/src/main/java/org/apache/impala/authorization/Authorizable.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizableColumn.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizableDb.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizableFactory.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizableTable.java
M
fe/src/main/java/org/apache/impala/authorization/DefaultAuthorizableFactory.java
M fe/src/main/java/org/apache/impala/authorization/PrivilegeRequestBuilder.java
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaResourceBuilder.java
M
fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizableFactory.java
M fe/src/main/java/org/apache/impala/catalog/BuiltinsDb.java
M fe/src/main/java/org/apache/impala/catalog/Db.java
M fe/src/main/java/org/apache/impala/catalog/FeDb.java
M fe/src/main/java/org/apache/impala/catalog/FeTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java
M tests/authorization/test_ranger.py
30 files changed, 562 insertions(+), 101 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/14106/9
--
To view, visit http://gerrit.cloudera.org:8080/14106
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I737b7164a3e7afb9996b3402e6872effd663f7b4
Gerrit-Change-Number: 14106
Gerrit-PatchSet: 9
Gerrit-Owner: Bharath Vissapragada
Gerrit-Reviewer: Austin Nobis
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8228: Ownership support for Ranger authz
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14106 ) Change subject: IMPALA-8228: Ownership support for Ranger authz .. Patch Set 9: Added a delete-retry logic in creation of test policies. -- To view, visit http://gerrit.cloudera.org:8080/14106 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I737b7164a3e7afb9996b3402e6872effd663f7b4 Gerrit-Change-Number: 14106 Gerrit-PatchSet: 9 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 11 Sep 2019 16:49:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8940: Fix MetastoreEventsProcessorTest.testPartitionEvents in CDP builds
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14213 ) Change subject: IMPALA-8940: Fix MetastoreEventsProcessorTest.testPartitionEvents in CDP builds .. Patch Set 1: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/14213/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java: http://gerrit.cloudera.org:8080/#/c/14213/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@1382 PS1, Line 1382: if (val == null) parametersAfter.remove(parameter); nit: formatting, don't think we use single line if else -- To view, visit http://gerrit.cloudera.org:8080/14213 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie93e7237944142f616a80dec675280b397fe5995 Gerrit-Change-Number: 14213 Gerrit-PatchSet: 1 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 11 Sep 2019 16:53:31 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8228: Ownership support for Ranger authz
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14106 ) Change subject: IMPALA-8228: Ownership support for Ranger authz .. Patch Set 8: The test fails because of the remnants from earlier tests that granted privileges on the same functional.*.*. I'm looking into fixing it. -- To view, visit http://gerrit.cloudera.org:8080/14106 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I737b7164a3e7afb9996b3402e6872effd663f7b4 Gerrit-Change-Number: 14106 Gerrit-PatchSet: 8 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 11 Sep 2019 16:00:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8572: Log query events before unregister.
Hello radford nguyen, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14143 to look at the new patch set (#10). Change subject: IMPALA-8572: Log query events before unregister. .. IMPALA-8572: Log query events before unregister. Currently, the query events (audits and lineages) are logged as a part of query unregistration. This delays the event logging in cases where the Unregister() is delayed by client for some reason (ex: Hue does not call Unregister until the browser tab is closed) or the client goes away without calling Unregister and the query timeout kicks in. This patch moves this event logging to an earlier stage in the query lifecycle. Moved the event logging related code into ClientRequestState for easier code refactoring. The conditions under which the events are logged are slightly modified by this patch. Without the patch, events are logged for unsuccessful queries if atleast a single fetch is perfomed. This patch relaxes this guarantee to log events for any query that reaches the FINISHED state (rows are available to fetch by the client) and does not wait for a fetch to be performed. This simplifies the coordinator state machine by avoiding unnecessary synchronization. Added some test coverage for coordinator side code paths for writing lineages. fe specific lineage tests only verified the correctness of lineage created but did not test whether it was being flushed correctly to the disk. Change-Id: I639b9c1acb9806b29292cd85be2863688453ca2e --- M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-server.cc M be/src/service/impala-server.h A testdata/workloads/functional-query/queries/QueryTest/lineage.test M tests/common/impala_test_suite.py M tests/common/test_result_verifier.py M tests/custom_cluster/test_lineage.py M tests/unittests/test_file_parser.py M tests/util/test_file_parser.py 10 files changed, 6,009 insertions(+), 243 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/14143/10 -- To view, visit http://gerrit.cloudera.org:8080/14143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I639b9c1acb9806b29292cd85be2863688453ca2e Gerrit-Change-Number: 14143 Gerrit-PatchSet: 10 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen
[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/13932 )
Change subject: IMPALA-8661 : Add randomized tests to stress
MetastoreEventsProcessor
..
Patch Set 5:
(14 comments)
Great work, exhaustive coverage. I skimmed through most parts of the patch.
seems reasonable to me. I think it is difficult to review the entire patch in
a single go. We should probably commit it and keep fixing any issues
incrementally. Since most of the patch is isolated to test framework, any bugs
(like connection leaks, if any) shouldn't affect the product itself.
http://gerrit.cloudera.org:8080/#/c/13932/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13932/5//COMMIT_MSG@35
PS5, Line 35: 1. Ran the test with defaults. It generates about 2100 events
Should we consider lower defaults for core tests? 15mins seems like a
considerable increase in test execution time.
http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:
http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@299
PS5, Line 299: infoLog("Notification event received");
nit: isn't this too chatty for info?
http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/test/java/org/apache/impala/catalog/events/EventsProcessorStressTest.java
File
fe/src/test/java/org/apache/impala/catalog/events/EventsProcessorStressTest.java:
http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/test/java/org/apache/impala/catalog/events/EventsProcessorStressTest.java@46
PS5, Line 46: refresh
nit: grammar
http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/test/java/org/apache/impala/catalog/events/EventsProcessorStressTest.java@83
PS5, Line 83: else
do we want to override numClients in 3.x? in which case it should probably be
something like
if (version < 3 && numClients != null)...
http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
File
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java:
http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java@138
PS5, Line 138: import org.mockito.Mockito;
remove?
http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/test/java/org/apache/impala/testutil/HiveJdbcClientPool.java
File fe/src/test/java/org/apache/impala/testutil/HiveJdbcClientPool.java:
http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/test/java/org/apache/impala/testutil/HiveJdbcClientPool.java@28
PS5, Line 28: import java.util.concurrent.BlockingQueue;
nit: duplicate.
http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/test/java/org/apache/impala/testutil/HiveJdbcClientPool.java@36
PS5, Line 36: public class HiveJdbcClientPool implements Closeable {
class comment.
http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/test/java/org/apache/impala/testutil/HiveJdbcClientPool.java@71
PS5, Line 71: if (conn_ == null) {
: throw new RuntimeException("Connection not initialized.");
: } else if (conn_.isClosed()) {
: throw new RuntimeException("Connection not open.");
: }
Make them preconditions?
http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/test/java/org/apache/impala/testutil/HiveJdbcClientPool.java@94
PS5, Line 94: public boolean executeSql(String sql) throws SQLException {
method doc.
http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/test/java/org/apache/impala/testutil/HiveJdbcClientPool.java@146
PS5, Line 146: closedCount > 0
!freeClients_.isEmpty() ?
http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/test/java/org/apache/impala/testutil/TestUtils.java
File fe/src/test/java/org/apache/impala/testutil/TestUtils.java:
http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/test/java/org/apache/impala/testutil/TestUtils.java@42
PS5, Line 42: import org.apache.commons.lang3.RandomStringUtils;
remove?
http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/test/java/org/apache/impala/testutil/TestUtils.java@77
PS5, Line 77: DEFAULT_CONNECTION_TEMPLATE
nit:HS2_CONNECTION_TEMPLATE?
http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/test/java/org/apache/impala/util/RandomHiveQueryRunner.java
File fe/src/test/java/org/apache/impala/util/RandomHiveQueryRunner.java:
http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/test/java/org/apache/impala/util/RandomHiveQueryRunner.java@117
PS5, Line 117: for
nit: remove
http://gerrit.cloudera.org:8080/#/c/13932/5/fe/src/test/java/org/apache/impala/util/RandomHiveQueryRunner.java@345
PS5, Line 345: exists
exist
--
To view, visit http://gerrit.cloudera.org:8080/13932
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
[Impala-ASF-CR] IMPALA-8456: [DOCS] New HTTP protocol for Impala clients
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13960 ) Change subject: IMPALA-8456: [DOCS] New HTTP protocol for Impala clients .. Patch Set 2: Oops, I missed that commit. Thanks for the pointer. I think it is still useful to mention that the server endpoint supports basic and spnego auth. -- To view, visit http://gerrit.cloudera.org:8080/13960 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3101f8babc77a5a872778499a54ac479a66ad996 Gerrit-Change-Number: 13960 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 31 Jul 2019 21:38:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8456: [DOCS] New HTTP protocol for Impala clients
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13960 ) Change subject: IMPALA-8456: [DOCS] New HTTP protocol for Impala clients .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13960 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3101f8babc77a5a872778499a54ac479a66ad996 Gerrit-Change-Number: 13960 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 01 Aug 2019 01:53:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13932 ) Change subject: IMPALA-8661 : Add randomized tests to stress MetastoreEventsProcessor .. Patch Set 6: (1 comment) lgtm pending the test timings. http://gerrit.cloudera.org:8080/#/c/13932/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13932/5//COMMIT_MSG@35 PS5, Line 35: 1. Ran the test with defaults. It generates about 2100 events > I think the fe tests are not the long pole in the test execution, right? So Doesn't this job run everything serially? be tests -> fe-tests->e-e tests https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/6891/consoleFull (just picked up random job from verify-dry-run) -- To view, visit http://gerrit.cloudera.org:8080/13932 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8c85b83efd4f56b5ae0e8d1dc6a2ee2feb6721ce Gerrit-Change-Number: 13932 Gerrit-PatchSet: 6 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 01 Aug 2019 02:24:16 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8668: [DOCS] HS2 Support for Impala-Shell connection
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13961 ) Change subject: IMPALA-8668: [DOCS] HS2 Support for Impala-Shell connection .. Patch Set 2: (4 comments) http://gerrit.cloudera.org:8080/#/c/13961/2/docs/topics/impala_connecting.xml File docs/topics/impala_connecting.xml: http://gerrit.cloudera.org:8080/#/c/13961/2/docs/topics/impala_connecting.xml@85 PS2, Line 85: # When you are logged onto a different host, and impalad is listening I think you mean, "connecting to an impalad running on a remote machine".. or something like that. I don't think its logged onto a different host... http://gerrit.cloudera.org:8080/#/c/13961/2/docs/topics/impala_connecting.xml@87 PS2, Line 87: -i -i should be before the http://gerrit.cloudera.org:8080/#/c/13961/2/docs/topics/impala_connecting.xml@87 PS2, Line 87: -p -- (double dash) http://gerrit.cloudera.org:8080/#/c/13961/2/docs/topics/impala_shell_options.xml File docs/topics/impala_shell_options.xml: http://gerrit.cloudera.org:8080/#/c/13961/2/docs/topics/impala_shell_options.xml@573 PS2, Line 573: .< ..protocol. (same below too) -- To view, visit http://gerrit.cloudera.org:8080/13961 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie4e207d247409ac3069ce8a235be3f63e616007e Gerrit-Change-Number: 13961 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 01 Aug 2019 02:41:43 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8456: [DOCS] New HTTP protocol for Impala clients
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13960 ) Change subject: IMPALA-8456: [DOCS] New HTTP protocol for Impala clients .. Patch Set 2: Code-Review-1 Forgot about this. I think we should talk about authentication semantics over the new HTTP protocol. We support HTTP Basic auth. SPNEGO is not yet supported. -- To view, visit http://gerrit.cloudera.org:8080/13960 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3101f8babc77a5a872778499a54ac479a66ad996 Gerrit-Change-Number: 13960 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 31 Jul 2019 19:53:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8456: [DOCS] New HTTP protocol for Impala clients
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13960 ) Change subject: IMPALA-8456: [DOCS] New HTTP protocol for Impala clients .. Patch Set 2: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/13960/2/docs/topics/impala_ports.xml File docs/topics/impala_ports.xml: http://gerrit.cloudera.org:8080/#/c/13960/2/docs/topics/impala_ports.xml@401 PS2, Line 401: . ..using HiveServer2 protocol. -- To view, visit http://gerrit.cloudera.org:8080/13960 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3101f8babc77a5a872778499a54ac479a66ad996 Gerrit-Change-Number: 13960 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 31 Jul 2019 19:50:39 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8717: impala-shell support for HS2 HTTP endpoint
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13746 ) Change subject: IMPALA-8717: impala-shell support for HS2 HTTP endpoint .. Patch Set 16: Code-Review+2 Carrying +2. -- To view, visit http://gerrit.cloudera.org:8080/13746 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8323950857dfe1c1dfd5377fde79f87bc2ce9534 Gerrit-Change-Number: 13746 Gerrit-PatchSet: 16 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 29 Jul 2019 05:43:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8717: impala-shell support for HS2 HTTP endpoint
Bharath Vissapragada has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13746 ) Change subject: IMPALA-8717: impala-shell support for HS2 HTTP endpoint .. IMPALA-8717: impala-shell support for HS2 HTTP endpoint Adds impala-shell support to connect to HiveServer2 HTTP endpoint. Relies on toolchain change at https://gerrit.cloudera.org/#/c/13725/. Use --protocol='hs2-http' to enable this behavior. Example usages: --- impala-shell --protocol='hs2-http' (No auth) impala-shell --protocol='hs2-http' --ldap -u. (PLAIN auth) impala-shell --protocol-'hs2-http' --ssl --ca_cert... (TLS) impala-shell --protocol='hs2-http' --ldap --ssl --ca_cert... (LDAP + TLS) Limitations: --- - Does not support Kerberos (-k) due to lack ot SPNEGO support. Testing: - Parameterized existing shell tests to support this combination. - Added shell test coverage for LDAP auth. Change-Id: I8323950857dfe1c1dfd5377fde79f87bc2ce9534 Reviewed-on: http://gerrit.cloudera.org:8080/13746 Tested-by: Impala Public Jenkins Reviewed-by: Bharath Vissapragada --- M be/src/service/impala-server.cc M bin/impala-config.sh A fe/src/test/java/org/apache/impala/customcluster/LdapImpalaShellTest.java M shell/impala_client.py M shell/impala_shell.py M shell/option_parser.py M tests/common/impala_cluster.py M tests/common/impala_connection.py M tests/common/impala_service.py M tests/common/impala_test_suite.py M tests/common/test_dimensions.py M tests/conftest.py M tests/custom_cluster/test_client_ssl.py M tests/custom_cluster/test_shell_interactive.py M tests/custom_cluster/test_shell_interactive_reconnect.py M tests/run-tests.py M tests/shell/test_shell_commandline.py M tests/shell/test_shell_interactive.py M tests/shell/util.py 19 files changed, 326 insertions(+), 62 deletions(-) Approvals: Impala Public Jenkins: Verified Bharath Vissapragada: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/13746 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I8323950857dfe1c1dfd5377fde79f87bc2ce9534 Gerrit-Change-Number: 13746 Gerrit-PatchSet: 17 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8717: impala-shell support for HS2 HTTP endpoint
Hello Thomas Tauber-Marshall, David Knupp, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13746 to look at the new patch set (#15). Change subject: IMPALA-8717: impala-shell support for HS2 HTTP endpoint .. IMPALA-8717: impala-shell support for HS2 HTTP endpoint Adds impala-shell support to connect to HiveServer2 HTTP endpoint. Relies on toolchain change at https://gerrit.cloudera.org/#/c/13725/. Use --protocol='hs2-http' to enable this behavior. Example usages: --- impala-shell --protocol='hs2-http' (No auth) impala-shell --protocol='hs2-http' --ldap -u. (PLAIN auth) impala-shell --protocol-'hs2-http' --ssl --ca_cert... (TLS) impala-shell --protocol='hs2-http' --ldap --ssl --ca_cert... (LDAP + TLS) Limitations: --- - Does not support Kerberos (-k) due to lack ot SPNEGO support. Testing: - Parameterized existing shell tests to support this combination. - Added shell test coverage for LDAP auth. Change-Id: I8323950857dfe1c1dfd5377fde79f87bc2ce9534 --- M be/src/service/impala-server.cc M bin/impala-config.sh A fe/src/test/java/org/apache/impala/customcluster/LdapImpalaShellTest.java M shell/impala_client.py M shell/impala_shell.py M shell/option_parser.py M tests/common/impala_cluster.py M tests/common/impala_connection.py M tests/common/impala_service.py M tests/common/impala_test_suite.py M tests/common/test_dimensions.py M tests/conftest.py M tests/custom_cluster/test_client_ssl.py M tests/custom_cluster/test_shell_interactive.py M tests/custom_cluster/test_shell_interactive_reconnect.py M tests/run-tests.py M tests/shell/test_shell_commandline.py M tests/shell/test_shell_interactive.py M tests/shell/util.py 19 files changed, 326 insertions(+), 62 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/46/13746/15 -- To view, visit http://gerrit.cloudera.org:8080/13746 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8323950857dfe1c1dfd5377fde79f87bc2ce9534 Gerrit-Change-Number: 13746 Gerrit-PatchSet: 15 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8717: impala-shell support for HS2 HTTP endpoint
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13746 ) Change subject: IMPALA-8717: impala-shell support for HS2 HTTP endpoint .. Patch Set 15: Fixed the one final test thats failing. Kicking off another run. -- To view, visit http://gerrit.cloudera.org:8080/13746 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8323950857dfe1c1dfd5377fde79f87bc2ce9534 Gerrit-Change-Number: 13746 Gerrit-PatchSet: 15 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sun, 28 Jul 2019 22:51:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8717: impala-shell support for HS2 HTTP endpoint
Hello Thomas Tauber-Marshall, David Knupp, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13746 to look at the new patch set (#13). Change subject: IMPALA-8717: impala-shell support for HS2 HTTP endpoint .. IMPALA-8717: impala-shell support for HS2 HTTP endpoint Adds impala-shell support to connect to HiveServer2 HTTP endpoint. Relies on toolchain change at https://gerrit.cloudera.org/#/c/13725/. Use --protocol='hs2-http' to enable this behavior. Example usages: --- impala-shell --protocol='hs2-http' (No auth) impala-shell --protocol='hs2-http' --ldap -u. (PLAIN auth) impala-shell --protocol-'hs2-http' --ssl --ca_cert... (TLS) impala-shell --protocol='hs2-http' --ldap --ssl --ca_cert... (LDAP + TLS) Limitations: --- - Does not support Kerberos (-k) due to lack ot SPNEGO support. Testing: - Parameterized existing shell tests to support this combination. - Added shell test coverage for LDAP auth. Change-Id: I8323950857dfe1c1dfd5377fde79f87bc2ce9534 --- M be/src/service/impala-server.cc M bin/impala-config.sh A fe/src/test/java/org/apache/impala/customcluster/LdapImpalaShellTest.java M shell/impala_client.py M shell/impala_shell.py M shell/option_parser.py M tests/common/impala_cluster.py M tests/common/impala_connection.py M tests/common/impala_service.py M tests/common/impala_test_suite.py M tests/common/test_dimensions.py M tests/conftest.py M tests/custom_cluster/test_client_ssl.py M tests/custom_cluster/test_shell_interactive.py M tests/custom_cluster/test_shell_interactive_reconnect.py M tests/run-tests.py M tests/shell/test_shell_commandline.py M tests/shell/test_shell_interactive.py M tests/shell/util.py 19 files changed, 321 insertions(+), 62 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/46/13746/13 -- To view, visit http://gerrit.cloudera.org:8080/13746 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8323950857dfe1c1dfd5377fde79f87bc2ce9534 Gerrit-Change-Number: 13746 Gerrit-PatchSet: 13 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] Create ranger cache directory in containers.
Bharath Vissapragada has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14007 Change subject: Create ranger cache directory in containers. .. Create ranger cache directory in containers. Create a ranger cache directory used by ranger clients when ranger is enabled. For simplicity, it is added to the base image. It is used only on the coordinators/catalogd. Change-Id: Iad134636e1566a44acf7b010e6b6067a972798c6 --- M docker/impala_base/Dockerfile 1 file changed, 3 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/14007/1 -- To view, visit http://gerrit.cloudera.org:8080/14007 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iad134636e1566a44acf7b010e6b6067a972798c6 Gerrit-Change-Number: 14007 Gerrit-PatchSet: 1 Gerrit-Owner: Bharath Vissapragada
[Impala-ASF-CR] IMPALA-8848: fix UNION missing input cardinality bug
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14036 ) Change subject: IMPALA-8848: fix UNION missing input cardinality bug .. Patch Set 2: (4 comments) lgtm, just a bunch of clarifying questions and nits. http://gerrit.cloudera.org:8080/#/c/14036/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/14036/2//COMMIT_MSG@9 PS2, Line 9: If a UNION has input of unknown size, we should nit: I got confused by this statement for a minute. Rephrase may be? (to clarify that atleast one child with a valid cardinality is needed) http://gerrit.cloudera.org:8080/#/c/14036/2/fe/src/main/java/org/apache/impala/planner/UnionNode.java File fe/src/main/java/org/apache/impala/planner/UnionNode.java: http://gerrit.cloudera.org:8080/#/c/14036/2/fe/src/main/java/org/apache/impala/planner/UnionNode.java@124 PS2, Line 124: nit:..atleast..? http://gerrit.cloudera.org:8080/#/c/14036/2/fe/src/main/java/org/apache/impala/planner/UnionNode.java@127 PS2, Line 127: cardinality_ = checkedAdd(totalChildCardinality, constExprLists_.size()); nit:Preconditions.check(constExprLists_.size() > 0)? http://gerrit.cloudera.org:8080/#/c/14036/2/fe/src/test/java/org/apache/impala/planner/CardinalityTest.java File fe/src/test/java/org/apache/impala/planner/CardinalityTest.java: http://gerrit.cloudera.org:8080/#/c/14036/2/fe/src/test/java/org/apache/impala/planner/CardinalityTest.java@724 PS2, Line 724: path, UnionNode.class); Should we also add a test to cover something like select * from tbl_has_valid_cards union select * from tbl_without_valid_cards ? -- To view, visit http://gerrit.cloudera.org:8080/14036 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic3ed670ffb685d8ff24824933ca303f3219737bb Gerrit-Change-Number: 14036 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 09 Aug 2019 16:25:06 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7935: Disable /catalog object in local catalog mode.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/12443 ) Change subject: IMPALA-7935: Disable /catalog_object in local catalog mode. .. Patch Set 7: (6 comments) http://gerrit.cloudera.org:8080/#/c/12443/7/be/src/service/impala-http-handler.cc File be/src/service/impala-http-handler.cc: http://gerrit.cloudera.org:8080/#/c/12443/7/be/src/service/impala-http-handler.cc@109 PS7, Line 109: // The /catalog_object endpoint is disabled if local_catalog_mode is used. nit: add why http://gerrit.cloudera.org:8080/#/c/12443/7/tests/webserver/test_web_pages.py File tests/webserver/test_web_pages.py: http://gerrit.cloudera.org:8080/#/c/12443/7/tests/webserver/test_web_pages.py@605 PS7, Line 605: execute_serially why? http://gerrit.cloudera.org:8080/#/c/12443/7/tests/webserver/test_web_pages.py@607 PS7, Line 607: A positive test for /catalog_ob Instead say, check /catalog_obj endpoint is not accessible or something? http://gerrit.cloudera.org:8080/#/c/12443/7/tests/webserver/test_web_pages.py@608 PS7, Line 608: http://localhost:25000 use the URL templates above? http://gerrit.cloudera.org:8080/#/c/12443/7/tests/webserver/test_web_pages.py@609 PS7, Line 609: assert 'No URI handler for /catalog_object' \ I'm confused, the description says something else? *not* in catalog mode? http://gerrit.cloudera.org:8080/#/c/12443/7/tests/webserver/test_web_pages.py@610 PS7, Line 610:not in response Can you also scrape the webpage source and make sure it doesn't have a "catalog_object" anywhere? -- To view, visit http://gerrit.cloudera.org:8080/12443 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia04797b32964c2edaa2e860dcf510d6f9cccd81c Gerrit-Change-Number: 12443 Gerrit-PatchSet: 7 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 09 Aug 2019 15:54:24 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4551: Limit the size of SQL statements
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/14012 )
Change subject: IMPALA-4551: Limit the size of SQL statements
..
Patch Set 1:
(4 comments)
lgtm overall. Some general comments.
http://gerrit.cloudera.org:8080/#/c/14012/1/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
File fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java:
http://gerrit.cloudera.org:8080/#/c/14012/1/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java@462
PS1, Line 462: analysisResult_.analyzer_.checkStmtExprLimit();
nit: Probably worth mentioning that we want to enforce this before the rewrites
because rewrites are costly with long expression chains.
http://gerrit.cloudera.org:8080/#/c/14012/1/fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java:
http://gerrit.cloudera.org:8080/#/c/14012/1/fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java@2435
PS1, Line 2435: String repCols20 = getRepeatedColumnReference("int_col",
20, true);
Verify that analyzer.numStmtExprs_ is accounted properly?
http://gerrit.cloudera.org:8080/#/c/14012/1/fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java@2493
PS1, Line 2493: StringBuilder inList = new StringBuilder();
what about constant expressions in the IN lists? foo IN (2*3, 3*4...) ? It
looks to me like they are accounted, add tests for a mix of Literal and Const
expressions?
http://gerrit.cloudera.org:8080/#/c/14012/1/tests/query_test/test_exprs.py
File tests/query_test/test_exprs.py:
http://gerrit.cloudera.org:8080/#/c/14012/1/tests/query_test/test_exprs.py@139
PS1, Line 139: # This takes 20+ minutes, so only run it on exhaustive.
Is the intention to test the default limits here? If not, we can probably set a
non-default lower expression limit to save time and memory.
--
To view, visit http://gerrit.cloudera.org:8080/14012
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5675fb4a08c1dc51ae5bcf467cbb969cc064602c
Gerrit-Change-Number: 14012
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Quanlong Huang
Gerrit-Comment-Date: Fri, 09 Aug 2019 17:11:34 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7322: Add storage wait time to profile
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13786 ) Change subject: IMPALA-7322: Add storage wait time to profile .. Patch Set 1: Can you rebase? I can submit for a GVO. -- To view, visit http://gerrit.cloudera.org:8080/13786 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7447f8c8e7e50eb71d18643859d2e3de865368d2 Gerrit-Change-Number: 13786 Gerrit-PatchSet: 1 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Fri, 09 Aug 2019 17:14:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8817: Fix TestTestcaseBuilder tests on non-HDFS environment
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13963 ) Change subject: IMPALA-8817: Fix TestTestcaseBuilder tests on non-HDFS environment .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13963 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibf6767c3275dc5deec75a36f797d0963f83839cf Gerrit-Change-Number: 13963 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 01 Aug 2019 16:16:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8772: Import Testcase failed for SQL without table refs
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13893 ) Change subject: IMPALA-8772: Import Testcase failed for SQL without table refs .. Patch Set 1: Code-Review+2 (4 comments) Thanks for fixing this. http://gerrit.cloudera.org:8080/#/c/13893/1/tests/metadata/test_testcase_builder.py File tests/metadata/test_testcase_builder.py: http://gerrit.cloudera.org:8080/#/c/13893/1/tests/metadata/test_testcase_builder.py@41 PS1, Line 41: execute_query execute_query_expect_success http://gerrit.cloudera.org:8080/#/c/13893/1/tests/metadata/test_testcase_builder.py@43 PS1, Line 43: # Test load testcase works assert len(result.data) == 1 ? http://gerrit.cloudera.org:8080/#/c/13893/1/tests/metadata/test_testcase_builder.py@48 PS1, Line 48: # TODO: Delete testcase file from tmp Implement the TODO? http://gerrit.cloudera.org:8080/#/c/13893/1/tests/metadata/test_testcase_builder.py@49 PS1, Line 49: > flake8: W391 blank line at end of file Yep, remove these blank lines? -- To view, visit http://gerrit.cloudera.org:8080/13893 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I50d77d7de44bb140598a1e8db4da85a5ec87f31e Gerrit-Change-Number: 13893 Gerrit-PatchSet: 1 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 23 Jul 2019 02:46:22 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13874 ) Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode .. Patch Set 9: Code-Review+2 Thanks for fixing this. -- To view, visit http://gerrit.cloudera.org:8080/13874 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb Gerrit-Change-Number: 13874 Gerrit-PatchSet: 9 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 23 Jul 2019 02:52:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8746: [DOCS] Document the DEFAULT HINTS INSERT STATEMENT query option
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13885 ) Change subject: IMPALA-8746: [DOCS] Document the DEFAULT_HINTS_INSERT_STATEMENT query option .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13885 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia376721f46eb507901f9f64b5c3341dc0f36475b Gerrit-Change-Number: 13885 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 23 Jul 2019 02:50:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8717: impala-shell support for HS2 HTTP endpoint
Hello Thomas Tauber-Marshall, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13746 to look at the new patch set (#6). Change subject: IMPALA-8717: impala-shell support for HS2 HTTP endpoint .. IMPALA-8717: impala-shell support for HS2 HTTP endpoint Adds impala-shell support to connect to HiveServer2 HTTP endpoint. Relies on toolchain change at https://gerrit.cloudera.org/#/c/13725/. Use --protocol='hs2-http' to enable this behavior. Example usages: --- impala-shell --protocol='hs2-http' (No auth) impala-shell --protocol='hs2-http' --ldap -u. (PLAIN auth) impala-shell --protocol-'hs2-http' --ssl --ca_cert... (TLS) impala-shell --protocol='hs2-http' --ldap --ssl --ca_cert... (LDAP + TLS) Limitations: --- - Does not support Kerberos (-k) due to lack ot SPNEGO support. Testing: - Parameterized existing shell tests to support this combination. - Added shell test coverage for LDAP auth. Change-Id: I8323950857dfe1c1dfd5377fde79f87bc2ce9534 --- M be/src/service/impala-server.cc M bin/impala-config.sh M fe/src/test/java/org/apache/impala/customcluster/LdapJdbcTest.java M shell/impala_client.py M shell/impala_shell.py M shell/option_parser.py M tests/common/impala_cluster.py M tests/common/impala_service.py M tests/common/impala_test_suite.py M tests/common/test_dimensions.py M tests/conftest.py M tests/custom_cluster/test_client_ssl.py M tests/run-tests.py M tests/shell/test_shell_commandline.py M tests/shell/test_shell_interactive.py M tests/shell/util.py 16 files changed, 209 insertions(+), 44 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/46/13746/6 -- To view, visit http://gerrit.cloudera.org:8080/13746 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8323950857dfe1c1dfd5377fde79f87bc2ce9534 Gerrit-Change-Number: 13746 Gerrit-PatchSet: 6 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8717: impala-shell support for HS2 HTTP endpoint
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/13746 )
Change subject: IMPALA-8717: impala-shell support for HS2 HTTP endpoint
..
Patch Set 6:
(10 comments)
http://gerrit.cloudera.org:8080/#/c/13746/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13746/5//COMMIT_MSG@29
PS5, Line 29: - Added shell test coverage for LDAP auth.
> If you're interested, it should be possible to write automated tests for th
Done. Added some test coverage.
http://gerrit.cloudera.org:8080/#/c/13746/5/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/13746/5/be/src/service/impala-server.cc@2081
PS5, Line 2081: << " The connection had " <<
disconnected_sessions.size()
> I feel like the "with" ... " sessions closed" makes it sound like the sessi
Fair point. Done.
http://gerrit.cloudera.org:8080/#/c/13746/5/shell/impala_client.py
File shell/impala_client.py:
http://gerrit.cloudera.org:8080/#/c/13746/5/shell/impala_client.py@367
PS5, Line 367: # TODO: Investigate connection reuse in THttpClient and
revisit this.
> What would happen if you set the socket timeout and then just didn't reset
Yes. that is not desirable for longer duration RPCs, say waiting on results etc.
http://gerrit.cloudera.org:8080/#/c/13746/5/shell/impala_client.py@369
PS5, Line 369: print_to_stderr("Warning: --connect_timeout_ms is
currently ignored with" +
switched this to a warning.
http://gerrit.cloudera.org:8080/#/c/13746/5/shell/impala_client.py@392
PS5, Line 392:
> flake8: E501 line too long (93 > 90 characters)
Done
http://gerrit.cloudera.org:8080/#/c/13746/5/shell/impala_shell.py
File shell/impala_shell.py:
http://gerrit.cloudera.org:8080/#/c/13746/5/shell/impala_shell.py@531
PS5, Line 531: self.
> Maybe include the param name here and below, i.e. "use_http_base_transport=
Done
http://gerrit.cloudera.org:8080/#/c/13746/5/shell/impala_shell.py@544
PS5, Line 544: _ms
> nit: comma
Done
http://gerrit.cloudera.org:8080/#/c/13746/5/tests/run-tests.py
File tests/run-tests.py:
http://gerrit.cloudera.org:8080/#/c/13746/5/tests/run-tests.py@233
PS5, Line 233:
> flake8: E502 the backslash is redundant between brackets
Done
http://gerrit.cloudera.org:8080/#/c/13746/5/tests/run-tests.py@234
PS5, Line 234:
webserver_certificate_file=cert).read_debug_webpage('metrics?json'))
> Why is this needed?
This is not related to patch but I ran into this during my local testing. So I
thought it'd nice to fix it while I'm here.
So if you have a mini-cluster with TLS for web endpoints enabled, any
subsequent run-tests.py command gets stuck here in a loop, trying to print
metrics, because it fails to connect to the http end points without the
certificate. Essentially no tests run until you manually kill the cluster or
restart it without TLS for web endpoints.
http://gerrit.cloudera.org:8080/#/c/13746/5/tests/shell/test_shell_interactive.py
File tests/shell/test_shell_interactive.py:
http://gerrit.cloudera.org:8080/#/c/13746/5/tests/shell/test_shell_interactive.py@268
PS5, Line 268: assert protocol == "beeswax"
> assert protocol == "beeswax"
Done
--
To view, visit http://gerrit.cloudera.org:8080/13746
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8323950857dfe1c1dfd5377fde79f87bc2ce9534
Gerrit-Change-Number: 13746
Gerrit-PatchSet: 6
Gerrit-Owner: Bharath Vissapragada
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Thomas Tauber-Marshall
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Sat, 20 Jul 2019 03:23:38 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/13874 )
Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local
catalog mode
..
Patch Set 5:
(4 comments)
Patch makes sense to me. Some comments.
http://gerrit.cloudera.org:8080/#/c/13874/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13874/5//COMMIT_MSG@23
PS5, Line 23: - Testing in a HMS with 100 dbs and 3000 tables, without this
patch it
Mind deploying this jar on a cluster with Hue and quickly check that nothing
breaks?
http://gerrit.cloudera.org:8080/#/c/13874/5/fe/src/main/java/org/apache/impala/catalog/Catalog.java
File fe/src/main/java/org/apache/impala/catalog/Catalog.java:
http://gerrit.cloudera.org:8080/#/c/13874/5/fe/src/main/java/org/apache/impala/catalog/Catalog.java@167
PS5, Line 167: Catalog v1
I think the code never refers to Catalog "v1" and Catalog "v2". So this could
be confusing and better to get rid of them (multiple places).
http://gerrit.cloudera.org:8080/#/c/13874/5/fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java:
http://gerrit.cloudera.org:8080/#/c/13874/5/fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java@169
PS5, Line 169: newMap.put(tableName.toLowerCase(), null);
Should we instead load LocalIncompleteTable here?
http://gerrit.cloudera.org:8080/#/c/13874/5/fe/src/test/java/org/apache/impala/service/JdbcTest.java
File fe/src/test/java/org/apache/impala/service/JdbcTest.java:
http://gerrit.cloudera.org:8080/#/c/13874/5/fe/src/test/java/org/apache/impala/service/JdbcTest.java@390
PS5, Line 390: public void testMetaDataGetColumnComments() throws Exception {
Mind adding a test to LocalCatalogTest#testGetTables()? Check that the table
remains in an incomplete state after GET_TABLES request?
--
To view, visit http://gerrit.cloudera.org:8080/13874
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb
Gerrit-Change-Number: 13874
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Comment-Date: Thu, 18 Jul 2019 17:28:36 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13874 ) Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode .. Patch Set 5: (2 comments) Couple of nits, but can +2 once fixed. http://gerrit.cloudera.org:8080/#/c/13874/5/fe/src/main/java/org/apache/impala/catalog/Catalog.java File fe/src/main/java/org/apache/impala/catalog/Catalog.java: http://gerrit.cloudera.org:8080/#/c/13874/5/fe/src/main/java/org/apache/impala/catalog/Catalog.java@167 PS5, Line 167: Catalog v1 > I'm trying to explain why the implementation here is just delegating to get Right, I'm talking about the terminology. In the code, we never see these terms like v1 and v2. Maybe say legacy implementation vs LocalCatalog implementation or some such thing? http://gerrit.cloudera.org:8080/#/c/13874/6/fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java File fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java: http://gerrit.cloudera.org:8080/#/c/13874/6/fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java@385 PS6, Line 385: Why this -- To view, visit http://gerrit.cloudera.org:8080/13874 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb Gerrit-Change-Number: 13874 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Fri, 19 Jul 2019 16:16:52 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/13860 )
Change subject: IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to
create insert events with IllegalStateException.
..
Patch Set 2:
(3 comments)
I won't be able to review this, I'll let Vihang +2 it.
http://gerrit.cloudera.org:8080/#/c/13860/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/13860/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3936
PS2, Line 3936: private Map
getPartitionNamesMap(Collectionhttp://gerrit.cloudera.org:8080/#/c/13860/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3939
PS2, Line 3939: for (FeFsPartition part: partitions) {
you could simply this (and below) with stream() and collect()
http://gerrit.cloudera.org:8080/#/c/13860/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3948
PS2, Line 3948: private Map>
getPartitionNameFilesMap(Collectionhttp://gerrit.cloudera.org:8080/13860
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9
Gerrit-Change-Number: 13860
Gerrit-PatchSet: 2
Gerrit-Owner: Anurag Mantripragada
Gerrit-Reviewer: Anurag Mantripragada
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Fri, 19 Jul 2019 16:44:20 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5149: Provide query profile in JSON format
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/13801 )
Change subject: IMPALA-5149: Provide query profile in JSON format
..
Patch Set 8: Code-Review+1
(5 comments)
Some minor comments, but generally looks good to me. I spoke to Michael Ho
offline who is going to take another look into the change and take it to a +2.
Thanks for your patience so far.
http://gerrit.cloudera.org:8080/#/c/13801/8//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13801/8//COMMIT_MSG@22
PS8, Line 22:
Add a section on future compatibility guarantees?
I think the idea is that this can evolve over time until we standardize the
counters and the profile structure. So, until then if there are some Impal a
changes, clients can expect the JSON structure to change too (keys could be
added, deleted etc.)
http://gerrit.cloudera.org:8080/#/c/13801/8/be/src/util/runtime-profile-counters.h
File be/src/util/runtime-profile-counters.h:
http://gerrit.cloudera.org:8080/#/c/13801/8/be/src/util/runtime-profile-counters.h@292
PS8, Line 292: val->RemoveMember("kind");
Should we not update the kind ? Why remove it?
http://gerrit.cloudera.org:8080/#/c/13801/8/tests/hs2/test_hs2.py
File tests/hs2/test_hs2.py:
http://gerrit.cloudera.org:8080/#/c/13801/8/tests/hs2/test_hs2.py@668
PS8, Line 668: json_res["contents"]["child_profiles"][0]["info_strings"]:
This looks flaky. Instead assert that they exist before you can loop through
them?
http://gerrit.cloudera.org:8080/#/c/13801/8/tests/hs2/test_hs2.py@670
PS8, Line 670: assert statement in info_string["value"]
dump the json_res if the assert fails, helps with debugging test failures.
http://gerrit.cloudera.org:8080/#/c/13801/8/tests/webserver/test_web_pages.py
File tests/webserver/test_web_pages.py:
http://gerrit.cloudera.org:8080/#/c/13801/8/tests/webserver/test_web_pages.py@594
PS8, Line 594: self.fail("Download JSON format query profile cannot
be parsed.")
dump json
--
To view, visit http://gerrit.cloudera.org:8080/13801
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8181ac818bf22207ca1deabd9220c397ae723ec1
Gerrit-Change-Number: 13801
Gerrit-PatchSet: 8
Gerrit-Owner: Jiawei Wang
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: David Knupp
Gerrit-Reviewer: David Rorke
Gerrit-Reviewer: Greg Rahn
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jiawei Wang
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Sahil Takiar
Gerrit-Comment-Date: Fri, 19 Jul 2019 17:20:25 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8717: impala-shell support for HS2 HTTP endpoint
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13746 ) Change subject: IMPALA-8717: impala-shell support for HS2 HTTP endpoint .. Patch Set 7: (6 comments) http://gerrit.cloudera.org:8080/#/c/13746/6/fe/src/test/java/org/apache/impala/customcluster/LdapJdbcTest.java File fe/src/test/java/org/apache/impala/customcluster/LdapJdbcTest.java: http://gerrit.cloudera.org:8080/#/c/13746/6/fe/src/test/java/org/apache/impala/customcluster/LdapJdbcTest.java@116 PS6, Line 116: > line has trailing whitespace Done http://gerrit.cloudera.org:8080/#/c/13746/6/fe/src/test/java/org/apache/impala/customcluster/LdapJdbcTest.java@133 PS6, Line 133: : > Instead of catching the assertion failure to log and rethrow it, I think yo Done http://gerrit.cloudera.org:8080/#/c/13746/6/fe/src/test/java/org/apache/impala/customcluster/LdapJdbcTest.java@140 PS6, Line 140: > nit: I probably got this wrong in some places in my patches around this, bu removed it here. Basic is specific to http, but here were are testing all the protocols. http://gerrit.cloudera.org:8080/#/c/13746/6/fe/src/test/java/org/apache/impala/customcluster/LdapJdbcTest.java@143 PS6, Line 143: > We don't really want to inherit all of the JDBC related setup stuff here, e ya, I should've seen that. Fixed it now. http://gerrit.cloudera.org:8080/#/c/13746/6/fe/src/test/java/org/apache/impala/customcluster/LdapJdbcTest.java@149 PS6, Line 149: > Could you make the 'select logged_in_user()' and actually verify that its c Done http://gerrit.cloudera.org:8080/#/c/13746/6/shell/impala_shell.py File shell/impala_shell.py: http://gerrit.cloudera.org:8080/#/c/13746/6/shell/impala_shell.py@804 PS6, Line 804: assert options.protocol.lower() == 'beeswax' : port = str(DEFAULT_BEESWAX_PORT) > Probably only want to log if its actually wrong, and we should probably ret oops, debugging message leaked here. thanks for catching. I don't think anyone would ever hit this assertion unless they are changing something in the code. So I think its ok to keep it as is. -- To view, visit http://gerrit.cloudera.org:8080/13746 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8323950857dfe1c1dfd5377fde79f87bc2ce9534 Gerrit-Change-Number: 13746 Gerrit-PatchSet: 7 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Jul 2019 00:29:24 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8717: impala-shell support for HS2 HTTP endpoint
Hello Thomas Tauber-Marshall, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13746 to look at the new patch set (#7). Change subject: IMPALA-8717: impala-shell support for HS2 HTTP endpoint .. IMPALA-8717: impala-shell support for HS2 HTTP endpoint Adds impala-shell support to connect to HiveServer2 HTTP endpoint. Relies on toolchain change at https://gerrit.cloudera.org/#/c/13725/. Use --protocol='hs2-http' to enable this behavior. Example usages: --- impala-shell --protocol='hs2-http' (No auth) impala-shell --protocol='hs2-http' --ldap -u. (PLAIN auth) impala-shell --protocol-'hs2-http' --ssl --ca_cert... (TLS) impala-shell --protocol='hs2-http' --ldap --ssl --ca_cert... (LDAP + TLS) Limitations: --- - Does not support Kerberos (-k) due to lack ot SPNEGO support. Testing: - Parameterized existing shell tests to support this combination. - Added shell test coverage for LDAP auth. Change-Id: I8323950857dfe1c1dfd5377fde79f87bc2ce9534 --- M be/src/service/impala-server.cc M bin/impala-config.sh A fe/src/test/java/org/apache/impala/customcluster/LdapImpalaShellTest.java M shell/impala_client.py M shell/impala_shell.py M shell/option_parser.py M tests/common/impala_cluster.py M tests/common/impala_service.py M tests/common/impala_test_suite.py M tests/common/test_dimensions.py M tests/conftest.py M tests/custom_cluster/test_client_ssl.py M tests/run-tests.py M tests/shell/test_shell_commandline.py M tests/shell/test_shell_interactive.py M tests/shell/util.py 16 files changed, 284 insertions(+), 44 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/46/13746/7 -- To view, visit http://gerrit.cloudera.org:8080/13746 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8323950857dfe1c1dfd5377fde79f87bc2ce9534 Gerrit-Change-Number: 13746 Gerrit-PatchSet: 7 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8717: impala-shell support for HS2 HTTP endpoint
Hello Thomas Tauber-Marshall, David Knupp, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13746 to look at the new patch set (#8). Change subject: IMPALA-8717: impala-shell support for HS2 HTTP endpoint .. IMPALA-8717: impala-shell support for HS2 HTTP endpoint Adds impala-shell support to connect to HiveServer2 HTTP endpoint. Relies on toolchain change at https://gerrit.cloudera.org/#/c/13725/. Use --protocol='hs2-http' to enable this behavior. Example usages: --- impala-shell --protocol='hs2-http' (No auth) impala-shell --protocol='hs2-http' --ldap -u. (PLAIN auth) impala-shell --protocol-'hs2-http' --ssl --ca_cert... (TLS) impala-shell --protocol='hs2-http' --ldap --ssl --ca_cert... (LDAP + TLS) Limitations: --- - Does not support Kerberos (-k) due to lack ot SPNEGO support. Testing: - Parameterized existing shell tests to support this combination. - Added shell test coverage for LDAP auth. Change-Id: I8323950857dfe1c1dfd5377fde79f87bc2ce9534 --- M be/src/service/impala-server.cc M bin/impala-config.sh A fe/src/test/java/org/apache/impala/customcluster/LdapImpalaShellTest.java M shell/impala_client.py M shell/impala_shell.py M shell/option_parser.py M tests/common/impala_cluster.py M tests/common/impala_service.py M tests/common/impala_test_suite.py M tests/common/test_dimensions.py M tests/conftest.py M tests/custom_cluster/test_client_ssl.py M tests/run-tests.py M tests/shell/test_shell_commandline.py M tests/shell/test_shell_interactive.py M tests/shell/util.py 16 files changed, 287 insertions(+), 45 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/46/13746/8 -- To view, visit http://gerrit.cloudera.org:8080/13746 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8323950857dfe1c1dfd5377fde79f87bc2ce9534 Gerrit-Change-Number: 13746 Gerrit-PatchSet: 8 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8717: impala-shell support for HS2 HTTP endpoint
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/13746 )
Change subject: IMPALA-8717: impala-shell support for HS2 HTTP endpoint
..
Patch Set 6:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13746/6/shell/impala_shell.py
File shell/impala_shell.py:
http://gerrit.cloudera.org:8080/#/c/13746/6/shell/impala_shell.py@804
PS6, Line 804: print_to_stderr("protocol: " + options.protocol.lower())
: assert options.protocol.lower() == 'beeswax'
> Maybe I'm wrong, but it doesn't look like this value is validated anywhere
Done
--
To view, visit http://gerrit.cloudera.org:8080/13746
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8323950857dfe1c1dfd5377fde79f87bc2ce9534
Gerrit-Change-Number: 13746
Gerrit-PatchSet: 6
Gerrit-Owner: Bharath Vissapragada
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: David Knupp
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Thomas Tauber-Marshall
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Wed, 24 Jul 2019 17:02:15 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8486: fix stale libCache entries in LocalCatalog mode
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13849 ) Change subject: IMPALA-8486: fix stale libCache entries in LocalCatalog mode .. Patch Set 4: Yep, agreed. Dedicated executor libcache invalidation seems broken. -- To view, visit http://gerrit.cloudera.org:8080/13849 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie4812fb8737de3ba6074ffeb9007927bfbbbaf9b Gerrit-Change-Number: 13849 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 15 Jul 2019 16:51:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5149: Provide query profile in JSON format
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/13801 )
Change subject: IMPALA-5149: Provide query profile in JSON format
..
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13801/2/be/src/service/impala-hs2-server.cc
File be/src/service/impala-hs2-server.cc:
http://gerrit.cloudera.org:8080/#/c/13801/2/be/src/service/impala-hs2-server.cc@979
PS2, Line 979: if (request.format == TRuntimeProfileFormat::THRIFT) {
: return_val.__set_thrift_profile(thrift_profile);
: } else {
: DCHECK(request.format == TRuntimeProfileFormat::STRING
: || request.format == TRuntimeProfileFormat::BASE64);
: return_val.__set_profile(ss.str());
: }
> not sure; I'm not too familiar with this part of the code, I guess it depen
I don't think we need to fix this for beeswax server as it will be deprecated
in the future, especially given that shell now supports HS2 protocol.
--
To view, visit http://gerrit.cloudera.org:8080/13801
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8181ac818bf22207ca1deabd9220c397ae723ec1
Gerrit-Change-Number: 13801
Gerrit-PatchSet: 2
Gerrit-Owner: Jiawei Wang
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: David Knupp
Gerrit-Reviewer: David Rorke
Gerrit-Reviewer: Greg Rahn
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jiawei Wang
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Sahil Takiar
Gerrit-Comment-Date: Mon, 15 Jul 2019 22:43:44 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13860 ) Change subject: IMPALA-8489: Partitions created by RECOVER PARTITIONS fail to create insert events with IllegalStateException. .. Patch Set 1: Do you have the full IllegalStateException stack trace? (Don't see it in the jira). I'm trying to understand what the problem is here. -- To view, visit http://gerrit.cloudera.org:8080/13860 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idef7f6aadff2868047c861ebfcc05d65f080eab9 Gerrit-Change-Number: 13860 Gerrit-PatchSet: 1 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 15 Jul 2019 20:39:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8486: fix stale libCache entries in LocalCatalog mode coordinators
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13849 ) Change subject: IMPALA-8486: fix stale libCache entries in LocalCatalog mode coordinators .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13849 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie4812fb8737de3ba6074ffeb9007927bfbbbaf9b Gerrit-Change-Number: 13849 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 15 Jul 2019 17:57:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8717: impala-shell support for HS2 HTTP endpoint
Hello Thomas Tauber-Marshall, David Knupp, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13746 to look at the new patch set (#9). Change subject: IMPALA-8717: impala-shell support for HS2 HTTP endpoint .. IMPALA-8717: impala-shell support for HS2 HTTP endpoint Adds impala-shell support to connect to HiveServer2 HTTP endpoint. Relies on toolchain change at https://gerrit.cloudera.org/#/c/13725/. Use --protocol='hs2-http' to enable this behavior. Example usages: --- impala-shell --protocol='hs2-http' (No auth) impala-shell --protocol='hs2-http' --ldap -u. (PLAIN auth) impala-shell --protocol-'hs2-http' --ssl --ca_cert... (TLS) impala-shell --protocol='hs2-http' --ldap --ssl --ca_cert... (LDAP + TLS) Limitations: --- - Does not support Kerberos (-k) due to lack ot SPNEGO support. Testing: - Parameterized existing shell tests to support this combination. - Added shell test coverage for LDAP auth. Change-Id: I8323950857dfe1c1dfd5377fde79f87bc2ce9534 --- M be/src/service/impala-server.cc M bin/impala-config.sh A fe/src/test/java/org/apache/impala/customcluster/LdapImpalaShellTest.java M shell/impala_client.py M shell/impala_shell.py M shell/option_parser.py M tests/common/impala_cluster.py M tests/common/impala_service.py M tests/common/impala_test_suite.py M tests/common/test_dimensions.py M tests/conftest.py M tests/custom_cluster/test_client_ssl.py M tests/run-tests.py M tests/shell/test_shell_commandline.py M tests/shell/test_shell_interactive.py M tests/shell/util.py 16 files changed, 288 insertions(+), 45 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/46/13746/9 -- To view, visit http://gerrit.cloudera.org:8080/13746 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8323950857dfe1c1dfd5377fde79f87bc2ce9534 Gerrit-Change-Number: 13746 Gerrit-PatchSet: 9 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8717: impala-shell support for HS2 HTTP endpoint
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/13746 )
Change subject: IMPALA-8717: impala-shell support for HS2 HTTP endpoint
..
Patch Set 8:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13746/8/fe/src/test/java/org/apache/impala/customcluster/LdapImpalaShellTest.java
File fe/src/test/java/org/apache/impala/customcluster/LdapImpalaShellTest.java:
http://gerrit.cloudera.org:8080/#/c/13746/8/fe/src/test/java/org/apache/impala/customcluster/LdapImpalaShellTest.java@123
PS8, Line 123: String[] commandWithoutAuth = {"impala-shell.sh", "",
String.format("--query=%s", query)};
> line too long (94 > 90)
Done
--
To view, visit http://gerrit.cloudera.org:8080/13746
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8323950857dfe1c1dfd5377fde79f87bc2ce9534
Gerrit-Change-Number: 13746
Gerrit-PatchSet: 8
Gerrit-Owner: Bharath Vissapragada
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: David Knupp
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Thomas Tauber-Marshall
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Wed, 24 Jul 2019 17:07:00 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8606: Don't load table meta for GET TABLES in local catalog mode
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13874 ) Change subject: IMPALA-8606: Don't load table meta for GET_TABLES in local catalog mode .. Patch Set 11: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13874 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia8bbab7efdf8e629abe09d89ae3bd770e3feaccb Gerrit-Change-Number: 13874 Gerrit-PatchSet: 11 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 25 Jul 2019 01:03:43 + Gerrit-HasComments: No
