date:20190612

Hello Michael Ho, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/13592

to look at the new patch set (#8).

Change subject: IMPALA-8659: Allow self-RPCs for KRPC to go via loopback
..

IMPALA-8659: Allow self-RPCs for KRPC to go via loopback

Adds a flag --rpc_use_loopback that causes two differences
in behaviour when enabled:
1. KRPC will listen on all interfaces, i.e. bind the socket
   to INADDR_ANY.
2. KRPC RPCs to --hostname are sent to 127.0.0.1 instead of
   the IP (maybe external) that --hostname resolves to.

There is no change in default behaviour, except in containers,
where this flag is enabled by default.

Testing:
* Added a custom cluster test, which runs in exhaustive,
  as a sanity test for the behaviour of the flag.

Change-Id: I9dbd477769ed49c05e624f06da4e51afaaf1670d
---
M be/src/rpc/rpc-mgr.cc
M be/src/rpc/rpc-mgr.inline.h
M be/src/util/network-util.cc
M be/src/util/network-util.h
M docker/coord_exec/Dockerfile
M docker/coordinator/Dockerfile
M docker/executor/Dockerfile
A tests/custom_cluster/test_krpc_options.py
8 files changed, 79 insertions(+), 7 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/92/13592/8
--
To view, visit http://gerrit.cloudera.org:8080/13592
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9dbd477769ed49c05e624f06da4e51afaaf1670d
Gerrit-Change-Number: 13592
Gerrit-PatchSet: 8
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Michael Ho

[Impala-ASF-CR] IMPALA-8659: Allow self-RPCs for KRPC to go via loopback

Hello Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/13592

to look at the new patch set (#7).

Change subject: IMPALA-8659: Allow self-RPCs for KRPC to go via loopback
..

IMPALA-8659: Allow self-RPCs for KRPC to go via loopback

Adds a flag --rpc_use_loopback that causes two differences
in behaviour when enabled:
1. KRPC will listen on all interfaces, i.e. bind the socket
   to INADDR_ANY.
2. KRPC RPCs to --hostname are sent to 127.0.0.1 instead of
   the IP (maybe external) that --hostname resolves to.

There is no change in default behaviour, except in containers,
where this flag is enabled by default.

Testing:
* Added a custom cluster test, which runs in exhaustive,
  as a sanity test for the behaviour of the flag.

Change-Id: I9dbd477769ed49c05e624f06da4e51afaaf1670d
---
M be/src/rpc/rpc-mgr.cc
M be/src/rpc/rpc-mgr.inline.h
M be/src/util/network-util.cc
M be/src/util/network-util.h
M docker/coord_exec/Dockerfile
M docker/coordinator/Dockerfile
M docker/executor/Dockerfile
A tests/custom_cluster/test_krpc_options.py
8 files changed, 78 insertions(+), 7 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/92/13592/7
--
To view, visit http://gerrit.cloudera.org:8080/13592
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9dbd477769ed49c05e624f06da4e51afaaf1670d
Gerrit-Change-Number: 13592
Gerrit-PatchSet: 7
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Impala Public Jenkins

[Impala-ASF-CR] IMPALA-8659: Allow self-RPCs for KRPC to go via loopback

2019-06-12 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13592 )

Change subject: IMPALA-8659: Allow self-RPCs for KRPC to go via loopback
..


Patch Set 7:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4465/ 
DRY_RUN=true


--
To view, visit http://gerrit.cloudera.org:8080/13592
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9dbd477769ed49c05e624f06da4e51afaaf1670d
Gerrit-Change-Number: 13592
Gerrit-PatchSet: 7
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Michael Ho 
Gerrit-Comment-Date: Wed, 12 Jun 2019 23:50:47 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7290: part 2: Add HS2 support to Impala shell

Tim Armstrong has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12884 )

Change subject: IMPALA-7290: part 2: Add HS2 support to Impala shell
..


Patch Set 20:

Hit the new metric checks when running e2e tests where sessions from 
test_ddl_queries_are_closed were leaked. I couldn't reproduce locally, but it 
seems to be a side-effect of the shell tests that launch subprocesses not 
consistently cleaning up those processes (i.e. not sending EOF or exit; or 
whatever). I'm going to rerun the tests.


--
To view, visit http://gerrit.cloudera.org:8080/12884
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d5cc83d545aacc659523f29b1d6feed672e2a12
Gerrit-Change-Number: 12884
Gerrit-PatchSet: 20
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Thomas Marshall 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 12 Jun 2019 23:21:12 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8650: Docker build should not depend on test config

2019-06-12 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13597 )

Change subject: IMPALA-8650: Docker build should not depend on test config
..


Patch Set 2: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/13597
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa70864f5d047d1ff5f21e69d8f6358306424c0b
Gerrit-Change-Number: 13597
Gerrit-PatchSet: 2
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Comment-Date: Wed, 12 Jun 2019 23:15:23 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8536: Add Scalable Pool Configuration to Admission Controller.

2019-06-12 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13307 )

Change subject: IMPALA-8536: Add Scalable Pool Configuration to Admission 
Controller.
..


Patch Set 12:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/3588/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/13307
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If47508728124076f3b9200c27cffc989f7a4f188
Gerrit-Change-Number: 13307
Gerrit-PatchSet: 12
Gerrit-Owner: Andrew Sherman 
Gerrit-Reviewer: Andrew Sherman 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 12 Jun 2019 22:15:56 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8658: Populate missing Ranger audit fields

2019-06-12 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13601 )

Change subject: IMPALA-8658: Populate missing Ranger audit fields
..


Patch Set 2:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/3589/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/13601
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I167bc35411ad9b4164c292077ff082671967cff8
Gerrit-Change-Number: 13601
Gerrit-PatchSet: 2
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Todd Lipcon 
Gerrit-Comment-Date: Wed, 12 Jun 2019 22:15:35 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8617: Add support for lz4 in parquet

2019-06-12 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13582 )

Change subject: IMPALA-8617: Add support for lz4 in parquet
..


Patch Set 6:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/3587/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/13582
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia6850a39ef3f1e0e7ba48e08eef1d4f7cbb74d0c
Gerrit-Change-Number: 13582
Gerrit-PatchSet: 6
Gerrit-Owner: Abhishek Rawat 
Gerrit-Reviewer: Abhishek Rawat 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Todd Lipcon 
Gerrit-Comment-Date: Wed, 12 Jun 2019 22:13:49 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8588: Fix revoke grant option with Ranger

2019-06-12 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13450 )

Change subject: IMPALA-8588: Fix revoke grant option with Ranger
..


Patch Set 4: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/13450
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iddfccb442c3be3c266dbc2d8ae85c5674c534d7c
Gerrit-Change-Number: 13450
Gerrit-PatchSet: 4
Gerrit-Owner: Austin Nobis 
Gerrit-Reviewer: Austin Nobis 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Wed, 12 Jun 2019 21:36:48 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8588: Fix revoke grant option with Ranger

2019-06-12 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/13450 )

Change subject: IMPALA-8588: Fix revoke grant option with Ranger
..

IMPALA-8588: Fix revoke grant option with Ranger

Previously, the REVOKE GRANT OPTION type statements would not only
revoke the grant option, but also the privilege as well. The behavior
has been updated to only revoke the grant option. In Ranger UI, this is
seen as the delegate admin option. Examples:

REVOKE SELECT ON DATABASE  FROM USER 

This will revoke the SELECT privilege on the database resource, but
if there are other privileges on that resource the grant option will
remain for those privileges.

REVOKE GRANT OPTION FOR SELECT ON DATABASE  FROM USER 

This will revoke the grant option for all privileges on this database
resource. It will not revoke the SELECT privilege on the resource.

Testing:
- Ran all FE tests
- Ran all E2E tests
- Updated test_ranger to test behavior for when REVOKE GRANT OPTION
  statements are submitted.

Change-Id: Iddfccb442c3be3c266dbc2d8ae85c5674c534d7c
Reviewed-on: http://gerrit.cloudera.org:8080/13450
Reviewed-by: Impala Public Jenkins 
Tested-by: Impala Public Jenkins 
---
M 
fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java
M tests/authorization/test_ranger.py
3 files changed, 47 insertions(+), 26 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

--
To view, visit http://gerrit.cloudera.org:8080/13450
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Iddfccb442c3be3c266dbc2d8ae85c5674c534d7c
Gerrit-Change-Number: 13450
Gerrit-PatchSet: 5
Gerrit-Owner: Austin Nobis 
Gerrit-Reviewer: Austin Nobis 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins

[Impala-ASF-CR] IMPALA-8649: Fix confusing SHOW GRANT error messages

2019-06-12 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13587 )

Change subject: IMPALA-8649: Fix confusing SHOW GRANT error messages
..


Patch Set 4: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/13587
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibb88bdc19cd1223902b44e3634f756d086332266
Gerrit-Change-Number: 13587
Gerrit-PatchSet: 4
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: Austin Nobis 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Todd Lipcon 
Gerrit-Comment-Date: Wed, 12 Jun 2019 21:11:21 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8649: Fix confusing SHOW GRANT error messages

2019-06-12 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/13587 )

Change subject: IMPALA-8649: Fix confusing SHOW GRANT error messages
..

IMPALA-8649: Fix confusing SHOW GRANT error messages

Before this patch, PrivilegeSpec had error messages tailored to
GRANT and REVOKE statements. However, PrivilegeSpec is also used in SHOW
GRANT statement. This patch updates the error messages to be less
confusing in SHOW GRANT statement.

Testing:
- Added new tests in AnalyzeAuthStmtsTest
- Ran AnalyzeAuthStmtsTest

Change-Id: Ibb88bdc19cd1223902b44e3634f756d086332266
Reviewed-on: http://gerrit.cloudera.org:8080/13587
Reviewed-by: Impala Public Jenkins 
Tested-by: Impala Public Jenkins 
---
M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeAuthStmtsTest.java
2 files changed, 33 insertions(+), 18 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

--
To view, visit http://gerrit.cloudera.org:8080/13587
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ibb88bdc19cd1223902b44e3634f756d086332266
Gerrit-Change-Number: 13587
Gerrit-PatchSet: 5
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: Austin Nobis 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Todd Lipcon

[Impala-ASF-CR] IMPALA-8658: Populate missing Ranger audit fields

2019-06-12 Thread Fredy Wijaya (Code Review)

Fredy Wijaya has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/13601


Change subject: IMPALA-8658: Populate missing Ranger audit fields
..

IMPALA-8658: Populate missing Ranger audit fields

This patch adds missing Ranger audit fields, such as:
- Client IP
- Cluster name

This patch also updates the access type to be in upper case to be
consistent with Hive Ranger audit.

Testing:
- Updated the tests in RangerAuditLogTest
- Ran all FE tests
- Tested the changes against Solr cluster

Change-Id: I167bc35411ad9b4164c292077ff082671967cff8
---
M 
fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
M 
fe/src/main/java/org/apache/impala/authorization/ranger/RangerBufferAuditHandler.java
M 
fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java
M fe/src/test/resources/ranger-hive-audit.xml
4 files changed, 26 insertions(+), 4 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/13601/2
--
To view, visit http://gerrit.cloudera.org:8080/13601
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I167bc35411ad9b4164c292077ff082671967cff8
Gerrit-Change-Number: 13601
Gerrit-PatchSet: 2
Gerrit-Owner: Fredy Wijaya

[Impala-ASF-CR] IMPALA-8536: Add Scalable Pool Configuration to Admission Controller.

2019-06-12 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13307 )

Change subject: IMPALA-8536: Add Scalable Pool Configuration to Admission 
Controller.
..


Patch Set 11:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/3585/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/13307
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If47508728124076f3b9200c27cffc989f7a4f188
Gerrit-Change-Number: 13307
Gerrit-PatchSet: 11
Gerrit-Owner: Andrew Sherman 
Gerrit-Reviewer: Andrew Sherman 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 12 Jun 2019 20:53:56 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8536: Add Scalable Pool Configuration to Admission Controller.

Andrew Sherman has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13307 )

Change subject: IMPALA-8536: Add Scalable Pool Configuration to Admission 
Controller.
..


Patch Set 12:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/13307/11/be/src/scheduling/admission-controller.cc
File be/src/scheduling/admission-controller.cc:

http://gerrit.cloudera.org:8080/#/c/13307/11/be/src/scheduling/admission-controller.cc@1426
PS11, Line 1426: ca
> nit: can be
Done



--
To view, visit http://gerrit.cloudera.org:8080/13307
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If47508728124076f3b9200c27cffc989f7a4f188
Gerrit-Change-Number: 13307
Gerrit-PatchSet: 12
Gerrit-Owner: Andrew Sherman 
Gerrit-Reviewer: Andrew Sherman 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 12 Jun 2019 20:50:19 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8536: Add Scalable Pool Configuration to Admission Controller.

Andrew Sherman has uploaded a new patch set (#12). ( 
http://gerrit.cloudera.org:8080/13307 )

Change subject: IMPALA-8536: Add Scalable Pool Configuration to Admission 
Controller.
..

IMPALA-8536: Add Scalable Pool Configuration to Admission Controller.

Add 3 configuration parameters to Admission Controller that scale with
the number of hosts in the resource pool. These parameters are specified
to the Impalad through the -llama_site_path flag which points to a Llama
XML configuration file.

The new configuration parameters are:
+ Max Running Queries Multiple - this floating point number is
  multiplied by the current total number of executors at runtime to give
  the maximum number of concurrently running queries allowed in the
  pool. This calculation is rounded up to the nearest integer so the
  result will always be at least one as long as the parameter is
  non-zero.
+ Max Queued Queries Multiple - this floating point number is multiplied
  by the current total number of executors at runtime to give the
  maximum number of queries that can be queued in the pool. This
  calculation is rounded up to the nearest integer so the result will
  always be at least one as long as the parameter is non-zero.
+ Max Memory Multiple - this number of bytes is multiplied by the
  current total number of executors at runtime to give the maximum
  memory available across the cluster for the pool.
If any of these parameters have zero value then they will be ignored.
In this case the corresponding non-scalable parameters will be used, if
they are set.

The new parameters are exposed through the webui.

At various points in the code Admission Controller looks at the Pool
Config objects to find non-scalable parameters such as the max number of
queries that can run in the pool. These access have been encapsulated in
functions that return the scalable version of the configuration value if
the new scalable parameters are being used. Diagnostic messages are
enhanced to show the origin of the encapsulated parameters.

TESTING

All end-to-end tests are running clean with ASAN.

The unit test admission-controller-test.cc has been expanded to test the
newly added code.

Added an end-to-end test that adds and removes Impalads from a
minicluster.

Change-Id: If47508728124076f3b9200c27cffc989f7a4f188
---
M be/src/runtime/exec-env.cc
M be/src/scheduling/admission-controller-test.cc
M be/src/scheduling/admission-controller.cc
M be/src/scheduling/admission-controller.h
M common/thrift/ImpalaInternalService.thrift
M common/thrift/generate_metrics.py
M common/thrift/metrics.json
M fe/src/main/java/org/apache/impala/util/RequestPoolService.java
M fe/src/test/java/org/apache/impala/util/TestRequestPoolService.java
M fe/src/test/resources/fair-scheduler-test.xml
M fe/src/test/resources/fair-scheduler-test2.xml
M fe/src/test/resources/llama-site-test.xml
M fe/src/test/resources/llama-site-test2.xml
M 
testdata/workloads/functional-query/queries/QueryTest/admission-max-min-mem-limits.test
M 
testdata/workloads/functional-query/queries/QueryTest/admission-reject-mem-estimate.test
M 
testdata/workloads/functional-query/queries/QueryTest/admission-reject-min-reservation.test
M tests/common/impala_cluster.py
M tests/custom_cluster/test_admission_controller.py
M tests/webserver/test_web_pages.py
M www/admission_controller.tmpl
20 files changed, 1,400 insertions(+), 304 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/13307/12
--
To view, visit http://gerrit.cloudera.org:8080/13307
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If47508728124076f3b9200c27cffc989f7a4f188
Gerrit-Change-Number: 13307
Gerrit-PatchSet: 12
Gerrit-Owner: Andrew Sherman 
Gerrit-Reviewer: Andrew Sherman 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Tim Armstrong

[Impala-ASF-CR] IMPALA-8617: Add support for lz4 in parquet

2019-06-12 Thread Abhishek Rawat (Code Review)

Abhishek Rawat has uploaded a new patch set (#6). ( 
http://gerrit.cloudera.org:8080/13582 )

Change subject: IMPALA-8617: Add support for lz4 in parquet
..

IMPALA-8617: Add support for lz4 in parquet

A new enum value LZ4_BLOCKED was added to the THdfsCompression enum, to
distinguish it from the existing LZ4 codec. LZ4_BLOCKED codec represents
the block compression scheme used by Hadoop. Its similar to
SNAPPY_BLOCKED as far as the block format is concerned, with the only
difference being the codec used for compression and decompression.

Added Lz4BlockCompressor and Lz4BlockDecompressor classes for
compressing and decompressing parquet data using Hadoop's
lz4 block compression scheme.

The Lz4BlockCompressor treats the input
as a single block and generates a compressed block with following layout
  <4 byte big endian uncompressed size>
  <4 byte big endian compressed size>
  
The hdfs parquet table writer should call the Lz4BlockCompressor
using the ideal input size (unit of compression in parquet is a page),
and so the Lz4BlockCompressor does not further break down the input
into smaller blocks.

The Lz4BlockDecompressor on the other hand should be compatible with
blocks written by Impala and other engines in Hadoop ecosystem. It can
decompress compressed data in following format
  <4 byte big endian uncompressed size>
  <4 byte big endian compressed size>
  
  ...
  <4 byte big endian compressed size>
  
  ...
  

Externally users can now set the lz4 codec for parquet using:
  set COMPRESSION_CODEC=lz4
This gets translated into LZ4_BLOCKED codec for the
HdfsParquetTableWriter. Similarly, when reading lz4 compressed parquet
data, the LZ4_BLOCKED codec is used.

Testing:
 - Added unit tests for LZ4_BLOCKED in decompress-test.cc
 - Added unit tests for Hadoop compatibility in decompress-test.cc,
   basically being able to decompress an outer block with multiple inner
   blocks (the Lz4BlockDecompressor description above)
 - Added interoperability tests for Hive and Impala for all parquet
   codecs. New test added to
   tests/custom_cluster/test_hive_parquet_codec_interop.py

Change-Id: Ia6850a39ef3f1e0e7ba48e08eef1d4f7cbb74d0c
---
M be/src/exec/parquet/hdfs-parquet-table-writer.cc
M be/src/exec/parquet/parquet-column-readers.cc
M be/src/exec/parquet/parquet-common.cc
M be/src/exec/parquet/parquet-metadata-utils.cc
M be/src/service/query-options-test.cc
M be/src/util/codec.cc
M be/src/util/compress.cc
M be/src/util/compress.h
M be/src/util/decompress-test.cc
M be/src/util/decompress.cc
M be/src/util/decompress.h
M common/thrift/CatalogObjects.thrift
M common/thrift/generate_error_codes.py
M testdata/workloads/functional-query/queries/QueryTest/set.test
M tests/common/test_dimensions.py
A tests/custom_cluster/test_hive_parquet_codec_interop.py
M tests/query_test/test_insert.py
M tests/query_test/test_insert_parquet.py
18 files changed, 434 insertions(+), 59 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/13582/6
--
To view, visit http://gerrit.cloudera.org:8080/13582
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia6850a39ef3f1e0e7ba48e08eef1d4f7cbb74d0c
Gerrit-Change-Number: 13582
Gerrit-PatchSet: 6
Gerrit-Owner: Abhishek Rawat 
Gerrit-Reviewer: Abhishek Rawat 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Todd Lipcon

[Impala-ASF-CR] IMPALA-8617: Add support for lz4 in parquet

2019-06-12 Thread Abhishek Rawat (Code Review)

Abhishek Rawat has posted comments on this change. (
http://gerrit.cloudera.org:8080/13582 )

Change subject: IMPALA-8617: Add support for lz4 in parquet
..

Patch Set 2:

(1 comment)

Updated the error messages to include file-path and column name. I ended up
adding a new member `column_name_` in the class `BaseColumnWriter`. I was able
to easily get the file name and column name in the reader so no changes there.

http://gerrit.cloudera.org:8080/#/c/13582/2/common/thrift/generate_error_codes.py
File common/thrift/generate_error_codes.py:

http://gerrit.cloudera.org:8080/#/c/13582/2/common/thrift/generate_error_codes.py@421
PS2, Line 421:"LZ4Block: Decompressed size is not correct."),
> Yes! We should include the file path and column name in the error message.
I made the change to include file-path and column name in the error message.
Examples of error message during read/write below.

During read:
```
Parquet file
'hdfs://localhost:20500/test-warehouse/t1_lz4/75420bd1e4493fd8-7bd2b1cc_335390868_data.0.parq'
column 'c1' hit an error. LZ4Block: Decompressed size is not correct.
```

During write:
```
Parquet file
'hdfs://localhost:20500/test-warehouse/t1/_impala_insert_staging/6845d8b48f3c98fe_161dbf20/.6845d8b48f3c98fe-161dbf20_1495120350_dir/6845d8b48f3c98fe-161dbf20_825475957_data.0.parq'
column 'c1' hit an error. ZSTD_compress failed with error
```

--
To view, visit http://gerrit.cloudera.org:8080/13582
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia6850a39ef3f1e0e7ba48e08eef1d4f7cbb74d0c
Gerrit-Change-Number: 13582
Gerrit-PatchSet: 2
Gerrit-Owner: Abhishek Rawat
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Todd Lipcon
Gerrit-Comment-Date: Wed, 12 Jun 2019 20:47:46 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8633 : Insert event should not error when table does not exists

2019-06-12 Thread Vihang Karajgaonkar (Code Review)

Vihang Karajgaonkar has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13548 )

Change subject: IMPALA-8633 : Insert event should not error when table does not 
exists
..


Patch Set 4:

> (1 comment)
 >
 > Is there any e2e test we could write to stress these paths? eg if
 > you were to run the metastore poller with a 5 second frequency, and
 > you did some random sequence of hive operations, would it end up
 > triggering it? I'm wondering whether we have similar issues in
 > other events. For example, if you ALTER TABLE followed by DROP,
 > will Impala get desynchronized?

Created IMPALA-8661 to create random batch of event to stress event processor.


--
To view, visit http://gerrit.cloudera.org:8080/13548
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I961cd7cbede4c248dba538c7fabb4bc708e49693
Gerrit-Change-Number: 13548
Gerrit-PatchSet: 4
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Todd Lipcon 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Wed, 12 Jun 2019 20:44:20 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8629: (part 2) Adjust new KuduStorageHandler package

2019-06-12 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13541 )

Change subject: IMPALA-8629: (part 2) Adjust new KuduStorageHandler package
..


Patch Set 9:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/3586/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/13541
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4d0c505643247498383472704a37d27c9e1ce473
Gerrit-Change-Number: 13541
Gerrit-PatchSet: 9
Gerrit-Owner: Grant Henke 
Gerrit-Reviewer: Grant Henke 
Gerrit-Reviewer: Hao Hao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Thomas Marshall 
Gerrit-Comment-Date: Wed, 12 Jun 2019 20:43:26 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8536: Add Scalable Pool Configuration to Admission Controller.

2019-06-12 Thread Bikramjeet Vig (Code Review)

Bikramjeet Vig has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13307 )

Change subject: IMPALA-8536: Add Scalable Pool Configuration to Admission 
Controller.
..


Patch Set 11: Code-Review+1

(2 comments)

Looks good to me, I'll let Lars +2

http://gerrit.cloudera.org:8080/#/c/13307/3/be/src/scheduling/admission-controller.cc
File be/src/scheduling/admission-controller.cc:

http://gerrit.cloudera.org:8080/#/c/13307/3/be/src/scheduling/admission-controller.cc@1403
PS3, Line 1403: string, pair In HasAvailableMemResources() we effectively check the query against both
yup you are right, the per host admit_mem_limit would prevent that case.


http://gerrit.cloudera.org:8080/#/c/13307/11/be/src/scheduling/admission-controller.cc
File be/src/scheduling/admission-controller.cc:

http://gerrit.cloudera.org:8080/#/c/13307/11/be/src/scheduling/admission-controller.cc@1426
PS11, Line 1426: is
nit: can be



--
To view, visit http://gerrit.cloudera.org:8080/13307
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If47508728124076f3b9200c27cffc989f7a4f188
Gerrit-Change-Number: 13307
Gerrit-PatchSet: 11
Gerrit-Owner: Andrew Sherman 
Gerrit-Reviewer: Andrew Sherman 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 12 Jun 2019 20:41:34 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8143: Enhance DoRpcWithRetry().

2019-06-12 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12672 )

Change subject: IMPALA-8143: Enhance DoRpcWithRetry().
..


Patch Set 13: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/12672
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia9693151c35e02235665b3c285a48c585973d390
Gerrit-Change-Number: 12672
Gerrit-PatchSet: 13
Gerrit-Owner: Andrew Sherman 
Gerrit-Reviewer: Andrew Sherman 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Thomas Marshall 
Gerrit-Comment-Date: Wed, 12 Jun 2019 20:27:35 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8143: Enhance DoRpcWithRetry().

2019-06-12 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12672 )

Change subject: IMPALA-8143: Enhance DoRpcWithRetry().
..


Patch Set 13:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4462/ 
DRY_RUN=false


--
To view, visit http://gerrit.cloudera.org:8080/12672
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia9693151c35e02235665b3c285a48c585973d390
Gerrit-Change-Number: 12672
Gerrit-PatchSet: 13
Gerrit-Owner: Andrew Sherman 
Gerrit-Reviewer: Andrew Sherman 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Thomas Marshall 
Gerrit-Comment-Date: Wed, 12 Jun 2019 20:27:36 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8629: (part 2) Adjust new KuduStorageHandler package

2019-06-12 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13541 )

Change subject: IMPALA-8629: (part 2) Adjust new KuduStorageHandler package
..


Patch Set 10:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4461/ 
DRY_RUN=true


--
To view, visit http://gerrit.cloudera.org:8080/13541
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4d0c505643247498383472704a37d27c9e1ce473
Gerrit-Change-Number: 13541
Gerrit-PatchSet: 10
Gerrit-Owner: Grant Henke 
Gerrit-Reviewer: Grant Henke 
Gerrit-Reviewer: Hao Hao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Thomas Marshall 
Gerrit-Comment-Date: Wed, 12 Jun 2019 20:26:32 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8629: (part 2) Adjust new KuduStorageHandler package

2019-06-12 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13541 )

Change subject: IMPALA-8629: (part 2) Adjust new KuduStorageHandler package
..


Patch Set 10:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4460/ 
DRY_RUN=false


--
To view, visit http://gerrit.cloudera.org:8080/13541
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4d0c505643247498383472704a37d27c9e1ce473
Gerrit-Change-Number: 13541
Gerrit-PatchSet: 10
Gerrit-Owner: Grant Henke 
Gerrit-Reviewer: Grant Henke 
Gerrit-Reviewer: Hao Hao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Thomas Marshall 
Gerrit-Comment-Date: Wed, 12 Jun 2019 20:26:10 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8629: (part 2) Adjust new KuduStorageHandler package

2019-06-12 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13541 )

Change subject: IMPALA-8629: (part 2) Adjust new KuduStorageHandler package
..


Patch Set 10: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/13541
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4d0c505643247498383472704a37d27c9e1ce473
Gerrit-Change-Number: 13541
Gerrit-PatchSet: 10
Gerrit-Owner: Grant Henke 
Gerrit-Reviewer: Grant Henke 
Gerrit-Reviewer: Hao Hao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Thomas Marshall 
Gerrit-Comment-Date: Wed, 12 Jun 2019 20:26:09 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8629: (part 2) Adjust new KuduStorageHandler package

2019-06-12 Thread Thomas Marshall (Code Review)

Thomas Marshall has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13541 )

Change subject: IMPALA-8629: (part 2) Adjust new KuduStorageHandler package
..


Patch Set 9: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/13541
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4d0c505643247498383472704a37d27c9e1ce473
Gerrit-Change-Number: 13541
Gerrit-PatchSet: 9
Gerrit-Owner: Grant Henke 
Gerrit-Reviewer: Grant Henke 
Gerrit-Reviewer: Hao Hao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Thomas Marshall 
Gerrit-Comment-Date: Wed, 12 Jun 2019 20:25:50 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8629: (part 2) Adjust new KuduStorageHandler package

2019-06-12 Thread Grant Henke (Code Review)

Hello Thomas Marshall, Hao Hao, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/13541

to look at the new patch set (#9).

Change subject: IMPALA-8629: (part 2) Adjust new KuduStorageHandler package
..

IMPALA-8629: (part 2) Adjust new KuduStorageHandler package

This patch changes the new KuduStorageHandler
package from “org.apache.kudu.hive” to
“org.apache.hadoop.hive.kudu”.

This is done to ensure the stand-in storage handler
can be a real storage handler when a Hive integration
is added in the future. The “org.apache.hadoop.hive”
package is the standard package all Hive storage
handlers lives under.

Additionally this patch updates the stand-in InputFormat,
OutputFormat, and SerDe entries for Kudu. This allows a
future Hive integration to read HMS tables/entries created
by Impala.

This patch also includes PlannerTest fixes to handle the
ScanToken stringification changes in the newer version
of Kudu.

Change-Id: I4d0c505643247498383472704a37d27c9e1ce473
---
M bin/impala-config.sh
M fe/src/main/java/org/apache/impala/catalog/HdfsFileFormat.java
M fe/src/main/java/org/apache/impala/catalog/KuduTable.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java
M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java
M testdata/workloads/functional-planner/queries/PlannerTest/kudu.test
M tests/custom_cluster/test_kudu.py
M tests/query_test/test_kudu.py
9 files changed, 45 insertions(+), 50 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/41/13541/9
--
To view, visit http://gerrit.cloudera.org:8080/13541
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4d0c505643247498383472704a37d27c9e1ce473
Gerrit-Change-Number: 13541
Gerrit-PatchSet: 9
Gerrit-Owner: Grant Henke 
Gerrit-Reviewer: Grant Henke 
Gerrit-Reviewer: Hao Hao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Thomas Marshall

[Impala-ASF-CR] IMPALA-8654: Log the SQL statement in the Ranger audit log

2019-06-12 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13590 )

Change subject: IMPALA-8654: Log the SQL statement in the Ranger audit log
..


Patch Set 4:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/3584/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/13590
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id9f584ac4209604675eb13b6d6f349c6cbb1a387
Gerrit-Change-Number: 13590
Gerrit-PatchSet: 4
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Todd Lipcon 
Gerrit-Comment-Date: Wed, 12 Jun 2019 20:21:37 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8536: Add Scalable Pool Configuration to Admission Controller.

2019-06-12 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13307 )

Change subject: IMPALA-8536: Add Scalable Pool Configuration to Admission 
Controller.
..


Patch Set 11:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4459/ 
DRY_RUN=true


--
To view, visit http://gerrit.cloudera.org:8080/13307
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If47508728124076f3b9200c27cffc989f7a4f188
Gerrit-Change-Number: 13307
Gerrit-PatchSet: 11
Gerrit-Owner: Andrew Sherman 
Gerrit-Reviewer: Andrew Sherman 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 12 Jun 2019 20:15:32 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8536: Add Scalable Pool Configuration to Admission Controller.

Andrew Sherman has uploaded a new patch set (#11). ( 
http://gerrit.cloudera.org:8080/13307 )

Change subject: IMPALA-8536: Add Scalable Pool Configuration to Admission 
Controller.
..

IMPALA-8536: Add Scalable Pool Configuration to Admission Controller.

Add 3 configuration parameters to Admission Controller that scale with
the number of hosts in the resource pool. These parameters are specified
to the Impalad through the -llama_site_path flag which points to a Llama
XML configuration file.

The new configuration parameters are:
+ Max Running Queries Multiple - this floating point number is
  multiplied by the current total number of executors at runtime to give
  the maximum number of concurrently running queries allowed in the
  pool. This calculation is rounded up to the nearest integer so the
  result will always be at least one as long as the parameter is
  non-zero.
+ Max Queued Queries Multiple - this floating point number is multiplied
  by the current total number of executors at runtime to give the
  maximum number of queries that can be queued in the pool. This
  calculation is rounded up to the nearest integer so the result will
  always be at least one as long as the parameter is non-zero.
+ Max Memory Multiple - this number of bytes is multiplied by the
  current total number of executors at runtime to give the maximum
  memory available across the cluster for the pool.
If any of these parameters have zero value then they will be ignored.
In this case the corresponding non-scalable parameters will be used, if
they are set.

The new parameters are exposed through the webui.

At various points in the code Admission Controller looks at the Pool
Config objects to find non-scalable parameters such as the max number of
queries that can run in the pool. These access have been encapsulated in
functions that return the scalable version of the configuration value if
the new scalable parameters are being used. Diagnostic messages are
enhanced to show the origin of the encapsulated parameters.

TESTING

All end-to-end tests are running clean with ASAN.

The unit test admission-controller-test.cc has been expanded to test the
newly added code.

Added an end-to-end test that adds and removes Impalads from a
minicluster.

Change-Id: If47508728124076f3b9200c27cffc989f7a4f188
---
M be/src/runtime/exec-env.cc
M be/src/scheduling/admission-controller-test.cc
M be/src/scheduling/admission-controller.cc
M be/src/scheduling/admission-controller.h
M common/thrift/ImpalaInternalService.thrift
M common/thrift/generate_metrics.py
M common/thrift/metrics.json
M fe/src/main/java/org/apache/impala/util/RequestPoolService.java
M fe/src/test/java/org/apache/impala/util/TestRequestPoolService.java
M fe/src/test/resources/fair-scheduler-test.xml
M fe/src/test/resources/fair-scheduler-test2.xml
M fe/src/test/resources/llama-site-test.xml
M fe/src/test/resources/llama-site-test2.xml
M 
testdata/workloads/functional-query/queries/QueryTest/admission-max-min-mem-limits.test
M 
testdata/workloads/functional-query/queries/QueryTest/admission-reject-mem-estimate.test
M 
testdata/workloads/functional-query/queries/QueryTest/admission-reject-min-reservation.test
M tests/common/impala_cluster.py
M tests/custom_cluster/test_admission_controller.py
M tests/webserver/test_web_pages.py
M www/admission_controller.tmpl
20 files changed, 1,400 insertions(+), 304 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/13307/11
--
To view, visit http://gerrit.cloudera.org:8080/13307
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If47508728124076f3b9200c27cffc989f7a4f188
Gerrit-Change-Number: 13307
Gerrit-PatchSet: 11
Gerrit-Owner: Andrew Sherman 
Gerrit-Reviewer: Andrew Sherman 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Tim Armstrong

[Impala-ASF-CR] IMPALA-8536: Add Scalable Pool Configuration to Admission Controller.

Andrew Sherman has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13307 )

Change subject: IMPALA-8536: Add Scalable Pool Configuration to Admission 
Controller.
..


Patch Set 10:

(1 comment)

New patch coming to remove the log file I included in my change

http://gerrit.cloudera.org:8080/#/c/13307/10/logs/cluster/impalad.INFO
File logs/cluster/impalad.INFO:

http://gerrit.cloudera.org:8080/#/c/13307/10/logs/cluster/impalad.INFO@1
PS10, Line 1: impalad.asherman-desktop.asherman.log.INFO.20190611-162407.11081
This file should not be here :-(



--
To view, visit http://gerrit.cloudera.org:8080/13307
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If47508728124076f3b9200c27cffc989f7a4f188
Gerrit-Change-Number: 13307
Gerrit-PatchSet: 10
Gerrit-Owner: Andrew Sherman 
Gerrit-Reviewer: Andrew Sherman 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 12 Jun 2019 20:02:59 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8629: (part 2) Adjust new KuduStorageHandler package

2019-06-12 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13541 )

Change subject: IMPALA-8629: (part 2) Adjust new KuduStorageHandler package
..


Patch Set 8: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4453/


--
To view, visit http://gerrit.cloudera.org:8080/13541
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4d0c505643247498383472704a37d27c9e1ce473
Gerrit-Change-Number: 13541
Gerrit-PatchSet: 8
Gerrit-Owner: Grant Henke 
Gerrit-Reviewer: Grant Henke 
Gerrit-Reviewer: Hao Hao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Thomas Marshall 
Gerrit-Comment-Date: Wed, 12 Jun 2019 19:41:44 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8654: Log the SQL statement in the Ranger audit log

2019-06-12 Thread Fredy Wijaya (Code Review)

Fredy Wijaya has uploaded a new patch set (#4). ( 
http://gerrit.cloudera.org:8080/13590 )

Change subject: IMPALA-8654: Log the SQL statement in the Ranger audit log
..

IMPALA-8654: Log the SQL statement in the Ranger audit log

This patch logs the SQL statement to be authorized in the Ranger audit
log since it was a required field prior to the fix in RANGER-2463 to
avoid a NullPointerException in the Ranger admin that could cause the
Ranger audit logs to not show up in the Ranger web UI.

Testing:
- Updated the RangerAuditLogTest
- Tested the changes against Solr cluster
- Ran all FE tests

Change-Id: Id9f584ac4209604675eb13b6d6f349c6cbb1a387
---
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/CreateDbStmt.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java
M fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java
M fe/src/main/java/org/apache/impala/authorization/NoopAuthorizationFactory.java
M 
fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
M 
fe/src/main/java/org/apache/impala/authorization/ranger/RangerBufferAuditHandler.java
M 
fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java
M 
fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java
M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
10 files changed, 71 insertions(+), 12 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/13590/4
--
To view, visit http://gerrit.cloudera.org:8080/13590
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id9f584ac4209604675eb13b6d6f349c6cbb1a387
Gerrit-Change-Number: 13590
Gerrit-PatchSet: 4
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Todd Lipcon

[Impala-ASF-CR] IMPALA-7154: Increase metastore client time out for tests

2019-06-12 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13586 )

Change subject: IMPALA-7154: Increase metastore client time out for tests
..


Patch Set 5:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/3583/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/13586
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I253fc00618b28d8d6e626e1b6bf9c5c8290006d7
Gerrit-Change-Number: 13586
Gerrit-PatchSet: 5
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Wed, 12 Jun 2019 19:12:08 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7154: Increase metastore client time out for tests

2019-06-12 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13586 )

Change subject: IMPALA-7154: Increase metastore client time out for tests
..


Patch Set 4:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/3582/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/13586
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I253fc00618b28d8d6e626e1b6bf9c5c8290006d7
Gerrit-Change-Number: 13586
Gerrit-PatchSet: 4
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Wed, 12 Jun 2019 19:10:08 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7608: Estimate row count from file size when no stats available

2019-06-12 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12974 )

Change subject: IMPALA-7608: Estimate row count from file size when no stats 
available
..


Patch Set 15:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4458/ 
DRY_RUN=false


--
To view, visit http://gerrit.cloudera.org:8080/12974
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic414121c8df0d5222e4aeea096b5365beb04568a
Gerrit-Change-Number: 12974
Gerrit-PatchSet: 15
Gerrit-Owner: Fang-Yu Rao 
Gerrit-Reviewer: Fang-Yu Rao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Paul Rogers 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Todd Lipcon 
Gerrit-Comment-Date: Wed, 12 Jun 2019 18:54:26 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8593: Prohibit write operations for bucketed tables

2019-06-12 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13558 )

Change subject: IMPALA-8593: Prohibit write operations for bucketed tables
..


Patch Set 3:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/3581/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/13558
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia08d01168660830b6e0d08b55a95eac129889cec
Gerrit-Change-Number: 13558
Gerrit-PatchSet: 3
Gerrit-Owner: Yongzhi Chen 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Sudhanshu Arora 
Gerrit-Reviewer: Todd Lipcon 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Reviewer: Yongzhi Chen 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Wed, 12 Jun 2019 18:57:57 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8536: Add Scalable Pool Configuration to Admission Controller.

2019-06-12 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13307 )

Change subject: IMPALA-8536: Add Scalable Pool Configuration to Admission 
Controller.
..


Patch Set 10:

Build Failed

https://jenkins.impala.io/job/gerrit-code-review-checks/3580/ : Initial code 
review checks failed. See linked job for details on the failure.


--
To view, visit http://gerrit.cloudera.org:8080/13307
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If47508728124076f3b9200c27cffc989f7a4f188
Gerrit-Change-Number: 13307
Gerrit-PatchSet: 10
Gerrit-Owner: Andrew Sherman 
Gerrit-Reviewer: Andrew Sherman 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 12 Jun 2019 18:55:42 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7608: Estimate row count from file size when no stats available

Tim Armstrong has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12974 )

Change subject: IMPALA-7608: Estimate row count from file size when no stats 
available
..


Patch Set 14: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/12974
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic414121c8df0d5222e4aeea096b5365beb04568a
Gerrit-Change-Number: 12974
Gerrit-PatchSet: 14
Gerrit-Owner: Fang-Yu Rao 
Gerrit-Reviewer: Fang-Yu Rao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Paul Rogers 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Todd Lipcon 
Gerrit-Comment-Date: Wed, 12 Jun 2019 18:54:03 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7608: Estimate row count from file size when no stats available

2019-06-12 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12974 )

Change subject: IMPALA-7608: Estimate row count from file size when no stats 
available
..


Patch Set 15: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/12974
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic414121c8df0d5222e4aeea096b5365beb04568a
Gerrit-Change-Number: 12974
Gerrit-PatchSet: 15
Gerrit-Owner: Fang-Yu Rao 
Gerrit-Reviewer: Fang-Yu Rao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Paul Rogers 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Todd Lipcon 
Gerrit-Comment-Date: Wed, 12 Jun 2019 18:54:25 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7154: Increase metastore client time out for tests

2019-06-12 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13586 )

Change subject: IMPALA-7154: Increase metastore client time out for tests
..


Patch Set 6: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/13586
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I253fc00618b28d8d6e626e1b6bf9c5c8290006d7
Gerrit-Change-Number: 13586
Gerrit-PatchSet: 6
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Wed, 12 Jun 2019 18:36:50 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7154: Increase metastore client time out for tests

2019-06-12 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13586 )

Change subject: IMPALA-7154: Increase metastore client time out for tests
..


Patch Set 6:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4457/ 
DRY_RUN=false


--
To view, visit http://gerrit.cloudera.org:8080/13586
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I253fc00618b28d8d6e626e1b6bf9c5c8290006d7
Gerrit-Change-Number: 13586
Gerrit-PatchSet: 6
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Wed, 12 Jun 2019 18:36:51 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7154: Increase metastore client time out for tests

2019-06-12 Thread Vihang Karajgaonkar (Code Review)

Vihang Karajgaonkar has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13586 )

Change subject: IMPALA-7154: Increase metastore client time out for tests
..


Patch Set 5: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/13586
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I253fc00618b28d8d6e626e1b6bf9c5c8290006d7
Gerrit-Change-Number: 13586
Gerrit-PatchSet: 5
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Wed, 12 Jun 2019 18:32:02 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7154: Increase metastore client time out for tests

2019-06-12 Thread Vihang Karajgaonkar (Code Review)

Vihang Karajgaonkar has uploaded a new patch set (#5). ( 
http://gerrit.cloudera.org:8080/13586 )

Change subject: IMPALA-7154: Increase metastore client time out for tests
..

IMPALA-7154: Increase metastore client time out for tests

Some of the e2e tests create tables and databases on S3. HMS operations
like dropDatabaseCascade can take significant time (3-5 min) in such
tests. If the HMS Client times out, it retries for a configurable number
of times. Unfortunately, HMS client is not smart enough to understand
which operations should be retried. Eg, a drop database operation which
is timedout on the client side, should not be retried since the server
thread may have succeeded before the next retry. This would cause the
client to receive a NoSuchObjectException which would look like a flaky
test failure on the Impala side.

This patch increases the metastore client.socket.timeout for
the tests so that such flaky test failures are reduced.

Note:
1. The timeout value of 10 min was chosen based on looking at logs for
one such failure where the HMS API took about 3.5 min. Once this patch
is deployed we should continue to see if such failures reoccur and
readjust the client timeout value accordingly.

Change-Id: I253fc00618b28d8d6e626e1b6bf9c5c8290006d7
---
M fe/src/test/resources/hive-site.xml.py
1 file changed, 9 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/13586/5
--
To view, visit http://gerrit.cloudera.org:8080/13586
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I253fc00618b28d8d6e626e1b6bf9c5c8290006d7
Gerrit-Change-Number: 13586
Gerrit-PatchSet: 5
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Vihang Karajgaonkar

[Impala-ASF-CR] IMPALA-7154: Increase metastore client time out for tests

2019-06-12 Thread Vihang Karajgaonkar (Code Review)

Vihang Karajgaonkar has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13586 )

Change subject: IMPALA-7154: Increase metastore client time out for tests
..


Patch Set 2:

Carrying forward Lar's +2 from earlier.


--
To view, visit http://gerrit.cloudera.org:8080/13586
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I253fc00618b28d8d6e626e1b6bf9c5c8290006d7
Gerrit-Change-Number: 13586
Gerrit-PatchSet: 2
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Wed, 12 Jun 2019 18:31:35 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7154: Increase metastore client time out for tests

2019-06-12 Thread Vihang Karajgaonkar (Code Review)

Vihang Karajgaonkar has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13586 )

Change subject: IMPALA-7154: Increase metastore client time out for tests
..


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/13586/2/fe/src/test/resources/hive-site.xml.py
File fe/src/test/resources/hive-site.xml.py:

http://gerrit.cloudera.org:8080/#/c/13586/2/fe/src/test/resources/hive-site.xml.py@36
PS2, Line 36: retry.delay
> Ah, thanks for the explanation. It seems non-obvious enough to warrant a br
added a comment. Thanks



--
To view, visit http://gerrit.cloudera.org:8080/13586
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I253fc00618b28d8d6e626e1b6bf9c5c8290006d7
Gerrit-Change-Number: 13586
Gerrit-PatchSet: 2
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Wed, 12 Jun 2019 18:31:19 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7154: Increase metastore client time out for tests

2019-06-12 Thread Lars Volker (Code Review)

Lars Volker has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13586 )

Change subject: IMPALA-7154: Increase metastore client time out for tests
..


Patch Set 4: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/13586
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I253fc00618b28d8d6e626e1b6bf9c5c8290006d7
Gerrit-Change-Number: 13586
Gerrit-PatchSet: 4
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Wed, 12 Jun 2019 18:27:10 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7154: Increase metastore client time out for tests

2019-06-12 Thread Vihang Karajgaonkar (Code Review)

Vihang Karajgaonkar has uploaded a new patch set (#4). ( 
http://gerrit.cloudera.org:8080/13586 )

Change subject: IMPALA-7154: Increase metastore client time out for tests
..

IMPALA-7154: Increase metastore client time out for tests

Some of the e2e tests create tables and databases on S3. HMS operations
like dropDatabaseCascade can take significant time (3-5 min) in such
tests. If the HMS Client times out, it retries for a configurable number
of times. Unfortunately, HMS client is not smart enough to understand
which operations should be retried. Eg, a drop database operation which
is timedout on the client side, should not be retried since the server
thread may have succeeded before the next retry. This would cause the
client to receive a NoSuchObjectException which would look like a flaky
test failure on the Impala side.

This patch increases the metastore client.socket.timeout for
the tests so that such flaky test failures are reduced.

Note:
1. The timeout value of 10 min was chosen based on looking at logs for
one such failure where the HMS API took about 3.5 min. Once this patch
is deployed we should continue to see if such failures reoccur and
readjust the client timeout value accordingly.

Change-Id: I253fc00618b28d8d6e626e1b6bf9c5c8290006d7
---
M fe/src/test/resources/hive-site.xml.py
1 file changed, 5 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/13586/4
--
To view, visit http://gerrit.cloudera.org:8080/13586
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I253fc00618b28d8d6e626e1b6bf9c5c8290006d7
Gerrit-Change-Number: 13586
Gerrit-PatchSet: 4
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Vihang Karajgaonkar

[Impala-ASF-CR] IMPALA-7154: Increase metastore client time out for tests

2019-06-12 Thread Vihang Karajgaonkar (Code Review)

Vihang Karajgaonkar has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13586 )

Change subject: IMPALA-7154: Increase metastore client time out for tests
..


Patch Set 4:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/13586/2//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/13586/2//COMMIT_MSG@7
PS2, Line 7: IMPALA-7154: Increase metastore client time out for tests
> The first line of the commit message should contain a brief description of
Done


http://gerrit.cloudera.org:8080/#/c/13586/2//COMMIT_MSG@23
PS2, Line 23: 10 mi
> Stale comment?
yeah, Thanks for catching this.



--
To view, visit http://gerrit.cloudera.org:8080/13586
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I253fc00618b28d8d6e626e1b6bf9c5c8290006d7
Gerrit-Change-Number: 13586
Gerrit-PatchSet: 4
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Wed, 12 Jun 2019 18:26:52 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7154: Increase metastore client time out for tests

2019-06-12 Thread Lars Volker (Code Review)

Lars Volker has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13586 )

Change subject: IMPALA-7154: Increase metastore client time out for tests
..


Patch Set 3:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/13586/2//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/13586/2//COMMIT_MSG@7
PS2, Line 7: IMPALA-7154 : Error making 'dropDatabase' RPC to Hive Metastore. 
NoSuchObjectException thrown
> The first line of the commit message should contain a brief description of
ping?


http://gerrit.cloudera.org:8080/#/c/13586/2/fe/src/test/resources/hive-site.xml.py
File fe/src/test/resources/hive-site.xml.py:

http://gerrit.cloudera.org:8080/#/c/13586/2/fe/src/test/resources/hive-site.xml.py@36
PS2, Line 36: retry.delay
> It does not.  This is the delay between successive retry attempts. Although
Ah, thanks for the explanation. It seems non-obvious enough to warrant a brief 
comment in the file itself, but I'll let you decide.



--
To view, visit http://gerrit.cloudera.org:8080/13586
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I253fc00618b28d8d6e626e1b6bf9c5c8290006d7
Gerrit-Change-Number: 13586
Gerrit-PatchSet: 3
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Wed, 12 Jun 2019 18:25:37 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7154 : Error making 'dropDatabase' RPC to Hive Metastore. NoSuchObjectException thrown

2019-06-12 Thread Vihang Karajgaonkar (Code Review)

Vihang Karajgaonkar has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13586 )

Change subject: IMPALA-7154 : Error making 'dropDatabase' RPC to Hive 
Metastore. NoSuchObjectException thrown
..


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/13586/2/fe/src/test/resources/hive-site.xml.py
File fe/src/test/resources/hive-site.xml.py:

http://gerrit.cloudera.org:8080/#/c/13586/2/fe/src/test/resources/hive-site.xml.py@36
PS2, Line 36: retry.delay
> Does this enable retries? >From the commit message I understood that we shou
It does not.  This is the delay between successive retry attempts. Although 
this change is really unrelated to the issue, I thought of changing it since I 
found it unusual to have a retry delay of 0.

By default, Metastore client retries once. I don't want to disable retries just 
yet since I am not really sure it is indeed helpful in some cases like 
connectivity issues with HMS or the DB during the tests. If we see more retry 
related failures, then perhaps we should disable the retries.



--
To view, visit http://gerrit.cloudera.org:8080/13586
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I253fc00618b28d8d6e626e1b6bf9c5c8290006d7
Gerrit-Change-Number: 13586
Gerrit-PatchSet: 2
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Wed, 12 Jun 2019 18:23:49 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7154 : Error making 'dropDatabase' RPC to Hive Metastore. NoSuchObjectException thrown

2019-06-12 Thread Vihang Karajgaonkar (Code Review)

Hello Lars Volker, Fredy Wijaya, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/13586

to look at the new patch set (#3).

Change subject: IMPALA-7154 : Error making 'dropDatabase' RPC to Hive 
Metastore. NoSuchObjectException thrown
..

IMPALA-7154 : Error making 'dropDatabase' RPC to Hive Metastore. 
NoSuchObjectException thrown

Some of the e2e tests create tables and databases on S3. HMS operations
like dropDatabaseCascade can take significant time (3-5 min) in such
tests. If the HMS Client times out, it retries for a configurable number
of times. Unfortunately, HMS client is not smart enough to understand
which operations should be retried. Eg, a drop database operation which
is timed out on the client side, should not be retried since the server
thread may have succeeded before the next retry. This would cause the
client to receive a NoSuchObjectException which would look like a flaky
test failure on the Impala side.

This patch increases the metastore client.socket.timeout for
the tests so that such flaky test failures are reduced.

Note:
1. The timeout value of 10 min was chosen based on looking at logs for
one such failure where the HMS API took about 3.5 min. Once this patch
is deployed we should continue to see if such failures reoccur and
readjust the client timeout value accordingly.

Change-Id: I253fc00618b28d8d6e626e1b6bf9c5c8290006d7
---
M fe/src/test/resources/hive-site.xml.py
1 file changed, 5 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/13586/3
--
To view, visit http://gerrit.cloudera.org:8080/13586
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I253fc00618b28d8d6e626e1b6bf9c5c8290006d7
Gerrit-Change-Number: 13586
Gerrit-PatchSet: 3
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Vihang Karajgaonkar

[Impala-ASF-CR] IMPALA-8650: Docker build should not depend on test config

2019-06-12 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13597 )

Change subject: IMPALA-8650: Docker build should not depend on test config
..


Patch Set 1:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/3579/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/13597
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa70864f5d047d1ff5f21e69d8f6358306424c0b
Gerrit-Change-Number: 13597
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Wed, 12 Jun 2019 18:22:39 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8593: Prohibit write operations for bucketed tables

2019-06-12 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13558 )

Change subject: IMPALA-8593: Prohibit write operations for bucketed tables
..


Patch Set 3:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/13558/3/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:

http://gerrit.cloudera.org:8080/#/c/13558/3/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@20
PS3, Line 20: import static 
org.apache.hadoop.hive.metastore.api.hive_metastoreConstants.ACCESSTYPE_NONE;
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/13558/3/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@21
PS3, Line 21: import static 
org.apache.hadoop.hive.metastore.api.hive_metastoreConstants.ACCESSTYPE_READONLY;
line too long (95 > 90)


http://gerrit.cloudera.org:8080/#/c/13558/3/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@22
PS3, Line 22: import static 
org.apache.hadoop.hive.metastore.api.hive_metastoreConstants.ACCESSTYPE_READWRITE;
line too long (96 > 90)


http://gerrit.cloudera.org:8080/#/c/13558/3/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@23
PS3, Line 23: import static 
org.apache.hadoop.hive.metastore.api.hive_metastoreConstants.ACCESSTYPE_WRITEONLY;
line too long (96 > 90)



--
To view, visit http://gerrit.cloudera.org:8080/13558
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia08d01168660830b6e0d08b55a95eac129889cec
Gerrit-Change-Number: 13558
Gerrit-PatchSet: 3
Gerrit-Owner: Yongzhi Chen 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Sudhanshu Arora 
Gerrit-Reviewer: Todd Lipcon 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Reviewer: Yongzhi Chen 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Wed, 12 Jun 2019 18:16:26 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8593: Prohibit write operations for bucketed tables

2019-06-12 Thread Yongzhi Chen (Code Review)

Hello Vihang Karajgaonkar, Sudhanshu Arora, Zoltan Borok-Nagy, Todd Lipcon, Joe 
McDonnell, Csaba Ringhofer, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/13558

to look at the new patch set (#3).

Change subject: IMPALA-8593: Prohibit write operations for bucketed tables
..

IMPALA-8593: Prohibit write operations for bucketed tables

This patch adds a  method to check if a table bucketed.
For Hive 3, integrates with HMS translation layer for
capabilities checks.
Implements methods ensureTableWriteSupported and
ensureTableReadSupported.

Tests:
Added unit tests to ParserTest and AnalyzerTest.
Added bucketed tables which are required by IMPALA-8439.
Ran core tests.

ToDo:
Integrate checking bucketed tables capabilities and creating
error messages with HMS translation after Hive provides the
required functions.

Change-Id: Ia08d01168660830b6e0d08b55a95eac129889cec
---
M bin/impala-config.sh
M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeStmt.java
M fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java
M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java
M fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java
M fe/src/test/java/org/apache/impala/analysis/ParserTest.java
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
19 files changed, 354 insertions(+), 13 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/58/13558/3
--
To view, visit http://gerrit.cloudera.org:8080/13558
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia08d01168660830b6e0d08b55a95eac129889cec
Gerrit-Change-Number: 13558
Gerrit-PatchSet: 3
Gerrit-Owner: Yongzhi Chen 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Sudhanshu Arora 
Gerrit-Reviewer: Todd Lipcon 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Reviewer: Yongzhi Chen 
Gerrit-Reviewer: Zoltan Borok-Nagy

[Impala-ASF-CR] IMPALA-8593: Prohibit write operations for bucketed tables

2019-06-12 Thread Yongzhi Chen (Code Review)

Yongzhi Chen has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13558 )

Change subject: IMPALA-8593: Prohibit write operations for bucketed tables
..


Patch Set 2:

(12 comments)

Patch 3 will address the issues.

http://gerrit.cloudera.org:8080/#/c/13558/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:

http://gerrit.cloudera.org:8080/#/c/13558/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@78
PS1, Line 78:   private static final String CONNECTORREAD = 
"CONNECTORREAD".intern();
> Just to be clear I was saying to do the following
These strings are only used to be packed into a string array and sent to hive 
as capabilities. I will use enum in the future if we need enum operations.
I will remove the intern (just copied from hive code)


http://gerrit.cloudera.org:8080/#/c/13558/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@92
PS1, Line 92:   private static final long MAJOR_VERSION = 3;
> nit: HIVE_MAJOR_VERSION?
It is Impala shim's major version.


http://gerrit.cloudera.org:8080/#/c/13558/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@437
PS1, Line 437:* Set impala capabilities to hive client
> nit: could you elaborate a little? E.g. why is it needed, what are these ca
Done


http://gerrit.cloudera.org:8080/#/c/13558/1/fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java
File fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java:

http://gerrit.cloudera.org:8080/#/c/13558/1/fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java@93
PS1, Line 93: analyzer.ensureTableWriteSupported(table_);
> 1) If Hive does not define Metadata_modification and we want to fold this o
Bitwise can combine two checks into one (for example check write along with 
read/write).  The major purpose is to integrate with hive's translation layer, 
hive access type is in byte, bitwise operations work fine with it and may have 
better performance and simpler (save a couple of switch statements)
This function is frequently called,  checking annotation value for operation 
type may be roundabout.  I will accept your first option to use enum .


http://gerrit.cloudera.org:8080/#/c/13558/1/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:

http://gerrit.cloudera.org:8080/#/c/13558/1/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@129
PS1, Line 129:   public static final byte REQUIRE_READ = (byte)2;
> Why not import hive_metastoreConstants access types?
It is auto-generated code and Hive 2 does not have these constant. We do not 
want to break hive 2 impala.


http://gerrit.cloudera.org:8080/#/c/13558/1/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@150
PS1, Line 150: "Table %s not supported. Bucketed tables are only supported " +
 :   "for read."
> nit: How about "%s is a bucketed table. Only read operations are supported
Done


http://gerrit.cloudera.org:8080/#/c/13558/1/fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java:

http://gerrit.cloudera.org:8080/#/c/13558/1/fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java@876
PS1, Line 876: if (MetastoreShim.getMajorVersion() > 2) {
> Maybe you could add tests for the other statements as well, e.g. COMPUTE ST
Done


http://gerrit.cloudera.org:8080/#/c/13558/1/testdata/datasets/functional/functional_schema_template.sql
File testdata/datasets/functional/functional_schema_template.sql:

http://gerrit.cloudera.org:8080/#/c/13558/1/testdata/datasets/functional/functional_schema_template.sql@2183
PS1, Line 2183:  LOAD
  : INSERT OVERWRITE TABLE {db_name}{db_suffix}.{table_name}
  : SELECT id, int_col from functional.alltypes;
> This will only run for the non-text versions of the tables. If you want the
Done


http://gerrit.cloudera.org:8080/#/c/13558/2/testdata/datasets/functional/functional_schema_template.sql
File testdata/datasets/functional/functional_schema_template.sql:

http://gerrit.cloudera.org:8080/#/c/13558/2/testdata/datasets/functional/functional_schema_template.sql@2172
PS2, Line 2172: CLUSTERED BY (col1) INTO 5 BUCKETS
> Without STORED AS {file_format} all tables will be text files.
Done


http://gerrit.cloudera.org:8080/#/c/13558/2/testdata/datasets/functional/functional_schema_template.sql@2173
PS2, Line 2173: LOCATION '/test-warehouse/{table_name}';
> Adding {db_name}{db_suffix} would ensure that tables in different databases
Done


http://gerrit.cloudera.org:8080/#/c/13558/2/testdata/datasets/functional/functional_schema_template.sql@2182
PS2, Line 2182: CLUSTERED BY (col1) INTO 5 BUCKETS;
> Same as line 2172.
Done

[Impala-ASF-CR] IMPALA-8536: Add Scalable Pool Configuration to Admission Controller.

Andrew Sherman has uploaded a new patch set (#10). ( 
http://gerrit.cloudera.org:8080/13307 )

Change subject: IMPALA-8536: Add Scalable Pool Configuration to Admission 
Controller.
..

IMPALA-8536: Add Scalable Pool Configuration to Admission Controller.

Add 3 configuration parameters to Admission Controller that scale with
the number of hosts in the resource pool. These parameters are specified
to the Impalad through the -llama_site_path flag which points to a Llama
XML configuration file.

The new configuration parameters are:
+ Max Running Queries Multiple - this floating point number is
  multiplied by the current total number of executors at runtime to give
  the maximum number of concurrently running queries allowed in the
  pool. This calculation is rounded up to the nearest integer so the
  result will always be at least one as long as the parameter is
  non-zero.
+ Max Queued Queries Multiple - this floating point number is multiplied
  by the current total number of executors at runtime to give the
  maximum number of queries that can be queued in the pool. This
  calculation is rounded up to the nearest integer so the result will
  always be at least one as long as the parameter is non-zero.
+ Max Memory Multiple - this number of bytes is multiplied by the
  current total number of executors at runtime to give the maximum
  memory available across the cluster for the pool.
If any of these parameters have zero value then they will be ignored.
In this case the corresponding non-scalable parameters will be used, if
they are set.

The new parameters are exposed through the webui.

At various points in the code Admission Controller looks at the Pool
Config objects to find non-scalable parameters such as the max number of
queries that can run in the pool. These access have been encapsulated in
functions that return the scalable version of the configuration value if
the new scalable parameters are being used. Diagnostic messages are
enhanced to show the origin of the encapsulated parameters.

TESTING

All end-to-end tests are running clean with ASAN.

The unit test admission-controller-test.cc has been expanded to test the
newly added code.

Added an end-to-end test that adds and removes Impalads from a
minicluster.

Change-Id: If47508728124076f3b9200c27cffc989f7a4f188
---
M be/src/runtime/exec-env.cc
M be/src/scheduling/admission-controller-test.cc
M be/src/scheduling/admission-controller.cc
M be/src/scheduling/admission-controller.h
M common/thrift/ImpalaInternalService.thrift
M common/thrift/generate_metrics.py
M common/thrift/metrics.json
M fe/src/main/java/org/apache/impala/util/RequestPoolService.java
M fe/src/test/java/org/apache/impala/util/TestRequestPoolService.java
M fe/src/test/resources/fair-scheduler-test.xml
M fe/src/test/resources/fair-scheduler-test2.xml
M fe/src/test/resources/llama-site-test.xml
M fe/src/test/resources/llama-site-test2.xml
A logs/cluster/impalad.INFO
M 
testdata/workloads/functional-query/queries/QueryTest/admission-max-min-mem-limits.test
M 
testdata/workloads/functional-query/queries/QueryTest/admission-reject-mem-estimate.test
M 
testdata/workloads/functional-query/queries/QueryTest/admission-reject-min-reservation.test
M tests/common/impala_cluster.py
M tests/custom_cluster/test_admission_controller.py
M tests/webserver/test_web_pages.py
M www/admission_controller.tmpl
21 files changed, 1,401 insertions(+), 304 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/13307/10
--
To view, visit http://gerrit.cloudera.org:8080/13307
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If47508728124076f3b9200c27cffc989f7a4f188
Gerrit-Change-Number: 13307
Gerrit-PatchSet: 10
Gerrit-Owner: Andrew Sherman 
Gerrit-Reviewer: Andrew Sherman 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Tim Armstrong

[Impala-ASF-CR] IMPALA-8536: Add Scalable Pool Configuration to Admission Controller.

Andrew Sherman has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13307 )

Change subject: IMPALA-8536: Add Scalable Pool Configuration to Admission 
Controller.
..


Patch Set 10:

(17 comments)

Thanks for the reviews

http://gerrit.cloudera.org:8080/#/c/13307/8/be/src/scheduling/admission-controller-test.cc
File be/src/scheduling/admission-controller-test.cc:

http://gerrit.cloudera.org:8080/#/c/13307/8/be/src/scheduling/admission-controller-test.cc@317
PS8, Line 317:  CheckRoundingForPool(ac, /*expected*/ 3, /*parameter*/ 0.3, 
/*num hosts*/ 10
> nit: repeated test
Done


http://gerrit.cloudera.org:8080/#/c/13307/8/be/src/scheduling/admission-controller-test.cc@323
PS8, Line 323:   CheckRoundingForPool(ac, /*expected*/ 1, /*parameter*/ 
100, /*num hosts*/ 100);
> how about another one: CheckRoundingForPool(ac, /*expected*/ 2, /*parameter
Thanks


http://gerrit.cloudera.org:8080/#/c/13307/8/be/src/scheduling/admission-controller-test.cc@327
PS8, Line 327: CanAdmitRequest
> this test is only testing the functionality inside HasAvailableMemResources
Done


http://gerrit.cloudera.org:8080/#/c/13307/8/be/src/scheduling/admission-controller-test.cc@353
PS8, Line 353:   // Check that the query can be admitted.
 :   string not_admitted_reason;
 :   ASSERT_TRUE(admission_controller->CanAdmitRequest(
 :   *query_schedule, config_d, host_count, true, 
_admitted_reason));
 :
 :   // The query scales with cluster size of 1000.
 :   host_count = 1000;
 :   QuerySchedule* query_schedule1000 =
 :   MakeQuerySchedule(QUEUE_D, config_d, host_count, 30L * 
MEGABYTE);
> nit: would it make sense if in the first case host_count=2 the query cannot
That would be a nice test but the memory needed does simply scale with the 
number of hosts.


http://gerrit.cloudera.org:8080/#/c/13307/9/be/src/scheduling/admission-controller.h
File be/src/scheduling/admission-controller.h:

http://gerrit.cloudera.org:8080/#/c/13307/9/be/src/scheduling/admission-controller.h@639
PS9, Line 639:   const TPoolConfig& pool_cfg, int64_t cluster_size, 
std::string* reason);
> I think the method name should reflect that the config now depends on the c
Done


http://gerrit.cloudera.org:8080/#/c/13307/3/be/src/scheduling/admission-controller.cc
File be/src/scheduling/admission-controller.cc:

http://gerrit.cloudera.org:8080/#/c/13307/3/be/src/scheduling/admission-controller.cc@1403
PS3, Line 1403: string, pair pool_config.max_memory_multiple represents the max memory per node allocate
In HasAvailableMemResources() we effectively check the query against both
case 1: the max pool memory
case 2: the per host admit_mem_limit (from 
QuerySchedule.per_backend_mem_to_admit).
So I'm not sure we will overadmit.
I could extend IsPoolConfigValid to check for this case at the cost of another 
loop through the backend exec params. Do you think that would be useful?
It is slightly confusing as max_memory_multiple is used to calculate an 
aggregate memory size, whereas admit_mem_limit could in theory be different on 
each host, e.g. perhaps a coordinator might have more memory.


http://gerrit.cloudera.org:8080/#/c/13307/8/be/src/scheduling/admission-controller.cc
File be/src/scheduling/admission-controller.cc:

http://gerrit.cloudera.org:8080/#/c/13307/8/be/src/scheduling/admission-controller.cc@402
PS8, Line 402: Resources(const Que
> I'm in favor of naming this one cluster_size. That should imply that it's t
Done


http://gerrit.cloudera.org:8080/#/c/13307/8/be/src/scheduling/admission-controller.cc@484
PS8, Line 484: stats->local_stats().num_queued, 
GetStalenessDetailLocked(" "));
 : return false;
 :   } else if (max_requests >= 0 && stats->agg_num_running() >= 
max_requests) {
 : *not_admitted_reason = Substitute(QUEUED_NUM_RUNNING, 
stats->agg_num_running(),
 : max_requests, GetMaxRequestsForPoolDescription(pool_cfg, 
cluster_size),
 : GetStalenessDetailLocked(" "));
> nit: should we add info similar to GetMaxQueuedForPoolDescription here for
Done for QUEUED_NUM_RUNNING, but QUEUED_QUEUE_NOT_EMPTY seems OK as is.


http://gerrit.cloudera.org:8080/#/c/13307/8/be/src/scheduling/admission-controller.cc@1129
PS8, Line 1129:   pool_cfg.min_query_mem_limit, max_mem);
> We could replace this call with its return value, given that the pool_cfg u
Done


http://gerrit.cloudera.org:8080/#/c/13307/8/be/src/scheduling/admission-controller.cc@1422
PS8, Line 1422:
  :   // If we exclude the quiescing executors, and a query at the 
head of the queue is
  :   // scheduled to run on those, then the scaled down limits of 
the pool would prevent it
  :   // from being admitted, and hold up the rest of the queue
> nit: how about=> If we exclude the

[Impala-ASF-CR] IMPALA-7154 : Error making 'dropDatabase' RPC to Hive Metastore. NoSuchObjectException thrown

2019-06-12 Thread Lars Volker (Code Review)

Lars Volker has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13586 )

Change subject: IMPALA-7154 : Error making 'dropDatabase' RPC to Hive 
Metastore. NoSuchObjectException thrown
..


Patch Set 2:

(3 comments)

Thanks for looking into this.

http://gerrit.cloudera.org:8080/#/c/13586/2//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/13586/2//COMMIT_MSG@7
PS2, Line 7: IMPALA-7154 : Error making 'dropDatabase' RPC to Hive Metastore. 
NoSuchObjectException thrown
The first line of the commit message should contain a brief description of what 
the change does.

Nit: Remove space before ":", see other commit messages.


http://gerrit.cloudera.org:8080/#/c/13586/2//COMMIT_MSG@23
PS2, Line 23: 5 min
Stale comment?


http://gerrit.cloudera.org:8080/#/c/13586/2/fe/src/test/resources/hive-site.xml.py
File fe/src/test/resources/hive-site.xml.py:

http://gerrit.cloudera.org:8080/#/c/13586/2/fe/src/test/resources/hive-site.xml.py@36
PS2, Line 36: retry.delay
Does this enable retries? From the commit message I understood that we 
shouldn't retry but instead wait longer.



--
To view, visit http://gerrit.cloudera.org:8080/13586
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I253fc00618b28d8d6e626e1b6bf9c5c8290006d7
Gerrit-Change-Number: 13586
Gerrit-PatchSet: 2
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Wed, 12 Jun 2019 18:00:56 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7154 : Error making 'dropDatabase' RPC to Hive Metastore. NoSuchObjectException thrown

2019-06-12 Thread Vihang Karajgaonkar (Code Review)

Hello Lars Volker, Fredy Wijaya, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/13586

to look at the new patch set (#2).

Change subject: IMPALA-7154 : Error making 'dropDatabase' RPC to Hive 
Metastore. NoSuchObjectException thrown
..

IMPALA-7154 : Error making 'dropDatabase' RPC to Hive Metastore. 
NoSuchObjectException thrown

Some of the e2e tests create tables and databases on S3. HMS operations
like dropDatabaseCascade can take significant time (3-5 min) in such
tests. If the HMS Client times out, it retries for a configurable number
of times. Unfortunately, HMS client is not smart enough to understand
which operations should be retried. Eg, a drop database operation which
is timed out on the client side, should not be retried since the server
thread may have succeeded before the next retry. This would cause the
client to receive a NoSuchObjectException which would look like a flaky
test failure on the Impala side.

This patch increases the metastore client.socket.timeout for
the tests so that such flaky test failures are reduced.

Note:
1. The timeout value of 5 min was chosen based on looking at logs for
one such failure where the HMS API took about 3.5 min. Once this patch
is deployed we should continue to see if such failures reoccur and
readjust the client timeout value accordingly.

Change-Id: I253fc00618b28d8d6e626e1b6bf9c5c8290006d7
---
M fe/src/test/resources/hive-site.xml.py
1 file changed, 5 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/13586/2
--
To view, visit http://gerrit.cloudera.org:8080/13586
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I253fc00618b28d8d6e626e1b6bf9c5c8290006d7
Gerrit-Change-Number: 13586
Gerrit-PatchSet: 2
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Vihang Karajgaonkar

[Impala-ASF-CR] IMPALA-7154 : Error making 'dropDatabase' RPC to Hive Metastore. NoSuchObjectException thrown

2019-06-12 Thread Vihang Karajgaonkar (Code Review)

Vihang Karajgaonkar has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13586 )

Change subject: IMPALA-7154 : Error making 'dropDatabase' RPC to Hive 
Metastore. NoSuchObjectException thrown
..


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/13586/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/13586/1//COMMIT_MSG@14
PS1, Line 14: timedout
> typo: timed out
Done



--
To view, visit http://gerrit.cloudera.org:8080/13586
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I253fc00618b28d8d6e626e1b6bf9c5c8290006d7
Gerrit-Change-Number: 13586
Gerrit-PatchSet: 1
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Wed, 12 Jun 2019 17:48:57 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8650: Docker build should not depend on test config

2019-06-12 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13597 )

Change subject: IMPALA-8650: Docker build should not depend on test config
..


Patch Set 2:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4456/ 
DRY_RUN=false


--
To view, visit http://gerrit.cloudera.org:8080/13597
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa70864f5d047d1ff5f21e69d8f6358306424c0b
Gerrit-Change-Number: 13597
Gerrit-PatchSet: 2
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Wed, 12 Jun 2019 17:44:57 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8650: Docker build should not depend on test config

Tim Armstrong has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/13597


Change subject: IMPALA-8650: Docker build should not depend on test config
..

IMPALA-8650: Docker build should not depend on test config

Change-Id: Iaa70864f5d047d1ff5f21e69d8f6358306424c0b
---
M bin/create-test-configuration.sh
M docker/CMakeLists.txt
M fe/CMakeLists.txt
3 files changed, 1 insertion(+), 17 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/13597/1
--
To view, visit http://gerrit.cloudera.org:8080/13597
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Iaa70864f5d047d1ff5f21e69d8f6358306424c0b
Gerrit-Change-Number: 13597
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong

[Impala-ASF-CR] IMPALA-8629: (part 2) Adjust new KuduStorageHandler package

2019-06-12 Thread Hao Hao (Code Review)

Hao Hao has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13541 )

Change subject: IMPALA-8629: (part 2) Adjust new KuduStorageHandler package
..


Patch Set 8: Code-Review+1


--
To view, visit http://gerrit.cloudera.org:8080/13541
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4d0c505643247498383472704a37d27c9e1ce473
Gerrit-Change-Number: 13541
Gerrit-PatchSet: 8
Gerrit-Owner: Grant Henke 
Gerrit-Reviewer: Grant Henke 
Gerrit-Reviewer: Hao Hao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Thomas Marshall 
Gerrit-Comment-Date: Wed, 12 Jun 2019 17:34:37 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8629: (part 2) Adjust new KuduStorageHandler package

2019-06-12 Thread Thomas Marshall (Code Review)

Thomas Marshall has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13541 )

Change subject: IMPALA-8629: (part 2) Adjust new KuduStorageHandler package
..


Patch Set 8: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/13541
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4d0c505643247498383472704a37d27c9e1ce473
Gerrit-Change-Number: 13541
Gerrit-PatchSet: 8
Gerrit-Owner: Grant Henke 
Gerrit-Reviewer: Grant Henke 
Gerrit-Reviewer: Hao Hao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Thomas Marshall 
Gerrit-Comment-Date: Wed, 12 Jun 2019 17:32:31 +
Gerrit-HasComments: No

[Impala-ASF-CR] WIP: IMPALA-8636: Implement INSERT for insert-only ACID tables

2019-06-12 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13559 )

Change subject: WIP: IMPALA-8636: Implement INSERT for insert-only ACID tables
..


Patch Set 3:

Build Failed

https://jenkins.impala.io/job/gerrit-code-review-checks/3578/ : Initial code 
review checks failed. See linked job for details on the failure.


--
To view, visit http://gerrit.cloudera.org:8080/13559
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id6c36fa6902676f06b4e38730f737becfc7c06ad
Gerrit-Change-Number: 13559
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Wed, 12 Jun 2019 17:15:45 +
Gerrit-HasComments: No

[Impala-ASF-CR] WIP: IMPALA-8636: Implement INSERT for insert-only ACID tables

Hello Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/13559

to look at the new patch set (#3).

Change subject: WIP: IMPALA-8636: Implement INSERT for insert-only ACID tables
..

WIP: IMPALA-8636: Implement INSERT for insert-only ACID tables

Initial prototype of INSERT into ACID tables.

Change-Id: Id6c36fa6902676f06b4e38730f737becfc7c06ad
---
M be/src/exec/hdfs-table-sink.cc
M be/src/exec/hdfs-table-sink.h
M be/src/runtime/coordinator.cc
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/util/jni-util.h
M common/thrift/DataSinks.thrift
M common/thrift/Frontend.thrift
M common/thrift/ImpalaInternalService.thrift
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
A fe/src/main/java/org/apache/impala/common/TransactionException.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/planner/TableSink.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
18 files changed, 232 insertions(+), 19 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/13559/3
--
To view, visit http://gerrit.cloudera.org:8080/13559
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id6c36fa6902676f06b4e38730f737becfc7c06ad
Gerrit-Change-Number: 13559
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy 
Gerrit-Reviewer: Impala Public Jenkins

[Impala-ASF-CR] IMPALA-8588: Fix revoke grant option with Ranger

2019-06-12 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13450 )

Change subject: IMPALA-8588: Fix revoke grant option with Ranger
..


Patch Set 4: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/13450
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iddfccb442c3be3c266dbc2d8ae85c5674c534d7c
Gerrit-Change-Number: 13450
Gerrit-PatchSet: 4
Gerrit-Owner: Austin Nobis 
Gerrit-Reviewer: Austin Nobis 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Wed, 12 Jun 2019 16:07:04 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8588: Fix revoke grant option with Ranger

2019-06-12 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13450 )

Change subject: IMPALA-8588: Fix revoke grant option with Ranger
..


Patch Set 4:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4455/ 
DRY_RUN=false


--
To view, visit http://gerrit.cloudera.org:8080/13450
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iddfccb442c3be3c266dbc2d8ae85c5674c534d7c
Gerrit-Change-Number: 13450
Gerrit-PatchSet: 4
Gerrit-Owner: Austin Nobis 
Gerrit-Reviewer: Austin Nobis 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Wed, 12 Jun 2019 16:07:05 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8593: Prohibit write operations for bucketed tables

2019-06-12 Thread Csaba Ringhofer (Code Review)

Csaba Ringhofer has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13558 )

Change subject: IMPALA-8593: Prohibit write operations for bucketed tables
..


Patch Set 2:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/13558/2/testdata/datasets/functional/functional_schema_template.sql
File testdata/datasets/functional/functional_schema_template.sql:

http://gerrit.cloudera.org:8080/#/c/13558/2/testdata/datasets/functional/functional_schema_template.sql@2172
PS2, Line 2172: CLUSTERED BY (col1) INTO 5 BUCKETS
Without STORED AS {file_format} all tables will be text files.


http://gerrit.cloudera.org:8080/#/c/13558/2/testdata/datasets/functional/functional_schema_template.sql@2173
PS2, Line 2173: LOCATION '/test-warehouse/{table_name}';
Adding {db_name}{db_suffix} would ensure that tables in different databases use 
different locations. This probably doesn't matter if the table is empty, but it 
is still weird if e.g. a Parquet and a text table use the same directory.


http://gerrit.cloudera.org:8080/#/c/13558/2/testdata/datasets/functional/functional_schema_template.sql@2182
PS2, Line 2182: CLUSTERED BY (col1) INTO 5 BUCKETS;
Same as line 2172.



--
To view, visit http://gerrit.cloudera.org:8080/13558
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia08d01168660830b6e0d08b55a95eac129889cec
Gerrit-Change-Number: 13558
Gerrit-PatchSet: 2
Gerrit-Owner: Yongzhi Chen 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Sudhanshu Arora 
Gerrit-Reviewer: Todd Lipcon 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Reviewer: Yongzhi Chen 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Wed, 12 Jun 2019 15:40:21 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8649: Fix confusing SHOW GRANT error messages

2019-06-12 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13587 )

Change subject: IMPALA-8649: Fix confusing SHOW GRANT error messages
..


Patch Set 4:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4454/ 
DRY_RUN=false


--
To view, visit http://gerrit.cloudera.org:8080/13587
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibb88bdc19cd1223902b44e3634f756d086332266
Gerrit-Change-Number: 13587
Gerrit-PatchSet: 4
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: Austin Nobis 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Todd Lipcon 
Gerrit-Comment-Date: Wed, 12 Jun 2019 15:34:51 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8649: Fix confusing SHOW GRANT error messages

2019-06-12 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13587 )

Change subject: IMPALA-8649: Fix confusing SHOW GRANT error messages
..


Patch Set 4: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/13587
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibb88bdc19cd1223902b44e3634f756d086332266
Gerrit-Change-Number: 13587
Gerrit-PatchSet: 4
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: Austin Nobis 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Todd Lipcon 
Gerrit-Comment-Date: Wed, 12 Jun 2019 15:34:50 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8629: (part 2) Adjust new KuduStorageHandler package

2019-06-12 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13541 )

Change subject: IMPALA-8629: (part 2) Adjust new KuduStorageHandler package
..


Patch Set 8:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4453/ 
DRY_RUN=true


--
To view, visit http://gerrit.cloudera.org:8080/13541
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4d0c505643247498383472704a37d27c9e1ce473
Gerrit-Change-Number: 13541
Gerrit-PatchSet: 8
Gerrit-Owner: Grant Henke 
Gerrit-Reviewer: Grant Henke 
Gerrit-Reviewer: Hao Hao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Thomas Marshall 
Gerrit-Comment-Date: Wed, 12 Jun 2019 14:09:22 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8629: (part 2) Adjust new KuduStorageHandler package

2019-06-12 Thread Grant Henke (Code Review)

Hello Thomas Marshall, Hao Hao, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/13541

to look at the new patch set (#8).

Change subject: IMPALA-8629: (part 2) Adjust new KuduStorageHandler package
..

IMPALA-8629: (part 2) Adjust new KuduStorageHandler package

This patch changes the new KuduStorageHandler
package from “org.apache.kudu.hive” to
“org.apache.hadoop.hive.kudu”.

This is done to ensure the stand-in storage handler
can be a real storage handler when a Hive integration
is added in the future. The “org.apache.hadoop.hive”
package is the standard package all Hive storage
handlers lives under.

Additionally this patch updates the stand-in InputFormat,
OutputFormat, and SerDe entries for Kudu. This allows a
future Hive integration to read HMS tables/entries created
by Impala.

Change-Id: I4d0c505643247498383472704a37d27c9e1ce473
---
M bin/impala-config.sh
M fe/src/main/java/org/apache/impala/catalog/HdfsFileFormat.java
M fe/src/main/java/org/apache/impala/catalog/KuduTable.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java
M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java
M tests/custom_cluster/test_kudu.py
M tests/query_test/test_kudu.py
7 files changed, 26 insertions(+), 34 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/41/13541/8
--
To view, visit http://gerrit.cloudera.org:8080/13541
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4d0c505643247498383472704a37d27c9e1ce473
Gerrit-Change-Number: 13541
Gerrit-PatchSet: 8
Gerrit-Owner: Grant Henke 
Gerrit-Reviewer: Grant Henke 
Gerrit-Reviewer: Hao Hao 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Thomas Marshall

[Impala-ASF-CR] IMPALA-8436: Prohibit write/alter operations on materialized view

Zoltan Borok-Nagy has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/13503 )

Change subject: IMPALA-8436: Prohibit write/alter operations on materialized 
view
..

IMPALA-8436: Prohibit write/alter operations on materialized view

Instead of creating an in memory instance of View, we were
creating instance of HdfsTable. Modified the code to create instance of
View for materialized view.

Testing Done:
- Added tests in AnalyzerTest.

Change-Id: Idcd619303e19b5a2551876a63d67569c76bd22f0
Reviewed-on: http://gerrit.cloudera.org:8080/13503
Tested-by: Impala Public Jenkins 
Reviewed-by: Zoltan Borok-Nagy 
---
M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java
M testdata/datasets/functional/functional_schema_template.sql
4 files changed, 56 insertions(+), 1 deletion(-)

Approvals:
  Impala Public Jenkins: Verified
  Zoltan Borok-Nagy: Looks good to me, approved

--
To view, visit http://gerrit.cloudera.org:8080/13503
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Idcd619303e19b5a2551876a63d67569c76bd22f0
Gerrit-Change-Number: 13503
Gerrit-PatchSet: 6
Gerrit-Owner: Sudhanshu Arora 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Sudhanshu Arora 
Gerrit-Reviewer: Todd Lipcon 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Reviewer: Yongzhi Chen 
Gerrit-Reviewer: Zoltan Borok-Nagy

[Impala-ASF-CR] IMPALA-8436: Prohibit write/alter operations on materialized view

Zoltan Borok-Nagy has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/13503 )

Change subject: IMPALA-8436: Prohibit write/alter operations on materialized 
view
..


Patch Set 5: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/13503
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idcd619303e19b5a2551876a63d67569c76bd22f0
Gerrit-Change-Number: 13503
Gerrit-PatchSet: 5
Gerrit-Owner: Sudhanshu Arora 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Sudhanshu Arora 
Gerrit-Reviewer: Todd Lipcon 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Reviewer: Yongzhi Chen 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Wed, 12 Jun 2019 13:21:46 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8593: Prohibit write operations for bucketed tables