[Impala-ASF-CR] This change bumps the CDH BUILD VERSION to a version that includes the change to have full-thrift object for DropDatabaseMessage in Metastore notifications. This change is needed for t
Bharath Krishna has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12965 Change subject: This change bumps the CDH_BUILD_VERSION to a version that includes the change to have full-thrift object for DropDatabaseMessage in Metastore notifications. This change is needed for the upcoming patch for IMPALA-8338. .. This change bumps the CDH_BUILD_VERSION to a version that includes the change to have full-thrift object for DropDatabaseMessage in Metastore notifications. This change is needed for the upcoming patch for IMPALA-8338. Change-Id: Id38bae921c4d93421c6c72cdccef6a4783e2588e --- M bin/impala-config.sh 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/12965/1 -- To view, visit http://gerrit.cloudera.org:8080/12965 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Id38bae921c4d93421c6c72cdccef6a4783e2588e Gerrit-Change-Number: 12965 Gerrit-PatchSet: 1 Gerrit-Owner: Bharath Krishna
[Impala-ASF-CR] Add impala group.json to .gitignore
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12964 ) Change subject: Add impala_group.json to .gitignore .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12964 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I20ab5080fe931ac37af2403d7b85adb78916660a Gerrit-Change-Number: 12964 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 09 Apr 2019 05:23:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8336: fix flaky ORC memory limit test
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12961 ) Change subject: IMPALA-8336: fix flaky ORC memory limit test .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3999/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12961 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddcc34183ddcc7c48489739269881bffb1dc85e7 Gerrit-Change-Number: 12961 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 09 Apr 2019 05:22:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8385: Refactor Sentry admin user check
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12963 ) Change subject: IMPALA-8385: Refactor Sentry admin user check .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2697/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12963 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I911228b09af7eed5d5dc002b20591ef64dc625d3 Gerrit-Change-Number: 12963 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 09 Apr 2019 05:42:43 + Gerrit-HasComments: No
[Impala-ASF-CR] Add impala group.json to .gitignore
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12964 ) Change subject: Add impala_group.json to .gitignore .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4001/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12964 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I20ab5080fe931ac37af2403d7b85adb78916660a Gerrit-Change-Number: 12964 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 09 Apr 2019 05:30:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8380: Bump Postgres JDBC driver version to 42.2.5
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/12894 ) Change subject: IMPALA-8380: Bump Postgres JDBC driver version to 42.2.5 .. Patch Set 4: Laszlo, do you have time for another review here? -- To view, visit http://gerrit.cloudera.org:8080/12894 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ica5423c18a9f8346dda7dae617b1764638b57b6c Gerrit-Change-Number: 12894 Gerrit-PatchSet: 4 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Tue, 09 Apr 2019 05:35:28 + Gerrit-HasComments: No
[Impala-ASF-CR] Add impala group.json to .gitignore
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12964 ) Change subject: Add impala_group.json to .gitignore .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12964 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I20ab5080fe931ac37af2403d7b85adb78916660a Gerrit-Change-Number: 12964 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 09 Apr 2019 05:30:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5051: Add INT64 timestamp write support in Parquet
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12247 ) Change subject: IMPALA-5051: Add INT64 timestamp write support in Parquet .. Patch Set 18: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12247 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib41ad532ec902ed5a9a1528513726eac1c11441f Gerrit-Change-Number: 12247 Gerrit-PatchSet: 18 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: Zoltan Ivanfi Gerrit-Comment-Date: Tue, 09 Apr 2019 05:27:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8395: Parse older formats of /proc/net/dev correctly
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12954 ) Change subject: IMPALA-8395: Parse older formats of /proc/net/dev correctly .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12954 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic804955d8e4269e787037a6dc68bef2d70382426 Gerrit-Change-Number: 12954 Gerrit-PatchSet: 5 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Tue, 09 Apr 2019 05:24:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8395: Parse older formats of /proc/net/dev correctly
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12954 ) Change subject: IMPALA-8395: Parse older formats of /proc/net/dev correctly .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4000/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12954 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic804955d8e4269e787037a6dc68bef2d70382426 Gerrit-Change-Number: 12954 Gerrit-PatchSet: 5 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Tue, 09 Apr 2019 05:24:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8336: fix flaky ORC memory limit test
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12961 ) Change subject: IMPALA-8336: fix flaky ORC memory limit test .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12961 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddcc34183ddcc7c48489739269881bffb1dc85e7 Gerrit-Change-Number: 12961 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 09 Apr 2019 05:22:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8336: fix flaky ORC memory limit test
Hello Lars Volker, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12961 to look at the new patch set (#2). Change subject: IMPALA-8336: fix flaky ORC memory limit test .. IMPALA-8336: fix flaky ORC memory limit test Reduces the mem_limit for the ORC version of the test, which has proven to be flaky. Testing: Looped the test for a while locally without any failures. I was unable to reproduce the failure seen on CentOS6 jenkins locally, so we'll just try this tweak and see if it improves this. If not I can look deeper into it. Change-Id: Iddcc34183ddcc7c48489739269881bffb1dc85e7 --- M testdata/workloads/functional-query/queries/QueryTest/nested-types-tpch-mem-limit-single-node.test M tests/query_test/test_nested_types.py 2 files changed, 5 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/61/12961/2 -- To view, visit http://gerrit.cloudera.org:8080/12961 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iddcc34183ddcc7c48489739269881bffb1dc85e7 Gerrit-Change-Number: 12961 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] IMPALA-8336: fix flaky ORC memory limit test
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12961 ) Change subject: IMPALA-8336: fix flaky ORC memory limit test .. Patch Set 2: Code-Review+2 (2 comments) Carry http://gerrit.cloudera.org:8080/#/c/12961/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/12961/1//COMMIT_MSG@7 PS1, Line 7: IMPALA > typo: IMPALA Done http://gerrit.cloudera.org:8080/#/c/12961/1//COMMIT_MSG@13 PS1, Line 13: > Nit: a character Done -- To view, visit http://gerrit.cloudera.org:8080/12961 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddcc34183ddcc7c48489739269881bffb1dc85e7 Gerrit-Change-Number: 12961 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 09 Apr 2019 05:22:43 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Add impala group.json to .gitignore
Austin Nobis has posted comments on this change. ( http://gerrit.cloudera.org:8080/12964 ) Change subject: Add impala_group.json to .gitignore .. Patch Set 1: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/12964 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I20ab5080fe931ac37af2403d7b85adb78916660a Gerrit-Change-Number: 12964 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 09 Apr 2019 05:13:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8385: Refactor Sentry admin user check
Fredy Wijaya has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/12963 ) Change subject: IMPALA-8385: Refactor Sentry admin user check .. IMPALA-8385: Refactor Sentry admin user check This patch moves the hardcoded Sentry admin user check from the generic "show roles" and "show grant" catalog operations to Sentry authorization plugin, i.e. SentryImpaladAuthorizationManager. This patch also removes isAdmin() from AuthorizationManager interface. Testing: - Added a new authorization E2E test to test for Sentry admin check - Ran all FE tests - Ran all authorization E2E tests Change-Id: I911228b09af7eed5d5dc002b20591ef64dc625d3 --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog.cc M be/src/catalog/catalog.h M be/src/exec/catalog-op-executor.cc M be/src/exec/catalog-op-executor.h M be/src/service/client-request-state.cc M be/src/service/fe-support.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/authorization/AuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/NoneAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryImpaladAuthorizationManager.java M fe/src/main/java/org/apache/impala/service/FeSupport.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java A tests/authorization/test_sentry.py 17 files changed, 197 insertions(+), 82 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/12963/5 -- To view, visit http://gerrit.cloudera.org:8080/12963 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I911228b09af7eed5d5dc002b20591ef64dc625d3 Gerrit-Change-Number: 12963 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8385: Refactor Sentry admin user check
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12963 ) Change subject: IMPALA-8385: Refactor Sentry admin user check .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2696/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12963 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I911228b09af7eed5d5dc002b20591ef64dc625d3 Gerrit-Change-Number: 12963 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 09 Apr 2019 04:03:53 + Gerrit-HasComments: No
[Impala-ASF-CR] Add impala group.json to .gitignore
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12964 ) Change subject: Add impala_group.json to .gitignore .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2695/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12964 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I20ab5080fe931ac37af2403d7b85adb78916660a Gerrit-Change-Number: 12964 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 09 Apr 2019 03:58:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8385: Refactor Sentry admin user check
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12963 Change subject: IMPALA-8385: Refactor Sentry admin user check .. IMPALA-8385: Refactor Sentry admin user check This patch moves the hardcoded Sentry admin user check from the generic "show roles" and "show grant" catalog operations to Sentry authorization plugin, i.e. SentryImpaladAuthorizationManager. This patch also removes isAdmin() from AuthorizationManager interface. Testing: - Added a new authorization E2E test to test for Sentry admin check - Ran all FE tests - Ran all authorization E2E tests Change-Id: I911228b09af7eed5d5dc002b20591ef64dc625d3 --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog.cc M be/src/catalog/catalog.h M be/src/exec/catalog-op-executor.cc M be/src/exec/catalog-op-executor.h M be/src/service/client-request-state.cc M be/src/service/fe-support.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/authorization/AuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/NoneAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryImpaladAuthorizationManager.java M fe/src/main/java/org/apache/impala/service/FeSupport.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java A tests/authorization/test_sentry.py 16 files changed, 186 insertions(+), 76 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/12963/4 -- To view, visit http://gerrit.cloudera.org:8080/12963 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I911228b09af7eed5d5dc002b20591ef64dc625d3 Gerrit-Change-Number: 12963 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] Add impala group.json to .gitignore
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12964 Change subject: Add impala_group.json to .gitignore .. Add impala_group.json to .gitignore This patch adds ranger/setup/impala_group.json into .gitignore. Change-Id: I20ab5080fe931ac37af2403d7b85adb78916660a --- M testdata/cluster/.gitignore 1 file changed, 1 insertion(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/64/12964/1 -- To view, visit http://gerrit.cloudera.org:8080/12964 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I20ab5080fe931ac37af2403d7b85adb78916660a Gerrit-Change-Number: 12964 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-8395: Parse older formats of /proc/net/dev correctly
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/12954 ) Change subject: IMPALA-8395: Parse older formats of /proc/net/dev correctly .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12954 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic804955d8e4269e787037a6dc68bef2d70382426 Gerrit-Change-Number: 12954 Gerrit-PatchSet: 4 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Tue, 09 Apr 2019 02:05:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8371: Return appropriate error code for unified backend tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12885 ) Change subject: IMPALA-8371: Return appropriate error code for unified backend tests .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12885 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia146d026d42f76d5ea12d92798f299182de03eef Gerrit-Change-Number: 12885 Gerrit-PatchSet: 6 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Tue, 09 Apr 2019 01:38:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8393: Skip ranger setup for unsupported environments
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12957 ) Change subject: IMPALA-8393: Skip ranger setup for unsupported environments .. IMPALA-8393: Skip ranger setup for unsupported environments Previously, the setup-ranger step in create-load-data.sh was hard coded with localhost as the host for Ranger. This patch makes it possible to skip the setup for Ranger by using the flag -skip_ranger. The script was also updated to set the SKIP_RANGER variable when the REMOTE_LOAD environment variable is set. Testing: - Testing was performed by calling the script with and without the setup-ranger flag set as well as calling the script with and without the REMOTE_LOAD environment variable set. Change-Id: Ie81dda992cf29792468580b182e570132d5ce0a1 Reviewed-on: http://gerrit.cloudera.org:8080/12957 Reviewed-by: Fredy Wijaya Tested-by: Impala Public Jenkins --- M testdata/bin/create-load-data.sh 1 file changed, 12 insertions(+), 1 deletion(-) Approvals: Fredy Wijaya: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/12957 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ie81dda992cf29792468580b182e570132d5ce0a1 Gerrit-Change-Number: 12957 Gerrit-PatchSet: 7 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8393: Skip ranger setup for unsupported environments
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12957 ) Change subject: IMPALA-8393: Skip ranger setup for unsupported environments .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12957 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie81dda992cf29792468580b182e570132d5ce0a1 Gerrit-Change-Number: 12957 Gerrit-PatchSet: 6 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 09 Apr 2019 01:17:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPAAL-8336: fix flaky ORC memory limit test
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/12961 ) Change subject: IMPAAL-8336: fix flaky ORC memory limit test .. Patch Set 1: Code-Review+2 (2 comments) Just some nits http://gerrit.cloudera.org:8080/#/c/12961/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/12961/1//COMMIT_MSG@7 PS1, Line 7: IMPAAL typo: IMPALA http://gerrit.cloudera.org:8080/#/c/12961/1//COMMIT_MSG@13 PS1, Line 13: a Nit: a character -- To view, visit http://gerrit.cloudera.org:8080/12961 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddcc34183ddcc7c48489739269881bffb1dc85e7 Gerrit-Change-Number: 12961 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Tue, 09 Apr 2019 01:02:31 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5051: Add INT64 timestamp write support in Parquet
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12247 ) Change subject: IMPALA-5051: Add INT64 timestamp write support in Parquet .. Patch Set 17: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2694/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12247 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib41ad532ec902ed5a9a1528513726eac1c11441f Gerrit-Change-Number: 12247 Gerrit-PatchSet: 17 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: Zoltan Ivanfi Gerrit-Comment-Date: Tue, 09 Apr 2019 00:51:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPAAL-8336: fix flaky ORC memory limit test
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12961 ) Change subject: IMPAAL-8336: fix flaky ORC memory limit test .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2692/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12961 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddcc34183ddcc7c48489739269881bffb1dc85e7 Gerrit-Change-Number: 12961 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 09 Apr 2019 00:34:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5051: Add INT64 timestamp write support in Parquet
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12247 ) Change subject: IMPALA-5051: Add INT64 timestamp write support in Parquet .. Patch Set 18: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3998/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12247 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib41ad532ec902ed5a9a1528513726eac1c11441f Gerrit-Change-Number: 12247 Gerrit-PatchSet: 18 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: Zoltan Ivanfi Gerrit-Comment-Date: Tue, 09 Apr 2019 00:28:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7995: part 2: Jenkins script to automate e2e tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12937 ) Change subject: IMPALA-7995: part 2: Jenkins script to automate e2e tests .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2693/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12937 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I67a3562904c959b51f4bde52107193c4002cb1ce Gerrit-Change-Number: 12937 Gerrit-PatchSet: 7 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Tue, 09 Apr 2019 00:38:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6718: Add support for column-level permissions on views
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/12959 )
Change subject: IMPALA-6718: Add support for column-level permissions on views
..
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/12959/3/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
File fe/src/main/java/org/apache/impala/analysis/SelectStmt.java:
http://gerrit.cloudera.org:8080/#/c/12959/3/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@545
PS3, Line 545: private void registerViewColumnPrivileges() {
: for (TableRef tableRef: fromClause_.getTableRefs()) {
: if (!(tableRef instanceof InlineViewRef)) continue;
: InlineViewRef inlineViewRef = (InlineViewRef) tableRef;
: FeView view = inlineViewRef.getView();
: boolean isCatalogView = view != null &&
!view.isLocalView();
: if (!isCatalogView) continue;
: for (Expr expr : getResultExprs()) {
: if (!(expr instanceof SlotRef)) continue;
: SlotRef slotRef = (SlotRef) expr;
: analyzer_.registerPrivReq(builder -> builder
: .allOf(Privilege.SELECT)
: .onColumn(view.getDb().getName(), view.getName(),
: slotRef.getDesc().getLabel())
: .build());
: }
: }
: }
Had a chat with Fredy offline. We were discussing if there is a more cleaner
way to do it in the InlineViewRef analysis. We could mark the view slots as
materialized and add column privileges just like how we do for table columns.
--
To view, visit http://gerrit.cloudera.org:8080/12959
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If81e683212cba22cc0fa5fc091ec3c799fa33e14
Gerrit-Change-Number: 12959
Gerrit-PatchSet: 3
Gerrit-Owner: Fredy Wijaya
Gerrit-Reviewer: Austin Nobis
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Todd Lipcon
Gerrit-Comment-Date: Tue, 09 Apr 2019 00:28:25 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5051: Add INT64 timestamp write support in Parquet
Hello Lars Volker, Zoltan Borok-Nagy, Zoltan Ivanfi, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12247 to look at the new patch set (#17). Change subject: IMPALA-5051: Add INT64 timestamp write support in Parquet .. IMPALA-5051: Add INT64 timestamp write support in Parquet Add query option "parquet_timestamp_type" that chooses the Parquet type used when writing TIMESTAMP columns. This is an experimental feature at the moment, because these types are not widely adopted in other Hadoop components yet. For this reason the query option is added as "development" level, and the default behavior is not changed. The following options can be used: INT96_NANOS (default): This is the same as the old behavior, can represent any timestamp that can be handled by Impala. INT64_MILLIS, INT64_MICROS: Can encode the whole [1400..1) range handled by Impala at the cost of reduced precision. Values are rounded towards minus infinity during writing. INT64_NANOS: Can encode a reduced range without losing nanosecond precision: [1677-09-21 00:12:43.145224192 .. 2262-04-11 23:47:16.854775807] Values outside this range are converted to NULLs without warning. The change was done completely in the backend and all TIMESTAMP columns are written using the type set in the query option. An alternative design would have been to implement some parts in the fronted by adding TIMESTAMP->BIGINT conversion functions to the query plan, which would make it easier to add the possibility of per-column setting in the future. I choose the current design because it seemed much simpler and there are no clear plans for the per-column setting. Most of the code will be still useful if we decide to go the other way in the future. All types are written without conversion to UTC (the way Impala always wrote timestamps), and this information is expressed in the new Parquet logical types by setting isAdjustedToUTC to false. The old logical type (converted_type) is not set, because old readers do not read isAdjustedToUTC, and assume that TIMESTAMP_MILLIS and TIMESTAMP_MICROS are written in UTC. These readers can still read int64 timestamp columns as INT_64. Testing: - added unit tests for new TimestampValue->int64 functions - add EE tests for checking values / min-max stats / metadata written for int64 Parquet timestamps - ran core tests Change-Id: Ib41ad532ec902ed5a9a1528513726eac1c11441f --- M be/src/exec/parquet/hdfs-parquet-table-writer.cc M be/src/exec/parquet/hdfs-parquet-table-writer.h M be/src/exec/parquet/parquet-common.cc M be/src/exec/parquet/parquet-common.h M be/src/exec/parquet/parquet-metadata-utils.cc M be/src/exec/parquet/parquet-metadata-utils.h M be/src/runtime/timestamp-test.cc M be/src/runtime/timestamp-value.h M be/src/runtime/timestamp-value.inline.h M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/debug-util.cc M be/src/util/debug-util.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M testdata/workloads/functional-query/queries/QueryTest/parquet-int64-timestamps.test M tests/query_test/test_insert_parquet.py M tests/util/get_parquet_metadata.py 18 files changed, 611 insertions(+), 117 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/12247/17 -- To view, visit http://gerrit.cloudera.org:8080/12247 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib41ad532ec902ed5a9a1528513726eac1c11441f Gerrit-Change-Number: 12247 Gerrit-PatchSet: 17 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: Zoltan Ivanfi
[Impala-ASF-CR] IMPAAL-8336: fix flaky ORC memory limit test
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12961 Change subject: IMPAAL-8336: fix flaky ORC memory limit test .. IMPAAL-8336: fix flaky ORC memory limit test Reduces the mem_limit for the ORC version of the test, which has proven to be flaky. Testing: Looped the test for a while locally without any failures.a I was unable to reproduce the failure seen on CentOS6 jenkins locally, so we'll just try this tweak and see if it improves this. If not I can look deeper into it. Change-Id: Iddcc34183ddcc7c48489739269881bffb1dc85e7 --- M testdata/workloads/functional-query/queries/QueryTest/nested-types-tpch-mem-limit-single-node.test M tests/query_test/test_nested_types.py 2 files changed, 5 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/61/12961/1 -- To view, visit http://gerrit.cloudera.org:8080/12961 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iddcc34183ddcc7c48489739269881bffb1dc85e7 Gerrit-Change-Number: 12961 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-7995: part 2: Jenkins script to automate e2e tests
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12937 Change subject: IMPALA-7995: part 2: Jenkins script to automate e2e tests .. IMPALA-7995: part 2: Jenkins script to automate e2e tests Testing: Ran on https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/ Change-Id: I67a3562904c959b51f4bde52107193c4002cb1ce --- A bin/jenkins/dockerized-impala-bootstrap-and-test.sh A bin/jenkins/dockerized-impala-run-tests.sh M docker/configure_test_network.sh 3 files changed, 131 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/37/12937/7 -- To view, visit http://gerrit.cloudera.org:8080/12937 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I67a3562904c959b51f4bde52107193c4002cb1ce Gerrit-Change-Number: 12937 Gerrit-PatchSet: 7 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-8375: Add metrics for spill disk usage
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/12956 )
Change subject: IMPALA-8375: Add metrics for spill disk usage
..
Patch Set 3:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/12956/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/12956/3//COMMIT_MSG@20
PS3, Line 20:
Can you add a brief "Testing:" section to describe the automated tests you
added and any manual tests you ran (e.g. looked at the metrics page.
I find that helpful for reviewing.
http://gerrit.cloudera.org:8080/#/c/12956/3/be/src/runtime/tmp-file-mgr.cc
File be/src/runtime/tmp-file-mgr.cc:
http://gerrit.cloudera.org:8080/#/c/12956/3/be/src/runtime/tmp-file-mgr.cc@285
PS3, Line 285: -1 * (scratch_space_bytes_used_counter_->value()));
nit: unnecessary parentheses
http://gerrit.cloudera.org:8080/#/c/12956/3/be/src/util/metrics-test.cc
File be/src/util/metrics-test.cc:
http://gerrit.cloudera.org:8080/#/c/12956/3/be/src/util/metrics-test.cc@112
PS3, Line 112: IntHWMGauge* int_hwm_gauge = metrics.AddHWMGauge("gauge", 0);
I think if you cast this to the base class IntGauge the behaviour will change
because of the non-virtual methods.
http://gerrit.cloudera.org:8080/#/c/12956/3/be/src/util/metrics.h
File be/src/util/metrics.h:
http://gerrit.cloudera.org:8080/#/c/12956/3/be/src/util/metrics.h@255
PS3, Line 255: class AtomicHighWaterMarkMetric : public IntGauge {
I'm wondering if it's better to have this implemented as two metrics - one
which is the underlying metric and other which is the high water mark. I think
if the HWM is useful then the other metric is also useful to expose. E.g. this
could be a wrapper around an AtomicMetric similar to NegatedGauge below.
http://gerrit.cloudera.org:8080/#/c/12956/3/be/src/util/metrics.h@263
PS3, Line 263: int64_t GetValue() { return hwm_value_.Load(); }
Let's mark it is an override so it's clear that it's overriding something from
the base class.
http://gerrit.cloudera.org:8080/#/c/12956/3/be/src/util/metrics.h@269
PS3, Line 269: void SetValue(const int64_t& value) {
I think we have a problem here - SetValue() is not a virtual function so we can
easily call the wrong implementation if we have a pointer to IntGauge rather
than AtomicHighWaterMarkMetric.
http://gerrit.cloudera.org:8080/#/c/12956/3/be/src/util/metrics.h@276
PS3, Line 276: int64_t Increment(int64_t delta) {
Same issue here with it being non-virtual.
I think if we implemented this as a subclass of ScalarMetric<> wrapping an
AtomicMetric, we would avoid this issue since it wouldn't be in the same
hierarchy as AtomicMetric.
--
To view, visit http://gerrit.cloudera.org:8080/12956
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia1b3dd604c7234a8d8af34d70ca731544a46d298
Gerrit-Change-Number: 12956
Gerrit-PatchSet: 3
Gerrit-Owner: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Mon, 08 Apr 2019 23:50:25 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8375: Add metrics for spill disk usage
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12956 ) Change subject: IMPALA-8375: Add metrics for spill disk usage .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2691/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12956 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia1b3dd604c7234a8d8af34d70ca731544a46d298 Gerrit-Change-Number: 12956 Gerrit-PatchSet: 3 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 08 Apr 2019 23:40:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2690/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 6 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 08 Apr 2019 23:22:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8224: [DOCS] Add the missing contents about Impala web UI pages
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/12934 ) Change subject: IMPALA-8224: [DOCS] Add the missing contents about Impala web UI pages .. Patch Set 2: Can someone do a final review and approve the changes, please? -- To view, visit http://gerrit.cloudera.org:8080/12934 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic30c26fe1d19c1a81704723cd5ebed667982d5b6 Gerrit-Change-Number: 12934 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Balazs Jeszenszky Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Mon, 08 Apr 2019 22:54:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7322: Add storage wait time to profile
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/12940 )
Change subject: IMPALA-7322: Add storage wait time to profile
..
Patch Set 3:
(14 comments)
http://gerrit.cloudera.org:8080/#/c/12940/3/common/thrift/CatalogObjects.thrift
File common/thrift/CatalogObjects.thrift:
http://gerrit.cloudera.org:8080/#/c/12940/3/common/thrift/CatalogObjects.thrift@474
PS3, Line 474: Set iff this is a table needs storage access.
I think this is vague. Better description?
http://gerrit.cloudera.org:8080/#/c/12940/3/common/thrift/CatalogObjects.thrift@476
PS3, Line 476: storage_ld_time
Try to use meaningful variable names whenever possible (for readability). say
something like filesystem_metadata_load_time or something along those lines?
http://gerrit.cloudera.org:8080/#/c/12940/3/common/thrift/CatalogObjects.thrift@476
PS3, Line 476: optional
Why is it optional?
http://gerrit.cloudera.org:8080/#/c/12940/3/fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java
File fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java:
http://gerrit.cloudera.org:8080/#/c/12940/3/fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java@229
PS3, Line 229: strgTmNano
meaningful variable names (for readability)
http://gerrit.cloudera.org:8080/#/c/12940/3/fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java@232
PS3, Line 232: ldTbl
Better variable names
http://gerrit.cloudera.org:8080/#/c/12940/3/fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java@232
PS3, Line 232: loadedTbls_
How do you distinguish between tables loaded now vs tables already loaded.
For ex: if 3 of 5 tables need loading (meaning 2 already cached), this loop
seems to sum the metric for all the 5 tables.
http://gerrit.cloudera.org:8080/#/c/12940/3/fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java@233
PS3, Line 233: ldTbl instanceof Table
This is kinda obvious, force a downcast?
http://gerrit.cloudera.org:8080/#/c/12940/3/fe/src/main/java/org/apache/impala/catalog/HBaseTable.java
File fe/src/main/java/org/apache/impala/catalog/HBaseTable.java:
http://gerrit.cloudera.org:8080/#/c/12940/3/fe/src/main/java/org/apache/impala/catalog/HBaseTable.java@105
PS3, Line 105: ctxStg
better variable names
http://gerrit.cloudera.org:8080/#/c/12940/3/fe/src/main/java/org/apache/impala/catalog/HBaseTable.java@109
PS3, Line 109: hbaseTableName_ = Util.getHBaseTableName(getMetaStoreTable());
: // Warm up the connection and verify the table exists.
: Util.getHBaseTable(hbaseTableName_).close();
: columnFamilies_ = null;
: cols = Util.loadColumns(msTable_);
: } finally {
: storageLdTime_ = ctxStg.stop();
I don't think any of this calls into HBase
http://gerrit.cloudera.org:8080/#/c/12940/3/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/12940/3/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1215
PS3, Line 1215: storageLdTm
Better variable names
http://gerrit.cloudera.org:8080/#/c/12940/3/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1237
PS3, Line 1237: final Timer.Context ctxStg =
I think the scopes for metrics are incorrect. I'd suggest to do this on top of
https://gerrit.cloudera.org/#/c/12950/ which refactors the code nicely.
http://gerrit.cloudera.org:8080/#/c/12940/3/fe/src/main/java/org/apache/impala/catalog/Table.java
File fe/src/main/java/org/apache/impala/catalog/Table.java:
http://gerrit.cloudera.org:8080/#/c/12940/3/fe/src/main/java/org/apache/impala/catalog/Table.java@115
PS3, Line 115: // Storage load time for the table
Time spent in the source systems loading the fs metadata... (or something
similar?)
http://gerrit.cloudera.org:8080/#/c/12940/3/fe/src/main/java/org/apache/impala/catalog/Table.java@116
PS3, Line 116: storageLdTime_
better name? fileSystemMdLoadTime or something along those lines?
http://gerrit.cloudera.org:8080/#/c/12940/3/fe/src/main/java/org/apache/impala/catalog/Table.java@193
PS3, Line 193: STORAGE_LOAD_DURATION_METRIC
better name
--
To view, visit http://gerrit.cloudera.org:8080/12940
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6dde7e394b7c1c396d835ef6aa0a55930c0a8660
Gerrit-Change-Number: 12940
Gerrit-PatchSet: 3
Gerrit-Owner: Yongzhi Chen
Gerrit-Reviewer: Anurag Mantripragada
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Mon, 08 Apr 2019 23:16:19 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. Patch Set 6: (19 comments) http://gerrit.cloudera.org:8080/#/c/12889/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/12889/5//COMMIT_MSG@14 PS5, Line 14: partitions. : : Known Issues: > This may not be applicable anymore Done http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java: http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@413 PS5, Line 413: ts(dbName_, tb > I think this call is not correct since this will be a no-op if the table is Thanks for this catch. Used reloadTable() instead which forces a reload every time. reloadPartition() http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@601 PS5, Line 601:* Metastore event for INSERT events. > These following two lines can be Done http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java: http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@204 PS5, Line 204: > Typo sofar Done http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@38 PS5, Line 38: import org.apache.hadoop.hive.metastore.PartitionDropOptions; > unused? Done http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3551 PS5, Line 3551: // partition key will be empty. > nit : comma after tables. Done http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3557 PS5, Line 3557: InsertForTable = new HashSet<>(); > may be a better name would be to suggest that this contains files before in Done http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3558 PS5, Line 3558: eConf = new HiveConf(this. > is this unused? Done http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3580 PS5, Line 3580: rtition.isMarkedCached()) > Is there a concern here of running into CastException? I see that FeFsParti This code path is taken by only HDFSTables as you can see on L3522. http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3673 PS5, Line 3673: // For > may be do a else if(catalog.isExternalEventProcessingEnabled()) here so tha Done http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3674 PS5, Line 3674: Preconditions.checkState(parts.size() == 1); > May be add a Preconditions.checkState(parts.size == 1); here to make sure t Done http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3676 PS5, Line 3676: BeforeInsertForTable = ((( > same as above, Do we need to handle LocalFsPartition as well? The code path is taken by HdfsTable, so we do not need to handle LocalFsPartition. http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3699 PS5, Line 3699: createInsertEve > May be you can create 2 methods, one for partitioned case and another for n Done http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3716 PS5, Line 3716: } > Add to the description that this method is a no-op if event processing is d Done http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3719 PS5, Line 3719: nsert and ca > nit, change the name to isInsertOverwrite Done http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3736 PS5, Line 3736: List newFiles = new ArrayList<>(); > I think it would be helpful to add info log here which says how many new fi Done http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3750 PS5, Line 3750: } > add a info level log here which prints how many new files were added into t Done http://gerrit.cloudera.org:8080/#/c/12889/5/fe/src/main/java/org/apache/impala/service/Catal
[Impala-ASF-CR] IMPALA-8375: Add metrics for spill disk usage
Abhishek Rawat has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12956 Change subject: IMPALA-8375: Add metrics for spill disk usage .. IMPALA-8375: Add metrics for spill disk usage Added a new metric tmp-file-mgr.scratch-space-bytes-used-high-water-mark for tracking HWM for spilled bytes. A new class AtomicHighWaterMarkMetric was added to keep track of the current value and the HWM value. The current value for spilled bytes is incremented every time a new range is allocated from a temporary file. The current value for spilled bytes is decremented when a temporary file is closed. The new metric is not updated when ranges are recycled from a file. We can add a new metric in future for keeping track of actual spilled bytes. The HWM value is updated whenever the current value is greater than the HWM value. Change-Id: Ia1b3dd604c7234a8d8af34d70ca731544a46d298 --- M be/src/runtime/tmp-file-mgr.cc M be/src/runtime/tmp-file-mgr.h M be/src/util/metrics-test.cc M be/src/util/metrics.h M common/thrift/metrics.json 5 files changed, 98 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/56/12956/3 -- To view, visit http://gerrit.cloudera.org:8080/12956 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ia1b3dd604c7234a8d8af34d70ca731544a46d298 Gerrit-Change-Number: 12956 Gerrit-PatchSet: 3 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8395: Parse older formats of /proc/net/dev correctly
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12954 ) Change subject: IMPALA-8395: Parse older formats of /proc/net/dev correctly .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2689/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12954 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic804955d8e4269e787037a6dc68bef2d70382426 Gerrit-Change-Number: 12954 Gerrit-PatchSet: 4 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Mon, 08 Apr 2019 23:05:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8390: clean up test vectors in test cancellation.py
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12960 ) Change subject: IMPALA-8390: clean up test vectors in test_cancellation.py .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2687/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12960 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9673fe82bda5314aff6a51d1961759ff286fbf6f Gerrit-Change-Number: 12960 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Mon, 08 Apr 2019 22:40:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Anurag Mantripragada has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. IMPALA-7971: Add support for insert events in event processor. This patch adds support for detecting and processing insert events triggered by impala as well as external engines (eg.Hive). Inserts from Impala will fire an insert event notification. Using this event, event-processor will refresh table/partition. Both insert into and overwrite are supported for tables/partitions. Known Issues: 1. Inserts into tables from Hive are ignored by the event processor as these inserts create an ALTER event first followed by an INSERT event. The alter will invalidate table making the refresh a no-op. Insert into partitions from hive will create an INSERT event first followed by an ALTER event. In this case, there is an unnecessary table invalidate after a refresh. 2. Existing self-events logic cannot be used for insert events since firing insert event does not allow us to modify table parameters in HMS. This means we cannot get the CatalogServiceIdentifiers in insert events. Therefore, the event-processor will also refresh the tables for which insert operation is performed through Impala. Testing: 1. Added new custom cluster tests to run different insert commands from hive and verified new data is available in Impala without invalidate metadata. 2. Added a test in MetastoreEventsProcessor for testing insert events. Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 --- M be/src/service/client-request-state.cc M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java A tests/custom_cluster/test_event_processing.py 9 files changed, 518 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/12889/6 -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 6 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-6718: Add support for column-level permissions on views
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12959 ) Change subject: IMPALA-6718: Add support for column-level permissions on views .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2688/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12959 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If81e683212cba22cc0fa5fc091ec3c799fa33e14 Gerrit-Change-Number: 12959 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 08 Apr 2019 22:43:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8395: Parse older formats of /proc/net/dev correctly
Lars Volker has posted comments on this change. (
http://gerrit.cloudera.org:8080/12954 )
Change subject: IMPALA-8395: Parse older formats of /proc/net/dev correctly
..
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/12954/2/be/src/util/system-state-info.cc
File be/src/util/system-state-info.cc:
http://gerrit.cloudera.org:8080/#/c/12954/2/be/src/util/system-state-info.cc@168
PS2, Line 168: colon_idx = sp.find_first_of(':');
> Counters will come up empty in this case because substr() will return an em
I was wrong, counters will contain the whole line. I added a check, not finding
a colon is clearly a sign of a problem.
--
To view, visit http://gerrit.cloudera.org:8080/12954
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic804955d8e4269e787037a6dc68bef2d70382426
Gerrit-Change-Number: 12954
Gerrit-PatchSet: 4
Gerrit-Owner: Lars Volker
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Michael Ho
Gerrit-Comment-Date: Mon, 08 Apr 2019 22:24:07 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8395: Parse older formats of /proc/net/dev correctly
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12954 ) Change subject: IMPALA-8395: Parse older formats of /proc/net/dev correctly .. Patch Set 4: (4 comments) http://gerrit.cloudera.org:8080/#/c/12954/4/be/src/util/system-state-info-test.cc File be/src/util/system-state-info-test.cc: http://gerrit.cloudera.org:8080/#/c/12954/4/be/src/util/system-state-info-test.cc@96 PS4, Line 96: string dev_net = R"(Inter-| Receive | Transmit line too long (98 > 90) http://gerrit.cloudera.org:8080/#/c/12954/4/be/src/util/system-state-info-test.cc@97 PS4, Line 97: face |bytespackets errs drop fifo frame compressed multicast|bytespackets errs drop fifo colls carrier compressed line too long (122 > 90) http://gerrit.cloudera.org:8080/#/c/12954/4/be/src/util/system-state-info-test.cc@98 PS4, Line 98: lo:5829 53000 0 0 0 5829 53000 0 0 0 line too long (122 > 90) http://gerrit.cloudera.org:8080/#/c/12954/4/be/src/util/system-state-info-test.cc@99 PS4, Line 99: eth0:285025090 212208000 0 0 0 9137793 84770000 0 0 0)"; line too long (126 > 90) -- To view, visit http://gerrit.cloudera.org:8080/12954 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic804955d8e4269e787037a6dc68bef2d70382426 Gerrit-Change-Number: 12954 Gerrit-PatchSet: 4 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Mon, 08 Apr 2019 22:23:58 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8395: Parse older formats of /proc/net/dev correctly
Hello Bharath Vissapragada, Michael Ho, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12954 to look at the new patch set (#4). Change subject: IMPALA-8395: Parse older formats of /proc/net/dev correctly .. IMPALA-8395: Parse older formats of /proc/net/dev correctly Older kernel versions don't have a space between the interface name and the first counter value in /proc/net/dev. This change reworks the parsing logic to support such older formats and adds a unit test for it. Change-Id: Ic804955d8e4269e787037a6dc68bef2d70382426 --- M be/src/util/system-state-info-test.cc M be/src/util/system-state-info.cc M be/src/util/system-state-info.h 3 files changed, 46 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/12954/4 -- To view, visit http://gerrit.cloudera.org:8080/12954 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic804955d8e4269e787037a6dc68bef2d70382426 Gerrit-Change-Number: 12954 Gerrit-PatchSet: 4 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho
[Impala-ASF-CR] IMPALA-7995: part 1: fixes for e2e dockerised impala tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12639 ) Change subject: IMPALA-7995: part 1: fixes for e2e dockerised impala tests .. Patch Set 15: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2686/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12639 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iee86cbd2c4631a014af1e8cef8e1cd523a812755 Gerrit-Change-Number: 12639 Gerrit-PatchSet: 15 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 08 Apr 2019 22:12:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6718: Add support for column-level permissions on views
Fredy Wijaya has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/12959 ) Change subject: IMPALA-6718: Add support for column-level permissions on views .. IMPALA-6718: Add support for column-level permissions on views This patch adds support for column-level permissions on views. The following statements are now supported. GRANT select(col) ON db.my_view TO ROLE my_role -- Sentry only REVOKE select(col) ON db.my_view FROM ROLE my_role -- Sentry only GRANT select(col) ON db.my_view TO USER my_user -- Ranger only REVOKE select(col) ON db.my_view FROM USER my_user -- Ranger only GRANT select(col) ON db.my_view TO GROUP my_group -- Ranger only REVOKE select(col) ON db.my_view FROM GROUP my_group -- Ranger only Testing: - Updated AuthorizationStmtTest to with new test cases - Ran all FE tests - Ran all E2E authorization tests Change-Id: If81e683212cba22cc0fa5fc091ec3c799fa33e14 --- M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeAuthStmtsTest.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java 4 files changed, 33 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/12959/3 -- To view, visit http://gerrit.cloudera.org:8080/12959 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If81e683212cba22cc0fa5fc091ec3c799fa33e14 Gerrit-Change-Number: 12959 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8346: Specify hadoop user explicitly during data load
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/12958 ) Change subject: IMPALA-8346: Specify hadoop user explicitly during data load .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/12958/1/testdata/bin/create-tpcds-testcase-files.sh File testdata/bin/create-tpcds-testcase-files.sh: http://gerrit.cloudera.org:8080/#/c/12958/1/testdata/bin/create-tpcds-testcase-files.sh@24 PS1, Line 24: # Directory ownership and permissions need to be set differently, I'm not super convinced this is the right solution. How about fixing the remote data load to run consistently as a single user (impala)? Am I missing something? -- To view, visit http://gerrit.cloudera.org:8080/12958 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe11d7cb50d9e2657152c94f8defcbc69ca7e1ba Gerrit-Change-Number: 12958 Gerrit-PatchSet: 1 Gerrit-Owner: David Knupp Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 08 Apr 2019 22:04:53 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8390: clean up test vectors in test cancellation.py
Thomas Marshall has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12960 Change subject: IMPALA-8390: clean up test vectors in test_cancellation.py .. IMPALA-8390: clean up test vectors in test_cancellation.py Due to changes to TestCancellation made in IMPALA-7205 that were not reflected in TestCancellationSerial and TestCancellationFullSort, test_cancel_insert has not been running at all and test_cancel_sort has been running with unintended parameters. This patch re-enables test_cancel_insert, while including a number of constraints on its parameters to keep test execution time reasonable. It also fixes an incorrect constraint on test_cancel_sort. The patch also makes some related improvements: - Removes an xfail on test_cancel_insert related to a bug that is fixed now. - When ImpalaTestVector.get_value() is called with a value name that does not actually exist in the vector, the result is a StopIteration exception. Due to python's questionable habit of using exceptions for flow control, StopIteration is frequently treated not as an error but as the normal end of iteration, which can result in unexpected behavior, eg. when pytest_generate_tests raises a StopIteration pytest just silently ignores it and drops the test case. This patch modifies get_value() to instead raise a ValueError in this situation. - When a test has no vectors generated for it, the name of the test is now included in the logged warning. Change-Id: I9673fe82bda5314aff6a51d1961759ff286fbf6f --- M tests/common/test_vector.py M tests/conftest.py M tests/query_test/test_cancellation.py 3 files changed, 27 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/60/12960/1 -- To view, visit http://gerrit.cloudera.org:8080/12960 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I9673fe82bda5314aff6a51d1961759ff286fbf6f Gerrit-Change-Number: 12960 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Marshall
[Impala-ASF-CR] IMPALA-7368: Add initial support for DATE type
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12481 ) Change subject: IMPALA-7368: Add initial support for DATE type .. Patch Set 19: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/12481 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iea8155ef09557e0afa2f8b2d0b2dc9d0896dc30f Gerrit-Change-Number: 12481 Gerrit-PatchSet: 19 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 08 Apr 2019 22:00:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8390: clean up test vectors in test cancellation.py
Thomas Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/12960 ) Change subject: IMPALA-8390: clean up test vectors in test_cancellation.py .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/12960/1/tests/common/test_vector.py File tests/common/test_vector.py: http://gerrit.cloudera.org:8080/#/c/12960/1/tests/common/test_vector.py@76 PS1, Line 76: > flake8: E502 the backslash is redundant between brackets Done http://gerrit.cloudera.org:8080/#/c/12960/1/tests/common/test_vector.py@79 PS1, Line 79: ( > flake8: W602 deprecated form of raising exception Done -- To view, visit http://gerrit.cloudera.org:8080/12960 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9673fe82bda5314aff6a51d1961759ff286fbf6f Gerrit-Change-Number: 12960 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Mon, 08 Apr 2019 21:59:18 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8390: clean up test vectors in test cancellation.py
Hello David Knupp, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12960 to look at the new patch set (#2). Change subject: IMPALA-8390: clean up test vectors in test_cancellation.py .. IMPALA-8390: clean up test vectors in test_cancellation.py Due to changes to TestCancellation made in IMPALA-7205 that were not reflected in TestCancellationSerial and TestCancellationFullSort, test_cancel_insert has not been running at all and test_cancel_sort has been running with unintended parameters. This patch re-enables test_cancel_insert, while including a number of constraints on its parameters to keep test execution time reasonable. It also fixes an incorrect constraint on test_cancel_sort. The patch also makes some related improvements: - Removes an xfail on test_cancel_insert related to a bug that is fixed now. - When ImpalaTestVector.get_value() is called with a value name that does not actually exist in the vector, the result is a StopIteration exception. Due to python's questionable habit of using exceptions for flow control, StopIteration is frequently treated not as an error but as the normal end of iteration, which can result in unexpected behavior, eg. when pytest_generate_tests raises a StopIteration pytest just silently ignores it and drops the test case. This patch modifies get_value() to instead raise a ValueError in this situation. - When a test has no vectors generated for it, the name of the test is now included in the logged warning. Change-Id: I9673fe82bda5314aff6a51d1961759ff286fbf6f --- M tests/common/test_vector.py M tests/conftest.py M tests/query_test/test_cancellation.py 3 files changed, 27 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/60/12960/2 -- To view, visit http://gerrit.cloudera.org:8080/12960 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9673fe82bda5314aff6a51d1961759ff286fbf6f Gerrit-Change-Number: 12960 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8390: clean up test vectors in test cancellation.py
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12960 ) Change subject: IMPALA-8390: clean up test vectors in test_cancellation.py .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2685/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12960 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9673fe82bda5314aff6a51d1961759ff286fbf6f Gerrit-Change-Number: 12960 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 08 Apr 2019 21:54:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5051: Add INT64 timestamp write support in Parquet
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12247 ) Change subject: IMPALA-5051: Add INT64 timestamp write support in Parquet .. Patch Set 16: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/3994/ -- To view, visit http://gerrit.cloudera.org:8080/12247 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib41ad532ec902ed5a9a1528513726eac1c11441f Gerrit-Change-Number: 12247 Gerrit-PatchSet: 16 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: Zoltan Ivanfi Gerrit-Comment-Date: Mon, 08 Apr 2019 21:53:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7995: part 1: fixes for e2e dockerised impala tests
Tim Armstrong has uploaded a new patch set (#15). ( http://gerrit.cloudera.org:8080/12639 ) Change subject: IMPALA-7995: part 1: fixes for e2e dockerised impala tests .. IMPALA-7995: part 1: fixes for e2e dockerised impala tests This fixes all core e2e tests running on my local dockerised minicluster build. I do not yet have a CI job or script running but I wanted to get feedback on these changes sooner. The second part of the change will include the CI script and any follow-on fixes required for the exhaustive tests. The following fixes were required: * Detect docker_network from TEST_START_CLUSTER_ARGS * get_webserver_port() does not depend on the caller passing in the default webserver port. It failed previously because it relied on start-impala-cluster.py setting -webserver_port for *all* processes. * Add SkipIf markers for tests that don't make sense or are non-trivial to fix for containerised Impala. * Support loading Impala-lzo plugin from host for tests that depend on it. * Fix some tests that had 'localhost' hardcoded - instead it should be $INTERNAL_LISTEN_HOST, which defaults to localhost. * Fix bug with sorting impala daemons by backend port, which is the same for all dockerised impalads. Testing: I ran tests locally as follows after having set up a docker network and starting other services: ./buildall.sh -noclean -notests -ninja ninja -j $IMPALA_BUILD_THREADS docker_images export TEST_START_CLUSTER_ARGS="--docker_network=impala-cluster" export FE_TEST=false export BE_TEST=false export JDBC_TEST=false export CLUSTER_TEST=false ./bin/run-all-tests.sh Change-Id: Iee86cbd2c4631a014af1e8cef8e1cd523a812755 --- M bin/start-impala-cluster.py M docker/daemon_entrypoint.sh M docker/impala_base/Dockerfile M testdata/workloads/functional-query/queries/QueryTest/kudu_alter.test M testdata/workloads/functional-query/queries/QueryTest/kudu_create.test M tests/catalog_service/test_catalog_service_client.py M tests/common/custom_cluster_test_suite.py M tests/common/environ.py M tests/common/impala_cluster.py M tests/common/impala_test_suite.py M tests/common/skip.py M tests/conftest.py M tests/custom_cluster/test_jvm_mem_tracking.py M tests/custom_cluster/test_krpc_mem_usage.py M tests/custom_cluster/test_rpc_timeout.py M tests/custom_cluster/test_udf_concurrency.py M tests/hs2/test_fetch_first.py M tests/hs2/test_hs2.py M tests/hs2/test_json_endpoints.py M tests/metadata/test_compute_stats.py M tests/metadata/test_ddl.py M tests/observability/test_log_fragments.py M tests/query_test/test_hash_join_timer.py M tests/query_test/test_hdfs_caching.py M tests/query_test/test_insert.py M tests/query_test/test_insert_behaviour.py M tests/query_test/test_kudu.py M tests/query_test/test_lifecycle.py M tests/query_test/test_local_fs.py M tests/query_test/test_mem_usage_scaling.py M tests/query_test/test_queries.py M tests/query_test/test_udfs.py M tests/run-tests.py M tests/statestore/test_statestore.py M tests/stress/test_mini_stress.py M tests/webserver/test_web_pages.py 36 files changed, 183 insertions(+), 80 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/12639/15 -- To view, visit http://gerrit.cloudera.org:8080/12639 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iee86cbd2c4631a014af1e8cef8e1cd523a812755 Gerrit-Change-Number: 12639 Gerrit-PatchSet: 15 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-6718: Add support for column-level permissions on views
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12959 ) Change subject: IMPALA-6718: Add support for column-level permissions on views .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2684/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12959 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If81e683212cba22cc0fa5fc091ec3c799fa33e14 Gerrit-Change-Number: 12959 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 08 Apr 2019 21:46:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8395: Parse older formats of /proc/net/dev correctly
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12954 ) Change subject: IMPALA-8395: Parse older formats of /proc/net/dev correctly .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2683/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12954 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic804955d8e4269e787037a6dc68bef2d70382426 Gerrit-Change-Number: 12954 Gerrit-PatchSet: 3 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Mon, 08 Apr 2019 21:45:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8390: clean up test vectors in test cancellation.py
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12960 ) Change subject: IMPALA-8390: clean up test vectors in test_cancellation.py .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/12960/1/tests/common/test_vector.py File tests/common/test_vector.py: http://gerrit.cloudera.org:8080/#/c/12960/1/tests/common/test_vector.py@76 PS1, Line 76: \ flake8: E502 the backslash is redundant between brackets http://gerrit.cloudera.org:8080/#/c/12960/1/tests/common/test_vector.py@79 PS1, Line 79: , flake8: W602 deprecated form of raising exception -- To view, visit http://gerrit.cloudera.org:8080/12960 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9673fe82bda5314aff6a51d1961759ff286fbf6f Gerrit-Change-Number: 12960 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 08 Apr 2019 21:38:52 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8346: Specify hadoop user explicitly during data load
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12958 ) Change subject: IMPALA-8346: Specify hadoop user explicitly during data load .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2682/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12958 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe11d7cb50d9e2657152c94f8defcbc69ca7e1ba Gerrit-Change-Number: 12958 Gerrit-PatchSet: 1 Gerrit-Owner: David Knupp Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 08 Apr 2019 21:30:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6718: Add support for column-level permissions on views
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12959 Change subject: IMPALA-6718: Add support for column-level permissions on views .. IMPALA-6718: Add support for column-level permissions on views This patch adds support for column-level permissions on views. The following statements are now supported. GRANT select(col) ON db.my_view TO ROLE my_role -- Sentry only REVOKE select(col) ON db.my_view FROM ROLE my_role -- Sentry only GRANT select(col) ON db.my_view TO USER my_user -- Ranger only REVOKE select(col) ON db.my_view FROM USER my_user -- Ranger only GRANT select(col) ON db.my_view TO GROUP my_group -- Ranger only REVOKE select(col) ON db.my_view FROM GROUP my_group -- Ranger only Testing: - Updated AuthorizationStmtTest to with new test cases - Ran all FE tests - Ran all E2E authorization tests Change-Id: If81e683212cba22cc0fa5fc091ec3c799fa33e14 --- M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java 3 files changed, 30 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/12959/2 -- To view, visit http://gerrit.cloudera.org:8080/12959 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: If81e683212cba22cc0fa5fc091ec3c799fa33e14 Gerrit-Change-Number: 12959 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-7981: Add host disk statistics to profile
Lars Volker has posted comments on this change. (
http://gerrit.cloudera.org:8080/12822 )
Change subject: IMPALA-7981: Add host disk statistics to profile
..
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/12822/4/be/src/util/system-state-info.h
File be/src/util/system-state-info.h:
http://gerrit.cloudera.org:8080/#/c/12822/4/be/src/util/system-state-info.h@255
PS4, Line 255: for (int i = 0; i < num_values; ++i) {
: int64_t v = -1;
: (*ss) >> v;
> I think I prefer the one over there, but I don't feel strongly about either
On the other hand, this one uses standard c++ stringstreams so it might be
easier to understand quickly.
--
To view, visit http://gerrit.cloudera.org:8080/12822
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I373e7da47a0d722938e6ca1572c49a502951ed57
Gerrit-Change-Number: 12822
Gerrit-PatchSet: 4
Gerrit-Owner: Lars Volker
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Michael Ho
Gerrit-Comment-Date: Mon, 08 Apr 2019 21:19:12 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7981: Add host disk statistics to profile
Lars Volker has posted comments on this change. (
http://gerrit.cloudera.org:8080/12822 )
Change subject: IMPALA-7981: Add host disk statistics to profile
..
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/12822/4/be/src/util/system-state-info.h
File be/src/util/system-state-info.h:
http://gerrit.cloudera.org:8080/#/c/12822/4/be/src/util/system-state-info.h@255
PS4, Line 255: for (int i = 0; i < num_values; ++i) {
: int64_t v = -1;
: (*ss) >> v;
> Do you plan to choose https://gerrit.cloudera.org/#/c/12954/2/be/src/util/s
I think I prefer the one over there, but I don't feel strongly about either. Do
you have a preference?
--
To view, visit http://gerrit.cloudera.org:8080/12822
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I373e7da47a0d722938e6ca1572c49a502951ed57
Gerrit-Change-Number: 12822
Gerrit-PatchSet: 4
Gerrit-Owner: Lars Volker
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Michael Ho
Gerrit-Comment-Date: Mon, 08 Apr 2019 21:17:58 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8395: Parse older formats of /proc/net/dev correctly
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12954 ) Change subject: IMPALA-8395: Parse older formats of /proc/net/dev correctly .. Patch Set 3: (4 comments) http://gerrit.cloudera.org:8080/#/c/12954/3/be/src/util/system-state-info-test.cc File be/src/util/system-state-info-test.cc: http://gerrit.cloudera.org:8080/#/c/12954/3/be/src/util/system-state-info-test.cc@96 PS3, Line 96: string dev_net = R"(Inter-| Receive | Transmit line too long (98 > 90) http://gerrit.cloudera.org:8080/#/c/12954/3/be/src/util/system-state-info-test.cc@97 PS3, Line 97: face |bytespackets errs drop fifo frame compressed multicast|bytespackets errs drop fifo colls carrier compressed line too long (122 > 90) http://gerrit.cloudera.org:8080/#/c/12954/3/be/src/util/system-state-info-test.cc@98 PS3, Line 98: lo:5829 53000 0 0 0 5829 53000 0 0 0 line too long (122 > 90) http://gerrit.cloudera.org:8080/#/c/12954/3/be/src/util/system-state-info-test.cc@99 PS3, Line 99: eth0:285025090 212208000 0 0 0 9137793 84770000 0 0 0)"; line too long (126 > 90) -- To view, visit http://gerrit.cloudera.org:8080/12954 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic804955d8e4269e787037a6dc68bef2d70382426 Gerrit-Change-Number: 12954 Gerrit-PatchSet: 3 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Mon, 08 Apr 2019 21:17:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8395: Parse older formats of /proc/net/dev correctly
Hello Bharath Vissapragada, Michael Ho, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12954 to look at the new patch set (#3). Change subject: IMPALA-8395: Parse older formats of /proc/net/dev correctly .. IMPALA-8395: Parse older formats of /proc/net/dev correctly Older kernel versions don't have a space between the interface name and the first counter value in /proc/net/dev. This change reworks the parsing logic to support such older formats and adds a unit test for it. Change-Id: Ic804955d8e4269e787037a6dc68bef2d70382426 --- M be/src/util/system-state-info-test.cc M be/src/util/system-state-info.cc M be/src/util/system-state-info.h 3 files changed, 40 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/12954/3 -- To view, visit http://gerrit.cloudera.org:8080/12954 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic804955d8e4269e787037a6dc68bef2d70382426 Gerrit-Change-Number: 12954 Gerrit-PatchSet: 3 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho
[Impala-ASF-CR] IMPALA-8395: Parse older formats of /proc/net/dev correctly
Lars Volker has posted comments on this change. (
http://gerrit.cloudera.org:8080/12954 )
Change subject: IMPALA-8395: Parse older formats of /proc/net/dev correctly
..
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/12954/2/be/src/util/system-state-info.cc
File be/src/util/system-state-info.cc:
http://gerrit.cloudera.org:8080/#/c/12954/2/be/src/util/system-state-info.cc@168
PS2, Line 168: colon_idx = sp.find_first_of(':');
> Should this also check for the case of colon_idx == npos and skip ?
Counters will come up empty in this case because substr() will return an empty
StringPiece. Should I explain this in a comment?
http://gerrit.cloudera.org:8080/#/c/12954/2/be/src/util/system-state-info.cc@170
PS2, Line 170:
> I wonder if we should just skip if counters.values() < NUM_NET_VALUES ? Wil
Done. I checked Centos6 and the manpage for proc lists the same fields.
--
To view, visit http://gerrit.cloudera.org:8080/12954
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic804955d8e4269e787037a6dc68bef2d70382426
Gerrit-Change-Number: 12954
Gerrit-PatchSet: 2
Gerrit-Owner: Lars Volker
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Michael Ho
Gerrit-Comment-Date: Mon, 08 Apr 2019 21:16:22 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8371: Return appropriate error code for unified backend tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12885 ) Change subject: IMPALA-8371: Return appropriate error code for unified backend tests .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2681/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12885 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia146d026d42f76d5ea12d92798f299182de03eef Gerrit-Change-Number: 12885 Gerrit-PatchSet: 6 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Mon, 08 Apr 2019 21:10:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8393: Skip ranger setup for unsupported environments
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12957 ) Change subject: IMPALA-8393: Skip ranger setup for unsupported environments .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2680/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12957 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie81dda992cf29792468580b182e570132d5ce0a1 Gerrit-Change-Number: 12957 Gerrit-PatchSet: 6 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 08 Apr 2019 21:05:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7981: Add host disk statistics to profile
Michael Ho has posted comments on this change. (
http://gerrit.cloudera.org:8080/12822 )
Change subject: IMPALA-7981: Add host disk statistics to profile
..
Patch Set 4:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/12822/4/be/src/util/system-state-info.h
File be/src/util/system-state-info.h:
http://gerrit.cloudera.org:8080/#/c/12822/4/be/src/util/system-state-info.h@255
PS4, Line 255: for (int i = 0; i < num_values; ++i) {
: int64_t v = -1;
: (*ss) >> v;
Do you plan to choose
https://gerrit.cloudera.org/#/c/12954/2/be/src/util/system-state-info.cc@169
over the current implementation ?
http://gerrit.cloudera.org:8080/#/c/12822/2/be/src/util/system-state-info.h
File be/src/util/system-state-info.h:
http://gerrit.cloudera.org:8080/#/c/12822/2/be/src/util/system-state-info.h@183
PS2, Line 183: /// The enum names correspond to the fields of /proc/diskstats
: /// https://www.kernel.org/doc/Documentation/iostats.txt
> These are the fields present in 2.6, while 4.8 added more fields that we're
Yes, I think the link to the doc should be sufficient.
--
To view, visit http://gerrit.cloudera.org:8080/12822
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I373e7da47a0d722938e6ca1572c49a502951ed57
Gerrit-Change-Number: 12822
Gerrit-PatchSet: 4
Gerrit-Owner: Lars Volker
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Michael Ho
Gerrit-Comment-Date: Mon, 08 Apr 2019 20:56:43 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8371: Return appropriate error code for unified backend tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12885 ) Change subject: IMPALA-8371: Return appropriate error code for unified backend tests .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3996/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/12885 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia146d026d42f76d5ea12d92798f299182de03eef Gerrit-Change-Number: 12885 Gerrit-PatchSet: 6 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Mon, 08 Apr 2019 20:47:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8346: Specify hadoop user explicitly during data load
David Knupp has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12958 Change subject: IMPALA-8346: Specify hadoop user explicitly during data load .. IMPALA-8346: Specify hadoop user explicitly during data load Directory ownership and permissions need to be set differently, depending upon whether the test data is being setup for running tests on a local mini-cluster vs. a remote deployed cluster. Tested by installing data successfully in both contexts. Change-Id: Ibe11d7cb50d9e2657152c94f8defcbc69ca7e1ba --- M testdata/bin/create-tpcds-testcase-files.sh 1 file changed, 8 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/58/12958/1 -- To view, visit http://gerrit.cloudera.org:8080/12958 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ibe11d7cb50d9e2657152c94f8defcbc69ca7e1ba Gerrit-Change-Number: 12958 Gerrit-PatchSet: 1 Gerrit-Owner: David Knupp Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: David Knupp
[Impala-ASF-CR] IMPALA-8371: Return appropriate error code for unified backend tests
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/12885 ) Change subject: IMPALA-8371: Return appropriate error code for unified backend tests .. Patch Set 5: (2 comments) http://gerrit.cloudera.org:8080/#/c/12885/2/bin/gen-backend-test-script.py File bin/gen-backend-test-script.py: http://gerrit.cloudera.org:8080/#/c/12885/2/bin/gen-backend-test-script.py@37 PS2, Line 37: """).lstrip() > I see. If you add a "\" at the end of the first line, it'll dedent the rest Good point, switched to use the "\" and removed the lstrip http://gerrit.cloudera.org:8080/#/c/12885/5/bin/gen-backend-test-script.py File bin/gen-backend-test-script.py: http://gerrit.cloudera.org:8080/#/c/12885/5/bin/gen-backend-test-script.py@24 PS5, Line 24: # This script is used by the build system and is not intended to be run directly. > I'd be inclined to drop the parameter description above and repeat them in Done -- To view, visit http://gerrit.cloudera.org:8080/12885 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia146d026d42f76d5ea12d92798f299182de03eef Gerrit-Change-Number: 12885 Gerrit-PatchSet: 5 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Mon, 08 Apr 2019 20:46:27 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8371: Return appropriate error code for unified backend tests
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/12885 ) Change subject: IMPALA-8371: Return appropriate error code for unified backend tests .. Patch Set 6: Code-Review+2 Carry +2 -- To view, visit http://gerrit.cloudera.org:8080/12885 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia146d026d42f76d5ea12d92798f299182de03eef Gerrit-Change-Number: 12885 Gerrit-PatchSet: 6 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Mon, 08 Apr 2019 20:46:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7981: Add host disk statistics to profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12822 ) Change subject: IMPALA-7981: Add host disk statistics to profile .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2679/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12822 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I373e7da47a0d722938e6ca1572c49a502951ed57 Gerrit-Change-Number: 12822 Gerrit-PatchSet: 3 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Mon, 08 Apr 2019 20:53:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8393: Skip ranger setup for unsupported environments
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12957 ) Change subject: IMPALA-8393: Skip ranger setup for unsupported environments .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2678/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12957 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie81dda992cf29792468580b182e570132d5ce0a1 Gerrit-Change-Number: 12957 Gerrit-PatchSet: 5 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 08 Apr 2019 20:49:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8371: Return appropriate error code for unified backend tests
Hello Andrew Sherman, Lars Volker, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12885 to look at the new patch set (#6). Change subject: IMPALA-8371: Return appropriate error code for unified backend tests .. IMPALA-8371: Return appropriate error code for unified backend tests Unified backend tests rely on generating bash scripts for each test that call the unified executable with a filter to run the appropriate subset of the tests. The generated script currently does not return the return code from the test execution. This changes the test execution scripts to return the appropriate return code. To do this, the script generator is changed from a bash implementation in bin/gen-backend-test-script.sh to a python implementation in bin/gen-backend-test-script.py. This makes it easier to handle shell variables in the script template correctly. Testing: - Ran backend tests on centos 6, centos 7 - Manually tested with a failing test and verified return value Change-Id: Ia146d026d42f76d5ea12d92798f299182de03eef --- M be/CMakeLists.txt A bin/gen-backend-test-script.py D bin/gen-backend-test-script.sh 3 files changed, 80 insertions(+), 51 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/12885/6 -- To view, visit http://gerrit.cloudera.org:8080/12885 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia146d026d42f76d5ea12d92798f299182de03eef Gerrit-Change-Number: 12885 Gerrit-PatchSet: 6 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] IMPALA-8393: Skip ranger setup for unsupported environments
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12957 ) Change subject: IMPALA-8393: Skip ranger setup for unsupported environments .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2677/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12957 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie81dda992cf29792468580b182e570132d5ce0a1 Gerrit-Change-Number: 12957 Gerrit-PatchSet: 4 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 08 Apr 2019 20:44:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8395: Parse older formats of /proc/net/dev correctly
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12954 ) Change subject: IMPALA-8395: Parse older formats of /proc/net/dev correctly .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2676/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12954 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic804955d8e4269e787037a6dc68bef2d70382426 Gerrit-Change-Number: 12954 Gerrit-PatchSet: 2 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Mon, 08 Apr 2019 20:41:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8395: Parse older formats of /proc/net/dev correctly
Michael Ho has posted comments on this change. (
http://gerrit.cloudera.org:8080/12954 )
Change subject: IMPALA-8395: Parse older formats of /proc/net/dev correctly
..
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/12954/2/be/src/util/system-state-info.cc
File be/src/util/system-state-info.cc:
http://gerrit.cloudera.org:8080/#/c/12954/2/be/src/util/system-state-info.cc@168
PS2, Line 168: colon_idx = sp.find_first_of(':');
Should this also check for the case of colon_idx == npos and skip ?
http://gerrit.cloudera.org:8080/#/c/12954/2/be/src/util/system-state-info.cc@170
PS2, Line 170:
I wonder if we should just skip if counters.values() < NUM_NET_VALUES ? Will
doing so cause us to always skip for certain versions of old kernels ?
--
To view, visit http://gerrit.cloudera.org:8080/12954
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic804955d8e4269e787037a6dc68bef2d70382426
Gerrit-Change-Number: 12954
Gerrit-PatchSet: 2
Gerrit-Owner: Lars Volker
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Michael Ho
Gerrit-Comment-Date: Mon, 08 Apr 2019 20:38:47 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8363: Deny access when column masking or row filtering is enabled in Ranger
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12927 ) Change subject: IMPALA-8363: Deny access when column masking or row filtering is enabled in Ranger .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/12927/4/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java File fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java: http://gerrit.cloudera.org:8080/#/c/12927/4/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java@160 PS4, Line 160: case COLUMN: > Ranger column masking in views apply on either the view or base table permi I think part of the problem here is column-level permissions on views are broken in Impala. So I'm fixing this one first: https://issues.apache.org/jira/browse/IMPALA-6718 and will come back to this and hopefully it will make things much simpler. -- To view, visit http://gerrit.cloudera.org:8080/12927 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If46b4bf24d916e4a4ea8a36ff4acfd95d5f45c8e Gerrit-Change-Number: 12927 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 08 Apr 2019 20:29:25 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8395: Parse older formats of /proc/net/dev correctly
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/12954 ) Change subject: IMPALA-8395: Parse older formats of /proc/net/dev correctly .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/12954/1/be/src/util/system-state-info-test.cc File be/src/util/system-state-info-test.cc: http://gerrit.cloudera.org:8080/#/c/12954/1/be/src/util/system-state-info-test.cc@80 PS1, Line 80:face |bytespackets errs drop fifo frame compressed multicast|bytespackets errs drop fifo colls carrier compressed > Do we currently do this in other tests? I like that they're are inlined her Not too strong, I thought putting in a file means better readability. Don't see places where we exactly do something like this for be-tests but we do store the certs and pem files for thrift-server-test in be/src/testutil. Your call, like I mentioned, not too strong. -- To view, visit http://gerrit.cloudera.org:8080/12954 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic804955d8e4269e787037a6dc68bef2d70382426 Gerrit-Change-Number: 12954 Gerrit-PatchSet: 1 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Mon, 08 Apr 2019 20:24:24 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8393: Skip ranger setup for unsupported environments
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12957 ) Change subject: IMPALA-8393: Skip ranger setup for unsupported environments .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3995/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12957 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie81dda992cf29792468580b182e570132d5ce0a1 Gerrit-Change-Number: 12957 Gerrit-PatchSet: 6 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 08 Apr 2019 20:23:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5843: Use page index in Parquet files to skip pages
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12065 ) Change subject: IMPALA-5843: Use page index in Parquet files to skip pages .. Patch Set 9: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/2675/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/12065 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0cc99f129f2048dbafbe7f5a51d1ea3a5005731a Gerrit-Change-Number: 12065 Gerrit-PatchSet: 9 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 08 Apr 2019 20:23:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8393: Skip ranger setup for unsupported environments
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12957 ) Change subject: IMPALA-8393: Skip ranger setup for unsupported environments .. Patch Set 6: Code-Review+2 Thanks for the fix. -- To view, visit http://gerrit.cloudera.org:8080/12957 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie81dda992cf29792468580b182e570132d5ce0a1 Gerrit-Change-Number: 12957 Gerrit-PatchSet: 6 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 08 Apr 2019 20:23:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8393: Skip ranger setup for unsupported environments
Austin Nobis has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/12957 ) Change subject: IMPALA-8393: Skip ranger setup for unsupported environments .. IMPALA-8393: Skip ranger setup for unsupported environments Previously, the setup-ranger step in create-load-data.sh was hard coded with localhost as the host for Ranger. This patch makes it possible to skip the setup for Ranger by using the flag -skip_ranger. The script was also updated to set the SKIP_RANGER variable when the REMOTE_LOAD environment variable is set. Testing: - Testing was performed by calling the script with and without the setup-ranger flag set as well as calling the script with and without the REMOTE_LOAD environment variable set. Change-Id: Ie81dda992cf29792468580b182e570132d5ce0a1 --- M testdata/bin/create-load-data.sh 1 file changed, 12 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/12957/6 -- To view, visit http://gerrit.cloudera.org:8080/12957 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie81dda992cf29792468580b182e570132d5ce0a1 Gerrit-Change-Number: 12957 Gerrit-PatchSet: 6 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-5843: Use page index in Parquet files to skip pages
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/12065 ) Change subject: IMPALA-5843: Use page index in Parquet files to skip pages .. Patch Set 7: (54 comments) In a few comments the row group level profile counters came up. Whether we should only have "NumStatsFilteredRowGroups" and count row groups that are filtered out by either row group-level stats or page-level stats. Or, we should have multiple variants of that counter, counting the row group-level and page-level filtering separately. What do you think? For debugging it's good to have more fine-grained data, but I'm not sure that the query profile is the right place for such information. http://gerrit.cloudera.org:8080/#/c/12065/8/be/src/exec/hdfs-scan-node-base.h File be/src/exec/hdfs-scan-node-base.h: http://gerrit.cloudera.org:8080/#/c/12065/8/be/src/exec/hdfs-scan-node-base.h@272 PS8, Line 272: io::ScanRange* AllocateScanRange(hdfsFS fs, const char* file, > nit: same line wrapping for both methods Done http://gerrit.cloudera.org:8080/#/c/12065/8/be/src/exec/parquet/hdfs-parquet-scanner.h File be/src/exec/parquet/hdfs-parquet-scanner.h: http://gerrit.cloudera.org:8080/#/c/12065/8/be/src/exec/parquet/hdfs-parquet-scanner.h@360 PS8, Line 360: name in LLVM IR. > nit: make this a const& or non-const ptr. If a const* is required to be nul The signature matches the constructor of ColumnStatsReader. I added a short comment about that 'col_order' might be null. http://gerrit.cloudera.org:8080/#/c/12065/8/be/src/exec/parquet/hdfs-parquet-scanner.h@428 PS8, Line 428: > nit: lower case Done http://gerrit.cloudera.org:8080/#/c/12065/8/be/src/exec/parquet/hdfs-parquet-scanner.h@429 PS8, Line 429: at are eli > nit: missing comma Done http://gerrit.cloudera.org:8080/#/c/12065/8/be/src/exec/parquet/hdfs-parquet-scanner.h@552 PS8, Line 552: /// Returns OK if the query is not cancelled and hasn't exceeded any mem limits. > This looks like a candidate to explain the various parameters in some detai Added comments. Instead of 'fn_name', I modified this function to take a 'stats_field' parameter. http://gerrit.cloudera.org:8080/#/c/12065/7/be/src/exec/parquet/hdfs-parquet-scanner.cc File be/src/exec/parquet/hdfs-parquet-scanner.cc: http://gerrit.cloudera.org:8080/#/c/12065/7/be/src/exec/parquet/hdfs-parquet-scanner.cc@119 PS7, Line 119: num_stats_filtered_row_groups_by_page_index_counter_ = > Should these be NumStatsFilteredRowGroups and NumStatsFilteredPages? See reply comment http://gerrit.cloudera.org:8080/#/c/12065/7/be/src/exec/parquet/hdfs-parquet-scanner.cc@119 PS7, Line 119: num_stats_filtered_row_groups_by_page_index_counter_ = > I am not sure how to track this, but it is also possible that some columns I'm checking if there's an offset index for the first column chunk. Offset index should be present for every column, including timestamp and double columns as well. http://gerrit.cloudera.org:8080/#/c/12065/7/be/src/exec/parquet/hdfs-parquet-scanner.cc@803 PS7, Line 803: CalculateCandidateRanges > That's a good point. One could pass skip_ranges by value and then let the c I'm passing a pointer so I can certainly avoid copying. http://gerrit.cloudera.org:8080/#/c/12065/7/be/src/exec/parquet/hdfs-parquet-scanner.cc@813 PS7, Line 813: d > You could pass the scalar reader into ComputeCandidatePages() to call the s Yeah I definetely don't want to introduce a dependency in parquet-common to ScalarReaders. And using a lambda feels a bit of an overkill to me for such a simple problem. But if you feel strongly about it I can settle with the lambda. What concerns me most about setting this member of ScalarReader here is that later we are calling InitScalarColumns() and even Reset() on the scalar readers. I just couldn't yet figure out a better control flow without using unnecessary boilerplate code and extra state. But maybe I'll just store the candidate pages in a hash map, then pass them to the scalar readers InitScalarColumns(). I could use a setter there, or extend the Reset() member function. http://gerrit.cloudera.org:8080/#/c/12065/8/be/src/exec/parquet/hdfs-parquet-scanner.cc File be/src/exec/parquet/hdfs-parquet-scanner.cc: http://gerrit.cloudera.org:8080/#/c/12065/8/be/src/exec/parquet/hdfs-parquet-scanner.cc@117 PS8, Line 117: NumStatsFilteredRowGroups > This name is not precise anymore, as it only means row groups filtered by c See reply comment. http://gerrit.cloudera.org:8080/#/c/12065/8/be/src/exec/parquet/hdfs-parquet-scanner.cc@120 PS8, Line 120: NumStatsFilteredRowGroupsByPageIndex > Maybe "completely" could be added to the name to make its meaning more exac See reply comment. http://gerrit.cloudera.org:8080/#/c/12065/8/be/src/exec/parquet/hdfs-parquet-scanner.cc@517 PS8, Line 517: const parquet::ColumnChunk& col_chunk = row_group.columns[col_idx]; : const ColumnTy
[Impala-ASF-CR] IMPALA-7981: Add host disk statistics to profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12822 ) Change subject: IMPALA-7981: Add host disk statistics to profile .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/12822/3/be/src/util/system-state-info-test.cc File be/src/util/system-state-info-test.cc: http://gerrit.cloudera.org:8080/#/c/12822/3/be/src/util/system-state-info-test.cc@106 PS3, Line 106: R"( 8 0 sda 17124835 222797 716029974 8414920 43758807 38432095 7867287712 630179264 0 32547524 638999340 line too long (117 > 90) http://gerrit.cloudera.org:8080/#/c/12822/3/be/src/util/system-state-info-test.cc@107 PS3, Line 107:8 1 sda1 17124482 222797 716027002 8414892 43546943 38432095 7867287712 629089180 0 31590972 637917344)"; line too long (118 > 90) -- To view, visit http://gerrit.cloudera.org:8080/12822 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I373e7da47a0d722938e6ca1572c49a502951ed57 Gerrit-Change-Number: 12822 Gerrit-PatchSet: 3 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Mon, 08 Apr 2019 20:06:45 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7981: Add host disk statistics to profile
Hello Michael Ho, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12822 to look at the new patch set (#4). Change subject: IMPALA-7981: Add host disk statistics to profile .. IMPALA-7981: Add host disk statistics to profile This change adds host disk statistics to profiles. For each host that participates in the query execution it adds the read and write bandwidth across all disks. This includes all data read or written by the host as part of the execution of a query (spilling), by the HDFS data node, and by other processes running on the same system. The change adds tests for the added functionality to the backend and end to end tests. Change-Id: I373e7da47a0d722938e6ca1572c49a502951ed57 --- M be/src/runtime/query-state.cc M be/src/util/disk-info.h M be/src/util/system-state-info-test.cc M be/src/util/system-state-info.cc M be/src/util/system-state-info.h M tests/query_test/test_observability.py 6 files changed, 245 insertions(+), 31 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/12822/4 -- To view, visit http://gerrit.cloudera.org:8080/12822 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I373e7da47a0d722938e6ca1572c49a502951ed57 Gerrit-Change-Number: 12822 Gerrit-PatchSet: 4 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho
[Impala-ASF-CR] IMPALA-7981: Add host disk usage to profile
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/12822 ) Change subject: IMPALA-7981: Add host disk usage to profile .. Patch Set 3: (10 comments) http://gerrit.cloudera.org:8080/#/c/12822/2/be/src/util/system-state-info-test.cc File be/src/util/system-state-info-test.cc: http://gerrit.cloudera.org:8080/#/c/12822/2/be/src/util/system-state-info-test.cc@105 PS2, Line 105: string disk_stats = > Do we also care about testing for random devices which show up in /proc/dis I added two of these lines but the code will ignore anything that has not been identified by DiskInfo, and then 0 values will not to the result. http://gerrit.cloudera.org:8080/#/c/12822/2/be/src/util/system-state-info-test.cc@177 PS2, Line 177: : // Tests the computation logic for disk statistics. > Is this comment meant for the earlier test at line 103 ? Yes, thx. http://gerrit.cloudera.org:8080/#/c/12822/2/be/src/util/system-state-info.h File be/src/util/system-state-info.h: http://gerrit.cloudera.org:8080/#/c/12822/2/be/src/util/system-state-info.h@183 PS2, Line 183: /// The enum names correspond to the fields of /proc/diskstats : /// https://www.kernel.org/doc/Documentation/iostats.txt > The followings are the metrics common to both 2.6 and 4.8+, right ? These are the fields present in 2.6, while 4.8 added more fields that we're not using. Would you like me to add a comment elaborating this? It's also explained in the linked doc. http://gerrit.cloudera.org:8080/#/c/12822/2/be/src/util/system-state-info.h@221 PS2, Line 221: Invalid : /// entries are set to 0. > Invalid entries are set to 0. Done http://gerrit.cloudera.org:8080/#/c/12822/2/be/src/util/system-state-info.h@228 PS2, Line 228: /// Computes the read and w > Computes the read and write rate for the most recent epoch. Done http://gerrit.cloudera.org:8080/#/c/12822/2/be/src/util/system-state-info.h@233 PS2, Line 233: DiskStats > nit: I kind of interpret it as disk space used on first read. DiskStats may I switched to DiskStats. http://gerrit.cloudera.org:8080/#/c/12822/2/be/src/util/system-state-info.cc File be/src/util/system-state-info.cc: http://gerrit.cloudera.org:8080/#/c/12822/2/be/src/util/system-state-info.cc@45 PS2, Line 45: memset(&cpu_ratios_, 0, sizeof(cpu_ratios_)); > memset(&disk_usage_, 0, sizeof(disk_usage_)); Done http://gerrit.cloudera.org:8080/#/c/12822/2/be/src/util/system-state-info.cc@81 PS2, Line 81: du::Status status = ReadFileToString(Env::Default(), path, buf); > Will this lead to spam in the log if we periodically try reading the same p Done http://gerrit.cloudera.org:8080/#/c/12822/2/be/src/util/system-state-info.cc@199 PS2, Line 199: } > DCHECK(disk_val_idx_ == 0 || disk_val_idx_ == 1); Done http://gerrit.cloudera.org:8080/#/c/12822/2/be/src/util/system-state-info.cc@232 PS2, Line 232: > <= 0 Done -- To view, visit http://gerrit.cloudera.org:8080/12822 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I373e7da47a0d722938e6ca1572c49a502951ed57 Gerrit-Change-Number: 12822 Gerrit-PatchSet: 3 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Mon, 08 Apr 2019 20:06:12 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7981: Add host disk usage to profile
Hello Michael Ho, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12822 to look at the new patch set (#3). Change subject: IMPALA-7981: Add host disk usage to profile .. IMPALA-7981: Add host disk usage to profile This change adds host disk usage to profiles. For each host that participates in the query execution it adds the read and write bandwidth across all disks. This includes all data read or written by the host as part of the execution of a query (spilling), by the HDFS data node, and by other processes running on the same system. The change adds tests for the added functionality to the backend and end to end tests. Change-Id: I373e7da47a0d722938e6ca1572c49a502951ed57 --- M be/src/runtime/query-state.cc M be/src/util/disk-info.h M be/src/util/system-state-info-test.cc M be/src/util/system-state-info.cc M be/src/util/system-state-info.h M tests/query_test/test_observability.py 6 files changed, 245 insertions(+), 31 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/12822/3 -- To view, visit http://gerrit.cloudera.org:8080/12822 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I373e7da47a0d722938e6ca1572c49a502951ed57 Gerrit-Change-Number: 12822 Gerrit-PatchSet: 3 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho
[Impala-ASF-CR] IMPALA-8393: Skip ranger setup for unsupported environments
Austin Nobis has posted comments on this change. ( http://gerrit.cloudera.org:8080/12957 ) Change subject: IMPALA-8393: Skip ranger setup for unsupported environments .. Patch Set 5: (2 comments) http://gerrit.cloudera.org:8080/#/c/12957/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/12957/2//COMMIT_MSG@7 PS2, Line 7: Skip ranger setup for unsupported environments > Agreed -- could be something like: Done http://gerrit.cloudera.org:8080/#/c/12957/2/testdata/bin/create-load-data.sh File testdata/bin/create-load-data.sh: http://gerrit.cloudera.org:8080/#/c/12957/2/testdata/bin/create-load-data.sh@105 PS2, Line 105: > use [[ -n $REMOTE_LOAD ]] instead? Some of the code in this file uses -z: h Done -- To view, visit http://gerrit.cloudera.org:8080/12957 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie81dda992cf29792468580b182e570132d5ce0a1 Gerrit-Change-Number: 12957 Gerrit-PatchSet: 5 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 08 Apr 2019 20:05:28 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8393: Skip ranger setup for unsupported environments
Austin Nobis has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/12957 ) Change subject: IMPALA-8393: Skip ranger setup for unsupported environments .. IMPALA-8393: Skip ranger setup for unsupported environments Previously, the setup-ranger step in create-load-data.sh was hard coded with localhost as the host for Ranger. This patch makes it possible to skip the setup for Ranger by using the flag -skip_ranger. The script was also updated to set the SKIP_RANGER variable when the REMOTE_LOAD environment variable is set. Testing: - Testing was performed by calling the script with and without the setup-ranger flag set as well as calling the script with and without the REMOTE_LOAD environment variable set. Change-Id: Ie81dda992cf29792468580b182e570132d5ce0a1 --- M testdata/bin/create-load-data.sh 1 file changed, 13 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/12957/5 -- To view, visit http://gerrit.cloudera.org:8080/12957 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie81dda992cf29792468580b182e570132d5ce0a1 Gerrit-Change-Number: 12957 Gerrit-PatchSet: 5 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8393: Skip ranger setup for unsupported environments
Austin Nobis has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/12957 ) Change subject: IMPALA-8393: Skip ranger setup for unsupported environments .. IMPALA-8393: Skip ranger setup for unsupported environments Previously, the setup-ranger step in create-load-data.sh was hard coded with localhost as the host for Ranger. This patch makes it possible to skip the setup for Ranger by using the flag -skip_ranger. The script was also updated to set the SKIP_RANGER variable when the REMOTE_LOAD environment variable is set. Testing: - Testing was performed by calling the script with and without the setup-ranger flag set as well as calling the script with and without the REMOTE_LOAD environment variable set. Change-Id: Ie81dda992cf29792468580b182e570132d5ce0a1 --- M testdata/bin/create-load-data.sh 1 file changed, 14 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/12957/4 -- To view, visit http://gerrit.cloudera.org:8080/12957 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie81dda992cf29792468580b182e570132d5ce0a1 Gerrit-Change-Number: 12957 Gerrit-PatchSet: 4 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8395: Parse older formats of /proc/net/dev correctly
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12954 ) Change subject: IMPALA-8395: Parse older formats of /proc/net/dev correctly .. Patch Set 2: (4 comments) http://gerrit.cloudera.org:8080/#/c/12954/2/be/src/util/system-state-info-test.cc File be/src/util/system-state-info-test.cc: http://gerrit.cloudera.org:8080/#/c/12954/2/be/src/util/system-state-info-test.cc@96 PS2, Line 96: string dev_net = R"(Inter-| Receive | Transmit line too long (98 > 90) http://gerrit.cloudera.org:8080/#/c/12954/2/be/src/util/system-state-info-test.cc@97 PS2, Line 97: face |bytespackets errs drop fifo frame compressed multicast|bytespackets errs drop fifo colls carrier compressed line too long (122 > 90) http://gerrit.cloudera.org:8080/#/c/12954/2/be/src/util/system-state-info-test.cc@98 PS2, Line 98: lo:5829 53000 0 0 0 5829 53000 0 0 0 line too long (122 > 90) http://gerrit.cloudera.org:8080/#/c/12954/2/be/src/util/system-state-info-test.cc@99 PS2, Line 99: eth0:285025090 212208000 0 0 0 9137793 84770000 0 0 0)"; line too long (126 > 90) -- To view, visit http://gerrit.cloudera.org:8080/12954 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic804955d8e4269e787037a6dc68bef2d70382426 Gerrit-Change-Number: 12954 Gerrit-PatchSet: 2 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Mon, 08 Apr 2019 20:01:00 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8395: Parse older formats of /proc/net/dev correctly
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/12954 ) Change subject: IMPALA-8395: Parse older formats of /proc/net/dev correctly .. Patch Set 2: Code-Review+1 (2 comments) Carrying Bharath's +1. http://gerrit.cloudera.org:8080/#/c/12954/1/be/src/util/system-state-info-test.cc File be/src/util/system-state-info-test.cc: http://gerrit.cloudera.org:8080/#/c/12954/1/be/src/util/system-state-info-test.cc@80 PS1, Line 80:face |bytespackets errs drop fifo frame compressed multicast|bytespackets errs drop fifo colls carrier compressed > nit: I think you could also dump these contents to some files and read them Do we currently do this in other tests? I like that they're are inlined here which prevents an additional indirection, but I agree that the long lines are not ideal. Let me know if you feel strongly about it, otherwise I'd keep them here. http://gerrit.cloudera.org:8080/#/c/12954/1/be/src/util/system-state-info.cc File be/src/util/system-state-info.cc: http://gerrit.cloudera.org:8080/#/c/12954/1/be/src/util/system-state-info.cc@172 PS1, Line 172: > nit: trace log something? Done -- To view, visit http://gerrit.cloudera.org:8080/12954 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic804955d8e4269e787037a6dc68bef2d70382426 Gerrit-Change-Number: 12954 Gerrit-PatchSet: 2 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Mon, 08 Apr 2019 20:00:19 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8395: Parse older formats of /proc/net/dev correctly
Hello Bharath Vissapragada, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12954 to look at the new patch set (#2). Change subject: IMPALA-8395: Parse older formats of /proc/net/dev correctly .. IMPALA-8395: Parse older formats of /proc/net/dev correctly Older kernel versions don't have a space between the interface name and the first counter value in /proc/net/dev. This change reworks the parsing logic to support such older formats and adds a unit test for it. Change-Id: Ic804955d8e4269e787037a6dc68bef2d70382426 --- M be/src/util/system-state-info-test.cc M be/src/util/system-state-info.cc M be/src/util/system-state-info.h 3 files changed, 36 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/12954/2 -- To view, visit http://gerrit.cloudera.org:8080/12954 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic804955d8e4269e787037a6dc68bef2d70382426 Gerrit-Change-Number: 12954 Gerrit-PatchSet: 2 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-5843: Use page index in Parquet files to skip pages
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/12065 )
Change subject: IMPALA-5843: Use page index in Parquet files to skip pages
..
Patch Set 9:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/12065/9/be/src/exec/parquet/parquet-page-index-test.cc
File be/src/exec/parquet/parquet-page-index-test.cc:
http://gerrit.cloudera.org:8080/#/c/12065/9/be/src/exec/parquet/parquet-page-index-test.cc@108
PS9, Line 108: ValidatePageIndexRange(-1, -1, 30, 25}, {-1, -1, 5, 5},
{10, 5, 15, 5, true, 5, 50);
line too long (94 > 90)
http://gerrit.cloudera.org:8080/#/c/12065/9/tests/query_test/test_parquet_stats.py
File tests/query_test/test_parquet_stats.py:
http://gerrit.cloudera.org:8080/#/c/12065/9/tests/query_test/test_parquet_stats.py@94
PS9, Line 94:
flake8: W292 no newline at end of file
--
To view, visit http://gerrit.cloudera.org:8080/12065
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0cc99f129f2048dbafbe7f5a51d1ea3a5005731a
Gerrit-Change-Number: 12065
Gerrit-PatchSet: 9
Gerrit-Owner: Zoltan Borok-Nagy
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Pooja Nilangekar
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Mon, 08 Apr 2019 19:42:35 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5843: Use page index in Parquet files to skip pages
Hello Michael Ho, Lars Volker, Pooja Nilangekar, Tim Armstrong, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12065 to look at the new patch set (#9). Change subject: IMPALA-5843: Use page index in Parquet files to skip pages .. IMPALA-5843: Use page index in Parquet files to skip pages This commit implements page filtering based on the Parquet page index. The read and evaluation of the page index is done by the HdfsParquetScanner. At first, we determine the row ranges we are interested in, and based on the row ranges we determine the candidate pages for each column that we are reading. We still issue one ScanRange per column chunk, but we specify sub-ranges that store the candidate pages, i.e. we don't read the whole column chunk, but only fractions of it. Pages are not aligned across column chunks, i.e. page #2 of column A might store completely different rows than page #2 of column B. It means we need to implement some kind of row-skipping logic when we read the data pages. This logic is implemented in BaseScalarColumnReader and ScalarColumnReader. Collection column readers know nothing about page filtering. Page filtering can be turned off by setting the query option 'read_parquet_page_index' to false. Testing: * added some unit tests for the row range and page selection logic * generated various Parquet files with Parquet-MR * enabled Page index writing and wrote selective queries against tables written by Impala. Current tests are likely to use page filtering transparently. Performance: * measured locally, observed 3x to 10x speedup for selective queries TODO: * run standard benchmarks * measure performance for remote reads Change-Id: I0cc99f129f2048dbafbe7f5a51d1ea3a5005731a --- M be/src/common/global-flags.cc M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/parquet/CMakeLists.txt M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/parquet/hdfs-parquet-scanner.h M be/src/exec/parquet/parquet-column-readers.cc M be/src/exec/parquet/parquet-column-readers.h M be/src/exec/parquet/parquet-column-stats.cc M be/src/exec/parquet/parquet-column-stats.h A be/src/exec/parquet/parquet-common-test.cc M be/src/exec/parquet/parquet-common.cc M be/src/exec/parquet/parquet-common.h M be/src/exec/parquet/parquet-level-decoder.h A be/src/exec/parquet/parquet-page-index-test.cc A be/src/exec/parquet/parquet-page-index.cc A be/src/exec/parquet/parquet-page-index.h M be/src/exprs/literal.cc M be/src/runtime/scoped-buffer.h M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M testdata/data/README A testdata/data/alltypes_tiny_pages.parquet A testdata/data/alltypes_tiny_pages_plain.parquet A testdata/data/decimals_1_10.parquet A testdata/data/double_nested_decimals.parquet A testdata/data/nested_decimals.parquet A testdata/workloads/functional-query/queries/QueryTest/nested-types-parquet-page-index.test A testdata/workloads/functional-query/queries/QueryTest/parquet-page-index-alltypes-tiny-pages-plain.test A testdata/workloads/functional-query/queries/QueryTest/parquet-page-index-alltypes-tiny-pages.test A testdata/workloads/functional-query/queries/QueryTest/parquet-page-index-large.test A testdata/workloads/functional-query/queries/QueryTest/parquet-page-index.test M testdata/workloads/functional-query/queries/QueryTest/stats-extrapolation.test M tests/query_test/test_parquet_stats.py 36 files changed, 3,235 insertions(+), 85 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/12065/9 -- To view, visit http://gerrit.cloudera.org:8080/12065 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0cc99f129f2048dbafbe7f5a51d1ea3a5005731a Gerrit-Change-Number: 12065 Gerrit-PatchSet: 9 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8393: setup-ranger in create-load-data.sh breaks for clusters
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12957 ) Change subject: IMPALA-8393: setup-ranger in create-load-data.sh breaks for clusters .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2674/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12957 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie81dda992cf29792468580b182e570132d5ce0a1 Gerrit-Change-Number: 12957 Gerrit-PatchSet: 2 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 08 Apr 2019 19:06:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8393: setup-ranger in create-load-data.sh breaks for clusters
David Knupp has posted comments on this change. ( http://gerrit.cloudera.org:8080/12957 ) Change subject: IMPALA-8393: setup-ranger in create-load-data.sh breaks for clusters .. Patch Set 2: Code-Review+1 (1 comment) Looks basically good to me. I'll let Fredy give the final +2 after his comments get addressed. http://gerrit.cloudera.org:8080/#/c/12957/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/12957/2//COMMIT_MSG@7 PS2, Line 7: setup-ranger in create-load-data.sh breaks for clusters > nit: it's a bit hard to parse. Reword? Agreed -- could be something like: "Skip ranger setup steps in environments that don't support it." -- To view, visit http://gerrit.cloudera.org:8080/12957 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie81dda992cf29792468580b182e570132d5ce0a1 Gerrit-Change-Number: 12957 Gerrit-PatchSet: 2 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 08 Apr 2019 18:45:40 + Gerrit-HasComments: Yes
