[Impala-ASF-CR] Add missing sudo call to `service postgresql start`
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12724 ) Change subject: Add missing sudo call to `service postgresql start` .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3956/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12724 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie3859ad1308b7cd1d483d043b9f71d9f95abdf75 Gerrit-Change-Number: 12724 Gerrit-PatchSet: 2 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Tue, 26 Mar 2019 14:09:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8308: [DOCS] Impala 3.2 Release
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/12838 ) Change subject: IMPALA-8308: [DOCS] Impala 3.2 Release .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12838 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I939e391d9546bbe868fd3045af81882fcaee3b03 Gerrit-Change-Number: 12838 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 26 Mar 2019 14:30:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8330: Impala shell config file should use flag names
Fredy Wijaya has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/12823 ) Change subject: IMPALA-8330: Impala shell config file should use flag names .. IMPALA-8330: Impala shell config file should use flag names This patch changes the file format in Impala shell config file to accept both short and long flag names instead of optparse's dest names (variable names to store flag values) because dest names are internal to Impala shell. This patch does not change the format and behavior of Impala shell config file with [impala.query_options] section. Format: [impala] flag_name=flag_value Example: [impala] ; This is long flag. query=select 1 ; This is short flag. Q=DEFAULT_FILE_FORMAT=parquet ; Flags can be repeated with , var=msg1=hello,var=msg2=world Testing: - Ran all E2E shell tests on Python 2.6 and 2.7. Change-Id: Ic43603c1b538af08fddcab1b2c1f6ad1af1a6cb9 --- M shell/impala_shell.py M shell/option_parser.py M tests/shell/good_impalarc 3 files changed, 31 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/12823/5 -- To view, visit http://gerrit.cloudera.org:8080/12823 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic43603c1b538af08fddcab1b2c1f6ad1af1a6cb9 Gerrit-Change-Number: 12823 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8330: Impala shell config file should use flag names
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12823 ) Change subject: IMPALA-8330: Impala shell config file should use flag names .. Patch Set 5: (2 comments) http://gerrit.cloudera.org:8080/#/c/12823/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/12823/2//COMMIT_MSG@10 PS2, Line 10: both short and long flag names instead of optparse's dest names > My life would be easier if you explained what a 'dest name' is here. Done http://gerrit.cloudera.org:8080/#/c/12823/2//COMMIT_MSG@24 PS2, Line 24: Q=DEFAULT_FILE_FORMAT=parquet > Flags can be repeated. So if I have Impala shell allows it, e.g. impala-shell.sh -query_option=DEFAULT=parquet -Q DEFAULT_FILE_FORMAT=orc The behavior is undefined, it can either be parquet or orc. -- To view, visit http://gerrit.cloudera.org:8080/12823 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic43603c1b538af08fddcab1b2c1f6ad1af1a6cb9 Gerrit-Change-Number: 12823 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 26 Mar 2019 14:40:35 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8330: Impala shell config file should use flag names
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12823 ) Change subject: IMPALA-8330: Impala shell config file should use flag names .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2539/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12823 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic43603c1b538af08fddcab1b2c1f6ad1af1a6cb9 Gerrit-Change-Number: 12823 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 26 Mar 2019 15:13:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8359: Fix coverage data generation for impalads
Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12858 Change subject: IMPALA-8359: Fix coverage data generation for impalads .. IMPALA-8359: Fix coverage data generation for impalads impala::InitCommonRuntime() sets a signal handler for SIGTERM. Sometimes we already have a signal handler that we want to use. E.g. when building with code coverage, the gcov runtime also registers a signal handler that writes the coverage data. I added a global 'old_sigaction' variable that stores the old signal handler and invoke it in HandleSigTerm(). I tested manually. Change-Id: I9be1e1e73b6cfc3557077f763aee4dbfcc7a2d27 --- M be/src/common/init.cc 1 file changed, 19 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/58/12858/1 -- To view, visit http://gerrit.cloudera.org:8080/12858 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I9be1e1e73b6cfc3557077f763aee4dbfcc7a2d27 Gerrit-Change-Number: 12858 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8359: Fix coverage data generation for impalads
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12858 ) Change subject: IMPALA-8359: Fix coverage data generation for impalads .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/2540/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/12858 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9be1e1e73b6cfc3557077f763aee4dbfcc7a2d27 Gerrit-Change-Number: 12858 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Tue, 26 Mar 2019 15:45:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8362: Increase authorization test coverage with local catalog
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12859 Change subject: IMPALA-8362: Increase authorization test coverage with local catalog .. IMPALA-8362: Increase authorization test coverage with local catalog This patch adds authorization test that runs with local catalog by reusing the existing grant and revoke tests. Testing: - Added a new authorization test with local catalog - Ran all authorization tests Change-Id: I1efc56709bc3da16978589e20d7c611db4fa271a --- M tests/authorization/test_grant_revoke.py 1 file changed, 20 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/12859/2 -- To view, visit http://gerrit.cloudera.org:8080/12859 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I1efc56709bc3da16978589e20d7c611db4fa271a Gerrit-Change-Number: 12859 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-8359: Fix coverage data generation for impalads
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/12858 ) Change subject: IMPALA-8359: Fix coverage data generation for impalads .. Patch Set 1: Code-Review-1 Turned out the problem is not with the new signal handler, but with the call to _exit(). -- To view, visit http://gerrit.cloudera.org:8080/12858 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9be1e1e73b6cfc3557077f763aee4dbfcc7a2d27 Gerrit-Change-Number: 12858 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 26 Mar 2019 17:05:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8359: Fix coverage data generation for impalads
Hello Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12858 to look at the new patch set (#2). Change subject: IMPALA-8359: Fix coverage data generation for impalads .. IMPALA-8359: Fix coverage data generation for impalads impala::InitCommonRuntime() sets a signal handler for SIGTERM. It calls _exit(0) which causes normal program termination without cleaning up, i.e. no destructors are called etc. Gcov writes the coverage data in this cleanup phase, so calling _exit() prevents flushing coverage data. Now the '-codecoverage' flag also defines a macro named CODE_COVERAGE_ENABLED. If this macro is defined we call exit() instead of _exit(), because exit() cleans up the process properly. I tested manually. Change-Id: I9be1e1e73b6cfc3557077f763aee4dbfcc7a2d27 --- M be/CMakeLists.txt M be/src/common/init.cc 2 files changed, 7 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/58/12858/2 -- To view, visit http://gerrit.cloudera.org:8080/12858 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9be1e1e73b6cfc3557077f763aee4dbfcc7a2d27 Gerrit-Change-Number: 12858 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8362: Increase authorization test coverage with local catalog
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/12859 ) Change subject: IMPALA-8362: Increase authorization test coverage with local catalog .. Patch Set 2: Code-Review+2 (4 comments) http://gerrit.cloudera.org:8080/#/c/12859/2/tests/authorization/test_grant_revoke.py File tests/authorization/test_grant_revoke.py: http://gerrit.cloudera.org:8080/#/c/12859/2/tests/authorization/test_grant_revoke.py@112 PS2, Line 112: global_catalog Don't think the code refers to it as global_catalog. Call it default_catalog or v1_catalog or something? http://gerrit.cloudera.org:8080/#/c/12859/2/tests/authorization/test_grant_revoke.py@113 PS2, Line 113: non-local catalog If local catalog becomes default, this is stale. May be can just call default? http://gerrit.cloudera.org:8080/#/c/12859/2/tests/authorization/test_grant_revoke.py@118 PS2, Line 118: 1 nit: true (to be consistent with other tests) http://gerrit.cloudera.org:8080/#/c/12859/2/tests/authorization/test_grant_revoke.py@120 PS2, Line 120: 1 same. -- To view, visit http://gerrit.cloudera.org:8080/12859 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1efc56709bc3da16978589e20d7c611db4fa271a Gerrit-Change-Number: 12859 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 26 Mar 2019 17:42:45 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8362: Increase authorization test coverage with local catalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12859 ) Change subject: IMPALA-8362: Increase authorization test coverage with local catalog .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2541/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12859 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1efc56709bc3da16978589e20d7c611db4fa271a Gerrit-Change-Number: 12859 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 26 Mar 2019 17:40:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8308: [DOCS] Impala 3.2 Release
Alex Rodoni has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12838 ) Change subject: IMPALA-8308: [DOCS] Impala 3.2 Release .. IMPALA-8308: [DOCS] Impala 3.2 Release The following notes were added: - New features - Fixed issues - Incompatible changes - Upgrade notes Change-Id: I939e391d9546bbe868fd3045af81882fcaee3b03 Reviewed-on: http://gerrit.cloudera.org:8080/12838 Tested-by: Impala Public Jenkins Reviewed-by: Gabor Kaszab --- M docs/topics/impala_fixed_issues.xml M docs/topics/impala_incompatible_changes.xml M docs/topics/impala_new_features.xml M docs/topics/impala_upgrading.xml 4 files changed, 249 insertions(+), 3 deletions(-) Approvals: Impala Public Jenkins: Verified Gabor Kaszab: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/12838 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I939e391d9546bbe868fd3045af81882fcaee3b03 Gerrit-Change-Number: 12838 Gerrit-PatchSet: 5 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8362: Increase authorization test coverage with local catalog
Fredy Wijaya has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/12859 ) Change subject: IMPALA-8362: Increase authorization test coverage with local catalog .. IMPALA-8362: Increase authorization test coverage with local catalog This patch adds authorization test that runs with local catalog by reusing the existing grant and revoke tests. Testing: - Added a new authorization test with local catalog - Ran all authorization tests Change-Id: I1efc56709bc3da16978589e20d7c611db4fa271a --- M tests/authorization/test_grant_revoke.py 1 file changed, 20 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/12859/3 -- To view, visit http://gerrit.cloudera.org:8080/12859 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1efc56709bc3da16978589e20d7c611db4fa271a Gerrit-Change-Number: 12859 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8362: Increase authorization test coverage with local catalog
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12859 ) Change subject: IMPALA-8362: Increase authorization test coverage with local catalog .. Patch Set 3: Code-Review+2 (4 comments) Carry Bharath's +2. http://gerrit.cloudera.org:8080/#/c/12859/2/tests/authorization/test_grant_revoke.py File tests/authorization/test_grant_revoke.py: http://gerrit.cloudera.org:8080/#/c/12859/2/tests/authorization/test_grant_revoke.py@112 PS2, Line 112: catalog_v1(sel > Don't think the code refers to it as global_catalog. Call it default_catalo Done http://gerrit.cloudera.org:8080/#/c/12859/2/tests/authorization/test_grant_revoke.py@113 PS2, Line 113: catalog v1.""" > If local catalog becomes default, this is stale. May be can just call defau Done http://gerrit.cloudera.org:8080/#/c/12859/2/tests/authorization/test_grant_revoke.py@118 PS2, Line 118: t > nit: true (to be consistent with other tests) Done http://gerrit.cloudera.org:8080/#/c/12859/2/tests/authorization/test_grant_revoke.py@120 PS2, Line 120: t > same. Done -- To view, visit http://gerrit.cloudera.org:8080/12859 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1efc56709bc3da16978589e20d7c611db4fa271a Gerrit-Change-Number: 12859 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 26 Mar 2019 18:00:20 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8362: Increase authorization test coverage with local catalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12859 ) Change subject: IMPALA-8362: Increase authorization test coverage with local catalog .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12859 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1efc56709bc3da16978589e20d7c611db4fa271a Gerrit-Change-Number: 12859 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 26 Mar 2019 18:00:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8362: Increase authorization test coverage with local catalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12859 ) Change subject: IMPALA-8362: Increase authorization test coverage with local catalog .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3957/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12859 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1efc56709bc3da16978589e20d7c611db4fa271a Gerrit-Change-Number: 12859 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 26 Mar 2019 18:00:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8359: Fix coverage data generation for impalads
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12858 ) Change subject: IMPALA-8359: Fix coverage data generation for impalads .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2542/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12858 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9be1e1e73b6cfc3557077f763aee4dbfcc7a2d27 Gerrit-Change-Number: 12858 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 26 Mar 2019 18:02:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8225: Add ranger support for grant/revoke privilege to/from user
Austin Nobis has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/12769 ) Change subject: IMPALA-8225: Add ranger support for grant/revoke privilege to/from user .. IMPALA-8225: Add ranger support for grant/revoke privilege to/from user This patch adds support for GRANT privilege statements to USER and REVOKE privilege statements from USER. The RangerAuthorizationManager class has been created and will throw UnsupportedOperationException when an unimplemented method is called. The grammar has been updated to support FROM USER and TO USER for GRANT/REVOKE statements. Previously, privileges could be granted to a ROLE via GRANT/REVOKE statements even when the ROLE keyword was omitted. This is still the case for ROLE based authorization to preserve backward compatibility, but Ranger will throw an exception when a GRANT/REVOKE statement excludes the USER keyword. An additional end to end test test_ranger.py was added. A single test was added that grants and revokes for a user and asserts permissions on a table. The test uses sleep statements to work with Ranger's polling interval for policy changes. More end to end tests will be added in the future when the refresh authorization statement works properly. Testing: - Ran all FE tests - Ran all E2E authorization tests Change-Id: I6ee97bf41546d63385026c0e2b19545565402462 --- M common/thrift/JniCatalog.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/GrantRevokePrivStmt.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/NoneAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryImpaladAuthorizationManager.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java A fe/src/test/resources/xasecure-audit.xml M tests/authorization/test_grant_revoke.py A tests/authorization/test_ranger.py 15 files changed, 860 insertions(+), 227 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/12769/3 -- To view, visit http://gerrit.cloudera.org:8080/12769 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6ee97bf41546d63385026c0e2b19545565402462 Gerrit-Change-Number: 12769 Gerrit-PatchSet: 3 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12769 ) Change subject: IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/12769/4/tests/authorization/test_grant_revoke.py File tests/authorization/test_grant_revoke.py: http://gerrit.cloudera.org:8080/#/c/12769/4/tests/authorization/test_grant_revoke.py@369 PS4, Line 369: @ flake8: F811 redefinition of unused 'test_grant_revoke' from line 88 -- To view, visit http://gerrit.cloudera.org:8080/12769 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6ee97bf41546d63385026c0e2b19545565402462 Gerrit-Change-Number: 12769 Gerrit-PatchSet: 4 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 26 Mar 2019 18:20:59 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8362: Increase authorization test coverage with local catalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12859 ) Change subject: IMPALA-8362: Increase authorization test coverage with local catalog .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2543/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12859 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1efc56709bc3da16978589e20d7c611db4fa271a Gerrit-Change-Number: 12859 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 26 Mar 2019 18:24:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8143: Enhance DoRpcWithRetry().
Andrew Sherman has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/12672 ) Change subject: IMPALA-8143: Enhance DoRpcWithRetry(). .. IMPALA-8143: Enhance DoRpcWithRetry(). Allow callers of RpcMgr::DoRpcWithRetry to specify a time to sleep if the remote service is busy. Allow callers to specify a logging function that is called for rpc failures. Reviewers should note that the following changes to retry parameters were made: Cancel: add a 3 second sleep if service is busy. RemoteShutdown: add a 3 second sleep if service is busy. TESTING: Ran all end-to-end tests. Add a new test to rpc-mgr-test.cc which tests RpcMgr::DoRpcWithRetry in two ways. Firstly we test the case where the remote service is busy (which we force by filling up the queue), and secondly we exercise DoRpcWithRetry by using a fake Proxy that simulates failures. Clean up unused values of FaultInjectionUtil.RpcCallType and match values with test_rpc_timeout.py. Change-Id: Ia9693151c35e02235665b3c285a48c585973d390 --- M be/src/rpc/impala-service-pool.cc M be/src/rpc/impala-service-pool.h M be/src/rpc/rpc-mgr-test.cc M be/src/rpc/rpc-mgr-test.h M be/src/rpc/rpc-mgr.h M be/src/rpc/rpc-mgr.inline.h M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/query-state.h M be/src/service/client-request-state.cc M be/src/service/control-service.cc M be/src/service/control-service.h M be/src/testutil/fault-injection-util.h M tests/custom_cluster/test_rpc_timeout.py 13 files changed, 309 insertions(+), 58 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/12672/3 -- To view, visit http://gerrit.cloudera.org:8080/12672 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia9693151c35e02235665b3c285a48c585973d390 Gerrit-Change-Number: 12672 Gerrit-PatchSet: 3 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR] IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user
Austin Nobis has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/12769 ) Change subject: IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user .. IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user This patch adds support for GRANT privilege statements to USER and REVOKE privilege statements from USER. The RangerAuthorizationManager class has been created and will throw UnsupportedOperationException when an unimplemented method is called. The grammar has been updated to support FROM USER and TO USER for GRANT/REVOKE statements. Previously, privileges could be granted to a ROLE via GRANT/REVOKE statements even when the ROLE keyword was omitted. This is still the case for ROLE based authorization to preserve backward compatibility, but Ranger will throw an exception when a GRANT/REVOKE statement excludes the USER keyword. The syntax for the new statement is: GRANT ON TO USER REVOKE ON FROM USER Sentry does not support grant/revoke to/from user. Testing: An additional end to end test, test_ranger.py, was added. A single test was added that grants and revokes for a user and asserts permissions on a table. The test uses sleep statements to work with Ranger's polling interval for policy changes. More end to end tests will be added in the future when the refresh authorization statement works properly. - Ran all FE tests - Ran all E2E authorization tests Change-Id: I6ee97bf41546d63385026c0e2b19545565402462 --- M common/thrift/JniCatalog.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/GrantRevokePrivStmt.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/NoneAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryImpaladAuthorizationManager.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java A fe/src/test/resources/xasecure-audit.xml M tests/authorization/test_grant_revoke.py A tests/authorization/test_ranger.py 15 files changed, 860 insertions(+), 227 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/12769/4 -- To view, visit http://gerrit.cloudera.org:8080/12769 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6ee97bf41546d63385026c0e2b19545565402462 Gerrit-Change-Number: 12769 Gerrit-PatchSet: 4 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] Add missing sudo call to `service postgresql start`
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12724 ) Change subject: Add missing sudo call to `service postgresql start` .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12724 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie3859ad1308b7cd1d483d043b9f71d9f95abdf75 Gerrit-Change-Number: 12724 Gerrit-PatchSet: 2 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Tue, 26 Mar 2019 18:37:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user
Austin Nobis has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/12769 ) Change subject: IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user .. IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user This patch adds support for GRANT privilege statements to USER and REVOKE privilege statements from USER. The RangerAuthorizationManager class has been created and will throw UnsupportedOperationException when an unimplemented method is called. The grammar has been updated to support FROM USER and TO USER for GRANT/REVOKE statements. Previously, privileges could be granted to a ROLE via GRANT/REVOKE statements even when the ROLE keyword was omitted. This is still the case for ROLE based authorization to preserve backward compatibility, but Ranger will throw an exception when a GRANT/REVOKE statement excludes the USER keyword. The syntax for the new statement is: GRANT ON TO USER REVOKE ON FROM USER Sentry does not support grant/revoke to/from user. Testing: An additional end to end test, test_ranger.py, was added. A single test was added that grants and revokes for a user and asserts permissions on a table. The test uses sleep statements to work with Ranger's polling interval for policy changes. More end to end tests will be added in the future when the refresh authorization statement works properly. - Ran all FE tests - Ran all E2E authorization tests Change-Id: I6ee97bf41546d63385026c0e2b19545565402462 --- M common/thrift/JniCatalog.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/GrantRevokePrivStmt.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/NoneAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryImpaladAuthorizationManager.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java A fe/src/test/resources/xasecure-audit.xml A tests/authorization/test_ranger.py 14 files changed, 856 insertions(+), 227 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/12769/5 -- To view, visit http://gerrit.cloudera.org:8080/12769 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6ee97bf41546d63385026c0e2b19545565402462 Gerrit-Change-Number: 12769 Gerrit-PatchSet: 5 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8345 : Add option to set up minicluster to use Hive 3
Vihang Karajgaonkar has uploaded a new patch set (#6). (
http://gerrit.cloudera.org:8080/12846 )
Change subject: IMPALA-8345 : Add option to set up minicluster to use Hive 3
..
IMPALA-8345 : Add option to set up minicluster to use Hive 3
As a first step to integrate Impala with Hive 3.1.0 this patch modifies
the minicluster scripts to use Hive 3.1.0 instead of CDH Hive 2.1.1.
In order to make sure that existing setups don't break this is
enabled via a environment variable override to bin/impala-config.sh.
When the environment variable USE_CDP_HIVE is set to true the
bootstrap_toolchain script downloads Hive 3.1.0 tarballs and extracts it
in the toolchain directory. These binaries are used to start the Hive
services (Hiveserver2 and metastore). The default is still CDH Hive 2.1.1
Also, since Hive 3.1.0 uses a upgraded metastore schema, this patch
makes use of a different database name so that it is easy to switch from
working from one environment which uses Hive 2.1.1 metastore to another
which usese Hive 3.1.0 metastore.
In order to start a minicluster which uses Hive 3.1.0 users should
follow the steps below:
1. Make sure that minicluster, if running, is stopped
before you run the following commands.
2. Open a new terminal and run following commands.
> export USE_CDP_HIVE=true
> source bin/impala-config.sh
> bin/bootstrap_toolchain.py
The above command downloads the Hive 3.1.0 tarballs and extracts them
in toolchain/cdp_components-${CDP_BUILD_NUMBER} directory. This is a
no-op if the CDP_BUILD_NUMBER has not changed and if the cdp_components
are already downloaded by a previous invocation of the script.
> source bin/create-test-configuration.sh -create-metastore
The above step should provide "-create-metastore" only the first time
so that a new metastore db is created and the Hive 3.1.0 schema is
initialized. For all subsequent invocations, the "-create-metastore"
argument can be skipped. We should still source this script since the
hive-site.xml of Hive 3.1.0 is different than Hive 2.1.0 and
needs to be regenerated.
> testdata/bin/run-all.sh
Note that the testing was performed locally by downloading the Hive 3.1
binaries into
toolchain/cdp_components-976603/apache-hive-3.1.0.6.0.99.0-9-bin. Once
the binaries are available in S3 bucket, the bootstrap_toolchain script
should automatically do this for you.
Testing Done:
1. Made sure that the cluster comes up with Hive 3.1 when the steps
above are performed.
2. Made sure that existing scripts work as they do currently when
argument is not provided.
3. Impala cluster comes and connects to HMS 3.1.0 (Note that Impala
still uses Hive 2.1.1 client. Upgrading client libraries in Impala will
be done as a separate change)
Change-Id: Icfed856c1f5429ed45fd3d9cb08a5d1bb96a9605
---
M bin/bootstrap_toolchain.py
M bin/create-test-configuration.sh
M bin/impala-config.sh
A fe/src/test/resources/postgresql-hive-site.xml.cdp.template
M testdata/bin/run-hive-server.sh
5 files changed, 297 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/46/12846/6
--
To view, visit http://gerrit.cloudera.org:8080/12846
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icfed856c1f5429ed45fd3d9cb08a5d1bb96a9605
Gerrit-Change-Number: 12846
Gerrit-PatchSet: 6
Gerrit-Owner: Vihang Karajgaonkar
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8345 : Add option to set up minicluster to use Hive 3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12846 ) Change subject: IMPALA-8345 : Add option to set up minicluster to use Hive 3 .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/12846/6/bin/bootstrap_toolchain.py File bin/bootstrap_toolchain.py: http://gerrit.cloudera.org:8080/#/c/12846/6/bin/bootstrap_toolchain.py@432 PS6, Line 432: def download_cdp_hive(toolchain_root): flake8: E302 expected 2 blank lines, found 1 -- To view, visit http://gerrit.cloudera.org:8080/12846 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icfed856c1f5429ed45fd3d9cb08a5d1bb96a9605 Gerrit-Change-Number: 12846 Gerrit-PatchSet: 6 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 26 Mar 2019 18:44:54 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/12769 )
Change subject: IMPALA-8225: Add Ranger support for grant/revoke privilege
to/from user
..
Patch Set 5:
(18 comments)
http://gerrit.cloudera.org:8080/#/c/12769/4//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/12769/4//COMMIT_MSG@14
PS4, Line 14: ROLE via GRANT/REVOKE statements even
: when the ROLE keyword was omitted.
some example would be good
http://gerrit.cloudera.org:8080/#/c/12769/4//COMMIT_MSG@30
PS4, Line 30: works properly
nit: works properly with Ranger?
http://gerrit.cloudera.org:8080/#/c/12769/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/12769/5//COMMIT_MSG@26
PS5, Line 26: An additional end to end test, test_ranger.py, was added. A
single test
: was added that grants and revokes for a user and asserts
permissions on
: a table. The test uses sleep statements to work with Ranger's
polling
: interval for policy changes. More end to end tests will be added
in the
: future when the refresh authorization statement works properly.
nit: use a bullet point to be consistent with the other two bullet points
http://gerrit.cloudera.org:8080/#/c/12769/5//COMMIT_MSG@26
PS5, Line 26: An additional end to end test, test_ranger.py, was added. A
single test
: was added that grants and revokes for a user and asserts
permissions on
: a table. The test uses sleep statements to work with Ranger's
polling
: interval for policy changes. More end to end tests will be added
in the
: future when the refresh authorization statement works properly.
:
mention that AuthorizationStmtTest has been refactored to use the grant/revoke
methods from the AuthorizationManager to provide greater code coverage.
http://gerrit.cloudera.org:8080/#/c/12769/4/fe/src/main/java/org/apache/impala/authorization/NoneAuthorizationFactory.java
File
fe/src/main/java/org/apache/impala/authorization/NoneAuthorizationFactory.java:
http://gerrit.cloudera.org:8080/#/c/12769/4/fe/src/main/java/org/apache/impala/authorization/NoneAuthorizationFactory.java@127
PS4, Line 127: @Override
: public void grantPrivilegeToUser(User requestingUser,
TGrantRevokePrivParams params,
: TDdlExecResponse response) throws ImpalaException {
: }
:
: @Override
: public void revokePrivilegeFromUser(User requestingUser,
: TGrantRevokePrivParams params, TDdlExecResponse
response) throws ImpalaException {
: }
throw UnsupportedOperationException
http://gerrit.cloudera.org:8080/#/c/12769/4/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationManager.java
File
fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationManager.java:
http://gerrit.cloudera.org:8080/#/c/12769/4/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationManager.java@50
PS4, Line 50: RangerAuthorizationManager
I believe this is meant for Catalogd. We should rename this to
RangerCatalogdAuthorizationManager.
http://gerrit.cloudera.org:8080/#/c/12769/4/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationManager.java@53
PS4, Line 53:
: public RangerAuthorizationManager(
: Supplier authzChecker) {
: Preconditions.checkNotNull(authzChecker);
: Preconditions.checkArgument(authzChecker.get() instanceof
RangerAuthorizationChecker);
: auditHandler_ = new RangerDefaultAuditHandler();
: plugin_ = () -> ((RangerAuthorizationChecker)
authzChecker.get())
: .getRangerImpalaPlugin();
: }
:
: public RangerAuthorizationManager(RangerImpalaPlugin plugin) {
: auditHandler_ = new RangerDefaultAuditHandler();
: plugin_ = () -> plugin;
: }
I think we can just have one constructor.
public RangerAuthorizationManager(Supplier plugin)
http://gerrit.cloudera.org:8080/#/c/12769/5/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationManager.java
File
fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationManager.java:
http://gerrit.cloudera.org:8080/#/c/12769/5/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationManager.java@43
PS5, Line 43: An implementation of {@link AuthorizationManager} using Ranger.
update the comment that saying that it's used by catalogd
http://gerrit.cloudera.org:8080/#/c/12769/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/12769/4/fe/src/main/jav
[Impala-ASF-CR] IMPALA-8345 : Add option to set up minicluster to use Hive 3
Vihang Karajgaonkar has uploaded a new patch set (#7). (
http://gerrit.cloudera.org:8080/12846 )
Change subject: IMPALA-8345 : Add option to set up minicluster to use Hive 3
..
IMPALA-8345 : Add option to set up minicluster to use Hive 3
As a first step to integrate Impala with Hive 3.1.0 this patch modifies
the minicluster scripts to use Hive 3.1.0 instead of CDH Hive 2.1.1.
In order to make sure that existing setups don't break this is
enabled via a environment variable override to bin/impala-config.sh.
When the environment variable USE_CDP_HIVE is set to true the
bootstrap_toolchain script downloads Hive 3.1.0 tarballs and extracts it
in the toolchain directory. These binaries are used to start the Hive
services (Hiveserver2 and metastore). The default is still CDH Hive 2.1.1
Also, since Hive 3.1.0 uses a upgraded metastore schema, this patch
makes use of a different database name so that it is easy to switch from
working from one environment which uses Hive 2.1.1 metastore to another
which usese Hive 3.1.0 metastore.
In order to start a minicluster which uses Hive 3.1.0 users should
follow the steps below:
1. Make sure that minicluster, if running, is stopped
before you run the following commands.
2. Open a new terminal and run following commands.
> export USE_CDP_HIVE=true
> source bin/impala-config.sh
> bin/bootstrap_toolchain.py
The above command downloads the Hive 3.1.0 tarballs and extracts them
in toolchain/cdp_components-${CDP_BUILD_NUMBER} directory. This is a
no-op if the CDP_BUILD_NUMBER has not changed and if the cdp_components
are already downloaded by a previous invocation of the script.
> source bin/create-test-configuration.sh -create-metastore
The above step should provide "-create-metastore" only the first time
so that a new metastore db is created and the Hive 3.1.0 schema is
initialized. For all subsequent invocations, the "-create-metastore"
argument can be skipped. We should still source this script since the
hive-site.xml of Hive 3.1.0 is different than Hive 2.1.0 and
needs to be regenerated.
> testdata/bin/run-all.sh
Note that the testing was performed locally by downloading the Hive 3.1
binaries into
toolchain/cdp_components-976603/apache-hive-3.1.0.6.0.99.0-9-bin. Once
the binaries are available in S3 bucket, the bootstrap_toolchain script
should automatically do this for you.
Testing Done:
1. Made sure that the cluster comes up with Hive 3.1 when the steps
above are performed.
2. Made sure that existing scripts work as they do currently when
argument is not provided.
3. Impala cluster comes and connects to HMS 3.1.0 (Note that Impala
still uses Hive 2.1.1 client. Upgrading client libraries in Impala will
be done as a separate change)
Change-Id: Icfed856c1f5429ed45fd3d9cb08a5d1bb96a9605
---
M bin/bootstrap_toolchain.py
M bin/create-test-configuration.sh
M bin/impala-config.sh
A fe/src/test/resources/postgresql-hive-site.xml.cdp.template
M testdata/bin/run-hive-server.sh
5 files changed, 298 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/46/12846/7
--
To view, visit http://gerrit.cloudera.org:8080/12846
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icfed856c1f5429ed45fd3d9cb08a5d1bb96a9605
Gerrit-Change-Number: 12846
Gerrit-PatchSet: 7
Gerrit-Owner: Vihang Karajgaonkar
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8345 : Add option to set up minicluster to use Hive 3
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/12846 ) Change subject: IMPALA-8345 : Add option to set up minicluster to use Hive 3 .. Patch Set 7: (1 comment) http://gerrit.cloudera.org:8080/#/c/12846/6/bin/bootstrap_toolchain.py File bin/bootstrap_toolchain.py: http://gerrit.cloudera.org:8080/#/c/12846/6/bin/bootstrap_toolchain.py@432 PS6, Line 432: > flake8: E302 expected 2 blank lines, found 1 Done -- To view, visit http://gerrit.cloudera.org:8080/12846 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icfed856c1f5429ed45fd3d9cb08a5d1bb96a9605 Gerrit-Change-Number: 12846 Gerrit-PatchSet: 7 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 26 Mar 2019 18:56:28 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12769 ) Change subject: IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2544/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12769 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6ee97bf41546d63385026c0e2b19545565402462 Gerrit-Change-Number: 12769 Gerrit-PatchSet: 3 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 26 Mar 2019 19:03:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12769 ) Change subject: IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2547/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12769 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6ee97bf41546d63385026c0e2b19545565402462 Gerrit-Change-Number: 12769 Gerrit-PatchSet: 5 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 26 Mar 2019 19:04:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8143: Enhance DoRpcWithRetry().
Thomas Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/12672 ) Change subject: IMPALA-8143: Enhance DoRpcWithRetry(). .. Patch Set 3: Code-Review+1 (4 comments) Looks good, thanks for doing this. +1 for now to give Michael a change to take a look http://gerrit.cloudera.org:8080/#/c/12672/3/be/src/rpc/rpc-mgr.h File be/src/rpc/rpc-mgr.h: http://gerrit.cloudera.org:8080/#/c/12672/3/be/src/rpc/rpc-mgr.h@29 PS3, Line 29: using kudu::MonoDelta; please don't use 'using' in header files http://gerrit.cloudera.org:8080/#/c/12672/3/be/src/rpc/rpc-mgr.h@181 PS3, Line 181: noop maybe name this something more descriptive, eg. logging_noop() and/or mention in the comment that this is used as the logging function http://gerrit.cloudera.org:8080/#/c/12672/3/be/src/rpc/rpc-mgr.inline.h File be/src/rpc/rpc-mgr.inline.h: http://gerrit.cloudera.org:8080/#/c/12672/3/be/src/rpc/rpc-mgr.inline.h@30 PS3, Line 30: using kudu::MonoDelta; please don't use 'using' in header files http://gerrit.cloudera.org:8080/#/c/12672/3/be/src/rpc/rpc-mgr.inline.h@61 PS3, Line 61: using namespace std remove this -- To view, visit http://gerrit.cloudera.org:8080/12672 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia9693151c35e02235665b3c285a48c585973d390 Gerrit-Change-Number: 12672 Gerrit-PatchSet: 3 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 26 Mar 2019 19:04:28 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8345 : Add option to set up minicluster to use Hive 3
Andrew Sherman has posted comments on this change. (
http://gerrit.cloudera.org:8080/12846 )
Change subject: IMPALA-8345 : Add option to set up minicluster to use Hive 3
..
Patch Set 7: Code-Review+1
(5 comments)
a few nits but looks good
http://gerrit.cloudera.org:8080/#/c/12846/6//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/12846/6//COMMIT_MSG@10
PS6, Line 10: the minicluster scripts to use Hive 3.1.0 instead of CDH Hive
2.1.1.
Nit: to optionally use
http://gerrit.cloudera.org:8080/#/c/12846/6/bin/create-test-configuration.sh
File bin/create-test-configuration.sh:
http://gerrit.cloudera.org:8080/#/c/12846/6/bin/create-test-configuration.sh@132
PS6, Line 132: # Certain configurations (like
SentrySyncHMSNotificationsPostListener) does not work
s/does not/do not/
http://gerrit.cloudera.org:8080/#/c/12846/6/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/12846/6/bin/impala-config.sh@163
PS6, Line 163: export CDH_BUILD_NUMBER=909265
Nit: I wonder if CDP_BUILD_NUMBER should go here so that the 2 build numbers
are together
http://gerrit.cloudera.org:8080/#/c/12846/6/bin/impala-config.sh@534
PS6, Line 534: export
HIVE_HOME="$CDP_COMPONENTS_HOME/apache-hive-${IMPALA_HIVE_VERSION}-bin"
Why in one case is the home apache-hive-xxx and in the other it is hive-xxx ?
http://gerrit.cloudera.org:8080/#/c/12846/6/bin/impala-config.sh@748
PS6, Line 748: echo "CDP_BUILD_NUMBER= $CDP_BUILD_NUMBER"
It could be confusing that you always echo CDH_BUILD_NUMBER but you only echo
CDP_BUILD_NUMBER when doing a CDP build.
--
To view, visit http://gerrit.cloudera.org:8080/12846
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icfed856c1f5429ed45fd3d9cb08a5d1bb96a9605
Gerrit-Change-Number: 12846
Gerrit-PatchSet: 7
Gerrit-Owner: Vihang Karajgaonkar
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Tue, 26 Mar 2019 19:12:08 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8143: Enhance DoRpcWithRetry().
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12672 ) Change subject: IMPALA-8143: Enhance DoRpcWithRetry(). .. Patch Set 3: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/2546/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/12672 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia9693151c35e02235665b3c285a48c585973d390 Gerrit-Change-Number: 12672 Gerrit-PatchSet: 3 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 26 Mar 2019 19:06:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12769 ) Change subject: IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/12769/3/tests/authorization/test_grant_revoke.py File tests/authorization/test_grant_revoke.py: http://gerrit.cloudera.org:8080/#/c/12769/3/tests/authorization/test_grant_revoke.py@369 PS3, Line 369: @ flake8: F811 redefinition of unused 'test_grant_revoke' from line 88 -- To view, visit http://gerrit.cloudera.org:8080/12769 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6ee97bf41546d63385026c0e2b19545565402462 Gerrit-Change-Number: 12769 Gerrit-PatchSet: 3 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 26 Mar 2019 18:20:19 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12769 ) Change subject: IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2545/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12769 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6ee97bf41546d63385026c0e2b19545565402462 Gerrit-Change-Number: 12769 Gerrit-PatchSet: 4 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 26 Mar 2019 19:08:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8345 : Add option to set up minicluster to use Hive 3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12846 ) Change subject: IMPALA-8345 : Add option to set up minicluster to use Hive 3 .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2548/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12846 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icfed856c1f5429ed45fd3d9cb08a5d1bb96a9605 Gerrit-Change-Number: 12846 Gerrit-PatchSet: 6 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 26 Mar 2019 19:27:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8345 : Add option to set up minicluster to use Hive 3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12846 ) Change subject: IMPALA-8345 : Add option to set up minicluster to use Hive 3 .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2549/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12846 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icfed856c1f5429ed45fd3d9cb08a5d1bb96a9605 Gerrit-Change-Number: 12846 Gerrit-PatchSet: 7 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 26 Mar 2019 19:28:34 + Gerrit-HasComments: No
[Impala-ASF-CR] Move DEBIAN FRONTEND=noninteractive variable to the ubuntu section of the script
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12723 ) Change subject: Move DEBIAN_FRONTEND=noninteractive variable to the ubuntu section of the script .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3958/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/12723 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifab2f56d3e119e4e73685c3cbd5249760ceb33ac Gerrit-Change-Number: 12723 Gerrit-PatchSet: 2 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Tue, 26 Mar 2019 20:14:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8345 : Add option to set up minicluster to use Hive 3
Vihang Karajgaonkar has uploaded a new patch set (#9). (
http://gerrit.cloudera.org:8080/12846 )
Change subject: IMPALA-8345 : Add option to set up minicluster to use Hive 3
..
IMPALA-8345 : Add option to set up minicluster to use Hive 3
As a first step to integrate Impala with Hive 3.1.0 this patch modifies
the minicluster scripts to optionally use Hive 3.1.0 instead of
CDH Hive 2.1.1.
In order to make sure that existing setups don't break this is
enabled via a environment variable override to bin/impala-config.sh.
When the environment variable USE_CDP_HIVE is set to true the
bootstrap_toolchain script downloads Hive 3.1.0 tarballs and extracts it
in the toolchain directory. These binaries are used to start the Hive
services (Hiveserver2 and metastore). The default is still CDH Hive 2.1.1
Also, since Hive 3.1.0 uses a upgraded metastore schema, this patch
makes use of a different database name so that it is easy to switch from
working from one environment which uses Hive 2.1.1 metastore to another
which usese Hive 3.1.0 metastore.
In order to start a minicluster which uses Hive 3.1.0 users should
follow the steps below:
1. Make sure that minicluster, if running, is stopped
before you run the following commands.
2. Open a new terminal and run following commands.
> export USE_CDP_HIVE=true
> source bin/impala-config.sh
> bin/bootstrap_toolchain.py
The above command downloads the Hive 3.1.0 tarballs and extracts them
in toolchain/cdp_components-${CDP_BUILD_NUMBER} directory. This is a
no-op if the CDP_BUILD_NUMBER has not changed and if the cdp_components
are already downloaded by a previous invocation of the script.
> source bin/create-test-configuration.sh -create-metastore
The above step should provide "-create-metastore" only the first time
so that a new metastore db is created and the Hive 3.1.0 schema is
initialized. For all subsequent invocations, the "-create-metastore"
argument can be skipped. We should still source this script since the
hive-site.xml of Hive 3.1.0 is different than Hive 2.1.0 and
needs to be regenerated.
> testdata/bin/run-all.sh
Note that the testing was performed locally by downloading the Hive 3.1
binaries into
toolchain/cdp_components-976603/apache-hive-3.1.0.6.0.99.0-9-bin. Once
the binaries are available in S3 bucket, the bootstrap_toolchain script
should automatically do this for you.
Testing Done:
1. Made sure that the cluster comes up with Hive 3.1 when the steps
above are performed.
2. Made sure that existing scripts work as they do currently when
argument is not provided.
3. Impala cluster comes and connects to HMS 3.1.0 (Note that Impala
still uses Hive 2.1.1 client. Upgrading client libraries in Impala will
be done as a separate change)
Change-Id: Icfed856c1f5429ed45fd3d9cb08a5d1bb96a9605
---
M bin/bootstrap_toolchain.py
M bin/create-test-configuration.sh
M bin/impala-config.sh
A fe/src/test/resources/postgresql-hive-site.xml.cdp.template
M testdata/bin/run-hive-server.sh
5 files changed, 303 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/46/12846/9
--
To view, visit http://gerrit.cloudera.org:8080/12846
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icfed856c1f5429ed45fd3d9cb08a5d1bb96a9605
Gerrit-Change-Number: 12846
Gerrit-PatchSet: 9
Gerrit-Owner: Vihang Karajgaonkar
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8307 Use test -x to check for ntp-wait
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12726 ) Change subject: IMPALA-8307 Use test -x to check for ntp-wait .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3959/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/12726 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I53c63dfa651ac242050171da70540d24c4caf32c Gerrit-Change-Number: 12726 Gerrit-PatchSet: 2 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Tue, 26 Mar 2019 20:15:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7800: Reject new connections after --fe service threads
Hello Michael Ho, Andrew Sherman, Thomas Marshall, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12579 to look at the new patch set (#10). Change subject: IMPALA-7800: Reject new connections after --fe_service_threads .. IMPALA-7800: Reject new connections after --fe_service_threads The current implementation of the FE thrift server waits indefinitely to open the new session, if the maximum number of FE service threads specified by --fe_service_threads has been allocated. This patch introduces a startup flag to control how the server should treat new connection requests if we have run out of the configured number of server threads. If --accepted_cnxn_timeout > 0, new connection requests are rejected by the server if we can't get a server thread within the specified timeout. We set the default timeout to be 5 minutes. The old behavior can be restored by setting --accepted_cnxn_timeout=0, i.e., no timeout. The timeout applies only to client facing thrift servers, i.e., HS2 and Beeswax servers. Testing: Added a new custom cluster test suite to exercise the new code. Ran core and exhaustive tests. Change-Id: Idb345c1d84cc2f691f54ded467f253e758f87e64 --- M be/src/rpc/TAcceptQueueServer.cpp M be/src/rpc/TAcceptQueueServer.h M be/src/rpc/thrift-server.cc M be/src/rpc/thrift-server.h M be/src/service/impala-server.cc M common/thrift/metrics.json A tests/custom_cluster/test_frontend_connection_limit.py 7 files changed, 248 insertions(+), 36 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/12579/10 -- To view, visit http://gerrit.cloudera.org:8080/12579 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Idb345c1d84cc2f691f54ded467f253e758f87e64 Gerrit-Change-Number: 12579 Gerrit-PatchSet: 10 Gerrit-Owner: Zoram Thanga Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Zoram Thanga
[Impala-ASF-CR] IMPALA-7800: Reject new connections after --fe service threads
Zoram Thanga has posted comments on this change. ( http://gerrit.cloudera.org:8080/12579 ) Change subject: IMPALA-7800: Reject new connections after --fe_service_threads .. Patch Set 10: Code-Review+1 Carrying Andrew's +1 -- To view, visit http://gerrit.cloudera.org:8080/12579 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idb345c1d84cc2f691f54ded467f253e758f87e64 Gerrit-Change-Number: 12579 Gerrit-PatchSet: 10 Gerrit-Owner: Zoram Thanga Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Tue, 26 Mar 2019 20:20:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8345 : Add option to set up minicluster to use Hive 3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12846 ) Change subject: IMPALA-8345 : Add option to set up minicluster to use Hive 3 .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2550/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12846 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icfed856c1f5429ed45fd3d9cb08a5d1bb96a9605 Gerrit-Change-Number: 12846 Gerrit-PatchSet: 9 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 26 Mar 2019 20:55:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7800: Reject new connections after --fe service threads
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12579 ) Change subject: IMPALA-7800: Reject new connections after --fe_service_threads .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2551/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12579 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idb345c1d84cc2f691f54ded467f253e758f87e64 Gerrit-Change-Number: 12579 Gerrit-PatchSet: 10 Gerrit-Owner: Zoram Thanga Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Tue, 26 Mar 2019 20:59:16 + Gerrit-HasComments: No
[Impala-ASF-CR](asf-site) [DOCS] Impala 3.2 docs to be published
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12860 ) Change subject: [DOCS] Impala 3.2 docs to be published .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/289/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/12860 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: I3564270ab8488cd3482c40070f756e6d4b2a056e Gerrit-Change-Number: 12860 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 26 Mar 2019 21:05:37 + Gerrit-HasComments: No
[native-toolchain-CR] Parallelize image building process
Hector Acosta has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12864 Change subject: Parallelize image building process .. Parallelize image building process Images can now be built in parallel when using the new buildall.py script. Other than the parallelism and logging changes, the script behaves the same way. Change-Id: I526b0088020bb7c647f1918b3eb3258cc1e3ddf9 --- M docker/README.md A docker/buildall.py D docker/buildall.sh 3 files changed, 57 insertions(+), 39 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/64/12864/1 -- To view, visit http://gerrit.cloudera.org:8080/12864 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I526b0088020bb7c647f1918b3eb3258cc1e3ddf9 Gerrit-Change-Number: 12864 Gerrit-PatchSet: 1 Gerrit-Owner: Hector Acosta
[native-toolchain-CR] Install newer git version in centos 6.6
Hector Acosta has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12862 Change subject: Install newer git version in centos 6.6 .. Install newer git version in centos 6.6 Centos 6.6 contains a git version that doesn't work over dumb-http. To fix it we include a new Centos-Vault repo file with disabled repositories and explicitly install git from centos 6.9, which works. Change-Id: I66549477ec9378faeda46a98cf6556a3207979eb --- M docker/all/assert-dependencies-present.py A docker/redhat/Centos6-Vault.repo M docker/redhat6.df 3 files changed, 389 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/62/12862/1 -- To view, visit http://gerrit.cloudera.org:8080/12862 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I66549477ec9378faeda46a98cf6556a3207979eb Gerrit-Change-Number: 12862 Gerrit-PatchSet: 1 Gerrit-Owner: Hector Acosta
[native-toolchain-CR] Fix debian:7 and debian:8 image builds
Hector Acosta has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12863 Change subject: Fix debian:7 and debian:8 image builds .. Fix debian:7 and debian:8 image builds Wheezy and Jessie were moved out of the main debian mirrors: https://lists.debian.org/debian-devel-announce/2019/03/msg6.html So we're fetching from archive now. Change-Id: Id85298dc83e419aa8fbff6bd89b13f446adc93a8 --- A docker/debian/7/sources.list A docker/debian/8/sources.list M docker/debian7.df M docker/debian8.df 4 files changed, 14 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/63/12863/1 -- To view, visit http://gerrit.cloudera.org:8080/12863 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Id85298dc83e419aa8fbff6bd89b13f446adc93a8 Gerrit-Change-Number: 12863 Gerrit-PatchSet: 1 Gerrit-Owner: Hector Acosta
[native-toolchain-CR] Prepend docker image to both stderr and stdout
Hector Acosta has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12861 Change subject: Prepend docker image to both stderr and stdout .. Prepend docker image to both stderr and stdout Sometimes, it may be hard to debug which distro produced an error, this commit ensures that both stderr and stdout get piped through sed. Change-Id: I16dd725b0c98599867892801c7146c4fe60f46e1 --- M Makefile 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/61/12861/1 -- To view, visit http://gerrit.cloudera.org:8080/12861 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I16dd725b0c98599867892801c7146c4fe60f46e1 Gerrit-Change-Number: 12861 Gerrit-PatchSet: 1 Gerrit-Owner: Hector Acosta
[Impala-ASF-CR](asf-site) [DOCS] Impala 3.2 docs to be published
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12860 Change subject: [DOCS] Impala 3.2 docs to be published .. [DOCS] Impala 3.2 docs to be published Add Impala docs from branch master, commit hash 36a67c467dbaf713f19c9423a9b4de1f0fc7a5a0 Change-Id: I3564270ab8488cd3482c40070f756e6d4b2a056e --- M docs/build/html/index.html M docs/build/html/topics/impala_abort_on_error.html M docs/build/html/topics/impala_adls.html M docs/build/html/topics/impala_admin.html M docs/build/html/topics/impala_admission.html A docs/build/html/topics/impala_admission_config.html M docs/build/html/topics/impala_aggregate_functions.html M docs/build/html/topics/impala_aliases.html M docs/build/html/topics/impala_allow_erasure_coded_files.html M docs/build/html/topics/impala_allow_unsupported_formats.html M docs/build/html/topics/impala_alter_database.html M docs/build/html/topics/impala_alter_table.html M docs/build/html/topics/impala_alter_view.html M docs/build/html/topics/impala_analytic_functions.html M docs/build/html/topics/impala_appx_count_distinct.html M docs/build/html/topics/impala_appx_median.html M docs/build/html/topics/impala_array.html M docs/build/html/topics/impala_auditing.html M docs/build/html/topics/impala_authentication.html M docs/build/html/topics/impala_authorization.html M docs/build/html/topics/impala_avg.html M docs/build/html/topics/impala_avro.html M docs/build/html/topics/impala_batch_size.html M docs/build/html/topics/impala_bigint.html M docs/build/html/topics/impala_bit_functions.html M docs/build/html/topics/impala_boolean.html M docs/build/html/topics/impala_breakpad.html M docs/build/html/topics/impala_buffer_pool_limit.html M docs/build/html/topics/impala_char.html M docs/build/html/topics/impala_comment.html M docs/build/html/topics/impala_comments.html M docs/build/html/topics/impala_complex_types.html M docs/build/html/topics/impala_components.html M docs/build/html/topics/impala_compression_codec.html M docs/build/html/topics/impala_compute_stats.html M docs/build/html/topics/impala_compute_stats_min_sample_size.html M docs/build/html/topics/impala_concepts.html M docs/build/html/topics/impala_conditional_functions.html M docs/build/html/topics/impala_config.html M docs/build/html/topics/impala_config_options.html M docs/build/html/topics/impala_config_performance.html M docs/build/html/topics/impala_connecting.html M docs/build/html/topics/impala_conversion_functions.html M docs/build/html/topics/impala_count.html M docs/build/html/topics/impala_create_database.html M docs/build/html/topics/impala_create_function.html M docs/build/html/topics/impala_create_role.html M docs/build/html/topics/impala_create_table.html M docs/build/html/topics/impala_create_view.html M docs/build/html/topics/impala_custom_timezones.html M docs/build/html/topics/impala_databases.html M docs/build/html/topics/impala_datatypes.html M docs/build/html/topics/impala_datetime_functions.html M docs/build/html/topics/impala_ddl.html M docs/build/html/topics/impala_debug_action.html M docs/build/html/topics/impala_decimal.html M docs/build/html/topics/impala_decimal_v2.html M docs/build/html/topics/impala_dedicated_coordinator.html A docs/build/html/topics/impala_default_file_format.html M docs/build/html/topics/impala_default_join_distribution_mode.html M docs/build/html/topics/impala_default_spillable_buffer_size.html M docs/build/html/topics/impala_delegation.html M docs/build/html/topics/impala_delete.html M docs/build/html/topics/impala_describe.html M docs/build/html/topics/impala_development.html M docs/build/html/topics/impala_disable_codegen.html M docs/build/html/topics/impala_disable_codegen_rows_threshold.html M docs/build/html/topics/impala_disable_row_runtime_filtering.html M docs/build/html/topics/impala_disable_streaming_preaggregations.html M docs/build/html/topics/impala_disable_unsafe_spills.html M docs/build/html/topics/impala_disk_space.html M docs/build/html/topics/impala_distinct.html M docs/build/html/topics/impala_dml.html M docs/build/html/topics/impala_double.html M docs/build/html/topics/impala_drop_database.html M docs/build/html/topics/impala_drop_function.html M docs/build/html/topics/impala_drop_role.html M docs/build/html/topics/impala_drop_stats.html M docs/build/html/topics/impala_drop_table.html M docs/build/html/topics/impala_drop_view.html M docs/build/html/topics/impala_enable_expr_rewrites.html M docs/build/html/topics/impala_exec_single_node_rows_threshold.html M docs/build/html/topics/impala_exec_time_limit_s.html M docs/build/html/topics/impala_explain.html M docs/build/html/topics/impala_explain_level.html M docs/build/html/topics/impala_explain_plan.html M docs/build/html/topics/impala_faq.html M docs/build/html/topics/impala_file_formats.html M docs/build/html/topics/impala_fixed_issues.html M docs/build/html/topics/impala_float.html M docs/build/html/topics/impala
[Impala-ASF-CR](asf-site) [DOCS] Impala 3.2 docs to be published
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12860 ) Change subject: [DOCS] Impala 3.2 docs to be published .. Patch Set 1: Verified-1 Build Failed https://jenkins.impala.io/job/gerrit-docs-auto-test/289/ : Doc tests failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/12860 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: I3564270ab8488cd3482c40070f756e6d4b2a056e Gerrit-Change-Number: 12860 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Comment-Date: Tue, 26 Mar 2019 21:15:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8143: Enhance DoRpcWithRetry().
Andrew Sherman has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/12672 ) Change subject: IMPALA-8143: Enhance DoRpcWithRetry(). .. IMPALA-8143: Enhance DoRpcWithRetry(). Allow callers of RpcMgr::DoRpcWithRetry to specify a time to sleep if the remote service is busy. Allow callers to specify a logging function that is called for rpc failures. Reviewers should note that the following changes to retry parameters were made: Cancel: add a 3 second sleep if service is busy. RemoteShutdown: add a 3 second sleep if service is busy. TESTING: Ran all end-to-end tests. Add a new test to rpc-mgr-test.cc which tests RpcMgr::DoRpcWithRetry in two ways. Firstly we test the case where the remote service is busy (which we force by filling up the queue), and secondly we exercise DoRpcWithRetry by using a fake Proxy that simulates failures. Clean up unused values of FaultInjectionUtil.RpcCallType and match values with test_rpc_timeout.py. Change-Id: Ia9693151c35e02235665b3c285a48c585973d390 --- M be/src/rpc/impala-service-pool.cc M be/src/rpc/impala-service-pool.h M be/src/rpc/rpc-mgr-test.cc M be/src/rpc/rpc-mgr-test.h M be/src/rpc/rpc-mgr.h M be/src/rpc/rpc-mgr.inline.h M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/query-state.h M be/src/service/client-request-state.cc M be/src/service/control-service.cc M be/src/service/control-service.h M be/src/testutil/fault-injection-util.h M tests/custom_cluster/test_rpc_timeout.py 13 files changed, 303 insertions(+), 58 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/12672/4 -- To view, visit http://gerrit.cloudera.org:8080/12672 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia9693151c35e02235665b3c285a48c585973d390 Gerrit-Change-Number: 12672 Gerrit-PatchSet: 4 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR] IMPALA-8143: Enhance DoRpcWithRetry().
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/12672 ) Change subject: IMPALA-8143: Enhance DoRpcWithRetry(). .. Patch Set 3: (4 comments) Thanks for the quick review http://gerrit.cloudera.org:8080/#/c/12672/3/be/src/rpc/rpc-mgr.h File be/src/rpc/rpc-mgr.h: http://gerrit.cloudera.org:8080/#/c/12672/3/be/src/rpc/rpc-mgr.h@29 PS3, Line 29: using kudu::MonoDelta; > please don't use 'using' in header files Done http://gerrit.cloudera.org:8080/#/c/12672/3/be/src/rpc/rpc-mgr.h@181 PS3, Line 181: noop > maybe name this something more descriptive, eg. logging_noop() and/or menti Done http://gerrit.cloudera.org:8080/#/c/12672/3/be/src/rpc/rpc-mgr.inline.h File be/src/rpc/rpc-mgr.inline.h: http://gerrit.cloudera.org:8080/#/c/12672/3/be/src/rpc/rpc-mgr.inline.h@30 PS3, Line 30: using kudu::MonoDelta; > please don't use 'using' in header files Done http://gerrit.cloudera.org:8080/#/c/12672/3/be/src/rpc/rpc-mgr.inline.h@61 PS3, Line 61: using namespace std > remove this Done -- To view, visit http://gerrit.cloudera.org:8080/12672 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia9693151c35e02235665b3c285a48c585973d390 Gerrit-Change-Number: 12672 Gerrit-PatchSet: 3 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 26 Mar 2019 21:43:23 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8143: Enhance DoRpcWithRetry().
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12672 ) Change subject: IMPALA-8143: Enhance DoRpcWithRetry(). .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/12672/4/be/src/rpc/rpc-mgr.inline.h File be/src/rpc/rpc-mgr.inline.h: http://gerrit.cloudera.org:8080/#/c/12672/4/be/src/rpc/rpc-mgr.inline.h@56 PS4, Line 56: const int times_to_try, const kudu::MonoDelta& timeout, const int server_busy_backoff_s, line too long (92 > 90) -- To view, visit http://gerrit.cloudera.org:8080/12672 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia9693151c35e02235665b3c285a48c585973d390 Gerrit-Change-Number: 12672 Gerrit-PatchSet: 4 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 26 Mar 2019 21:44:00 + Gerrit-HasComments: Yes
[native-toolchain-CR] Install newer git version in centos 6.6
Hector Acosta has abandoned this change. ( http://gerrit.cloudera.org:8080/12862 ) Change subject: Install newer git version in centos 6.6 .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/12862 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I66549477ec9378faeda46a98cf6556a3207979eb Gerrit-Change-Number: 12862 Gerrit-PatchSet: 1 Gerrit-Owner: Hector Acosta
[Impala-ASF-CR] IMPALA-8143: Enhance DoRpcWithRetry().
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/12672 ) Change subject: IMPALA-8143: Enhance DoRpcWithRetry(). .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/12672/4/be/src/rpc/rpc-mgr.inline.h File be/src/rpc/rpc-mgr.inline.h: http://gerrit.cloudera.org:8080/#/c/12672/4/be/src/rpc/rpc-mgr.inline.h@56 PS4, Line 56: const int times_to_try, const kudu::MonoDelta& timeout, const int server_busy_backoff_s, > line too long (92 > 90) Done -- To view, visit http://gerrit.cloudera.org:8080/12672 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia9693151c35e02235665b3c285a48c585973d390 Gerrit-Change-Number: 12672 Gerrit-PatchSet: 4 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 26 Mar 2019 21:51:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8143: Enhance DoRpcWithRetry().
Andrew Sherman has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/12672 ) Change subject: IMPALA-8143: Enhance DoRpcWithRetry(). .. IMPALA-8143: Enhance DoRpcWithRetry(). Allow callers of RpcMgr::DoRpcWithRetry to specify a time to sleep if the remote service is busy. Allow callers to specify a logging function that is called for rpc failures. Reviewers should note that the following changes to retry parameters were made: Cancel: add a 3 second sleep if service is busy. RemoteShutdown: add a 3 second sleep if service is busy. TESTING: Ran all end-to-end tests. Add a new test to rpc-mgr-test.cc which tests RpcMgr::DoRpcWithRetry in two ways. Firstly we test the case where the remote service is busy (which we force by filling up the queue), and secondly we exercise DoRpcWithRetry by using a fake Proxy that simulates failures. Clean up unused values of FaultInjectionUtil.RpcCallType and match values with test_rpc_timeout.py. Change-Id: Ia9693151c35e02235665b3c285a48c585973d390 --- M be/src/rpc/impala-service-pool.cc M be/src/rpc/impala-service-pool.h M be/src/rpc/rpc-mgr-test.cc M be/src/rpc/rpc-mgr-test.h M be/src/rpc/rpc-mgr.h M be/src/rpc/rpc-mgr.inline.h M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/query-state.h M be/src/service/client-request-state.cc M be/src/service/control-service.cc M be/src/service/control-service.h M be/src/testutil/fault-injection-util.h M tests/custom_cluster/test_rpc_timeout.py 13 files changed, 302 insertions(+), 58 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/12672/5 -- To view, visit http://gerrit.cloudera.org:8080/12672 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia9693151c35e02235665b3c285a48c585973d390 Gerrit-Change-Number: 12672 Gerrit-PatchSet: 5 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR] IMPALA-7800: Reject new connections after --fe service threads
Michael Ho has posted comments on this change. (
http://gerrit.cloudera.org:8080/12579 )
Change subject: IMPALA-7800: Reject new connections after --fe_service_threads
..
Patch Set 10:
(9 comments)
LGTM. Mostly nits.
http://gerrit.cloudera.org:8080/#/c/12579/10//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/12579/10//COMMIT_MSG@7
PS10, Line 7: Reject new connections after --fe_service_threads
Does this need to be updated too ?
http://gerrit.cloudera.org:8080/#/c/12579/10/be/src/rpc/TAcceptQueueServer.h
File be/src/rpc/TAcceptQueueServer.h:
http://gerrit.cloudera.org:8080/#/c/12579/10/be/src/rpc/TAcceptQueueServer.h@65
PS10, Line 65: timeout
nit: timeout_ms
http://gerrit.cloudera.org:8080/#/c/12579/10/be/src/rpc/TAcceptQueueServer.h@116
PS10, Line 116: /// Amount of time after which a connection request will be
timed out.
in milliseconds
http://gerrit.cloudera.org:8080/#/c/12579/10/be/src/rpc/TAcceptQueueServer.cpp
File be/src/rpc/TAcceptQueueServer.cpp:
http://gerrit.cloudera.org:8080/#/c/12579/10/be/src/rpc/TAcceptQueueServer.cpp@263
PS10, Line 263: if (queue_timeout_ms_ != 0)
: entry->expiration_time_ = MonotonicMillis() +
queue_timeout_ms_;
May need to guard against negative value although it's user's fault in that
case.
if (queue_timeout_ms > 0) {
}
http://gerrit.cloudera.org:8080/#/c/12579/10/be/src/rpc/thrift-server.h
File be/src/rpc/thrift-server.h:
http://gerrit.cloudera.org:8080/#/c/12579/10/be/src/rpc/thrift-server.h@149
PS10, Line 149: int64_t queue_timeout_ms = 0);
May make sense to document the meaning of this argument too.
nit: this line seems to fit into the previous line too.
http://gerrit.cloudera.org:8080/#/c/12579/10/be/src/rpc/thrift-server.h@197
PS10, Line 197: /// Amount of time an accepted client connection will be kept
in the accept
: /// queue before it is timed out. Used in TAcceptQueueServer.
May want to document that if it's 0, it means there is no timeout (similar to
max_concurrent_connections_ above).
http://gerrit.cloudera.org:8080/#/c/12579/10/be/src/rpc/thrift-server.h@277
PS10, Line 277: }
nit: missing blank line after this function.
http://gerrit.cloudera.org:8080/#/c/12579/10/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/12579/10/be/src/service/impala-server.cc@239
PS10, Line 239: accepted_cnxn_timeout
Should this have a more specific name such as "accepted_client_cnxn_timeout" as
we use Thrift server for various internal services too ?
http://gerrit.cloudera.org:8080/#/c/12579/10/tests/custom_cluster/test_frontend_connection_limit.py
File tests/custom_cluster/test_frontend_connection_limit.py:
http://gerrit.cloudera.org:8080/#/c/12579/10/tests/custom_cluster/test_frontend_connection_limit.py@90
PS10, Line 90: except Exception as e:
How do we catch cases in which the test passes without raising the exception ?
--
To view, visit http://gerrit.cloudera.org:8080/12579
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb345c1d84cc2f691f54ded467f253e758f87e64
Gerrit-Change-Number: 12579
Gerrit-PatchSet: 10
Gerrit-Owner: Zoram Thanga
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Thomas Marshall
Gerrit-Reviewer: Zoram Thanga
Gerrit-Comment-Date: Tue, 26 Mar 2019 22:05:22 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8143: Enhance DoRpcWithRetry().
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12672 ) Change subject: IMPALA-8143: Enhance DoRpcWithRetry(). .. Patch Set 4: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/2552/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/12672 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia9693151c35e02235665b3c285a48c585973d390 Gerrit-Change-Number: 12672 Gerrit-PatchSet: 4 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 26 Mar 2019 22:27:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8143: Enhance DoRpcWithRetry().
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12672 ) Change subject: IMPALA-8143: Enhance DoRpcWithRetry(). .. Patch Set 5: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/2553/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/12672 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia9693151c35e02235665b3c285a48c585973d390 Gerrit-Change-Number: 12672 Gerrit-PatchSet: 5 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 26 Mar 2019 22:31:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-2990: timeout unresponsive queries in coordinator
Hello Michael Ho, Philip Zeyliger, Todd Lipcon, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12299 to look at the new patch set (#7). Change subject: IMPALA-2990: timeout unresponsive queries in coordinator .. IMPALA-2990: timeout unresponsive queries in coordinator The coordinator currently waits indefinitely if it does not receive a status report from a backend. This could cause a query to hang indefinitely in certain situations, for example if the backend decides to cancel itself as a result of failed status report rpcs. This patch adds a thread to ImpalaServer which periodically iterates over all queries for which that server is the coordinator and cancels any that haven't had a report from a backend in a certain amount of time. The timeout is calculated as the longest a backend will attempt to retry sending status reports before giving up and cancelling itself. With the default flags, this timeout is about 15 minutes. The thread wakes up at an interval of the calculated timeout + 10% TODO: - Write functional tests once the appropriate mechanisms are in place to simulate errors (IMPALA-8138) Testing: Ran tests on a 10 node cluster loaded with tpch 500: - Ran the stress test for 1000 queries with the debug actions: 'REPORT_EXEC_STATUS_SEND:FAIL@0.1|REPORT_EXEC_STATUS_RECV:FAIL@0.1' Prior to this patch, this setup results in hanging queries. With this patch, no hangs were observed. - Ran perf tests with 4 concurrent streams, 3 iterations per query. Found no change in performance. Change-Id: I196c8c6a5633b1960e2c3a3884777be9b3824987 --- M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/runtime/query-state.cc M be/src/runtime/query-state.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M common/thrift/ImpalaInternalService.thrift M common/thrift/generate_error_codes.py 10 files changed, 155 insertions(+), 41 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/12299/7 -- To view, visit http://gerrit.cloudera.org:8080/12299 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I196c8c6a5633b1960e2c3a3884777be9b3824987 Gerrit-Change-Number: 12299 Gerrit-PatchSet: 7 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8362: Increase authorization test coverage with local catalog
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12859 ) Change subject: IMPALA-8362: Increase authorization test coverage with local catalog .. IMPALA-8362: Increase authorization test coverage with local catalog This patch adds authorization test that runs with local catalog by reusing the existing grant and revoke tests. Testing: - Added a new authorization test with local catalog - Ran all authorization tests Change-Id: I1efc56709bc3da16978589e20d7c611db4fa271a Reviewed-on: http://gerrit.cloudera.org:8080/12859 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M tests/authorization/test_grant_revoke.py 1 file changed, 20 insertions(+), 5 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/12859 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I1efc56709bc3da16978589e20d7c611db4fa271a Gerrit-Change-Number: 12859 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8143: Enhance DoRpcWithRetry().
Andrew Sherman has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/12672 ) Change subject: IMPALA-8143: Enhance DoRpcWithRetry(). .. IMPALA-8143: Enhance DoRpcWithRetry(). Allow callers of RpcMgr::DoRpcWithRetry to specify a time to sleep if the remote service is busy. Allow callers to specify a logging function that is called for rpc failures. Reviewers should note that the following changes to retry parameters were made: Cancel: add a 3 second sleep if service is busy. RemoteShutdown: add a 3 second sleep if service is busy. TESTING: Ran all end-to-end tests. Add a new test to rpc-mgr-test.cc which tests RpcMgr::DoRpcWithRetry in two ways. Firstly we test the case where the remote service is busy (which we force by filling up the queue), and secondly we exercise DoRpcWithRetry by using a fake Proxy that simulates failures. Clean up unused values of FaultInjectionUtil.RpcCallType and match values with test_rpc_timeout.py. Change-Id: Ia9693151c35e02235665b3c285a48c585973d390 --- M be/src/rpc/impala-service-pool.cc M be/src/rpc/impala-service-pool.h M be/src/rpc/rpc-mgr-test.cc M be/src/rpc/rpc-mgr-test.h M be/src/rpc/rpc-mgr.h M be/src/rpc/rpc-mgr.inline.h M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/query-state.h M be/src/service/client-request-state.cc M be/src/service/control-service.cc M be/src/service/control-service.h M be/src/testutil/fault-injection-util.h M tests/custom_cluster/test_rpc_timeout.py 13 files changed, 302 insertions(+), 58 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/12672/6 -- To view, visit http://gerrit.cloudera.org:8080/12672 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia9693151c35e02235665b3c285a48c585973d390 Gerrit-Change-Number: 12672 Gerrit-PatchSet: 6 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall
[native-toolchain-CR] Install newer git version in centos 6.6
Hector Acosta has restored this change. ( http://gerrit.cloudera.org:8080/12862 ) Change subject: Install newer git version in centos 6.6 .. Restored -- To view, visit http://gerrit.cloudera.org:8080/12862 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: restore Gerrit-Change-Id: I66549477ec9378faeda46a98cf6556a3207979eb Gerrit-Change-Number: 12862 Gerrit-PatchSet: 1 Gerrit-Owner: Hector Acosta
[Impala-ASF-CR] IMPALA-8362: Increase authorization test coverage with local catalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12859 ) Change subject: IMPALA-8362: Increase authorization test coverage with local catalog .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12859 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1efc56709bc3da16978589e20d7c611db4fa271a Gerrit-Change-Number: 12859 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 26 Mar 2019 22:42:31 + Gerrit-HasComments: No
[native-toolchain-CR] Prepend docker image to both stderr and stdout
Hector Acosta has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/12861 ) Change subject: Prepend docker image to both stderr and stdout .. Prepend docker image to both stderr and stdout Sometimes, it may be hard to debug which distro produced an error, this commit ensures that both stderr and stdout get piped through sed. Change-Id: I16dd725b0c98599867892801c7146c4fe60f46e1 --- M Makefile 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/61/12861/2 -- To view, visit http://gerrit.cloudera.org:8080/12861 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I16dd725b0c98599867892801c7146c4fe60f46e1 Gerrit-Change-Number: 12861 Gerrit-PatchSet: 2 Gerrit-Owner: Hector Acosta
[native-toolchain-CR] Install newer nss version in centos 6.6
Hector Acosta has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/12862 ) Change subject: Install newer nss version in centos 6.6 .. Install newer nss version in centos 6.6 Centos 6.6 ships with an nss version that can't clone from github. To fix it we include a new Centos-Vault repo file with disabled repositories and explicitly install nss from centos 6.9, which works. Change-Id: I66549477ec9378faeda46a98cf6556a3207979eb --- M docker/all/assert-dependencies-present.py A docker/redhat/Centos6-Vault.repo M docker/redhat6.df 3 files changed, 390 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/62/12862/2 -- To view, visit http://gerrit.cloudera.org:8080/12862 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I66549477ec9378faeda46a98cf6556a3207979eb Gerrit-Change-Number: 12862 Gerrit-PatchSet: 2 Gerrit-Owner: Hector Acosta
[Impala-ASF-CR] IMPALA-7800: Reject new connections after --fe service threads
Zoram Thanga has posted comments on this change. ( http://gerrit.cloudera.org:8080/12579 ) Change subject: IMPALA-7800: Reject new connections after --fe_service_threads .. Patch Set 10: (9 comments) Thanks Michael. Uploading PS 11 http://gerrit.cloudera.org:8080/#/c/12579/10//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/12579/10//COMMIT_MSG@7 PS10, Line 7: Reject new connections after --fe_service_threads > Does this need to be updated too ? Done http://gerrit.cloudera.org:8080/#/c/12579/10/be/src/rpc/TAcceptQueueServer.h File be/src/rpc/TAcceptQueueServer.h: http://gerrit.cloudera.org:8080/#/c/12579/10/be/src/rpc/TAcceptQueueServer.h@65 PS10, Line 65: timeout > nit: timeout_ms Done http://gerrit.cloudera.org:8080/#/c/12579/10/be/src/rpc/TAcceptQueueServer.h@116 PS10, Line 116: /// Amount of time after which a connection request will be timed out. > in milliseconds Done http://gerrit.cloudera.org:8080/#/c/12579/10/be/src/rpc/TAcceptQueueServer.cpp File be/src/rpc/TAcceptQueueServer.cpp: http://gerrit.cloudera.org:8080/#/c/12579/10/be/src/rpc/TAcceptQueueServer.cpp@263 PS10, Line 263: if (queue_timeout_ms_ != 0) : entry->expiration_time_ = MonotonicMillis() + queue_timeout_ms_; > May need to guard against negative value although it's user's fault in that Done http://gerrit.cloudera.org:8080/#/c/12579/10/be/src/rpc/thrift-server.h File be/src/rpc/thrift-server.h: http://gerrit.cloudera.org:8080/#/c/12579/10/be/src/rpc/thrift-server.h@149 PS10, Line 149: int64_t queue_timeout_ms = 0); > May make sense to document the meaning of this argument too. Done http://gerrit.cloudera.org:8080/#/c/12579/10/be/src/rpc/thrift-server.h@197 PS10, Line 197: /// Amount of time an accepted client connection will be kept in the accept : /// queue before it is timed out. Used in TAcceptQueueServer. > May want to document that if it's 0, it means there is no timeout (similar Done http://gerrit.cloudera.org:8080/#/c/12579/10/be/src/rpc/thrift-server.h@277 PS10, Line 277: } > nit: missing blank line after this function. Done http://gerrit.cloudera.org:8080/#/c/12579/10/be/src/service/impala-server.cc File be/src/service/impala-server.cc: http://gerrit.cloudera.org:8080/#/c/12579/10/be/src/service/impala-server.cc@239 PS10, Line 239: accepted_cnxn_timeout > Should this have a more specific name such as "accepted_client_cnxn_timeout Changed. http://gerrit.cloudera.org:8080/#/c/12579/10/tests/custom_cluster/test_frontend_connection_limit.py File tests/custom_cluster/test_frontend_connection_limit.py: http://gerrit.cloudera.org:8080/#/c/12579/10/tests/custom_cluster/test_frontend_connection_limit.py@90 PS10, Line 90: except Exception as e: > How do we catch cases in which the test passes without raising the exceptio The test will be marked as failed in that case (XFAIL). Added an else block with client.close() to not make test verify step wait. -- To view, visit http://gerrit.cloudera.org:8080/12579 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idb345c1d84cc2f691f54ded467f253e758f87e64 Gerrit-Change-Number: 12579 Gerrit-PatchSet: 10 Gerrit-Owner: Zoram Thanga Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Tue, 26 Mar 2019 22:58:23 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7800: Time out new connections after --fe service threads
Hello Michael Ho, Andrew Sherman, Thomas Marshall, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12579 to look at the new patch set (#11). Change subject: IMPALA-7800: Time out new connections after --fe_service_threads .. IMPALA-7800: Time out new connections after --fe_service_threads The current implementation of the FE thrift server waits indefinitely to open the new session, if the maximum number of FE service threads specified by --fe_service_threads has been allocated. This patch introduces a startup flag to control how the server should treat new connection requests if we have run out of the configured number of server threads. If --accepted_client_cnxn_timeout > 0, new connection requests are rejected by the server if we can't get a server thread within the specified timeout. We set the default timeout to be 5 minutes. The old behavior can be restored by setting --accepted_client_cnxn_timeout=0, i.e., no timeout. The timeout applies only to client facing thrift servers, i.e., HS2 and Beeswax servers. Testing: Added a new custom cluster test suite to exercise the new code. Ran core and exhaustive tests. Change-Id: Idb345c1d84cc2f691f54ded467f253e758f87e64 --- M be/src/rpc/TAcceptQueueServer.cpp M be/src/rpc/TAcceptQueueServer.h M be/src/rpc/thrift-server.cc M be/src/rpc/thrift-server.h M be/src/service/impala-server.cc M common/thrift/metrics.json A tests/custom_cluster/test_frontend_connection_limit.py 7 files changed, 255 insertions(+), 36 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/12579/11 -- To view, visit http://gerrit.cloudera.org:8080/12579 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Idb345c1d84cc2f691f54ded467f253e758f87e64 Gerrit-Change-Number: 12579 Gerrit-PatchSet: 11 Gerrit-Owner: Zoram Thanga Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Zoram Thanga
[Impala-ASF-CR] IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user
Austin Nobis has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/12769 ) Change subject: IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user .. IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user This patch adds support for GRANT privilege statements to USER and REVOKE privilege statements from USER. The RangerAuthorizationManager class has been created and will throw UnsupportedOperationException when an unimplemented method is called. The grammar has been updated to support FROM USER and TO USER for GRANT/REVOKE statements. Previously, privileges could be granted to a ROLE via GRANT/REVOKE statements even when the ROLE keyword was omitted, i.e: GRANT ON TO This is still the case for ROLE based authorization to preserve backward compatibility, but Ranger will throw an exception when a GRANT/REVOKE statement excludes the USER keyword. The syntax for the new statement is: GRANT ON TO USER REVOKE ON FROM USER Sentry does not support grant/revoke to/from user. Testing: - An additional end to end test, test_ranger.py, was added. A single test was added that grants and revokes for a user and asserts permissions on a table. The test uses sleep statements to work with Ranger's polling interval for policy changes. More end to end tests will be added in the future when the refresh authorization statement works properly with Ranger. - AuthorizationStmtTest has been refactored to use the new RangerCatalogdAuthorizationManager grant/revoke methods for better test coverage. - Ran all FE tests - Ran all E2E authorization tests Change-Id: I6ee97bf41546d63385026c0e2b19545565402462 --- M common/thrift/JniCatalog.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/GrantRevokePrivStmt.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/NoneAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java A fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryCatalogdAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryImpaladAuthorizationManager.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java A tests/authorization/test_ranger.py 13 files changed, 599 insertions(+), 223 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/12769/6 -- To view, visit http://gerrit.cloudera.org:8080/12769 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6ee97bf41546d63385026c0e2b19545565402462 Gerrit-Change-Number: 12769 Gerrit-PatchSet: 6 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-2990: timeout unresponsive queries in coordinator
Todd Lipcon has posted comments on this change. (
http://gerrit.cloudera.org:8080/12299 )
Change subject: IMPALA-2990: timeout unresponsive queries in coordinator
..
Patch Set 7:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/12299/4/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/12299/4/be/src/service/impala-server.cc@215
PS4, Line 215: DEFINE_int32(status_report_interval_ms, 5000,
> I think you're saying to set this to 1.1 * GetMaxReportRetryMs()?
right, I think I meant 0.1 * GetMaxReportRetryMs(). ie we need to check much
more frequently than the timeout, or else we could overshoot it by a factor of 2
http://gerrit.cloudera.org:8080/#/c/12299/4/be/src/service/impala-server.cc@398
PS4, Line 398:
> Of course, even in this configuration, backends will report their final sta
k. We could deprecate it in a separate patch I suppose. Fine with this as is.
http://gerrit.cloudera.org:8080/#/c/12299/4/be/src/service/impala-server.cc@2288
PS4, Line 2288: return Status::OK();
: }
:
: void ImpalaServer::ExpireQuery(ClientRequestState* crs, const
Status& status) {
: DCHECK(!status.ok());
: cancellation_thread_pool_->Offer(
> Yes, all of these flags are documented and settable.
k, would be interested what Michael thinks about that idea
http://gerrit.cloudera.org:8080/#/c/12299/7/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/12299/7/be/src/service/impala-server.cc@218
PS7, Line 218: DEFINE_int32(status_report_max_retries, 3,
Is our timeout too aggressive here? If one of the backends goes into some kind
of 15s process-wide pause (eg as we've seen in the past with tcmalloc releasing
memory in https://issues.apache.org/jira/browse/IMPALA-2800) we'll end up
cancelling. Am afraid that being too aggressive will end up cancelling queries
incorrectly. Curious for Michael's thoughts
http://gerrit.cloudera.org:8080/#/c/12299/7/be/src/service/impala-server.cc@2258
PS7, Line 2258: max_lag_ms * 1.1
per comment elsewhere: don't we need to wake up more often than the timeout?
otherwise we can overshoot by a factor of 2 here.
--
To view, visit http://gerrit.cloudera.org:8080/12299
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I196c8c6a5633b1960e2c3a3884777be9b3824987
Gerrit-Change-Number: 12299
Gerrit-PatchSet: 7
Gerrit-Owner: Thomas Marshall
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Reviewer: Thomas Marshall
Gerrit-Reviewer: Todd Lipcon
Gerrit-Comment-Date: Tue, 26 Mar 2019 23:16:03 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12769 ) Change subject: IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user .. Patch Set 6: (4 comments) Mostly nits. LGTM. http://gerrit.cloudera.org:8080/#/c/12769/6/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java File fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java: http://gerrit.cloudera.org:8080/#/c/12769/6/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java@84 PS6, Line 84: nit: remove extra new line http://gerrit.cloudera.org:8080/#/c/12769/6/fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java File fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java: http://gerrit.cloudera.org:8080/#/c/12769/6/fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java@1495 PS6, Line 1495: principal_types nit: use camelCase http://gerrit.cloudera.org:8080/#/c/12769/6/tests/authorization/test_ranger.py File tests/authorization/test_ranger.py: http://gerrit.cloudera.org:8080/#/c/12769/6/tests/authorization/test_ranger.py@44 PS6, Line 44: anobis_client nit: non_admin_client? http://gerrit.cloudera.org:8080/#/c/12769/6/tests/authorization/test_ranger.py@53 PS6, Line 53: user="admin") nit: move to L52 -- To view, visit http://gerrit.cloudera.org:8080/12769 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6ee97bf41546d63385026c0e2b19545565402462 Gerrit-Change-Number: 12769 Gerrit-PatchSet: 6 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 26 Mar 2019 23:15:41 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8143: Enhance DoRpcWithRetry().
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12672 ) Change subject: IMPALA-8143: Enhance DoRpcWithRetry(). .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2554/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12672 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia9693151c35e02235665b3c285a48c585973d390 Gerrit-Change-Number: 12672 Gerrit-PatchSet: 6 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 26 Mar 2019 23:17:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12769 ) Change subject: IMPALA-8225: Add Ranger support for grant/revoke privilege to/from user .. Patch Set 6: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/2555/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/12769 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6ee97bf41546d63385026c0e2b19545565402462 Gerrit-Change-Number: 12769 Gerrit-PatchSet: 6 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 26 Mar 2019 23:36:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7800: Time out new connections after --fe service threads
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12579 ) Change subject: IMPALA-7800: Time out new connections after --fe_service_threads .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2556/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12579 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idb345c1d84cc2f691f54ded467f253e758f87e64 Gerrit-Change-Number: 12579 Gerrit-PatchSet: 11 Gerrit-Owner: Zoram Thanga Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Tue, 26 Mar 2019 23:40:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8330: Impala shell config file should use flag names
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/12823 ) Change subject: IMPALA-8330: Impala shell config file should use flag names .. Patch Set 5: Code-Review+1 (1 comment) Other reviewers probably have more comments. So +1 ing this. LGTM. http://gerrit.cloudera.org:8080/#/c/12823/5/shell/option_parser.py File shell/option_parser.py: http://gerrit.cloudera.org:8080/#/c/12823/5/shell/option_parser.py@132 PS5, Line 132: config = ConfigParser.ConfigParser() Do we need this? we convert it to uppercase anyway (L144) -- To view, visit http://gerrit.cloudera.org:8080/12823 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic43603c1b538af08fddcab1b2c1f6ad1af1a6cb9 Gerrit-Change-Number: 12823 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 26 Mar 2019 23:54:38 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8312 : Alter database operations have race condition
Vihang Karajgaonkar has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/12789 ) Change subject: IMPALA-8312 : Alter database operations have race condition .. IMPALA-8312 : Alter database operations have race condition This patch fixes a race condition in the alter database implementation in the catalogOpExecutor. The original implementation did a in-place modification of the metastore database object in the Db. This can lead to partially updated database object becoming visible to a reading thread causing potential problems. In order to fix the race, the patch makes a copy of the existing database object, modifies the copy and then atomically switches the actual database object with the modified copy. This is done while holding the metastoreddlLock, and then taking the write lock on the catalog version object which makes it consistent with the other catalog write operations. Added a test which consistently reproduces the race. The test creating many reader threads and a writer thread which continuously keeps changing the owner name and its type by issuing a alter database operation. The test fails without the patch. After the patch the test passes. The race also applies to the alter database set comment operation, although its hard to write a test for that code-path. Change-Id: I32c8c96a6029bf9d9db37ea8315f6c9603b5a2fc --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java A fe/src/test/java/org/apache/impala/catalog/AlterDatabaseTest.java 4 files changed, 311 insertions(+), 41 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/12789/7 -- To view, visit http://gerrit.cloudera.org:8080/12789 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I32c8c96a6029bf9d9db37ea8315f6c9603b5a2fc Gerrit-Change-Number: 12789 Gerrit-PatchSet: 7 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8312 : Alter database operations have race condition
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12789 ) Change subject: IMPALA-8312 : Alter database operations have race condition .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2557/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12789 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32c8c96a6029bf9d9db37ea8315f6c9603b5a2fc Gerrit-Change-Number: 12789 Gerrit-PatchSet: 7 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 27 Mar 2019 00:39:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8314 Set correct permissions for ~/.ssh files
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12725 ) Change subject: IMPALA-8314 Set correct permissions for ~/.ssh files .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3960/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/12725 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3dd6b12318dd99c67127bff10a89adb931f44087 Gerrit-Change-Number: 12725 Gerrit-PatchSet: 2 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 27 Mar 2019 00:42:02 + Gerrit-HasComments: No
[Impala-ASF-CR] Move DEBIAN FRONTEND=noninteractive variable to the ubuntu section of the script
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12723 ) Change subject: Move DEBIAN_FRONTEND=noninteractive variable to the ubuntu section of the script .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12723 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifab2f56d3e119e4e73685c3cbd5249760ceb33ac Gerrit-Change-Number: 12723 Gerrit-PatchSet: 2 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 27 Mar 2019 00:42:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8307 Use test -x to check for ntp-wait
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12726 ) Change subject: IMPALA-8307 Use test -x to check for ntp-wait .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12726 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I53c63dfa651ac242050171da70540d24c4caf32c Gerrit-Change-Number: 12726 Gerrit-PatchSet: 2 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 27 Mar 2019 00:43:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8345 : Add option to set up minicluster to use Hive 3
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12846 ) Change subject: IMPALA-8345 : Add option to set up minicluster to use Hive 3 .. Patch Set 9: Thanks for addressing my feedback. I'll set someone else review to a +2 -- To view, visit http://gerrit.cloudera.org:8080/12846 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icfed856c1f5429ed45fd3d9cb08a5d1bb96a9605 Gerrit-Change-Number: 12846 Gerrit-PatchSet: 9 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 27 Mar 2019 01:18:04 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-7095: clean up scan node profiles
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12848 ) Change subject: IMPALA-7095: clean up scan node profiles .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12848 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I77286282d42e7764bfdf94c7ec47cec9d743f787 Gerrit-Change-Number: 12848 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 27 Mar 2019 01:19:38 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-7095: clean up scan node profiles
Quanlong Huang has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12848 ) Change subject: IMPALA-7095: clean up scan node profiles .. IMPALA-7095: clean up scan node profiles Add counters to scan node implementations where they make sense but were missing (e.g. row batch queue counters for multithread Kudu scans) and remove them where they don't make sense (e.g. scanner thread counters for non-multithreaded scans). Refactors the multithreaded Kudu and HDFS scans to share logic via composition (single inheritance doesn't work for this case), which enables the same set of counters to be maintained with shared code. The row batch queueing and thread tracking is now shared. I looked at combining the logic around 'status_', 'lock_' and 'done_' between the two but the details were different enough that it didn't seem worth abstracting. Adds a PeakScannerThreadConcurrency counter - this answers a common question. Fixes RowsRead for data source scans. Fix some of the comments to be more accurate/useful. Testing: Ran exhaustive tests. Ran various types of scans (HDFS, Kudu, HBase, Data source) and inspected the profile output manually. Change-Id: I77286282d42e7764bfdf94c7ec47cec9d743f787 Reviewed-on: http://gerrit.cloudera.org:8080/10810 Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins Reviewed-on: http://gerrit.cloudera.org:8080/12848 --- M be/src/exec/data-source-scan-node.cc M be/src/exec/data-source-scan-node.h M be/src/exec/hbase-scan-node.cc M be/src/exec/hbase-scan-node.h M be/src/exec/hbase-table-scanner.cc M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/hdfs-scan-node-mt.h M be/src/exec/hdfs-scan-node.cc M be/src/exec/hdfs-scan-node.h M be/src/exec/hdfs-text-scanner.cc M be/src/exec/kudu-scan-node-base.h M be/src/exec/kudu-scan-node-mt.cc M be/src/exec/kudu-scan-node.cc M be/src/exec/kudu-scan-node.h M be/src/exec/scan-node.cc M be/src/exec/scan-node.h M be/src/runtime/fragment-instance-state.cc M be/src/util/blocking-queue.h M be/src/util/thread.h 20 files changed, 496 insertions(+), 357 deletions(-) Approvals: Impala Public Jenkins: Verified Tim Armstrong: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/12848 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: merged Gerrit-Change-Id: I77286282d42e7764bfdf94c7ec47cec9d743f787 Gerrit-Change-Number: 12848 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8345 : Add option to set up minicluster to use Hive 3
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/12846 )
Change subject: IMPALA-8345 : Add option to set up minicluster to use Hive 3
..
Patch Set 9:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/12846/9/bin/bootstrap_toolchain.py
File bin/bootstrap_toolchain.py:
http://gerrit.cloudera.org:8080/#/c/12846/9/bin/bootstrap_toolchain.py@433
PS9, Line 433: def download_cdp_hive(toolchain_root):
We don't have to do it now, but at some point, we should refactor this function
to be more generic, like downloading Ranger, Hive, etc.
http://gerrit.cloudera.org:8080/#/c/12846/9/bin/create-test-configuration.sh
File bin/create-test-configuration.sh:
http://gerrit.cloudera.org:8080/#/c/12846/9/bin/create-test-configuration.sh@132
PS9, Line 132: # Certain configurations (like
SentrySyncHMSNotificationsPostListener) do not work
: # with HMS 3.1.0. Use a cdp specific configuration template
: generate_config postgresql-hive-site.xml.cdp.template
hive-site.xml
will this cause Sentry tests to fail when USE_CDP_HIVE=true?
http://gerrit.cloudera.org:8080/#/c/12846/9/bin/create-test-configuration.sh@149
PS9, Line 149: 1>${IMPALA_CLUSTER_LOGS_DIR}/schematool.log 2>&1
it may be better to use tee
2>&1 | tee ${IMPALA_CLUSTER_LOGS_DIR}/schematool.log
http://gerrit.cloudera.org:8080/#/c/12846/9/fe/src/test/resources/postgresql-hive-site.xml.cdp.template
File fe/src/test/resources/postgresql-hive-site.xml.cdp.template:
http://gerrit.cloudera.org:8080/#/c/12846/9/fe/src/test/resources/postgresql-hive-site.xml.cdp.template@27
PS9, Line 27:
nit: formatting is off in this file, a lot of mixed 1 space vs 2 spaces.
http://gerrit.cloudera.org:8080/#/c/12846/9/testdata/bin/run-hive-server.sh
File testdata/bin/run-hive-server.sh:
http://gerrit.cloudera.org:8080/#/c/12846/9/testdata/bin/run-hive-server.sh@72
PS9, Line 72: if [[ $USE_CDP_HIVE && -n "$SENTRY_HOME" ]]; then
: for f in ${SENTRY_HOME}/lib/sentry-binding-hive*.jar; do
: FILE_NAME=$(basename $f)
: # exclude all the hive jars from being included in the
classpath since Sentry
: # depends on Hive 2.1.1
: if [[ ! $FILE_NAME == hive* ]]; then
: export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:${f}
: fi
: done
: fi
nit: use 2 spaces
--
To view, visit http://gerrit.cloudera.org:8080/12846
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icfed856c1f5429ed45fd3d9cb08a5d1bb96a9605
Gerrit-Change-Number: 12846
Gerrit-PatchSet: 9
Gerrit-Owner: Vihang Karajgaonkar
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Wed, 27 Mar 2019 01:50:55 +
Gerrit-HasComments: Yes
[Impala-ASF-CR](asf-site) [DOCS] Impala 3.2 docs to be published
Anonymous Coward (479) has posted comments on this change. ( http://gerrit.cloudera.org:8080/12860 ) Change subject: [DOCS] Impala 3.2 docs to be published .. Patch Set 1: Code-Review+1 LGTM. Waiting on gerrit.cloudera.org admin to give me +2 rights with my new username. -- To view, visit http://gerrit.cloudera.org:8080/12860 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: I3564270ab8488cd3482c40070f756e6d4b2a056e Gerrit-Change-Number: 12860 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Anonymous Coward (479) Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Comment-Date: Wed, 27 Mar 2019 03:05:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8330: Impala shell config file should use flag names
Jim Apple has posted comments on this change. ( http://gerrit.cloudera.org:8080/12823 ) Change subject: IMPALA-8330: Impala shell config file should use flag names .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/12823/5/tests/shell/good_impalarc File tests/shell/good_impalarc: PS5: If this breaks existing .impalarc files, I'd prefer to wait until 4.0. -- To view, visit http://gerrit.cloudera.org:8080/12823 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic43603c1b538af08fddcab1b2c1f6ad1af1a6cb9 Gerrit-Change-Number: 12823 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Comment-Date: Wed, 27 Mar 2019 03:26:03 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7211: Fix the between predicate for decimals
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10898 ) Change subject: IMPALA-7211: Fix the between predicate for decimals .. Patch Set 5: This change did not cherrypick successfully into branch 2.x. To resolve this, please do the cherry-pick manually and submit it to Gerrit at refs/for/2.x or add an exception to the branch 2.x copy of bin/ignored_commits.json. Thanks, your friendly bot at https://jenkins.impala.io/job/cherrypick-2.x-and-test/666/ . -- To view, visit http://gerrit.cloudera.org:8080/10898 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iac89d62082052b41bfa698e4de09aec26df5c312 Gerrit-Change-Number: 10898 Gerrit-PatchSet: 5 Gerrit-Owner: Taras Bobrovytsky Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 27 Mar 2019 05:01:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8314 Set correct permissions for ~/.ssh files
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12725 ) Change subject: IMPALA-8314 Set correct permissions for ~/.ssh files .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12725 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3dd6b12318dd99c67127bff10a89adb931f44087 Gerrit-Change-Number: 12725 Gerrit-PatchSet: 2 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 27 Mar 2019 05:13:08 + Gerrit-HasComments: No
