Hello Quanlong Huang, Aman Sinha, Joe McDonnell, Impala Public Jenkins, John Sherman,
I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21160 to look at the new patch set (#14). Change subject: IMPALA-12921, IMPALA-12985: Support locally built Ranger ...................................................................... IMPALA-12921, IMPALA-12985: Support locally built Ranger This patch does the following. 1. Add the support for locally built Ranger. 2. Resolve IMPALA-12985 by switching to the new constructor when instantiating RangerAccessRequestImpl. The instructions on how to build Ranger locally and point Impala to the locally built Ranger server are provided as below. Suppose the Ranger project is under the folder $RANGER_SRC_DIR. We could execute the following to build Ranger. By default, the compressed tarball is produced under $RANGER_SRC_DIR/target. mvn clean compile -B -nsu -DskipCheck=true -Dcheckstyle.skip=true \ package install -DskipITs -DskipTests -Dmaven.javadoc.skip=true After building Ranger, we need to build Impala's Java code so that Impala's Java code could consume the locally produced Ranger classes. We will need to export the following environment variables before building Impala. This prevents bootstrap_toolchain.py from trying to download the compressed Ranger tarball. Moreover, we need to apply IMPALA-12921_pom_patching.diff on Impala if the locally built Ranger pulls in an incompatible version of hive-storage-api. 1. export RANGER_VERSION_OVERRIDE=\ $(mvn -f $RANGER_SRC_DIR/pom.xml -q help:evaluate \ -Dexpression=project.version -DforceStdout) 2. export RANGER_HOME_OVERRIDE=$RANGER_SRC_DIR/target/\ ranger-${RANGER_VERSION_OVERRIDE}-admin It then suffices to execute the following to point Impala to the locally built Ranger server before starting Impala. 1. source $IMPALA_HOME/bin/impala-config.sh 2. tar zxv -f $RANGER_SRC_DIR/target/\ ranger-${IMPALA_RANGER_VERSION}-admin.tar.gz \ -C $RANGER_SRC_DIR/target/ 3. $IMPALA_HOME/bin/create-test-configuration.sh 4. $IMPALA_HOME/bin/create-test-configuration.sh \ -create_ranger_policy_db 5. $IMPALA_HOME/testdata/bin/run-ranger.sh (run-all.sh has to be executed instead if other underlying services have not been started) 6. $IMPALA_HOME/testdata/bin/setup-ranger.sh Testing: - Manually verified that we could point Impala to a locally built Apache Ranger on the master branch (with tip being RANGER-4745). - Manually verified that with RANGER-4771.diff and IMPALA-12921_addendum.diff, only 3 authorization-related tests failed. They failed because the resource type of 'storage-type' is not supported in Apache Ranger yet and thus the test cases added in IMPALA-10436 could fail. - Manually verified that the log files of Apache and CDP Ranger's Admin server could be created under ${RANGER_LOG_DIR} after we start the Ranger service. - Verified that this patch passed the core tests when CDP Ranger is used. Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27 --- M README-build.md M bin/bootstrap_toolchain.py M bin/create-test-configuration.sh M bin/impala-config.sh M bin/rat_exclude_files.txt M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java M testdata/bin/setup-ranger.sh A testdata/cluster/ranger/IMPALA-12921_addendum.diff A testdata/cluster/ranger/RANGER-4771.diff A testdata/cluster/ranger/README R testdata/cluster/ranger/setup/all_database_policy_revised.json.template A testdata/cluster/ranger/setup/impala_group_non_owner_2.json M testdata/cluster/ranger/setup/impala_user_non_owner.json.template A testdata/cluster/ranger/setup/impala_user_non_owner_2.json.template M testdata/cluster/ranger/setup/impala_user_owner.json.template 16 files changed, 384 insertions(+), 16 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/60/21160/14 -- To view, visit http://gerrit.cloudera.org:8080/21160 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27 Gerrit-Change-Number: 21160 Gerrit-PatchSet: 14 Gerrit-Owner: Fang-Yu Rao <fangyu....@cloudera.com> Gerrit-Reviewer: Aman Sinha <amsi...@cloudera.com> Gerrit-Reviewer: Fang-Yu Rao <fangyu....@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Joe McDonnell <joemcdonn...@cloudera.com> Gerrit-Reviewer: John Sherman <j...@cloudera.com> Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com>