Review Request 64707: AMBARI-22669 Ranger stack script changes to fix missing directory failure for blueprint installation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/64707/ --- Review request for Ambari, Gautam Borad, Jonathan Hurley, and Vishal Suvagia. Bugs: AMBARI-22669 https://issues.apache.org/jira/browse/AMBARI-22669 Repository: ambari Description --- Updating stack scripts code to remove use of commandParams/version to build system path. For upgrade, using upgrade_summary module to get upgrade-to version ie (target version). Diffs - ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py 5731e6c ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py d0a725a ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py 9b1f6e2 ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms.py bbc438b ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms_server.py 0b37489 ambari-server/src/test/python/stacks/2.6/RANGER/test_ranger_admin.py e8bacbd Diff: https://reviews.apache.org/r/64707/diff/1/ Testing --- Tested Ranger Installation. Ran 255 tests in 7.877s OK -- Total run:1210 Total errors:0 Total failures:0 OK Thanks, Mugdha Varadkar
Review Request 63907: AMBARI-22463 Removing secure reference configs entries from stack for Ranger KMS service
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63907/ --- Review request for Ambari, Robert Levas and Vishal Suvagia. Bugs: AMBARI-22463 https://issues.apache.org/jira/browse/AMBARI-22463 Repository: ambari Description --- Removing below properties entries from stack, hadoop.kms.authentication.kerberos.keytab hadoop.kms.authentication.kerberos.principal hadoop.kms.authentication.signer.secret.provider.zookeeper.kerberos.keytab hadoop.kms.authentication.signer.secret.provider.zookeeper.kerberos.principal These properties will be added in secure cluster using Ranger KMS Kerberos descriptor. Diffs - ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/configuration/kms-site.xml 59a6952 ambari-server/src/main/resources/common-services/RANGER_KMS/1.0.0.3.0/configuration/kms-site.xml 1e6f7b5 Diff: https://reviews.apache.org/r/63907/diff/1/ Testing --- Ran 255 tests in 8.409s OK -- Total run:1205 Total errors:0 Total failures:0 OK Thanks, Mugdha Varadkar
Review Request 63505: AMBARI-22350 Updating Ranger & Atlas stack features for stack 3.0
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/63505/ --- Review request for Ambari, Gautam Borad, Jayush Luniya, and Vishal Suvagia. Bugs: AMBARI-22350 https://issues.apache.org/jira/browse/AMBARI-22350 Repository: ambari Description --- Adding missing Ranger & Atlas stack features for stack 3.0 and also adding AMBARI-21227 fix for Ranger & Ranger Kms stack 3.0. Diffs - ambari-server/src/main/resources/common-services/RANGER/1.0.0.3.0/package/scripts/ranger_admin.py 777e95d ambari-server/src/main/resources/common-services/RANGER/1.0.0.3.0/package/scripts/setup_ranger_xml.py c84ae96 ambari-server/src/main/resources/common-services/RANGER_KMS/1.0.0.3.0/package/scripts/kms.py 6e4a171 ambari-server/src/main/resources/common-services/RANGER_KMS/1.0.0.3.0/package/scripts/kms_server.py 4708da1 ambari-server/src/main/resources/stacks/HDP/3.0/properties/stack_features.json afd5183 Diff: https://reviews.apache.org/r/63505/diff/1/ Testing --- Unit test: Ran 255 tests in 7.733s OK -- Total run:1204 Total errors:0 Total failures:0 OK Thanks, Mugdha Varadkar
Review Request 63036: AMBARI-22234 Optimizing Ranger KMS imports
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63036/
---
Review request for Ambari, Gautam Borad and Jonathan Hurley.
Bugs: AMBARI-22234
https://issues.apache.org/jira/browse/AMBARI-22234
Repository: ambari
Description
---
Updating Ranger Kms script file and using proper methods for importing modules.
Diffs
-
ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms_server.py
fcf2478
ambari-server/src/main/resources/common-services/RANGER_KMS/1.0.0.3.0/package/scripts/kms_server.py
4c313c4
Diff: https://reviews.apache.org/r/63036/diff/1/
Testing
---
Running tests for stack:None service:RANGER_KMS
2017-10-13 17:44:18,232 - Stack Feature Version Info: Cluster Stack=2.5,
Command Stack=None, Command Version=2.5.0.0-777 -> 2.5.0.0-777
2017-10-13 17:44:18,255 - Using hadoop conf dir:
/usr/hdp/2.5.0.0-777/hadoop/conf
test_db_flavor_0_5_0_2_3 (test_db_flavor_config.TestDbFlavorConfig) ...
2017-10-13 17:44:18,262 - Error! Sorry, but we can't find jdbc driver with
default name mysql-connector-java.jar in ranger kms lib dir. So, db connection
check can fail. Please run 'ambari-server setup --jdbc-db={db_name}
--jdbc-driver={path_to_jdbc} on server host.'
2017-10-13 17:44:18,262 - Error! Sorry, but we can't find jdbc driver with
default name mysql-connector-java.jar in ranger kms lib dir. So, db connection
check can fail. Please run 'ambari-server setup --jdbc-db={db_name}
--jdbc-driver={path_to_jdbc} on server host.'
ok
2017-10-13 17:44:18,275 - Stack Feature Version Info: Cluster Stack=2.5,
Command Stack=None, Command Version=2.5.0.0-777 -> 2.5.0.0-777
2017-10-13 17:44:18,298 - Using hadoop conf dir:
/usr/hdp/2.5.0.0-777/hadoop/conf
test_db_flavor_1_0_0_3_0 (test_db_flavor_config.TestDbFlavorConfig) ...
2017-10-13 17:44:18,305 - Error! Sorry, but we can't find jdbc driver with
default name mysql-connector-java.jar in ranger kms lib dir. So, db connection
check can fail. Please run 'ambari-server setup --jdbc-db={db_name}
--jdbc-driver={path_to_jdbc} on server host.'
2017-10-13 17:44:18,305 - Error! Sorry, but we can't find jdbc driver with
default name mysql-connector-java.jar in ranger kms lib dir. So, db connection
check can fail. Please run 'ambari-server setup --jdbc-db={db_name}
--jdbc-driver={path_to_jdbc} on server host.'
ok
2017-10-13 17:44:18,314 - Stack Feature Version Info: Cluster Stack=2.5,
Command Stack=None, Command Version=2.5.0.0-777 -> 2.5.0.0-777
2017-10-13 17:44:18,337 - Using hadoop conf dir:
/usr/hdp/2.5.0.0-777/hadoop/conf
test_unsupported_db_flavor_0_5_0_2_3 (test_db_flavor_config.TestDbFlavorConfig)
... ok
2017-10-13 17:44:18,344 - Stack Feature Version Info: Cluster Stack=2.5,
Command Stack=None, Command Version=2.5.0.0-777 -> 2.5.0.0-777
2017-10-13 17:44:18,367 - Using hadoop conf dir:
/usr/hdp/2.5.0.0-777/hadoop/conf
test_unsupported_db_flavor_1_0_0_3_0 (test_db_flavor_config.TestDbFlavorConfig)
... ok
------
Ran 4 tests in 0.147s
OK
Thanks,
Mugdha Varadkar
Re: Review Request 62985: AMBARI-22232 : Need to add a new property to support proxy users property for Atlas service
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62985/#review188094 --- Ship it! Ship It! - Mugdha Varadkar On Oct. 14, 2017, 11:17 a.m., Vishal Suvagia wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62985/ > --- > > (Updated Oct. 14, 2017, 11:17 a.m.) > > > Review request for Ambari, Gautam Borad, Jonathan Hurley, Madhan Neethiraj, > Mugdha Varadkar, Nate Cole, and Nixon Rodrigues. > > > Bugs: AMBARI-22232 > https://issues.apache.org/jira/browse/AMBARI-22232 > > > Repository: ambari > > > Description > --- > > When Atlas is enabled via proxy needs to know the proxy-users to authorize > them while accessing the Atlas service via proxy support using property > atlas.proxyusers with default value knox. Additional details on ATLAS-2166. > > PS: Attached patch is applicable for trunk branch, patch for branch-2.6 is > attached on Apache JIRA. > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/AtlasProxyUserConfigCalculation.java > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml > 1af3b18 > ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml > 0a01a04 > > ambari-server/src/main/resources/stacks/HDP/2.6/services/ATLAS/configuration/application-properties.xml > c271dc3 > ambari-server/src/main/resources/stacks/HDP/2.6/services/stack_advisor.py > 94f28db > > ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml > ebb81d9 > ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml > ae5972e > > ambari-server/src/test/java/org/apache/ambari/server/serveraction/upgrades/AtlasProxyUserConfigCalculationTest.java > PRE-CREATION > ambari-server/src/test/python/stacks/2.6/common/test_stack_advisor.py > ade08c1 > > > Diff: https://reviews.apache.org/r/62985/diff/2/ > > > Testing > --- > > Verified with fresh-install and EU on CentOS 6. > > > Thanks, > > Vishal Suvagia > >
Re: Review Request 62890: AMBARI-22204 Updating Ranger Admin pre-upgrade task
> On Oct. 11, 2017, 2:08 p.m., Jonathan Hurley wrote: > > ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py > > Lines 201-205 (original), 201-207 (patched) > > <https://reviews.apache.org/r/62890/diff/1/?file=1852401#file1852401line201> > > > > Why not just use the upgrade_summary module for all of this - would > > protect you against changes in the future. Updated in latest patch - Mugdha --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62890/#review187657 ------- On Oct. 12, 2017, 9 a.m., Mugdha Varadkar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62890/ > --- > > (Updated Oct. 12, 2017, 9 a.m.) > > > Review request for Ambari, Alejandro Fernandez, Gautam Borad, Jonathan > Hurley, Nate Cole, and Velmurugan Periasamy. > > > Bugs: AMBARI-22204 > https://issues.apache.org/jira/browse/AMBARI-22204 > > > Repository: ambari > > > Description > --- > > After AMBARI-21581 changes, need to update Ranger Admin pre-upgrade task. > > > Diffs > - > > > ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py > f779c18 > > ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/ranger_tagsync.py > 74439bf > > ambari-server/src/main/resources/stacks/BigInsights/4.2/services/RANGER/package/scripts/ranger_admin.py > 37555cc > > > Diff: https://reviews.apache.org/r/62890/diff/2/ > > > Testing > --- > > Tested EU & RU for Ranger Admin. > > > Thanks, > > Mugdha Varadkar > >
Re: Review Request 62890: AMBARI-22204 Updating Ranger Admin pre-upgrade task
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62890/ --- (Updated Oct. 12, 2017, 9 a.m.) Review request for Ambari, Alejandro Fernandez, Gautam Borad, Jonathan Hurley, Nate Cole, and Velmurugan Periasamy. Changes --- Handle Jonathan Hurley changes Bugs: AMBARI-22204 https://issues.apache.org/jira/browse/AMBARI-22204 Repository: ambari Description --- After AMBARI-21581 changes, need to update Ranger Admin pre-upgrade task. Diffs (updated) - ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py f779c18 ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/ranger_tagsync.py 74439bf ambari-server/src/main/resources/stacks/BigInsights/4.2/services/RANGER/package/scripts/ranger_admin.py 37555cc Diff: https://reviews.apache.org/r/62890/diff/2/ Changes: https://reviews.apache.org/r/62890/diff/1-2/ Testing --- Tested EU & RU for Ranger Admin. Thanks, Mugdha Varadkar
Review Request 62890: AMBARI-22204 Updating Ranger Admin pre-upgrade task
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62890/ --- Review request for Ambari, Alejandro Fernandez, Gautam Borad, Jonathan Hurley, Nate Cole, and Velmurugan Periasamy. Bugs: AMBARI-22204 https://issues.apache.org/jira/browse/AMBARI-22204 Repository: ambari Description --- After AMBARI-21581 changes, need to update Ranger Admin pre-upgrade task. Diffs - ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py f779c18 ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/ranger_tagsync.py 74439bf ambari-server/src/main/resources/stacks/BigInsights/4.2/services/RANGER/package/scripts/ranger_admin.py 37555cc Diff: https://reviews.apache.org/r/62890/diff/1/ Testing --- Tested EU & RU for Ranger Admin. Thanks, Mugdha Varadkar
Review Request 62712: AMBARI-22102 Ranger KMS should add proxy user for Spark2 user
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62712/ --- Review request for Ambari, Bikas Saha, Gautam Borad, Jayush Luniya, and Velmurugan Periasamy. Bugs: AMBARI-22102 https://issues.apache.org/jira/browse/AMBARI-22102 Repository: ambari Description --- Spark2 user needs to be added to Ranger KMS proxy users in cluster. Updating required_services list with SPARK2 config livy2_user/livy2-env to recommendKMSProxyUsers() Diffs - ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py be615db Diff: https://reviews.apache.org/r/62712/diff/1/ Testing --- Unit Test passed. Thanks, Mugdha Varadkar
Re: Review Request 62351: AMBARI-21910 Ranger Usersync config to support nested group evaluation for LDAP Sync source property
> On Sept. 21, 2017, 3:19 p.m., Jonathan Hurley wrote: > > ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml > > Lines 398 (patched) > > <https://reviews.apache.org/r/62351/diff/2/?file=1831545#file1831545line398> > > > > Maybe a better title? Like "Enabling Nested Group Sync for Ranger" Updated in latest patch - Mugdha --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62351/#review185894 --- On Sept. 22, 2017, 12:40 p.m., Mugdha Varadkar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62351/ > --- > > (Updated Sept. 22, 2017, 12:40 p.m.) > > > Review request for Ambari, Alejandro Fernandez, Gautam Borad, Jonathan > Hurley, Jayush Luniya, and Velmurugan Periasamy. > > > Bugs: AMBARI-21910 > https://issues.apache.org/jira/browse/AMBARI-21910 > > > Repository: ambari > > > Description > --- > > As part of RANGER-1735, we are supporting nested groups in ranger usersync. > For this, introduced one new property > "ranger.usersync.ldap.grouphierarchylevels". The value should be an integer > with default value as 0. This property should be under Ranger --> Configs --> > Ranger User Info --> Group Configs tab. > This property can be enabled with a flag labelled as "Sync Nested Groups". > 1. Default value for this flag is "No" > 2. This flag is always shown under Group Configs (even when "Enable group > Sync" is set to "No") > 3. If this flag is set to "No", then the value for > ranger.usersync.ldap.grouphierarchylevels is hidden and is set to "0". > 4. If this flag is set to "Yes", then > ranger.usersync.ldap.grouphierarchylevels is shown and allowed to set any > integer greater than 0. > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/RangerUsersyncConfigCalculation.java > PRE-CREATION > > ambari-server/src/main/resources/common-services/RANGER/0.7.0/configuration/ranger-env.xml > 627216e > > ambari-server/src/main/resources/common-services/RANGER/0.7.0/configuration/ranger-ugsync-site.xml > a994856 > > ambari-server/src/main/resources/common-services/RANGER/0.7.0/themes/theme_version_5.json > 8068a38 > > ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml > 535ac2e > ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml > 74271cc > > ambari-server/src/test/java/org/apache/ambari/server/serveraction/upgrades/RangerUsersyncConfigCalculationTest.java > PRE-CREATION > > > Diff: https://reviews.apache.org/r/62351/diff/3/ > > > Testing > --- > > Test Fresh Installs. > Ambari upgrade with stack 2.6. > Unit Test passed for UpgradeCatalog260Test. > > > Thanks, > > Mugdha Varadkar > >
Re: Review Request 62351: AMBARI-21910 Ranger Usersync config to support nested group evaluation for LDAP Sync source property
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62351/ --- (Updated Sept. 22, 2017, 12:40 p.m.) Review request for Ambari, Alejandro Fernandez, Gautam Borad, Jonathan Hurley, Jayush Luniya, and Velmurugan Periasamy. Bugs: AMBARI-21910 https://issues.apache.org/jira/browse/AMBARI-21910 Repository: ambari Description --- As part of RANGER-1735, we are supporting nested groups in ranger usersync. For this, introduced one new property "ranger.usersync.ldap.grouphierarchylevels". The value should be an integer with default value as 0. This property should be under Ranger --> Configs --> Ranger User Info --> Group Configs tab. This property can be enabled with a flag labelled as "Sync Nested Groups". 1. Default value for this flag is "No" 2. This flag is always shown under Group Configs (even when "Enable group Sync" is set to "No") 3. If this flag is set to "No", then the value for ranger.usersync.ldap.grouphierarchylevels is hidden and is set to "0". 4. If this flag is set to "Yes", then ranger.usersync.ldap.grouphierarchylevels is shown and allowed to set any integer greater than 0. Diffs (updated) - ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/RangerUsersyncConfigCalculation.java PRE-CREATION ambari-server/src/main/resources/common-services/RANGER/0.7.0/configuration/ranger-env.xml 627216e ambari-server/src/main/resources/common-services/RANGER/0.7.0/configuration/ranger-ugsync-site.xml a994856 ambari-server/src/main/resources/common-services/RANGER/0.7.0/themes/theme_version_5.json 8068a38 ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml 535ac2e ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml 74271cc ambari-server/src/test/java/org/apache/ambari/server/serveraction/upgrades/RangerUsersyncConfigCalculationTest.java PRE-CREATION Diff: https://reviews.apache.org/r/62351/diff/3/ Changes: https://reviews.apache.org/r/62351/diff/2-3/ Testing --- Test Fresh Installs. Ambari upgrade with stack 2.6. Unit Test passed for UpgradeCatalog260Test. Thanks, Mugdha Varadkar
Re: Review Request 62351: AMBARI-21910 Ranger Usersync config to support nested group evaluation for LDAP Sync source property
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62351/ --- (Updated Sept. 20, 2017, 11:38 a.m.) Review request for Ambari, Alejandro Fernandez, Gautam Borad, Jonathan Hurley, Jayush Luniya, and Velmurugan Periasamy. Changes --- Removed config update logic from ambari-upgrade and added it during stack upgrade(2.6.x to 2.6.y) Testing done: - Test EU upgrade from 2.6.x to 2.6.y - Unit test report: mvn -DskipPythonTests -Drat.skip -Dtest=RangerUsersyncConfigCalculationTest test [INFO] --- [INFO] T E S T S [INFO] --- [INFO] Running org.apache.ambari.server.serveraction.upgrades.RangerUsersyncConfigCalculationTest [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.694 s - in org.apache.ambari.server.serveraction.upgrades.RangerUsersyncConfigCalculationTest [INFO] [INFO] Results: [INFO] [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] [INFO] --- apache-rat-plugin:0.11:check (default) @ ambari-server --- [INFO] RAT will not execute since it is configured to be skipped via system property 'rat.skip'. [INFO] [INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-server --- [INFO] skipping execute as per configuraion [INFO] [INFO] --- maven-checkstyle-plugin:2.17:check (checkstyle) @ ambari-server --- [INFO] Starting audit... Audit done. [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 01:01 min [INFO] Finished at: 2017-09-20T10:44:09+00:00 [INFO] Final Memory: 60M/870M [INFO] Bugs: AMBARI-21910 https://issues.apache.org/jira/browse/AMBARI-21910 Repository: ambari Description --- As part of RANGER-1735, we are supporting nested groups in ranger usersync. For this, introduced one new property "ranger.usersync.ldap.grouphierarchylevels". The value should be an integer with default value as 0. This property should be under Ranger --> Configs --> Ranger User Info --> Group Configs tab. This property can be enabled with a flag labelled as "Sync Nested Groups". 1. Default value for this flag is "No" 2. This flag is always shown under Group Configs (even when "Enable group Sync" is set to "No") 3. If this flag is set to "No", then the value for ranger.usersync.ldap.grouphierarchylevels is hidden and is set to "0". 4. If this flag is set to "Yes", then ranger.usersync.ldap.grouphierarchylevels is shown and allowed to set any integer greater than 0. Diffs (updated) - ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/RangerUsersyncConfigCalculation.java PRE-CREATION ambari-server/src/main/resources/common-services/RANGER/0.7.0/configuration/ranger-env.xml 627216e ambari-server/src/main/resources/common-services/RANGER/0.7.0/configuration/ranger-ugsync-site.xml a994856 ambari-server/src/main/resources/common-services/RANGER/0.7.0/themes/theme_version_5.json 8068a38 ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml 535ac2e ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml 74271cc ambari-server/src/test/java/org/apache/ambari/server/serveraction/upgrades/RangerUsersyncConfigCalculationTest.java PRE-CREATION Diff: https://reviews.apache.org/r/62351/diff/2/ Changes: https://reviews.apache.org/r/62351/diff/1-2/ Testing --- Test Fresh Installs. Ambari upgrade with stack 2.6. Unit Test passed for UpgradeCatalog260Test. Thanks, Mugdha Varadkar
Re: Review Request 62351: AMBARI-21910 Ranger Usersync config to support nested group evaluation for LDAP Sync source property
> On Sept. 15, 2017, 2:25 p.m., Jonathan Hurley wrote: > > ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog260.java > > Lines 621-658 (patched) > > <https://reviews.apache.org/r/62351/diff/1/?file=1827865#file1827865line621> > > > > This will cause Ranger configurations to change during upgrade, which > > will require a restart. We typically don't do this during Ambari upgrades > > since it would lead to extra downtime. These settings can be configured > > manually after upgrading Ambari, no? > > > > Also - we shouldn't be referring to stacks by name and version. This > > kind of tight coupling is what we're trying to get away from. You should, > > instead, query the current cluster stack to see if it supports those > > properties. Hi Jonathan Hurley, I added this change in ambari upgrade to handle scenario, where user have Ambari-2.5.0.3 and installing released HDP bits ie 2.6.3 (which will have this RANGER-1735 feature). Now they do upgrade to latest Ambari-2.6.0 so the theme will work as per these changes. It will maintain the values of existing properties added as custom. - Mugdha --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62351/#review185489 --- On Sept. 15, 2017, 12:41 p.m., Mugdha Varadkar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62351/ > --- > > (Updated Sept. 15, 2017, 12:41 p.m.) > > > Review request for Ambari, Alejandro Fernandez, Gautam Borad, Jonathan > Hurley, Jayush Luniya, and Velmurugan Periasamy. > > > Bugs: AMBARI-21910 > https://issues.apache.org/jira/browse/AMBARI-21910 > > > Repository: ambari > > > Description > --- > > As part of RANGER-1735, we are supporting nested groups in ranger usersync. > For this, introduced one new property > "ranger.usersync.ldap.grouphierarchylevels". The value should be an integer > with default value as 0. This property should be under Ranger --> Configs --> > Ranger User Info --> Group Configs tab. > This property can be enabled with a flag labelled as "Sync Nested Groups". > 1. Default value for this flag is "No" > 2. This flag is always shown under Group Configs (even when "Enable group > Sync" is set to "No") > 3. If this flag is set to "No", then the value for > ranger.usersync.ldap.grouphierarchylevels is hidden and is set to "0". > 4. If this flag is set to "Yes", then > ranger.usersync.ldap.grouphierarchylevels is shown and allowed to set any > integer greater than 0. > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog260.java > a56bda2 > > ambari-server/src/main/resources/common-services/RANGER/0.7.0/configuration/ranger-env.xml > 627216e > > ambari-server/src/main/resources/common-services/RANGER/0.7.0/configuration/ranger-ugsync-site.xml > a994856 > > ambari-server/src/main/resources/common-services/RANGER/0.7.0/themes/theme_version_5.json > 8068a38 > > ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog260Test.java > e363c8d > > > Diff: https://reviews.apache.org/r/62351/diff/1/ > > > Testing > --- > > Test Fresh Installs. > Ambari upgrade with stack 2.6. > Unit Test passed for UpgradeCatalog260Test. > > > Thanks, > > Mugdha Varadkar > >
Review Request 62351: AMBARI-21910 Ranger Usersync config to support nested group evaluation for LDAP Sync source property
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62351/ --- Review request for Ambari, Alejandro Fernandez, Gautam Borad, Jonathan Hurley, Jayush Luniya, and Velmurugan Periasamy. Bugs: AMBARI-21910 https://issues.apache.org/jira/browse/AMBARI-21910 Repository: ambari Description --- As part of RANGER-1735, we are supporting nested groups in ranger usersync. For this, introduced one new property "ranger.usersync.ldap.grouphierarchylevels". The value should be an integer with default value as 0. This property should be under Ranger --> Configs --> Ranger User Info --> Group Configs tab. This property can be enabled with a flag labelled as "Sync Nested Groups". 1. Default value for this flag is "No" 2. This flag is always shown under Group Configs (even when "Enable group Sync" is set to "No") 3. If this flag is set to "No", then the value for ranger.usersync.ldap.grouphierarchylevels is hidden and is set to "0". 4. If this flag is set to "Yes", then ranger.usersync.ldap.grouphierarchylevels is shown and allowed to set any integer greater than 0. Diffs - ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog260.java a56bda2 ambari-server/src/main/resources/common-services/RANGER/0.7.0/configuration/ranger-env.xml 627216e ambari-server/src/main/resources/common-services/RANGER/0.7.0/configuration/ranger-ugsync-site.xml a994856 ambari-server/src/main/resources/common-services/RANGER/0.7.0/themes/theme_version_5.json 8068a38 ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog260Test.java e363c8d Diff: https://reviews.apache.org/r/62351/diff/1/ Testing --- Test Fresh Installs. Ambari upgrade with stack 2.6. Unit Test passed for UpgradeCatalog260Test. Thanks, Mugdha Varadkar
Re: Review Request 62239: AMBARI-21925 : Facilitate a parameter to provide HBase conf directory for atlas-env configurations.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62239/#review185154 --- Ship it! Ship It! - Mugdha Varadkar On Sept. 12, 2017, 9:03 a.m., Vishal Suvagia wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62239/ > --- > > (Updated Sept. 12, 2017, 9:03 a.m.) > > > Review request for Ambari, Alejandro Fernandez, Gautam Borad, Jayush Luniya, > Madhan Neethiraj, Mugdha Varadkar, and Sumit Mohanty. > > > Bugs: AMBARI-21925 > https://issues.apache.org/jira/browse/AMBARI-21925 > > > Repository: ambari > > > Description > --- > > Current configuration for Atlas atlas-env contains hbase_conf_dir which is > set in environment as export HBASE_CONF_DIR=hbase_conf_dir > However, hbase_conf_dir is currently not exposed in ambari and hence its not > possible to set just hbase_conf_dir instead through blueprints we need to > pass full atlas-env.sh content to set the hbase_conf_dir. > > > Diffs > - > > > ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py > 96041e6 > > ambari-server/src/main/resources/common-services/ATLAS/0.7.0.3.0/configuration/atlas-env.xml > f97ca98 > > ambari-server/src/main/resources/common-services/ATLAS/0.7.0.3.0/package/scripts/params.py > 61b41d6 > > ambari-server/src/main/resources/stacks/HDP/2.6/services/ATLAS/configuration/atlas-env.xml > 821fafa > ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml > 4764297 > > ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml > 0d9a746 > ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml > b5c88de > > > Diff: https://reviews.apache.org/r/62239/diff/1/ > > > Testing > --- > > Verified with fresh install on Cent-OS 6. > > > Thanks, > > Vishal Suvagia > >
Re: Review Request 62157: AMBARI-21904 Remove redundant smokeuser entry from Ranger KMS Kerberos descriptor
> On Sept. 7, 2017, 5:40 p.m., Robert Levas wrote: > > ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/kerberos.json > > Lines 37-39 (original), 34-36 (patched) > > <https://reviews.apache.org/r/62157/diff/1/?file=1817582#file1817582line37> > > > > Shouldn't this be removed as well? Updated the change in latest patch - Mugdha --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62157/#review184846 --- On Sept. 8, 2017, 7 a.m., Mugdha Varadkar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62157/ > --- > > (Updated Sept. 8, 2017, 7 a.m.) > > > Review request for Ambari, Alejandro Fernandez, Gautam Borad, and Robert > Levas. > > > Bugs: AMBARI-21904 > https://issues.apache.org/jira/browse/AMBARI-21904 > > > Repository: ambari > > > Description > --- > > /smokeuser entry in RANGER_KMS_SERVER component section is not needed. It > wont break anything, but it is redundant and should eventually be removed. > (Feedback from review request - https://reviews.apache.org/r/34561/) > > > Diffs > - > > > ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog260.java > 07ae0c2 > > ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/kerberos.json > 69d6b6c > > ambari-server/src/main/resources/stacks/HDP/2.5/services/RANGER_KMS/kerberos.json > a54783e > > ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog260Test.java > 2e16754 > > ambari-server/src/test/resources/kerberos/test_kerberos_descriptor_ranger_kms.json > PRE-CREATION > > > Diff: https://reviews.apache.org/r/62157/diff/2/ > > > Testing > --- > > mvn -DskipPythonTests -Drat.skip -Dtest=UpgradeCatalog260Test test > > [INFO] --- > [INFO] T E S T S > [INFO] --- > [INFO] Running org.apache.ambari.server.upgrade.UpgradeCatalog260Test > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.772 > s - in org.apache.ambari.server.upgrade.UpgradeCatalog260Test > [INFO] > [INFO] Results: > [INFO] > [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0 > [INFO] > [INFO] > [INFO] --- apache-rat-plugin:0.11:check (default) @ ambari-server --- > [INFO] RAT will not execute since it is configured to be skipped via system > property 'rat.skip'. > [INFO] > [INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-server --- > [INFO] skipping execute as per configuraion > [INFO] > [INFO] --- maven-checkstyle-plugin:2.17:check (checkstyle) @ ambari-server --- > [INFO] Starting audit... > Audit done. > [INFO] > > [INFO] BUILD SUCCESS > [INFO] > ---- > [INFO] Total time: 42.247 s > [INFO] Finished at: 2017-09-07T14:28:04+00:00 > [INFO] Final Memory: 52M/803M > [INFO] > > > > Thanks, > > Mugdha Varadkar > >
Re: Review Request 62157: AMBARI-21904 Remove redundant smokeuser entry from Ranger KMS Kerberos descriptor
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62157/ --- (Updated Sept. 8, 2017, 7 a.m.) Review request for Ambari, Alejandro Fernandez, Gautam Borad, and Robert Levas. Bugs: AMBARI-21904 https://issues.apache.org/jira/browse/AMBARI-21904 Repository: ambari Description --- /smokeuser entry in RANGER_KMS_SERVER component section is not needed. It wont break anything, but it is redundant and should eventually be removed. (Feedback from review request - https://reviews.apache.org/r/34561/) Diffs (updated) - ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog260.java 07ae0c2 ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/kerberos.json 69d6b6c ambari-server/src/main/resources/stacks/HDP/2.5/services/RANGER_KMS/kerberos.json a54783e ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog260Test.java 2e16754 ambari-server/src/test/resources/kerberos/test_kerberos_descriptor_ranger_kms.json PRE-CREATION Diff: https://reviews.apache.org/r/62157/diff/2/ Changes: https://reviews.apache.org/r/62157/diff/1-2/ Testing --- mvn -DskipPythonTests -Drat.skip -Dtest=UpgradeCatalog260Test test [INFO] --- [INFO] T E S T S [INFO] --- [INFO] Running org.apache.ambari.server.upgrade.UpgradeCatalog260Test [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.772 s - in org.apache.ambari.server.upgrade.UpgradeCatalog260Test [INFO] [INFO] Results: [INFO] [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] [INFO] --- apache-rat-plugin:0.11:check (default) @ ambari-server --- [INFO] RAT will not execute since it is configured to be skipped via system property 'rat.skip'. [INFO] [INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-server --- [INFO] skipping execute as per configuraion [INFO] [INFO] --- maven-checkstyle-plugin:2.17:check (checkstyle) @ ambari-server --- [INFO] Starting audit... Audit done. [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 42.247 s [INFO] Finished at: 2017-09-07T14:28:04+00:00 [INFO] Final Memory: 52M/803M [INFO] Thanks, Mugdha Varadkar
Review Request 62157: AMBARI-21904 Remove redundant smokeuser entry from Ranger KMS Kerberos descriptor
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62157/ --- Review request for Ambari, Alejandro Fernandez, Gautam Borad, and Robert Levas. Bugs: AMBARI-21904 https://issues.apache.org/jira/browse/AMBARI-21904 Repository: ambari Description --- /smokeuser entry in RANGER_KMS_SERVER component section is not needed. It wont break anything, but it is redundant and should eventually be removed. (Feedback from review request - https://reviews.apache.org/r/34561/) Diffs - ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog260.java f2eac97 ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/kerberos.json 69d6b6c ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog260Test.java 427cadd ambari-server/src/test/resources/kerberos/test_kerberos_descriptor_ranger_kms.json PRE-CREATION Diff: https://reviews.apache.org/r/62157/diff/1/ Testing --- mvn -DskipPythonTests -Drat.skip -Dtest=UpgradeCatalog260Test test [INFO] --- [INFO] T E S T S [INFO] --- [INFO] Running org.apache.ambari.server.upgrade.UpgradeCatalog260Test [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.772 s - in org.apache.ambari.server.upgrade.UpgradeCatalog260Test [INFO] [INFO] Results: [INFO] [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] [INFO] --- apache-rat-plugin:0.11:check (default) @ ambari-server --- [INFO] RAT will not execute since it is configured to be skipped via system property 'rat.skip'. [INFO] [INFO] --- exec-maven-plugin:1.2.1:exec (python-test) @ ambari-server --- [INFO] skipping execute as per configuraion [INFO] [INFO] --- maven-checkstyle-plugin:2.17:check (checkstyle) @ ambari-server --- [INFO] Starting audit... Audit done. [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 42.247 s [INFO] Finished at: 2017-09-07T14:28:04+00:00 [INFO] Final Memory: 52M/803M [INFO] Thanks, Mugdha Varadkar
Review Request 62117: AMBARI-21889 MaxBackupIndex does not work with DailyRollingFileAppender in Ranger
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62117/ --- Review request for Ambari, Alejandro Fernandez and Gautam Borad. Bugs: AMBARI-21889 https://issues.apache.org/jira/browse/AMBARI-21889 Repository: ambari Description --- Ranger uses DailyRollingFileAppender and "maxbackupindex" is not in that class so need to remove the same from admin-log4j config. Diffs - ambari-server/src/main/resources/common-services/RANGER/0.6.0/configuration/admin-log4j.xml 6108c36 ambari-server/src/main/resources/common-services/RANGER/0.6.0/configuration/tagsync-log4j.xml 6384302 ambari-server/src/main/resources/common-services/RANGER/0.6.0/configuration/usersync-log4j.xml 8843a2a ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/configuration/kms-log4j.xml bac2e84 ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml 4b53c7c ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml 29fcac0 ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 0c4106e ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml 3e62fb1 ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/config-upgrade.xml 970e19f Diff: https://reviews.apache.org/r/62117/diff/1/ Testing --- Tested Ranger Installation. Thanks, Mugdha Varadkar
Re: Review Request 62015: Ranger should handle external Zookeeper properly
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62015/#review184342 --- Ship it! Ship It! - Mugdha Varadkar On Aug. 31, 2017, 8:41 p.m., Miklos Gergely wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62015/ > --- > > (Updated Aug. 31, 2017, 8:41 p.m.) > > > Review request for Ambari, Gautam Borad, Mugdha Varadkar, Oliver Szabo, > Robert Nettleton, Vishal Suvagia, and Velmurugan Periasamy. > > > Bugs: AMBARI-21863 > https://issues.apache.org/jira/browse/AMBARI-21863 > > > Repository: ambari > > > Description > --- > > Ranger tries to create it's configurations using the local ambari-infra Solr > even if it is specified to use an external one. > > > Diffs > - > > > ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py > bd45558 > > > Diff: https://reviews.apache.org/r/62015/diff/1/ > > > Testing > --- > > Tested on two clusters, one running the external Zookeeper, the other > installed Ranger, the config data were created. > > > Thanks, > > Miklos Gergely > >
Re: Review Request 61713: AMBARI-21743 Delete ranger-solr-plugin-enabled property during Cross Stack Upgrade
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61713/ --- (Updated Aug. 17, 2017, 1:33 p.m.) Review request for Ambari, Alejandro Fernandez, Gautam Borad, Oliver Szabo, and Sumit Mohanty. Bugs: AMBARI-21743 https://issues.apache.org/jira/browse/AMBARI-21743 Repository: ambari Description --- For Cross stack upgrade, after upgrading to Ambari-2.5 and then upgrading to stack 2.6, ranger-solr-plugin-enabled property in ranger-env config type is not used. Hence need to delete this config during upgrade. Diffs - ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml 6ed6a11 ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml e5f3690 Diff: https://reviews.apache.org/r/61713/diff/1/ Testing --- Tested EU from IOP-4.2.5 to HDP-2.6.x Thanks, Mugdha Varadkar
Review Request 61713: AMBARI-21743 Delete ranger-solr-plugin-enabled property during Cross Stack Upgrade
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61713/ --- Review request for Ambari, Alejandro Fernandez, Gautam Borad, Oliver Szabo, and Sumit Mohanty. Bugs: AMBARI-21743 https://issues.apache.org/jira/browse/AMBARI-21743 Repository: ambari Description --- For Cross stack upgrade, after upgrading to Ambari-2.5 and then upgrading to stack 2.6, ranger-solr-plugin-enabled property in ranger-env config type is not used. Hence need to delete this config during upgrade. Diffs - ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/config-upgrade.xml 6ed6a11 ambari-server/src/main/resources/stacks/BigInsights/4.2.5/upgrades/nonrolling-upgrade-to-hdp-2.6.xml e5f3690 Diff: https://reviews.apache.org/r/61713/diff/1/ Testing --- Tested EU from IOP-4.2.5 to HDP-2.6.x Thanks, Mugdha Varadkar
Re: Review Request 61525: AMBARI-21690 Updating get_stack_version pattern match
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61525/ --- (Updated Aug. 11, 2017, 4:44 a.m.) Review request for Ambari, Alejandro Fernandez, Gautam Borad, and Jonathan Hurley. Bugs: AMBARI-21690 https://issues.apache.org/jira/browse/AMBARI-21690 Repository: ambari Description --- Currently get_stack_version only accepts stack version with -buildnumber at the end. Need to change stack_version pattern match for stack_version without -buildnumber. For cross stack upgrade from IOP to HDP, if user restart services after Ambari upgrade ie before upgrading to HDP stack 2.6, the get_stack_version returns None. This affects Ranger-related functionality which depends on stack_version for generating file path. The problem occured for cluster with IOP - 4.2.0.0 with latest Ambari, which doesn't have -buildnumber at the end, so 4.2.0.0 is not matched and stack_version=None. Diffs - ambari-common/src/main/python/resource_management/libraries/functions/get_stack_version.py 7274a59 Diff: https://reviews.apache.org/r/61525/diff/2/ Testing --- Tested: 1)Services restart after ambari upgrade with IOP 4.2.0.0 2)Fresh installs with HDP. 3)EU from IOP to HDP-2.6.x and HDP-2.6.x to HDP-2.6.y Unit test report: Ran 274 tests in 6.820s OK -- Total run:1162 Total errors:0 Total failures:0 OK Thanks, Mugdha Varadkar
Re: Review Request 61527: AMBARI-21649 : Update Atlas log4j to reflect latest configurations
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61527/#review182559 --- Ship it! Ship It! - Mugdha Varadkar On Aug. 10, 2017, 5:42 a.m., Vishal Suvagia wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/61527/ > --- > > (Updated Aug. 10, 2017, 5:42 a.m.) > > > Review request for Ambari, Alejandro Fernandez, Gautam Borad, Jayush Luniya, > Madhan Neethiraj, Mugdha Varadkar, and Sumit Mohanty. > > > Bugs: AMBARI-21649 > https://issues.apache.org/jira/browse/AMBARI-21649 > > > Repository: ambari > > > Description > --- > > Need to update Atlas log4j configurations to reflect latest configurations as > per update in Atlas. > > PS: Attached patch is for trunk, seperate patch for branch-2.6 is attached on > Apache JIRA. > > > Diffs > - > > > ambari-server/src/main/resources/common-services/ATLAS/0.7.0.3.0/configuration/atlas-log4j.xml > bafd47d > ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/config-upgrade.xml > ec6ee55 > > ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml > e9490b2 > ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml > 840b17d > > ambari-server/src/main/resources/stacks/HDP/2.6/services/ATLAS/configuration/atlas-log4j.xml > 9e3abcf > ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml > c2c1532 > > ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml > 86b3956 > ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml > b376fa7 > > > Diff: https://reviews.apache.org/r/61527/diff/2/ > > > Testing > --- > > Verified with fresh install and upgrade scenarios on CentOS 6. > > > Thanks, > > Vishal Suvagia > >
Re: Review Request 61525: AMBARI-21690 Updating get_stack_version pattern match
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61525/ --- (Updated Aug. 10, 2017, 6:29 a.m.) Review request for Ambari, Alejandro Fernandez, Gautam Borad, and Jonathan Hurley. Bugs: AMBARI-21690 https://issues.apache.org/jira/browse/AMBARI-21690 Repository: ambari Description --- Currently get_stack_version only accepts stack version with -buildnumber at the end. Need to change stack_version pattern match for stack_version without -buildnumber. For cross stack upgrade from IOP to HDP, if user restart services after Ambari upgrade ie before upgrading to HDP stack 2.6, the get_stack_version returns None. This affects Ranger-related functionality which depends on stack_version for generating file path. The problem occured for cluster with IOP - 4.2.0.0 with latest Ambari, which doesn't have -buildnumber at the end, so 4.2.0.0 is not matched and stack_version=None. Diffs (updated) - ambari-common/src/main/python/resource_management/libraries/functions/get_stack_version.py 7274a59 Diff: https://reviews.apache.org/r/61525/diff/2/ Changes: https://reviews.apache.org/r/61525/diff/1-2/ Testing --- Tested: 1)Services restart after ambari upgrade with IOP 4.2.0.0 2)Fresh installs with HDP. 3)EU from IOP to HDP-2.6.x and HDP-2.6.x to HDP-2.6.y Unit test report: Ran 274 tests in 6.820s OK -- Total run:1162 Total errors:0 Total failures:0 OK Thanks, Mugdha Varadkar
Re: Review Request 61525: AMBARI-21690 Updating get_stack_version pattern match
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61525/ --- (Updated Aug. 9, 2017, 1:07 p.m.) Review request for Ambari, Alejandro Fernandez, Gautam Borad, and Jonathan Hurley. Bugs: AMBARI-21690 https://issues.apache.org/jira/browse/AMBARI-21690 Repository: ambari Description --- Currently get_stack_version only accepts stack version with -buildnumber at the end. Need to change stack_version pattern match for stack_version without -buildnumber. For cross stack upgrade from IOP to HDP, if user restart services after Ambari upgrade ie before upgrading to HDP stack 2.6, the get_stack_version returns None. This affects Ranger-related functionality which depends on stack_version for generating file path. The problem occured for cluster with IOP - 4.2.0.0 with latest Ambari, which doesn't have -buildnumber at the end, so 4.2.0.0 is not matched and stack_version=None. Diffs - ambari-common/src/main/python/resource_management/libraries/functions/get_stack_version.py 7274a59 Diff: https://reviews.apache.org/r/61525/diff/1/ Testing --- Tested: 1)Services restart after ambari upgrade with IOP 4.2.0.0 2)Fresh installs with HDP. 3)EU from IOP to HDP-2.6.x and HDP-2.6.x to HDP-2.6.y Unit test report: Ran 274 tests in 6.820s OK -- Total run:1162 Total errors:0 Total failures:0 OK Thanks, Mugdha Varadkar
Review Request 61525: AMBARI-21690 Updating get_stack_version pattern match
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61525/ --- Review request for Ambari, Alejandro Fernandez, Gautam Borad, and Jonathan Hurley. Repository: ambari Description --- Currently get_stack_version only accepts stack version with -buildnumber at the end. Need to change stack_version pattern match for stack_version without -buildnumber. For cross stack upgrade from IOP to HDP, if user restart services after Ambari upgrade ie before upgrading to HDP stack 2.6, the get_stack_version returns None. This affects Ranger-related functionality which depends on stack_version for generating file path. The problem occured for cluster with IOP - 4.2.0.0 with latest Ambari, which doesn't have -buildnumber at the end, so 4.2.0.0 is not matched and stack_version=None. Diffs - ambari-common/src/main/python/resource_management/libraries/functions/get_stack_version.py 7274a59 Diff: https://reviews.apache.org/r/61525/diff/1/ Testing --- Tested: 1)Services restart after ambari upgrade with IOP 4.2.0.0 2)Fresh installs with HDP. 3)EU from IOP to HDP-2.6.x and HDP-2.6.x to HDP-2.6.y Unit test report: Ran 274 tests in 6.820s OK -- Total run:1162 Total errors:0 Total failures:0 OK Thanks, Mugdha Varadkar
Review Request 61405: AMBARI-21650 Ranger Usersync process starts with older stack script during cross stack upgrade
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61405/ --- Review request for Ambari, Alejandro Fernandez, Gautam Borad, and Sumit Mohanty. Bugs: AMBARI-21650 https://issues.apache.org/jira/browse/AMBARI-21650 Repository: ambari Description --- EU from IOP(4.2.5) stack to HDP(2.6.2) starts Ranger Usersync process for IOP stack. This is because the symbolic link used for starting Ranger Usersync process points to IOP Ranger Usersync script file /usr/bin/ranger-usersync -> /usr/iop/current/ranger-usersync/ranger-usersync-services.sh Diffs - ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py 2fa7b2f Diff: https://reviews.apache.org/r/61405/diff/1/ Testing --- Tested EU from IOP to HDP and HDP-2.5.x to HDP-2.6.x Thanks, Mugdha Varadkar
Re: Review Request 61376: AMBARI-21635 Show Audit to DB removal pre-check message during cross stack upgrade
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61376/
---
(Updated Aug. 3, 2017, 5:09 a.m.)
Review request for Ambari, Alejandro Fernandez and Gautam Borad.
Bugs: AMBARI-21635
https://issues.apache.org/jira/browse/AMBARI-21635
Repository: ambari
Description
---
Need to add "Audit to DB removal "pre-check message during cross stack upgrade
from IOP 4.2 to HDP-2.6.2.
Also providing below changes:
Removing hdp_2_5_0_0_set_external_solrCloud_flag task from config-upgrade.xml
for IOP 4.2 to HDP-2.6.2
Adding below missing features which were added in Ambari-2.5 in HDP stack for
Ranger
{
"name": "ranger_admin_password_change",
"description": "Allow ranger admin credentials to be specified during cluster
creation (AMBARI-17000)",
"min_version": "4.2.5.0"
},
{
"name": "ranger_hive_plugin_jdbc_url",
"description": "Handle Ranger hive repo config jdbc url change for stack 2.5
(AMBARI-18386)",
"min_version": "4.2.5.0"
},
{
"name": "ranger_xml_configuration",
"description": "Ranger code base support xml configurations",
"min_version": "4.2.0.0"
},
{
"name": "kafka_ranger_plugin_support",
"description": "Ambari stack changes for Ranger Kafka Plugin (AMBARI-11299)",
"min_version": "4.2.0.0"
},
{
"name": "yarn_ranger_plugin_support",
"description": "Implement Stack changes for Ranger Yarn Plugin integration
(AMBARI-10866)",
"min_version": "4.2.0.0"
}
Diffs
-
ambari-server/src/main/resources/stacks/BigInsights/4.0/properties/stack_features.json
a6672e4
ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml
17e80b5
ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml
55736cf
Diff: https://reviews.apache.org/r/61376/diff/1/
Testing
---
Total run:1161
Total errors:0
Total failures:0
OK
Thanks,
Mugdha Varadkar
Review Request 61376: AMBARI-21635 Show Audit to DB removal pre-check message during cross stack upgrade
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61376/
---
Review request for Ambari, Alejandro Fernandez and Gautam Borad.
Bugs: AMBARI-21635
https://issues.apache.org/jira/browse/AMBARI-21635
Repository: ambari
Description
---
Need to add "Audit to DB removal "pre-check message during cross stack upgrade
from IOP 4.2 to HDP-2.6.2.
Also providing below changes:
Removing hdp_2_5_0_0_set_external_solrCloud_flag task from config-upgrade.xml
for IOP 4.2 to HDP-2.6.2
Adding below missing features which were added in Ambari-2.5 in HDP stack for
Ranger
{
"name": "ranger_admin_password_change",
"description": "Allow ranger admin credentials to be specified during cluster
creation (AMBARI-17000)",
"min_version": "4.2.5.0"
},
{
"name": "ranger_hive_plugin_jdbc_url",
"description": "Handle Ranger hive repo config jdbc url change for stack 2.5
(AMBARI-18386)",
"min_version": "4.2.5.0"
},
{
"name": "ranger_xml_configuration",
"description": "Ranger code base support xml configurations",
"min_version": "4.2.0.0"
},
{
"name": "kafka_ranger_plugin_support",
"description": "Ambari stack changes for Ranger Kafka Plugin (AMBARI-11299)",
"min_version": "4.2.0.0"
},
{
"name": "yarn_ranger_plugin_support",
"description": "Implement Stack changes for Ranger Yarn Plugin integration
(AMBARI-10866)",
"min_version": "4.2.0.0"
}
Diffs
-
ambari-server/src/main/resources/stacks/BigInsights/4.0/properties/stack_features.json
a6672e4
ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/config-upgrade.xml
17e80b5
ambari-server/src/main/resources/stacks/BigInsights/4.2/upgrades/nonrolling-upgrade-to-hdp-2.6.xml
55736cf
Diff: https://reviews.apache.org/r/61376/diff/1/
Testing
---
Total run:1161
Total errors:0
Total failures:0
OK
Thanks,
Mugdha Varadkar
Re: Review Request 61217: AMBARI-21588 : MetadataServer start fails if Atlas service is started with custom service user.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61217/#review181671 --- Ship it! Ship It! - Mugdha Varadkar On July 28, 2017, 2:13 p.m., Vishal Suvagia wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/61217/ > --- > > (Updated July 28, 2017, 2:13 p.m.) > > > Review request for Ambari, Alejandro Fernandez, Gautam Borad, Jayush Luniya, > Madhan Neethiraj, Mugdha Varadkar, and Sumit Mohanty. > > > Bugs: AMBARI-21588 > https://issues.apache.org/jira/browse/AMBARI-21588 > > > Repository: ambari > > > Description > --- > > File permissons for stack_root/current/atlas-server/server/webapp/atlas is > root:root, when atlas user is custom, Atlas Metadata Server can not start. > After changing permissions to the custom atlas user then Atlas was > successfully started. > > > Diffs > - > > > ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py > 74b1688 > ambari-server/src/test/python/stacks/2.3/ATLAS/test_metadata_server.py > d1d8b7f > ambari-server/src/test/python/stacks/2.5/ATLAS/test_atlas_server.py 539bef5 > > > Diff: https://reviews.apache.org/r/61217/diff/1/ > > > Testing > --- > > Verified with install on CentOS 6. > > > Thanks, > > Vishal Suvagia > >
Review Request 61105: Auto populate zookeeper.connect in Ranger Kafka plugin properties
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61105/ --- Review request for Ambari, Alejandro Fernandez and Gautam Borad. Bugs: AMBARI-21560 https://issues.apache.org/jira/browse/AMBARI-21560 Repository: ambari Description --- Auto populate zookeeper.connect in ranger-kafka-plugin-properties which always points to the localhost:2181, that cause test connection and resource lookup failure in Kafka service created on Ranger Admin. Patch for trunk is attached on https://issues.apache.org/jira/secure/attachment/12878770/AMBARI-21560-trunk.patch Diffs - ambari-server/src/main/resources/common-services/KAFKA/0.9.0/configuration/ranger-kafka-plugin-properties.xml 3949402 ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py d0f82af ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py fe3d26d Diff: https://reviews.apache.org/r/61105/diff/1/ Testing --- Unit Test Report: test_recommendKAFKAConfigurations (test_stack_advisor.TestHDP23StackAdvisor) ... ok Thanks, Mugdha Varadkar
Re: Review Request 61083: AMBARI-21563 : Copy Hadoop core-site.xml to Atlas conf when hadoop is configured for LDAP/AD.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61083/#review181318 --- Ship it! Ship It! - Mugdha Varadkar On July 24, 2017, 2:51 p.m., Vishal Suvagia wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/61083/ > --- > > (Updated July 24, 2017, 2:51 p.m.) > > > Review request for Ambari, Alejandro Fernandez, Gautam Borad, Jayush Luniya, > Madhan Neethiraj, Mugdha Varadkar, Nixon Rodrigues, and Sumit Mohanty. > > > Bugs: AMBARI-21563 > https://issues.apache.org/jira/browse/AMBARI-21563 > > > Repository: ambari > > > Description > --- > > Atlas use user groups synced in hadoopUGI for authentication and > authorization process for LDAP/Kerberos type. > As core-site.xml was available in HBase conf which is in Atlas classpath and > so core.site.xml configs were available. But now as core-site.xml is not > available in Hbase conf, to fix this issue hadoop core-site.xml need to be > copied to Atlas conf directory by Ambari. > > PS: Attached patch is applicable for trunk branch, patch for branch-2.5 is > attached on Apache JIRA. > > > Diffs > - > > > ambari-common/src/main/python/resource_management/libraries/functions/constants.py > c74e029 > > ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py > d424f5b > > ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py > 3ed469a > > ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json > 31cf0c8 > ambari-server/src/main/resources/stacks/HDP/2.6/services/ATLAS/metainfo.xml > 6e1a52e > ambari-server/src/test/python/stacks/2.3/ATLAS/test_metadata_server.py > d1d8b7f > ambari-server/src/test/python/stacks/2.5/ATLAS/test_atlas_server.py 539bef5 > > > Diff: https://reviews.apache.org/r/61083/diff/1/ > > > Testing > --- > > Verified with installation on CentOS 6. > > > Thanks, > > Vishal Suvagia > >
Re: Review Request 60842: AMBARI-21460 : Add new kafka client properties to the ambari managed atlas config
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/60842/#review180511 --- Ship it! Ship It! - Mugdha Varadkar On July 13, 2017, 3:24 p.m., Vishal Suvagia wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/60842/ > --- > > (Updated July 13, 2017, 3:24 p.m.) > > > Review request for Ambari, Alejandro Fernandez, Ayub Pathan, Gautam Borad, > Jayush Luniya, Madhan Neethiraj, Mugdha Varadkar, Nixon Rodrigues, and Sumit > Mohanty. > > > Bugs: AMBARI-21460 > https://issues.apache.org/jira/browse/AMBARI-21460 > > > Repository: ambari > > > Description > --- > > As part of latest changes in Atlas Kafka configuration below new kafka client > properties are introduced. > > atlas.kafka.enable.auto.commit=false > atlas.kafka.session.timeout.ms=3 > > Need to add this to the Ambari managed Atlas configurations. > PS: Attached patch is for trunk, seperate patch for branch-2.5 is attached on > Apache JIRA. > > > Diffs > - > > ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/config-upgrade.xml > 30796cc > > ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml > 1f37389 > ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml > 22c9a8d > > ambari-server/src/main/resources/stacks/HDP/2.6/services/ATLAS/configuration/application-properties.xml > 91de1b0 > ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml > 6dd2129 > > ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml > e262971 > ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml > 6b01ce9 > > > Diff: https://reviews.apache.org/r/60842/diff/1/ > > > Testing > --- > > Verified with installation on CentOS 6. > > > Thanks, > > Vishal Suvagia > >
Re: Review Request 60560: AMBARI-21374 : Replication factor should be configurable in Atlas to create solr collections
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/60560/#review179340 --- Ship it! Ship It! - Mugdha Varadkar On June 30, 2017, 8:49 a.m., Vishal Suvagia wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/60560/ > --- > > (Updated June 30, 2017, 8:49 a.m.) > > > Review request for Ambari, Alejandro Fernandez, Gautam Borad, Jayush Luniya, > Madhan Neethiraj, Mugdha Varadkar, Nixon Rodrigues, and Sumit Mohanty. > > > Bugs: AMBARI-21374 > https://issues.apache.org/jira/browse/AMBARI-21374 > > > Repository: ambari > > > Description > --- > > In current scenarios, Solr collections created for Atlas, has > replication-factor dependent on the no of Infra-Solr servers present in > Ambari, However to be more modular we should allow the replication factor to > be configurable for the user. > PS: Attached patch is for trunk branch, patch for branch-2.5 is attached to > Apache JIRA. > > > Diffs > - > > > ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py > 111a248 > > ambari-server/src/main/resources/common-services/ATLAS/0.7.0.3.0/configuration/atlas-env.xml > c5a4fd6 > > ambari-server/src/main/resources/common-services/ATLAS/0.7.0.3.0/package/scripts/params.py > d26df33 > > ambari-server/src/main/resources/stacks/HDP/2.6/services/ATLAS/configuration/atlas-env.xml > d54cf15 > > > Diff: https://reviews.apache.org/r/60560/diff/1/ > > > Testing > --- > > Verified with installation on Cent-OS 6. > > > Thanks, > > Vishal Suvagia > >
Review Request 60460: AMBARI-21058 HDP 3.0 - Changing common service version for Ranger & Ranger Kms
/0.5.0.3.0/configuration/ranger-kms-policymgr-ssl.xml 9eedc73 ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.3.0/configuration/ranger-kms-security.xml 13adcb4 ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.3.0/configuration/ranger-kms-site.xml 1d32f72 ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.3.0/kerberos.json a54783e ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.3.0/metainfo.xml 24ac51f ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.3.0/package/scripts/kms.py 5a25b92 ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.3.0/package/scripts/kms_server.py 44d61da ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.3.0/package/scripts/kms_service.py 2ff48c3 ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.3.0/package/scripts/params.py 2445f2e ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.3.0/package/scripts/service_check.py 84e4e73 ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.3.0/package/scripts/status_params.py 34d0082 ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.3.0/package/scripts/upgrade.py 8478bb8 ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.3.0/package/templates/input.config-ranger-kms.json.j2 306fade ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.3.0/role_command_order.json 7ddab41 ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.3.0/service_advisor.py 9c33218 ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.3.0/themes/theme_version_1.json c08a56c ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.3.0/themes/theme_version_2.json be50dad ambari-server/src/main/resources/common-services/RANGER_KMS/1.0.0.3.0/alerts.json PRE-CREATION ambari-server/src/main/resources/common-services/RANGER_KMS/1.0.0.3.0/configuration/dbks-site.xml PRE-CREATION ambari-server/src/main/resources/common-services/RANGER_KMS/1.0.0.3.0/configuration/kms-env.xml PRE-CREATION ambari-server/src/main/resources/common-services/RANGER_KMS/1.0.0.3.0/configuration/kms-log4j.xml PRE-CREATION ambari-server/src/main/resources/common-services/RANGER_KMS/1.0.0.3.0/configuration/kms-properties.xml PRE-CREATION ambari-server/src/main/resources/common-services/RANGER_KMS/1.0.0.3.0/configuration/kms-site.xml PRE-CREATION ambari-server/src/main/resources/common-services/RANGER_KMS/1.0.0.3.0/configuration/ranger-kms-audit.xml PRE-CREATION ambari-server/src/main/resources/common-services/RANGER_KMS/1.0.0.3.0/configuration/ranger-kms-policymgr-ssl.xml PRE-CREATION ambari-server/src/main/resources/common-services/RANGER_KMS/1.0.0.3.0/configuration/ranger-kms-security.xml PRE-CREATION ambari-server/src/main/resources/common-services/RANGER_KMS/1.0.0.3.0/configuration/ranger-kms-site.xml PRE-CREATION ambari-server/src/main/resources/common-services/RANGER_KMS/1.0.0.3.0/kerberos.json PRE-CREATION ambari-server/src/main/resources/common-services/RANGER_KMS/1.0.0.3.0/metainfo.xml PRE-CREATION ambari-server/src/main/resources/common-services/RANGER_KMS/1.0.0.3.0/package/scripts/kms.py PRE-CREATION ambari-server/src/main/resources/common-services/RANGER_KMS/1.0.0.3.0/package/scripts/kms_server.py PRE-CREATION ambari-server/src/main/resources/common-services/RANGER_KMS/1.0.0.3.0/package/scripts/kms_service.py PRE-CREATION ambari-server/src/main/resources/common-services/RANGER_KMS/1.0.0.3.0/package/scripts/params.py PRE-CREATION ambari-server/src/main/resources/common-services/RANGER_KMS/1.0.0.3.0/package/scripts/service_check.py PRE-CREATION ambari-server/src/main/resources/common-services/RANGER_KMS/1.0.0.3.0/package/scripts/status_params.py PRE-CREATION ambari-server/src/main/resources/common-services/RANGER_KMS/1.0.0.3.0/package/scripts/upgrade.py PRE-CREATION ambari-server/src/main/resources/common-services/RANGER_KMS/1.0.0.3.0/package/templates/input.config-ranger-kms.json.j2 PRE-CREATION ambari-server/src/main/resources/common-services/RANGER_KMS/1.0.0.3.0/role_command_order.json PRE-CREATION ambari-server/src/main/resources/common-services/RANGER_KMS/1.0.0.3.0/service_advisor.py PRE-CREATION ambari-server/src/main/resources/common-services/RANGER_KMS/1.0.0.3.0/themes/theme_version_1.json PRE-CREATION ambari-server/src/main/resources/stacks/HDP/3.0/services/RANGER/metainfo.xml c8b3d65 ambari-server/src/main/resources/stacks/HDP/3.0/services/RANGER_KMS/metainfo.xml 3375d90 Diff: https://reviews.apache.org/r/60460/diff/1/ Testing --- mvn clean test Thanks, Mugdha Varadkar
Re: Review Request 60388: AMBARI-21256 : As part of START_ALL of services Ranger kms starts after hbase and hive causing operation failure
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/60388/#review178872 --- Ship it! Ship It! - Mugdha Varadkar On June 23, 2017, 1:13 p.m., Vishal Suvagia wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/60388/ > --- > > (Updated June 23, 2017, 1:13 p.m.) > > > Review request for Ambari, Alejandro Fernandez, Gautam Borad, Jayush Luniya, > Mugdha Varadkar, Sumit Mohanty, and Velmurugan Periasamy. > > > Bugs: AMBARI-21256 > https://issues.apache.org/jira/browse/AMBARI-21256 > > > Repository: ambari > > > Description > --- > > As part of START_ALL of services Ranger-KMS starts after Hbase and Hive > causing their start failure. > In an encrypted cluster where TDE is on, Hbase service seems to be dependent > on the Ranger-KMS service, but for a start-all services action Hbase service > starts before Ranger-KMS. > So Hbase master goes down while connecting to KMS, since at the time of Hbase > start KMS is down , connection to KMS fails, and Hbase-Master fails to become > active. > This in-turn causes Atlas service start failure which actually depends on > HBase and start_all flow fails for HBASE and HIVE. > > Attached patch is for branch-2.5, addendum trunk patch is attached to Apache > JIRA. > > > Diffs > - > > > ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/role_command_order.json > PRE-CREATION > > > Diff: https://reviews.apache.org/r/60388/diff/1/ > > > Testing > --- > > Verified with installation on Cent-OS 6. > > > Thanks, > > Vishal Suvagia > >
Re: Review Request 59701: AMBARI-21154 : Add JAAS config properties for Atlas Hive hook in HiveCli to use kerberos ticket-cache
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/59701/#review178859 --- Ship it! Ship It! - Mugdha Varadkar On June 23, 2017, 1:04 p.m., Vishal Suvagia wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/59701/ > --- > > (Updated June 23, 2017, 1:04 p.m.) > > > Review request for Ambari, Alejandro Fernandez, Gautam Borad, Madhan > Neethiraj, Mugdha Varadkar, Nixon Rodrigues, Robert Levas, and Sumit Mohanty. > > > Bugs: AMBARI-21154 > https://issues.apache.org/jira/browse/AMBARI-21154 > > > Repository: ambari > > > Description > --- > > In a kerberized environment, Atlas hook uses JAAS configuration section named > "KakfaClient" to authenticate with Kafka broker. In a typical Hive deployment > this configuration section is set to use the keytab and principal of > HiveServer2 process. The hook running in HiveCLI might fail to authenticate > with Kafka if the user can't read the configured keytab. > > Given that HiveCLI users would have performed kinit, the hook in HiveCLI > should use the ticket-cache generated by kinit. When ticket cache is not > available (for example in HiveServer2), the hook should use the configuration > provided in KafkaClient JAAS section > > As a solution need to add below in hive atlas-application.properties by > default if atlas-hive hook is enabled in secure mode > > atlas.jaas.ticketBased-KafkaClient.loginModuleControlFlag=required > atlas.jaas.ticketBased-KafkaClient.loginModuleName=com.sun.security.auth.module.Krb5LoginModule > atlas.jaas.ticketBased-KafkaClient.option.useTicketCache=true > > The attached patch is for trunk branch, patch for branch-2.5 is attached to > Apache Jira > > > Diffs > - > > > ambari-server/src/main/resources/common-services/HIVE/2.1.0.3.0/service_advisor.py > 6d3e13d > ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/config-upgrade.xml > a29f74b > > ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml > 8c659ee > ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml > 3054ca3 > ambari-server/src/main/resources/stacks/HDP/2.6/services/stack_advisor.py > f8bbca5 > ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml > 1cbd78b > > ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml > ede267a > ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml > b70943b > ambari-server/src/test/python/stacks/2.6/common/test_stack_advisor.py > d4d28c9 > > > Diff: https://reviews.apache.org/r/59701/diff/3/ > > > Testing > --- > > Verified fresh install and upgrade on Cent-OS-6. > > > Thanks, > > Vishal Suvagia > >
Re: Review Request 60078: AMBARI-21190 Populate urlauth filesystem schemes in Ranger hive plugin
> On June 16, 2017, 11:20 p.m., Alejandro Fernandez wrote: > > ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml > > Lines 61 (patched) > > <https://reviews.apache.org/r/60078/diff/1/?file=1750495#file1750495line61> > > > > If an existing cluster with Hive and Ranger is Kerberized, this logic > > may also need to exist in Stack Advisor so that config changes cause it to > > be saved. Hi Alejandro, This property is used for Hive when Hive is configured to use different file-system schemes supported by HDFS, it is also independent of whether the cluster is kerberized or not. It is a new property introduced recently. For existing cluster, stack advisor logic will not be needed to recommend. If user want this property, they can add it via custom-config or it will be added during minor upgrade of 2.6.x to 2.6.y - Mugdha --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/60078/#review178136 --- On June 14, 2017, 1:37 p.m., Mugdha Varadkar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/60078/ > --- > > (Updated June 14, 2017, 1:37 p.m.) > > > Review request for Ambari, Alejandro Fernandez, Gautam Borad, and Jonathan > Hurley. > > > Bugs: AMBARI-21190 > https://issues.apache.org/jira/browse/AMBARI-21190 > > > Repository: ambari > > > Description > --- > > populate ranger.plugin.hive.urlauth.filesystem.schemes default value to > hdfs:,file:,wasb:,adl: > 1) it will be better if we keep default value of > ranger.plugin.hive.urlauth.filesystem.schemes=hdfs:,file:,wasb:,adl:, if it > does not harm to have all of the fs values in this properly it will be good > to add them. > 2) it will be good to add this properly in advanced-ranger-hive-security > > > Diffs > - > > > ambari-server/src/main/resources/common-services/HIVE/2.1.0.3.0/configuration/ranger-hive-security.xml > d98bc21 > > ambari-server/src/main/resources/common-services/RANGER/0.5.0/configuration/ranger-ugsync-site.xml > 439c495 > > ambari-server/src/main/resources/common-services/RANGER/0.7.0.3.0/configuration/ranger-ugsync-site.xml > 2c62851 > > ambari-server/src/main/resources/stacks/HDP/2.6/services/HIVE/configuration/ranger-hive-security.xml > PRE-CREATION > ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml > 1610bb5 > > ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml > 1cdd184 > ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml > 3e7e3d7 > > > Diff: https://reviews.apache.org/r/60078/diff/1/ > > > Testing > --- > > Tested Ranger Install + Enabled Hive Plugin. > > > Thanks, > > Mugdha Varadkar > >
Review Request 60118: AMBARI-21259 Need to use provided password when usersync sync source and authentication method are different
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/60118/ --- Review request for Ambari, Alejandro Fernandez, Gautam Borad, and Jayush Luniya. Bugs: AMBARI-21259 https://issues.apache.org/jira/browse/AMBARI-21259 Repository: ambari Description --- In Ambari for Ranger, the password for user authentication is taken from Ranger User Info Tab for LDAP/AD when sync-source and authentication method is LDAP/AD. For the scenario where sync-source and authentication method are different or when credentials for sync-source and authentication are different will need to take password from advanced section for LDAP/AD. Diffs - ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py 094d239 ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py 8f89416 Diff: https://reviews.apache.org/r/60118/diff/1/ Testing --- Unit test report: Ran 272 tests in 9.308s OK -- Total run:1160 Total errors:0 Total failures:0 OK Thanks, Mugdha Varadkar
Re: Review Request 60078: AMBARI-21190 Populate urlauth filesystem schemes in Ranger hive plugin
> On June 14, 2017, 3:12 p.m., Jonathan Hurley wrote: > > ambari-server/src/main/resources/stacks/HDP/2.6/services/HIVE/configuration/ranger-hive-security.xml > > Lines 1-33 (patched) > > <https://reviews.apache.org/r/60078/diff/1/?file=1750494#file1750494line1> > > > > This file and the new property already exist at the common-services > > level; why add it here too? It should just inherit from common-services. Hi Jonathan, Hive service for stack 3.0 is extending new common-service version 2.1.0.3.0. All configs from HDP 2.0.6 – 2.6 are flattend out in common-services version 2.1.0.3.0 for stack 3.0. So adding the same property in common-services/HIVE/2.1.0.3.0/configuration/ranger-hive-security.xml. Adding this property here for stack 2.6 - Mugdha --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/60078/#review177899 ------- On June 14, 2017, 1:37 p.m., Mugdha Varadkar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/60078/ > --- > > (Updated June 14, 2017, 1:37 p.m.) > > > Review request for Ambari, Alejandro Fernandez, Gautam Borad, and Jonathan > Hurley. > > > Bugs: AMBARI-21190 > https://issues.apache.org/jira/browse/AMBARI-21190 > > > Repository: ambari > > > Description > --- > > populate ranger.plugin.hive.urlauth.filesystem.schemes default value to > hdfs:,file:,wasb:,adl: > 1) it will be better if we keep default value of > ranger.plugin.hive.urlauth.filesystem.schemes=hdfs:,file:,wasb:,adl:, if it > does not harm to have all of the fs values in this properly it will be good > to add them. > 2) it will be good to add this properly in advanced-ranger-hive-security > > > Diffs > - > > > ambari-server/src/main/resources/common-services/HIVE/2.1.0.3.0/configuration/ranger-hive-security.xml > d98bc21 > > ambari-server/src/main/resources/common-services/RANGER/0.5.0/configuration/ranger-ugsync-site.xml > 439c495 > > ambari-server/src/main/resources/common-services/RANGER/0.7.0.3.0/configuration/ranger-ugsync-site.xml > 2c62851 > > ambari-server/src/main/resources/stacks/HDP/2.6/services/HIVE/configuration/ranger-hive-security.xml > PRE-CREATION > ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml > 1610bb5 > > ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml > 1cdd184 > ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml > 3e7e3d7 > > > Diff: https://reviews.apache.org/r/60078/diff/1/ > > > Testing > --- > > Tested Ranger Install + Enabled Hive Plugin. > > > Thanks, > > Mugdha Varadkar > >
Review Request 60078: AMBARI-21190 Populate urlauth filesystem schemes in Ranger hive plugin
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/60078/ --- Review request for Ambari, Alejandro Fernandez, Gautam Borad, and Jonathan Hurley. Bugs: AMBARI-21190 https://issues.apache.org/jira/browse/AMBARI-21190 Repository: ambari Description --- populate ranger.plugin.hive.urlauth.filesystem.schemes default value to hdfs:,file:,wasb:,adl: 1) it will be better if we keep default value of ranger.plugin.hive.urlauth.filesystem.schemes=hdfs:,file:,wasb:,adl:, if it does not harm to have all of the fs values in this properly it will be good to add them. 2) it will be good to add this properly in advanced-ranger-hive-security Diffs - ambari-server/src/main/resources/common-services/HIVE/2.1.0.3.0/configuration/ranger-hive-security.xml d98bc21 ambari-server/src/main/resources/common-services/RANGER/0.5.0/configuration/ranger-ugsync-site.xml 439c495 ambari-server/src/main/resources/common-services/RANGER/0.7.0.3.0/configuration/ranger-ugsync-site.xml 2c62851 ambari-server/src/main/resources/stacks/HDP/2.6/services/HIVE/configuration/ranger-hive-security.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml 1610bb5 ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml 1cdd184 ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml 3e7e3d7 Diff: https://reviews.apache.org/r/60078/diff/1/ Testing --- Tested Ranger Install + Enabled Hive Plugin. Thanks, Mugdha Varadkar
Review Request 59997: AMBARI-21227 Updating Ranger install.properties file
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/59997/ --- Review request for Ambari, Alejandro Fernandez, Gautam Borad, and Velmurugan Periasamy. Bugs: AMBARI-21227 https://issues.apache.org/jira/browse/AMBARI-21227 Repository: ambari Description --- Updating Ranger and Ranger Kms services install.properties. Also backing the files in ambari-agent tmp directory. Diffs - ambari-common/src/main/python/resource_management/libraries/providers/modify_properties_file.py 53d7d2f ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py b849d58 ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py 8f89416 ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms.py 5a25b92 ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms_server.py 44d61da ambari-server/src/test/python/stacks/2.2/RANGER/test_ranger_admin.py d338c28 ambari-server/src/test/python/stacks/2.5/RANGER/test_ranger_admin.py 8f2bd2e ambari-server/src/test/python/stacks/2.5/RANGER_KMS/test_kms_server.py f6f0af3 ambari-server/src/test/python/stacks/2.6/RANGER/test_ranger_admin.py 9167017 Diff: https://reviews.apache.org/r/59997/diff/1/ Testing --- Unit test report: Ran 272 tests in 8.526s OK -- Total run:1160 Total errors:0 Total failures:0 OK Thanks, Mugdha Varadkar
Re: Review Request 59701: AMBARI-21154 : Add JAAS config properties for Atlas Hive hook in HiveCli to use kerberos ticket-cache
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/59701/#review177146 --- ambari-server/src/main/resources/stacks/HDP/2.6/services/HIVE/kerberos.json Lines 31 (patched) <https://reviews.apache.org/r/59701/#comment250687> This will be required in trunk branch in https://github.com/apache/ambari/blob/trunk/ambari-server/src/main/resources/common-services/HIVE/2.1.0.3.0/kerberos.json for stack 3.0 - Mugdha Varadkar On June 1, 2017, 5:04 a.m., Vishal Suvagia wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/59701/ > --- > > (Updated June 1, 2017, 5:04 a.m.) > > > Review request for Ambari, Alejandro Fernandez, Gautam Borad, Madhan > Neethiraj, Mugdha Varadkar, Nixon Rodrigues, and Sumit Mohanty. > > > Bugs: AMBARI-21154 > https://issues.apache.org/jira/browse/AMBARI-21154 > > > Repository: ambari > > > Description > --- > > In a kerberized environment, Atlas hook uses JAAS configuration section named > "KakfaClient" to authenticate with Kafka broker. In a typical Hive deployment > this configuration section is set to use the keytab and principal of > HiveServer2 process. The hook running in HiveCLI might fail to authenticate > with Kafka if the user can't read the configured keytab. > > Given that HiveCLI users would have performed kinit, the hook in HiveCLI > should use the ticket-cache generated by kinit. When ticket cache is not > available (for example in HiveServer2), the hook should use the configuration > provided in KafkaClient JAAS section > > As a solution need to add below in hive atlas-application.properties by > default if atlas-hive hook is enabled in secure mode > > atlas.jaas.ticketBased-KafkaClient.loginModuleControlFlag=required > atlas.jaas.ticketBased-KafkaClient.loginModuleName=com.sun.security.auth.module.Krb5LoginModule > atlas.jaas.ticketBased-KafkaClient.option.useTicketCache=true > > > Diffs > - > > ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/config-upgrade.xml > a29f74b > > ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml > 8c659ee > ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml > 3054ca3 > ambari-server/src/main/resources/stacks/HDP/2.6/services/HIVE/kerberos.json > PRE-CREATION > ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml > 1610bb5 > > ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml > 1cdd184 > ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml > 3e7e3d7 > > > Diff: https://reviews.apache.org/r/59701/diff/1/ > > > Testing > --- > > Verified fresh install and upgrade on Cent-OS-6. > > > Thanks, > > Vishal Suvagia > >
Re: Review Request 59701: AMBARI-21154 : Add JAAS config properties for Atlas Hive hook in HiveCli to use kerberos ticket-cache
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/59701/#review176733 --- Ship it! Ship It! - Mugdha Varadkar On June 1, 2017, 5:04 a.m., Vishal Suvagia wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/59701/ > --- > > (Updated June 1, 2017, 5:04 a.m.) > > > Review request for Ambari, Alejandro Fernandez, Gautam Borad, Madhan > Neethiraj, Mugdha Varadkar, Nixon Rodrigues, and Sumit Mohanty. > > > Bugs: AMBARI-21154 > https://issues.apache.org/jira/browse/AMBARI-21154 > > > Repository: ambari > > > Description > --- > > In a kerberized environment, Atlas hook uses JAAS configuration section named > "KakfaClient" to authenticate with Kafka broker. In a typical Hive deployment > this configuration section is set to use the keytab and principal of > HiveServer2 process. The hook running in HiveCLI might fail to authenticate > with Kafka if the user can't read the configured keytab. > > Given that HiveCLI users would have performed kinit, the hook in HiveCLI > should use the ticket-cache generated by kinit. When ticket cache is not > available (for example in HiveServer2), the hook should use the configuration > provided in KafkaClient JAAS section > > As a solution need to add below in hive atlas-application.properties by > default if atlas-hive hook is enabled in secure mode > > atlas.jaas.ticketBased-KafkaClient.loginModuleControlFlag=required > atlas.jaas.ticketBased-KafkaClient.loginModuleName=com.sun.security.auth.module.Krb5LoginModule > atlas.jaas.ticketBased-KafkaClient.option.useTicketCache=true > > > Diffs > - > > ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/config-upgrade.xml > a29f74b > > ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml > 8c659ee > ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml > 3054ca3 > ambari-server/src/main/resources/stacks/HDP/2.6/services/HIVE/kerberos.json > PRE-CREATION > ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml > 1610bb5 > > ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml > 1cdd184 > ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml > 3e7e3d7 > > > Diff: https://reviews.apache.org/r/59701/diff/1/ > > > Testing > --- > > Verified fresh install and upgrade on Cent-OS-6. > > > Thanks, > > Vishal Suvagia > >
Re: Review Request 59364: AMBARI-21053 Reverting Beacon stack advisor recommendation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/59364/ --- (Updated May 18, 2017, 3:03 p.m.) Review request for Ambari, Alejandro Fernandez, Jayush Luniya, Sumit Mohanty, Xi Wang, and Yusaku Sako. Changes --- Updating correct jira ID Summary (updated) - AMBARI-21053 Reverting Beacon stack advisor recommendation Bugs: AMBARI-21053 https://issues.apache.org/jira/browse/AMBARI-21053 Repository: ambari Description --- Beacon is an add-on-service installed via management pack. For services installed via mpack need to use service advisor recommendation, which should be provided in mpack. Because of the stack_advisor code available, service_advisor.py code logic from mpack is not getting executed. Diffs - ambari-server/src/main/resources/stacks/HDP/2.6/services/stack_advisor.py 488562b Diff: https://reviews.apache.org/r/59364/diff/1/ Testing --- Ran 273 tests in 7.202s OK -- Total run:1171 Total errors:0 Total failures:0 OK Thanks, Mugdha Varadkar
Review Request 59364: AMBARI-20762 Reverting Beacon stack advisor recommendation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/59364/ --- Review request for Ambari, Alejandro Fernandez, Jayush Luniya, Sumit Mohanty, Xi Wang, and Yusaku Sako. Bugs: AMBARI-20762 https://issues.apache.org/jira/browse/AMBARI-20762 Repository: ambari Description --- Beacon is an add-on-service installed via management pack. For services installed via mpack need to use service advisor recommendation, which should be provided in mpack. Because of the stack_advisor code available, service_advisor.py code logic from mpack is not getting executed. Diffs - ambari-server/src/main/resources/stacks/HDP/2.6/services/stack_advisor.py 488562b Diff: https://reviews.apache.org/r/59364/diff/1/ Testing --- Ran 273 tests in 7.202s OK -- Total run:1171 Total errors:0 Total failures:0 OK Thanks, Mugdha Varadkar
Re: Review Request 58995: AMBARI-20932 : Ranger plugin setup should handle Nifi service
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58995/#review174139 --- Ship it! Ship It! - Mugdha Varadkar On May 6, 2017, 6:45 p.m., Vishal Suvagia wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/58995/ > --- > > (Updated May 6, 2017, 6:45 p.m.) > > > Review request for Ambari, Alejandro Fernandez, Gautam Borad, Jayush Luniya, > Mugdha Varadkar, Sumit Mohanty, and Velmurugan Periasamy. > > > Bugs: AMBARI-20932 > https://issues.apache.org/jira/browse/AMBARI-20932 > > > Repository: ambari > > > Description > --- > > Add support to enable Ranger Nifi plugin. > > > Diffs > - > > > ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin_xml.py > c510dac > > ambari-server/src/main/resources/common-services/RANGER/0.7.0/configuration/ranger-env.xml > 661089a > > ambari-server/src/main/resources/common-services/RANGER/0.7.0/themes/theme_version_5.json > 87fccbe > > > Diff: https://reviews.apache.org/r/58995/diff/3/ > > > Testing > --- > > Verified fresh installation on Cent-OS 6. > > > Thanks, > > Vishal Suvagia > >
Re: Review Request 58993: AMBARI-20930 : stackadvisor recommendations for atlas hooks should be disabled for external Atlas
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58993/#review174019 --- Ship it! Ship It! - Mugdha Varadkar On May 5, 2017, 12:17 p.m., Vishal Suvagia wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/58993/ > --- > > (Updated May 5, 2017, 12:17 p.m.) > > > Review request for Ambari, Alejandro Fernandez, Gautam Borad, Madhan > Neethiraj, Mugdha Varadkar, and Sumit Mohanty. > > > Bugs: AMBARI-20930 > https://issues.apache.org/jira/browse/AMBARI-20930 > > > Repository: ambari > > > Description > --- > > In cases where Atlas is not setup on the cluster but the Atlas hooks are > enabled for external Atlas. > Ambari recommendations show to disable the Atlas hooks. > This behaviour needs to be changed to support externally used Atlas service. > > PS: Attached patch is for trunk, patch for branch-2.5 is attached to > AMBARI-20930 Apache JIRA. > > > Diffs > - > > > ambari-server/src/main/resources/common-services/HIVE/2.1.0.3.0/service_advisor.py > ef3e340 > ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py > eb7389d > > > Diff: https://reviews.apache.org/r/58993/diff/2/ > > > Testing > --- > > Verified fresh installation on Cent-OS 6. > > > Thanks, > > Vishal Suvagia > >
Re: Review Request 58989: AMBARI-20779 ADDENDUM Create Ranger KMS HDFS audit folder as part of install
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58989/ --- (Updated May 5, 2017, 11:55 a.m.) Review request for Ambari, Alejandro Fernandez, Gautam Borad, Sumit Mohanty, and Velmurugan Periasamy. Bugs: AMBARI-20779 https://issues.apache.org/jira/browse/AMBARI-20779 Repository: ambari Description --- Addendum patch to change ordering of RANGER-KMS service for RU and EU. Need to upgrade RANGER-KMS service after HDFS upgrade, as if Audit to HDFS is set to true, WebHDFS RESTAPI tries to create audit directory in HDFS this will fail RANGER-KMS restart task during upgrade as HDFS service will not be started. Diffs (updated) - ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.3.xml e111dbf ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml 6230921 ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml 47a7094 ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.6.xml 18c0c15 ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.4.xml 5120740 ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml 358fb6e ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.6.xml 368bbe2 ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/host-upgrade-2.5.xml c180b61 ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.5.xml 3ceea8f ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml c3436b9 ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/host-upgrade-2.6.xml f923702 ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml a74d085 Diff: https://reviews.apache.org/r/58989/diff/2/ Changes: https://reviews.apache.org/r/58989/diff/1-2/ Testing --- Tested EU upgrade to stack 2.6 Thanks, Mugdha Varadkar
Re: Review Request 58989: AMBARI-20779 ADDENDUM Create Ranger KMS HDFS audit folder as part of install
> On May 4, 2017, 7:43 p.m., Alejandro Fernandez wrote: > > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.3.xml > > Line 344 (original) > > <https://reviews.apache.org/r/58989/diff/1/?file=1708261#file1708261line344> > > > > I thought Ranger had to be upgraded before HDFS due to the plugins. Hi Alejandro, In Ambari Ranger and Ranger Kms are two different services. Current Upgrade order is Zookeeper > Ranger > Ranger Kms > HDFS Due to changes in Ranger Kms service for creating directory in HDFS, need to upgrade Ranger Kms after HDFS as during EU all services are stopped Ranger Kms restart fails to create directory in HDFS. I can skip the changes for RU since there is no stop task for services as in EU. So for RU the HDFS service will be up. - Mugdha --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58989/#review173947 ------- On May 4, 2017, 1:36 p.m., Mugdha Varadkar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/58989/ > --- > > (Updated May 4, 2017, 1:36 p.m.) > > > Review request for Ambari, Alejandro Fernandez, Gautam Borad, Sumit Mohanty, > and Velmurugan Periasamy. > > > Bugs: AMBARI-20779 > https://issues.apache.org/jira/browse/AMBARI-20779 > > > Repository: ambari > > > Description > --- > > Addendum patch to change ordering of RANGER-KMS service for RU and EU. Need > to upgrade RANGER-KMS service after HDFS upgrade, as if Audit to HDFS is set > to true, WebHDFS RESTAPI tries to create audit directory in HDFS this will > fail RANGER-KMS restart task during upgrade as HDFS service will not be > started. > > > Diffs > - > > > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.3.xml > e111dbf > > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml > 6230921 > > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml > 47a7094 > > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.6.xml > 18c0c15 > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.3.xml > 19031a8 > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml > 8cfaa52 > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml > f7192ae > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.6.xml > d98bb53 > > ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.4.xml > 5120740 > > ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml > 358fb6e > > ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.6.xml > 368bbe2 > ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.4.xml > 80436bf > ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.5.xml > ad9ae33 > ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.6.xml > 1eb9836 > > ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/host-upgrade-2.5.xml > c180b61 > > ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.5.xml > 3ceea8f > > ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml > c3436b9 > ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.5.xml > f3b1fa6 > ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml > 0470b7a > > ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/host-upgrade-2.6.xml > f923702 > > ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml > a74d085 > ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml > 35d3da2 > > > Diff: https://reviews.apache.org/r/58989/diff/1/ > > > Testing > --- > > Tested EU upgrade to stack 2.6 > > > Thanks, > > Mugdha Varadkar > >
Re: Review Request 58797: AMBARI-20868 : Ranger admin start fails when Ambari Infra is turned off.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58797/#review173904 --- Ship it! Ship It! - Mugdha Varadkar On May 4, 2017, 1:27 p.m., Vishal Suvagia wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/58797/ > --- > > (Updated May 4, 2017, 1:27 p.m.) > > > Review request for Ambari, Alejandro Fernandez, Gautam Borad, Mugdha > Varadkar, Oliver Szabo, and Velmurugan Periasamy. > > > Bugs: AMBARI-20868 > https://issues.apache.org/jira/browse/AMBARI-20868 > > > Repository: ambari > > > Description > --- > > Ranger Admin start fails when Ambari Infra or External Solr service is down,. > need to handle errors and ensure Ranger start works, despite solr service > being down. > > > Diffs > - > > > ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py > a8b7f24 > > ambari-server/src/main/resources/common-services/RANGER/0.7.0/properties/ranger-solrconfig.xml.j2 > 34ce70d > ambari-server/src/main/resources/stacks/HDP/2.6/services/stack_advisor.py > ad44033 > > > Diff: https://reviews.apache.org/r/58797/diff/3/ > > > Testing > --- > > Verified fresh install on Cent-OS 6. > > > Thanks, > > Vishal Suvagia > >
Re: Review Request 58989: AMBARI-20779 ADDENDUM Create Ranger KMS HDFS audit folder as part of install
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58989/ --- (Updated May 4, 2017, 1:36 p.m.) Review request for Ambari, Alejandro Fernandez, Gautam Borad, Sumit Mohanty, and Velmurugan Periasamy. Bugs: AMBARI-20779 https://issues.apache.org/jira/browse/AMBARI-20779 Repository: ambari Description (updated) --- Addendum patch to change ordering of RANGER-KMS service for RU and EU. Need to upgrade RANGER-KMS service after HDFS upgrade, as if Audit to HDFS is set to true, WebHDFS RESTAPI tries to create audit directory in HDFS this will fail RANGER-KMS restart task during upgrade as HDFS service will not be started. Diffs - ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.3.xml e111dbf ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml 6230921 ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml 47a7094 ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.6.xml 18c0c15 ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.3.xml 19031a8 ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 8cfaa52 ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml f7192ae ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.6.xml d98bb53 ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.4.xml 5120740 ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml 358fb6e ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.6.xml 368bbe2 ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.4.xml 80436bf ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.5.xml ad9ae33 ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.6.xml 1eb9836 ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/host-upgrade-2.5.xml c180b61 ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.5.xml 3ceea8f ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml c3436b9 ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.5.xml f3b1fa6 ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml 0470b7a ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/host-upgrade-2.6.xml f923702 ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml a74d085 ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml 35d3da2 Diff: https://reviews.apache.org/r/58989/diff/1/ Testing --- Tested EU upgrade to stack 2.6 Thanks, Mugdha Varadkar
Review Request 58989: AMBARI-20779 ADDENDUM Create Ranger KMS HDFS audit folder as part of install
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58989/ --- Review request for Ambari, Alejandro Fernandez, Gautam Borad, Sumit Mohanty, and Velmurugan Periasamy. Bugs: AMBARI-20779 https://issues.apache.org/jira/browse/AMBARI-20779 Repository: ambari Description --- Addendum patch to change ordering of RANGER-KMS service for RU and EU. Need to upgrade RANGER-KMS service after HDFS upgrade, as if Audit to HDFS is set to true, WebHDFS RESTAPI tries to create audit directory in hadoop this will fail as HDFS service will not be started. Diffs - ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.3.xml e111dbf ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml 6230921 ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.5.xml 47a7094 ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.6.xml 18c0c15 ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.3.xml 19031a8 ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml 8cfaa52 ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.5.xml f7192ae ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.6.xml d98bb53 ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.4.xml 5120740 ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.5.xml 358fb6e ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.6.xml 368bbe2 ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.4.xml 80436bf ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.5.xml ad9ae33 ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.6.xml 1eb9836 ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/host-upgrade-2.5.xml c180b61 ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.5.xml 3ceea8f ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml c3436b9 ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.5.xml f3b1fa6 ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml 0470b7a ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/host-upgrade-2.6.xml f923702 ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml a74d085 ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml 35d3da2 Diff: https://reviews.apache.org/r/58989/diff/1/ Testing --- Tested EU upgrade to stack 2.6 Thanks, Mugdha Varadkar
Re: Review Request 58965: AMBARI-20920 : Atlas restart fails due to missing hbase dependency in role-command
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58965/#review173871 --- Ship it! Ship It! - Mugdha Varadkar On May 3, 2017, 3:12 p.m., Vishal Suvagia wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/58965/ > --- > > (Updated May 3, 2017, 3:12 p.m.) > > > Review request for Ambari, Alejandro Fernandez, Gautam Borad, Madhan > Neethiraj, Mugdha Varadkar, and Sumit Mohanty. > > > Bugs: AMBARI-20920 > https://issues.apache.org/jira/browse/AMBARI-20920 > > > Repository: ambari > > > Description > --- > > ATLAS component STARTs or RESTARTs are dependent on HBASE STARTs/RESTARTs, so > the commands will be scheduled appropriately. These should be captured in > (role command order) for Atlas. > > PS: Attached patch is for branch-2.5, trunk-addendum patch is attached to > AMBARI-20920 Apache JIRA. > > > Diffs > - > > > ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/role_command_order.json > PRE-CREATION > > ambari-server/src/main/resources/common-services/ATLAS/0.7.0.2.5/role_command_order.json > PRE-CREATION > ambari-server/src/main/resources/stacks/HDP/2.5/role_command_order.json > 58dcf8d > > > Diff: https://reviews.apache.org/r/58965/diff/1/ > > > Testing > --- > > Verified fresh install on Cent-OS 6. > > > Thanks, > > Vishal Suvagia > >
Re: Review Request 58918: AMBARI-20911 Add property in Ranger for including SSL protocols
> On May 2, 2017, 2:16 p.m., Jonathan Hurley wrote: > > ambari-server/src/main/resources/common-services/RANGER/0.7.0/configuration/ranger-admin-site.xml > > Lines 33 (patched) > > <https://reviews.apache.org/r/58918/diff/1/?file=1705929#file1705929line33> > > > > cURL on several supported OS's don't support TLSv1.2 Closing the Review request as fix for AMBARI-20911: https://issues.apache.org/jira/browse/AMBARI-20911 is no more required. - Mugdha --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58918/#review173585 ------- On May 2, 2017, 11:58 a.m., Mugdha Varadkar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/58918/ > --- > > (Updated May 2, 2017, 11:58 a.m.) > > > Review request for Ambari, Alejandro Fernandez, Gautam Borad, Jonathan > Hurley, and Sumit Mohanty. > > > Bugs: AMBARI-20911 > https://issues.apache.org/jira/browse/AMBARI-20911 > > > Repository: ambari > > > Description > --- > > Add ranger.service.https.attrib.ssl.enabled.protocols property in > ranger-admin-site config type for stack 2.6 > > For fresh install the value required is: TLSv1.1,TLSv1.2 > For upgrade: SSLv2Hello,TLSv1,TLSv1.1,TLSv1.2 > > > Diffs > - > > > ambari-server/src/main/resources/common-services/RANGER/0.7.0/configuration/ranger-admin-site.xml > ebf8517 > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml > 8b5c07d > > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.6.xml > 18c0c15 > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.6.xml > d98bb53 > ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml > b3d19d4 > > ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.6.xml > 368bbe2 > ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.6.xml > 1eb9836 > ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/config-upgrade.xml > 559d8dc > > ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml > c3436b9 > ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml > 0470b7a > ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml > a6b7523 > > ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml > a74d085 > ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml > 35d3da2 > > > Diff: https://reviews.apache.org/r/58918/diff/1/ > > > Testing > --- > > Tested EU Upgrade with Ranger. > > > Thanks, > > Mugdha Varadkar > >
Review Request 58918: AMBARI-20911 Add property in Ranger for including SSL protocols
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58918/ --- Review request for Ambari, Alejandro Fernandez, Gautam Borad, Jonathan Hurley, and Sumit Mohanty. Bugs: AMBARI-20911 https://issues.apache.org/jira/browse/AMBARI-20911 Repository: ambari Description --- Add ranger.service.https.attrib.ssl.enabled.protocols property in ranger-admin-site config type for stack 2.6 For fresh install the value required is: TLSv1.1,TLSv1.2 For upgrade: SSLv2Hello,TLSv1,TLSv1.1,TLSv1.2 Diffs - ambari-server/src/main/resources/common-services/RANGER/0.7.0/configuration/ranger-admin-site.xml ebf8517 ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 8b5c07d ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.6.xml 18c0c15 ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.6.xml d98bb53 ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml b3d19d4 ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.6.xml 368bbe2 ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.6.xml 1eb9836 ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/config-upgrade.xml 559d8dc ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml c3436b9 ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml 0470b7a ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml a6b7523 ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml a74d085 ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml 35d3da2 Diff: https://reviews.apache.org/r/58918/diff/1/ Testing --- Tested EU Upgrade with Ranger. Thanks, Mugdha Varadkar
Re: Review Request 58497: AMBARI-20779 Create Ranger KMS HDFS audit folder as part of install
> On April 26, 2017, 6:37 p.m., Alejandro Fernandez wrote: > > ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms.py > > Lines 510 (patched) > > <https://reviews.apache.org/r/58497/diff/2-3/?file=1699268#file1699268line510> > > > > I think you still need has_hdfs_client_on_node > > Mugdha Varadkar wrote: > Hi Alejandro, > > create_kms_audit_dir is calcluated using security_enabled and > has_hdfs_client_on_node flags, if has_hdfs_client_on_node is false then > create_kms_audit_dir will be false, so the resulting HDFS resource will not > be created. > > Alejandro Fernandez wrote: > If security is disabled and HDFS Client is not present, > create_kms_audit_dir will still be True, which will fail. > instead, initialize create_kms_audit_dir to false, and set to true if > security is diabled and has hdfs client. Hi Alejandro, If there is no HDFS_CLIENT on node but the node can have any other components of HDFS service, then in simple environment, the create resource call will work. Hence using create_kms_audit_dir flag. So there will be only one condition where we can set the flag create_kms_audit_dir as false when security is enabled for the cluster and but there is no HDFS_CLIENT installed on the node. I have updated the patch to use has_hdfs_client_on_node flag. As during upgrade we don't know whether the node where Ranger Kms is installed will have any Hdfs service components installed. - Mugdha --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58497/#review173073 ------- On April 28, 2017, 11:53 a.m., Mugdha Varadkar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/58497/ > --- > > (Updated April 28, 2017, 11:53 a.m.) > > > Review request for Ambari, Alejandro Fernandez, Gautam Borad, Sumit Mohanty, > and Velmurugan Periasamy. > > > Bugs: AMBARI-20779 > https://issues.apache.org/jira/browse/AMBARI-20779 > > > Repository: ambari > > > Description > --- > > Create audit directory /ranger/audit/kms in hdfs, if > xasecure.audit.destination.hdfs (Audit To HDFS) property is set to true. > > > Diffs > - > > > ambari-server/src/main/resources/common-services/RANGER/0.5.0/role_command_order.json > df62dfd > > ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/metainfo.xml > b20201c > > ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms.py > 423cdec > > ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/params.py > 9fe0a61 > > ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/role_command_order.json > 006d177 > ambari-server/src/main/resources/stacks/HDP/2.3/role_command_order.json > 4f279c6 > ambari-server/src/test/python/stacks/2.5/RANGER_KMS/test_kms_server.py > 7082a33 > ambari-server/src/test/python/stacks/2.5/configs/ranger-kms-secured.json > 7054e8f > > > Diff: https://reviews.apache.org/r/58497/diff/4/ > > > Testing > --- > > Unit Test Result: > > Running tests for stack:2.5 service:RANGER_KMS > test_configure_default (test_kms_server.TestRangerKMS) ... 2017-04-18 > 13:13:22,030 - Stack Feature Version Info: stack_version=2.5, > version=2.5.0.0-777, current_cluster_version=2.5.0.0-777 -> 2.5.0.0-777 > 2017-04-18 13:13:22,046 - Using hadoop conf dir: > /usr/hdp/current/hadoop-client/conf > ok > test_configure_secured (test_kms_server.TestRangerKMS) ... 2017-04-18 > 13:13:22,062 - Stack Feature Version Info: stack_version=2.5, > version=2.5.0.0-801, current_cluster_version=2.5.0.0-801 -> 2.5.0.0-801 > 2017-04-18 13:13:22,080 - Using hadoop conf dir: > /usr/hdp/current/hadoop-client/conf > ok > test_start_default (test_kms_server.TestRangerKMS) ... 2017-04-18 > 13:13:22,095 - Stack Feature Version Info: stack_version=2.5, > version=2.5.0.0-777, current_cluster_version=2.5.0.0-777 -> 2.5.0.0-777 > 2017-04-18 13:13:22,111 - Using hadoop conf dir: > /usr/hdp/current/hadoop-client/conf > 2017-04-18 13:13:22,116 - Rangeradmin: Skip ranger admin if it's down ! > ok > test_start_secured (test_kms_server.TestRangerKMS) ... 2017-04-18 > 13:13:22,129 - Stack Feature Version Info: stack_version=2.5, > version=2.5.0.0-801, current_cluster_version=2.5.0.0-801 -> 2.5.0.0-801 &g
Re: Review Request 58497: AMBARI-20779 Create Ranger KMS HDFS audit folder as part of install
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58497/ --- (Updated April 28, 2017, 11:53 a.m.) Review request for Ambari, Alejandro Fernandez, Gautam Borad, Sumit Mohanty, and Velmurugan Periasamy. Bugs: AMBARI-20779 https://issues.apache.org/jira/browse/AMBARI-20779 Repository: ambari Description --- Create audit directory /ranger/audit/kms in hdfs, if xasecure.audit.destination.hdfs (Audit To HDFS) property is set to true. Diffs (updated) - ambari-server/src/main/resources/common-services/RANGER/0.5.0/role_command_order.json df62dfd ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/metainfo.xml b20201c ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms.py 423cdec ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/params.py 9fe0a61 ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/role_command_order.json 006d177 ambari-server/src/main/resources/stacks/HDP/2.3/role_command_order.json 4f279c6 ambari-server/src/test/python/stacks/2.5/RANGER_KMS/test_kms_server.py 7082a33 ambari-server/src/test/python/stacks/2.5/configs/ranger-kms-secured.json 7054e8f Diff: https://reviews.apache.org/r/58497/diff/4/ Changes: https://reviews.apache.org/r/58497/diff/3-4/ Testing --- Unit Test Result: Running tests for stack:2.5 service:RANGER_KMS test_configure_default (test_kms_server.TestRangerKMS) ... 2017-04-18 13:13:22,030 - Stack Feature Version Info: stack_version=2.5, version=2.5.0.0-777, current_cluster_version=2.5.0.0-777 -> 2.5.0.0-777 2017-04-18 13:13:22,046 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf ok test_configure_secured (test_kms_server.TestRangerKMS) ... 2017-04-18 13:13:22,062 - Stack Feature Version Info: stack_version=2.5, version=2.5.0.0-801, current_cluster_version=2.5.0.0-801 -> 2.5.0.0-801 2017-04-18 13:13:22,080 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf ok test_start_default (test_kms_server.TestRangerKMS) ... 2017-04-18 13:13:22,095 - Stack Feature Version Info: stack_version=2.5, version=2.5.0.0-777, current_cluster_version=2.5.0.0-777 -> 2.5.0.0-777 2017-04-18 13:13:22,111 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf 2017-04-18 13:13:22,116 - Rangeradmin: Skip ranger admin if it's down ! ok test_start_secured (test_kms_server.TestRangerKMS) ... 2017-04-18 13:13:22,129 - Stack Feature Version Info: stack_version=2.5, version=2.5.0.0-801, current_cluster_version=2.5.0.0-801 -> 2.5.0.0-801 2017-04-18 13:13:22,145 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf 2017-04-18 13:13:22,150 - RangeradminV2: Skip ranger admin if it's down ! 2017-04-18 13:13:22,151 - KMS repository c1_kms exist ok test_stop_default (test_kms_server.TestRangerKMS) ... 2017-04-18 13:13:22,163 - Stack Feature Version Info: stack_version=2.5, version=2.5.0.0-777, current_cluster_version=2.5.0.0-777 -> 2.5.0.0-777 2017-04-18 13:13:22,179 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf ok -- Ran 5 tests in 0.164s OK Thanks, Mugdha Varadkar
Re: Review Request 58797: AMBARI-20868 : Ranger admin start fails when Ambari Infra is turned off.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58797/#review173287 --- Ship it! Ship It! - Mugdha Varadkar On April 27, 2017, 3:51 p.m., Vishal Suvagia wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/58797/ > --- > > (Updated April 27, 2017, 3:51 p.m.) > > > Review request for Ambari, Alejandro Fernandez, Gautam Borad, Mugdha > Varadkar, Oliver Szabo, and Velmurugan Periasamy. > > > Bugs: AMBARI-20868 > https://issues.apache.org/jira/browse/AMBARI-20868 > > > Repository: ambari > > > Description > --- > > Ranger Admin start fails when Ambari Infra or External Solr service is down,. > need to handle errors and ensure Ranger start works, despite solr service > being down. > > > Diffs > - > > > ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py > e56a705 > > ambari-server/src/main/resources/common-services/RANGER/0.7.0/properties/ranger-solrconfig.xml.j2 > 34ce70d > > > Diff: https://reviews.apache.org/r/58797/diff/2/ > > > Testing > --- > > Verified fresh install on Cent-OS 6. > > > Thanks, > > Vishal Suvagia > >
Re: Review Request 58798: AMBARI-20873 : Atlas gc-worker logs are not generated due to incorrect configuration
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58798/#review173286 --- Ship it! Ship It! - Mugdha Varadkar On April 27, 2017, 2:39 p.m., Vishal Suvagia wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/58798/ > --- > > (Updated April 27, 2017, 2:39 p.m.) > > > Review request for Ambari, Alejandro Fernandez, Gautam Borad, Madhan > Neethiraj, Mugdha Varadkar, and Sumit Mohanty. > > > Bugs: AMBARI-20873 > https://issues.apache.org/jira/browse/AMBARI-20873 > > > Repository: ambari > > > Description > --- > > Atlas gc-worker logs are not generated due to missing / in atlas > configurations. > Currently it is set as -Xloggc:$ATLAS_LOG_DIRgc-worker.log it should actually > be -Xloggc:$ATLAS_LOG_DIR/gc-worker.log > > > Diffs > - > > > ambari-server/src/main/resources/stacks/HDP/2.5/services/ATLAS/configuration/atlas-env.xml > bd0da9f > ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/config-upgrade.xml > 1c2ceff > > ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.5.xml > be8b750 > > ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml > d822778 > ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.5.xml > 210c4e2 > ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml > 9a06dd0 > ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml > f5d87e9 > > ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml > f7c276b > ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml > 81757c1 > > > Diff: https://reviews.apache.org/r/58798/diff/1/ > > > Testing > --- > > Verified fresh install and upgrade on Cent-OS 6. > > > Thanks, > > Vishal Suvagia > >
Re: Review Request 58497: AMBARI-20779 Create Ranger KMS HDFS audit folder as part of install
> On April 26, 2017, 6:37 p.m., Alejandro Fernandez wrote: > > ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms.py > > Lines 510 (patched) > > <https://reviews.apache.org/r/58497/diff/2-3/?file=1699268#file1699268line510> > > > > I think you still need has_hdfs_client_on_node Hi Alejandro, create_kms_audit_dir is calcluated using security_enabled and has_hdfs_client_on_node flags, if has_hdfs_client_on_node is false then create_kms_audit_dir will be false, so the resulting HDFS resource will not be created. - Mugdha --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58497/#review173073 --- On April 26, 2017, 9:56 a.m., Mugdha Varadkar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/58497/ > --- > > (Updated April 26, 2017, 9:56 a.m.) > > > Review request for Ambari, Alejandro Fernandez, Gautam Borad, Sumit Mohanty, > and Velmurugan Periasamy. > > > Bugs: AMBARI-20779 > https://issues.apache.org/jira/browse/AMBARI-20779 > > > Repository: ambari > > > Description > --- > > Create audit directory /ranger/audit/kms in hdfs, if > xasecure.audit.destination.hdfs (Audit To HDFS) property is set to true. > > > Diffs > - > > > ambari-server/src/main/resources/common-services/RANGER/0.5.0/role_command_order.json > df62dfd > > ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/metainfo.xml > b20201c > > ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms.py > 423cdec > > ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/params.py > 9fe0a61 > > ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/role_command_order.json > 006d177 > ambari-server/src/main/resources/stacks/HDP/2.3/role_command_order.json > 4f279c6 > ambari-server/src/test/python/stacks/2.5/RANGER_KMS/test_kms_server.py > 7082a33 > ambari-server/src/test/python/stacks/2.5/configs/ranger-kms-secured.json > 7054e8f > > > Diff: https://reviews.apache.org/r/58497/diff/3/ > > > Testing > --- > > Unit Test Result: > > Running tests for stack:2.5 service:RANGER_KMS > test_configure_default (test_kms_server.TestRangerKMS) ... 2017-04-18 > 13:13:22,030 - Stack Feature Version Info: stack_version=2.5, > version=2.5.0.0-777, current_cluster_version=2.5.0.0-777 -> 2.5.0.0-777 > 2017-04-18 13:13:22,046 - Using hadoop conf dir: > /usr/hdp/current/hadoop-client/conf > ok > test_configure_secured (test_kms_server.TestRangerKMS) ... 2017-04-18 > 13:13:22,062 - Stack Feature Version Info: stack_version=2.5, > version=2.5.0.0-801, current_cluster_version=2.5.0.0-801 -> 2.5.0.0-801 > 2017-04-18 13:13:22,080 - Using hadoop conf dir: > /usr/hdp/current/hadoop-client/conf > ok > test_start_default (test_kms_server.TestRangerKMS) ... 2017-04-18 > 13:13:22,095 - Stack Feature Version Info: stack_version=2.5, > version=2.5.0.0-777, current_cluster_version=2.5.0.0-777 -> 2.5.0.0-777 > 2017-04-18 13:13:22,111 - Using hadoop conf dir: > /usr/hdp/current/hadoop-client/conf > 2017-04-18 13:13:22,116 - Rangeradmin: Skip ranger admin if it's down ! > ok > test_start_secured (test_kms_server.TestRangerKMS) ... 2017-04-18 > 13:13:22,129 - Stack Feature Version Info: stack_version=2.5, > version=2.5.0.0-801, current_cluster_version=2.5.0.0-801 -> 2.5.0.0-801 > 2017-04-18 13:13:22,145 - Using hadoop conf dir: > /usr/hdp/current/hadoop-client/conf > 2017-04-18 13:13:22,150 - RangeradminV2: Skip ranger admin if it's down ! > 2017-04-18 13:13:22,151 - KMS repository c1_kms exist > ok > test_stop_default (test_kms_server.TestRangerKMS) ... 2017-04-18 13:13:22,163 > - Stack Feature Version Info: stack_version=2.5, version=2.5.0.0-777, > current_cluster_version=2.5.0.0-777 -> 2.5.0.0-777 > 2017-04-18 13:13:22,179 - Using hadoop conf dir: > /usr/hdp/current/hadoop-client/conf > ok > > -- > Ran 5 tests in 0.164s > > OK > > > Thanks, > > Mugdha Varadkar > >
Re: Review Request 58497: AMBARI-20779 Create Ranger KMS HDFS audit folder as part of install
> On April 25, 2017, 8:10 p.m., Alejandro Fernandez wrote: > > ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms.py > > Lines 509 (patched) > > <https://reviews.apache.org/r/58497/diff/2/?file=1699268#file1699268line509> > > > > For upgraded clusters, HDFS Client is not guaranteed to be on this host. > > This should still check if HDFS Client is installed on this host, and > > only then run the commands. > > > > Also, add a comment as to why this is needed Updated in latest patch - Mugdha --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58497/#review172965 ------- On April 26, 2017, 9:56 a.m., Mugdha Varadkar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/58497/ > --- > > (Updated April 26, 2017, 9:56 a.m.) > > > Review request for Ambari, Alejandro Fernandez, Gautam Borad, Sumit Mohanty, > and Velmurugan Periasamy. > > > Bugs: AMBARI-20779 > https://issues.apache.org/jira/browse/AMBARI-20779 > > > Repository: ambari > > > Description > --- > > Create audit directory /ranger/audit/kms in hdfs, if > xasecure.audit.destination.hdfs (Audit To HDFS) property is set to true. > > > Diffs > - > > > ambari-server/src/main/resources/common-services/RANGER/0.5.0/role_command_order.json > df62dfd > > ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/metainfo.xml > b20201c > > ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms.py > 423cdec > > ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/params.py > 9fe0a61 > > ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/role_command_order.json > 006d177 > ambari-server/src/main/resources/stacks/HDP/2.3/role_command_order.json > 4f279c6 > ambari-server/src/test/python/stacks/2.5/RANGER_KMS/test_kms_server.py > 7082a33 > ambari-server/src/test/python/stacks/2.5/configs/ranger-kms-secured.json > 7054e8f > > > Diff: https://reviews.apache.org/r/58497/diff/3/ > > > Testing > --- > > Unit Test Result: > > Running tests for stack:2.5 service:RANGER_KMS > test_configure_default (test_kms_server.TestRangerKMS) ... 2017-04-18 > 13:13:22,030 - Stack Feature Version Info: stack_version=2.5, > version=2.5.0.0-777, current_cluster_version=2.5.0.0-777 -> 2.5.0.0-777 > 2017-04-18 13:13:22,046 - Using hadoop conf dir: > /usr/hdp/current/hadoop-client/conf > ok > test_configure_secured (test_kms_server.TestRangerKMS) ... 2017-04-18 > 13:13:22,062 - Stack Feature Version Info: stack_version=2.5, > version=2.5.0.0-801, current_cluster_version=2.5.0.0-801 -> 2.5.0.0-801 > 2017-04-18 13:13:22,080 - Using hadoop conf dir: > /usr/hdp/current/hadoop-client/conf > ok > test_start_default (test_kms_server.TestRangerKMS) ... 2017-04-18 > 13:13:22,095 - Stack Feature Version Info: stack_version=2.5, > version=2.5.0.0-777, current_cluster_version=2.5.0.0-777 -> 2.5.0.0-777 > 2017-04-18 13:13:22,111 - Using hadoop conf dir: > /usr/hdp/current/hadoop-client/conf > 2017-04-18 13:13:22,116 - Rangeradmin: Skip ranger admin if it's down ! > ok > test_start_secured (test_kms_server.TestRangerKMS) ... 2017-04-18 > 13:13:22,129 - Stack Feature Version Info: stack_version=2.5, > version=2.5.0.0-801, current_cluster_version=2.5.0.0-801 -> 2.5.0.0-801 > 2017-04-18 13:13:22,145 - Using hadoop conf dir: > /usr/hdp/current/hadoop-client/conf > 2017-04-18 13:13:22,150 - RangeradminV2: Skip ranger admin if it's down ! > 2017-04-18 13:13:22,151 - KMS repository c1_kms exist > ok > test_stop_default (test_kms_server.TestRangerKMS) ... 2017-04-18 13:13:22,163 > - Stack Feature Version Info: stack_version=2.5, version=2.5.0.0-777, > current_cluster_version=2.5.0.0-777 -> 2.5.0.0-777 > 2017-04-18 13:13:22,179 - Using hadoop conf dir: > /usr/hdp/current/hadoop-client/conf > ok > > -- > Ran 5 tests in 0.164s > > OK > > > Thanks, > > Mugdha Varadkar > >
Re: Review Request 58497: AMBARI-20779 Create Ranger KMS HDFS audit folder as part of install
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58497/ --- (Updated April 26, 2017, 9:56 a.m.) Review request for Ambari, Alejandro Fernandez, Gautam Borad, Sumit Mohanty, and Velmurugan Periasamy. Bugs: AMBARI-20779 https://issues.apache.org/jira/browse/AMBARI-20779 Repository: ambari Description --- Create audit directory /ranger/audit/kms in hdfs, if xasecure.audit.destination.hdfs (Audit To HDFS) property is set to true. Diffs (updated) - ambari-server/src/main/resources/common-services/RANGER/0.5.0/role_command_order.json df62dfd ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/metainfo.xml b20201c ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms.py 423cdec ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/params.py 9fe0a61 ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/role_command_order.json 006d177 ambari-server/src/main/resources/stacks/HDP/2.3/role_command_order.json 4f279c6 ambari-server/src/test/python/stacks/2.5/RANGER_KMS/test_kms_server.py 7082a33 ambari-server/src/test/python/stacks/2.5/configs/ranger-kms-secured.json 7054e8f Diff: https://reviews.apache.org/r/58497/diff/3/ Changes: https://reviews.apache.org/r/58497/diff/2-3/ Testing --- Unit Test Result: Running tests for stack:2.5 service:RANGER_KMS test_configure_default (test_kms_server.TestRangerKMS) ... 2017-04-18 13:13:22,030 - Stack Feature Version Info: stack_version=2.5, version=2.5.0.0-777, current_cluster_version=2.5.0.0-777 -> 2.5.0.0-777 2017-04-18 13:13:22,046 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf ok test_configure_secured (test_kms_server.TestRangerKMS) ... 2017-04-18 13:13:22,062 - Stack Feature Version Info: stack_version=2.5, version=2.5.0.0-801, current_cluster_version=2.5.0.0-801 -> 2.5.0.0-801 2017-04-18 13:13:22,080 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf ok test_start_default (test_kms_server.TestRangerKMS) ... 2017-04-18 13:13:22,095 - Stack Feature Version Info: stack_version=2.5, version=2.5.0.0-777, current_cluster_version=2.5.0.0-777 -> 2.5.0.0-777 2017-04-18 13:13:22,111 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf 2017-04-18 13:13:22,116 - Rangeradmin: Skip ranger admin if it's down ! ok test_start_secured (test_kms_server.TestRangerKMS) ... 2017-04-18 13:13:22,129 - Stack Feature Version Info: stack_version=2.5, version=2.5.0.0-801, current_cluster_version=2.5.0.0-801 -> 2.5.0.0-801 2017-04-18 13:13:22,145 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf 2017-04-18 13:13:22,150 - RangeradminV2: Skip ranger admin if it's down ! 2017-04-18 13:13:22,151 - KMS repository c1_kms exist ok test_stop_default (test_kms_server.TestRangerKMS) ... 2017-04-18 13:13:22,163 - Stack Feature Version Info: stack_version=2.5, version=2.5.0.0-777, current_cluster_version=2.5.0.0-777 -> 2.5.0.0-777 2017-04-18 13:13:22,179 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf ok -- Ran 5 tests in 0.164s OK Thanks, Mugdha Varadkar
Re: Review Request 58497: AMBARI-20779 Create Ranger KMS HDFS audit folder as part of install
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58497/ --- (Updated April 25, 2017, 12:12 p.m.) Review request for Ambari, Alejandro Fernandez, Gautam Borad, Sumit Mohanty, and Velmurugan Periasamy. Bugs: AMBARI-20779 https://issues.apache.org/jira/browse/AMBARI-20779 Repository: ambari Description --- Create audit directory /ranger/audit/kms in hdfs, if xasecure.audit.destination.hdfs (Audit To HDFS) property is set to true. Diffs (updated) - ambari-server/src/main/resources/common-services/RANGER/0.5.0/role_command_order.json df62dfd ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/metainfo.xml b20201c ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms.py 423cdec ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/params.py 9fe0a61 ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/role_command_order.json 006d177 ambari-server/src/main/resources/stacks/HDP/2.3/role_command_order.json 4f279c6 ambari-server/src/test/python/stacks/2.5/RANGER_KMS/test_kms_server.py 7082a33 ambari-server/src/test/python/stacks/2.5/configs/ranger-kms-secured.json 7054e8f Diff: https://reviews.apache.org/r/58497/diff/2/ Changes: https://reviews.apache.org/r/58497/diff/1-2/ Testing --- Unit Test Result: Running tests for stack:2.5 service:RANGER_KMS test_configure_default (test_kms_server.TestRangerKMS) ... 2017-04-18 13:13:22,030 - Stack Feature Version Info: stack_version=2.5, version=2.5.0.0-777, current_cluster_version=2.5.0.0-777 -> 2.5.0.0-777 2017-04-18 13:13:22,046 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf ok test_configure_secured (test_kms_server.TestRangerKMS) ... 2017-04-18 13:13:22,062 - Stack Feature Version Info: stack_version=2.5, version=2.5.0.0-801, current_cluster_version=2.5.0.0-801 -> 2.5.0.0-801 2017-04-18 13:13:22,080 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf ok test_start_default (test_kms_server.TestRangerKMS) ... 2017-04-18 13:13:22,095 - Stack Feature Version Info: stack_version=2.5, version=2.5.0.0-777, current_cluster_version=2.5.0.0-777 -> 2.5.0.0-777 2017-04-18 13:13:22,111 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf 2017-04-18 13:13:22,116 - Rangeradmin: Skip ranger admin if it's down ! ok test_start_secured (test_kms_server.TestRangerKMS) ... 2017-04-18 13:13:22,129 - Stack Feature Version Info: stack_version=2.5, version=2.5.0.0-801, current_cluster_version=2.5.0.0-801 -> 2.5.0.0-801 2017-04-18 13:13:22,145 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf 2017-04-18 13:13:22,150 - RangeradminV2: Skip ranger admin if it's down ! 2017-04-18 13:13:22,151 - KMS repository c1_kms exist ok test_stop_default (test_kms_server.TestRangerKMS) ... 2017-04-18 13:13:22,163 - Stack Feature Version Info: stack_version=2.5, version=2.5.0.0-777, current_cluster_version=2.5.0.0-777 -> 2.5.0.0-777 2017-04-18 13:13:22,179 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf ok -- Ran 5 tests in 0.164s OK Thanks, Mugdha Varadkar
Re: Review Request 58497: AMBARI-20779 Create Ranger KMS HDFS audit folder as part of install
> On April 24, 2017, 6:48 p.m., Alejandro Fernandez wrote: > > ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms.py > > Lines 509 (patched) > > <https://reviews.apache.org/r/58497/diff/1/?file=1693617#file1693617line509> > > > > Can you confirm this will work even if HDFS Client is not installed on > > this host? > > That scenario may happen on upgraded clusters. Hi Alejandro, For Simple env it will work. It uses WebHDFS RESTAPI to create Hdfs related operation using curl. For Kerberos env will require HDFS Client to be installed on same node because before executing WebHDFS RESTAPI using curl, script does kinit using hdfs pricncipal. Correct principal to use is only hdfs user principal for authentication. If used other pricipal WebHDFS RESTAPI gives 403 Forbidden response for creation of directory. I can skip the creation of directory during upgrade. As Ranger KMS service(dependent on HDFS) works with HDFS service, I have added HDFS_CLIENT dependency for host where Ranger KMS will be installed. - Mugdha --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58497/#review172220 ------- On April 18, 2017, 7:51 a.m., Mugdha Varadkar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/58497/ > --- > > (Updated April 18, 2017, 7:51 a.m.) > > > Review request for Ambari, Alejandro Fernandez, Gautam Borad, Sumit Mohanty, > and Velmurugan Periasamy. > > > Bugs: AMBARI-20779 > https://issues.apache.org/jira/browse/AMBARI-20779 > > > Repository: ambari > > > Description > --- > > Create audit directory /ranger/audit/kms in hdfs, if > xasecure.audit.destination.hdfs (Audit To HDFS) property is set to true. > > > Diffs > - > > > ambari-server/src/main/resources/common-services/RANGER/0.5.0/role_command_order.json > df62dfd > > ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/metainfo.xml > b20201c > > ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms.py > 423cdec > > ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/params.py > 9fe0a61 > > ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/role_command_order.json > 006d177 > ambari-server/src/main/resources/stacks/HDP/2.3/role_command_order.json > 4f279c6 > ambari-server/src/test/python/stacks/2.5/RANGER_KMS/test_kms_server.py > 7082a33 > ambari-server/src/test/python/stacks/2.5/configs/ranger-kms-secured.json > 7054e8f > > > Diff: https://reviews.apache.org/r/58497/diff/1/ > > > Testing > --- > > Unit Test Result: > > Running tests for stack:2.5 service:RANGER_KMS > test_configure_default (test_kms_server.TestRangerKMS) ... 2017-04-18 > 13:13:22,030 - Stack Feature Version Info: stack_version=2.5, > version=2.5.0.0-777, current_cluster_version=2.5.0.0-777 -> 2.5.0.0-777 > 2017-04-18 13:13:22,046 - Using hadoop conf dir: > /usr/hdp/current/hadoop-client/conf > ok > test_configure_secured (test_kms_server.TestRangerKMS) ... 2017-04-18 > 13:13:22,062 - Stack Feature Version Info: stack_version=2.5, > version=2.5.0.0-801, current_cluster_version=2.5.0.0-801 -> 2.5.0.0-801 > 2017-04-18 13:13:22,080 - Using hadoop conf dir: > /usr/hdp/current/hadoop-client/conf > ok > test_start_default (test_kms_server.TestRangerKMS) ... 2017-04-18 > 13:13:22,095 - Stack Feature Version Info: stack_version=2.5, > version=2.5.0.0-777, current_cluster_version=2.5.0.0-777 -> 2.5.0.0-777 > 2017-04-18 13:13:22,111 - Using hadoop conf dir: > /usr/hdp/current/hadoop-client/conf > 2017-04-18 13:13:22,116 - Rangeradmin: Skip ranger admin if it's down ! > ok > test_start_secured (test_kms_server.TestRangerKMS) ... 2017-04-18 > 13:13:22,129 - Stack Feature Version Info: stack_version=2.5, > version=2.5.0.0-801, current_cluster_version=2.5.0.0-801 -> 2.5.0.0-801 > 2017-04-18 13:13:22,145 - Using hadoop conf dir: > /usr/hdp/current/hadoop-client/conf > 2017-04-18 13:13:22,150 - RangeradminV2: Skip ranger admin if it's down ! > 2017-04-18 13:13:22,151 - KMS repository c1_kms exist > ok > test_stop_default (test_kms_server.TestRangerKMS) ... 2017-04-18 13:13:22,163 > - Stack Feature Version Info: stack_version=2.5, version=2.5.0.0-777, > current_cluster_version=2.5.0.0-777 -> 2.5.0.0-777 > 2017-04-18 13:13:22,179 - Using hadoop conf dir: > /usr/hdp/current/hadoop-client/conf > ok > > -- > Ran 5 tests in 0.164s > > OK > > > Thanks, > > Mugdha Varadkar > >
Review Request 58617: AMBARI-20816 Create Ranger Hive Service during Hive Metastore restart
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58617/ --- Review request for Ambari, Alejandro Fernandez, Gautam Borad, Sumit Mohanty, and Velmurugan Periasamy. Bugs: AMBARI-20816 https://issues.apache.org/jira/browse/AMBARI-20816 Repository: ambari Description --- Need to create Ranger Hive service in Cluster1 having Hive Metastore + Ranger Service. The Ranger Hive service created in Cluster1 will be used by Cluster2 having Ranger Hive Plugin enabled under Hive Server2. Solution: Need service creation logic using Ranger REST API. The REST API code is already available for use. If ranger.service.config.param.enable.hive.metastore.lookup property present in ranger-hive-plugin-properties with value true and Ranger configurations are present in the cluster(Ranger installed in same cluster) then only create Ranger Hive service. For installation via blueprint, two extra custom config parameters ranger.service.config.param.enable.hive.metastore.lookup with value true and ranger.service.config.param.hive.site.file.path with value /etc/hive/conf/hive-site.xml are required to be provided in blueprint under ranger-hive-plugin-properties config type. For installation via UI, need to add properties in custom section of ranger-hive-plugin-properties under Hive Service config tab. Diffs - ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py 8b69e45 ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py 84bac38 ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/setup_ranger_hive.py 80bd7c8 ambari-server/src/main/resources/common-services/HIVE/2.1.0.3.0/package/scripts/hive_metastore.py 8b69e45 ambari-server/src/main/resources/common-services/HIVE/2.1.0.3.0/package/scripts/params_linux.py 84bac38 ambari-server/src/main/resources/common-services/HIVE/2.1.0.3.0/package/scripts/setup_ranger_hive.py 80bd7c8 Diff: https://reviews.apache.org/r/58617/diff/1/ Testing --- Unit Test Report: Ran 272 tests in 8.920s OK -- Total run:1194 Total errors:0 Total failures:0 OK Thanks, Mugdha Varadkar
Re: Review Request 58576: AMBARI-20792 : Atlas admin username and password should be configurable through ambari
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58576/#review172588 --- Ship it! Ship It! - Mugdha Varadkar On April 20, 2017, 3:02 p.m., Vishal Suvagia wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/58576/ > --- > > (Updated April 20, 2017, 3:02 p.m.) > > > Review request for Ambari, Alejandro Fernandez, Gautam Borad, Madhan > Neethiraj, Mugdha Varadkar, Oliver Szabo, and Sumit Mohanty. > > > Bugs: AMBARI-20792 > https://issues.apache.org/jira/browse/AMBARI-20792 > > > Repository: ambari > > > Description > --- > > Need a feature to configure Atlas admin user-name and password in Ambari. > > > Diffs > - > > > ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py > c25445c > > ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py > e243662 > > ambari-server/src/main/resources/stacks/HDP/2.6/services/ATLAS/configuration/atlas-env.xml > PRE-CREATION > > > Diff: https://reviews.apache.org/r/58576/diff/1/ > > > Testing > --- > > Verified fresh install and upgrade scenario on centos. > > > Thanks, > > Vishal Suvagia > >
Review Request 58497: AMBARI-20779 Create Ranger KMS HDFS audit folder as part of install
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58497/ --- Review request for Ambari, Alejandro Fernandez, Gautam Borad, Sumit Mohanty, and Velmurugan Periasamy. Bugs: AMBARI-20779 https://issues.apache.org/jira/browse/AMBARI-20779 Repository: ambari Description --- Create audit directory /ranger/audit/kms in hdfs, if xasecure.audit.destination.hdfs (Audit To HDFS) property is set to true. Diffs - ambari-server/src/main/resources/common-services/RANGER/0.5.0/role_command_order.json df62dfd ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/metainfo.xml b20201c ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms.py 423cdec ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/params.py 9fe0a61 ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/role_command_order.json 006d177 ambari-server/src/main/resources/stacks/HDP/2.3/role_command_order.json 4f279c6 ambari-server/src/test/python/stacks/2.5/RANGER_KMS/test_kms_server.py 7082a33 ambari-server/src/test/python/stacks/2.5/configs/ranger-kms-secured.json 7054e8f Diff: https://reviews.apache.org/r/58497/diff/1/ Testing --- Unit Test Result: Running tests for stack:2.5 service:RANGER_KMS test_configure_default (test_kms_server.TestRangerKMS) ... 2017-04-18 13:13:22,030 - Stack Feature Version Info: stack_version=2.5, version=2.5.0.0-777, current_cluster_version=2.5.0.0-777 -> 2.5.0.0-777 2017-04-18 13:13:22,046 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf ok test_configure_secured (test_kms_server.TestRangerKMS) ... 2017-04-18 13:13:22,062 - Stack Feature Version Info: stack_version=2.5, version=2.5.0.0-801, current_cluster_version=2.5.0.0-801 -> 2.5.0.0-801 2017-04-18 13:13:22,080 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf ok test_start_default (test_kms_server.TestRangerKMS) ... 2017-04-18 13:13:22,095 - Stack Feature Version Info: stack_version=2.5, version=2.5.0.0-777, current_cluster_version=2.5.0.0-777 -> 2.5.0.0-777 2017-04-18 13:13:22,111 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf 2017-04-18 13:13:22,116 - Rangeradmin: Skip ranger admin if it's down ! ok test_start_secured (test_kms_server.TestRangerKMS) ... 2017-04-18 13:13:22,129 - Stack Feature Version Info: stack_version=2.5, version=2.5.0.0-801, current_cluster_version=2.5.0.0-801 -> 2.5.0.0-801 2017-04-18 13:13:22,145 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf 2017-04-18 13:13:22,150 - RangeradminV2: Skip ranger admin if it's down ! 2017-04-18 13:13:22,151 - KMS repository c1_kms exist ok test_stop_default (test_kms_server.TestRangerKMS) ... 2017-04-18 13:13:22,163 - Stack Feature Version Info: stack_version=2.5, version=2.5.0.0-777, current_cluster_version=2.5.0.0-777 -> 2.5.0.0-777 2017-04-18 13:13:22,179 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf ok -- Ran 5 tests in 0.164s OK Thanks, Mugdha Varadkar
Re: Review Request 58075: AMBARI-20629 Take Ambari Cluster Name as a property in Ranger plugin configs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58075/ --- (Updated April 4, 2017, 5:07 a.m.) Review request for Ambari, Alejandro Fernandez, Gautam Borad, Sumit Mohanty, and Velmurugan Periasamy. Changes --- Updated Review comments Bugs: AMBARI-20629 https://issues.apache.org/jira/browse/AMBARI-20629 Repository: ambari Description --- In order to support cloud scenarios as well as Tagsync to capture Ambari cluster name. Need to add a new property that is required by plugins as well as tagsync for picking up Ambari cluster name. For Plugins: ranger.plugin..ambari.cluster.name Diffs (updated) - ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py 268d81c ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params_linux.py 41f44c3 ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/configuration/ranger-hdfs-audit.xml 731b136 ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/package/scripts/params_linux.py 62a5edd ambari-server/src/main/resources/common-services/HIVE/2.1.0.3.0/configuration/ranger-hive-audit.xml 95edb38 ambari-server/src/main/resources/common-services/KAFKA/0.8.1/package/scripts/params.py b338add ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/params_linux.py 8beb2c0 ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/params.py db59973 ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py 03d3fb2 ambari-server/src/main/resources/common-services/YARN/3.0.0.3.0/configuration/ranger-yarn-audit.xml a6b1baa ambari-server/src/main/resources/common-services/YARN/3.0.0.3.0/package/scripts/params_linux.py d069722 ambari-server/src/main/resources/stacks/HDP/2.6/services/ATLAS/configuration/ranger-atlas-audit.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.6/services/HBASE/configuration/ranger-hbase-audit.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.6/services/HDFS/configuration/ranger-hdfs-audit.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.6/services/HIVE/configuration/ranger-hive-audit.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.6/services/KAFKA/configuration/ranger-kafka-audit.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.6/services/KNOX/configuration/ranger-knox-audit.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.6/services/RANGER_KMS/configuration/ranger-kms-audit.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.6/services/STORM/configuration/ranger-storm-audit.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.6/services/YARN/configuration/ranger-yarn-audit.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml 0d71244 ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml 4a641ec ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml 0587741 Diff: https://reviews.apache.org/r/58075/diff/3/ Changes: https://reviews.apache.org/r/58075/diff/2-3/ Testing --- Tested Ranger install in fresh and EU from 2.6.x to 2.6.y. Thanks, Mugdha Varadkar
Review Request 58106: AMBARI-20636 Allow users to add custom configs for Ranger service in all plugins
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58106/
---
Review request for Ambari, Alejandro Fernandez, Gautam Borad, Sumit Mohanty,
and Velmurugan Periasamy.
Bugs: AMBARI-20636
https://issues.apache.org/jira/browse/AMBARI-20636
Repository: ambari
Description
---
In order to help Ranger to communicate with Hive metastore (in cloud env) there
is a need to allow users to add custom configs from Ambari, which needs to be
added as part of Ranger service which gets created on Enabling plugins.
This will be helpful for all plugins to define n-number of custom configs from
Ambari.
To add those configs, need to follow prefix "ranger.service.config.param" +
config parameter.
For blueprint based installs need to give those configs as below
"ranger--plugin-properties": {
"ranger.service.config.param.configParameter1" : "vaule1",
"ranger.service.config.param.configParameter2" : "value2"
}
For example in case of Hive
"ranger-hive-plugin-properties": {
"ranger.service.config.param.enable.hive.metastore.lookup": "false",
"ranger.service.config.param.hive.site.file.path":
"/etc/hive/conf/hive-site.xml"
}
For UI based installs add those properties in custom section under
ranger--plugin-properties. So that first restart of the component
can take those value after enabling the plugin for the component.
Diffs
-
ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin_xml.py
04a5bb1
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py
21d172b
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py
268d81c
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params_linux.py
41f44c3
ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/package/scripts/params_linux.py
62a5edd
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py
880ddc5
ambari-server/src/main/resources/common-services/HIVE/2.1.0.3.0/package/scripts/params_linux.py
880ddc5
ambari-server/src/main/resources/common-services/KAFKA/0.8.1/package/scripts/params.py
b338add
ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/params_linux.py
8beb2c0
ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/params.py
db59973
ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/params_linux.py
5d8a5f3
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py
03d3fb2
ambari-server/src/main/resources/common-services/YARN/3.0.0.3.0/package/scripts/params_linux.py
d069722
Diff: https://reviews.apache.org/r/58106/diff/1/
Testing
---
Tested Enabling of plugins by adding custom configs.
Thanks,
Mugdha Varadkar
Re: Review Request 58075: AMBARI-20629 Take Ambari Cluster Name as a property in Ranger plugin configs
> On March 30, 2017, 6:37 p.m., Alejandro Fernandez wrote: > > ambari-server/src/main/resources/stacks/HDP/2.6/services/YARN/configuration/ranger-yarn-audit.xml > > Lines 23 (patched) > > <https://reviews.apache.org/r/58075/diff/1/?file=1680993#file1680993line23> > > > > Same change is needed in YARN for 3.0 Updated all above review comments on latest patch - Mugdha --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58075/#review170622 --- On March 31, 2017, 3:59 a.m., Mugdha Varadkar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/58075/ > --- > > (Updated March 31, 2017, 3:59 a.m.) > > > Review request for Ambari, Alejandro Fernandez, Gautam Borad, Sumit Mohanty, > and Velmurugan Periasamy. > > > Bugs: AMBARI-20629 > https://issues.apache.org/jira/browse/AMBARI-20629 > > > Repository: ambari > > > Description > --- > > In order to support cloud scenarios as well as Tagsync to capture Ambari > cluster name. Need to add a new property that is required by plugins as well > as tagsync for picking up Ambari cluster name. > > For Plugins: > ranger.plugin..ambari.cluster.name > > > Diffs > - > > > ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py > 268d81c > > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params_linux.py > 41f44c3 > > ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/configuration/ranger-hdfs-audit.xml > 731b136 > > ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/package/scripts/params_linux.py > 62a5edd > > ambari-server/src/main/resources/common-services/HIVE/2.1.0.3.0/configuration/ranger-hive-audit.xml > 95edb38 > > ambari-server/src/main/resources/common-services/KAFKA/0.8.1/package/scripts/params.py > b338add > > ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/params_linux.py > 8beb2c0 > > ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/params.py > db59973 > > ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py > 03d3fb2 > > ambari-server/src/main/resources/common-services/YARN/3.0.0.3.0/configuration/ranger-yarn-audit.xml > a6b1baa > > ambari-server/src/main/resources/common-services/YARN/3.0.0.3.0/package/scripts/params_linux.py > d069722 > > ambari-server/src/main/resources/stacks/HDP/2.6/services/ATLAS/configuration/ranger-atlas-audit.xml > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.6/services/HBASE/configuration/ranger-hbase-audit.xml > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.6/services/HDFS/configuration/ranger-hdfs-audit.xml > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.6/services/HIVE/configuration/ranger-hive-audit.xml > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.6/services/KAFKA/configuration/ranger-kafka-audit.xml > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.6/services/KNOX/configuration/ranger-knox-audit.xml > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.6/services/RANGER_KMS/configuration/ranger-kms-audit.xml > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.6/services/STORM/configuration/ranger-storm-audit.xml > PRE-CREATION > > ambari-server/src/main/resources/stacks/HDP/2.6/services/YARN/configuration/ranger-yarn-audit.xml > PRE-CREATION > ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml > 0d71244 > > ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml > 4a641ec > ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml > 0587741 > > > Diff: https://reviews.apache.org/r/58075/diff/2/ > > > Testing > --- > > Tested Ranger install in fresh and EU from 2.6.x to 2.6.y. > > > Thanks, > > Mugdha Varadkar > >
Re: Review Request 58075: AMBARI-20629 Take Ambari Cluster Name as a property in Ranger plugin configs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58075/ --- (Updated March 31, 2017, 3:59 a.m.) Review request for Ambari, Alejandro Fernandez, Gautam Borad, Sumit Mohanty, and Velmurugan Periasamy. Changes --- Updated Review Comments. Patch for branch-2.5: https://issues.apache.org/jira/secure/attachment/12861238/AMBARI-20629.patch Bugs: AMBARI-20629 https://issues.apache.org/jira/browse/AMBARI-20629 Repository: ambari Description --- In order to support cloud scenarios as well as Tagsync to capture Ambari cluster name. Need to add a new property that is required by plugins as well as tagsync for picking up Ambari cluster name. For Plugins: ranger.plugin..ambari.cluster.name Diffs (updated) - ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py 268d81c ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params_linux.py 41f44c3 ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/configuration/ranger-hdfs-audit.xml 731b136 ambari-server/src/main/resources/common-services/HDFS/3.0.0.3.0/package/scripts/params_linux.py 62a5edd ambari-server/src/main/resources/common-services/HIVE/2.1.0.3.0/configuration/ranger-hive-audit.xml 95edb38 ambari-server/src/main/resources/common-services/KAFKA/0.8.1/package/scripts/params.py b338add ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/params_linux.py 8beb2c0 ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/params.py db59973 ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py 03d3fb2 ambari-server/src/main/resources/common-services/YARN/3.0.0.3.0/configuration/ranger-yarn-audit.xml a6b1baa ambari-server/src/main/resources/common-services/YARN/3.0.0.3.0/package/scripts/params_linux.py d069722 ambari-server/src/main/resources/stacks/HDP/2.6/services/ATLAS/configuration/ranger-atlas-audit.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.6/services/HBASE/configuration/ranger-hbase-audit.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.6/services/HDFS/configuration/ranger-hdfs-audit.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.6/services/HIVE/configuration/ranger-hive-audit.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.6/services/KAFKA/configuration/ranger-kafka-audit.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.6/services/KNOX/configuration/ranger-knox-audit.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.6/services/RANGER_KMS/configuration/ranger-kms-audit.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.6/services/STORM/configuration/ranger-storm-audit.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.6/services/YARN/configuration/ranger-yarn-audit.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml 0d71244 ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml 4a641ec ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml 0587741 Diff: https://reviews.apache.org/r/58075/diff/2/ Changes: https://reviews.apache.org/r/58075/diff/1-2/ Testing --- Tested Ranger install in fresh and EU from 2.6.x to 2.6.y. Thanks, Mugdha Varadkar
Review Request 58075: AMBARI-20629 Take Ambari Cluster Name as a property in Ranger plugin configs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58075/ --- Review request for Ambari, Alejandro Fernandez, Gautam Borad, Sumit Mohanty, and Velmurugan Periasamy. Bugs: AMBARI-20629 https://issues.apache.org/jira/browse/AMBARI-20629 Repository: ambari Description --- In order to support cloud scenarios as well as Tagsync to capture Ambari cluster name. Need to add a new property that is required by plugins as well as tagsync for picking up Ambari cluster name. For Plugins: ranger.plugin..ambari.cluster.name Diffs - ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py ab8a4d9 ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params_linux.py 07cb409 ambari-server/src/main/resources/common-services/KAFKA/0.8.1/package/scripts/params.py 69d801a ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/params_linux.py f461f41 ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/params.py db59973 ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py f9228be ambari-server/src/main/resources/stacks/HDP/2.6/services/ATLAS/configuration/ranger-atlas-audit.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.6/services/HBASE/configuration/ranger-hbase-audit.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.6/services/HDFS/configuration/ranger-hdfs-audit.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.6/services/HIVE/configuration/ranger-hive-audit.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.6/services/KAFKA/configuration/ranger-kafka-audit.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.6/services/KNOX/configuration/ranger-knox-audit.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.6/services/RANGER_KMS/configuration/ranger-kms-audit.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.6/services/STORM/configuration/ranger-storm-audit.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.6/services/YARN/configuration/ranger-yarn-audit.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml 0d71244 ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml 035cb59 ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml 458667d Diff: https://reviews.apache.org/r/58075/diff/1/ Testing --- Tested Ranger install in fresh and EU from 2.6.x to 2.6.y. Thanks, Mugdha Varadkar
Review Request 57452: AMBARI-20369 Need hdfs-site for saving ranger audits to hdfs in namenode HA env
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/57452/ --- Review request for Ambari, Alejandro Fernandez, Gautam Borad, and Velmurugan Periasamy. Bugs: AMBARI-20369 https://issues.apache.org/jira/browse/AMBARI-20369 Repository: ambari Description --- For KNOX and RANGER_KMS services which supports ranger plugin, need to have hdfs-site.xml available in respective services conf directory for saving ranger audits to hdfs in namenode HA env. Diffs - ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/setup_ranger_knox.py 67a1670 ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms.py 1afe136 ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/params.py dc830d5 ambari-server/src/test/python/stacks/2.5/RANGER_KMS/test_kms_server.py 6f41b6d Diff: https://reviews.apache.org/r/57452/diff/1/ Testing --- Manualy tested audits to hdfs working in namenode HA env for KNOX and RANGER_KMS Thanks, Mugdha Varadkar
Re: Review Request 57396: AMBARI-20329 After restarting Ranger, PAM files are overwritten by default template
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/57396/#review168246 --- Ship it! Ship It! - Mugdha Varadkar On March 7, 2017, 10:45 p.m., Shi Wang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/57396/ > --- > > (Updated March 7, 2017, 10:45 p.m.) > > > Review request for Ambari, Mugdha Varadkar and Robert Levas. > > > Bugs: AMBARI-20329 > https://issues.apache.org/jira/browse/AMBARI-20329 > > > Repository: ambari > > > Description > --- > > AMBARI-18425 add PAM support for ranger authentication in Ambari, but every > time restart ranger-admin it will generate the files again, which will > overwrite the user change. Need to check first if these files already exist, > do not generate again. > > > Diffs > - > > > ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py > 15f7cf7 > > > Diff: https://reviews.apache.org/r/57396/diff/1/ > > > Testing > --- > > 1. Change ranger pam files > 2. Restart ranger > 3. Check if pam files settings will be overwritten by template. > > > Thanks, > > Shi Wang > >
Re: Review Request 57350: Kerberos identity reference not working for ranger-audit property in hbase
> On March 7, 2017, 1:24 p.m., Mugdha Varadkar wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog250.java
> > Lines 583 (patched)
> > <https://reviews.apache.org/r/57350/diff/1/?file=1656837#file1656837line588>
> >
> > Just curious to know whether principal/keytab descriptor should be
> > mapped to hbase principal/keytab rather than null ?
>
> Robert Levas wrote:
> Setting the principal value and the keytab file to `null` indicates that
> this Kerberos identitiy descriptor is to inherit the value from the reference
> identitiy - which is what we want in this case. For exampe, looking at
> https://github.com/apache/ambari/blob/trunk/ambari-server/src/main/resources/stacks/HDP/2.5/services/HBASE/kerberos.json#L68,
> the principal value will become `hbase/_HOST@${realm}`.
Okay, Thank you.
- Mugdha
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57350/#review168122
---
On March 7, 2017, 12:13 a.m., Robert Levas wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57350/
> ---
>
> (Updated March 7, 2017, 12:13 a.m.)
>
>
> Review request for Ambari, Attila Magyar, Balázs Bence Sári, Eugene
> Chekanskiy, Laszlo Puskas, Mugdha Varadkar, and Sebastian Toader.
>
>
> Bugs: AMBARI-20335
> https://issues.apache.org/jira/browse/AMBARI-20335
>
>
> Repository: ambari
>
>
> Description
> ---
>
> From stack 2.5 onwards
> `xasecure.audit.jaas.Client.option.principal/ranger-hbase-audit` needs to
> have principal value available under
> `hbase.master.kerberos.principal/hbase-site`
>
> To achieve that added below block of code under hbase
> [kerberos.json|https://github.com/apache/ambari/blob/branch-2.5/ambari-server/src/main/resources/stacks/HDP/2.5/services/HBASE/kerberos.json]
> ```
> {
> "name": "/HBASE/HBASE_MASTER/hbase_master_hbase",
> "principal": {
> "configuration":
> "ranger-hbase-audit/xasecure.audit.jaas.Client.option.principal"
> },
> "keytab": {
> "configuration":
> "ranger-hbase-audit/xasecure.audit.jaas.Client.option.keyTab"
> }
> }
> ```
>
> But on test cluster,
> `xasecure.audit.jaas.Client.option.principal/ranger-hbase-audit` property is
> not showing the expected value. It is showing the principal/keytab values of
> `ams_hbase_master_hbase` identity.
>
> Because of wrong reference of principal audit to solr is not working in
> kerberos environment, as security.json have below entry instead of
> `hb...@example.com`
> ```
> "amshb...@example.com":[
> "ranger_audit_user",
> "dev"]
> ```
>
>
> Diffs
> -
>
>
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog250.java
> 141e9cd
>
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HBASE/kerberos.json
> f510770
>
> ambari-server/src/main/resources/stacks/PERF/1.0/services/FAKEHBASE/kerberos.json
> b053779
>
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog250Test.java
> 2a59ccc
>
> ambari-server/src/test/resources/kerberos/test_kerberos_descriptor_2_5_infra_solr.json
> 0c2723e
>
>
> Diff: https://reviews.apache.org/r/57350/diff/1/
>
>
> Testing
> ---
>
> Manually tested in Ambari 2.5.0 cluster and upgrade from Ambari 2.4.2.
>
> # Local test results:
>
> ```
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 23:53.766s
> [INFO] Finished at: Mon Mar 06 16:55:35 EST 2017
> [INFO] Final Memory: 71M/772M
> [INFO]
>
> ```
>
> # Jenkins test results: PENDING
>
>
> Thanks,
>
> Robert Levas
>
>
Re: Review Request 57350: Kerberos identity reference not working for ranger-audit property in hbase
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57350/#review168126
---
Ship it!
Ship It!
- Mugdha Varadkar
On March 7, 2017, 12:13 a.m., Robert Levas wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57350/
> ---
>
> (Updated March 7, 2017, 12:13 a.m.)
>
>
> Review request for Ambari, Attila Magyar, Balázs Bence Sári, Eugene
> Chekanskiy, Laszlo Puskas, Mugdha Varadkar, and Sebastian Toader.
>
>
> Bugs: AMBARI-20335
> https://issues.apache.org/jira/browse/AMBARI-20335
>
>
> Repository: ambari
>
>
> Description
> ---
>
> From stack 2.5 onwards
> `xasecure.audit.jaas.Client.option.principal/ranger-hbase-audit` needs to
> have principal value available under
> `hbase.master.kerberos.principal/hbase-site`
>
> To achieve that added below block of code under hbase
> [kerberos.json|https://github.com/apache/ambari/blob/branch-2.5/ambari-server/src/main/resources/stacks/HDP/2.5/services/HBASE/kerberos.json]
> ```
> {
> "name": "/HBASE/HBASE_MASTER/hbase_master_hbase",
> "principal": {
> "configuration":
> "ranger-hbase-audit/xasecure.audit.jaas.Client.option.principal"
> },
> "keytab": {
> "configuration":
> "ranger-hbase-audit/xasecure.audit.jaas.Client.option.keyTab"
> }
> }
> ```
>
> But on test cluster,
> `xasecure.audit.jaas.Client.option.principal/ranger-hbase-audit` property is
> not showing the expected value. It is showing the principal/keytab values of
> `ams_hbase_master_hbase` identity.
>
> Because of wrong reference of principal audit to solr is not working in
> kerberos environment, as security.json have below entry instead of
> `hb...@example.com`
> ```
> "amshb...@example.com":[
> "ranger_audit_user",
> "dev"]
> ```
>
>
> Diffs
> -
>
>
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog250.java
> 141e9cd
>
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HBASE/kerberos.json
> f510770
>
> ambari-server/src/main/resources/stacks/PERF/1.0/services/FAKEHBASE/kerberos.json
> b053779
>
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog250Test.java
> 2a59ccc
>
> ambari-server/src/test/resources/kerberos/test_kerberos_descriptor_2_5_infra_solr.json
> 0c2723e
>
>
> Diff: https://reviews.apache.org/r/57350/diff/1/
>
>
> Testing
> ---
>
> Manually tested in Ambari 2.5.0 cluster and upgrade from Ambari 2.4.2.
>
> # Local test results:
>
> ```
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 23:53.766s
> [INFO] Finished at: Mon Mar 06 16:55:35 EST 2017
> [INFO] Final Memory: 71M/772M
> [INFO]
>
> ```
>
> # Jenkins test results: PENDING
>
>
> Thanks,
>
> Robert Levas
>
>
Re: Review Request 57350: Kerberos identity reference not working for ranger-audit property in hbase
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57350/#review168122
---
ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog250.java
Lines 583 (patched)
<https://reviews.apache.org/r/57350/#comment240254>
Just curious to know whether principal/keytab descriptor should be mapped
to hbase principal/keytab rather than null ?
- Mugdha Varadkar
On March 7, 2017, 12:13 a.m., Robert Levas wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57350/
> ---
>
> (Updated March 7, 2017, 12:13 a.m.)
>
>
> Review request for Ambari, Attila Magyar, Balázs Bence Sári, Eugene
> Chekanskiy, Laszlo Puskas, Mugdha Varadkar, and Sebastian Toader.
>
>
> Bugs: AMBARI-20335
> https://issues.apache.org/jira/browse/AMBARI-20335
>
>
> Repository: ambari
>
>
> Description
> ---
>
> From stack 2.5 onwards
> `xasecure.audit.jaas.Client.option.principal/ranger-hbase-audit` needs to
> have principal value available under
> `hbase.master.kerberos.principal/hbase-site`
>
> To achieve that added below block of code under hbase
> [kerberos.json|https://github.com/apache/ambari/blob/branch-2.5/ambari-server/src/main/resources/stacks/HDP/2.5/services/HBASE/kerberos.json]
> ```
> {
> "name": "/HBASE/HBASE_MASTER/hbase_master_hbase",
> "principal": {
> "configuration":
> "ranger-hbase-audit/xasecure.audit.jaas.Client.option.principal"
> },
> "keytab": {
> "configuration":
> "ranger-hbase-audit/xasecure.audit.jaas.Client.option.keyTab"
> }
> }
> ```
>
> But on test cluster,
> `xasecure.audit.jaas.Client.option.principal/ranger-hbase-audit` property is
> not showing the expected value. It is showing the principal/keytab values of
> `ams_hbase_master_hbase` identity.
>
> Because of wrong reference of principal audit to solr is not working in
> kerberos environment, as security.json have below entry instead of
> `hb...@example.com`
> ```
> "amshb...@example.com":[
> "ranger_audit_user",
> "dev"]
> ```
>
>
> Diffs
> -
>
>
> ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog250.java
> 141e9cd
>
> ambari-server/src/main/resources/stacks/HDP/2.5/services/HBASE/kerberos.json
> f510770
>
> ambari-server/src/main/resources/stacks/PERF/1.0/services/FAKEHBASE/kerberos.json
> b053779
>
> ambari-server/src/test/java/org/apache/ambari/server/upgrade/UpgradeCatalog250Test.java
> 2a59ccc
>
> ambari-server/src/test/resources/kerberos/test_kerberos_descriptor_2_5_infra_solr.json
> 0c2723e
>
>
> Diff: https://reviews.apache.org/r/57350/diff/1/
>
>
> Testing
> ---
>
> Manually tested in Ambari 2.5.0 cluster and upgrade from Ambari 2.4.2.
>
> # Local test results:
>
> ```
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 23:53.766s
> [INFO] Finished at: Mon Mar 06 16:55:35 EST 2017
> [INFO] Final Memory: 71M/772M
> [INFO]
>
> ```
>
> # Jenkins test results: PENDING
>
>
> Thanks,
>
> Robert Levas
>
>
Review Request 57337: AMBARI-20317 Update stack advisor logic for getting enable atlas hook flag value
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/57337/ --- Review request for Ambari, Alejandro Fernandez, Gautam Borad, and Sumit Mohanty. Bugs: AMBARI-20317 https://issues.apache.org/jira/browse/AMBARI-20317 Repository: ambari Description --- Stack advisor recommendation for atlas hooks not working as expected due to AMBARI-20304 changes. Diffs - ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/configuration/sqoop-site.xml 5d87c4d ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py f18bfe9 Diff: https://reviews.apache.org/r/57337/diff/1/ Testing --- test_recommendHiveConfigurations_with_atlas (test_stack_advisor.TestHDP23StackAdvisor) ... ok test_recommendSqoopConfigurations (test_stack_advisor.TestHDP23StackAdvisor) ... ok test_recommendStormConfigurations (test_stack_advisor.TestHDP23StackAdvisor) ... ok Thanks, Mugdha Varadkar
Re: Review Request 57139: AMBARI-20208 : Atlas kafka servers should be configured using kafka listeners.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/57139/#review167272 --- Ship it! Ship It! - Mugdha Varadkar On March 1, 2017, 12:36 p.m., Vishal Suvagia wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/57139/ > --- > > (Updated March 1, 2017, 12:36 p.m.) > > > Review request for Ambari, Alejandro Fernandez, Gautam Borad, Madhan > Neethiraj, Mugdha Varadkar, Sumit Mohanty, and Sriharsha Chintalapani. > > > Bugs: AMBARI-20208 > https://issues.apache.org/jira/browse/AMBARI-20208 > > > Repository: ambari > > > Description > --- > > Adding dependency of kafka broker's listeners config to populate accurate > port for Atlas atlas.kafka.bootstrap.servers property. > > > Diffs > - > > > ambari-server/src/main/resources/common-services/ATLAS/0.7.0.2.5/configuration/application-properties.xml > 15cc73b > ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py > 6f3dfa7 > ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py > cf1c0ee > > > Diff: https://reviews.apache.org/r/57139/diff/2/ > > > Testing > --- > > Verified dependency of : listeners propert for following cases: > > 1) Simple environment with listener's value as : PLAINTEXT://localhost: > > 2) Kerberos environment with listener's value as : > PLAINTEXTSASL://localhost: > > 3) Kerberos environment with listener's value as : > PLAINTEXTSASL://localhost:,PLAINTEXT://localhost: > > 4) Kerberos environment with listener's value as : > PLAINTEXT://localhost: :: in this case it will pickup the port given > for PLAINTEXT protocol only (as SASL port is not specified). > > > Thanks, > > Vishal Suvagia > >
Review Request 57140: AMBARI-20232 Updating text for Ranger Admin manual task for upgrade
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/57140/ --- Review request for Ambari, Alejandro Fernandez, Gautam Borad, and Velmurugan Periasamy. Bugs: AMBARI-20232 https://issues.apache.org/jira/browse/AMBARI-20232 Repository: ambari Description --- Updating text for Ranger Admin manual task for upgrade. Updating if for below stack upgrades: 2.5.x to 2.5.y 2.5.x to 2.6.x 2.6.x to 2.6.y Diffs - ambari-server/src/main/resources/common-services/RANGER/0.6.0/metainfo.xml 12fde7e ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.5.xml 84af62e ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml 3a4797b ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.5.xml 7205924 ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml c01d11e ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml 921e479 ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml 8669d2b Diff: https://reviews.apache.org/r/57140/diff/ Testing --- Tested EU from 2.5 to 2.6 Thanks, Mugdha Varadkar
Re: Review Request 56997: Use storm user principal instead of nimbus user principal for ranger audit
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/56997/#review166697 --- Ship it! Ship It! - Mugdha Varadkar On Feb. 23, 2017, 8:49 p.m., Oliver Szabo wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/56997/ > --- > > (Updated Feb. 23, 2017, 8:49 p.m.) > > > Review request for Ambari, Miklos Gergely, Mugdha Varadkar, Robert Levas, and > Robert Nettleton. > > > Bugs: AMBARI-20152 > https://issues.apache.org/jira/browse/AMBARI-20152 > > > Repository: ambari > > > Description > --- > > Use storm principal and keytab for ranger plugin instead of nimbus ones. > In storm code, storm user will be used globally anyway, ranger plugin will > use that. In Ambari 2.4 that not caused any issues, but from Ambari 2.5, Solr > authorization is supported, that can cause if storm is authenticated with the > worng user, it wont be able to access the ranger audit collection. > > > Diffs > - > > ambari-server/src/main/resources/common-services/STORM/1.0.1/kerberos.json > fecef7c > > Diff: https://reviews.apache.org/r/56997/diff/ > > > Testing > --- > > done. > > > Thanks, > > Oliver Szabo > >
Re: Review Request 56997: Use storm user principal instead of nimbus user principal for ranger audit
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/56997/#review15 --- ambari-server/src/main/resources/common-services/STORM/1.0.1/kerberos.json (line 109) <https://reviews.apache.org/r/56997/#comment238697> Will this property be updated after ambari upgrade to use storm_components principal ? - Mugdha Varadkar On Feb. 23, 2017, 8:49 p.m., Oliver Szabo wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/56997/ > --- > > (Updated Feb. 23, 2017, 8:49 p.m.) > > > Review request for Ambari, Miklos Gergely, Mugdha Varadkar, Robert Levas, and > Robert Nettleton. > > > Bugs: AMBARI-20152 > https://issues.apache.org/jira/browse/AMBARI-20152 > > > Repository: ambari > > > Description > --- > > Use storm principal and keytab for ranger plugin instead of nimbus ones. > In storm code, storm user will be used globally anyway, ranger plugin will > use that. In Ambari 2.4 that not caused any issues, but from Ambari 2.5, Solr > authorization is supported, that can cause if storm is authenticated with the > worng user, it wont be able to access the ranger audit collection. > > > Diffs > - > > ambari-server/src/main/resources/common-services/STORM/1.0.1/kerberos.json > fecef7c > > Diff: https://reviews.apache.org/r/56997/diff/ > > > Testing > --- > > done. > > > Thanks, > > Oliver Szabo > >
Re: Review Request 56956: Remove OrderedDict python module code from ranger stack
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/56956/#review166481 --- Ship it! Ship It! - Mugdha Varadkar On Feb. 22, 2017, 11:16 p.m., Oliver Szabo wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/56956/ > --- > > (Updated Feb. 22, 2017, 11:16 p.m.) > > > Review request for Ambari, Jonathan Hurley, Miklos Gergely, Mugdha Varadkar, > and Robert Nettleton. > > > Bugs: AMBARI-20124 > https://issues.apache.org/jira/browse/AMBARI-20124 > > > Repository: ambari > > > Description > --- > > remove collections.OrderedDict from ranger stack code. (it seems on my > environemnt that module was installed with python 2.6.6, but its supported > from 2.7, in 2.6 ordereddict.OrderedDict can be an option) > But: use list of tuples instead of it. > > > Diffs > - > > > ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py > b3eb919 > > Diff: https://reviews.apache.org/r/56956/diff/ > > > Testing > --- > > done. > > > Thanks, > > Oliver Szabo > >
Re: Review Request 56927: AMBARI-20116 : Stack Recommendation for Atlas not working on change of Kafka broker port.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/56927/#review166480 --- Ship it! Ship It! - Mugdha Varadkar On Feb. 22, 2017, 3:34 p.m., Vishal Suvagia wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/56927/ > --- > > (Updated Feb. 22, 2017, 3:34 p.m.) > > > Review request for Ambari, Alejandro Fernandez, Gautam Borad, Madhan > Neethiraj, Mugdha Varadkar, Oliver Szabo, and Sumit Mohanty. > > > Bugs: AMBARI-20116 > https://issues.apache.org/jira/browse/AMBARI-20116 > > > Repository: ambari > > > Description > --- > > When Kafka-broker port is changed, appropriate change should be made for > Atlas Kafla broker servers, this does not happen as stack-dependency for the > same is missing. > > > Diffs > - > > > ambari-server/src/main/resources/common-services/ATLAS/0.7.0.2.5/configuration/application-properties.xml > 36702e8 > > Diff: https://reviews.apache.org/r/56927/diff/ > > > Testing > --- > > Verified with a fresh install on CentOS-6. > > > Thanks, > > Vishal Suvagia > >
Re: Review Request 56759: AMBARI-20047 : Repository config user needs to set customized service user in Ranger when service plugin is enabled.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/56759/#review166103 --- Ship it! Ship It! - Mugdha Varadkar On Feb. 20, 2017, 3:48 p.m., Vishal Suvagia wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/56759/ > --- > > (Updated Feb. 20, 2017, 3:48 p.m.) > > > Review request for Ambari, Alejandro Fernandez, Gautam Borad, Mugdha > Varadkar, Oliver Szabo, Sumit Mohanty, and Velmurugan Periasamy. > > > Bugs: AMBARI-20047 > https://issues.apache.org/jira/browse/AMBARI-20047 > > > Repository: ambari > > > Description > --- > > The property REPOSITORY_CONFIG_USERNAME in Advanced > ranger-*-plugin-properties is not updated to the custom user-name when > services are using custom users. > This causes issues especially in cases of Test connection, and policy user > which is given default permissions in Ranger. > > > Diffs > - > > > ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin_xml.py > 56c46dd > > ambari-server/src/main/resources/stacks/HDP/2.6/services/ATLAS/configuration/ranger-atlas-plugin-properties.xml > d8885e5 > > ambari-server/src/main/resources/stacks/HDP/2.6/services/HBASE/configuration/ranger-hbase-plugin-properties.xml > d8885e5 > > ambari-server/src/main/resources/stacks/HDP/2.6/services/HDFS/configuration/ranger-hdfs-plugin-properties.xml > 6e93879 > > ambari-server/src/main/resources/stacks/HDP/2.6/services/HIVE/configuration/ranger-hive-plugin-properties.xml > d8885e5 > > ambari-server/src/main/resources/stacks/HDP/2.6/services/KAFKA/configuration/ranger-kafka-plugin-properties.xml > d8885e5 > > ambari-server/src/main/resources/stacks/HDP/2.6/services/YARN/configuration/ranger-yarn-plugin-properties.xml > d8885e5 > ambari-server/src/main/resources/stacks/HDP/2.6/services/stack_advisor.py > 3dd1bdd > ambari-server/src/test/python/stacks/2.6/common/test_stack_advisor.py > c15eaf1 > > Diff: https://reviews.apache.org/r/56759/diff/ > > > Testing > --- > > Verified with a fresh installation on CentOS-6. > > > Thanks, > > Vishal Suvagia > >
Re: Review Request 56671: Add Solr authorization settings during LogSearch/Atlas/Ranger startup
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56671/#review165727
---
Fix it, then Ship it!
ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py
(line 679)
<https://reviews.apache.org/r/56671/#comment237578>
Please add defaults for atlas, yarn and kms. For kms the default principal
name is rangerkms.
- Mugdha Varadkar
On Feb. 15, 2017, 2:36 p.m., Oliver Szabo wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56671/
> ---
>
> (Updated Feb. 15, 2017, 2:36 p.m.)
>
>
> Review request for Ambari, Miklos Gergely, Mugdha Varadkar, and Robert
> Nettleton.
>
>
> Bugs: AMBARI-20013
> https://issues.apache.org/jira/browse/AMBARI-20013
>
>
> Repository: ambari
>
>
> Description
> ---
>
> - make sure solr keytab and solr user is created on logsearch/ranger/atlas
> hosts
> - created a new solr_cloud_util method called add roles (curl for adding user
> roles)
> - updated secure znode method (adding infra-solr by default)
> - move generated security.json into a static file
> - added a custom security.json which can be used instead of the generated
> security.json
>
> In ranger/atlas stack, the following calls can be used for securing znode and
> update user-roles:
>
> solr_cloud_util.secure_znode(config=params.config,
> zookeeper_quorum=params.zookeeper_quorum,
>
> solr_znode=format("{infra_solr_znode}/collections/mycollectionznode"),
> jaas_file=params.logsearch_jaas_file,
> java64_home=params.java64_home,
> sasl_users=["myuser1@HOST", "myuser2"])
>
> and
>
> solr_cloud_util.add_solr_roles(params.config,
>roles = ["ranger_audit_user"],
>new_service_principals = ["audituser1",
> "audituser2"])
>
> about ranger plugins:
>
> there is a way to use these methods in other services like storm etc. , the
> problem is its needed to infra-solr be started, so if we include these in a
> lot of other services, it can really slows down the deployment. that is the
> reason why i kept the property dependencies in infra-solr-security-json
> config. other then that, with the changes ranger can start successfully if
> its added later to the cluster, and for plugins, infra-solr will be flagged
> to be restarted.
>
>
> Diffs
> -
>
>
> ambari-common/src/main/python/resource_management/libraries/functions/solr_cloud_util.py
> 4628211
>
> ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/configuration/infra-solr-security-json.xml
> e193a8c
>
> ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/package/scripts/params.py
> ab9aa61
>
> ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/package/scripts/setup_infra_solr.py
> 8d72f42
>
> ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/package/templates/infra-solr-security.json.j2
> PRE-CREATION
>
> ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/properties/infra-solr-security.json.j2
> ed764f0
>
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py
> 2232bb2
>
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py
> 682fc9f
>
> ambari-server/src/main/resources/common-services/ATLAS/0.7.0.2.5/kerberos.json
> bc8e351
>
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/kerberos.json
> 49d1b10
>
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/params.py
> fecd802
>
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/setup_logsearch.py
> ba91e20
>
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py
> 0b4532b
>
> ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py
> ae49c4f
> ambari-server/src/main/resources/common-services/RANGER/0.6.0/kerberos.json
> 253e32e
> ambari-server/src/test/python/stacks/2.3/ATLAS/test_metadata_server.py
> 1bbf75e
> am
Re: Review Request 56671: Add Solr authorization settings during LogSearch/Atlas/Ranger startup
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56671/#review165686
---
ambari-server/src/main/resources/common-services/RANGER/0.6.0/kerberos.json
(line 75)
<https://reviews.apache.org/r/56671/#comment237540>
0.6.0 version is extended from stack 2.5 which is released.
If user upgrading to ambari with this fix, will infra-solr principal get
generated for ranger-admin host after ambari upgrade ?
- Mugdha Varadkar
On Feb. 14, 2017, 7:06 p.m., Oliver Szabo wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56671/
> ---
>
> (Updated Feb. 14, 2017, 7:06 p.m.)
>
>
> Review request for Ambari, Miklos Gergely, Mugdha Varadkar, and Robert
> Nettleton.
>
>
> Bugs: AMBARI-20013
> https://issues.apache.org/jira/browse/AMBARI-20013
>
>
> Repository: ambari
>
>
> Description
> ---
>
> - make sure solr keytab and solr user is created on logsearch/ranger/atlas
> hosts
> - created a new solr_cloud_util method called add roles (curl for adding user
> roles)
> - updated secure znode method (adding infra-solr by default)
> - move generated security.json into a static file
> - added a custom security.json which can be used instead of the generated
> security.json
>
> This patch still does not contain atlas/ranger fixes (it will, that review is
> mosly for review, i will include ranger/atlas changes later).
>
> In ranger/atlas stack, the following calls can be used for securing znode and
> update user-roles:
>
> solr_cloud_util.secure_znode(config=params.config,
> zookeeper_quorum=params.zookeeper_quorum,
>
> solr_znode=format("{infra_solr_znode}/collections/mycollectionznode"),
> jaas_file=params.logsearch_jaas_file,
> java64_home=params.java64_home,
> sasl_users=["myuser1@HOST", "myuser2"])
>
> and
>
> solr_cloud_util.add_solr_roles(params.config,
>roles = ["ranger_audit_user"],
>new_service_principals = ["audituser1",
> "audituser2"])
>
> about ranger plugins:
>
> there is a way to use these methods in other services like storm etc. , the
> problem is its needed to infra-solr be started, so if we include these in a
> lot of other services, it can really slows down the deployment. that is the
> reason why i kept the property dependencies in infra-solr-security-json
> config. other then that, with the changes ranger can start successfully if
> its added later to the cluster, and for plugins, infra-solr will be flagged
> to be restarted.
>
>
> Diffs
> -
>
>
> ambari-common/src/main/python/resource_management/libraries/functions/solr_cloud_util.py
> 4628211
>
> ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/configuration/infra-solr-security-json.xml
> e193a8c
>
> ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/package/scripts/params.py
> ab9aa61
>
> ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/package/scripts/setup_infra_solr.py
> 8d72f42
>
> ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/package/templates/infra-solr-security.json.j2
> PRE-CREATION
>
> ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/properties/infra-solr-security.json.j2
> ed764f0
>
> ambari-server/src/main/resources/common-services/ATLAS/0.7.0.2.5/kerberos.json
> bc8e351
>
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/kerberos.json
> 49d1b10
>
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/params.py
> fecd802
>
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/setup_logsearch.py
> ba91e20
> ambari-server/src/main/resources/common-services/RANGER/0.6.0/kerberos.json
> 253e32e
> ambari-server/src/test/python/stacks/2.4/AMBARI_INFRA/test_infra_solr.py
> cd88fec
>
> Diff: https://reviews.apache.org/r/56671/diff/
>
>
> Testing
> ---
>
> unit tests done, still writing unit tests for atlas/ranger additions.
>
>
> Thanks,
>
> Oliver Szabo
>
>
Re: Review Request 56671: Add Solr authorization settings during LogSearch/Atlas/Ranger startup
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56671/#review165683
---
ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/package/scripts/setup_infra_solr.py
(line 75)
<https://reviews.apache.org/r/56671/#comment237536>
How is this going to get uploaded to zookeeper node ? or is this a manual
task for user using custom-security.json ?
If infra-solr is restarted the custom-security.json will get override with
static security.json template
- Mugdha Varadkar
On Feb. 14, 2017, 7:06 p.m., Oliver Szabo wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56671/
> ---
>
> (Updated Feb. 14, 2017, 7:06 p.m.)
>
>
> Review request for Ambari, Miklos Gergely, Mugdha Varadkar, and Robert
> Nettleton.
>
>
> Bugs: AMBARI-20013
> https://issues.apache.org/jira/browse/AMBARI-20013
>
>
> Repository: ambari
>
>
> Description
> ---
>
> - make sure solr keytab and solr user is created on logsearch/ranger/atlas
> hosts
> - created a new solr_cloud_util method called add roles (curl for adding user
> roles)
> - updated secure znode method (adding infra-solr by default)
> - move generated security.json into a static file
> - added a custom security.json which can be used instead of the generated
> security.json
>
> This patch still does not contain atlas/ranger fixes (it will, that review is
> mosly for review, i will include ranger/atlas changes later).
>
> In ranger/atlas stack, the following calls can be used for securing znode and
> update user-roles:
>
> solr_cloud_util.secure_znode(config=params.config,
> zookeeper_quorum=params.zookeeper_quorum,
>
> solr_znode=format("{infra_solr_znode}/collections/mycollectionznode"),
> jaas_file=params.logsearch_jaas_file,
> java64_home=params.java64_home,
> sasl_users=["myuser1@HOST", "myuser2"])
>
> and
>
> solr_cloud_util.add_solr_roles(params.config,
>roles = ["ranger_audit_user"],
>new_service_principals = ["audituser1",
> "audituser2"])
>
> about ranger plugins:
>
> there is a way to use these methods in other services like storm etc. , the
> problem is its needed to infra-solr be started, so if we include these in a
> lot of other services, it can really slows down the deployment. that is the
> reason why i kept the property dependencies in infra-solr-security-json
> config. other then that, with the changes ranger can start successfully if
> its added later to the cluster, and for plugins, infra-solr will be flagged
> to be restarted.
>
>
> Diffs
> -
>
>
> ambari-common/src/main/python/resource_management/libraries/functions/solr_cloud_util.py
> 4628211
>
> ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/configuration/infra-solr-security-json.xml
> e193a8c
>
> ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/package/scripts/params.py
> ab9aa61
>
> ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/package/scripts/setup_infra_solr.py
> 8d72f42
>
> ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/package/templates/infra-solr-security.json.j2
> PRE-CREATION
>
> ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/properties/infra-solr-security.json.j2
> ed764f0
>
> ambari-server/src/main/resources/common-services/ATLAS/0.7.0.2.5/kerberos.json
> bc8e351
>
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/kerberos.json
> 49d1b10
>
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/params.py
> fecd802
>
> ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/setup_logsearch.py
> ba91e20
> ambari-server/src/main/resources/common-services/RANGER/0.6.0/kerberos.json
> 253e32e
> ambari-server/src/test/python/stacks/2.4/AMBARI_INFRA/test_infra_solr.py
> cd88fec
>
> Diff: https://reviews.apache.org/r/56671/diff/
>
>
> Testing
> ---
>
> unit tests done, still writing unit tests for atlas/ranger additions.
>
>
> Thanks,
>
> Oliver Szabo
>
>
Re: Review Request 56607: AMBARI-19915 Add Ranger KMS SSL properties in ambari stack
> On Feb. 13, 2017, 7:18 p.m., Alejandro Fernandez wrote: > > ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms.py, > > line 148 > > <https://reviews.apache.org/r/56607/diff/1/?file=1632316#file1632316line148> > > > > Today, we have symlinks from /etc/$comp/conf -> > > /usr/hdp/current/$comp/conf so that configs still work during stack > > upgrades (RU/EU). This seems to be a flat directory. > > > > How is it going to be used? > > What will happen during stack upgrades or downgrades? > > Mugdha Varadkar wrote: > Hi Alejandro, > > For Ranger KMS: the "conf" symlinks is created under the path > /etc/ranger/kms/. This flat directory path is created by rpm package. I added > this code if in case the path is not created, it may be created from Ambari, > also this path is needed to create some additional files. > > Alejandro Fernandez wrote: > What is actually stored in that directory? > If any files are overriden by installing a different version, then that > will cause problems during stack upgrade. > We shouldn't be using fixed paths any more. Basically there are two directories created 1. /etc/ranger/kms/conf (created by rpm/deb package installation which helps for manual/non-ambari installation) 2. /etc/ranger-kms//0 (created by conf-select installation for ambari based installs) 3. /usr/hdp/current/ranger-kms/conf is also a symlink which points to /etc/ranger-kms//0 What is actually stored in that directory? rangerkms.jceks file is stored in that directory. The file is created/modified during every start of ranger-kms service. Below is the structure of the /etc/ranger/kms/ directory: - conf -> /usr/hdp/current/ranger-kms/conf - conf.backup - rangerkms.jceks During upgrade we do conf-select which will set versioned config directories and it copies /usr/hdp/current/ranger-kms/conf/* to /etc/ranger-kms//0 For Downgrade the versioned config directories will be present already on the machine. EU has been tested and it went through the upgrade process successfully. - Mugdha --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/56607/#review165378 --- On Feb. 14, 2017, 1:18 p.m., Mugdha Varadkar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/56607/ > --- > > (Updated Feb. 14, 2017, 1:18 p.m.) > > > Review request for Ambari, Alejandro Fernandez, Gautam Borad, Jayush Luniya, > and Velmurugan Periasamy. > > > Bugs: AMBARI-19915 > https://issues.apache.org/jira/browse/AMBARI-19915 > > > Repository: ambari > > > Description > --- > > Below SSL properties needs to be added in Ambari for Ranger KMS service > - ranger.service.https.attrib.keystore.file > - ranger.service.https.attrib.client.auth > - ranger.service.https.attrib.keystore.keyalias > - ranger.service.https.attrib.keystore.pass > - ranger.credential.provider.path > - ranger.service.https.attrib.keystore.credential.alias > > Along with addition of these properties, need to secure password property. > > > Diffs > - > > > ambari-common/src/main/python/resource_management/libraries/functions/constants.py > 8fd5c8d > ambari-server/src/main/resources/common-services/RANGER/0.6.0/metainfo.xml > 12fde7e > > ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/configuration/kms-env.xml > 7dea07f > > ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms.py > 742cb93 > > ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/params.py > 05e8881 > > ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json > 0fd1766 > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml > 54072ad > > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.6.xml > 31b20cf > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.6.xml > 88486e6 > ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml > 44a9b7c > > ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.6.xml > 4d70156 > ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.6.xml > 5b92e5f > ambari-server/src/main/resources/stacks/HDP/2.5/upgrades
Re: Review Request 56607: AMBARI-19915 Add Ranger KMS SSL properties in ambari stack
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/56607/ --- (Updated Feb. 14, 2017, 1:18 p.m.) Review request for Ambari, Alejandro Fernandez, Gautam Borad, Jayush Luniya, and Velmurugan Periasamy. Bugs: AMBARI-19915 https://issues.apache.org/jira/browse/AMBARI-19915 Repository: ambari Description --- Below SSL properties needs to be added in Ambari for Ranger KMS service - ranger.service.https.attrib.keystore.file - ranger.service.https.attrib.client.auth - ranger.service.https.attrib.keystore.keyalias - ranger.service.https.attrib.keystore.pass - ranger.credential.provider.path - ranger.service.https.attrib.keystore.credential.alias Along with addition of these properties, need to secure password property. Diffs (updated) - ambari-common/src/main/python/resource_management/libraries/functions/constants.py 8fd5c8d ambari-server/src/main/resources/common-services/RANGER/0.6.0/metainfo.xml 12fde7e ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/configuration/kms-env.xml 7dea07f ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms.py 742cb93 ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/params.py 05e8881 ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json 0fd1766 ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 54072ad ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.6.xml 31b20cf ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.6.xml 88486e6 ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml 44a9b7c ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.6.xml 4d70156 ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.6.xml 5b92e5f ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/config-upgrade.xml 23564ad ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml 24db720 ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml 818a6c0 ambari-server/src/main/resources/stacks/HDP/2.6/services/RANGER_KMS/configuration/ranger-kms-site.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.6/services/stack_advisor.py 969c3dd ambari-server/src/test/python/stacks/2.5/RANGER_KMS/test_kms_server.py 57f9f34 ambari-server/src/test/python/stacks/2.6/common/test_stack_advisor.py b7f8cbb Diff: https://reviews.apache.org/r/56607/diff/ Testing --- Test Case: Running tests for stack:2.5 service:RANGER_KMS test_configure_default (test_kms_server.TestRangerKMS) ... 2017-02-13 19:08:27,451 - Stack Feature Version Info: stack_version=2.5, version=2.5.0.0-777, current_cluster_version=2.5.0.0-777 -> 2.5.0.0-777 2017-02-13 19:08:27,467 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf ok test_configure_secured (test_kms_server.TestRangerKMS) ... 2017-02-13 19:08:27,502 - Stack Feature Version Info: stack_version=2.5, version=2.5.0.0-801, current_cluster_version=2.5.0.0-801 -> 2.5.0.0-801 2017-02-13 19:08:27,518 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf ok test_start_default (test_kms_server.TestRangerKMS) ... 2017-02-13 19:08:27,533 - Stack Feature Version Info: stack_version=2.5, version=2.5.0.0-777, current_cluster_version=2.5.0.0-777 -> 2.5.0.0-777 2017-02-13 19:08:27,550 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf 2017-02-13 19:08:27,555 - Rangeradmin: Skip ranger admin if it's down ! ok test_start_secured (test_kms_server.TestRangerKMS) ... 2017-02-13 19:08:27,568 - Stack Feature Version Info: stack_version=2.5, version=2.5.0.0-801, current_cluster_version=2.5.0.0-801 -> 2.5.0.0-801 2017-02-13 19:08:27,597 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf 2017-02-13 19:08:27,602 - RangeradminV2: Skip ranger admin if it's down ! 2017-02-13 19:08:27,602 - KMS repository c1_kms exist ok test_stop_default (test_kms_server.TestRangerKMS) ... 2017-02-13 19:08:27,618 - Stack Feature Version Info: stack_version=2.5, version=2.5.0.0-777, current_cluster_version=2.5.0.0-777 -> 2.5.0.0-777 2017-02-13 19:08:27,639 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf ok -- Ran 5 tests in 0.277s OK test_recommendRangerKMSConfigurations (test_stack_advisor.TestHDP26StackAdvisor) ... ok Thanks, Mugdha Varadkar
Re: Review Request 56607: AMBARI-19915 Add Ranger KMS SSL properties in ambari stack
> On Feb. 13, 2017, 7:18 p.m., Alejandro Fernandez wrote: > > ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms.py, > > line 148 > > <https://reviews.apache.org/r/56607/diff/1/?file=1632316#file1632316line148> > > > > Today, we have symlinks from /etc/$comp/conf -> > > /usr/hdp/current/$comp/conf so that configs still work during stack > > upgrades (RU/EU). This seems to be a flat directory. > > > > How is it going to be used? > > What will happen during stack upgrades or downgrades? Hi Alejandro, For Ranger KMS: the "conf" symlinks is created under the path /etc/ranger/kms/. This flat directory path is created by rpm package. I added this code if in case the path is not created, it may be created from Ambari, also this path is needed to create some additional files. - Mugdha --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/56607/#review165378 ------- On Feb. 13, 2017, 7:14 p.m., Mugdha Varadkar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/56607/ > --- > > (Updated Feb. 13, 2017, 7:14 p.m.) > > > Review request for Ambari, Alejandro Fernandez, Gautam Borad, Jayush Luniya, > and Velmurugan Periasamy. > > > Bugs: AMBARI-19915 > https://issues.apache.org/jira/browse/AMBARI-19915 > > > Repository: ambari > > > Description > --- > > Below SSL properties needs to be added in Ambari for Ranger KMS service > - ranger.service.https.attrib.keystore.file > - ranger.service.https.attrib.client.auth > - ranger.service.https.attrib.keystore.keyalias > - ranger.service.https.attrib.keystore.pass > - ranger.credential.provider.path > - ranger.service.https.attrib.keystore.credential.alias > > Along with addition of these properties, need to secure password property. > > > Diffs > - > > > ambari-common/src/main/python/resource_management/libraries/functions/constants.py > 8fd5c8d > > ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/configuration/kms-env.xml > 7dea07f > > ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms.py > 742cb93 > > ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/params.py > 05e8881 > > ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json > 0fd1766 > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml > 54072ad > > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.6.xml > c64d6ed > ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.6.xml > 58db4a9 > ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml > 44a9b7c > > ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.6.xml > 92c5902 > ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.6.xml > 30b5d13 > ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/config-upgrade.xml > 23564ad > > ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml > 75515a5 > ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml > b83525a > > ambari-server/src/main/resources/stacks/HDP/2.6/services/RANGER_KMS/configuration/ranger-kms-site.xml > PRE-CREATION > ambari-server/src/main/resources/stacks/HDP/2.6/services/stack_advisor.py > 969c3dd > ambari-server/src/test/python/stacks/2.5/RANGER_KMS/test_kms_server.py > 57f9f34 > ambari-server/src/test/python/stacks/2.6/common/test_stack_advisor.py > b7f8cbb > > Diff: https://reviews.apache.org/r/56607/diff/ > > > Testing > --- > > Test Case: > > Running tests for stack:2.5 service:RANGER_KMS > test_configure_default (test_kms_server.TestRangerKMS) ... 2017-02-13 > 19:08:27,451 - Stack Feature Version Info: stack_version=2.5, > version=2.5.0.0-777, current_cluster_version=2.5.0.0-777 -> 2.5.0.0-777 > 2017-02-13 19:08:27,467 - Using hadoop conf dir: > /usr/hdp/current/hadoop-client/conf > ok > test_configure_secured (test_kms_server.TestRangerKMS) ... 2017-02-13 > 19:08:27,502 - Stack Feature Version Info: stack_version=2.5, > version=2.5.0.0-801, current_cluster_version=2.5.0.0-801 -> 2.5.0.0-801 > 2017-02-13 19:08:27,51
Re: Review Request 56622: AMBARI-19993 : Update atlas log-4j.xml configurations in ambari
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/56622/#review165454 --- Ship it! Ship It! - Mugdha Varadkar On Feb. 13, 2017, 10:37 p.m., Vishal Suvagia wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/56622/ > --- > > (Updated Feb. 13, 2017, 10:37 p.m.) > > > Review request for Ambari, Alejandro Fernandez, Gautam Borad, Madhan > Neethiraj, Mugdha Varadkar, Oliver Szabo, and Sumit Mohanty. > > > Bugs: AMBARI-19993 > https://issues.apache.org/jira/browse/AMBARI-19993 > > > Repository: ambari > > > Description > --- > > Need to update atlas-log4j configurations in ambari to reflect latest in > atlas. > > > Diffs > - > > > ambari-server/src/main/resources/stacks/HDP/2.6/services/ATLAS/configuration/atlas-log4j.xml > 95bc0ea > > Diff: https://reviews.apache.org/r/56622/diff/ > > > Testing > --- > > Verified fresh install on Cent-os 6. > > > Thanks, > > Vishal Suvagia > >
Review Request 56607: AMBARI-19915 Add Ranger KMS SSL properties in ambari stack
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/56607/ --- Review request for Ambari, Alejandro Fernandez, Gautam Borad, Jayush Luniya, and Velmurugan Periasamy. Bugs: AMBARI-19915 https://issues.apache.org/jira/browse/AMBARI-19915 Repository: ambari Description --- Below SSL properties needs to be added in Ambari for Ranger KMS service - ranger.service.https.attrib.keystore.file - ranger.service.https.attrib.client.auth - ranger.service.https.attrib.keystore.keyalias - ranger.service.https.attrib.keystore.pass - ranger.credential.provider.path - ranger.service.https.attrib.keystore.credential.alias Along with addition of these properties, need to secure password property. Diffs - ambari-common/src/main/python/resource_management/libraries/functions/constants.py 8fd5c8d ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/configuration/kms-env.xml 7dea07f ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms.py 742cb93 ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/params.py 05e8881 ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json 0fd1766 ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/config-upgrade.xml 54072ad ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.6.xml c64d6ed ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.6.xml 58db4a9 ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml 44a9b7c ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/nonrolling-upgrade-2.6.xml 92c5902 ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/upgrade-2.6.xml 30b5d13 ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/config-upgrade.xml 23564ad ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml 75515a5 ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml b83525a ambari-server/src/main/resources/stacks/HDP/2.6/services/RANGER_KMS/configuration/ranger-kms-site.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.6/services/stack_advisor.py 969c3dd ambari-server/src/test/python/stacks/2.5/RANGER_KMS/test_kms_server.py 57f9f34 ambari-server/src/test/python/stacks/2.6/common/test_stack_advisor.py b7f8cbb Diff: https://reviews.apache.org/r/56607/diff/ Testing --- Test Case: Running tests for stack:2.5 service:RANGER_KMS test_configure_default (test_kms_server.TestRangerKMS) ... 2017-02-13 19:08:27,451 - Stack Feature Version Info: stack_version=2.5, version=2.5.0.0-777, current_cluster_version=2.5.0.0-777 -> 2.5.0.0-777 2017-02-13 19:08:27,467 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf ok test_configure_secured (test_kms_server.TestRangerKMS) ... 2017-02-13 19:08:27,502 - Stack Feature Version Info: stack_version=2.5, version=2.5.0.0-801, current_cluster_version=2.5.0.0-801 -> 2.5.0.0-801 2017-02-13 19:08:27,518 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf ok test_start_default (test_kms_server.TestRangerKMS) ... 2017-02-13 19:08:27,533 - Stack Feature Version Info: stack_version=2.5, version=2.5.0.0-777, current_cluster_version=2.5.0.0-777 -> 2.5.0.0-777 2017-02-13 19:08:27,550 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf 2017-02-13 19:08:27,555 - Rangeradmin: Skip ranger admin if it's down ! ok test_start_secured (test_kms_server.TestRangerKMS) ... 2017-02-13 19:08:27,568 - Stack Feature Version Info: stack_version=2.5, version=2.5.0.0-801, current_cluster_version=2.5.0.0-801 -> 2.5.0.0-801 2017-02-13 19:08:27,597 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf 2017-02-13 19:08:27,602 - RangeradminV2: Skip ranger admin if it's down ! 2017-02-13 19:08:27,602 - KMS repository c1_kms exist ok test_stop_default (test_kms_server.TestRangerKMS) ... 2017-02-13 19:08:27,618 - Stack Feature Version Info: stack_version=2.5, version=2.5.0.0-777, current_cluster_version=2.5.0.0-777 -> 2.5.0.0-777 2017-02-13 19:08:27,639 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf ok -- Ran 5 tests in 0.277s OK test_recommendRangerKMSConfigurations (test_stack_advisor.TestHDP26StackAdvisor) ... ok Thanks, Mugdha Varadkar
Re: Review Request 56276: AMBARI-19845 Secure Ranger passwords in Ambari Stacks
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/56276/ --- (Updated Feb. 9, 2017, 5:38 a.m.) Review request for Ambari, Alejandro Fernandez, Gautam Borad, Jayush Luniya, Sumit Mohanty, and Velmurugan Periasamy. Bugs: AMBARI-19845 https://issues.apache.org/jira/browse/AMBARI-19845 Repository: ambari Description --- Make sure plain-text password is not persisted in XML and always stored in jceks Diffs - ambari-common/src/main/python/resource_management/libraries/functions/constants.py 8fd5c8d ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin_xml.py a12116d ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py 0b4532b ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py 3cb225f ambari-server/src/main/resources/common-services/RANGER/0.5.0/configuration/ranger-admin-site.xml 0067520 ambari-server/src/main/resources/common-services/RANGER/0.7.0/configuration/ranger-admin-site.xml PRE-CREATION ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms.py 742cb93 ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/params.py 05e8881 ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json 0fd1766 ambari-server/src/test/python/stacks/2.5/RANGER/test_ranger_admin.py b01e7da ambari-server/src/test/python/stacks/2.5/RANGER/test_ranger_usersync.py 22e84fc ambari-server/src/test/python/stacks/2.5/RANGER_KMS/test_kms_server.py 57f9f34 ambari-server/src/test/python/stacks/2.6/RANGER/test_ranger_admin.py 8dda363 ambari-server/src/test/python/stacks/2.6/RANGER/test_ranger_tagsync.py bf5128e ambari-server/src/test/python/stacks/2.6/configs/ranger-admin-default.json 2c4815b Diff: https://reviews.apache.org/r/56276/diff/ Testing --- Unit test result: Ran 270 tests in 11.685s OK -- Total run:1172 Total errors:0 Total failures:0 OK Thanks, Mugdha Varadkar
Re: Review Request 56179: Add infra-solr-plugin for authorization (with Kerberos)
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56179/#review164497
---
ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/properties/infra-solr-security.json.j2
(line 30)
<https://reviews.apache.org/r/56179/#comment236265>
The infra_solr_ranger_audit_service_users variable will give
{default_ranger_audit_users} instead of calculated list of users in variable
default_ranger_audit_users.
ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/configuration/infra-solr-security-json.xml
(line 27)
<https://reviews.apache.org/r/56179/#comment236266>
This should be {{default_ranger_audit_users}}
- Mugdha Varadkar
On Feb. 6, 2017, 8:32 p.m., Oliver Szabo wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56179/
> ---
>
> (Updated Feb. 6, 2017, 8:32 p.m.)
>
>
> Review request for Ambari, Miklos Gergely, Robert Nettleton, and Sebastian
> Toader.
>
>
> Bugs: AMBARI-19822
> https://issues.apache.org/jira/browse/AMBARI-19822
>
>
> Repository: ambari
>
>
> Description
> ---
>
> Problem:
> If an ambari cluster is secured and kerberos authentication is used for Solr,
> we need (default) authorizations as well to make sure only the specific
> service users (ranger, atlas, logsearch) can access their collections (and
> solr user as well)
>
> Solution:
> Although RuleBasedAuthorizationPlugin seems to be a good solution here, to
> map default users to default permissions, unfortunately, permissions and
> roles using principal name for mapping (not username) from the authentication
> tokens. Also Solr name rules applied on the username and not on the
> principal, therefore we need the fully qualified hostname as well in the
> role-permission mapping. In order to avoid that issue, I added an own plugin
> (org.apache.ambari.infra.security.InfraRuleBasedAuthorizationPlugin), to map
> users with @ format.
>
> to problem is in here in RuleBasedAuthorizationPlugin.java:
> https://github.com/apache/lucene-solr/blob/releases/lucene-solr/5.5.2/solr/core/src/java/org/apache/solr/security/RuleBasedAuthorizationPlugin.java#L153
>
> notice that InfraRuleBasedAuthorizationPlugin is only a copy of that file
> (InfraUserRolesLookupStrategy class which I added and included in the new
> plugin class)
>
> In case of we need strict host validations i added 2 new json properties for
> that:
> 1. { "user-host" : {"" : []} }
> 2. {"user-host-regex" : {"" : "hostname-regex"} }
>
> {{user-host-regex}} has higher precedence then {{user-host}}
>
>
> Diffs
> -
>
> ambari-logsearch/ambari-infra-solr-plugin/pom.xml PRE-CREATION
>
> ambari-logsearch/ambari-infra-solr-plugin/src/main/java/org.apache.ambari.infra.security/InfraKerberosHostValidator.java
> PRE-CREATION
>
> ambari-logsearch/ambari-infra-solr-plugin/src/main/java/org.apache.ambari.infra.security/InfraRuleBasedAuthorizationPlugin.java
> PRE-CREATION
>
> ambari-logsearch/ambari-infra-solr-plugin/src/main/java/org.apache.ambari.infra.security/InfraUserRolesLookupStrategy.java
> PRE-CREATION
>
> ambari-logsearch/ambari-infra-solr-plugin/src/test/java/org/apache/ambari/infra/security/InfraKerberosHostValidatorTest.java
> PRE-CREATION
>
> ambari-logsearch/ambari-infra-solr-plugin/src/test/java/org/apache/ambari/infra/security/InfraRuleBasedAuthorizationPluginTest.java
> PRE-CREATION
>
> ambari-logsearch/ambari-infra-solr-plugin/src/test/java/org/apache/ambari/infra/security/InfraUserRolesLookupStrategyTest.java
> PRE-CREATION
> ambari-logsearch/ambari-logsearch-assembly/pom.xml c486050
> ambari-logsearch/pom.xml 7aeb4a7
>
> ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/configuration/infra-solr-security-json.xml
> ed623df
>
> ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/package/scripts/params.py
> 526baea
>
> ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/properties/infra-solr-security.json.j2
> d8aea24
> ambari-server/src/test/python/stacks/2.4/configs/default.json 7a940d3
>
> Diff: https://reviews.apache.org/r/56179/diff/
>
>
> Testing
> ---
>
> unit tests done, behavior validated with unit tests. FT: validated with
> logsearch and atlas as well.
>
>
> Thanks,
>
> Oliver Szabo
>
>
Re: Review Request 56278: AMBARI-19864 : Zookeeper namespace for Hive service should be taken from hive-interactive-site, when only Hive-server Interactive is installed.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/56278/#review164339 --- Ship it! Ship It! - Mugdha Varadkar On Feb. 6, 2017, 6:31 a.m., Vishal Suvagia wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/56278/ > --- > > (Updated Feb. 6, 2017, 6:31 a.m.) > > > Review request for Ambari, Alejandro Fernandez, Gautam Borad, Mugdha > Varadkar, Oliver Szabo, Sumit Mohanty, and Velmurugan Periasamy. > > > Bugs: AMBARI-19864 > https://issues.apache.org/jira/browse/AMBARI-19864 > > > Repository: ambari > > > Description > --- > > If only Hive-Server Interactive is installed, then zoo-keeper name space for > hive service configured for Ranger should be taken from hive-interactive-site > and not from hive-site. > > > Diffs > - > > > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py > 211fe0a > > Diff: https://reviews.apache.org/r/56278/diff/ > > > Testing > --- > > Verified installation on CentOS-6. > > > Thanks, > > Vishal Suvagia > >
Review Request 56276: AMBARI-19845 Secure Ranger passwords in Ambari Stacks
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/56276/ --- Review request for Ambari, Alejandro Fernandez, Gautam Borad, Jayush Luniya, and Velmurugan Periasamy. Bugs: AMBARI-19845 https://issues.apache.org/jira/browse/AMBARI-19845 Repository: ambari Description --- Make sure plain-text password is not persisted in XML and always stored in jceks Diffs - ambari-common/src/main/python/resource_management/libraries/functions/constants.py 8fd5c8d ambari-common/src/main/python/resource_management/libraries/functions/setup_ranger_plugin_xml.py a12116d ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py 0b4532b ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py 3cb225f ambari-server/src/main/resources/common-services/RANGER/0.5.0/configuration/ranger-admin-site.xml 0067520 ambari-server/src/main/resources/common-services/RANGER/0.7.0/configuration/ranger-admin-site.xml PRE-CREATION ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms.py 742cb93 ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/params.py 05e8881 ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_features.json 0fd1766 ambari-server/src/test/python/stacks/2.5/RANGER/test_ranger_admin.py b01e7da ambari-server/src/test/python/stacks/2.5/RANGER/test_ranger_usersync.py 22e84fc ambari-server/src/test/python/stacks/2.5/RANGER_KMS/test_kms_server.py 57f9f34 ambari-server/src/test/python/stacks/2.6/RANGER/test_ranger_admin.py 8dda363 ambari-server/src/test/python/stacks/2.6/RANGER/test_ranger_tagsync.py bf5128e ambari-server/src/test/python/stacks/2.6/configs/ranger-admin-default.json 2c4815b Diff: https://reviews.apache.org/r/56276/diff/ Testing --- Unit test result: Ran 270 tests in 11.685s OK -- Total run:1172 Total errors:0 Total failures:0 OK Thanks, Mugdha Varadkar
Review Request 56124: AMBARI-19795 Ambari upgrade to not add ranger plugin configs under ranger plugin supported services
-hbase-plugin-properties.xml 3450970 ambari-server/src/main/resources/stacks/HDP/2.5/services/HDFS/configuration/ranger-hdfs-plugin-properties.xml 953e42e ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/ranger-hive-plugin-properties.xml 3450970 ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/ranger-hive-security.xml 640234b ambari-server/src/main/resources/stacks/HDP/2.5/services/KAFKA/configuration/ranger-kafka-plugin-properties.xml 3450970 ambari-server/src/main/resources/stacks/HDP/2.5/services/KNOX/configuration/ranger-knox-plugin-properties.xml 3450970 ambari-server/src/main/resources/stacks/HDP/2.5/services/STORM/configuration/ranger-storm-policymgr-ssl.xml 21658e7 ambari-server/src/main/resources/stacks/HDP/2.5/services/STORM/configuration/ranger-storm-security.xml 8a3dd2e ambari-server/src/main/resources/stacks/HDP/2.5/services/YARN/configuration/ranger-yarn-plugin-properties.xml 3450970 ambari-server/src/main/resources/stacks/HDP/2.6/services/ATLAS/configuration/ranger-atlas-plugin-properties.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.6/services/HBASE/configuration/ranger-hbase-plugin-properties.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.6/services/HDFS/configuration/ranger-hdfs-plugin-properties.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.6/services/HIVE/configuration/ranger-hive-plugin-properties.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.6/services/KAFKA/configuration/ranger-kafka-plugin-properties.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.6/services/KNOX/configuration/ranger-knox-plugin-properties.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.6/services/STORM/configuration/ranger-storm-plugin-properties.xml PRE-CREATION ambari-server/src/main/resources/stacks/HDP/2.6/services/YARN/configuration/ranger-yarn-plugin-properties.xml PRE-CREATION Diff: https://reviews.apache.org/r/56124/diff/ Testing --- Verified Ranger Installation. Thanks, Mugdha Varadkar
