Review Request 53129: AMBARI-18671 Ranger KMS should add proxy users for yarn and livy

[Mugdha Varadkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53129/
---

Review request for Ambari, Alejandro Fernandez, Gautam Borad, Sumit Mohanty, 
and Velmurugan Periasamy.


Bugs: AMBARI-18671
https://issues.apache.org/jira/browse/AMBARI-18671


Repository: ambari


Description
---

Add below properties under kms-site.xml having value as '*', if YARN|SPARK 
services are installed

- hadoop.kms.proxyuser.[livy|yarn].groups
- hadoop.kms.proxyuser.[livy|yarn].hosts
- hadoop.kms.proxyuser.[livy|yarn].users

Also with above changes need to revert changes done for AMBARI-18096


Diffs
-

  ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
24b9f6e 
  ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
986ca69 
  ambari-server/src/test/python/stacks/2.0.6/common/test_stack_advisor.py 
cd69dc3 
  ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 72e59aa 

Diff: https://reviews.apache.org/r/53129/diff/


Testing
---

Tested Ranger Kms installation with Yarn|Spark on centos 6


Thanks,

Mugdha Varadkar

Re: Review Request 53129: AMBARI-18671 Ranger KMS should add proxy users for yarn and livy

[Mugdha Varadkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53129/
---

(Updated Oct. 25, 2016, 2:20 p.m.)


Review request for Ambari, Alejandro Fernandez, Gautam Borad, Sumit Mohanty, 
and Velmurugan Periasamy.


Bugs: AMBARI-18671
https://issues.apache.org/jira/browse/AMBARI-18671


Repository: ambari


Description
---

Add below properties under kms-site.xml having value as '*', if YARN|SPARK 
services are installed

- hadoop.kms.proxyuser.[livy|yarn].groups
- hadoop.kms.proxyuser.[livy|yarn].hosts
- hadoop.kms.proxyuser.[livy|yarn].users

Also with above changes need to revert changes done for AMBARI-18096


Diffs (updated)
-

  ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
24b9f6e 
  ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
986ca69 
  ambari-server/src/test/python/stacks/2.0.6/common/test_stack_advisor.py 
cd69dc3 
  ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 72e59aa 

Diff: https://reviews.apache.org/r/53129/diff/


Testing
---

Tested Ranger Kms installation with Yarn|Spark on centos 6


Thanks,

Mugdha Varadkar

Re: Review Request 53129: AMBARI-18671 Ranger KMS should add proxy users for yarn and livy

[Alejandro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53129/#review153806
---




ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py (line 
1771)


Technically item is an "index"


- Alejandro Fernandez


On Oct. 25, 2016, 2:20 p.m., Mugdha Varadkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53129/
> ---
> 
> (Updated Oct. 25, 2016, 2:20 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Gautam Borad, Sumit Mohanty, 
> and Velmurugan Periasamy.
> 
> 
> Bugs: AMBARI-18671
> https://issues.apache.org/jira/browse/AMBARI-18671
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Add below properties under kms-site.xml having value as '*', if YARN|SPARK 
> services are installed
> 
> - hadoop.kms.proxyuser.[livy|yarn].groups
> - hadoop.kms.proxyuser.[livy|yarn].hosts
> - hadoop.kms.proxyuser.[livy|yarn].users
> 
> Also with above changes need to revert changes done for AMBARI-18096
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> 24b9f6e 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
> 986ca69 
>   ambari-server/src/test/python/stacks/2.0.6/common/test_stack_advisor.py 
> cd69dc3 
>   ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 
> 72e59aa 
> 
> Diff: https://reviews.apache.org/r/53129/diff/
> 
> 
> Testing
> ---
> 
> Tested Ranger Kms installation with Yarn|Spark on centos 6
> 
> 
> Thanks,
> 
> Mugdha Varadkar
> 
>

Re: Review Request 53129: AMBARI-18671 Ranger KMS should add proxy users for yarn and livy

[Mugdha Varadkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53129/
---

(Updated Oct. 26, 2016, 10:53 a.m.)


Review request for Ambari, Alejandro Fernandez, Gautam Borad, Sumit Mohanty, 
and Velmurugan Periasamy.


Bugs: AMBARI-18671
https://issues.apache.org/jira/browse/AMBARI-18671


Repository: ambari


Description
---

Add below properties under kms-site.xml having value as '*', if YARN|SPARK 
services are installed

- hadoop.kms.proxyuser.[livy|yarn].groups
- hadoop.kms.proxyuser.[livy|yarn].hosts
- hadoop.kms.proxyuser.[livy|yarn].users

Also with above changes need to revert changes done for AMBARI-18096


Diffs (updated)
-

  ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
24b9f6e 
  ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
986ca69 
  ambari-server/src/test/python/stacks/2.0.6/common/test_stack_advisor.py 
cd69dc3 
  ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 72e59aa 

Diff: https://reviews.apache.org/r/53129/diff/


Testing
---

Tested Ranger Kms installation with Yarn|Spark on centos 6


Thanks,

Mugdha Varadkar

Re: Review Request 53129: AMBARI-18671 Ranger KMS should add proxy users for yarn and livy

[Mugdha Varadkar]

> On Oct. 25, 2016, 4:32 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py, 
> > line 1771
> > 
> >
> > Technically item is an "index"

yes item is meant to be an index for the requiredServices. Updated in latest 
patch


- Mugdha


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53129/#review153806
---


On Oct. 26, 2016, 10:53 a.m., Mugdha Varadkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53129/
> ---
> 
> (Updated Oct. 26, 2016, 10:53 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Gautam Borad, Sumit Mohanty, 
> and Velmurugan Periasamy.
> 
> 
> Bugs: AMBARI-18671
> https://issues.apache.org/jira/browse/AMBARI-18671
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Add below properties under kms-site.xml having value as '*', if YARN|SPARK 
> services are installed
> 
> - hadoop.kms.proxyuser.[livy|yarn].groups
> - hadoop.kms.proxyuser.[livy|yarn].hosts
> - hadoop.kms.proxyuser.[livy|yarn].users
> 
> Also with above changes need to revert changes done for AMBARI-18096
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> 24b9f6e 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
> 986ca69 
>   ambari-server/src/test/python/stacks/2.0.6/common/test_stack_advisor.py 
> cd69dc3 
>   ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 
> 72e59aa 
> 
> Diff: https://reviews.apache.org/r/53129/diff/
> 
> 
> Testing
> ---
> 
> Tested Ranger Kms installation with Yarn|Spark on centos 6
> 
> 
> Thanks,
> 
> Mugdha Varadkar
> 
>

Re: Review Request 53129: AMBARI-18671 Ranger KMS should add proxy users for yarn and livy

[Alejandro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53129/#review153916
---



How will this impact existing clusters? Do they need to update these configs 
once they upgrade Ambari? What about when they upgrade the stack?

- Alejandro Fernandez


On Oct. 26, 2016, 10:53 a.m., Mugdha Varadkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53129/
> ---
> 
> (Updated Oct. 26, 2016, 10:53 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Gautam Borad, Sumit Mohanty, 
> and Velmurugan Periasamy.
> 
> 
> Bugs: AMBARI-18671
> https://issues.apache.org/jira/browse/AMBARI-18671
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Add below properties under kms-site.xml having value as '*', if YARN|SPARK 
> services are installed
> 
> - hadoop.kms.proxyuser.[livy|yarn].groups
> - hadoop.kms.proxyuser.[livy|yarn].hosts
> - hadoop.kms.proxyuser.[livy|yarn].users
> 
> Also with above changes need to revert changes done for AMBARI-18096
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> 24b9f6e 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
> 986ca69 
>   ambari-server/src/test/python/stacks/2.0.6/common/test_stack_advisor.py 
> cd69dc3 
>   ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 
> 72e59aa 
> 
> Diff: https://reviews.apache.org/r/53129/diff/
> 
> 
> Testing
> ---
> 
> Tested Ranger Kms installation with Yarn|Spark on centos 6
> 
> 
> Thanks,
> 
> Mugdha Varadkar
> 
>

Re: Review Request 53129: AMBARI-18671 Ranger KMS should add proxy users for yarn and livy

[Mugdha Varadkar]

> On Oct. 26, 2016, 6:11 p.m., Alejandro Fernandez wrote:
> > How will this impact existing clusters? Do they need to update these 
> > configs once they upgrade Ambari? What about when they upgrade the stack?

For existing clusters this properties will be available as custom properties. 
After ambari/stack upgrade the custom properties will be retained.

Currently the properties needs to be added manually, this fix will 
automatically add it using Stack Advisor.


- Mugdha


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53129/#review153916
---


On Oct. 26, 2016, 10:53 a.m., Mugdha Varadkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53129/
> ---
> 
> (Updated Oct. 26, 2016, 10:53 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Gautam Borad, Sumit Mohanty, 
> and Velmurugan Periasamy.
> 
> 
> Bugs: AMBARI-18671
> https://issues.apache.org/jira/browse/AMBARI-18671
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Add below properties under kms-site.xml having value as '*', if YARN|SPARK 
> services are installed
> 
> - hadoop.kms.proxyuser.[livy|yarn].groups
> - hadoop.kms.proxyuser.[livy|yarn].hosts
> - hadoop.kms.proxyuser.[livy|yarn].users
> 
> Also with above changes need to revert changes done for AMBARI-18096
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> 24b9f6e 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
> 986ca69 
>   ambari-server/src/test/python/stacks/2.0.6/common/test_stack_advisor.py 
> cd69dc3 
>   ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 
> 72e59aa 
> 
> Diff: https://reviews.apache.org/r/53129/diff/
> 
> 
> Testing
> ---
> 
> Tested Ranger Kms installation with Yarn|Spark on centos 6
> 
> 
> Thanks,
> 
> Mugdha Varadkar
> 
>

Re: Review Request 53129: AMBARI-18671 Ranger KMS should add proxy users for yarn and livy

[Gautam Borad]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53129/#review155122
---


Ship it!




Ship It!

- Gautam Borad


On Oct. 26, 2016, 10:53 a.m., Mugdha Varadkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53129/
> ---
> 
> (Updated Oct. 26, 2016, 10:53 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Gautam Borad, Sumit Mohanty, 
> and Velmurugan Periasamy.
> 
> 
> Bugs: AMBARI-18671
> https://issues.apache.org/jira/browse/AMBARI-18671
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Add below properties under kms-site.xml having value as '*', if YARN|SPARK 
> services are installed
> 
> - hadoop.kms.proxyuser.[livy|yarn].groups
> - hadoop.kms.proxyuser.[livy|yarn].hosts
> - hadoop.kms.proxyuser.[livy|yarn].users
> 
> Also with above changes need to revert changes done for AMBARI-18096
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> 24b9f6e 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
> 986ca69 
>   ambari-server/src/test/python/stacks/2.0.6/common/test_stack_advisor.py 
> cd69dc3 
>   ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 
> 72e59aa 
> 
> Diff: https://reviews.apache.org/r/53129/diff/
> 
> 
> Testing
> ---
> 
> Tested Ranger Kms installation with Yarn|Spark on centos 6
> 
> 
> Thanks,
> 
> Mugdha Varadkar
> 
>

Re: Review Request 53129: AMBARI-18671 Ranger KMS should add proxy users for yarn and livy

[Alejandro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53129/#review155324
---



What should the behavior be for customers with Ranger and Kerberos that are 
upgrading their stack from an existing version with the old logic (could be HDP 
2.3, 2.4, 2.5) to a higher HDP version?

- Alejandro Fernandez


On Oct. 26, 2016, 10:53 a.m., Mugdha Varadkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53129/
> ---
> 
> (Updated Oct. 26, 2016, 10:53 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Gautam Borad, Sumit Mohanty, 
> and Velmurugan Periasamy.
> 
> 
> Bugs: AMBARI-18671
> https://issues.apache.org/jira/browse/AMBARI-18671
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Add below properties under kms-site.xml having value as '*', if YARN|SPARK 
> services are installed
> 
> - hadoop.kms.proxyuser.[livy|yarn].groups
> - hadoop.kms.proxyuser.[livy|yarn].hosts
> - hadoop.kms.proxyuser.[livy|yarn].users
> 
> Also with above changes need to revert changes done for AMBARI-18096
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> 24b9f6e 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
> 986ca69 
>   ambari-server/src/test/python/stacks/2.0.6/common/test_stack_advisor.py 
> cd69dc3 
>   ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 
> 72e59aa 
> 
> Diff: https://reviews.apache.org/r/53129/diff/
> 
> 
> Testing
> ---
> 
> Tested Ranger Kms installation with Yarn|Spark on centos 6
> 
> 
> Thanks,
> 
> Mugdha Varadkar
> 
>

Re: Review Request 53129: AMBARI-18671 Ranger KMS should add proxy users for yarn and livy

[Mugdha Varadkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53129/
---

(Updated Nov. 9, 2016, 1:25 p.m.)


Review request for Ambari, Alejandro Fernandez, Gautam Borad, Sumit Mohanty, 
and Velmurugan Periasamy.


Changes
---

Shifted the logic of stack-advisor.py from 2.5 to 2.3. As AMBARI-18096 is 
reverted and the proxy-user needs to be added in stack 2.3


Bugs: AMBARI-18671
https://issues.apache.org/jira/browse/AMBARI-18671


Repository: ambari


Description
---

Add below properties under kms-site.xml having value as '*', if YARN|SPARK 
services are installed

- hadoop.kms.proxyuser.[livy|yarn].groups
- hadoop.kms.proxyuser.[livy|yarn].hosts
- hadoop.kms.proxyuser.[livy|yarn].users

Also with above changes need to revert changes done for AMBARI-18096


Diffs (updated)
-

  ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
24b9f6e 
  ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
ec92f3a 
  ambari-server/src/test/python/stacks/2.0.6/common/test_stack_advisor.py 
f759b6c 
  ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py ad6842f 
  ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 6fe357a 

Diff: https://reviews.apache.org/r/53129/diff/


Testing
---

Tested Ranger Kms installation with Yarn|Spark on centos 6


Thanks,

Mugdha Varadkar

Re: Review Request 53129: AMBARI-18671 Ranger KMS should add proxy users for yarn and livy

[Mugdha Varadkar]

> On Nov. 8, 2016, 6:48 p.m., Alejandro Fernandez wrote:
> > What should the behavior be for customers with Ranger and Kerberos that are 
> > upgrading their stack from an existing version with the old logic (could be 
> > HDP 2.3, 2.4, 2.5) to a higher HDP version?

Thanks for pointing it out, will need Ranger proxy related properties if 
upgrading from 2.3/2.4 to 2.5 in config-upgrade.xml. Have created a Jira 
(https://issues.apache.org/jira/browse/AMBARI-18834) to track the fix.


- Mugdha


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53129/#review155324
---


On Nov. 9, 2016, 1:25 p.m., Mugdha Varadkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53129/
> ---
> 
> (Updated Nov. 9, 2016, 1:25 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Gautam Borad, Sumit Mohanty, 
> and Velmurugan Periasamy.
> 
> 
> Bugs: AMBARI-18671
> https://issues.apache.org/jira/browse/AMBARI-18671
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Add below properties under kms-site.xml having value as '*', if YARN|SPARK 
> services are installed
> 
> - hadoop.kms.proxyuser.[livy|yarn].groups
> - hadoop.kms.proxyuser.[livy|yarn].hosts
> - hadoop.kms.proxyuser.[livy|yarn].users
> 
> Also with above changes need to revert changes done for AMBARI-18096
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> 24b9f6e 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
> ec92f3a 
>   ambari-server/src/test/python/stacks/2.0.6/common/test_stack_advisor.py 
> f759b6c 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> ad6842f 
>   ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 
> 6fe357a 
> 
> Diff: https://reviews.apache.org/r/53129/diff/
> 
> 
> Testing
> ---
> 
> Tested Ranger Kms installation with Yarn|Spark on centos 6
> 
> 
> Thanks,
> 
> Mugdha Varadkar
> 
>

Re: Review Request 53129: AMBARI-18671 Ranger KMS should add proxy users for yarn and livy

[Alejandro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53129/#review155501
---


Ship it!




Ship It!

- Alejandro Fernandez


On Nov. 9, 2016, 1:25 p.m., Mugdha Varadkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53129/
> ---
> 
> (Updated Nov. 9, 2016, 1:25 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Gautam Borad, Sumit Mohanty, 
> and Velmurugan Periasamy.
> 
> 
> Bugs: AMBARI-18671
> https://issues.apache.org/jira/browse/AMBARI-18671
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Add below properties under kms-site.xml having value as '*', if YARN|SPARK 
> services are installed
> 
> - hadoop.kms.proxyuser.[livy|yarn].groups
> - hadoop.kms.proxyuser.[livy|yarn].hosts
> - hadoop.kms.proxyuser.[livy|yarn].users
> 
> Also with above changes need to revert changes done for AMBARI-18096
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> 24b9f6e 
>   ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py 
> ec92f3a 
>   ambari-server/src/test/python/stacks/2.0.6/common/test_stack_advisor.py 
> f759b6c 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> ad6842f 
>   ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py 
> 6fe357a 
> 
> Diff: https://reviews.apache.org/r/53129/diff/
> 
> 
> Testing
> ---
> 
> Tested Ranger Kms installation with Yarn|Spark on centos 6
> 
> 
> Thanks,
> 
> Mugdha Varadkar
> 
>