Re: Review Request 61251: Dynamically determine what keytab files have been distributed

2017-08-03 Thread Robert Levas 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61251/#review182085
---


Ship it!




Ship It!

- Robert Levas


On Aug. 3, 2017, 4:14 a.m., Attila Magyar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61251/
> ---
> 
> (Updated Aug. 3, 2017, 4:14 a.m.)
> 
> 
> Review request for Ambari, Balázs Bence Sári, Robert Levas, and Sebastian 
> Toader.
> 
> 
> Bugs: AMBARI-21613
> https://issues.apache.org/jira/browse/AMBARI-21613
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Dynamically determine what keytab files have been distributed to hosts. A 
> custom command should be available via the KERBEROS_CLIENT to query for the 
> keytab files installed on the relevant host. The communication between the 
> Ambari server and the agents should generate data needed to determine what 
> keytab files exist.
> 
> 
> Diffs
> -
> 
>   ambari-common/src/main/python/resource_management/core/resources/klist.py 
> PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatHandler.java
>  1bc4c36 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartbeatProcessor.java
>  a08abab 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
>  e5b7afd 
>   
> ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/metainfo.xml
>  6a2dd09 
>   
> ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/package/scripts/kerberos_client.py
>  39fdcf5 
>   
> ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/package/scripts/kerberos_common.py
>  fcd57af 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/metainfo.xml
>  0e42bda 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/package/scripts/kerberos_client.py
>  b2cdaa6 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/package/scripts/kerberos_common.py
>  abf58ee 
>   
> ambari-server/src/test/java/org/apache/ambari/server/agent/TestHeartbeatHandler.java
>  baa9bae 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java
>  4508527 
>   ambari-server/src/test/python/stacks/2.2/KERBEROS/test_kerberos_client.py 
> f638845 
> 
> 
> Diff: https://reviews.apache.org/r/61251/diff/4/
> 
> 
> Testing
> ---
> 
> 1.
> - created a cluster with kerberos enabled
> - deleted a keytab file from a host
> - regenerated missing keytabs
> - checked that the deleted keytab file was recreated
> 
> 2.
> - regenerated missing keytabs when no keytab files were missing
> - checked that no keytab file was regenerated
> 
> 3.
> - created a 15 nodes cluster on openstack with services: HDFS, YARN, 
> ZOOKEEPER, SPARK1, SPARK2, HIVE
> - measured the time taken to regenerate all keytabs and only missing keytabs:
> 
>   regenerate missing keytabs only: 12.84 seconds (?=2.33)
>   regenerate all keytabs: 39.1 seconds (?=5.37)
> 
> existing tests:
> 
> server:
> Ran 246 tests in 6.999s
> OK
> --
> Total run:1146
> Total errors:0
> Total failures:0
> 
> Results :
> Tests run: 4816, Failures: 0, Errors: 0, Skipped: 35
> 
> agent:
> Ran 470 tests in 98.589s
> OK
> 
> 
> Thanks,
> 
> Attila Magyar
> 
>

Re: Review Request 61251: Dynamically determine what keytab files have been distributed

2017-08-03 Thread Attila Magyar 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61251/
---

(Updated Aug. 3, 2017, 8:14 a.m.)


Review request for Ambari, Balázs Bence Sári, Robert Levas, and Sebastian 
Toader.


Changes
---

sudo for running klist


Bugs: AMBARI-21613
https://issues.apache.org/jira/browse/AMBARI-21613


Repository: ambari


Description
---

Dynamically determine what keytab files have been distributed to hosts. A 
custom command should be available via the KERBEROS_CLIENT to query for the 
keytab files installed on the relevant host. The communication between the 
Ambari server and the agents should generate data needed to determine what 
keytab files exist.


Diffs (updated)
-

  ambari-common/src/main/python/resource_management/core/resources/klist.py 
PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatHandler.java
 1bc4c36 
  
ambari-server/src/main/java/org/apache/ambari/server/agent/HeartbeatProcessor.java
 a08abab 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
 e5b7afd 
  
ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/metainfo.xml
 6a2dd09 
  
ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/package/scripts/kerberos_client.py
 39fdcf5 
  
ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/package/scripts/kerberos_common.py
 fcd57af 
  
ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/metainfo.xml 
0e42bda 
  
ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/package/scripts/kerberos_client.py
 b2cdaa6 
  
ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/package/scripts/kerberos_common.py
 abf58ee 
  
ambari-server/src/test/java/org/apache/ambari/server/agent/TestHeartbeatHandler.java
 baa9bae 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java
 4508527 
  ambari-server/src/test/python/stacks/2.2/KERBEROS/test_kerberos_client.py 
f638845 


Diff: https://reviews.apache.org/r/61251/diff/4/

Changes: https://reviews.apache.org/r/61251/diff/3-4/


Testing (updated)
---

1.
- created a cluster with kerberos enabled
- deleted a keytab file from a host
- regenerated missing keytabs
- checked that the deleted keytab file was recreated

2.
- regenerated missing keytabs when no keytab files were missing
- checked that no keytab file was regenerated

3.
- created a 15 nodes cluster on openstack with services: HDFS, YARN, ZOOKEEPER, 
SPARK1, SPARK2, HIVE
- measured the time taken to regenerate all keytabs and only missing keytabs:

  regenerate missing keytabs only: 12.84 seconds (?=2.33)
  regenerate all keytabs: 39.1 seconds (?=5.37)

existing tests:

server:
Ran 246 tests in 6.999s
OK
--
Total run:1146
Total errors:0
Total failures:0

Results :
Tests run: 4816, Failures: 0, Errors: 0, Skipped: 35

agent:
Ran 470 tests in 98.589s
OK


Thanks,

Attila Magyar

Re: Review Request 61251: Dynamically determine what keytab files have been distributed

2017-08-02 Thread Robert Levas 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61251/#review182016
---


Fix it, then Ship it!




This looks great!  Thanks for making those changes.


ambari-common/src/main/python/resource_management/core/resources/klist.py
Lines 37 (patched)


This should be execued via sudo...

```
 code, out, err = shell.call(self._command('-k', keytab_path), stdout = 
PIPE, stderr = PIPE, timeout = 5, sudo = True)
```


- Robert Levas


On Aug. 2, 2017, 7:59 a.m., Attila Magyar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61251/
> ---
> 
> (Updated Aug. 2, 2017, 7:59 a.m.)
> 
> 
> Review request for Ambari, Balázs Bence Sári, Robert Levas, and Sebastian 
> Toader.
> 
> 
> Bugs: AMBARI-21613
> https://issues.apache.org/jira/browse/AMBARI-21613
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Dynamically determine what keytab files have been distributed to hosts. A 
> custom command should be available via the KERBEROS_CLIENT to query for the 
> keytab files installed on the relevant host. The communication between the 
> Ambari server and the agents should generate data needed to determine what 
> keytab files exist.
> 
> 
> Diffs
> -
> 
>   ambari-common/src/main/python/resource_management/core/resources/klist.py 
> PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatHandler.java
>  1bc4c36 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartbeatProcessor.java
>  a08abab 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
>  e5b7afd 
>   
> ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/metainfo.xml
>  6a2dd09 
>   
> ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/package/scripts/kerberos_client.py
>  39fdcf5 
>   
> ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/package/scripts/kerberos_common.py
>  fcd57af 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/metainfo.xml
>  0e42bda 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/package/scripts/kerberos_client.py
>  b2cdaa6 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/package/scripts/kerberos_common.py
>  abf58ee 
>   
> ambari-server/src/test/java/org/apache/ambari/server/agent/TestHeartbeatHandler.java
>  baa9bae 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java
>  4508527 
>   ambari-server/src/test/python/stacks/2.2/KERBEROS/test_kerberos_client.py 
> f638845 
> 
> 
> Diff: https://reviews.apache.org/r/61251/diff/3/
> 
> 
> Testing
> ---
> 
> 1.
> - created a cluster with kerberos enabled
> - deleted a keytab file from a host
> - regenerated missing keytabs
> - checked that the deleted keytab file was recreated
> 
> 2.
> - regenerated missing keytabs when no keytab files were missing
> - checked that no keytab file was regenerated
> 
> 3.
> - created a 15 nodes cluster on openstack with services: HDFS, YARN, 
> ZOOKEEPER, SPARK1, SPARK2, HIVE
> - measured the time taken to regenerate all keytabs and only missing keytabs:
> 
>   regenerate missing keytabs only: 12.84 seconds (?=2.33)
>   regenerate all keytabs: 39.1 seconds (?=5.37)
> 
> existing tests: pending
> 
> 
> Thanks,
> 
> Attila Magyar
> 
>

Re: Review Request 61251: Dynamically determine what keytab files have been distributed

2017-08-02 Thread Sebastian Toader 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61251/#review181972
---


Ship it!




Ship It!

- Sebastian Toader


On Aug. 2, 2017, 1:59 p.m., Attila Magyar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61251/
> ---
> 
> (Updated Aug. 2, 2017, 1:59 p.m.)
> 
> 
> Review request for Ambari, Balázs Bence Sári, Robert Levas, and Sebastian 
> Toader.
> 
> 
> Bugs: AMBARI-21613
> https://issues.apache.org/jira/browse/AMBARI-21613
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Dynamically determine what keytab files have been distributed to hosts. A 
> custom command should be available via the KERBEROS_CLIENT to query for the 
> keytab files installed on the relevant host. The communication between the 
> Ambari server and the agents should generate data needed to determine what 
> keytab files exist.
> 
> 
> Diffs
> -
> 
>   ambari-common/src/main/python/resource_management/core/resources/klist.py 
> PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatHandler.java
>  1bc4c36 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartbeatProcessor.java
>  a08abab 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
>  e5b7afd 
>   
> ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/metainfo.xml
>  6a2dd09 
>   
> ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/package/scripts/kerberos_client.py
>  39fdcf5 
>   
> ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/package/scripts/kerberos_common.py
>  fcd57af 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/metainfo.xml
>  0e42bda 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/package/scripts/kerberos_client.py
>  b2cdaa6 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/package/scripts/kerberos_common.py
>  abf58ee 
>   
> ambari-server/src/test/java/org/apache/ambari/server/agent/TestHeartbeatHandler.java
>  baa9bae 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java
>  4508527 
>   ambari-server/src/test/python/stacks/2.2/KERBEROS/test_kerberos_client.py 
> f638845 
> 
> 
> Diff: https://reviews.apache.org/r/61251/diff/3/
> 
> 
> Testing
> ---
> 
> 1.
> - created a cluster with kerberos enabled
> - deleted a keytab file from a host
> - regenerated missing keytabs
> - checked that the deleted keytab file was recreated
> 
> 2.
> - regenerated missing keytabs when no keytab files were missing
> - checked that no keytab file was regenerated
> 
> 3.
> - created a 15 nodes cluster on openstack with services: HDFS, YARN, 
> ZOOKEEPER, SPARK1, SPARK2, HIVE
> - measured the time taken to regenerate all keytabs and only missing keytabs:
> 
>   regenerate missing keytabs only: 12.84 seconds (?=2.33)
>   regenerate all keytabs: 39.1 seconds (?=5.37)
> 
> existing tests: pending
> 
> 
> Thanks,
> 
> Attila Magyar
> 
>

Re: Review Request 61251: Dynamically determine what keytab files have been distributed

2017-08-02 Thread Attila Magyar 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61251/
---

(Updated Aug. 2, 2017, 11:59 a.m.)


Review request for Ambari, Balázs Bence Sári, Robert Levas, and Sebastian 
Toader.


Changes
---

added additional check for testing if the keytab file has the correct principal


Bugs: AMBARI-21613
https://issues.apache.org/jira/browse/AMBARI-21613


Repository: ambari


Description
---

Dynamically determine what keytab files have been distributed to hosts. A 
custom command should be available via the KERBEROS_CLIENT to query for the 
keytab files installed on the relevant host. The communication between the 
Ambari server and the agents should generate data needed to determine what 
keytab files exist.


Diffs (updated)
-

  ambari-common/src/main/python/resource_management/core/resources/klist.py 
PRE-CREATION 
  
ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatHandler.java
 1bc4c36 
  
ambari-server/src/main/java/org/apache/ambari/server/agent/HeartbeatProcessor.java
 a08abab 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
 e5b7afd 
  
ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/metainfo.xml
 6a2dd09 
  
ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/package/scripts/kerberos_client.py
 39fdcf5 
  
ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/package/scripts/kerberos_common.py
 fcd57af 
  
ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/metainfo.xml 
0e42bda 
  
ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/package/scripts/kerberos_client.py
 b2cdaa6 
  
ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/package/scripts/kerberos_common.py
 abf58ee 
  
ambari-server/src/test/java/org/apache/ambari/server/agent/TestHeartbeatHandler.java
 baa9bae 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java
 4508527 
  ambari-server/src/test/python/stacks/2.2/KERBEROS/test_kerberos_client.py 
f638845 


Diff: https://reviews.apache.org/r/61251/diff/3/

Changes: https://reviews.apache.org/r/61251/diff/2-3/


Testing
---

1.
- created a cluster with kerberos enabled
- deleted a keytab file from a host
- regenerated missing keytabs
- checked that the deleted keytab file was recreated

2.
- regenerated missing keytabs when no keytab files were missing
- checked that no keytab file was regenerated

3.
- created a 15 nodes cluster on openstack with services: HDFS, YARN, ZOOKEEPER, 
SPARK1, SPARK2, HIVE
- measured the time taken to regenerate all keytabs and only missing keytabs:

  regenerate missing keytabs only: 12.84 seconds (?=2.33)
  regenerate all keytabs: 39.1 seconds (?=5.37)

existing tests: pending


Thanks,

Attila Magyar

Re: Review Request 61251: Dynamically determine what keytab files have been distributed

2017-08-01 Thread Sebastian Toader 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61251/#review181897
---


Ship it!




Ship It!

- Sebastian Toader


On Aug. 1, 2017, 4:56 p.m., Attila Magyar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61251/
> ---
> 
> (Updated Aug. 1, 2017, 4:56 p.m.)
> 
> 
> Review request for Ambari, Balázs Bence Sári, Robert Levas, and Sebastian 
> Toader.
> 
> 
> Bugs: AMBARI-21613
> https://issues.apache.org/jira/browse/AMBARI-21613
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Dynamically determine what keytab files have been distributed to hosts. A 
> custom command should be available via the KERBEROS_CLIENT to query for the 
> keytab files installed on the relevant host. The communication between the 
> Ambari server and the agents should generate data needed to determine what 
> keytab files exist.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatHandler.java
>  1bc4c36 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartbeatProcessor.java
>  a08abab 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
>  e5b7afd 
>   
> ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/metainfo.xml
>  6a2dd09 
>   
> ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/package/scripts/kerberos_client.py
>  39fdcf5 
>   
> ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/package/scripts/kerberos_common.py
>  fcd57af 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/metainfo.xml
>  0e42bda 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/package/scripts/kerberos_client.py
>  b2cdaa6 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/package/scripts/kerberos_common.py
>  abf58ee 
>   
> ambari-server/src/test/java/org/apache/ambari/server/agent/TestHeartbeatHandler.java
>  baa9bae 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java
>  4508527 
>   ambari-server/src/test/python/stacks/2.2/KERBEROS/test_kerberos_client.py 
> f638845 
> 
> 
> Diff: https://reviews.apache.org/r/61251/diff/2/
> 
> 
> Testing
> ---
> 
> 1.
> - created a cluster with kerberos enabled
> - deleted a keytab file from a host
> - regenerated missing keytabs
> - checked that the deleted keytab file was recreated
> 
> 2.
> - regenerated missing keytabs when no keytab files were missing
> - checked that no keytab file was regenerated
> 
> 3.
> - created a 15 nodes cluster on openstack with services: HDFS, YARN, 
> ZOOKEEPER, SPARK1, SPARK2, HIVE
> - measured the time taken to regenerate all keytabs and only missing keytabs:
> 
>   regenerate missing keytabs only: 12.84 seconds (?=2.33)
>   regenerate all keytabs: 39.1 seconds (?=5.37)
> 
> existing tests: pending
> 
> 
> Thanks,
> 
> Attila Magyar
> 
>

Re: Review Request 61251: Dynamically determine what keytab files have been distributed

2017-08-01 Thread Attila Magyar 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61251/
---

(Updated Aug. 1, 2017, 2:56 p.m.)


Review request for Ambari, Balázs Bence Sári, Robert Levas, and Sebastian 
Toader.


Changes
---

- os.path.isfile ->sudo.path_exists
- MISSING_KEYTABS -> CHECK_KEYTABS
- added javadoc, fixd typo


Bugs: AMBARI-21613
https://issues.apache.org/jira/browse/AMBARI-21613


Repository: ambari


Description
---

Dynamically determine what keytab files have been distributed to hosts. A 
custom command should be available via the KERBEROS_CLIENT to query for the 
keytab files installed on the relevant host. The communication between the 
Ambari server and the agents should generate data needed to determine what 
keytab files exist.


Diffs (updated)
-

  
ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatHandler.java
 1bc4c36 
  
ambari-server/src/main/java/org/apache/ambari/server/agent/HeartbeatProcessor.java
 a08abab 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
 e5b7afd 
  
ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/metainfo.xml
 6a2dd09 
  
ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/package/scripts/kerberos_client.py
 39fdcf5 
  
ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/package/scripts/kerberos_common.py
 fcd57af 
  
ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/metainfo.xml 
0e42bda 
  
ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/package/scripts/kerberos_client.py
 b2cdaa6 
  
ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/package/scripts/kerberos_common.py
 abf58ee 
  
ambari-server/src/test/java/org/apache/ambari/server/agent/TestHeartbeatHandler.java
 baa9bae 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java
 4508527 
  ambari-server/src/test/python/stacks/2.2/KERBEROS/test_kerberos_client.py 
f638845 


Diff: https://reviews.apache.org/r/61251/diff/2/

Changes: https://reviews.apache.org/r/61251/diff/1-2/


Testing
---

1.
- created a cluster with kerberos enabled
- deleted a keytab file from a host
- regenerated missing keytabs
- checked that the deleted keytab file was recreated

2.
- regenerated missing keytabs when no keytab files were missing
- checked that no keytab file was regenerated

3.
- created a 15 nodes cluster on openstack with services: HDFS, YARN, ZOOKEEPER, 
SPARK1, SPARK2, HIVE
- measured the time taken to regenerate all keytabs and only missing keytabs:

  regenerate missing keytabs only: 12.84 seconds (?=2.33)
  regenerate all keytabs: 39.1 seconds (?=5.37)

existing tests: pending


Thanks,

Attila Magyar

Re: Review Request 61251: Dynamically determine what keytab files have been distributed

2017-08-01 Thread Robert Levas 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61251/#review181891
---



This looks great. I like the concept.


ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
Lines 145 (patched)


How about renaming this to something like `CHECK_KEYTAB`.  It is more like 
an actions like... SET and REMOVE.



ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
Lines 3047 (patched)


`addGettingMisingKeytabsStage` --> `addGettingMissingKeytabsStage`  
(spelling issue: Mising)

Also, missing JavaDoc.



ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/package/scripts/kerberos_common.py
Lines 456-457 (patched)


In the (not so distanct) future we may want to place multiple principals in 
the same keytab file, or at least allow the user to set the configurations for 
this.

It appears that the input and output structures should support this, 
however the test is not checking for principal. It only checks for the 
existance of the keytab file.  It would be nice to make sure the specifiec 
principal exists in the specified keytab file. Additional work will be needed 
if we want to verify that the requied keytabs are in the keytab file.  For 
example AES256, RC4, etc... 

However, for now, maybe a `klist` needs to be performed on the keytab file 
to ensure at least one keytab entry for the requested principal exists. 

Sonehow this should be done via sudo in the event the agent is not running 
as root.  Maybe a KeytabFile resource can be created for this? See 
`resource_management.core.providers.system.FileProvider` and 
`resource_management.core.resources.system.File`.


- Robert Levas


On Aug. 1, 2017, 5:58 a.m., Attila Magyar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61251/
> ---
> 
> (Updated Aug. 1, 2017, 5:58 a.m.)
> 
> 
> Review request for Ambari, Balázs Bence Sári, Robert Levas, and Sebastian 
> Toader.
> 
> 
> Bugs: AMBARI-21613
> https://issues.apache.org/jira/browse/AMBARI-21613
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Dynamically determine what keytab files have been distributed to hosts. A 
> custom command should be available via the KERBEROS_CLIENT to query for the 
> keytab files installed on the relevant host. The communication between the 
> Ambari server and the agents should generate data needed to determine what 
> keytab files exist.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatHandler.java
>  1bc4c36 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartbeatProcessor.java
>  a08abab 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
>  e5b7afd 
>   
> ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/metainfo.xml
>  6a2dd09 
>   
> ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/package/scripts/kerberos_client.py
>  39fdcf5 
>   
> ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/package/scripts/kerberos_common.py
>  fcd57af 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/metainfo.xml
>  0e42bda 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/package/scripts/kerberos_client.py
>  b2cdaa6 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/package/scripts/kerberos_common.py
>  abf58ee 
>   
> ambari-server/src/test/java/org/apache/ambari/server/agent/TestHeartbeatHandler.java
>  baa9bae 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java
>  4508527 
>   ambari-server/src/test/python/stacks/2.2/KERBEROS/test_kerberos_client.py 
> f638845 
> 
> 
> Diff: https://reviews.apache.org/r/61251/diff/1/
> 
> 
> Testing
> ---
> 
> 1.
> - created a cluster with kerberos enabled
> - deleted a keytab file from a host
> - regenerated missing keytabs
> - checked that the deleted keytab file was recreated
> 
> 2.
> - regenerated missing keytabs when no keytab files were missing
> - checked that no keytab file was regenerated
> 
> 3.
> - created a 15 nodes cluster on openstack with services: HDFS, YARN, 
> ZOOKEEPER, SPARK1, SPARK2, HIVE
> - measured the time taken to regenerate all keytabs and only missing keytabs:
> 
>   regenerate missing keytabs only: 12.84 seconds (?=2.33)
>   regenerate all keytabs: 39.1 seconds (?=5.37)
> 
> existing tests: pending
> 
> 
> Thanks,
> 
> Attila M

Re: Review Request 61251: Dynamically determine what keytab files have been distributed

2017-08-01 Thread Sebastian Toader 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61251/#review181888
---




ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/package/scripts/kerberos_common.py
Lines 471 (patched)


If ambari-agent runs as non-root user, will the agent have access to the 
path of the key tab file to verify it exists? Note access to a keytab file may 
be restricted to specific users and groups (defined in kerberos.json).


- Sebastian Toader


On Aug. 1, 2017, 11:58 a.m., Attila Magyar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61251/
> ---
> 
> (Updated Aug. 1, 2017, 11:58 a.m.)
> 
> 
> Review request for Ambari, Balázs Bence Sári, Robert Levas, and Sebastian 
> Toader.
> 
> 
> Bugs: AMBARI-21613
> https://issues.apache.org/jira/browse/AMBARI-21613
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Dynamically determine what keytab files have been distributed to hosts. A 
> custom command should be available via the KERBEROS_CLIENT to query for the 
> keytab files installed on the relevant host. The communication between the 
> Ambari server and the agents should generate data needed to determine what 
> keytab files exist.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatHandler.java
>  1bc4c36 
>   
> ambari-server/src/main/java/org/apache/ambari/server/agent/HeartbeatProcessor.java
>  a08abab 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
>  e5b7afd 
>   
> ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/metainfo.xml
>  6a2dd09 
>   
> ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/package/scripts/kerberos_client.py
>  39fdcf5 
>   
> ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/package/scripts/kerberos_common.py
>  fcd57af 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/metainfo.xml
>  0e42bda 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/package/scripts/kerberos_client.py
>  b2cdaa6 
>   
> ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/package/scripts/kerberos_common.py
>  abf58ee 
>   
> ambari-server/src/test/java/org/apache/ambari/server/agent/TestHeartbeatHandler.java
>  baa9bae 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java
>  4508527 
>   ambari-server/src/test/python/stacks/2.2/KERBEROS/test_kerberos_client.py 
> f638845 
> 
> 
> Diff: https://reviews.apache.org/r/61251/diff/1/
> 
> 
> Testing
> ---
> 
> 1.
> - created a cluster with kerberos enabled
> - deleted a keytab file from a host
> - regenerated missing keytabs
> - checked that the deleted keytab file was recreated
> 
> 2.
> - regenerated missing keytabs when no keytab files were missing
> - checked that no keytab file was regenerated
> 
> 3.
> - created a 15 nodes cluster on openstack with services: HDFS, YARN, 
> ZOOKEEPER, SPARK1, SPARK2, HIVE
> - measured the time taken to regenerate all keytabs and only missing keytabs:
> 
>   regenerate missing keytabs only: 12.84 seconds (?=2.33)
>   regenerate all keytabs: 39.1 seconds (?=5.37)
> 
> existing tests: pending
> 
> 
> Thanks,
> 
> Attila Magyar
> 
>

Review Request 61251: Dynamically determine what keytab files have been distributed

2017-08-01 Thread Attila Magyar 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61251/
---

Review request for Ambari, Balázs Bence Sári, Robert Levas, and Sebastian 
Toader.


Bugs: AMBARI-21613
https://issues.apache.org/jira/browse/AMBARI-21613


Repository: ambari


Description
---

Dynamically determine what keytab files have been distributed to hosts. A 
custom command should be available via the KERBEROS_CLIENT to query for the 
keytab files installed on the relevant host. The communication between the 
Ambari server and the agents should generate data needed to determine what 
keytab files exist.


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/agent/HeartBeatHandler.java
 1bc4c36 
  
ambari-server/src/main/java/org/apache/ambari/server/agent/HeartbeatProcessor.java
 a08abab 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
 e5b7afd 
  
ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/metainfo.xml
 6a2dd09 
  
ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/package/scripts/kerberos_client.py
 39fdcf5 
  
ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/package/scripts/kerberos_common.py
 fcd57af 
  
ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/metainfo.xml 
0e42bda 
  
ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/package/scripts/kerberos_client.py
 b2cdaa6 
  
ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/package/scripts/kerberos_common.py
 abf58ee 
  
ambari-server/src/test/java/org/apache/ambari/server/agent/TestHeartbeatHandler.java
 baa9bae 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/KerberosHelperTest.java
 4508527 
  ambari-server/src/test/python/stacks/2.2/KERBEROS/test_kerberos_client.py 
f638845 


Diff: https://reviews.apache.org/r/61251/diff/1/


Testing
---

1.
- created a cluster with kerberos enabled
- deleted a keytab file from a host
- regenerated missing keytabs
- checked that the deleted keytab file was recreated

2.
- regenerated missing keytabs when no keytab files were missing
- checked that no keytab file was regenerated

3.
- created a 15 nodes cluster on openstack with services: HDFS, YARN, ZOOKEEPER, 
SPARK1, SPARK2, HIVE
- measured the time taken to regenerate all keytabs and only missing keytabs:

  regenerate missing keytabs only: 12.84 seconds (?=2.33)
  regenerate all keytabs: 39.1 seconds (?=5.37)

existing tests: pending


Thanks,

Attila Magyar