Re: Review Request 59992: AMBARI-21226. Add two new Spark thrift server related configurations in Spark stack

[Saisai Shao]

> On June 13, 2017, 3:18 p.m., Sumit Mohanty wrote:
> > One more thing: If user upgrades from HDP-2.6.x to 2.6.2, do you want the 
> > configs to be added? If yes then that upgrade xml change is needed.
> 
> Saisai Shao wrote:
> Yes, I want. So do I only need to change the code under HDP/2.6/upgrades?
> 
> Sumit Mohanty wrote:
> Yes.
> 
> Saisai Shao wrote:
> Hi Sumit, what is the usage of host-upgrade-2.6.xml file under 
> HDP/2.6/upgrades, should I also change this file?
> 
> Sumit Mohanty wrote:
> Something similar to 
> https://github.com/apache/ambari/commit/ccd6b25ed2f053a9b1b2045fd0cf41f4472e657e
>  (see changes in the 2.6 folder)

Thanks! Looks like the current patch is enough :)


- Saisai


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59992/#review177746
---


On June 14, 2017, 1:31 a.m., Saisai Shao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59992/
> ---
> 
> (Updated June 14, 2017, 1:31 a.m.)
> 
> 
> Review request for Ambari and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-21226
> https://issues.apache.org/jira/browse/AMBARI-21226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> AMBARI-21226. Add two new Spark thrift server related configurations in Spark 
> stack
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/configuration/spark-thrift-sparkconf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK2/configuration/spark2-thrift-sparkconf.xml
>  PRE-CREATION 
>   ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml 
> 1610bb5 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml
>  1cdd184 
>   ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml 
> 3e7e3d7 
> 
> 
> Diff: https://reviews.apache.org/r/59992/diff/3/
> 
> 
> Testing
> ---
> 
> Test is done manually on:
> 
> updating HDP from 2.3 -> 2.6, 2.4 -> 2.6, 2.5 -> 2.6, and fresh install of 2.6
> 
> 
> Thanks,
> 
> Saisai Shao
> 
>

Re: Review Request 59992: AMBARI-21226. Add two new Spark thrift server related configurations in Spark stack

[Sumit Mohanty]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59992/#review177855
---


Ship it!




Ship It!

- Sumit Mohanty


On June 14, 2017, 1:31 a.m., Saisai Shao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59992/
> ---
> 
> (Updated June 14, 2017, 1:31 a.m.)
> 
> 
> Review request for Ambari and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-21226
> https://issues.apache.org/jira/browse/AMBARI-21226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> AMBARI-21226. Add two new Spark thrift server related configurations in Spark 
> stack
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/configuration/spark-thrift-sparkconf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK2/configuration/spark2-thrift-sparkconf.xml
>  PRE-CREATION 
>   ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml 
> 1610bb5 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml
>  1cdd184 
>   ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml 
> 3e7e3d7 
> 
> 
> Diff: https://reviews.apache.org/r/59992/diff/3/
> 
> 
> Testing
> ---
> 
> Test is done manually on:
> 
> updating HDP from 2.3 -> 2.6, 2.4 -> 2.6, 2.5 -> 2.6, and fresh install of 2.6
> 
> 
> Thanks,
> 
> Saisai Shao
> 
>

Re: Review Request 59992: AMBARI-21226. Add two new Spark thrift server related configurations in Spark stack

[Sumit Mohanty]

> On June 13, 2017, 3:18 p.m., Sumit Mohanty wrote:
> > One more thing: If user upgrades from HDP-2.6.x to 2.6.2, do you want the 
> > configs to be added? If yes then that upgrade xml change is needed.
> 
> Saisai Shao wrote:
> Yes, I want. So do I only need to change the code under HDP/2.6/upgrades?
> 
> Sumit Mohanty wrote:
> Yes.
> 
> Saisai Shao wrote:
> Hi Sumit, what is the usage of host-upgrade-2.6.xml file under 
> HDP/2.6/upgrades, should I also change this file?

Something similar to 
https://github.com/apache/ambari/commit/ccd6b25ed2f053a9b1b2045fd0cf41f4472e657e
 (see changes in the 2.6 folder)


- Sumit


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59992/#review177746
---


On June 14, 2017, 1:31 a.m., Saisai Shao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59992/
> ---
> 
> (Updated June 14, 2017, 1:31 a.m.)
> 
> 
> Review request for Ambari and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-21226
> https://issues.apache.org/jira/browse/AMBARI-21226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> AMBARI-21226. Add two new Spark thrift server related configurations in Spark 
> stack
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/configuration/spark-thrift-sparkconf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK2/configuration/spark2-thrift-sparkconf.xml
>  PRE-CREATION 
>   ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml 
> 1610bb5 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml
>  1cdd184 
>   ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml 
> 3e7e3d7 
> 
> 
> Diff: https://reviews.apache.org/r/59992/diff/3/
> 
> 
> Testing
> ---
> 
> Test is done manually on:
> 
> updating HDP from 2.3 -> 2.6, 2.4 -> 2.6, 2.5 -> 2.6, and fresh install of 2.6
> 
> 
> Thanks,
> 
> Saisai Shao
> 
>

Re: Review Request 59992: AMBARI-21226. Add two new Spark thrift server related configurations in Spark stack

[Saisai Shao]

> On June 13, 2017, 3:18 p.m., Sumit Mohanty wrote:
> > One more thing: If user upgrades from HDP-2.6.x to 2.6.2, do you want the 
> > configs to be added? If yes then that upgrade xml change is needed.
> 
> Saisai Shao wrote:
> Yes, I want. So do I only need to change the code under HDP/2.6/upgrades?
> 
> Sumit Mohanty wrote:
> Yes.

Hi Sumit, what is the usage of host-upgrade-2.6.xml file under 
HDP/2.6/upgrades, should I also change this file?


- Saisai


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59992/#review177746
---


On June 14, 2017, 1:31 a.m., Saisai Shao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59992/
> ---
> 
> (Updated June 14, 2017, 1:31 a.m.)
> 
> 
> Review request for Ambari and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-21226
> https://issues.apache.org/jira/browse/AMBARI-21226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> AMBARI-21226. Add two new Spark thrift server related configurations in Spark 
> stack
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/configuration/spark-thrift-sparkconf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK2/configuration/spark2-thrift-sparkconf.xml
>  PRE-CREATION 
>   ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml 
> 1610bb5 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml
>  1cdd184 
>   ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml 
> 3e7e3d7 
> 
> 
> Diff: https://reviews.apache.org/r/59992/diff/3/
> 
> 
> Testing
> ---
> 
> Test is done manually on:
> 
> updating HDP from 2.3 -> 2.6, 2.4 -> 2.6, 2.5 -> 2.6, and fresh install of 2.6
> 
> 
> Thanks,
> 
> Saisai Shao
> 
>

Re: Review Request 59992: AMBARI-21226. Add two new Spark thrift server related configurations in Spark stack

[Saisai Shao]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59992/
---

(Updated June 14, 2017, 1:31 a.m.)


Review request for Ambari and Sumit Mohanty.


Bugs: AMBARI-21226
https://issues.apache.org/jira/browse/AMBARI-21226


Repository: ambari


Description
---

AMBARI-21226. Add two new Spark thrift server related configurations in Spark 
stack


Diffs (updated)
-

  
ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/configuration/spark-thrift-sparkconf.xml
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK2/configuration/spark2-thrift-sparkconf.xml
 PRE-CREATION 
  ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/config-upgrade.xml 
1610bb5 
  
ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/nonrolling-upgrade-2.6.xml
 1cdd184 
  ambari-server/src/main/resources/stacks/HDP/2.6/upgrades/upgrade-2.6.xml 
3e7e3d7 


Diff: https://reviews.apache.org/r/59992/diff/3/

Changes: https://reviews.apache.org/r/59992/diff/2-3/


Testing
---

Test is done manually on:

updating HDP from 2.3 -> 2.6, 2.4 -> 2.6, 2.5 -> 2.6, and fresh install of 2.6


Thanks,

Saisai Shao

Re: Review Request 59992: AMBARI-21226. Add two new Spark thrift server related configurations in Spark stack

[Saisai Shao]

> On June 13, 2017, 3:18 p.m., Sumit Mohanty wrote:
> > One more thing: If user upgrades from HDP-2.6.x to 2.6.2, do you want the 
> > configs to be added? If yes then that upgrade xml change is needed.

Yes, I want. So do I only need to change the code under HDP/2.6/upgrades?


- Saisai


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59992/#review177746
---


On June 13, 2017, 5:44 a.m., Saisai Shao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59992/
> ---
> 
> (Updated June 13, 2017, 5:44 a.m.)
> 
> 
> Review request for Ambari and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-21226
> https://issues.apache.org/jira/browse/AMBARI-21226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> AMBARI-21226. Add two new Spark thrift server related configurations in Spark 
> stack
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/configuration/spark-thrift-sparkconf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK2/configuration/spark2-thrift-sparkconf.xml
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/59992/diff/2/
> 
> 
> Testing
> ---
> 
> Test is done manually on:
> 
> updating HDP from 2.3 -> 2.6, 2.4 -> 2.6, 2.5 -> 2.6, and fresh install of 2.6
> 
> 
> Thanks,
> 
> Saisai Shao
> 
>

Re: Review Request 59838: Stack advisor needs to recommend dependency for slaves and masters

[Tim Thorpe]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59838/
---

(Updated June 13, 2017, 9:56 p.m.)


Review request for Ambari, Alejandro Fernandez, Di Li, Jaimin Jetly, Jonathan 
Hurley, Jayush Luniya, and Nate Cole.


Changes
---

Updated diff to fix the UI issue: 
  File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py", 
line 901, in sortServicesByDependencies
for component in service["components"]:
TypeError: 'NoneType' object is unsubscriptable


Bugs: AMBARI-20122
https://issues.apache.org/jira/browse/AMBARI-20122


Repository: ambari


Description
---

After resolution of AMBARI-19685, stack advisor validates if stack defined 
dependency is not satisfied but recommendation API does not account for this.

Stack defined dependencies are service/component based and has a scope 
CLUSTER|HOST.  

During recommendation the services to install have already been selected.  We 
can't really utilize the cluster scope because either the dependent service was 
selected or it was not.  If it was not selected it will be caught during 
validation.  We can only recommend based on HOST scope.

This JIRA is also limited to only handling those which don't have conditional 
dependencies.


Diffs (updated)
-

  ambari-server/src/main/resources/stacks/stack_advisor.py 4a81dc6 
  ambari-server/src/test/python/stacks/2.0.6/common/test_stack_advisor.py 
41c57f6 


Diff: https://reviews.apache.org/r/59838/diff/4/

Changes: https://reviews.apache.org/r/59838/diff/3-4/


Testing
---

Added new test case to test the recommendation of dependencies for slaves and 
masters

mvn clean test -DskipSurefireTests


New trunk without my change

Total run:1168
Total errors:13
Total failures:0

New trunk with my change

Total run:1169
Total errors:13
Total failures:0


Thanks,

Tim Thorpe

Re: Review Request 59929: Extend Swagger Maven pluging to handle nested API's

[Balázs Bence Sári]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59929/
---

(Updated June 13, 2017, 9:39 p.m.)


Review request for Ambari, Attila Doroszlai, Jaimin Jetly, Oliver Szabo, Sumit 
Mohanty, and Sebastian Toader.


Changes
---

organize dependencies more nicely


Bugs: AMBARI-21207
https://issues.apache.org/jira/browse/AMBARI-21207


Repository: ambari


Description
---

Wrote a custom SwaggerReader in the utility project that handles nested API's. 
There is currently one nested API which is annotated for swagger: 
org.apache.ambari.server.api.services.ServiceService (nested under 
ClusterService). Using the custom reader operation paths and parameters are 
generated correctly for the nested API.


Diffs (updated)
-

  ambari-project/pom.xml 64d1416 
  ambari-server/pom.xml 669f90f 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/ClusterService.java
 f61fb2a 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/ServiceService.java
 a28c4aa 
  utility/pom.xml 7d5eb93 
  utility/src/main/java/org/apache/ambari/swagger/AmbariSwaggerReader.java 
PRE-CREATION 
  utility/src/test/java/org/apache/ambari/swagger/AmbariSwaggerReaderTest.java 
PRE-CREATION 
  utility/src/test/resources/log4j.properties PRE-CREATION 


Diff: https://reviews.apache.org/r/59929/diff/3/

Changes: https://reviews.apache.org/r/59929/diff/2-3/


Testing
---

Tested Swagger API generation manually. Wrote new unit tests. Run all unit 
tests for the ambari-server and utility projects, no errors or failures.


Thanks,

Balázs Bence Sári

Re: Review Request 60055: Support revert for patch upgrades

[Nate Cole]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60055/
---

(Updated June 13, 2017, 5:19 p.m.)


Review request for Ambari, Alejandro Fernandez, Dmitro Lisnichenko, and 
Jonathan Hurley.


Changes
---

Update for test fixes


Bugs: AMBARI-21241
https://issues.apache.org/jira/browse/AMBARI-21241


Repository: ambari


Description
---

Entails adding an option to specify a previous upgrade to revert.  Checks to 
make sure it's only a patch.


Diffs (updated)
-

  
ambari-server/src/main/java/org/apache/ambari/server/checks/HostsRepositoryVersionCheck.java
 a66db3c008 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/PrereqCheckRequest.java
 c8c9f9ed2b 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/PreUpgradeCheckResourceProvider.java
 ea8fb372dc 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java
 60665f7b2b 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/UpgradeEntity.java
 43b2e088d7 
  
ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/FinalizeUpgradeAction.java
 e70f1ce8b2 
  
ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/UpdateDesiredStackAction.java
 8a4820d7f4 
  
ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeContext.java 
3ecf64dc0c 
  ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql e34140593a 
  ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql ce8736e62d 
  ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql ace3738d2c 
  ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql 5899179aba 
  ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql 6164b2de5c 
  ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql 52d2b87c74 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackUpgradeConfigurationMergeTest.java
 a37e4f5d12 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UpgradeResourceProviderTest.java
 014ab4223f 
  
ambari-server/src/test/java/org/apache/ambari/server/state/UpgradeHelperTest.java
 921322b68c 
  ambari-server/src/test/resources/hbase_version_test.xml 58c87015b5 


Diff: https://reviews.apache.org/r/60055/diff/2/

Changes: https://reviews.apache.org/r/60055/diff/1-2/


Testing (updated)
---

Manual.  Automated:

[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time: 25:08 min
[INFO] Finished at: 2017-06-13T17:18:07-04:00
[INFO] Final Memory: 73M/673M
[INFO] 


Thanks,

Nate Cole

Re: Review Request 60055: Support revert for patch upgrades

[Alejandro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60055/#review177793
---


Ship it!




Ship It!

- Alejandro Fernandez


On June 13, 2017, 7:24 p.m., Nate Cole wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60055/
> ---
> 
> (Updated June 13, 2017, 7:24 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Dmitro Lisnichenko, and 
> Jonathan Hurley.
> 
> 
> Bugs: AMBARI-21241
> https://issues.apache.org/jira/browse/AMBARI-21241
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Entails adding an option to specify a previous upgrade to revert.  Checks to 
> make sure it's only a patch.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/checks/HostsRepositoryVersionCheck.java
>  a66db3c008 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/PrereqCheckRequest.java
>  c8c9f9ed2b 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/PreUpgradeCheckResourceProvider.java
>  ea8fb372dc 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java
>  60665f7b2b 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/UpgradeEntity.java
>  43b2e088d7 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/FinalizeUpgradeAction.java
>  e70f1ce8b2 
>   
> ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/UpdateDesiredStackAction.java
>  8a4820d7f4 
>   
> ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeContext.java
>  3ecf64dc0c 
>   ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql e34140593a 
>   ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql ce8736e62d 
>   ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql ace3738d2c 
>   ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql 5899179aba 
>   ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql 
> 6164b2de5c 
>   ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql 52d2b87c74 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackUpgradeConfigurationMergeTest.java
>  a37e4f5d12 
>   
> ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UpgradeResourceProviderTest.java
>  014ab4223f 
>   
> ambari-server/src/test/java/org/apache/ambari/server/state/UpgradeHelperTest.java
>  921322b68c 
>   ambari-server/src/test/resources/hbase_version_test.xml 58c87015b5 
> 
> 
> Diff: https://reviews.apache.org/r/60055/diff/1/
> 
> 
> Testing
> ---
> 
> Manual.  Automated pending.
> 
> 
> Thanks,
> 
> Nate Cole
> 
>

Re: Review Request 59838: Stack advisor needs to recommend dependency for slaves and masters

[Tim Thorpe]

> On June 13, 2017, 7:09 p.m., Nate Cole wrote:
> > Ship It!

Hi Nate, still working on this to fix a deploy with UI issue will hopefully 
have another patch up soon.  Thanks


- Tim


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59838/#review177782
---


On June 7, 2017, 12:26 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59838/
> ---
> 
> (Updated June 7, 2017, 12:26 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Di Li, Jaimin Jetly, Jonathan 
> Hurley, Jayush Luniya, and Nate Cole.
> 
> 
> Bugs: AMBARI-20122
> https://issues.apache.org/jira/browse/AMBARI-20122
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> After resolution of AMBARI-19685, stack advisor validates if stack defined 
> dependency is not satisfied but recommendation API does not account for this.
> 
> Stack defined dependencies are service/component based and has a scope 
> CLUSTER|HOST.  
> 
> During recommendation the services to install have already been selected.  We 
> can't really utilize the cluster scope because either the dependent service 
> was selected or it was not.  If it was not selected it will be caught during 
> validation.  We can only recommend based on HOST scope.
> 
> This JIRA is also limited to only handling those which don't have conditional 
> dependencies.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 4a81dc6 
>   ambari-server/src/test/python/stacks/2.0.6/common/test_stack_advisor.py 
> 41c57f6 
> 
> 
> Diff: https://reviews.apache.org/r/59838/diff/3/
> 
> 
> Testing
> ---
> 
> Added new test case to test the recommendation of dependencies for slaves and 
> masters
> 
> mvn clean test -DskipSurefireTests
> 
> 
> New trunk without my change
> 
> Total run:1168
> Total errors:13
> Total failures:0
> 
> New trunk with my change
> 
> Total run:1169
> Total errors:13
> Total failures:0
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>

Review Request 60055: Support revert for patch upgrades

[Nate Cole]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60055/
---

Review request for Ambari, Alejandro Fernandez, Dmitro Lisnichenko, and 
Jonathan Hurley.


Bugs: AMBARI-21241
https://issues.apache.org/jira/browse/AMBARI-21241


Repository: ambari


Description
---

Entails adding an option to specify a previous upgrade to revert.  Checks to 
make sure it's only a patch.


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/checks/HostsRepositoryVersionCheck.java
 a66db3c008 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/PrereqCheckRequest.java
 c8c9f9ed2b 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/PreUpgradeCheckResourceProvider.java
 ea8fb372dc 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UpgradeResourceProvider.java
 60665f7b2b 
  
ambari-server/src/main/java/org/apache/ambari/server/orm/entities/UpgradeEntity.java
 43b2e088d7 
  
ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/FinalizeUpgradeAction.java
 e70f1ce8b2 
  
ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/UpdateDesiredStackAction.java
 8a4820d7f4 
  
ambari-server/src/main/java/org/apache/ambari/server/state/UpgradeContext.java 
3ecf64dc0c 
  ambari-server/src/main/resources/Ambari-DDL-Derby-CREATE.sql e34140593a 
  ambari-server/src/main/resources/Ambari-DDL-MySQL-CREATE.sql ce8736e62d 
  ambari-server/src/main/resources/Ambari-DDL-Oracle-CREATE.sql ace3738d2c 
  ambari-server/src/main/resources/Ambari-DDL-Postgres-CREATE.sql 5899179aba 
  ambari-server/src/main/resources/Ambari-DDL-SQLAnywhere-CREATE.sql 6164b2de5c 
  ambari-server/src/main/resources/Ambari-DDL-SQLServer-CREATE.sql 52d2b87c74 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/StackUpgradeConfigurationMergeTest.java
 a37e4f5d12 
  
ambari-server/src/test/java/org/apache/ambari/server/controller/internal/UpgradeResourceProviderTest.java
 014ab4223f 
  
ambari-server/src/test/java/org/apache/ambari/server/state/UpgradeHelperTest.java
 921322b68c 
  ambari-server/src/test/resources/hbase_version_test.xml 58c87015b5 


Diff: https://reviews.apache.org/r/60055/diff/1/


Testing
---

Manual.  Automated pending.


Thanks,

Nate Cole

Re: Review Request 59838: Stack advisor needs to recommend dependency for slaves and masters

[Nate Cole]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59838/#review177782
---


Ship it!




Ship It!

- Nate Cole


On June 7, 2017, 8:26 a.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59838/
> ---
> 
> (Updated June 7, 2017, 8:26 a.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Di Li, Jaimin Jetly, Jonathan 
> Hurley, Jayush Luniya, and Nate Cole.
> 
> 
> Bugs: AMBARI-20122
> https://issues.apache.org/jira/browse/AMBARI-20122
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> After resolution of AMBARI-19685, stack advisor validates if stack defined 
> dependency is not satisfied but recommendation API does not account for this.
> 
> Stack defined dependencies are service/component based and has a scope 
> CLUSTER|HOST.  
> 
> During recommendation the services to install have already been selected.  We 
> can't really utilize the cluster scope because either the dependent service 
> was selected or it was not.  If it was not selected it will be caught during 
> validation.  We can only recommend based on HOST scope.
> 
> This JIRA is also limited to only handling those which don't have conditional 
> dependencies.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 4a81dc6 
>   ambari-server/src/test/python/stacks/2.0.6/common/test_stack_advisor.py 
> 41c57f6 
> 
> 
> Diff: https://reviews.apache.org/r/59838/diff/3/
> 
> 
> Testing
> ---
> 
> Added new test case to test the recommendation of dependencies for slaves and 
> masters
> 
> mvn clean test -DskipSurefireTests
> 
> 
> New trunk without my change
> 
> Total run:1168
> Total errors:13
> Total failures:0
> 
> New trunk with my change
> 
> Total run:1169
> Total errors:13
> Total failures:0
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>

Re: Review Request 59992: AMBARI-21226. Add two new Spark thrift server related configurations in Spark stack

[Alejandro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59992/#review14
---


Ship it!




Ship It!

- Alejandro Fernandez


On June 13, 2017, 5:44 a.m., Saisai Shao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59992/
> ---
> 
> (Updated June 13, 2017, 5:44 a.m.)
> 
> 
> Review request for Ambari and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-21226
> https://issues.apache.org/jira/browse/AMBARI-21226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> AMBARI-21226. Add two new Spark thrift server related configurations in Spark 
> stack
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/configuration/spark-thrift-sparkconf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK2/configuration/spark2-thrift-sparkconf.xml
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/59992/diff/2/
> 
> 
> Testing
> ---
> 
> Test is done manually on:
> 
> updating HDP from 2.3 -> 2.6, 2.4 -> 2.6, 2.5 -> 2.6, and fresh install of 2.6
> 
> 
> Thanks,
> 
> Saisai Shao
> 
>

Re: Review Request 59951: AMBARI-21214 : Use a uuid vs long row key for metrics in AMS schema

[Dmytro Sen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59951/#review177762
---




ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/PhoenixHBaseAccessor.java
Line 480 (original), 484 (patched)


Let's remove this if it's not needed.



ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/PhoenixHBaseAccessor.java
Line 1197 (original), 1207 (patched)


Looks like the last parameter is always null and not needed anymore


- Dmytro Sen


On Июнь 9, 2017, 6:52 п.п., Aravindan Vijayan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59951/
> ---
> 
> (Updated Июнь 9, 2017, 6:52 п.п.)
> 
> 
> Review request for Ambari, Dmytro Sen, Sumit Mohanty, and Sid Wagle.
> 
> 
> Bugs: AMBARI-21214
> https://issues.apache.org/jira/browse/AMBARI-21214
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Presently AMS stores (metricName, hostname, serverTime, appId, instanceId) 
> for every row. Although this makes SQL query easy to write, it adds 
> duplication when storing it in the Regions memory. The plan is to have a 
> unique integer ID for an application-instance metric on a host and use that 
> as a lookup table with a write through cache when inserting or reading data 
> in the METRIC_RECORD table.
> 
> WORK DONE
> Added new logic for UUID management
> Integrated the UUID management into metadata manager. 
> Added UUID handling into write, read and aggregate paths. 
> 
> TODO
> Fix downsampling use case for UUID change.
> 
> 
> Diffs
> -
> 
>   
> ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/TimelineMetric.java
>  edace52 
>   
> ambari-metrics/ambari-metrics-common/src/main/java/org/apache/hadoop/metrics2/sink/timeline/TimelineMetricMetadata.java
>  727becc 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/HBaseTimelineMetricsService.java
>  f962f44 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/PhoenixHBaseAccessor.java
>  15b0bb8 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricConfiguration.java
>  de33bd1 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricStore.java
>  d052d54 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/TimelineMetricsFilter.java
>  1446ec2 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/AbstractTimelineAggregator.java
>  cb131d3 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineClusterMetric.java
>  b7d9110 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricAggregatorFactory.java
>  2eb3553 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricAppAggregator.java
>  9eaf456 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricClusterAggregator.java
>  74d4013 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricClusterAggregatorSecond.java
>  34b1f9b 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricHostAggregator.java
>  a17433b 
>   
> ambari-metrics/ambari-metrics-timelineservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/metrics/timeline/aggregators/TimelineMetricReadHelper.java
>  672f85f 
>   
> 

Review Request 60050: AMBARI-21240. Some topology request data retained after host removed

[Attila Doroszlai]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60050/
---

Review request for Ambari, Sandor Magyari, Sumit Mohanty, and Sebastian Toader.


Bugs: AMBARI-21240
https://issues.apache.org/jira/browse/AMBARI-21240


Repository: ambari


Description
---

Remove orphaned `topology_logical_request` and `topology_request`.


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClustersImpl.java
 e0a496509e9256d5ca550e7dd79ae0cd0fb5882b 
  
ambari-server/src/main/java/org/apache/ambari/server/topology/PersistedStateImpl.java
 a8b202e31b390f483d6ed1aee470179f519388a8 


Diff: https://reviews.apache.org/r/60050/diff/1/


Testing
---

Functional test:

 * up/downscale by 1 host (verify topology request is removed)
 * upscale by 2 hosts, downscale by 1 host (verify topology request is retained)
 * upscale by 2 hosts with 1 pending, downscale by 1 host (verify topology 
request is retained)
 * upscale by 3 hosts with 1 pending, remove pending host assignments (verify 
topology request is retained)
 * upscale by 3 hosts with 3 pending, remove pending host assignments (verify 
topology request is removed)
 * restart Ambari Server (verify no database error reported)

Unit tests:

```
Tests run: 4982, Failures: 0, Errors: 0, Skipped: 39
```


Thanks,

Attila Doroszlai

Re: Review Request 59992: AMBARI-21226. Add two new Spark thrift server related configurations in Spark stack

[Sumit Mohanty]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59992/#review177746
---



One more thing: If user upgrades from HDP-2.6.x to 2.6.2, do you want the 
configs to be added? If yes then that upgrade xml change is needed.

- Sumit Mohanty


On June 13, 2017, 5:44 a.m., Saisai Shao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59992/
> ---
> 
> (Updated June 13, 2017, 5:44 a.m.)
> 
> 
> Review request for Ambari and Sumit Mohanty.
> 
> 
> Bugs: AMBARI-21226
> https://issues.apache.org/jira/browse/AMBARI-21226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> AMBARI-21226. Add two new Spark thrift server related configurations in Spark 
> stack
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK/configuration/spark-thrift-sparkconf.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.6/services/SPARK2/configuration/spark2-thrift-sparkconf.xml
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/59992/diff/2/
> 
> 
> Testing
> ---
> 
> Test is done manually on:
> 
> updating HDP from 2.3 -> 2.6, 2.4 -> 2.6, 2.5 -> 2.6, and fresh install of 2.6
> 
> 
> Thanks,
> 
> Saisai Shao
> 
>

Re: Review Request 59954: Drop JDK 7 support for Ambari Server and Ambari managed services (AMS, LogSearch, Infra)

[Oliver Szabo]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59954/
---

(Updated June 13, 2017, 2:44 p.m.)


Review request for Ambari, Jonathan Hurley, Nate Cole, Robert Levas, Robert 
Nettleton, Sumit Mohanty, Sebastian Toader, and Sid Wagle.


Changes
---

remove steps (they have already done)


Bugs: AMBARI-21099
https://issues.apache.org/jira/browse/AMBARI-21099


Repository: ambari


Description (updated)
---

Requirements:
- ambari should drop JDK 7 support for ambari-server and ambari-managed 
services (infra, ams, logsearch), so force to use jdk8
- HDP 2.x should be still able to use JDK 1.7
Changes in ambari setup:
1. First the usual ambari setup

Choose JDK for Ambari and Stack services:
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
[3] Custom JDK

2. If JDK 7 is picked, that cannot be used for ambari (only for the stack 
services)

JDK 7 is detected (JDK 8 required for ambari-server), install JDK 8 as well:
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[3] Custom JDK


(this commands can be used is silent mode: ambari-server setup --java-home=... 
--stack-java-home=...)

After the commands are executed, the following properties would appear in 
ambari.properties

stack.java.home=/usr/jdk64/jdk1.7.0_67
stack.jdk.name=jdk-7u67linux-x64.tar.gz
stack.jce.name=UnlimitedJCEPolicyJDK7.zip

During command.json generating, commandParameters would be filled with proper 
ambari jdk,jce info (different for infra/logsearch/ams from other services) - 
not hostLevelParams, as multiple command could use the same hostLevelParams 
(like start all services: that can start zookeeper and ams as well with the 
same hostLevelParams)

On ambari-upgrade, ambari process wont be able to start with jdk7 so it will be 
required to run ambari-server setup first (you can skip the first change jdk 
part, but then the user will see its needed to setup a different jdk for ambari)

In the future, it will be recommended to do the JDK upgrade before upgrading to 
HDP 3.0 (to avoid some issues during HDP upgrade)

next step: 
- add function to ambari common to validate jdk version (inorder to fail 
starting any service which require jdk 8)

open question:
- there can be a more flexible solution to include different jdks in ambari 
(based on jdk releases list) and send all of them in the command.json, and 
chose the proper one for different services. also that can cause some issues 
with custom jdks (adding a name to custom ones?), and also because the jdk 
names can be anything, we would need to validate/get the jdk version during 
sending a command to the agent. all of this work has done about to support 
older version of HDP, so im not sure about to create a flexible solution at 
all, as managing multiple JDKs can be really hard for a customer, so i would 
not recommend to use multiple ones.


Diffs
-

  ambari-server/docs/configuration/index.md 2394264 
  
ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java
 60df8cf 
  
ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
 fb06e6d 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariActionExecutionHelper.java
 1b0e0e0 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java
 07f6e30 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
 8d262e2 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClientConfigResourceProvider.java
 21cf16c 
  ambari-server/src/main/java/org/apache/ambari/server/utils/StageUtils.java 
9930148 
  ambari-server/src/main/python/ambari-server.py d84e833 
  ambari-server/src/main/python/ambari_server/serverConfiguration.py 4780338 
  ambari-server/src/main/python/ambari_server/serverSetup.py c6de088 
  
ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/package/scripts/params.py
 6eb3ba8 
  
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py
 b8c14f4 
  
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py
 36c4598 
  
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py
 d26df33 
  
ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/params.py
 6738c5c 
  
ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py
 094d239 
  
ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py
 85eb796 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-ANY/scripts/params.py
 4052d1d 
  

Re: Review Request 59954: Drop JDK 7 support for Ambari Server and Ambari managed services (AMS, LogSearch, Infra)

[Oliver Szabo]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59954/
---

(Updated June 13, 2017, 2:38 p.m.)


Review request for Ambari, Jonathan Hurley, Nate Cole, Robert Levas, Robert 
Nettleton, Sumit Mohanty, Sebastian Toader, and Sid Wagle.


Changes
---

- add minJdk 1.8 to HDP 3.0 metainfo
- use ambari_java_home for solr_cloud_util commands in atlas/ranger below HDP 
3.0 (not adding to HDP 3.0 params.py)
- use ambari_java_home in ams/logsearch/infa


Bugs: AMBARI-21099
https://issues.apache.org/jira/browse/AMBARI-21099


Repository: ambari


Description
---

Requirements:
- ambari should drop JDK 7 support for ambari-server and ambari-managed 
services (infra, ams, logsearch), so force to use jdk8
- HDP 2.x should be still able to use JDK 1.7
Changes in ambari setup:
1. First the usual ambari setup

Choose JDK for Ambari and Stack services:
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
[3] Custom JDK

2. If JDK 7 is picked, that cannot be used for ambari (only for the stack 
services)

JDK 7 is detected (JDK 8 required for ambari-server), install JDK 8 as well:
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[3] Custom JDK


(this commands can be used is silent mode: ambari-server setup --java-home=... 
--stack-java-home=...)

After the commands are executed, the following properties would appear in 
ambari.properties

stack.java.home=/usr/jdk64/jdk1.7.0_67
stack.jdk.name=jdk-7u67linux-x64.tar.gz
stack.jce.name=UnlimitedJCEPolicyJDK7.zip

During command.json generating, commandParameters would be filled with proper 
ambari jdk,jce info (different for infra/logsearch/ams from other services) - 
not hostLevelParams, as multiple command could use the same hostLevelParams 
(like start all services: that can start zookeeper and ams as well with the 
same hostLevelParams)

On ambari-upgrade, ambari process wont be able to start with jdk7 so it will be 
required to run ambari-server setup first (you can skip the first change jdk 
part, but then the user will see its needed to setup a different jdk for ambari)

In the future, it will be recommended to do the JDK upgrade before upgrading to 
HDP 3.0 (to avoid some issues during HDP upgrade)

next step: 
- ams, logsearch, infra should use ambari_java_home (if it exists) 
- add function to ambari common to validate jdk version (inorder to fail 
starting any service which require jdk 8)
- make sure during HDP upgrade, minJdk and maxJdk is using the stack java 
versions, not ambari ones. (also add 1.8 as minJdk to HDP 3.0)
- there is one java version check in hive llap stack code, that should use 
stack.java.version (which i added), if that is included. (not ambari one - that 
will get the java.version from the ambari-server process, which can be invalid 
in that case)

open question:
- there can be a more flexible solution to include different jdks in ambari 
(based on jdk releases list) and send all of them in the command.json, and 
chose the proper one for different services. also that can cause some issues 
with custom jdks (adding a name to custom ones?), and also because the jdk 
names can be anything, we would need to validate/get the jdk version during 
sending a command to the agent. all of this work has done about to support 
older version of HDP, so im not sure about to create a flexible solution at 
all, as managing multiple JDKs can be really hard for a customer, so i would 
not recommend to use multiple ones.


Diffs (updated)
-

  ambari-server/docs/configuration/index.md 2394264 
  
ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java
 60df8cf 
  
ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
 fb06e6d 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariActionExecutionHelper.java
 1b0e0e0 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java
 07f6e30 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
 8d262e2 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClientConfigResourceProvider.java
 21cf16c 
  ambari-server/src/main/java/org/apache/ambari/server/utils/StageUtils.java 
9930148 
  ambari-server/src/main/python/ambari-server.py d84e833 
  ambari-server/src/main/python/ambari_server/serverConfiguration.py 4780338 
  ambari-server/src/main/python/ambari_server/serverSetup.py c6de088 
  
ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/package/scripts/params.py
 6eb3ba8 
  
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py
 b8c14f4 
  

Re: Review Request 59954: Drop JDK 7 support for Ambari Server and Ambari managed services (AMS, LogSearch, Infra)

[Oliver Szabo]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59954/
---

(Updated June 13, 2017, 2:42 p.m.)


Review request for Ambari, Jonathan Hurley, Nate Cole, Robert Levas, Robert 
Nettleton, Sumit Mohanty, Sebastian Toader, and Sid Wagle.


Changes
---

FT done


Bugs: AMBARI-21099
https://issues.apache.org/jira/browse/AMBARI-21099


Repository: ambari


Description
---

Requirements:
- ambari should drop JDK 7 support for ambari-server and ambari-managed 
services (infra, ams, logsearch), so force to use jdk8
- HDP 2.x should be still able to use JDK 1.7
Changes in ambari setup:
1. First the usual ambari setup

Choose JDK for Ambari and Stack services:
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
[3] Custom JDK

2. If JDK 7 is picked, that cannot be used for ambari (only for the stack 
services)

JDK 7 is detected (JDK 8 required for ambari-server), install JDK 8 as well:
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[3] Custom JDK


(this commands can be used is silent mode: ambari-server setup --java-home=... 
--stack-java-home=...)

After the commands are executed, the following properties would appear in 
ambari.properties

stack.java.home=/usr/jdk64/jdk1.7.0_67
stack.jdk.name=jdk-7u67linux-x64.tar.gz
stack.jce.name=UnlimitedJCEPolicyJDK7.zip

During command.json generating, commandParameters would be filled with proper 
ambari jdk,jce info (different for infra/logsearch/ams from other services) - 
not hostLevelParams, as multiple command could use the same hostLevelParams 
(like start all services: that can start zookeeper and ams as well with the 
same hostLevelParams)

On ambari-upgrade, ambari process wont be able to start with jdk7 so it will be 
required to run ambari-server setup first (you can skip the first change jdk 
part, but then the user will see its needed to setup a different jdk for ambari)

In the future, it will be recommended to do the JDK upgrade before upgrading to 
HDP 3.0 (to avoid some issues during HDP upgrade)

next step: 
- ams, logsearch, infra should use ambari_java_home (if it exists) 
- add function to ambari common to validate jdk version (inorder to fail 
starting any service which require jdk 8)
- make sure during HDP upgrade, minJdk and maxJdk is using the stack java 
versions, not ambari ones. (also add 1.8 as minJdk to HDP 3.0)
- there is one java version check in hive llap stack code, that should use 
stack.java.version (which i added), if that is included. (not ambari one - that 
will get the java.version from the ambari-server process, which can be invalid 
in that case)

open question:
- there can be a more flexible solution to include different jdks in ambari 
(based on jdk releases list) and send all of them in the command.json, and 
chose the proper one for different services. also that can cause some issues 
with custom jdks (adding a name to custom ones?), and also because the jdk 
names can be anything, we would need to validate/get the jdk version during 
sending a command to the agent. all of this work has done about to support 
older version of HDP, so im not sure about to create a flexible solution at 
all, as managing multiple JDKs can be really hard for a customer, so i would 
not recommend to use multiple ones.


Diffs
-

  ambari-server/docs/configuration/index.md 2394264 
  
ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java
 60df8cf 
  
ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
 fb06e6d 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariActionExecutionHelper.java
 1b0e0e0 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java
 07f6e30 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
 8d262e2 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClientConfigResourceProvider.java
 21cf16c 
  ambari-server/src/main/java/org/apache/ambari/server/utils/StageUtils.java 
9930148 
  ambari-server/src/main/python/ambari-server.py d84e833 
  ambari-server/src/main/python/ambari_server/serverConfiguration.py 4780338 
  ambari-server/src/main/python/ambari_server/serverSetup.py c6de088 
  
ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/package/scripts/params.py
 6eb3ba8 
  
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py
 b8c14f4 
  
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py
 36c4598 
  
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py
 d26df33 
  

Re: Review Request 60049: Ambari leaves a lot of processes from Hive alert on the machine

[Vitalyi Brodetskyi]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60049/#review177738
---


Ship it!




Ship It!

- Vitalyi Brodetskyi


On Червень 13, 2017, 1:57 після полудня, Andrew Onischuk wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60049/
> ---
> 
> (Updated Червень 13, 2017, 1:57 після полудня)
> 
> 
> Review request for Ambari and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-21239
> https://issues.apache.org/jira/browse/AMBARI-21239
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> I had a running Ambari instance on version 2.5.1.0-105. Unfortunately i lost
> the database VM storing the metadata after the cluster had been runing for
> almost 2 weeks without issues. What I observed is that the instance in a few
> hours got really slow. Upon investigation found hundreds of Beeline processes
> like this:
> 
> 
> 
> ambari-+ 3439770 3439769  0 06:19 ?00:00:00 -bash -c export  
> PATH='/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent:/bin/:/usr/bin/:/usr/lib/hive/bin/:/usr/sbin/'
>  ; ! beeline -u 
> 'jdbc:hive2://ctr-e133-1493418528701-31316-01-08.hwx.site:10500/;transportMode=binary'
>   -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL'
> ambari-+ 3439814 3439770  0 06:19 ?00:00:42 
> /usr/lib/jvm/java-openjdk/bin/java -Xmx1024m -Dhdp.version=2.6.1.0-64 
> -Djava.net.preferIPv4Stack=true -Dhdp.version=2.6.1.0-64 
> -Dhadoop.log.dir=/grid/0/log/hdfs/ambari-qa -Dhadoop.log.file=hadoop.log 
> -Dhadoop.home.dir=/usr/hdp/2.6.1.0-64/hadoop -Dhadoop.id.str=ambari-qa 
> -Dhadoop.root.logger=INFO,console 
> -Djava.library.path=:/usr/hdp/2.6.1.0-64/hadoop/lib/native/Linux-amd64-64:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-64/hadoop/lib/native
>  -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true 
> -Xmx1024m -Xmx1024m 
> -Djava.util.logging.config.file=/usr/hdp/2.6.1.0-64/hive/conf/parquet-logging.properties
>  -Dlog4j.configuration=beeline-log4j.properties 
> -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar 
> /usr/hdp/2.6.1.0-64/hive/lib/hive-beeline-1.2.1000.2.6.1.0-64.jar 
> org.apache.hive.beeline.BeeLine -u 
> jdbc:hive2://ctr-e133-1493418528701-31316-01-08.hwx.site:1050
 0/;transportMode=binary -e
> ambari-+ 3439816 3439770  0 06:19 ?00:00:00 awk {print}
> ambari-+ 3439817 3439770  0 06:19 ?00:00:00 grep -i -e Connection 
> refused -e Invalid URL
> ambari-+ 3440645 3440644  0 06:22 ?00:00:00 -bash -c export  
> PATH='/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent:/bin/:/usr/bin/:/usr/lib/hive/bin/:/usr/sbin/'
>  ; ! beeline -u 
> 'jdbc:hive2://ctr-e133-1493418528701-31316-01-08.hwx.site:10500/;transportMode=binary'
>   -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL'
> ambari-+ 3440685 3440645  0 06:22 ?00:00:42 
> /usr/lib/jvm/java-openjdk/bin/java -Xmx1024m -Dhdp.version=2.6.1.0-64 
> -Djava.net.preferIPv4Stack=true -Dhdp.version=2.6.1.0-64 
> -Dhadoop.log.dir=/grid/0/log/hdfs/ambari-qa -Dhadoop.log.file=hadoop.log 
> -Dhadoop.home.dir=/usr/hdp/2.6.1.0-64/hadoop -Dhadoop.id.str=ambari-qa 
> -Dhadoop.root.logger=INFO,console 
> -Djava.library.path=:/usr/hdp/2.6.1.0-64/hadoop/lib/native/Linux-amd64-64:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-64/hadoop/lib/native
>  -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true 
> -Xmx1024m -Xmx1024m 
> -Djava.util.logging.config.file=/usr/hdp/2.6.1.0-64/hive/conf/parquet-logging.properties
>  -Dlog4j.configuration=beeline-log4j.properties 
> -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar 
> /usr/hdp/2.6.1.0-64/hive/lib/hive-beeline-1.2.1000.2.6.1.0-64.jar 
> org.apache.hive.beeline.BeeLine -u 
> jdbc:hive2://ctr-e133-1493418528701-31316-01-08.hwx.site:1050
 0/;transportMode=binary -e
> ambari-+ 3440686 3440645  0 06:22 ?00:00:00 awk {print}
> ambari-+ 3440687 3440645  0 06:22 ?00:00:00 grep -i -e Connection 
> refused -e Invalid URL
> ambari-+ 3441185 3441184  0 06:25 ?00:00:00 -bash -c export  
> PATH='/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent:/bin/:/usr/bin/:/usr/lib/hive/bin/:/usr/sbin/'
>  ; ! beeline -u 
> 'jdbc:hive2://ctr-e133-1493418528701-31316-01-08.hwx.site:10500/;transportMode=binary'
>   -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL'
> ambari-+ 3441224 3441185  0 06:25 ?00:00:46 
> /usr/lib/jvm/java-openjdk/bin/java 

Review Request 60049: Ambari leaves a lot of processes from Hive alert on the machine

[Andrew Onischuk]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60049/
---

Review request for Ambari and Vitalyi Brodetskyi.


Bugs: AMBARI-21239
https://issues.apache.org/jira/browse/AMBARI-21239


Repository: ambari


Description
---

I had a running Ambari instance on version 2.5.1.0-105. Unfortunately i lost
the database VM storing the metadata after the cluster had been runing for
almost 2 weeks without issues. What I observed is that the instance in a few
hours got really slow. Upon investigation found hundreds of Beeline processes
like this:



ambari-+ 3439770 3439769  0 06:19 ?00:00:00 -bash -c export  
PATH='/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent:/bin/:/usr/bin/:/usr/lib/hive/bin/:/usr/sbin/'
 ; ! beeline -u 
'jdbc:hive2://ctr-e133-1493418528701-31316-01-08.hwx.site:10500/;transportMode=binary'
  -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL'
ambari-+ 3439814 3439770  0 06:19 ?00:00:42 
/usr/lib/jvm/java-openjdk/bin/java -Xmx1024m -Dhdp.version=2.6.1.0-64 
-Djava.net.preferIPv4Stack=true -Dhdp.version=2.6.1.0-64 
-Dhadoop.log.dir=/grid/0/log/hdfs/ambari-qa -Dhadoop.log.file=hadoop.log 
-Dhadoop.home.dir=/usr/hdp/2.6.1.0-64/hadoop -Dhadoop.id.str=ambari-qa 
-Dhadoop.root.logger=INFO,console 
-Djava.library.path=:/usr/hdp/2.6.1.0-64/hadoop/lib/native/Linux-amd64-64:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-64/hadoop/lib/native
 -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true 
-Xmx1024m -Xmx1024m 
-Djava.util.logging.config.file=/usr/hdp/2.6.1.0-64/hive/conf/parquet-logging.properties
 -Dlog4j.configuration=beeline-log4j.properties 
-Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar 
/usr/hdp/2.6.1.0-64/hive/lib/hive-beeline-1.2.1000.2.6.1.0-64.jar 
org.apache.hive.beeline.BeeLine -u 
jdbc:hive2://ctr-e133-1493418528701-31316-01-08.hwx.site:10500/
 ;transportMode=binary -e
ambari-+ 3439816 3439770  0 06:19 ?00:00:00 awk {print}
ambari-+ 3439817 3439770  0 06:19 ?00:00:00 grep -i -e Connection 
refused -e Invalid URL
ambari-+ 3440645 3440644  0 06:22 ?00:00:00 -bash -c export  
PATH='/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent:/bin/:/usr/bin/:/usr/lib/hive/bin/:/usr/sbin/'
 ; ! beeline -u 
'jdbc:hive2://ctr-e133-1493418528701-31316-01-08.hwx.site:10500/;transportMode=binary'
  -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL'
ambari-+ 3440685 3440645  0 06:22 ?00:00:42 
/usr/lib/jvm/java-openjdk/bin/java -Xmx1024m -Dhdp.version=2.6.1.0-64 
-Djava.net.preferIPv4Stack=true -Dhdp.version=2.6.1.0-64 
-Dhadoop.log.dir=/grid/0/log/hdfs/ambari-qa -Dhadoop.log.file=hadoop.log 
-Dhadoop.home.dir=/usr/hdp/2.6.1.0-64/hadoop -Dhadoop.id.str=ambari-qa 
-Dhadoop.root.logger=INFO,console 
-Djava.library.path=:/usr/hdp/2.6.1.0-64/hadoop/lib/native/Linux-amd64-64:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-64/hadoop/lib/native
 -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true 
-Xmx1024m -Xmx1024m 
-Djava.util.logging.config.file=/usr/hdp/2.6.1.0-64/hive/conf/parquet-logging.properties
 -Dlog4j.configuration=beeline-log4j.properties 
-Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar 
/usr/hdp/2.6.1.0-64/hive/lib/hive-beeline-1.2.1000.2.6.1.0-64.jar 
org.apache.hive.beeline.BeeLine -u 
jdbc:hive2://ctr-e133-1493418528701-31316-01-08.hwx.site:10500/
 ;transportMode=binary -e
ambari-+ 3440686 3440645  0 06:22 ?00:00:00 awk {print}
ambari-+ 3440687 3440645  0 06:22 ?00:00:00 grep -i -e Connection 
refused -e Invalid URL
ambari-+ 3441185 3441184  0 06:25 ?00:00:00 -bash -c export  
PATH='/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent:/bin/:/usr/bin/:/usr/lib/hive/bin/:/usr/sbin/'
 ; ! beeline -u 
'jdbc:hive2://ctr-e133-1493418528701-31316-01-08.hwx.site:10500/;transportMode=binary'
  -e '' 2>&1| awk '{print}'|grep -i -e 'Connection refused' -e 'Invalid URL'
ambari-+ 3441224 3441185  0 06:25 ?00:00:46 
/usr/lib/jvm/java-openjdk/bin/java -Xmx1024m -Dhdp.version=2.6.1.0-64 
-Djava.net.preferIPv4Stack=true -Dhdp.version=2.6.1.0-64 
-Dhadoop.log.dir=/grid/0/log/hdfs/ambari-qa -Dhadoop.log.file=hadoop.log 
-Dhadoop.home.dir=/usr/hdp/2.6.1.0-64/hadoop -Dhadoop.id.str=ambari-qa 
-Dhadoop.root.logger=INFO,console 
-Djava.library.path=:/usr/hdp/2.6.1.0-64/hadoop/lib/native/Linux-amd64-64:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:/usr/hdp/2.6.1.0-64/hadoop/lib/native
 -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true 
-Xmx1024m -Xmx1024m 

Re: Review Request 60048: Kafka userprincipal to shortname is not using AUTH_TO_LOCAL rules for authorization

[Dmitro Lisnichenko]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60048/#review177736
---


Ship it!




Ship It!

- Dmitro Lisnichenko


On June 13, 2017, 4:18 p.m., Eugene Chekanskiy wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60048/
> ---
> 
> (Updated June 13, 2017, 4:18 p.m.)
> 
> 
> Review request for Ambari, Dmitro Lisnichenko, Robert Levas, and Vitalyi 
> Brodetskyi.
> 
> 
> Bugs: AMBARI-21238
> https://issues.apache.org/jira/browse/AMBARI-21238
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> add comma separated rules to sasl.kerberos.principal.to.local.rules property 
> of server.properties
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AuthToLocalBuilder.java
>  33c8f3b 
>   
> ambari-server/src/main/resources/common-services/KAFKA/0.10.0.3.0/kerberos.json
>  eb31ad6 
>   ambari-server/src/main/resources/common-services/KAFKA/0.10.0/kerberos.json 
> eb31ad6 
>   ambari-server/src/main/resources/common-services/KAFKA/0.9.0/kerberos.json 
> 7500891 
>   
> contrib/management-packs/hdf-ambari-mpack/src/main/resources/stacks/HDF/2.0/services/KAFKA/kerberos.json
>  e1e6461 
> 
> 
> Diff: https://reviews.apache.org/r/60048/diff/1/
> 
> 
> Testing
> ---
> 
> mvn clean test, cluster deploy
> 
> 
> Thanks,
> 
> Eugene Chekanskiy
> 
>

Review Request 60048: Kafka userprincipal to shortname is not using AUTH_TO_LOCAL rules for authorization

[Eugene Chekanskiy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60048/
---

Review request for Ambari, Dmitro Lisnichenko, Robert Levas, and Vitalyi 
Brodetskyi.


Bugs: AMBARI-21238
https://issues.apache.org/jira/browse/AMBARI-21238


Repository: ambari


Description
---

add comma separated rules to sasl.kerberos.principal.to.local.rules property of 
server.properties


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/controller/AuthToLocalBuilder.java
 33c8f3b 
  
ambari-server/src/main/resources/common-services/KAFKA/0.10.0.3.0/kerberos.json 
eb31ad6 
  ambari-server/src/main/resources/common-services/KAFKA/0.10.0/kerberos.json 
eb31ad6 
  ambari-server/src/main/resources/common-services/KAFKA/0.9.0/kerberos.json 
7500891 
  
contrib/management-packs/hdf-ambari-mpack/src/main/resources/stacks/HDF/2.0/services/KAFKA/kerberos.json
 e1e6461 


Diff: https://reviews.apache.org/r/60048/diff/1/


Testing
---

mvn clean test, cluster deploy


Thanks,

Eugene Chekanskiy

Re: Review Request 59956: Update Database Access Layer to Support New Database Schema for Improved User Account Management

[Attila Magyar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59956/#review177734
---




ambari-server/src/main/java/org/apache/ambari/server/orm/entities/UserEntity.java
Lines 98 (patched)


Can't we just initialize Collections like this to empty collections 
(instead of null)? This would make most of the null checks unnecessary.


- Attila Magyar


On June 12, 2017, 5:15 p.m., Robert Levas wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59956/
> ---
> 
> (Updated June 12, 2017, 5:15 p.m.)
> 
> 
> Review request for Ambari, Attila Magyar, Balázs Bence Sári, Eugene 
> Chekanskiy, Jonathan Hurley, Laszlo Puskas, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-21147
> https://issues.apache.org/jira/browse/AMBARI-21147
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Update Database Access Layer to Support New Database Schema for Improved User 
> Account Management.  
> 
> * Update `org.apache.ambari.server.orm.entities.UserEntity`
> * Update `org.apache.ambari.server.orm.dao.UserDAO`
> * Add `org.apache.ambari.server.orm.entities.UserAuthenticationEntity`
> * Add `org.apache.ambari.server.orm.dao.UserAuthenticationDAO`
> 
> Note: Some changes will be revisited when updating the different 
> authentication processes to work with the improved user account management 
> code.
> 
> 
> Diffs
> -
> 
>   ambari-server/docs/api/generated/index.html 7ea4297b99 
>   ambari-server/docs/api/generated/swagger.json d7d54a510f 
>   
> ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
>  fb06e6d8a5 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementController.java
>  807bded873 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
>  eb64030e45 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariServer.java
>  aeba739a6d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/ControllerModule.java
>  f3c2ec871b 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/ResourceProviderFactory.java
>  391213858e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/UserRequest.java
>  40818c8f48 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/UserResponse.java
>  5afacb70ef 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/AbstractControllerResourceProvider.java
>  b35b2a8612 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ActiveWidgetLayoutResourceProvider.java
>  389f0b2bf2 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UserPrivilegeResourceProvider.java
>  614f7abda1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/UserResourceProvider.java
>  c5c36e9942 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/UserAuthenticationDAO.java
>  PRE-CREATION 
>   ambari-server/src/main/java/org/apache/ambari/server/orm/dao/UserDAO.java 
> ce47c4c38c 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/UserAuthenticationEntity.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/orm/entities/UserEntity.java
>  9011eaecec 
>   
> ambari-server/src/main/java/org/apache/ambari/server/security/authentication/AmbariJWTAuthenticationFilter.java
>  195c55afa5 
>   
> ambari-server/src/main/java/org/apache/ambari/server/security/authentication/AuthenticationMethodNotAllowedException.java
>  PRE-CREATION 
>   
> ambari-server/src/main/java/org/apache/ambari/server/security/authentication/kerberos/AmbariAuthToLocalUserDetailsService.java
>  1e4f6ead08 
>   
> ambari-server/src/main/java/org/apache/ambari/server/security/authentication/kerberos/AmbariKerberosAuthenticationProperties.java
>  09422e51e3 
>   
> ambari-server/src/main/java/org/apache/ambari/server/security/authorization/AmbariAuthorizationFilter.java
>  ce9a79023d 
>   
> ambari-server/src/main/java/org/apache/ambari/server/security/authorization/AmbariLdapAuthenticationProvider.java
>  b7ff297ce5 
>   
> ambari-server/src/main/java/org/apache/ambari/server/security/authorization/AmbariLdapAuthoritiesPopulator.java
>  d38d44c16f 
>   
> ambari-server/src/main/java/org/apache/ambari/server/security/authorization/AmbariLocalUserProvider.java
>  37d5d49c37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/security/authorization/AmbariPamAuthenticationProvider.java
>  373552e6e1 
>   
> 

Re: Review Request 60047: Principal placeholder replacement in kerberos.json doesn't work with blueprint installs

[Sebastian Toader]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60047/#review177733
---


Ship it!




Ship It!

- Sebastian Toader


On June 13, 2017, 2:42 p.m., Attila Magyar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60047/
> ---
> 
> (Updated June 13, 2017, 2:42 p.m.)
> 
> 
> Review request for Ambari, Balázs Bence Sári, Laszlo Puskas, Robert Levas, 
> and Sebastian Toader.
> 
> 
> Bugs: AMBARI-21236
> https://issues.apache.org/jira/browse/AMBARI-21236
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Kerberos principal placeholders like
> sasl:${principals/YARN/RESOURCEMANAGER/resource_manager_rm|principalPrimary()}:rwcda
> are not getting replaced if the cluster is created with blueprints 
> (kerberos=enabled).
> 
> I moved the logic that populates the replacemap with kerberos principals to a 
> different place.
> 
> 
> Diffs
> -
> 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
>  802c93d 
> 
> 
> Diff: https://reviews.apache.org/r/60047/diff/1/
> 
> 
> Testing
> ---
> 
> - created a cluster with blueprint (kerberos=enabled)
> - checked yarn-site.xml and zookeeper ACLs
> 
> existing tests: Tests run: 4982, Failures: 0, Errors: 0, Skipped: 39
> 
> 
> Thanks,
> 
> Attila Magyar
> 
>

Review Request 60047: Principal placeholder replacement in kerberos.json doesn't work with blueprint installs

[Attila Magyar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60047/
---

Review request for Ambari, Balázs Bence Sári, Laszlo Puskas, Robert Levas, and 
Sebastian Toader.


Bugs: AMBARI-21236
https://issues.apache.org/jira/browse/AMBARI-21236


Repository: ambari


Description
---

Kerberos principal placeholders like
sasl:${principals/YARN/RESOURCEMANAGER/resource_manager_rm|principalPrimary()}:rwcda
are not getting replaced if the cluster is created with blueprints 
(kerberos=enabled).

I moved the logic that populates the replacemap with kerberos principals to a 
different place.


Diffs
-

  
ambari-server/src/main/java/org/apache/ambari/server/controller/KerberosHelperImpl.java
 802c93d 


Diff: https://reviews.apache.org/r/60047/diff/1/


Testing
---

- created a cluster with blueprint (kerberos=enabled)
- checked yarn-site.xml and zookeeper ACLs

existing tests: Tests run: 4982, Failures: 0, Errors: 0, Skipped: 39


Thanks,

Attila Magyar

Re: Review Request 59954: Drop JDK 7 support for Ambari Server and Ambari managed services (AMS, LogSearch, Infra)

[Oliver Szabo]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59954/
---

(Updated June 13, 2017, 12:14 p.m.)


Review request for Ambari, Jonathan Hurley, Nate Cole, Robert Levas, Robert 
Nettleton, Sumit Mohanty, Sebastian Toader, and Sid Wagle.


Changes
---

removed unused List


Bugs: AMBARI-21099
https://issues.apache.org/jira/browse/AMBARI-21099


Repository: ambari


Description
---

Requirements:
- ambari should drop JDK 7 support for ambari-server and ambari-managed 
services (infra, ams, logsearch), so force to use jdk8
- HDP 2.x should be still able to use JDK 1.7
Changes in ambari setup:
1. First the usual ambari setup

Choose JDK for Ambari and Stack services:
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
[3] Custom JDK

2. If JDK 7 is picked, that cannot be used for ambari (only for the stack 
services)

JDK 7 is detected (JDK 8 required for ambari-server), install JDK 8 as well:
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[3] Custom JDK


(this commands can be used is silent mode: ambari-server setup --java-home=... 
--stack-java-home=...)

After the commands are executed, the following properties would appear in 
ambari.properties

stack.java.home=/usr/jdk64/jdk1.7.0_67
stack.jdk.name=jdk-7u67linux-x64.tar.gz
stack.jce.name=UnlimitedJCEPolicyJDK7.zip

During command.json generating, commandParameters would be filled with proper 
ambari jdk,jce info (different for infra/logsearch/ams from other services) - 
not hostLevelParams, as multiple command could use the same hostLevelParams 
(like start all services: that can start zookeeper and ams as well with the 
same hostLevelParams)

On ambari-upgrade, ambari process wont be able to start with jdk7 so it will be 
required to run ambari-server setup first (you can skip the first change jdk 
part, but then the user will see its needed to setup a different jdk for ambari)

In the future, it will be recommended to do the JDK upgrade before upgrading to 
HDP 3.0 (to avoid some issues during HDP upgrade)

next step: 
- ams, logsearch, infra should use ambari_java_home (if it exists) 
- add function to ambari common to validate jdk version (inorder to fail 
starting any service which require jdk 8)
- make sure during HDP upgrade, minJdk and maxJdk is using the stack java 
versions, not ambari ones. (also add 1.8 as minJdk to HDP 3.0)
- there is one java version check in hive llap stack code, that should use 
stack.java.version (which i added), if that is included. (not ambari one - that 
will get the java.version from the ambari-server process, which can be invalid 
in that case)

open question:
- there can be a more flexible solution to include different jdks in ambari 
(based on jdk releases list) and send all of them in the command.json, and 
chose the proper one for different services. also that can cause some issues 
with custom jdks (adding a name to custom ones?), and also because the jdk 
names can be anything, we would need to validate/get the jdk version during 
sending a command to the agent. all of this work has done about to support 
older version of HDP, so im not sure about to create a flexible solution at 
all, as managing multiple JDKs can be really hard for a customer, so i would 
not recommend to use multiple ones.


Diffs (updated)
-

  ambari-server/docs/configuration/index.md 2394264 
  
ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java
 60df8cf 
  
ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java
 fb06e6d 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariActionExecutionHelper.java
 1b0e0e0 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java
 07f6e30 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java
 8d262e2 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClientConfigResourceProvider.java
 21cf16c 
  ambari-server/src/main/java/org/apache/ambari/server/utils/StageUtils.java 
9930148 
  ambari-server/src/main/python/ambari-server.py d84e833 
  ambari-server/src/main/python/ambari_server/serverConfiguration.py 4780338 
  ambari-server/src/main/python/ambari_server/serverSetup.py c6de088 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-ANY/scripts/params.py
 4052d1d 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-ANY/scripts/shared_initialization.py
 1f17cd1 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/params.py
 3488e75 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/shared_initialization.py
 148d235 
  

Review Request 60045: Add validation and recommendation if hive user value is different than hive

[Vitalyi Brodetskyi]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60045/
---

Review request for Ambari, Andrew Onischuk, Dmytro Sen, Sumit Mohanty, and Sid 
Wagle.


Bugs: AMBARI-21237
https://issues.apache.org/jira/browse/AMBARI-21237


Repository: ambari


Description
---

By default yarn.nodemanager.kill-escape.user is set to hive the default 
username for Hive. When custom hive user is used the above property should be 
modified via stack advisor and also validation error must be reported if the 
value is not correct.


Diffs
-

  
ambari-server/src/main/resources/stacks/HDP/2.6/services/YARN/configuration/yarn-site.xml
 cab0e65 
  ambari-server/src/main/resources/stacks/HDP/2.6/services/stack_advisor.py 
9cd4acf 


Diff: https://reviews.apache.org/r/60045/diff/1/


Testing
---

mvn clean test


Thanks,

Vitalyi Brodetskyi

Re: Review Request 60007: Rename userconfig resource to event history in Log Search Portal

[Sebastian Toader]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60007/#review177728
---


Ship it!




Ship It!

- Sebastian Toader


On June 13, 2017, 12:24 p.m., Oliver Szabo wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60007/
> ---
> 
> (Updated June 13, 2017, 12:24 p.m.)
> 
> 
> Review request for Ambari, Attila Doroszlai, Miklos Gergely, Robert 
> Nettleton, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-21228
> https://issues.apache.org/jira/browse/AMBARI-21228
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Get rid of userconfig name, use history nd eventHistory instead of it. (its 
> valid to do that as filters is now on zookeeper)
> 
> 
> Diffs
> -
> 
>   
> ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/common/LogFeederConstants.java
>  a7559aa 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/conf/SecurityConfig.java
>  973dc4b 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/conf/SolrConfig.java
>  f00e8c5 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/conf/SolrEventHistoryPropsConfig.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/conf/SolrUserPropsConfig.java
>  c8c96c8 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/conf/global/SolrEventHistoryState.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/conf/global/SolrUserConfigState.java
>  06af14c 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/converter/EventHistoryRequestQueryConverter.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/converter/UserConfigRequestQueryConverter.java
>  594f23b 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/dao/EventHistorySolrDao.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/dao/SolrSchemaFieldDao.java
>  01378c1 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/dao/UserConfigSolrDao.java
>  a0e01a3 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/doc/DocConstants.java
>  885771d 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/manager/EventHistoryManager.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/manager/UserConfigManager.java
>  1df9f5a 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/model/request/EventHistoryParamDefinition.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/model/request/UserConfigParamDefinition.java
>  dd74b6f 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/model/request/impl/EventHistoryRequest.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/model/request/impl/UserConfigRequest.java
>  cbbc6e6 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/model/response/EventHistoryData.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/model/response/EventHistoryDataListResponse.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/model/response/UserConfigData.java
>  ef4e0be 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/model/response/UserConfigDataListResponse.java
>  5c445b2 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/rest/EventHistoryResource.java
>  PRE-CREATION 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/rest/StatusResource.java
>  8cc6b94 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/rest/UserConfigResource.java
>  00b971a 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/solr/SolrConstants.java
>  39e134a 
>   
> ambari-logsearch/ambari-logsearch-server/src/main/java/org/apache/ambari/logsearch/web/filters/LogsearchEventHistoryStateFilter.java
>  PRE-CREATION 
>   
> 

Re: Review Request 59929: Extend Swagger Maven pluging to handle nested API's

[Balázs Bence Sári]

> On June 12, 2017, 10:48 a.m., Attila Doroszlai wrote:
> > utility/src/main/java/org/apache/ambari/swagger/AmbariSwaggerReader.java
> > Lines 56 (patched)
> > 
> >
> > Should be `LOG`, not `logger`

Unfortunately, LOG denotes an existing member variable which is a (less 
sophisticated than slfj) Maven logger. I chose slf4j for logging, but I had to 
chose a different variable name.


> On June 12, 2017, 10:48 a.m., Attila Doroszlai wrote:
> > utility/src/main/java/org/apache/ambari/swagger/AmbariSwaggerReader.java
> > Lines 80 (patched)
> > 
> >
> > Please avoid + in `debug` message, use placeholders (`{}`) instead.

This is a reainder of switching logger implementations back and forth (slf4j vs 
maven logging). I'll fix it.


- Balázs Bence


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59929/#review177602
---


On June 12, 2017, 10:09 p.m., Balázs Bence Sári wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59929/
> ---
> 
> (Updated June 12, 2017, 10:09 p.m.)
> 
> 
> Review request for Ambari, Attila Doroszlai, Jaimin Jetly, Oliver Szabo, 
> Sumit Mohanty, and Sebastian Toader.
> 
> 
> Bugs: AMBARI-21207
> https://issues.apache.org/jira/browse/AMBARI-21207
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Wrote a custom SwaggerReader in the utility project that handles nested 
> API's. There is currently one nested API which is annotated for swagger: 
> org.apache.ambari.server.api.services.ServiceService (nested under 
> ClusterService). Using the custom reader operation paths and parameters are 
> generated correctly for the nested API.
> 
> 
> Diffs
> -
> 
>   ambari-project/pom.xml 64d1416 
>   ambari-server/pom.xml 669f90f 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/ClusterService.java
>  f61fb2a 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/ServiceService.java
>  a28c4aa 
>   utility/pom.xml 7d5eb93 
>   utility/src/main/java/org/apache/ambari/swagger/AmbariSwaggerReader.java 
> PRE-CREATION 
>   
> utility/src/test/java/org/apache/ambari/swagger/AmbariSwaggerReaderTest.java 
> PRE-CREATION 
>   utility/src/test/resources/log4j.properties PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/59929/diff/2/
> 
> 
> Testing
> ---
> 
> Tested Swagger API generation manually. Wrote new unit tests. Run all unit 
> tests for the ambari-server and utility projects, no errors or failures.
> 
> 
> Thanks,
> 
> Balázs Bence Sári
> 
>