Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Jayush Luniya]

> On March 2, 2016, 6:58 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/stacks/default_stack_advisor.py, line 35
> > 
> >
> > Please include Srimanth Gunturi in the code review, thanks!

+1 on adding Srimanth to the CR.


- Jayush


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review121687
---


On March 1, 2016, 4:53 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated March 1, 2016, 4:53 p.m.)
> 
> 
> Review request for Ambari, Sumit Mohanty and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-agent/pom.xml c2c993f 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/scripts/stack_advisor.py cdd9acb 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/GANGLIA/service_advisor_BIGTOP08GANGLIA.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HBASE/service_advisor_BIGTOP08HBASE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/service_advisor_BIGTOP08HDFS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HIVE/service_advisor_BIGTOP08HIVE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/OOZIE/service_advisor_BIGTOP08OOZIE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/YARN/service_advisor_BIGTOP08YARN.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/ZOOKEEPER/service_advisor_BIGTOP08ZOOKEEPER.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/stack_advisor.py 
> 53591cd 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/AMBARI_METRICS/service_advisor_HDP206AMBARI_METRICS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/GANGLIA/service_advisor_HDP206GANGLIA.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HBASE/service_advisor_HDP206HBASE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HDFS/service_advisor_HDP206HDFS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/service_advisor_HDP206HIVE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/OOZIE/service_advisor_HDP206OOZIE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/YARN/service_advisor_HDP206YARN.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/ZOOKEEPER/service_advisor_HDP206ZOOKEEPER.py
>  PRE-CREATION 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> f6f8cde 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1/services/FALCON/service_advisor_HDP21FALCON.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1/services/HIVE/service_advisor_HDP21HIVE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1/services/OOZIE/service_advisor_HDP21OOZIE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1/services/STORM/service_advisor_HDP21STORM.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/service_advisor_HDP21TEZ.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1/services/YARN/service_advisor_HDP21YARN.py
>  PRE-CREATION 
>   ambari-server/src/main/resources/stacks/HDP/2.1/services/stack_advisor.py 
> 3a552b8 
>   
> ambari-server/src/main/resources/stacks/HDP/2.2/services/AMBARI_METRICS/metainfo.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.2/services/AMBARI_METRICS/service_advisor_HDP22AMBARI_METRICS.py
>  PRE-CREATIO

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Jayush Luniya]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review127144
---



I dont think we should go down the route of breaking up the stack_advisor from 
stack-level to service-level at one shot. It is too risky a change. Instead we 
should support service-level advisors as extensions to stack_advisors so that 
add-on/custom services can extend the stack_advisors. Similar to how you 
handled RCO.

- Jayush Luniya


On March 1, 2016, 4:53 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated March 1, 2016, 4:53 p.m.)
> 
> 
> Review request for Ambari, Sumit Mohanty and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-agent/pom.xml c2c993f 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/scripts/stack_advisor.py cdd9acb 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/GANGLIA/service_advisor_BIGTOP08GANGLIA.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HBASE/service_advisor_BIGTOP08HBASE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/service_advisor_BIGTOP08HDFS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HIVE/service_advisor_BIGTOP08HIVE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/OOZIE/service_advisor_BIGTOP08OOZIE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/YARN/service_advisor_BIGTOP08YARN.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/ZOOKEEPER/service_advisor_BIGTOP08ZOOKEEPER.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/stack_advisor.py 
> 53591cd 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/AMBARI_METRICS/service_advisor_HDP206AMBARI_METRICS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/GANGLIA/service_advisor_HDP206GANGLIA.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HBASE/service_advisor_HDP206HBASE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HDFS/service_advisor_HDP206HDFS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/service_advisor_HDP206HIVE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/OOZIE/service_advisor_HDP206OOZIE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/YARN/service_advisor_HDP206YARN.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/ZOOKEEPER/service_advisor_HDP206ZOOKEEPER.py
>  PRE-CREATION 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> f6f8cde 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1/services/FALCON/service_advisor_HDP21FALCON.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1/services/HIVE/service_advisor_HDP21HIVE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1/services/OOZIE/service_advisor_HDP21OOZIE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1/services/STORM/service_advisor_HDP21STORM.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/service_advisor_HDP21TEZ.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1/services/YARN/service_advisor_HDP21YARN.py
>  PRE-CREATION 
>   ambari-server/src/main/resources/stacks/HDP/2.1/services/stack_advisor.py 
> 3a552b8 
>   
> ambari-server/src/main/resources/stacks/HDP/2.2/services/AMBARI_METRICS/metainfo.xml
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.2/services/AMBARI_METRICS/service_advisor_HDP22AMBARI_METRICS.py
>  P

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Jayush Luniya]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review127148
---




ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/GANGLIA/service_advisor_BIGTOP08GANGLIA.py
 (line 1)


Naming each service_advisor script named with stackname-version-servicename 
doesnt quite feel elegant. Simply service_advisor.py is what should be there at 
service-level. 

Also, we should do cross-stack references. When we bundle a stack in an 
Ambari release, we only bundle one-stack. So for BIGTOP, the HDP stack 
directory will not be present. Otherwise if you include both HDP and BIGTOP in 
the Ambari release, in the UI, you will see both BIGTOP and HDP stack options.


- Jayush Luniya


On March 1, 2016, 4:53 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated March 1, 2016, 4:53 p.m.)
> 
> 
> Review request for Ambari, Sumit Mohanty and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-agent/pom.xml c2c993f 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/scripts/stack_advisor.py cdd9acb 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/GANGLIA/service_advisor_BIGTOP08GANGLIA.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HBASE/service_advisor_BIGTOP08HBASE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/service_advisor_BIGTOP08HDFS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HIVE/service_advisor_BIGTOP08HIVE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/OOZIE/service_advisor_BIGTOP08OOZIE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/YARN/service_advisor_BIGTOP08YARN.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/ZOOKEEPER/service_advisor_BIGTOP08ZOOKEEPER.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/stack_advisor.py 
> 53591cd 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/AMBARI_METRICS/service_advisor_HDP206AMBARI_METRICS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/GANGLIA/service_advisor_HDP206GANGLIA.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HBASE/service_advisor_HDP206HBASE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HDFS/service_advisor_HDP206HDFS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/service_advisor_HDP206HIVE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/OOZIE/service_advisor_HDP206OOZIE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/YARN/service_advisor_HDP206YARN.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/ZOOKEEPER/service_advisor_HDP206ZOOKEEPER.py
>  PRE-CREATION 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> f6f8cde 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1/services/FALCON/service_advisor_HDP21FALCON.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1/services/HIVE/service_advisor_HDP21HIVE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1/services/OOZIE/service_advisor_HDP21OOZIE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1/services/STORM/service_advisor_HDP21STORM.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/service_advisor_HDP21TEZ.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1/services/YARN/service_advisor_HDP21YARN.py
>  PR

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Srimanth Gunturi]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review127197
---



Tim, thanks for contributing to the stack-advisor feature.

When we started the stack-advisor feature, there was a lot of interconnect 
between base service component-layout and configurations (HDFS, YARN, 
MapReduce, Hive, HBase) where common calculations were used by multiple 
services. To maximise readability and reuse of code, we took the 1 
stack-advisor/stack-version approach. This has kept the number of scripts 
managable (8 scripts for all versions of HDP stack) to maintain and debug when 
problems occur. It is very useful for stacks where a lot of interconnected 
services exist and stack-advisor should account for all the usecases.

Now, coming to this JIRA, it is undeniable that individual service folders 
should also be able to contribute to stack-advisor logic - it is useful when 
the service is not so dependent on other services and can independently provide 
its component-layout and configurations. Also, it is makes it easier to 
contribute a rich service by dropping a folder into the stack.

With that requirement said, I think a smaller change would be to provide a 
mechanism by which a service can contribute logic to the stack-version's 
stack-advisor. This would allow for both worlds of highly-interconnected and 
highly-independent services to coexist in the stack-advisor framework. For an 
independent service, it can provide a stack-advisor script which only dealt 
with it's component-layout and configuration - that can be overlaid on top of 
the stack-version's recommendations/validations.

My vote would be to have such an optional service-level stack-advisor script, 
which can be provided only when needed. We can discuss its design and how best 
to implement it.

- Srimanth Gunturi


On March 1, 2016, 4:53 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated March 1, 2016, 4:53 p.m.)
> 
> 
> Review request for Ambari, Sumit Mohanty and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-agent/pom.xml c2c993f 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/scripts/stack_advisor.py cdd9acb 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/GANGLIA/service_advisor_BIGTOP08GANGLIA.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HBASE/service_advisor_BIGTOP08HBASE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/service_advisor_BIGTOP08HDFS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HIVE/service_advisor_BIGTOP08HIVE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/OOZIE/service_advisor_BIGTOP08OOZIE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/YARN/service_advisor_BIGTOP08YARN.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/ZOOKEEPER/service_advisor_BIGTOP08ZOOKEEPER.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/stack_advisor.py 
> 53591cd 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/AMBARI_METRICS/service_advisor_HDP206AMBARI_METRICS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/GANGLIA/service_advisor_HDP206GANGLIA.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HBASE/service_advisor_HDP206HBASE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HDFS/service_advisor_HDP206HDFS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/service_advisor_HDP206HIVE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/OOZIE/service_advisor

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Tim Thorpe]

> On March 2, 2016, 9:37 p.m., jun aoki wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HBASE/service_advisor_HDP206HBASE.py,
> >  line 1
> > 
> >
> > Hi Tim, nice effort to decouple stack_advisor!
> > 
> > Is this file naming service_advisor_HDP206HBASE.py a bit redundant? 
> > this is under 
> > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HBASE, any files 
> > under this folder is meant for HDP, version 2.0.6 and HBASE.
> 
> Tim Thorpe wrote:
> I tried having the name as you stated but had a problem with python.  For 
> example in 2.3/services/HDFS, I have the following:
> 
> SCRIPT_DIR = os.path.dirname(os.path.abspath(file))
> PARENT_DIR = os.path.join(SCRIPT_DIR, '../../../2.2/services/HDFS/')
> sys.path.append(PARENT_DIR)
> from service_advisor_HDP22HDFS import *
> 
> If the file name is the same in 2.3 and 2.2, then "from service_advisor 
> import *" could reference the 2.2 or the 2.3 file.  I tried it with import 
> HDP22HDFSServiceAdvisor as well and both of them failed to find the correct 
> parent class.
> 
> The other way I could have solved this is to load all the service advisor 
> files from the default_stack_advisor.py walking my way up the stack.  This 
> would have allowed you to avoid the inheritance code in the 
> service_advisor.py files but it would restrict you from inheriting from 
> outside of your stack (unless at that point you started including the 
> inheritance code and the change of file names).
> 
> jun aoki wrote:
> I believe it has to treated as python packages so that file path matters.
> ```
> sys.path.append("/stacks")
> from HDP.2_2.services.HDFS import service_advisor # you'd probably have 
> to deal with "." in versions.
> ```

I have figured out a solution for this.  So the naming of the service advisor 
file would always be service_advisor.py without being prefaced with the stack 
name, version and service name.


- Tim


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review121720
---


On March 1, 2016, 4:53 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated March 1, 2016, 4:53 p.m.)
> 
> 
> Review request for Ambari, Sumit Mohanty and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-agent/pom.xml c2c993f 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/scripts/stack_advisor.py cdd9acb 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/GANGLIA/service_advisor_BIGTOP08GANGLIA.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HBASE/service_advisor_BIGTOP08HBASE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/service_advisor_BIGTOP08HDFS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HIVE/service_advisor_BIGTOP08HIVE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/OOZIE/service_advisor_BIGTOP08OOZIE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/YARN/service_advisor_BIGTOP08YARN.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/ZOOKEEPER/service_advisor_BIGTOP08ZOOKEEPER.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/stack_advisor.py 
> 53591cd 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/AMBARI_METRICS/service_advisor_HDP206AMBARI_METRICS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/GANGLIA/service_advisor_HDP206GANGLIA.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HBASE/servi

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Tim Thorpe]

> On March 2, 2016, 4:56 a.m., Matt wrote:
> > ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py,
> >  line 1
> > 
> >
> > If I were to add HAWQ 2.0.0's metainfo.xml under HDP 2.3 stack, and say 
> > that it extends from common-services, will this service_advisor be honored?
> 
> Tim Thorpe wrote:
> I had some difficulty with HAWQ and PXF because they are not actually in 
> the stack and can be added by installing a separate RPM.  In order for HAWQ 
> and PXF to inherit the service_advisor.py files from common-services, they 
> would need to implement mostly empty service advisors under the 
> //services/ folder that explicitly inherit from the 
> ones in common-services.
> 
> Matt wrote:
> Is there a way we can avoid having this empty service_advisor?
> 
> service X in common-services has a service_advisor
> service X in HDP 2.2 has no service_advisor - should inherit from 
> common-services
> service X in HDP 2.3 has service_advisor - should inherit from 
> common-services + add new recommendations/validations from HDP 2.3 X's 
> service_advisor
> 
> Something similar to have themes can be managed or how metainfo.xml has 
> extends property for stack definiton

Hi Matt, I can add the service advisor in the code that reads the 
stack/services from the file system.  This would allow me to know which 
service_advisor.py file should be used (either because it is inherited or 
because it is defined in the current stack version).  This information would 
then need to be passed to the stack advisor in the services.json.  I don't 
think the service advisor should handle implicite inheritance between stack 
versions by that I mean in that your example:

service X in common-services has a service_advisor
service X in HDP 2.2 has no service_advisor - should inherit from 
common-services
service X in HDP 2.3 has service_advisor - only inherits from common-services 
if the service_advisor.py adds that logic 

Basically this means that service X in HDP 2.3 would have 2 options to replace 
the service_advisor.py in common-services or inherit from it and add new 
recommendations/validations.


> On March 2, 2016, 4:56 a.m., Matt wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/GANGLIA/service_advisor_HDP206GANGLIA.py,
> >  line 1
> > 
> >
> > If the service_advisor is under HDP 2.0.6's GANGLIA service folder, it 
> > is not required to repeat the stack name, the version and the service on 
> > the filename, for every combination of stack, version and service.
> > 
> > This would be good enough:
> > HDP/2.0.6/services/GANGLIA/service_advisor.py
> 
> Tim Thorpe wrote:
> I tried having the name as you stated but had a problem with python.  For 
> example in 2.3/services/HDFS, I have the following:
> 
> SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
> PARENT_DIR = os.path.join(SCRIPT_DIR, '../../../2.2/services/HDFS/')
> sys.path.append(PARENT_DIR)
> from service_advisor_HDP22HDFS import *
> 
> If the file name is the same in 2.3 and 2.2, then "from service_advisor 
> import *" could reference the 2.2 or the 2.3 file.  I tried it with import 
> HDP22HDFSServiceAdvisor as well and both of them failed to find the correct 
> parent class.
> 
> The other way I could have solved this is to load all the service advisor 
> files from the default_stack_advisor.py walking my way up the stack.  This 
> would have allowed you to avoid the inheritance code in the 
> service_advisor.py files but it would restrict you from inheriting from 
> outside of your stack (unless at that point you started including the 
> inheritance code and the change of file names).

I have figured out a solution for this.  So the naming of the service advisor 
file would always be service_advisor.py without being prefaced with the stack 
name, version and service name.


> On March 2, 2016, 4:56 a.m., Matt wrote:
> > ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py, line 
> > 22
> > 
> >
> > If all the stack_advisor.py files are being refactored into service 
> > level advisors, the unit tests should be at service level as well.
> > 
> > This would bring better modularity for introducing a new service:
> > - Add a new service to the stack
> > - Implement the service advisor for that service
> > - Implement unit test for the service advisor for that service (For the 
> > current patch, the tests are still with stack advisor and not service 
> > advisor)
> 
> Tim Thorpe wrote:
> I agree but hopefully that can be a future enhancement?  I'd rather get 
> this included and then work on refactoring the test code.

The stack_advisor.py is no longer 

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Tim Thorpe]

> On April 5, 2016, 5:32 p.m., Jayush Luniya wrote:
> > ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/GANGLIA/service_advisor_BIGTOP08GANGLIA.py,
> >  line 1
> > 
> >
> > Naming each service_advisor script named with 
> > stackname-version-servicename doesnt quite feel elegant. Simply 
> > service_advisor.py is what should be there at service-level. 
> > 
> > Also, we should do cross-stack references. When we bundle a stack in an 
> > Ambari release, we only bundle one-stack. So for BIGTOP, the HDP stack 
> > directory will not be present. Otherwise if you include both HDP and BIGTOP 
> > in the Ambari release, in the UI, you will see both BIGTOP and HDP stack 
> > options.

I have figured out a solution for this.  So the naming of the service advisor 
file would always be service_advisor.py without being prefaced with the stack 
name, version and service name.

Since the stack advisors are no longer being refactored, we will only allow for 
service advisors to be plugged in, there will be no dependence of BIGTOP to HDP.


- Tim


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review127148
---


On March 1, 2016, 4:53 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated March 1, 2016, 4:53 p.m.)
> 
> 
> Review request for Ambari, Sumit Mohanty and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-agent/pom.xml c2c993f 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/scripts/stack_advisor.py cdd9acb 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/GANGLIA/service_advisor_BIGTOP08GANGLIA.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HBASE/service_advisor_BIGTOP08HBASE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/service_advisor_BIGTOP08HDFS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HIVE/service_advisor_BIGTOP08HIVE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/OOZIE/service_advisor_BIGTOP08OOZIE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/YARN/service_advisor_BIGTOP08YARN.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/ZOOKEEPER/service_advisor_BIGTOP08ZOOKEEPER.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/stack_advisor.py 
> 53591cd 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/AMBARI_METRICS/service_advisor_HDP206AMBARI_METRICS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/GANGLIA/service_advisor_HDP206GANGLIA.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HBASE/service_advisor_HDP206HBASE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HDFS/service_advisor_HDP206HDFS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/service_advisor_HDP206HIVE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/OOZIE/service_advisor_HDP206OOZIE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/YARN/service_advisor_HDP206YARN.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/ZOOKEEPER/service_advisor_HDP206ZOOKEEPER.py
>  PRE-CREATION 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> f6f8cde 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1/services/FALCON/service_advisor_HDP21FALCON.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1/services/HIVE/service_advisor_HDP21HIVE.p

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Tim Thorpe]

> On April 5, 2016, 7:53 p.m., Srimanth Gunturi wrote:
> > Tim, thanks for contributing to the stack-advisor feature.
> > 
> > When we started the stack-advisor feature, there was a lot of interconnect 
> > between base service component-layout and configurations (HDFS, YARN, 
> > MapReduce, Hive, HBase) where common calculations were used by multiple 
> > services. To maximise readability and reuse of code, we took the 1 
> > stack-advisor/stack-version approach. This has kept the number of scripts 
> > managable (8 scripts for all versions of HDP stack) to maintain and debug 
> > when problems occur. It is very useful for stacks where a lot of 
> > interconnected services exist and stack-advisor should account for all the 
> > usecases.
> > 
> > Now, coming to this JIRA, it is undeniable that individual service folders 
> > should also be able to contribute to stack-advisor logic - it is useful 
> > when the service is not so dependent on other services and can 
> > independently provide its component-layout and configurations. Also, it is 
> > makes it easier to contribute a rich service by dropping a folder into the 
> > stack.
> > 
> > With that requirement said, I think a smaller change would be to provide a 
> > mechanism by which a service can contribute logic to the stack-version's 
> > stack-advisor. This would allow for both worlds of highly-interconnected 
> > and highly-independent services to coexist in the stack-advisor framework. 
> > For an independent service, it can provide a stack-advisor script which 
> > only dealt with it's component-layout and configuration - that can be 
> > overlaid on top of the stack-version's recommendations/validations.
> > 
> > My vote would be to have such an optional service-level stack-advisor 
> > script, which can be provided only when needed. We can discuss its design 
> > and how best to implement it.

Hi Srimanth, we had a meeting yesterday discussing this and that was the 
consensus.  Service advisors will be able to plug into the stack advisor code.  
It will require some changes to the stacks/stack_advisor.py.

a) For the recommendConfigGroupsConfigurations and recommendConfigurations 
functions, the code for looping through services will need to check for service 
advisors:

  for service in servicesList:
calculation = self.getServiceConfigurationRecommender(service)
if calculation is not None:
  calculation(configurations, clusterSummary, services, hosts)
else:
  advisor = createServiceAdvisor(service)
  if advisor is not None:
advisor.recommendConfigurations(...)

b) For createComponentLayoutRecommendations, the code which processes the 
master and slave services will need to be moved into separate functions: 
getHostsForMasterComponent and getHostsForSlaveComponent.  This will allow 
stack versions to overwrite the details of how those recommendations take place 
without messing up the code which processes service advisors.  The 
createComponentLayoutRecommendations would end up looking something like:

  advisor = createServiceAdvisor(service)
  if advisor is not None:
advisor.getHostsForMasterComponent(...)
  else
self.getHostsForMasterComponent(...)

c) Change def getComponentLayoutValidations from a no-op to the following:

  def getComponentLayoutValidations(self, services, hosts):

servicesList = [service["StackServices"]["service_name"] for service in 
services["services"]]
for service in servicesList:
  advisor = createServiceAdvisor(service)
  if advisor is not None:
advisor.getComponentLayoutValidations(...)

This also requires the HDP 2.0.6 stack advisor to call 
super.getComponentLayoutValidations

d) Change def getConfigurationsValidationItems from a no-op to the following:

  def getConfigurationsValidationItems(self, services, hosts):

servicesList = [service["StackServices"]["service_name"] for service in 
services["services"]]
for service in servicesList:
  advisor = createServiceAdvisor(service)
  if advisor is not None:
advisor.getConfigurationsValidationItems(...)

This also requires the HDP 2.0.6 stack advisor to call 
super.getConfigurationsValidationItems

I'm sure there are also a number of lesser functions which would need to handle 
things differently.  

One thing I am still struggling with is whether I should create a default 
service advisor class.  The problem with this class is that it would mostly 
duplicate a bunch of functions which are defined in the default stack advisor 
class.  There are basically two options:

1) Create the default service advisor class and live with the duplication (and 
the need to make changes in multiple places)

2) Inherit the custom service advisor classes from the default stack advisor 
and live with the fact that a service is inheriting from the stack.  
(See one of the comments above by Jun Aoki)

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Tim Thorpe]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/
---

(Updated April 12, 2016, 5:33 p.m.)


Review request for Ambari, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.


Bugs: AMBARI-15226
https://issues.apache.org/jira/browse/AMBARI-15226


Repository: ambari


Description
---

Currently the stack advisor is defined under each stack version such as 
HDP/2.3. The problem with this is that it restricts the services that can be 
added to the stack. If a custom service is to be added, they would need to 
modify the stack advisor. If the configuration recommendation and validation 
can be done at the service level then the custom service could just include 
their own recommendations and validations separately.


Diffs
-

  ambari-agent/pom.xml c2c993f 
  ambari-server/src/main/assemblies/server.xml e1a4919 
  
ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py 
PRE-CREATION 
  ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
PRE-CREATION 
  ambari-server/src/main/resources/scripts/stack_advisor.py cdd9acb 
  
ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/GANGLIA/service_advisor_BIGTOP08GANGLIA.py
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HBASE/service_advisor_BIGTOP08HBASE.py
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/service_advisor_BIGTOP08HDFS.py
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HIVE/service_advisor_BIGTOP08HIVE.py
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/OOZIE/service_advisor_BIGTOP08OOZIE.py
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/YARN/service_advisor_BIGTOP08YARN.py
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/ZOOKEEPER/service_advisor_BIGTOP08ZOOKEEPER.py
 PRE-CREATION 
  ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/stack_advisor.py 
53591cd 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/AMBARI_METRICS/service_advisor_HDP206AMBARI_METRICS.py
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/GANGLIA/service_advisor_HDP206GANGLIA.py
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HBASE/service_advisor_HDP206HBASE.py
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HDFS/service_advisor_HDP206HDFS.py
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/service_advisor_HDP206HIVE.py
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/OOZIE/service_advisor_HDP206OOZIE.py
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/YARN/service_advisor_HDP206YARN.py
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.0.6/services/ZOOKEEPER/service_advisor_HDP206ZOOKEEPER.py
 PRE-CREATION 
  ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
f6f8cde 
  
ambari-server/src/main/resources/stacks/HDP/2.1/services/FALCON/service_advisor_HDP21FALCON.py
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.1/services/HIVE/service_advisor_HDP21HIVE.py
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.1/services/OOZIE/service_advisor_HDP21OOZIE.py
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.1/services/STORM/service_advisor_HDP21STORM.py
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/service_advisor_HDP21TEZ.py
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.1/services/YARN/service_advisor_HDP21YARN.py
 PRE-CREATION 
  ambari-server/src/main/resources/stacks/HDP/2.1/services/stack_advisor.py 
3a552b8 
  
ambari-server/src/main/resources/stacks/HDP/2.2/services/AMBARI_METRICS/metainfo.xml
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.2/services/AMBARI_METRICS/service_advisor_HDP22AMBARI_METRICS.py
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.2/services/HBASE/service_advisor_HDP22HBASE.py
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.2/services/HDFS/service_advisor_HDP22HDFS.py
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.2/services/HIVE/service_advisor_HDP22HIVE.py
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.2/services/KAFKA/service_advisor_HDP22KAFKA.py
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.2/services/KNOX/service_advisor_HDP22KNOX.py
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.2/services/RANGER/service_advisor_HDP22RANGER.py
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.2/services/STORM/service_advisor_HDP22STORM.py
 PRE-CREATION 
  
ambari-server/src/main/resources/stacks/HDP/2.2/servic

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Tim Thorpe]

> On March 2, 2016, 6:58 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/stacks/default_stack_advisor.py, line 35
> > 
> >
> > Please include Srimanth Gunturi in the code review, thanks!
> 
> Jayush Luniya wrote:
> +1 on adding Srimanth to the CR.

Could you please recommend who could be added for HAWQ and PXF to review those 
specific changes?


- Tim


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review121687
---


On April 12, 2016, 5:33 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 12, 2016, 5:33 p.m.)
> 
> 
> Review request for Ambari, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-agent/pom.xml c2c993f 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/scripts/stack_advisor.py cdd9acb 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/GANGLIA/service_advisor_BIGTOP08GANGLIA.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HBASE/service_advisor_BIGTOP08HBASE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/service_advisor_BIGTOP08HDFS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HIVE/service_advisor_BIGTOP08HIVE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/OOZIE/service_advisor_BIGTOP08OOZIE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/YARN/service_advisor_BIGTOP08YARN.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/ZOOKEEPER/service_advisor_BIGTOP08ZOOKEEPER.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/stack_advisor.py 
> 53591cd 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/AMBARI_METRICS/service_advisor_HDP206AMBARI_METRICS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/GANGLIA/service_advisor_HDP206GANGLIA.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HBASE/service_advisor_HDP206HBASE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HDFS/service_advisor_HDP206HDFS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/service_advisor_HDP206HIVE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/OOZIE/service_advisor_HDP206OOZIE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/YARN/service_advisor_HDP206YARN.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/ZOOKEEPER/service_advisor_HDP206ZOOKEEPER.py
>  PRE-CREATION 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> f6f8cde 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1/services/FALCON/service_advisor_HDP21FALCON.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1/services/HIVE/service_advisor_HDP21HIVE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1/services/OOZIE/service_advisor_HDP21OOZIE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1/services/STORM/service_advisor_HDP21STORM.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1/services/TEZ/service_advisor_HDP21TEZ.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1/services/YARN/service_advisor_HDP21YARN.py
>  PRE-CREATION 
>   ambari-server/src/main/resources/stacks/HDP/2.1/services/stack_advisor.py 
> 3a552b8 
>   
> ambari-server/src/main/resources/stacks/HDP/2.2/services/AMBARI_METRICS/metainfo.xml
>

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Matt]

> On March 2, 2016, 1:37 p.m., jun aoki wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HBASE/service_advisor_HDP206HBASE.py,
> >  line 32
> > 
> >
> > Could this inheritate a brand new DefaultServiceAdvisor instead of the 
> > stack advisor? service and stack are different hierarchy and I feel somehow 
> > wrong if a service inheritate a stack.
> 
> Tim Thorpe wrote:
> Point well taken. I'll look at refactoring it that way but really would 
> prefer to get the code in the current way first.  There are code changes to 
> the stack advisor files on a daily basis and the longer it waits the more 
> merges will be needed.  I will already be refactoring the code to split the 
> test scripts to be on the service level rather than the stack level.  I will 
> attempt to do both of these tasks at the same time.

May be this should inherit from service_advisor for that service under 
common-services?


- Matt


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review121720
---


On April 12, 2016, 10:33 a.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 12, 2016, 10:33 a.m.)
> 
> 
> Review request for Ambari, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-agent/pom.xml c2c993f 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/scripts/stack_advisor.py cdd9acb 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/GANGLIA/service_advisor_BIGTOP08GANGLIA.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HBASE/service_advisor_BIGTOP08HBASE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/service_advisor_BIGTOP08HDFS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HIVE/service_advisor_BIGTOP08HIVE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/OOZIE/service_advisor_BIGTOP08OOZIE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/YARN/service_advisor_BIGTOP08YARN.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/ZOOKEEPER/service_advisor_BIGTOP08ZOOKEEPER.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/stack_advisor.py 
> 53591cd 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/AMBARI_METRICS/service_advisor_HDP206AMBARI_METRICS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/GANGLIA/service_advisor_HDP206GANGLIA.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HBASE/service_advisor_HDP206HBASE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HDFS/service_advisor_HDP206HDFS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/service_advisor_HDP206HIVE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/OOZIE/service_advisor_HDP206OOZIE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/YARN/service_advisor_HDP206YARN.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/ZOOKEEPER/service_advisor_HDP206ZOOKEEPER.py
>  PRE-CREATION 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> f6f8cde 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1/services/FALCON/service_advisor_HDP21FALCON.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1/services/HIVE/service_advisor_HDP21HIVE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1/services/OOZIE/service_adviso

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Tim Thorpe]

> On March 2, 2016, 9:37 p.m., jun aoki wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HBASE/service_advisor_HDP206HBASE.py,
> >  line 32
> > 
> >
> > Could this inheritate a brand new DefaultServiceAdvisor instead of the 
> > stack advisor? service and stack are different hierarchy and I feel somehow 
> > wrong if a service inheritate a stack.
> 
> Tim Thorpe wrote:
> Point well taken. I'll look at refactoring it that way but really would 
> prefer to get the code in the current way first.  There are code changes to 
> the stack advisor files on a daily basis and the longer it waits the more 
> merges will be needed.  I will already be refactoring the code to split the 
> test scripts to be on the service level rather than the stack level.  I will 
> attempt to do both of these tasks at the same time.
> 
> Matt wrote:
> May be this should inherit from service_advisor for that service under 
> common-services?

Hi Matt, so you are recommending that the resources/stacks/service_advisor.py 
be moved to resources/common-services/service_advisor.py?


- Tim


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review121720
---


On April 12, 2016, 5:33 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 12, 2016, 5:33 p.m.)
> 
> 
> Review request for Ambari, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-agent/pom.xml c2c993f 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/scripts/stack_advisor.py cdd9acb 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/GANGLIA/service_advisor_BIGTOP08GANGLIA.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HBASE/service_advisor_BIGTOP08HBASE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/service_advisor_BIGTOP08HDFS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HIVE/service_advisor_BIGTOP08HIVE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/OOZIE/service_advisor_BIGTOP08OOZIE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/YARN/service_advisor_BIGTOP08YARN.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/ZOOKEEPER/service_advisor_BIGTOP08ZOOKEEPER.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/stack_advisor.py 
> 53591cd 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/AMBARI_METRICS/service_advisor_HDP206AMBARI_METRICS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/GANGLIA/service_advisor_HDP206GANGLIA.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HBASE/service_advisor_HDP206HBASE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HDFS/service_advisor_HDP206HDFS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/service_advisor_HDP206HIVE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/OOZIE/service_advisor_HDP206OOZIE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/YARN/service_advisor_HDP206YARN.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/ZOOKEEPER/service_advisor_HDP206ZOOKEEPER.py
>  PRE-CREATION 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> f6f8cde 
>   
> ambari-server/src/main/resources/stacks/HDP/2.1/services/FALCON/service_advisor_HDP21FALCON.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Tim Thorpe]

> On March 2, 2016, 9:37 p.m., jun aoki wrote:
> > ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HBASE/service_advisor_HDP206HBASE.py,
> >  line 32
> > 
> >
> > Could this inheritate a brand new DefaultServiceAdvisor instead of the 
> > stack advisor? service and stack are different hierarchy and I feel somehow 
> > wrong if a service inheritate a stack.
> 
> Tim Thorpe wrote:
> Point well taken. I'll look at refactoring it that way but really would 
> prefer to get the code in the current way first.  There are code changes to 
> the stack advisor files on a daily basis and the longer it waits the more 
> merges will be needed.  I will already be refactoring the code to split the 
> test scripts to be on the service level rather than the stack level.  I will 
> attempt to do both of these tasks at the same time.
> 
> Matt wrote:
> May be this should inherit from service_advisor for that service under 
> common-services?
> 
> Tim Thorpe wrote:
> Hi Matt, so you are recommending that the 
> resources/stacks/service_advisor.py be moved to 
> resources/common-services/service_advisor.py?

Probably a more logical place to put it would be under resources/scripts


- Tim


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review121720
---


On April 12, 2016, 5:33 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 12, 2016, 5:33 p.m.)
> 
> 
> Review request for Ambari, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-agent/pom.xml c2c993f 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/scripts/stack_advisor.py cdd9acb 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/GANGLIA/service_advisor_BIGTOP08GANGLIA.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HBASE/service_advisor_BIGTOP08HBASE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HDFS/service_advisor_BIGTOP08HDFS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/HIVE/service_advisor_BIGTOP08HIVE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/OOZIE/service_advisor_BIGTOP08OOZIE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/YARN/service_advisor_BIGTOP08YARN.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/ZOOKEEPER/service_advisor_BIGTOP08ZOOKEEPER.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/BIGTOP/0.8/services/stack_advisor.py 
> 53591cd 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/AMBARI_METRICS/service_advisor_HDP206AMBARI_METRICS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/GANGLIA/service_advisor_HDP206GANGLIA.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HBASE/service_advisor_HDP206HBASE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HDFS/service_advisor_HDP206HDFS.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/HIVE/service_advisor_HDP206HIVE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/OOZIE/service_advisor_HDP206OOZIE.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/YARN/service_advisor_HDP206YARN.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/stacks/HDP/2.0.6/services/ZOOKEEPER/service_advisor_HDP206ZOOKEEPER.py
>  PRE-CREATION 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> f6f8cde 
>   
> ambari-server/src/main/resources/stacks/HDP/2

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Tim Thorpe]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/
---

(Updated April 22, 2016, 4:39 p.m.)


Review request for Ambari, Jayush Luniya, Sumit Mohanty, Srimanth Gunturi, and 
Yusaku Sako.


Bugs: AMBARI-15226
https://issues.apache.org/jira/browse/AMBARI-15226


Repository: ambari


Description
---

Currently the stack advisor is defined under each stack version such as 
HDP/2.3. The problem with this is that it restricts the services that can be 
added to the stack. If a custom service is to be added, they would need to 
modify the stack advisor. If the configuration recommendation and validation 
can be done at the service level then the custom service could just include 
their own recommendations and validations separately.


Diffs (updated)
-

  ambari-server/src/main/assemblies/server.xml e1a4919 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
 df65010 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
 00c8696 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
 ca1968e 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
 6c6fa91 
  
ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
 636de37 
  
ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
 d574d60 
  ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
b7e09a9 
  
ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
 d27e52a 
  ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
db95fec 
  
ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py 
PRE-CREATION 
  ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
PRE-CREATION 
  ambari-server/src/main/resources/properties.json eac0dbd 
  ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
0130483 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
3a65541 
  ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
  ambari-server/src/main/resources/stacks/stack_advisor.py 539bd25 
  ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 6c9fd46 

Diff: https://reviews.apache.org/r/44210/diff/


Testing
---

Ran all the non java unit tests.  

Total run:945
Total errors:0
Total failures:0

Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
their service advisors were called.


Thanks,

Tim Thorpe

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Srimanth Gunturi]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review130123
---




ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
 (line 77)


Trying to think if we really need to expose the 'advisor_name' and 
'advisor_path' in the stack-service API response? The files will be at known 
locations anyways and used when available... similar to the stack's 
advisor_path/advisor_name. 

I am thinking this can work without adding these 2 properties to a 
stack-service.



ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
 (line 88)


I think we can make this work without adding new properties to 
stack-service resource.


- Srimanth Gunturi


On April 22, 2016, 4:39 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 22, 2016, 4:39 p.m.)
> 
> 
> Review request for Ambari, Jayush Luniya, Sumit Mohanty, Srimanth Gunturi, 
> and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  d574d60 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> db95fec 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/properties.json eac0dbd 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 0130483 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> 3a65541 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 539bd25 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 6c9fd46 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
> their service advisors were called.
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Jayush Luniya]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review130124
---




ambari-server/src/main/resources/stacks/service_advisor.py (line 24)


Add documentation for all functions here.


- Jayush Luniya


On April 22, 2016, 4:39 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 22, 2016, 4:39 p.m.)
> 
> 
> Review request for Ambari, Jayush Luniya, Sumit Mohanty, Srimanth Gunturi, 
> and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  d574d60 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> db95fec 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/properties.json eac0dbd 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 0130483 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> 3a65541 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 539bd25 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 6c9fd46 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
> their service advisors were called.
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Jayush Luniya]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review130126
---




ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py 
(line 19)


Please add Pivotal folks to the review to look at this. Also since the 
patch is a bit old, please verify and revise the patch to include any latest 
changes.


- Jayush Luniya


On April 22, 2016, 4:39 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 22, 2016, 4:39 p.m.)
> 
> 
> Review request for Ambari, Jayush Luniya, Sumit Mohanty, Srimanth Gunturi, 
> and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  d574d60 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> db95fec 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/properties.json eac0dbd 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 0130483 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> 3a65541 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 539bd25 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 6c9fd46 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
> their service advisors were called.
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Jayush Luniya]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review130131
---




ambari-server/src/main/resources/stacks/stack_advisor.py (line 507)


loadServiceAdvisor() instead?


- Jayush Luniya


On April 22, 2016, 4:39 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 22, 2016, 4:39 p.m.)
> 
> 
> Review request for Ambari, Jayush Luniya, Sumit Mohanty, Srimanth Gunturi, 
> and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  d574d60 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> db95fec 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/properties.json eac0dbd 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 0130483 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> 3a65541 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 539bd25 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 6c9fd46 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
> their service advisors were called.
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Tim Thorpe]

> On April 22, 2016, 5:15 p.m., Jayush Luniya wrote:
> > ambari-server/src/main/resources/stacks/service_advisor.py, line 24
> > 
> >
> > Add documentation for all functions here.

I will make sure I document all the functions in the service advisor.


- Tim


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review130124
---


On April 22, 2016, 4:39 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 22, 2016, 4:39 p.m.)
> 
> 
> Review request for Ambari, Jayush Luniya, Sumit Mohanty, Srimanth Gunturi, 
> and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  d574d60 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> db95fec 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/properties.json eac0dbd 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 0130483 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> 3a65541 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 539bd25 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 6c9fd46 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
> their service advisors were called.
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Tim Thorpe]

> On April 22, 2016, 5:18 p.m., Jayush Luniya wrote:
> > ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py,
> >  line 19
> > 
> >
> > Please add Pivotal folks to the review to look at this. Also since the 
> > patch is a bit old, please verify and revise the patch to include any 
> > latest changes.

I will update the patch to the latest once I get all the feedback including the 
Pivotal folks.  Otherwise, I'm going to be updating the patch many times before 
I get to the point where it can be accept for commit.  Can you please let me 
know who I should add from Pivotal?  Thanks


- Tim


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review130126
---


On April 22, 2016, 4:39 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 22, 2016, 4:39 p.m.)
> 
> 
> Review request for Ambari, Jayush Luniya, Sumit Mohanty, Srimanth Gunturi, 
> and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  d574d60 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> db95fec 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/properties.json eac0dbd 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 0130483 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> 3a65541 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 539bd25 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 6c9fd46 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
> their service advisors were called.
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Tim Thorpe]

> On April 22, 2016, 5:10 p.m., Srimanth Gunturi wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java,
> >  line 77
> > 
> >
> > Trying to think if we really need to expose the 'advisor_name' and 
> > 'advisor_path' in the stack-service API response? The files will be at 
> > known locations anyways and used when available... similar to the stack's 
> > advisor_path/advisor_name. 
> > 
> > I am thinking this can work without adding these 2 properties to a 
> > stack-service.

The reason this works for the stack is because the stack_advisor.py in the 
scripts directory attempts to load all the stack_advisor.py files.  It uses the 
stack name, stack version and parent versions to determine what to load.  It 
starts with the oldest version and keeps loading until the most recent version. 
 In order to do this for services, we'd need to know the some sort of 
information to determine what to load from where.  I could have added something 
similar so that it would look into the stack versions and then somehow look to 
common services if that was required.  In my code I was testing with HAWQ and 
PXF with their service advisor's being loaded from the common-services 
directory.  It seemed much easier to me at least to get the service_advisor.py 
location and calculate the advisor name in the java code when reading the stack 
and services (StackManager/StackModule/StackDirectory etc...).  This way I know 
exactly where to load the py file and exactly what class name to use.


> On April 22, 2016, 5:10 p.m., Srimanth Gunturi wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java,
> >  line 88
> > 
> >
> > I think we can make this work without adding new properties to 
> > stack-service resource.

see my comment above.


- Tim


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review130123
---


On April 22, 2016, 4:39 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 22, 2016, 4:39 p.m.)
> 
> 
> Review request for Ambari, Jayush Luniya, Sumit Mohanty, Srimanth Gunturi, 
> and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  d574d60 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> db95fec 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/properties.json eac0dbd 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 0130483 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> 3a65541 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 539bd25 
>   ambari-server/src/test/python/stacks/2.3/common/te

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Tim Thorpe]

> On April 22, 2016, 5:33 p.m., Jayush Luniya wrote:
> > ambari-server/src/main/resources/stacks/stack_advisor.py, line 522
> > 
> >
> > loadServiceAdvisor() instead?

If I follow the tradition of the scripts/stack_advisor.py, I guess I should 
call it def instantiateServiceAdvisor().  Any one of those names are fine by 
me.  create or instantiate are more accurate names I think, because it actually 
does both load the script and create an instance.


- Tim


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review130131
---


On April 22, 2016, 4:39 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 22, 2016, 4:39 p.m.)
> 
> 
> Review request for Ambari, Jayush Luniya, Sumit Mohanty, Srimanth Gunturi, 
> and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  d574d60 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> db95fec 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/properties.json eac0dbd 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 0130483 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> 3a65541 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 539bd25 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 6c9fd46 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
> their service advisors were called.
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Jayush Luniya]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review130138
---




ambari-server/src/main/resources/stacks/stack_advisor.py (line 458)


I might be missing something, but it doesnt look like we are using 
getHostsForSlaveComponent


- Jayush Luniya


On April 22, 2016, 4:39 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 22, 2016, 4:39 p.m.)
> 
> 
> Review request for Ambari, Jayush Luniya, Sumit Mohanty, Srimanth Gunturi, 
> and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  d574d60 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> db95fec 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/properties.json eac0dbd 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 0130483 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> 3a65541 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 539bd25 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 6c9fd46 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
> their service advisors were called.
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Jayush Luniya]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review130141
---




ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
(line 31)


Lets say we have 
common-services/MYSERVICE/1.0.0
  |__ service_advisor.py
common-services/MYSERVICE/2.0.0 (<- extends common-services/MYSERVICE/1.0.0)
  |__ service_advisor.py  

stacks/HDP/2.3/services/MYSERVICE (<- extends 
common-services/MYSERVICE/1.0.0)
  |__ service_advisor.py

stacks/HDP/2.5/services/MYSERVICE (<- extends 
common-services/MYSERVICE/2.0.0)
  |__ service_advisor.py
  
I believe we wont support any sort of inheritance between them? I am ok 
with keeping things simple right now but we should call out this limitation and 
that there could be certain duplication.


- Jayush Luniya


On April 22, 2016, 4:39 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 22, 2016, 4:39 p.m.)
> 
> 
> Review request for Ambari, Jayush Luniya, Sumit Mohanty, Srimanth Gunturi, 
> and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  d574d60 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> db95fec 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/properties.json eac0dbd 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 0130483 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> 3a65541 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 539bd25 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 6c9fd46 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
> their service advisors were called.
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Jayush Luniya]

> On April 22, 2016, 6:15 p.m., Jayush Luniya wrote:
> > ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py,
> >  line 31
> > 
> >
> > Lets say we have 
> > common-services/MYSERVICE/1.0.0
> >   |__ service_advisor.py
> > common-services/MYSERVICE/2.0.0 (<- extends 
> > common-services/MYSERVICE/1.0.0)
> >   |__ service_advisor.py  
> > 
> > stacks/HDP/2.3/services/MYSERVICE (<- extends 
> > common-services/MYSERVICE/1.0.0)
> >   |__ service_advisor.py
> > 
> > stacks/HDP/2.5/services/MYSERVICE (<- extends 
> > common-services/MYSERVICE/2.0.0)
> >   |__ service_advisor.py
> >   
> > I believe we wont support any sort of inheritance between them? I am ok 
> > with keeping things simple right now but we should call out this limitation 
> > and that there could be certain duplication.

Sorry the formatting went off after publishing :(


- Jayush


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review130141
---


On April 22, 2016, 4:39 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 22, 2016, 4:39 p.m.)
> 
> 
> Review request for Ambari, Jayush Luniya, Sumit Mohanty, Srimanth Gunturi, 
> and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  d574d60 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> db95fec 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/properties.json eac0dbd 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 0130483 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> 3a65541 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 539bd25 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 6c9fd46 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
> their service advisors were called.
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Jayush Luniya]

> On April 22, 2016, 5:33 p.m., Jayush Luniya wrote:
> > ambari-server/src/main/resources/stacks/stack_advisor.py, line 522
> > 
> >
> > loadServiceAdvisor() instead?
> 
> Tim Thorpe wrote:
> If I follow the tradition of the scripts/stack_advisor.py, I guess I 
> should call it def instantiateServiceAdvisor().  Any one of those names are 
> fine by me.  create or instantiate are more accurate names I think, because 
> it actually does both load the script and create an instance.

instantiateServiceAdvisor() is fine.


- Jayush


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review130131
---


On April 22, 2016, 4:39 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 22, 2016, 4:39 p.m.)
> 
> 
> Review request for Ambari, Jayush Luniya, Sumit Mohanty, Srimanth Gunturi, 
> and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  d574d60 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> db95fec 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/properties.json eac0dbd 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 0130483 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> 3a65541 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 539bd25 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 6c9fd46 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
> their service advisors were called.
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Tim Thorpe]

> On April 22, 2016, 5:58 p.m., Jayush Luniya wrote:
> > ambari-server/src/main/resources/stacks/stack_advisor.py, line 473
> > 
> >
> > I might be missing something, but it doesnt look like we are using 
> > getHostsForSlaveComponent

Yep I missed something in that patch, when I updated an older patch over some 
later code.  The createComponentLayoutRecommendations code should have the 
client/slave section like this:
for service in services["services"]:
  slaveClientComponents = [component for component in service["components"]
   if self.isSlaveComponent(component) or 
self.isClientComponent(component)]
  serviceAdvisor = self.createServiceAdvisor(service)
  for component in slaveClientComponents:
componentName = component["StackServiceComponents"]["component_name"]
hostsForComponent = []
if serviceAdvisor is None:
  hostsForComponent = self.getHostsForSlaveComponent(services, hosts, 
component, hostsList, hostsComponentsMap)
else:
  hostsForComponent = serviceAdvisor.getHostsForSlaveComponent(self, 
services, hosts, component, hostsList, hostsComponentsMap)


- Tim


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review130138
---


On April 22, 2016, 4:39 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 22, 2016, 4:39 p.m.)
> 
> 
> Review request for Ambari, Jayush Luniya, Sumit Mohanty, Srimanth Gunturi, 
> and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  d574d60 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> db95fec 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/properties.json eac0dbd 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 0130483 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> 3a65541 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 539bd25 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 6c9fd46 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
> their service advisors were called.
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Jayush Luniya]

> On April 22, 2016, 5:18 p.m., Jayush Luniya wrote:
> > ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py,
> >  line 19
> > 
> >
> > Please add Pivotal folks to the review to look at this. Also since the 
> > patch is a bit old, please verify and revise the patch to include any 
> > latest changes.
> 
> Tim Thorpe wrote:
> I will update the patch to the latest once I get all the feedback 
> including the Pivotal folks.  Otherwise, I'm going to be updating the patch 
> many times before I get to the point where it can be accept for commit.  Can 
> you please let me know who I should add from Pivotal?  Thanks

https://reviews.apache.org/users/bhuvnesh2703/
https://reviews.apache.org/users/adenissov/
https://reviews.apache.org/users/odiachenko/


- Jayush


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review130126
---


On April 22, 2016, 4:39 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 22, 2016, 4:39 p.m.)
> 
> 
> Review request for Ambari, Jayush Luniya, Sumit Mohanty, Srimanth Gunturi, 
> and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  d574d60 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> db95fec 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/properties.json eac0dbd 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 0130483 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> 3a65541 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 539bd25 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 6c9fd46 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
> their service advisors were called.
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Tim Thorpe]

> On April 22, 2016, 6:15 p.m., Jayush Luniya wrote:
> > ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py,
> >  line 31
> > 
> >
> > Lets say we have 
> > common-services/MYSERVICE/1.0.0
> >   |__ service_advisor.py
> > common-services/MYSERVICE/2.0.0 (<- extends 
> > common-services/MYSERVICE/1.0.0)
> >   |__ service_advisor.py  
> > 
> > stacks/HDP/2.3/services/MYSERVICE (<- extends 
> > common-services/MYSERVICE/1.0.0)
> >   |__ service_advisor.py
> > 
> > stacks/HDP/2.5/services/MYSERVICE (<- extends 
> > common-services/MYSERVICE/2.0.0)
> >   |__ service_advisor.py
> >   
> > I believe we wont support any sort of inheritance between them? I am ok 
> > with keeping things simple right now but we should call out this limitation 
> > and that there could be certain duplication.
> 
> Jayush Luniya wrote:
> Sorry the formatting went off after publishing :(

Hi Jayush, you are right there won't be any automatic inheritance but I don't 
think that's really a bad thing.  You can still have explicit inheritance with 
code like this:

SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
PARENT_DIR = os.path.join(SCRIPT_DIR, '../../../2.2/services/YARN/')
PARENT_FILE = os.path.join(PARENT_DIR, 'service_advisor.py')

try:
  with open(PARENT_FILE, 'rb') as fp:
service_advisor = imp.load_module('service_advisor', fp, PARENT_FILE, 
('.py', 'rb', imp.PY_SOURCE))
except Exception as e:
  traceback.print_exc()
  print "Failed to load parent"


class 
HDP23MAPREDUCE2ServiceAdvisor(service_advisor.HDP22MAPREDUCE2ServiceAdvisor):


- Tim


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review130141
---


On April 22, 2016, 4:39 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 22, 2016, 4:39 p.m.)
> 
> 
> Review request for Ambari, Jayush Luniya, Sumit Mohanty, Srimanth Gunturi, 
> and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  d574d60 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> db95fec 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/properties.json eac0dbd 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 0130483 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> 3a65541 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 539bd25 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 6c9fd46 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
> their service advisors were called.
> 
> 
> 

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Srimanth Gunturi]

> On April 22, 2016, 5:10 p.m., Srimanth Gunturi wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java,
> >  line 77
> > 
> >
> > Trying to think if we really need to expose the 'advisor_name' and 
> > 'advisor_path' in the stack-service API response? The files will be at 
> > known locations anyways and used when available... similar to the stack's 
> > advisor_path/advisor_name. 
> > 
> > I am thinking this can work without adding these 2 properties to a 
> > stack-service.
> 
> Tim Thorpe wrote:
> The reason this works for the stack is because the stack_advisor.py in 
> the scripts directory attempts to load all the stack_advisor.py files.  It 
> uses the stack name, stack version and parent versions to determine what to 
> load.  It starts with the oldest version and keeps loading until the most 
> recent version.  In order to do this for services, we'd need to know the some 
> sort of information to determine what to load from where.  I could have added 
> something similar so that it would look into the stack versions and then 
> somehow look to common services if that was required.  In my code I was 
> testing with HAWQ and PXF with their service advisor's being loaded from the 
> common-services directory.  It seemed much easier to me at least to get the 
> service_advisor.py location and calculate the advisor name in the java code 
> when reading the stack and services (StackManager/StackModule/StackDirectory 
> etc...).  This way I know exactly where to load the py file and exactly what 
> class name t
 o use.

Yes, I agree that it makes it easy to determine location of files. But if you 
look at it from pure API perspective, it is not useful for any caller except 
ambari-server - None of the callers have access to those paths/files except 
ambari-server. It is ambari-server's internals exposed outside just for its own 
consumption. Also, like the stack's service-advisor.py, the service's files can 
be dynamically located and loaded.

If it is a question of efficiency, we can cache the result so that for a 
stack-version's service we only determine this once.

My vote would be to not have these properties on the stack service's response.


- Srimanth


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review130123
---


On April 22, 2016, 6:27 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 22, 2016, 6:27 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  d574d60 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> db95fec 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/properties.json eac0dbd 
>   ambari

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Jayush Luniya]

> On April 22, 2016, 6:15 p.m., Jayush Luniya wrote:
> > ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py,
> >  line 31
> > 
> >
> > Lets say we have 
> > common-services/MYSERVICE/1.0.0
> >   |__ service_advisor.py
> > common-services/MYSERVICE/2.0.0 (<- extends 
> > common-services/MYSERVICE/1.0.0)
> >   |__ service_advisor.py  
> > 
> > stacks/HDP/2.3/services/MYSERVICE (<- extends 
> > common-services/MYSERVICE/1.0.0)
> >   |__ service_advisor.py
> > 
> > stacks/HDP/2.5/services/MYSERVICE (<- extends 
> > common-services/MYSERVICE/2.0.0)
> >   |__ service_advisor.py
> >   
> > I believe we wont support any sort of inheritance between them? I am ok 
> > with keeping things simple right now but we should call out this limitation 
> > and that there could be certain duplication.
> 
> Jayush Luniya wrote:
> Sorry the formatting went off after publishing :(
> 
> Tim Thorpe wrote:
> Hi Jayush, you are right there won't be any automatic inheritance but I 
> don't think that's really a bad thing.  You can still have explicit 
> inheritance with code like this:
> 
> SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
> PARENT_DIR = os.path.join(SCRIPT_DIR, '../../../2.2/services/YARN/')
> PARENT_FILE = os.path.join(PARENT_DIR, 'service_advisor.py')
> 
> try:
>   with open(PARENT_FILE, 'rb') as fp:
> service_advisor = imp.load_module('service_advisor', fp, PARENT_FILE, 
> ('.py', 'rb', imp.PY_SOURCE))
> except Exception as e:
>   traceback.print_exc()
>   print "Failed to load parent"
> 
> 
> class 
> HDP23MAPREDUCE2ServiceAdvisor(service_advisor.HDP22MAPREDUCE2ServiceAdvisor):

Lets just document this in the abstract service_advisor.py. Since stack advisor 
supports such inheritance, just want to make sure that we call this out and 
devs dont assume it will just work :)


- Jayush


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review130141
---


On April 22, 2016, 6:27 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 22, 2016, 6:27 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  d574d60 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> db95fec 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/properties.json eac0dbd 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 0130483 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> 3a65541 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 539bd25 
>   am

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Tim Thorpe]

> On April 22, 2016, 5:10 p.m., Srimanth Gunturi wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java,
> >  line 77
> > 
> >
> > Trying to think if we really need to expose the 'advisor_name' and 
> > 'advisor_path' in the stack-service API response? The files will be at 
> > known locations anyways and used when available... similar to the stack's 
> > advisor_path/advisor_name. 
> > 
> > I am thinking this can work without adding these 2 properties to a 
> > stack-service.
> 
> Tim Thorpe wrote:
> The reason this works for the stack is because the stack_advisor.py in 
> the scripts directory attempts to load all the stack_advisor.py files.  It 
> uses the stack name, stack version and parent versions to determine what to 
> load.  It starts with the oldest version and keeps loading until the most 
> recent version.  In order to do this for services, we'd need to know the some 
> sort of information to determine what to load from where.  I could have added 
> something similar so that it would look into the stack versions and then 
> somehow look to common services if that was required.  In my code I was 
> testing with HAWQ and PXF with their service advisor's being loaded from the 
> common-services directory.  It seemed much easier to me at least to get the 
> service_advisor.py location and calculate the advisor name in the java code 
> when reading the stack and services (StackManager/StackModule/StackDirectory 
> etc...).  This way I know exactly where to load the py file and exactly what 
> class name t
 o use.
> 
> Srimanth Gunturi wrote:
> Yes, I agree that it makes it easy to determine location of files. But if 
> you look at it from pure API perspective, it is not useful for any caller 
> except ambari-server - None of the callers have access to those paths/files 
> except ambari-server. It is ambari-server's internals exposed outside just 
> for its own consumption. Also, like the stack's service-advisor.py, the 
> service's files can be dynamically located and loaded.
> 
> If it is a question of efficiency, we can cache the result so that for a 
> stack-version's service we only determine this once.
> 
> My vote would be to not have these properties on the stack service's 
> response.

Instead of adding them to the stack service's response, I can inject it only 
for the stack advisor calls.  I'll work on making the change.


- Tim


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review130123
---


On April 22, 2016, 6:27 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 22, 2016, 6:27 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  d574d60 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> db95fec 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/s

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Tim Thorpe]

> On April 22, 2016, 6:15 p.m., Jayush Luniya wrote:
> > ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py,
> >  line 31
> > 
> >
> > Lets say we have 
> > common-services/MYSERVICE/1.0.0
> >   |__ service_advisor.py
> > common-services/MYSERVICE/2.0.0 (<- extends 
> > common-services/MYSERVICE/1.0.0)
> >   |__ service_advisor.py  
> > 
> > stacks/HDP/2.3/services/MYSERVICE (<- extends 
> > common-services/MYSERVICE/1.0.0)
> >   |__ service_advisor.py
> > 
> > stacks/HDP/2.5/services/MYSERVICE (<- extends 
> > common-services/MYSERVICE/2.0.0)
> >   |__ service_advisor.py
> >   
> > I believe we wont support any sort of inheritance between them? I am ok 
> > with keeping things simple right now but we should call out this limitation 
> > and that there could be certain duplication.
> 
> Jayush Luniya wrote:
> Sorry the formatting went off after publishing :(
> 
> Tim Thorpe wrote:
> Hi Jayush, you are right there won't be any automatic inheritance but I 
> don't think that's really a bad thing.  You can still have explicit 
> inheritance with code like this:
> 
> SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
> PARENT_DIR = os.path.join(SCRIPT_DIR, '../../../2.2/services/YARN/')
> PARENT_FILE = os.path.join(PARENT_DIR, 'service_advisor.py')
> 
> try:
>   with open(PARENT_FILE, 'rb') as fp:
> service_advisor = imp.load_module('service_advisor', fp, PARENT_FILE, 
> ('.py', 'rb', imp.PY_SOURCE))
> except Exception as e:
>   traceback.print_exc()
>   print "Failed to load parent"
> 
> 
> class 
> HDP23MAPREDUCE2ServiceAdvisor(service_advisor.HDP22MAPREDUCE2ServiceAdvisor):
> 
> Jayush Luniya wrote:
> Lets just document this in the abstract service_advisor.py. Since stack 
> advisor supports such inheritance, just want to make sure that we call this 
> out and devs dont assume it will just work :)

I'll add that in when I do my documentation changes.


- Tim


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review130141
---


On April 22, 2016, 6:27 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 22, 2016, 6:27 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  d574d60 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> db95fec 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/properties.json eac0dbd 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 0130483 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> 3a65541 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Alexander Denissov]

> On April 22, 2016, 6:15 p.m., Jayush Luniya wrote:
> >

So, what will be the rules for SA logic inheritance ?

When MYSERVICE/2.0.0 ships and needs to reuse the stack_advisor logic from 
MYSERVICE/1.0.0 -- will we:
- rely on stack inheritance ?
- rely on SA inheritance within MYSERVICE ?
- combination of these ?


- Alexander


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review130141
---


On April 22, 2016, 6:27 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 22, 2016, 6:27 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  d574d60 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> db95fec 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/properties.json eac0dbd 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 0130483 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> 3a65541 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 539bd25 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 6c9fd46 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
> their service advisors were called.
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Tim Thorpe]

> On April 22, 2016, 6:15 p.m., Jayush Luniya wrote:
> >
> 
> Alexander Denissov wrote:
> So, what will be the rules for SA logic inheritance ?
> 
> When MYSERVICE/2.0.0 ships and needs to reuse the stack_advisor logic 
> from MYSERVICE/1.0.0 -- will we:
> - rely on stack inheritance ?
> - rely on SA inheritance within MYSERVICE ?
> - combination of these ?

There will be no implicit inheritance.  If you want to use inheritance you can 
but it will be left up to the service to handle it with code like:

SCRIPT_DIR = os.path.dirname(os.path.abspath(file))
PARENT_DIR = os.path.join(SCRIPT_DIR, '../../../2.2/services/YARN/')
PARENT_FILE = os.path.join(PARENT_DIR, 'service_advisor.py')

try:
  with open(PARENT_FILE, 'rb') as fp:
service_advisor = imp.load_module('service_advisor', fp, PARENT_FILE, 
('.py', 'rb', imp.PY_SOURCE))
except Exception as e:
  traceback.print_exc()
  print "Failed to load parent"


class 
HDP23MAPREDUCE2ServiceAdvisor(service_advisor.HDP22MAPREDUCE2ServiceAdvisor):


- Tim


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review130141
---


On April 22, 2016, 6:27 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 22, 2016, 6:27 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  d574d60 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> db95fec 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/properties.json eac0dbd 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 0130483 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> 3a65541 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 539bd25 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 6c9fd46 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
> their service advisors were called.
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Tim Thorpe]

> On April 22, 2016, 6:15 p.m., Jayush Luniya wrote:
> >
> 
> Alexander Denissov wrote:
> So, what will be the rules for SA logic inheritance ?
> 
> When MYSERVICE/2.0.0 ships and needs to reuse the stack_advisor logic 
> from MYSERVICE/1.0.0 -- will we:
> - rely on stack inheritance ?
> - rely on SA inheritance within MYSERVICE ?
> - combination of these ?
> 
> Tim Thorpe wrote:
> There will be no implicit inheritance.  If you want to use inheritance 
> you can but it will be left up to the service to handle it with code like:
> 
> SCRIPT_DIR = os.path.dirname(os.path.abspath(file))
> PARENT_DIR = os.path.join(SCRIPT_DIR, '../../../2.2/services/YARN/')
> PARENT_FILE = os.path.join(PARENT_DIR, 'service_advisor.py')
> 
> try:
>   with open(PARENT_FILE, 'rb') as fp:
> service_advisor = imp.load_module('service_advisor', fp, PARENT_FILE, 
> ('.py', 'rb', imp.PY_SOURCE))
> except Exception as e:
>   traceback.print_exc()
>   print "Failed to load parent"
> 
> 
> class 
> HDP23MAPREDUCE2ServiceAdvisor(service_advisor.HDP22MAPREDUCE2ServiceAdvisor):

So I guess the answer to your question is none of those.


- Tim


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review130141
---


On April 22, 2016, 6:27 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 22, 2016, 6:27 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  d574d60 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> db95fec 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/properties.json eac0dbd 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 0130483 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> 3a65541 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 539bd25 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 6c9fd46 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
> their service advisors were called.
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Matt]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review130191
---




ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py 
(line 44)


*return componentName in ('HAWQMASTER', 'HAWQSTANDBY')*



ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py 
(line 118)


There is some additional logic for HAWQ mentioned in 
recommendHDFSConfigurations: 
https://github.com/apache/ambari/blob/trunk/ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py#L368-#L370
 

I believe this would still work from the HDP 2.3 stack advisor. Would you 
like to move it here?



ambari-server/src/main/resources/stacks/service_advisor.py (line 48)


stackadvisor is being passed to many of the service advisor functions for 
using some utility functions like isLocalHost()

These utility functions should ideally reside outside both stack advisor 
and service advisor.



ambari-server/src/main/resources/stacks/stack_advisor.py (line 419)


If component A of service SA is co-located based on how component B of 
service SB is laid out, and services list is ['SA', 'SB'], then host assignment 
for A comes before B, which would be incorrect.

SB's components should be recommended first, then SA's components, because 
A depends on host assignment of B.



ambari-server/src/main/resources/stacks/stack_advisor.py (line 576)


createServiceAdvisor is called multiple times, in 
componentLayoutValidations, recommendComponentLayout, etc. 

Is it not possible to call it once, and cache it and use it again?



ambari-server/src/main/resources/stacks/stack_advisor.py (line 654)


How is this different from the function defined on line 586 (right side)?


- Matt


On April 22, 2016, 11:27 a.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 22, 2016, 11:27 a.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  d574d60 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> db95fec 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/properties.json eac0dbd 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 0130483 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> 3a65541 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 539bd25 
>   ambari-server/src/test/pytho

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Tim Thorpe]

> On April 22, 2016, 11:19 p.m., Matt wrote:
> > ambari-server/src/main/resources/stacks/stack_advisor.py, line 434
> > 
> >
> > If component A of service SA is co-located based on how component B of 
> > service SB is laid out, and services list is ['SA', 'SB'], then host 
> > assignment for A comes before B, which would be incorrect.
> > 
> > SB's components should be recommended first, then SA's components, 
> > because A depends on host assignment of B.

That is why I added a colocate function.  The idea here is that first all 
master components will be assigned their hosts, second all slave/client 
components will be assigned their hosts, and finally all components which need 
to be colocated will calculate their hosts.


> On April 22, 2016, 11:19 p.m., Matt wrote:
> > ambari-server/src/main/resources/stacks/service_advisor.py, line 48
> > 
> >
> > stackadvisor is being passed to many of the service advisor functions 
> > for using some utility functions like isLocalHost()
> > 
> > These utility functions should ideally reside outside both stack 
> > advisor and service advisor.

I agree but the point of my changes here were to minimise the changes on the 
original stack advisor code.  I can look at these in a future jira.


> On April 22, 2016, 11:19 p.m., Matt wrote:
> > ambari-server/src/main/resources/stacks/stack_advisor.py, line 591
> > 
> >
> > createServiceAdvisor is called multiple times, in 
> > componentLayoutValidations, recommendComponentLayout, etc. 
> > 
> > Is it not possible to call it once, and cache it and use it again?

It would be possible to cache like in a map by service.


> On April 22, 2016, 11:19 p.m., Matt wrote:
> > ambari-server/src/main/resources/stacks/stack_advisor.py, line 669
> > 
> >
> > How is this different from the function defined on line 586 (right 
> > side)?

Good catch, I'll go over this again and see which method has the correct 
behavior.  It could be that one of those methods should be removed or perhaps 
one should be renamed.


- Tim


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review130191
---


On April 22, 2016, 6:27 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 22, 2016, 6:27 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  d574d60 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> db95fec 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/properties.json eac0dbd 
>   

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Tim Thorpe]

> On April 22, 2016, 11:19 p.m., Matt wrote:
> > ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py,
> >  line 44
> > 
> >
> > *return componentName in ('HAWQMASTER', 'HAWQSTANDBY')*

That works better, think I just copied and pasted it from a different method.


- Tim


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review130191
---


On April 22, 2016, 6:27 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 22, 2016, 6:27 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  d574d60 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> db95fec 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/properties.json eac0dbd 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 0130483 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> 3a65541 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 539bd25 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 6c9fd46 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
> their service advisors were called.
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Tim Thorpe]

> On April 22, 2016, 11:19 p.m., Matt wrote:
> > ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py,
> >  line 118
> > 
> >
> > There is some additional logic for HAWQ mentioned in 
> > recommendHDFSConfigurations: 
> > https://github.com/apache/ambari/blob/trunk/ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py#L368-#L370
> >  
> > 
> > I believe this would still work from the HDP 2.3 stack advisor. Would 
> > you like to move it here?

I was a bit hesitant to make changes to the functions for the other components 
but it definitely could be.


- Tim


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review130191
---


On April 22, 2016, 6:27 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 22, 2016, 6:27 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  d574d60 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> db95fec 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/properties.json eac0dbd 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 0130483 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> 3a65541 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 539bd25 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 6c9fd46 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
> their service advisors were called.
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Tim Thorpe]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/
---

(Updated April 27, 2016, 8:48 p.m.)


Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.


Changes
---

I've updated the patch with changes for the items I've marked as fixed.


Bugs: AMBARI-15226
https://issues.apache.org/jira/browse/AMBARI-15226


Repository: ambari


Description
---

Currently the stack advisor is defined under each stack version such as 
HDP/2.3. The problem with this is that it restricts the services that can be 
added to the stack. If a custom service is to be added, they would need to 
modify the stack advisor. If the configuration recommendation and validation 
can be done at the service level then the custom service could just include 
their own recommendations and validations separately.


Diffs (updated)
-

  ambari-server/src/main/assemblies/server.xml e1a4919 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
 df65010 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
 00c8696 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
 ca1968e 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
 6c6fa91 
  
ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
 636de37 
  
ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
 356adb1 
  ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
b7e09a9 
  
ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
 d27e52a 
  ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
5a2bf84 
  
ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py 
PRE-CREATION 
  ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
PRE-CREATION 
  ambari-server/src/main/resources/properties.json eac0dbd 
  ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
6b3b1f5 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
9f77129 
  ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
  ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e 
  ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 2080c52 

Diff: https://reviews.apache.org/r/44210/diff/


Testing
---

Ran all the non java unit tests.  

Total run:945
Total errors:0
Total failures:0

Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
their service advisors were called.


Thanks,

Tim Thorpe

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Tim Thorpe]

> On April 22, 2016, 11:19 p.m., Matt wrote:
> > ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py,
> >  line 118
> > 
> >
> > There is some additional logic for HAWQ mentioned in 
> > recommendHDFSConfigurations: 
> > https://github.com/apache/ambari/blob/trunk/ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py#L368-#L370
> >  
> > 
> > I believe this would still work from the HDP 2.3 stack advisor. Would 
> > you like to move it here?
> 
> Tim Thorpe wrote:
> I was a bit hesitant to make changes to the functions for the other 
> components but it definitely could be.

So I made the change and ran the stack manager python tests and all of them 
passed except one:

FAIL: test_recommendHAWQConfigurations 
(test_stack_advisor.TestHDP23StackAdvisor)
--
Traceback (most recent call last):
  File 
"/home/tim/dev/workspaces/15226-test/ambari/ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py",
 line 2047, in test_recommendHAWQConfigurations
'hawq-site': {'properties': {}}})
AssertionError: {'hdfs-client': {'properties': 
{'output.replace-datanode-on-failure': 'false'}}, 'hawq-site': {'properties': 
{}}, 'hdfs-site': {'properties': {'dfs.allow.truncate': 'true'}}} != 
{'hdfs-client': {'properties': {'output.replace-datanode-on-failure': 
'false'}}, 'hawq-site': {'properties': {}}}

I could modify the test to include hdfs-site's dfs.allow.truncate = true.


- Tim


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review130191
---


On April 27, 2016, 8:48 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 27, 2016, 8:48 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  356adb1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> 5a2bf84 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/properties.json eac0dbd 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 6b3b1f5 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> 9f77129 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 2080c52 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
> their service advisors were called.
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Tim Thorpe]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/
---

(Updated April 28, 2016, 4:34 p.m.)


Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.


Bugs: AMBARI-15226
https://issues.apache.org/jira/browse/AMBARI-15226


Repository: ambari


Description
---

Currently the stack advisor is defined under each stack version such as 
HDP/2.3. The problem with this is that it restricts the services that can be 
added to the stack. If a custom service is to be added, they would need to 
modify the stack advisor. If the configuration recommendation and validation 
can be done at the service level then the custom service could just include 
their own recommendations and validations separately.


Diffs (updated)
-

  ambari-server/src/main/assemblies/server.xml e1a4919 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
 df65010 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
 00c8696 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
 ca1968e 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
 6c6fa91 
  
ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
 636de37 
  
ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
 356adb1 
  ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
b7e09a9 
  
ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
 d27e52a 
  ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
5a2bf84 
  
ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py 
PRE-CREATION 
  ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
PRE-CREATION 
  ambari-server/src/main/resources/properties.json eac0dbd 
  ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
1680f21 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
d0ce196 
  ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
  ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e 
  ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 2080c52 

Diff: https://reviews.apache.org/r/44210/diff/


Testing
---

Ran all the non java unit tests.  

Total run:945
Total errors:0
Total failures:0

Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
their service advisors were called.


Thanks,

Tim Thorpe

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Tim Thorpe]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/
---

(Updated April 28, 2016, 4:36 p.m.)


Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.


Changes
---

Uploaded a new patch file with all the changes done over a new workspace using 
today's trunk code


Bugs: AMBARI-15226
https://issues.apache.org/jira/browse/AMBARI-15226


Repository: ambari


Description
---

Currently the stack advisor is defined under each stack version such as 
HDP/2.3. The problem with this is that it restricts the services that can be 
added to the stack. If a custom service is to be added, they would need to 
modify the stack advisor. If the configuration recommendation and validation 
can be done at the service level then the custom service could just include 
their own recommendations and validations separately.


Diffs (updated)
-

  ambari-server/src/main/assemblies/server.xml e1a4919 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
 df65010 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
 00c8696 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
 ca1968e 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
 6c6fa91 
  
ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
 636de37 
  
ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
 356adb1 
  ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
b7e09a9 
  
ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
 d27e52a 
  ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
5a2bf84 
  
ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py 
PRE-CREATION 
  ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
PRE-CREATION 
  ambari-server/src/main/resources/properties.json eac0dbd 
  ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
1680f21 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
d0ce196 
  ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
  ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e 
  ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 2080c52 

Diff: https://reviews.apache.org/r/44210/diff/


Testing
---

Ran all the non java unit tests.  

Total run:945
Total errors:0
Total failures:0

Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
their service advisors were called.


Thanks,

Tim Thorpe

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Jayush Luniya]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review130977
---


Ship it!




Ship It!

- Jayush Luniya


On April 28, 2016, 4:36 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 28, 2016, 4:36 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  356adb1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> 5a2bf84 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/properties.json eac0dbd 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 1680f21 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> d0ce196 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 2080c52 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
> their service advisors were called.
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Jayush Luniya]

> On April 28, 2016, 7:12 p.m., Jayush Luniya wrote:
> > Ship It!

Thanks @Tim for your contribution.


- Jayush


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review130977
---


On April 28, 2016, 4:36 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 28, 2016, 4:36 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  356adb1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> 5a2bf84 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/properties.json eac0dbd 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 1680f21 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> d0ce196 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 2080c52 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
> their service advisors were called.
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Alexander Denissov]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review131002
---




ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py 
(line 110)


if we make isLocalHost utility function available in another utility, then 
there will be no need to pass stackAdvisor as parameter to all the functions in 
a ServiceAdvisor, as it promotes "spagetti-code", since StackAdvisor calls into 
ServiceAdvisor and having the call coming in the opposite directions is not 
desired.



ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py 
(line 115)


since ServiceAdvisor should be called only when the service is installed or 
is chosen to be installed, do we really need the extra check here ?


- Alexander Denissov


On April 28, 2016, 4:36 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 28, 2016, 4:36 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  356adb1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> 5a2bf84 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/properties.json eac0dbd 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 1680f21 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> d0ce196 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 2080c52 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
> their service advisors were called.
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Alexander Denissov]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review131009
---




ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
 (line 56)


should we not have a convention at all and rely on property specified in 
metainfo.xml ?

The service advisors can be called HAWQ200ServiceAdvisor for the service in 
common services and HDP23HAWQ200ServiceAdvisor for the version specific to 
HDP2.3 stack, if any.

This is the argument about inheritance between advisors of different 
versions of service and stack.



ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
 (line 65)


In case the service comes from the common services (like HAWQ) which can be 
versioned independently from the stack, I think we should include the service 
version:

HDP23HAWQ200ServiceAdvisor when HDP2.3 uses HAWQ 2.0.0

HDP23HAWQ210ServiceAdvisor when HDP2.3 uses HAWQ 2.1.0

This example assumes that HAWQ 2.1.0 can be release before the next version 
of Ambari or stack can be released.



ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py 
(line 33)


Should the name include the version, so the future versions can extend this:

HAWQ200ServiceAdvisor ?

Also applies to the file name, should we have HAWQ200_service_advisor.py ?



ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py 
(line 46)


we have a case (not coded yet) that might require more flexible lgoic here 
for which we will need more metadata. When we do thin, we might extend the 
interface to take a list of current component layout.


- Alexander Denissov


On April 28, 2016, 4:36 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 28, 2016, 4:36 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  356adb1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> 5a2bf84 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/properties.json eac0dbd 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 1680f21 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> d0ce196 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 2080c52 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Alexander Denissov]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review131020
---



Do we have a branch cut with these changes in so that we can test HAWQ and PXF 
with this new logic ? After testing the branch can be merged to trunk and we 
will avoid any major surprises.

- Alexander Denissov


On April 28, 2016, 4:36 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 28, 2016, 4:36 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  356adb1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> 5a2bf84 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/properties.json eac0dbd 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 1680f21 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> d0ce196 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 2080c52 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
> their service advisors were called.
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Tim Thorpe]

> On April 29, 2016, 12:12 a.m., Alexander Denissov wrote:
> > Do we have a branch cut with these changes in so that we can test HAWQ and 
> > PXF with this new logic ? After testing the branch can be merged to trunk 
> > and we will avoid any major surprises.

There isn't currently a branch.  Do we really need a branch, wouldn't it be 
just as easy to just apply the patch over trunk and try it out?  If you think 
we need a branch then, I'll ask that one get created.


- Tim


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review131020
---


On April 28, 2016, 4:36 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 28, 2016, 4:36 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  356adb1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> 5a2bf84 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/properties.json eac0dbd 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 1680f21 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> d0ce196 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 2080c52 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
> their service advisors were called.
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Tim Thorpe]

> On April 29, 2016, 12:11 a.m., Alexander Denissov wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java,
> >  line 56
> > 
> >
> > should we not have a convention at all and rely on property specified 
> > in metainfo.xml ?
> > 
> > The service advisors can be called HAWQ200ServiceAdvisor for the 
> > service in common services and HDP23HAWQ200ServiceAdvisor for the version 
> > specific to HDP2.3 stack, if any.
> > 
> > This is the argument about inheritance between advisors of different 
> > versions of service and stack.

I debated about this but I don't see what you really gain by allowing the 
advisor name or file name to be specified in the metainfo.xml.  Although the 
code there should be changed to serviceName + serviceVersion + 
"ServiceAdvisor".  This would support the situation you mentioned in your email 
about adding HAWQ 2.1.0 to common-services and it inheriting from 
HAWQ200ServiceAdvisor.

The file name should always be "service_advisor.py" much like we always have 
"stack_advisor.py".  The class name of the service advisor depends on whether 
it is from common-services or from the stack.  It should be 
ServiceAdvisor in common-services and 
ServiceAdvisor in the stack.


> On April 29, 2016, 12:11 a.m., Alexander Denissov wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java,
> >  line 65
> > 
> >
> > In case the service comes from the common services (like HAWQ) which 
> > can be versioned independently from the stack, I think we should include 
> > the service version:
> > 
> > HDP23HAWQ200ServiceAdvisor when HDP2.3 uses HAWQ 2.0.0
> > 
> > HDP23HAWQ210ServiceAdvisor when HDP2.3 uses HAWQ 2.1.0
> > 
> > This example assumes that HAWQ 2.1.0 can be release before the next 
> > version of Ambari or stack can be released.

I'm not really sure how this would help.  Are you thinking that you'd want 2 
service advisor classes defined in HDP/2.3/services/HAWQ/service_advisor.py or 
2 service advisor files in HDP/2.3/services/HAWQ?  I don't really understand 
why you'd want to do either of those.  You are either using 200 or 210.  I 
don't think it really matters if that switch is defined at the 
service_advisor.py level or the metainfo.xml level.

So you'd define it as 
HDP23HAWQServiceAdvisor(service_advisor.HAWQ200ServiceAdvisor) or 
HDP23HAWQServiceAdvisor(service_advisor.HAWQ210ServiceAdvisor)


> On April 29, 2016, 12:11 a.m., Alexander Denissov wrote:
> > ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py,
> >  line 33
> > 
> >
> > Should the name include the version, so the future versions can extend 
> > this:
> > 
> > HAWQ200ServiceAdvisor ?
> > 
> > Also applies to the file name, should we have 
> > HAWQ200_service_advisor.py ?

I agree for common-services the service version should be included.  I don't 
think there is a reason to have it in the file name.  I had a previous comment 
that recommended the file name always remain the same.  This is identical to 
the way stack_advisor.py is handled.


> On April 29, 2016, 12:11 a.m., Alexander Denissov wrote:
> > ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py,
> >  line 46
> > 
> >
> > we have a case (not coded yet) that might require more flexible lgoic 
> > here for which we will need more metadata. When we do thin, we might extend 
> > the interface to take a list of current component layout.

I think I would need to see a bit more about what you are thinking.  I'm not 
sure that function would necessarily be the best place to have that logic.  
Either way, if in the future you need to change things, all you need to do is 
modify the stacks/stack_advisor.py to pass in the extra parameter(s), modify 
the stacks/service_advisor.py to accept the extra parameter(s) and then modify 
your service_advisor.py file(s) as well.


- Tim


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review131009
---


On April 28, 2016, 4:36 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 28, 2016, 4:36 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sa

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Tim Thorpe]

> On April 28, 2016, 10:54 p.m., Alexander Denissov wrote:
> > ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py,
> >  line 115
> > 
> >
> > since ServiceAdvisor should be called only when the service is 
> > installed or is chosen to be installed, do we really need the extra check 
> > here ?

Line 115 and 116 could be dropped.  Matt had asked if I should port that over 
from the recommendHDFSConfigurations and I left it fully intact from there.


- Tim


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review131002
---


On April 28, 2016, 4:36 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 28, 2016, 4:36 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  356adb1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> 5a2bf84 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/properties.json eac0dbd 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 1680f21 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> d0ce196 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 2080c52 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
> their service advisors were called.
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Tim Thorpe]

> On April 29, 2016, 12:11 a.m., Alexander Denissov wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java,
> >  line 56
> > 
> >
> > should we not have a convention at all and rely on property specified 
> > in metainfo.xml ?
> > 
> > The service advisors can be called HAWQ200ServiceAdvisor for the 
> > service in common services and HDP23HAWQ200ServiceAdvisor for the version 
> > specific to HDP2.3 stack, if any.
> > 
> > This is the argument about inheritance between advisors of different 
> > versions of service and stack.
> 
> Tim Thorpe wrote:
> I debated about this but I don't see what you really gain by allowing the 
> advisor name or file name to be specified in the metainfo.xml.  Although the 
> code there should be changed to serviceName + serviceVersion + 
> "ServiceAdvisor".  This would support the situation you mentioned in your 
> email about adding HAWQ 2.1.0 to common-services and it inheriting from 
> HAWQ200ServiceAdvisor.
> 
> The file name should always be "service_advisor.py" much like we always 
> have "stack_advisor.py".  The class name of the service advisor depends on 
> whether it is from common-services or from the stack.  It should be 
> ServiceAdvisor in common-services and 
> ServiceAdvisor in the stack.

When I scanned over the stacks/service_advisor.py, I noticed that the naming 
convention was not documented.  I added that in.
HAWQ200ServiceAdvisor for the service in common services 
HDP23HAWQServiceAdvisor for the version specific to the HDP 2.3 stack


- Tim


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review131009
---


On April 28, 2016, 4:36 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 28, 2016, 4:36 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  356adb1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> 5a2bf84 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/properties.json eac0dbd 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 1680f21 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> d0ce196 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 2080c52 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
> their service advisors were c

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Tim Thorpe]

> On April 28, 2016, 10:54 p.m., Alexander Denissov wrote:
> > ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py,
> >  line 110
> > 
> >
> > if we make isLocalHost utility function available in another utility, 
> > then there will be no need to pass stackAdvisor as parameter to all the 
> > functions in a ServiceAdvisor, as it promotes "spagetti-code", since 
> > StackAdvisor calls into ServiceAdvisor and having the call coming in the 
> > opposite directions is not desired.

I have been asked to keep this patch as minimal as possible to avoid 
destabilizing the code.  I could have refactored many of the functions into a 
common file which both the stacks/stack_advisor.py and 
stacks/service_advisor.py could use but this would result in many more possible 
issues.  To play it safe I passed in the stackAdvisor.  In a previous 
iteration, when I had refactored most of the stack advisor even for the stack 
services like YARN, HDFS and HBASE, I found that sometimes I needed information 
in say the YARN service_advisor.py that was only available in the HDFS 
service_advisor.py.  In that case, I would create an HDFS service advisor and 
call the required functions.  Now with all of those functions being in the 
stackAdvisor, you can't do those sorts of things.  This means that for some 
service, you might need a reference to the stackAdvisor.  Obviously it needs to 
be used sparingly.

In the future I hope to come back to this code and give a better separation 
between stack and service advisors.


- Tim


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review131002
---


On April 28, 2016, 4:36 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 28, 2016, 4:36 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  356adb1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> 5a2bf84 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/properties.json eac0dbd 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 1680f21 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> d0ce196 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 2080c52 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
> their service advisors were called.
> 
> 
> Thanks,
> 
> Tim T

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Alexander Denissov]

> On April 28, 2016, 10:54 p.m., Alexander Denissov wrote:
> > ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py,
> >  line 110
> > 
> >
> > if we make isLocalHost utility function available in another utility, 
> > then there will be no need to pass stackAdvisor as parameter to all the 
> > functions in a ServiceAdvisor, as it promotes "spagetti-code", since 
> > StackAdvisor calls into ServiceAdvisor and having the call coming in the 
> > opposite directions is not desired.
> 
> Tim Thorpe wrote:
> I have been asked to keep this patch as minimal as possible to avoid 
> destabilizing the code.  I could have refactored many of the functions into a 
> common file which both the stacks/stack_advisor.py and 
> stacks/service_advisor.py could use but this would result in many more 
> possible issues.  To play it safe I passed in the stackAdvisor.  In a 
> previous iteration, when I had refactored most of the stack advisor even for 
> the stack services like YARN, HDFS and HBASE, I found that sometimes I needed 
> information in say the YARN service_advisor.py that was only available in the 
> HDFS service_advisor.py.  In that case, I would create an HDFS service 
> advisor and call the required functions.  Now with all of those functions 
> being in the stackAdvisor, you can't do those sorts of things.  This means 
> that for some service, you might need a reference to the stackAdvisor.  
> Obviously it needs to be used sparingly.
> 
> In the future I hope to come back to this code and give a better 
> separation between stack and service advisors.

Understood. I'm just afraid that the further rectoring might become problematic 
as it will require changes to the interface of the base service_advisor to 
remove passing in the stack advisor. Then we will have to refactor HAWq/PXF 
once again to consume the change. If the parameter is not there in the first 
place, service developers will take small steps to refactor pieces of logic 
they need from StackAdvisor into utils. However, we are OK with your approach 
as well.


- Alexander


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review131002
---


On April 28, 2016, 4:36 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 28, 2016, 4:36 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  356adb1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> 5a2bf84 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/properties.json eac0dbd 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 1680f21 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> d0ce196 
> 

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Alexander Denissov]

> On April 29, 2016, 12:11 a.m., Alexander Denissov wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java,
> >  line 56
> > 
> >
> > should we not have a convention at all and rely on property specified 
> > in metainfo.xml ?
> > 
> > The service advisors can be called HAWQ200ServiceAdvisor for the 
> > service in common services and HDP23HAWQ200ServiceAdvisor for the version 
> > specific to HDP2.3 stack, if any.
> > 
> > This is the argument about inheritance between advisors of different 
> > versions of service and stack.
> 
> Tim Thorpe wrote:
> I debated about this but I don't see what you really gain by allowing the 
> advisor name or file name to be specified in the metainfo.xml.  Although the 
> code there should be changed to serviceName + serviceVersion + 
> "ServiceAdvisor".  This would support the situation you mentioned in your 
> email about adding HAWQ 2.1.0 to common-services and it inheriting from 
> HAWQ200ServiceAdvisor.
> 
> The file name should always be "service_advisor.py" much like we always 
> have "stack_advisor.py".  The class name of the service advisor depends on 
> whether it is from common-services or from the stack.  It should be 
> ServiceAdvisor in common-services and 
> ServiceAdvisor in the stack.
> 
> Tim Thorpe wrote:
> When I scanned over the stacks/service_advisor.py, I noticed that the 
> naming convention was not documented.  I added that in.
> HAWQ200ServiceAdvisor for the service in common services 
> HDP23HAWQServiceAdvisor for the version specific to the HDP 2.3 stack

Can you please give an example of how the advisor path / name will be specified 
in metainfo.xml ?

Your naming convention makes sense. One case we were considering was when we 
would support multiple versions of HAWQ for the same stack. That might require 
changes to the service advisor. If the file is always called service_advisor.py 
and the class is HDP23HAWQServiceAdvisor, then there might be 2 versions of the 
file with the same name and the same class inside. The file will need to be 
managed by stand-alone plugin (extension in later design) along with 
metainfo.xml that goes in stacks/HDP/2.X/services/HAWQ The difference probably 
would be in inheritance:

HDP23HAWQServiceAdvisor (HAWQ200ServiceAdvisor) --> for the original
HDP23HAWQServiceAdvisor (HAWQ210ServiceAdvisor) --> for updated service version


- Alexander


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review131009
---


On April 28, 2016, 4:36 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 28, 2016, 4:36 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  356adb1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> 5a2bf84 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREA

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Alexander Denissov]

> On April 29, 2016, 12:11 a.m., Alexander Denissov wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java,
> >  line 65
> > 
> >
> > In case the service comes from the common services (like HAWQ) which 
> > can be versioned independently from the stack, I think we should include 
> > the service version:
> > 
> > HDP23HAWQ200ServiceAdvisor when HDP2.3 uses HAWQ 2.0.0
> > 
> > HDP23HAWQ210ServiceAdvisor when HDP2.3 uses HAWQ 2.1.0
> > 
> > This example assumes that HAWQ 2.1.0 can be release before the next 
> > version of Ambari or stack can be released.
> 
> Tim Thorpe wrote:
> I'm not really sure how this would help.  Are you thinking that you'd 
> want 2 service advisor classes defined in 
> HDP/2.3/services/HAWQ/service_advisor.py or 2 service advisor files in 
> HDP/2.3/services/HAWQ?  I don't really understand why you'd want to do either 
> of those.  You are either using 200 or 210.  I don't think it really matters 
> if that switch is defined at the service_advisor.py level or the metainfo.xml 
> level.
> 
> So you'd define it as 
> HDP23HAWQServiceAdvisor(service_advisor.HAWQ200ServiceAdvisor) or 
> HDP23HAWQServiceAdvisor(service_advisor.HAWQ210ServiceAdvisor)

yes, I think this will work, as I also replied in the comment above.


- Alexander


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review131009
---


On April 28, 2016, 4:36 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 28, 2016, 4:36 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  356adb1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> 5a2bf84 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/properties.json eac0dbd 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 1680f21 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> d0ce196 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 2080c52 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
> their service advisors were called.
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Tim Thorpe]

> On April 29, 2016, 12:11 a.m., Alexander Denissov wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java,
> >  line 56
> > 
> >
> > should we not have a convention at all and rely on property specified 
> > in metainfo.xml ?
> > 
> > The service advisors can be called HAWQ200ServiceAdvisor for the 
> > service in common services and HDP23HAWQ200ServiceAdvisor for the version 
> > specific to HDP2.3 stack, if any.
> > 
> > This is the argument about inheritance between advisors of different 
> > versions of service and stack.
> 
> Tim Thorpe wrote:
> I debated about this but I don't see what you really gain by allowing the 
> advisor name or file name to be specified in the metainfo.xml.  Although the 
> code there should be changed to serviceName + serviceVersion + 
> "ServiceAdvisor".  This would support the situation you mentioned in your 
> email about adding HAWQ 2.1.0 to common-services and it inheriting from 
> HAWQ200ServiceAdvisor.
> 
> The file name should always be "service_advisor.py" much like we always 
> have "stack_advisor.py".  The class name of the service advisor depends on 
> whether it is from common-services or from the stack.  It should be 
> ServiceAdvisor in common-services and 
> ServiceAdvisor in the stack.
> 
> Tim Thorpe wrote:
> When I scanned over the stacks/service_advisor.py, I noticed that the 
> naming convention was not documented.  I added that in.
> HAWQ200ServiceAdvisor for the service in common services 
> HDP23HAWQServiceAdvisor for the version specific to the HDP 2.3 stack
> 
> Alexander Denissov wrote:
> Can you please give an example of how the advisor path / name will be 
> specified in metainfo.xml ?
> 
> Your naming convention makes sense. One case we were considering was when 
> we would support multiple versions of HAWQ for the same stack. That might 
> require changes to the service advisor. If the file is always called 
> service_advisor.py and the class is HDP23HAWQServiceAdvisor, then there might 
> be 2 versions of the file with the same name and the same class inside. The 
> file will need to be managed by stand-alone plugin (extension in later 
> design) along with metainfo.xml that goes in stacks/HDP/2.X/services/HAWQ The 
> difference probably would be in inheritance:
> 
> HDP23HAWQServiceAdvisor (HAWQ200ServiceAdvisor) --> for the original
> HDP23HAWQServiceAdvisor (HAWQ210ServiceAdvisor) --> for updated service 
> version

Currently there is nothing in the metainfo.xml to specify the advisor name & 
path.  The ServiceDirectory java code will detect the presence of 
service_advisor.py files in the root of the service folder (whether 
common-services or in the stack).  It will use the existing inheritance method 
such that if the stack service doesn't contain the service_advisor.py file, it 
will inherit that file from common-services or its parent stack version (if the 
service exists there).  The name will be calculated in that same java code 
based on the naming convention mentioned above.  All inheritance from python 
file to python (as previously mentioned) will be handled by explicit 
inheritance included by the service developers.


- Tim


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review131009
---


On April 28, 2016, 4:36 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 28, 2016, 4:36 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Tim Thorpe]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/
---

(Updated April 29, 2016, 8:11 p.m.)


Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.


Changes
---

Did another quick change to change the common-services naming convention to 
include the service version number and remove lines 115 and 116 in the HAWQ 
service advisor:
servicesList = [service["StackServices"]["service_name"] for service in 
services["services"]]
if "HAWQ" in servicesList:


Bugs: AMBARI-15226
https://issues.apache.org/jira/browse/AMBARI-15226


Repository: ambari


Description
---

Currently the stack advisor is defined under each stack version such as 
HDP/2.3. The problem with this is that it restricts the services that can be 
added to the stack. If a custom service is to be added, they would need to 
modify the stack advisor. If the configuration recommendation and validation 
can be done at the service level then the custom service could just include 
their own recommendations and validations separately.


Diffs (updated)
-

  ambari-server/src/main/assemblies/server.xml e1a4919 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
 df65010 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
 00c8696 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
 ca1968e 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
 6c6fa91 
  
ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
 636de37 
  
ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
 356adb1 
  ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
b7e09a9 
  
ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
 d27e52a 
  ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
5a2bf84 
  
ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py 
PRE-CREATION 
  ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
PRE-CREATION 
  ambari-server/src/main/resources/properties.json eac0dbd 
  ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
1680f21 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
f475798 
  ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
  ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e 
  ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 2080c52 

Diff: https://reviews.apache.org/r/44210/diff/


Testing
---

Ran all the non java unit tests.  

Total run:945
Total errors:0
Total failures:0

Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
their service advisors were called.


Thanks,

Tim Thorpe

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Alexander Denissov]

> On April 29, 2016, 12:12 a.m., Alexander Denissov wrote:
> > Do we have a branch cut with these changes in so that we can test HAWQ and 
> > PXF with this new logic ? After testing the branch can be merged to trunk 
> > and we will avoid any major surprises.
> 
> Tim Thorpe wrote:
> There isn't currently a branch.  Do we really need a branch, wouldn't it 
> be just as easy to just apply the patch over trunk and try it out?  If you 
> think we need a branch then, I'll ask that one get created.

We will be testing this patch with HAWQ to see if it behaves as expected and 
there are no regressions.


- Alexander


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review131020
---


On April 29, 2016, 8:11 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 29, 2016, 8:11 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  356adb1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> 5a2bf84 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/properties.json eac0dbd 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 1680f21 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> f475798 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 2080c52 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
> their service advisors were called.
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Tim Thorpe]

> On April 29, 2016, 12:12 a.m., Alexander Denissov wrote:
> > Do we have a branch cut with these changes in so that we can test HAWQ and 
> > PXF with this new logic ? After testing the branch can be merged to trunk 
> > and we will avoid any major surprises.
> 
> Tim Thorpe wrote:
> There isn't currently a branch.  Do we really need a branch, wouldn't it 
> be just as easy to just apply the patch over trunk and try it out?  If you 
> think we need a branch then, I'll ask that one get created.
> 
> Alexander Denissov wrote:
> We will be testing this patch with HAWQ to see if it behaves as expected 
> and there are no regressions.

Hi Alex, please let me know when you have validated the patch with HAWQ so I 
can get it pushed in (after I try it out again over trunk). Thanks


- Tim


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review131020
---


On April 29, 2016, 8:11 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 29, 2016, 8:11 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  356adb1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> 5a2bf84 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/properties.json eac0dbd 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 1680f21 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> f475798 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 2080c52 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
> their service advisors were called.
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Alexander Denissov]

> On April 29, 2016, 12:12 a.m., Alexander Denissov wrote:
> > Do we have a branch cut with these changes in so that we can test HAWQ and 
> > PXF with this new logic ? After testing the branch can be merged to trunk 
> > and we will avoid any major surprises.
> 
> Tim Thorpe wrote:
> There isn't currently a branch.  Do we really need a branch, wouldn't it 
> be just as easy to just apply the patch over trunk and try it out?  If you 
> think we need a branch then, I'll ask that one get created.
> 
> Alexander Denissov wrote:
> We will be testing this patch with HAWQ to see if it behaves as expected 
> and there are no regressions.
> 
> Tim Thorpe wrote:
> Hi Alex, please let me know when you have validated the patch with HAWQ 
> so I can get it pushed in (after I try it out again over trunk). Thanks

Tim, the guys tried it on Friday and it did not quite work (custom 
recommendations did not come in UI). They are trying to dig deeper and debug 
today to pinpoint where the problem is.


- Alexander


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review131020
---


On April 29, 2016, 8:11 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 29, 2016, 8:11 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  356adb1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> 5a2bf84 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/properties.json eac0dbd 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 1680f21 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> f475798 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 2080c52 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
> their service advisors were called.
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Lav Jain]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review131409
---




ambari-server/src/main/resources/stacks/stack_advisor.py (line 619)


This method is not being called because the corresponding method in 2.0.6 
stack advisor is overriding it.


- Lav Jain


On April 29, 2016, 8:11 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 29, 2016, 8:11 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  356adb1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> 5a2bf84 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/properties.json eac0dbd 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 1680f21 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> f475798 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 2080c52 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
> their service advisors were called.
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Tim Thorpe]

> On May 2, 2016, 11:07 p.m., Lav Jain wrote:
> > ambari-server/src/main/resources/stacks/stack_advisor.py, line 644
> > 
> >
> > This method is not being called because the corresponding method in 
> > 2.0.6 stack advisor is overriding it.

Hi Lav, I don't see the 2.0.6 stack advisor overriding this method (def 
validateComponentLayout).  It does override the method 
getComponentLayoutValidations which gets called by validateComponentLayout but 
in that method it calls super getComponentLayoutValidations first:
  def getComponentLayoutValidations(self, services, hosts):
"""Returns array of Validation objects about issues with hostnames 
components assigned to"""
items = super(HDP206StackAdvisor, 
self).getComponentLayoutValidations(services, hosts)


- Tim


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review131409
---


On April 29, 2016, 8:11 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated April 29, 2016, 8:11 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  356adb1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> 5a2bf84 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/properties.json eac0dbd 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 1680f21 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> f475798 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 2080c52 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
> their service advisors were called.
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Tim Thorpe]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/
---

(Updated May 3, 2016, 2:21 p.m.)


Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.


Changes
---

This patch fixes the common-services naming convention so that it uses service 
name and version.  I attempted to install HAWQ on a 3 node cluster.  It 
recommended the standby on node 2 and the master on node 3.  So looks like its 
working properly.  Also I checked the services.json in 
/var/run/ambari-server/stack-recommendations/1 and it returns 
HAWQ200ServiceAdvisor.

cat services.json | grep -A 2 -B 2 advisor_name
  "stack_name" : "HDP",
  "stack_version" : "2.3",
  "advisor_name" : "HAWQ200ServiceAdvisor",
  "advisor_path" : 
"/var/lib/ambari-server/resources/common-services/HAWQ/2.0.0/service_advisor.py"
},


Bugs: AMBARI-15226
https://issues.apache.org/jira/browse/AMBARI-15226


Repository: ambari


Description
---

Currently the stack advisor is defined under each stack version such as 
HDP/2.3. The problem with this is that it restricts the services that can be 
added to the stack. If a custom service is to be added, they would need to 
modify the stack advisor. If the configuration recommendation and validation 
can be done at the service level then the custom service could just include 
their own recommendations and validations separately.


Diffs (updated)
-

  ambari-server/src/main/assemblies/server.xml e1a4919 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
 df65010 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
 00c8696 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
 ca1968e 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
 6c6fa91 
  
ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
 636de37 
  
ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
 356adb1 
  ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
b7e09a9 
  
ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
 d27e52a 
  ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
5a2bf84 
  
ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py 
PRE-CREATION 
  ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
PRE-CREATION 
  ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
1680f21 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
f475798 
  ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
  ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e 
  ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 2080c52 

Diff: https://reviews.apache.org/r/44210/diff/


Testing
---

Ran all the non java unit tests.  

Total run:945
Total errors:0
Total failures:0

Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
their service advisors were called.


Thanks,

Tim Thorpe

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Lav Jain]

> On May 2, 2016, 11:07 p.m., Lav Jain wrote:
> > ambari-server/src/main/resources/stacks/stack_advisor.py, line 644
> > 
> >
> > This method is not being called because the corresponding method in 
> > 2.0.6 stack advisor is overriding it.
> 
> Tim Thorpe wrote:
> Hi Lav, I don't see the 2.0.6 stack advisor overriding this method (def 
> validateComponentLayout).  It does override the method 
> getComponentLayoutValidations which gets called by validateComponentLayout 
> but in that method it calls super getComponentLayoutValidations first:
>   def getComponentLayoutValidations(self, services, hosts):
> """Returns array of Validation objects about issues with hostnames 
> components assigned to"""
> items = super(HDP206StackAdvisor, 
> self).getComponentLayoutValidations(services, hosts)

Hi Tim,

The method being overridden is getConfigurationsValidationItems():

https://github.com/apache/ambari/blob/trunk/ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py#L929-L956


- Lav


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review131409
---


On May 3, 2016, 2:21 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated May 3, 2016, 2:21 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  356adb1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> 5a2bf84 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 1680f21 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> f475798 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 2080c52 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
> their service advisors were called.
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Tim Thorpe]

> On May 2, 2016, 11:07 p.m., Lav Jain wrote:
> > ambari-server/src/main/resources/stacks/stack_advisor.py, line 644
> > 
> >
> > This method is not being called because the corresponding method in 
> > 2.0.6 stack advisor is overriding it.
> 
> Tim Thorpe wrote:
> Hi Lav, I don't see the 2.0.6 stack advisor overriding this method (def 
> validateComponentLayout).  It does override the method 
> getComponentLayoutValidations which gets called by validateComponentLayout 
> but in that method it calls super getComponentLayoutValidations first:
>   def getComponentLayoutValidations(self, services, hosts):
> """Returns array of Validation objects about issues with hostnames 
> components assigned to"""
> items = super(HDP206StackAdvisor, 
> self).getComponentLayoutValidations(services, hosts)
> 
> Lav Jain wrote:
> Hi Tim,
> 
> The method being overridden is getConfigurationsValidationItems():
> 
> 
> https://github.com/apache/ambari/blob/trunk/ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py#L929-L956

Thanks, just confirmed it.  I'll make sure the 2.0.6 stack advisor calls the 
super method.


- Tim


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review131409
---


On May 3, 2016, 2:21 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated May 3, 2016, 2:21 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  356adb1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> 5a2bf84 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 1680f21 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> f475798 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 2080c52 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
> their service advisors were called.
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Tim Thorpe]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/
---

(Updated May 3, 2016, 7:04 p.m.)


Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.


Changes
---

This fixes the issue Lav brought up about configuration validation not calling 
the service advisors.


Bugs: AMBARI-15226
https://issues.apache.org/jira/browse/AMBARI-15226


Repository: ambari


Description
---

Currently the stack advisor is defined under each stack version such as 
HDP/2.3. The problem with this is that it restricts the services that can be 
added to the stack. If a custom service is to be added, they would need to 
modify the stack advisor. If the configuration recommendation and validation 
can be done at the service level then the custom service could just include 
their own recommendations and validations separately.


Diffs (updated)
-

  ambari-server/src/main/assemblies/server.xml e1a4919 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
 df65010 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
 00c8696 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
 ca1968e 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
 6c6fa91 
  
ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
 636de37 
  
ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
 356adb1 
  ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
b7e09a9 
  
ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
 d27e52a 
  ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
5a2bf84 
  
ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py 
PRE-CREATION 
  ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
PRE-CREATION 
  ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
1680f21 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
f475798 
  ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
  ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e 
  ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 2080c52 

Diff: https://reviews.apache.org/r/44210/diff/


Testing
---

Ran all the non java unit tests.  

Total run:945
Total errors:0
Total failures:0

Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
their service advisors were called.


Thanks,

Tim Thorpe

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Matt]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review131579
---



The functionality of ServiceAdvisor works as expected. However more 
clarification is needed on how to use inheritance and override service advisor 
defined in common-services. 
For example:
HAWQ200ServiceAdvisor is defined under commmon-services. I'd like to override 
the method getComponentLayoutScheme with some new logic for HDP 2.3 stack. 
Followed the documentation in 
ambari-server/src/main/resources/stacks/service_advisor.py but it did not work 
as expected.

- Matt


On May 3, 2016, 12:04 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated May 3, 2016, 12:04 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  356adb1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> 5a2bf84 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 1680f21 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> f475798 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 2080c52 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
> their service advisors were called.
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Tim Thorpe]

> On May 3, 2016, 11:09 p.m., Matt wrote:
> > The functionality of ServiceAdvisor works as expected. However more 
> > clarification is needed on how to use inheritance and override service 
> > advisor defined in common-services. 
> > For example:
> > HAWQ200ServiceAdvisor is defined under commmon-services. I'd like to 
> > override the method getComponentLayoutScheme with some new logic for HDP 
> > 2.3 stack. Followed the documentation in 
> > ambari-server/src/main/resources/stacks/service_advisor.py but it did not 
> > work as expected.

Thanks Matt, for finding that issue.  There were actually 2 problems.  The 
first was with the advisor class naming convention for stack service's that 
contained the service_advisor.py rather than inherit it from common-services.  
It missed the stack version number in the advisor class name.  The second issue 
was with the HAWQ and PXF service_advisor.py files which added to the stacks 
directory to the system path to load the stacks/service_advisor.py (for the 
default service advisor).  I have modified that code to use the same dynamic 
loading I recommended for service advisor inheritance.  In other words, I 
changed it to this:

SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
STACKS_DIR = os.path.join(SCRIPT_DIR, '../../../stacks/')
PARENT_FILE = os.path.join(STACKS_DIR, 'service_advisor.py')

try:
  with open(PARENT_FILE, 'rb') as fp:
service_advisor = imp.load_module('service_advisor', fp, PARENT_FILE, 
('.py', 'rb', imp.PY_SOURCE))
except Exception as e:
  traceback.print_exc()
  print "Failed to load parent"

class HAWQ200ServiceAdvisor(service_advisor.ServiceAdvisor):

I presume the reason this change is required is due to a naming conflict 
because all the files are called service_advisor.py.


- Tim


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review131579
---


On May 3, 2016, 7:04 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated May 3, 2016, 7:04 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  356adb1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> 5a2bf84 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 1680f21 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> f475798 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 2080c52 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Tim Thorpe]

> On May 3, 2016, 11:09 p.m., Matt wrote:
> > The functionality of ServiceAdvisor works as expected. However more 
> > clarification is needed on how to use inheritance and override service 
> > advisor defined in common-services. 
> > For example:
> > HAWQ200ServiceAdvisor is defined under commmon-services. I'd like to 
> > override the method getComponentLayoutScheme with some new logic for HDP 
> > 2.3 stack. Followed the documentation in 
> > ambari-server/src/main/resources/stacks/service_advisor.py but it did not 
> > work as expected.
> 
> Tim Thorpe wrote:
> Thanks Matt, for finding that issue.  There were actually 2 problems.  
> The first was with the advisor class naming convention for stack service's 
> that contained the service_advisor.py rather than inherit it from 
> common-services.  It missed the stack version number in the advisor class 
> name.  The second issue was with the HAWQ and PXF service_advisor.py files 
> which added to the stacks directory to the system path to load the 
> stacks/service_advisor.py (for the default service advisor).  I have modified 
> that code to use the same dynamic loading I recommended for service advisor 
> inheritance.  In other words, I changed it to this:
> 
> SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
> STACKS_DIR = os.path.join(SCRIPT_DIR, '../../../stacks/')
> PARENT_FILE = os.path.join(STACKS_DIR, 'service_advisor.py')
> 
> try:
>   with open(PARENT_FILE, 'rb') as fp:
> service_advisor = imp.load_module('service_advisor', fp, PARENT_FILE, 
> ('.py', 'rb', imp.PY_SOURCE))
> except Exception as e:
>   traceback.print_exc()
>   print "Failed to load parent"
> 
> class HAWQ200ServiceAdvisor(service_advisor.ServiceAdvisor):
> 
> I presume the reason this change is required is due to a naming conflict 
> because all the files are called service_advisor.py.

I have also added a comment in stacks/service_advisor.py to look at the HAWQ 
and PXF service advisors for examples on how to create your own service advisor.


- Tim


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review131579
---


On May 3, 2016, 7:04 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated May 3, 2016, 7:04 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  356adb1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> 5a2bf84 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 1680f21 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> f475798 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e 
>   ambari-server/

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Tim Thorpe]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/
---

(Updated May 4, 2016, 2:17 p.m.)


Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.


Changes
---

Fixes issues with stack service's advisor class naming convention and with HAWQ 
and PXF common services inheritance.


Bugs: AMBARI-15226
https://issues.apache.org/jira/browse/AMBARI-15226


Repository: ambari


Description
---

Currently the stack advisor is defined under each stack version such as 
HDP/2.3. The problem with this is that it restricts the services that can be 
added to the stack. If a custom service is to be added, they would need to 
modify the stack advisor. If the configuration recommendation and validation 
can be done at the service level then the custom service could just include 
their own recommendations and validations separately.


Diffs (updated)
-

  ambari-server/src/main/assemblies/server.xml e1a4919 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
 df65010 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
 00c8696 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
 ca1968e 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
 6c6fa91 
  
ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
 636de37 
  
ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
 356adb1 
  ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
b7e09a9 
  
ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
 d27e52a 
  ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
5a2bf84 
  
ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py 
PRE-CREATION 
  ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
PRE-CREATION 
  ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
1680f21 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
f475798 
  ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
  ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e 
  ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 2080c52 

Diff: https://reviews.apache.org/r/44210/diff/


Testing
---

Ran all the non java unit tests.  

Total run:945
Total errors:0
Total failures:0

Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
their service advisors were called.


Thanks,

Tim Thorpe

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Matt]

> On May 3, 2016, 4:09 p.m., Matt wrote:
> > The functionality of ServiceAdvisor works as expected. However more 
> > clarification is needed on how to use inheritance and override service 
> > advisor defined in common-services. 
> > For example:
> > HAWQ200ServiceAdvisor is defined under commmon-services. I'd like to 
> > override the method getComponentLayoutScheme with some new logic for HDP 
> > 2.3 stack. Followed the documentation in 
> > ambari-server/src/main/resources/stacks/service_advisor.py but it did not 
> > work as expected.
> 
> Tim Thorpe wrote:
> Thanks Matt, for finding that issue.  There were actually 2 problems.  
> The first was with the advisor class naming convention for stack service's 
> that contained the service_advisor.py rather than inherit it from 
> common-services.  It missed the stack version number in the advisor class 
> name.  The second issue was with the HAWQ and PXF service_advisor.py files 
> which added to the stacks directory to the system path to load the 
> stacks/service_advisor.py (for the default service advisor).  I have modified 
> that code to use the same dynamic loading I recommended for service advisor 
> inheritance.  In other words, I changed it to this:
> 
> SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
> STACKS_DIR = os.path.join(SCRIPT_DIR, '../../../stacks/')
> PARENT_FILE = os.path.join(STACKS_DIR, 'service_advisor.py')
> 
> try:
>   with open(PARENT_FILE, 'rb') as fp:
> service_advisor = imp.load_module('service_advisor', fp, PARENT_FILE, 
> ('.py', 'rb', imp.PY_SOURCE))
> except Exception as e:
>   traceback.print_exc()
>   print "Failed to load parent"
> 
> class HAWQ200ServiceAdvisor(service_advisor.ServiceAdvisor):
> 
> I presume the reason this change is required is due to a naming conflict 
> because all the files are called service_advisor.py.
> 
> Tim Thorpe wrote:
> I have also added a comment in stacks/service_advisor.py to look at the 
> HAWQ and PXF service advisors for examples on how to create your own service 
> advisor.

Thank you Tim for making the updates. We are testing it at the moment...


- Matt


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review131579
---


On May 4, 2016, 7:17 a.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated May 4, 2016, 7:17 a.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  356adb1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> 5a2bf84 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 1680f21 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> f475798 
>   ambari-server/src/main/resources/stacks/service_adviso

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Matt]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review131747
---


Ship it!




We have tested the following changes:
- The logic for stack_advisor which was pulled into service_advisor works well.
- Inheritance between stacks works well. We tested inheritance from 
common-services into stack 2.3, and then inheriting from 2.3 to stack 2.4 with 
overrides.

Enhancements:
- Factor out utility functions that are common for both stack_advisor and 
service_advisor.
- Cache service_advisors to avoid loading service_advisor multiple times for 
the same service.
- At the moment, the service advisor class name under the stack should be of 
the format **ServiceAdvisor**. It 
would be great if this class can have a different name in the 
service_advisor.py file and specify the file path and class name in 
metainfo.xml under that stack, so that the class gets loaded.

- Matt


On May 4, 2016, 7:17 a.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated May 4, 2016, 7:17 a.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  356adb1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> 5a2bf84 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 1680f21 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> f475798 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> 2080c52 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
> their service advisors were called.
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Tim Thorpe]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/
---

(Updated May 5, 2016, 2:50 p.m.)


Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.


Changes
---

Final patch file attached, which was merged into latest trunk.


Bugs: AMBARI-15226
https://issues.apache.org/jira/browse/AMBARI-15226


Repository: ambari


Description
---

Currently the stack advisor is defined under each stack version such as 
HDP/2.3. The problem with this is that it restricts the services that can be 
added to the stack. If a custom service is to be added, they would need to 
modify the stack advisor. If the configuration recommendation and validation 
can be done at the service level then the custom service could just include 
their own recommendations and validations separately.


Diffs (updated)
-

  ambari-server/src/main/assemblies/server.xml e1a4919 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
 df65010 
  
ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
 00c8696 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
 ca1968e 
  
ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
 6c6fa91 
  
ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
 636de37 
  
ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
 356adb1 
  ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
b7e09a9 
  
ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
 d27e52a 
  ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
5a2bf84 
  
ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py 
PRE-CREATION 
  ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
PRE-CREATION 
  ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
1680f21 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
cf0990d 
  ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
  ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e 
  ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py fcb5407 

Diff: https://reviews.apache.org/r/44210/diff/


Testing
---

Ran all the non java unit tests.  

Total run:945
Total errors:0
Total failures:0

Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
their service advisors were called.


Thanks,

Tim Thorpe

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Tim Thorpe]

> On May 4, 2016, 9:37 p.m., Matt wrote:
> > We have tested the following changes:
> > - The logic for stack_advisor which was pulled into service_advisor works 
> > well.
> > - Inheritance between stacks works well. We tested inheritance from 
> > common-services into stack 2.3, and then inheriting from 2.3 to stack 2.4 
> > with overrides.
> > 
> > Enhancements:
> > - Factor out utility functions that are common for both stack_advisor and 
> > service_advisor.
> > - Cache service_advisors to avoid loading service_advisor multiple times 
> > for the same service.
> > - At the moment, the service advisor class name under the stack should be 
> > of the format **ServiceAdvisor**. 
> > It would be great if this class can have a different name in the 
> > service_advisor.py file and specify the file path and class name in 
> > metainfo.xml under that stack, so that the class gets loaded.

Hi Matt, are you able to push the patch in?  I don't have committer access.  If 
not can someone else?  Thanks


- Tim


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review131747
---


On May 5, 2016, 2:50 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated May 5, 2016, 2:50 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  356adb1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> 5a2bf84 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 1680f21 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> cf0990d 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> fcb5407 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
> their service advisors were called.
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Matt]

> On May 4, 2016, 2:37 p.m., Matt wrote:
> > We have tested the following changes:
> > - The logic for stack_advisor which was pulled into service_advisor works 
> > well.
> > - Inheritance between stacks works well. We tested inheritance from 
> > common-services into stack 2.3, and then inheriting from 2.3 to stack 2.4 
> > with overrides.
> > 
> > Enhancements:
> > - Factor out utility functions that are common for both stack_advisor and 
> > service_advisor.
> > - Cache service_advisors to avoid loading service_advisor multiple times 
> > for the same service.
> > - At the moment, the service advisor class name under the stack should be 
> > of the format **ServiceAdvisor**. 
> > It would be great if this class can have a different name in the 
> > service_advisor.py file and specify the file path and class name in 
> > metainfo.xml under that stack, so that the class gets loaded.
> 
> Tim Thorpe wrote:
> Hi Matt, are you able to push the patch in?  I don't have committer 
> access.  If not can someone else?  Thanks

I'll look into this within a couple of hours, when I get back to my laptop.


- Matt


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review131747
---


On May 5, 2016, 7:50 a.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated May 5, 2016, 7:50 a.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  356adb1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> 5a2bf84 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 1680f21 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> cf0990d 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> fcb5407 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
> their service advisors were called.
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Jayush Luniya]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review131962
---


Ship it!




Ship It!

- Jayush Luniya


On May 5, 2016, 2:50 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated May 5, 2016, 2:50 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  356adb1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> 5a2bf84 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 1680f21 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> cf0990d 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> fcb5407 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
> their service advisors were called.
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Matt]

> On May 5, 2016, 3:52 p.m., Jayush Luniya wrote:
> > Ship It!

Jayush, shall I commit this or are you working on it?


- Matt


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review131962
---


On May 5, 2016, 7:50 a.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated May 5, 2016, 7:50 a.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  356adb1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> 5a2bf84 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 1680f21 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> cf0990d 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> fcb5407 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
> their service advisors were called.
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>

Re: Review Request 44210: AMBARI-15226 - The stack advisor should be pushed down to the services

[Jayush Luniya]

> On May 5, 2016, 10:52 p.m., Jayush Luniya wrote:
> > Ship It!
> 
> Matt wrote:
> Jayush, shall I commit this or are you working on it?

Committed to trunk


- Jayush


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44210/#review131962
---


On May 5, 2016, 2:50 p.m., Tim Thorpe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44210/
> ---
> 
> (Updated May 5, 2016, 2:50 p.m.)
> 
> 
> Review request for Ambari, Alexander Denissov, bhuvnesh chaudhary, Jayush 
> Luniya, Oleksandr Diachenko, Sumit Mohanty, Srimanth Gunturi, and Yusaku Sako.
> 
> 
> Bugs: AMBARI-15226
> https://issues.apache.org/jira/browse/AMBARI-15226
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Currently the stack advisor is defined under each stack version such as 
> HDP/2.3. The problem with this is that it restricts the services that can be 
> added to the stack. If a custom service is to be added, they would need to 
> modify the stack advisor. If the configuration recommendation and validation 
> can be done at the service level then the custom service could just include 
> their own recommendations and validations separately.
> 
> 
> Diffs
> -
> 
>   ambari-server/src/main/assemblies/server.xml e1a4919 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/AmbariMetaInfo.java
>  df65010 
>   
> ambari-server/src/main/java/org/apache/ambari/server/api/services/stackadvisor/commands/StackAdvisorCommand.java
>  00c8696 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/StackServiceResponse.java
>  ca1968e 
>   
> ambari-server/src/main/java/org/apache/ambari/server/controller/internal/StackServiceResourceProvider.java
>  6c6fa91 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/CommonServiceDirectory.java
>  636de37 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceDirectory.java
>  356adb1 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/ServiceModule.java 
> b7e09a9 
>   
> ambari-server/src/main/java/org/apache/ambari/server/stack/StackServiceDirectory.java
>  d27e52a 
>   ambari-server/src/main/java/org/apache/ambari/server/state/ServiceInfo.java 
> 5a2bf84 
>   
> ambari-server/src/main/resources/common-services/HAWQ/2.0.0/service_advisor.py
>  PRE-CREATION 
>   
> ambari-server/src/main/resources/common-services/PXF/3.0.0/service_advisor.py 
> PRE-CREATION 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py 
> 1680f21 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/stack_advisor.py 
> cf0990d 
>   ambari-server/src/main/resources/stacks/service_advisor.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/stack_advisor.py 9979e7e 
>   ambari-server/src/test/python/stacks/2.3/common/test_stack_advisor.py 
> fcb5407 
> 
> Diff: https://reviews.apache.org/r/44210/diff/
> 
> 
> Testing
> ---
> 
> Ran all the non java unit tests.  
> 
> Total run:945
> Total errors:0
> Total failures:0
> 
> Manually configured HAWQ and PXF as part of the HDP 2.3 stack and made sure 
> their service advisors were called.
> 
> 
> Thanks,
> 
> Tim Thorpe
> 
>