Re: Review Request 64221: AMBARI-22567 : Integrate Spark lifecycle management into AMS AD Manager.

2017-11-30 Thread Aravindan Vijayan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64221/
---

(Updated Dec. 1, 2017, 12:07 a.m.)


Review request for Ambari, Jayush Luniya and Sid Wagle.


Changes
---

Fixed review comments.


Bugs: AMBARI-22567
https://issues.apache.org/jira/browse/AMBARI-22567


Repository: ambari


Description
---

Package spark tar ball along with AD Manager jar.
Implement START/STOP of embedded spark master + worker as a step in AD Manager 
start/stop.
Build ambari-metrics-admanager RPM
Verify ability to configure and deploy AD Manager using Ambari.


Diffs (updated)
-

  
ambari-metrics/ambari-metrics-anomaly-detection-service/conf/unix/ambari-metrics-admanager
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/conf/unix/ambari-metrics-admanager.sh
 f1a1ae3 
  ambari-metrics/ambari-metrics-anomaly-detection-service/conf/unix/config.yaml 
PRE-CREATION 
  ambari-metrics/ambari-metrics-anomaly-detection-service/pom.xml c6927dd 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/assemblies/empty.xml
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/resources/config.yml
 7de06b4 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/app/AnomalyDetectionAppConfig.scala
 f9ed4b2 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/common/Range.scala
 003c18f 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/common/Season.scala
 f875e3b 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/common/SeasonType.scala
 067972c 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/common/TimeRange.scala
 50df658 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/configuration/MetricDefinitionServiceConfiguration.scala
 b560713 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/configuration/SparkConfiguration.scala
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/db/PhoenixAnomalyStoreAccessor.scala
 147d1f7 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/model/PointInTimeAnomalyInstance.scala
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/model/Range.scala
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/model/Season.scala
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/model/SeasonType.scala
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/model/TimeRange.scala
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/model/TrendAnomalyInstance.scala
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/subsystem/pointintime/PointInTimeAnomalyInstance.scala
 56ca2c1 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/subsystem/trend/TrendAnomalyInstance.scala
 7392d59 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/resources/config.yaml
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/scala/org/apache/ambari/metrics/adservice/app/AnomalyDetectionAppConfigTest.scala
 989ba21 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/scala/org/apache/ambari/metrics/adservice/app/DefaultADResourceSpecTest.scala
 e38ea40 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/scala/org/apache/ambari/metrics/adservice/common/RangeTest.scala
 b610b97 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/scala/org/apache/ambari/metrics/adservice/common/SeasonTest.scala
 a823c73 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/scala/org/apache/ambari/metrics/adservice/model/RangeTest.scala
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/scala/org/apache/ambari/metrics/adservice/model/SeasonTest.scala
 PRE-CREATION 
  ambari-metrics/ambari-metrics-assembly/pom.xml d9875ce 
  
ambari-metrics/ambari-metrics-assembly/src/main/assembly/anomaly-detection.xml 
PRE-CREATION

Re: Review Request 64221: AMBARI-22567 : Integrate Spark lifecycle management into AMS AD Manager.

2017-11-30 Thread Sid Wagle 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64221/#review192380
---


Fix it, then Ship it!





ambari-metrics/ambari-metrics-anomaly-detection-service/conf/unix/ambari-metrics-admanager
Lines 33 (patched)


If this script retruns a ret code we should be using it here.



ambari-metrics/ambari-metrics-anomaly-detection-service/conf/unix/ambari-metrics-admanager
Lines 35 (patched)


Have another if, maybe we want to add status too if spark script supports 
it.



ambari-metrics/ambari-metrics-anomaly-detection-service/conf/unix/log4j.properties
Line 25 (original), 25 (patched)


Uncommon path usage



ambari-metrics/ambari-metrics-assembly/pom.xml
Lines 662 (patched)


Need to make sure we add debian support.



ambari-metrics/ambari-metrics-assembly/src/main/assembly/anomaly-detection.xml
Lines 21 (patched)


formatting: Tabs are 4 spaces instead of 2.



ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-ad-spark-env.xml
Lines 22 (patched)


formatting



ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-ad-spark-env.xml
Lines 24 (patched)


MAster should have less memory that worker by default.



ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-ad-spark-env.xml
Lines 72 (patched)


Move lower than master and worker settings for readbility.



ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/metainfo.xml
Lines 141 (patched)


ams-admanager-spark-env



ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/ams.py
Lines 574 (patched)


Better to do fixed loc from rpm.


- Sid Wagle


On Nov. 30, 2017, 9:34 p.m., Aravindan Vijayan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64221/
> ---
> 
> (Updated Nov. 30, 2017, 9:34 p.m.)
> 
> 
> Review request for Ambari, Jayush Luniya and Sid Wagle.
> 
> 
> Bugs: AMBARI-22567
> https://issues.apache.org/jira/browse/AMBARI-22567
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> Package spark tar ball along with AD Manager jar.
> Implement START/STOP of embedded spark master + worker as a step in AD 
> Manager start/stop.
> Build ambari-metrics-admanager RPM
> Verify ability to configure and deploy AD Manager using Ambari.
> 
> 
> Diffs
> -
> 
>   
> ambari-metrics/ambari-metrics-anomaly-detection-service/conf/unix/ambari-metrics-admanager
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-anomaly-detection-service/conf/unix/ambari-metrics-admanager.sh
>  f1a1ae3 
>   
> ambari-metrics/ambari-metrics-anomaly-detection-service/conf/unix/config.yaml 
> PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-anomaly-detection-service/conf/unix/log4j.properties
>  9dba1da 
>   ambari-metrics/ambari-metrics-anomaly-detection-service/pom.xml c6927dd 
>   
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/assemblies/empty.xml
>  PRE-CREATION 
>   
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/resources/config.yml
>  7de06b4 
>   
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/app/AnomalyDetectionAppConfig.scala
>  f9ed4b2 
>   
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/common/Range.scala
>  003c18f 
>   
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/common/Season.scala
>  f875e3b 
>   
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/common/SeasonType.scala
>  067972c 
>   
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/common/TimeRange.scala
>  50df658 
>   
> ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/configuration/MetricDefinitionServiceConfiguration.scala
>  b560713 
>   
>

Re: Review Request 64221: AMBARI-22567 : Integrate Spark lifecycle management into AMS AD Manager.

2017-11-30 Thread Aravindan Vijayan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64221/
---

(Updated Nov. 30, 2017, 9:34 p.m.)


Review request for Ambari, Jayush Luniya and Sid Wagle.


Changes
---

Fixed unit tests.


Bugs: AMBARI-22567
https://issues.apache.org/jira/browse/AMBARI-22567


Repository: ambari


Description
---

Package spark tar ball along with AD Manager jar.
Implement START/STOP of embedded spark master + worker as a step in AD Manager 
start/stop.
Build ambari-metrics-admanager RPM
Verify ability to configure and deploy AD Manager using Ambari.


Diffs (updated)
-

  
ambari-metrics/ambari-metrics-anomaly-detection-service/conf/unix/ambari-metrics-admanager
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/conf/unix/ambari-metrics-admanager.sh
 f1a1ae3 
  ambari-metrics/ambari-metrics-anomaly-detection-service/conf/unix/config.yaml 
PRE-CREATION 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/conf/unix/log4j.properties
 9dba1da 
  ambari-metrics/ambari-metrics-anomaly-detection-service/pom.xml c6927dd 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/assemblies/empty.xml
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/resources/config.yml
 7de06b4 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/app/AnomalyDetectionAppConfig.scala
 f9ed4b2 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/common/Range.scala
 003c18f 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/common/Season.scala
 f875e3b 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/common/SeasonType.scala
 067972c 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/common/TimeRange.scala
 50df658 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/configuration/MetricDefinitionServiceConfiguration.scala
 b560713 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/configuration/SparkConfiguration.scala
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/db/PhoenixAnomalyStoreAccessor.scala
 147d1f7 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/model/PointInTimeAnomalyInstance.scala
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/model/Range.scala
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/model/Season.scala
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/model/SeasonType.scala
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/model/TimeRange.scala
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/model/TrendAnomalyInstance.scala
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/subsystem/pointintime/PointInTimeAnomalyInstance.scala
 56ca2c1 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/subsystem/trend/TrendAnomalyInstance.scala
 7392d59 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/resources/config.yaml
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/scala/org/apache/ambari/metrics/adservice/app/AnomalyDetectionAppConfigTest.scala
 989ba21 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/scala/org/apache/ambari/metrics/adservice/app/DefaultADResourceSpecTest.scala
 e38ea40 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/scala/org/apache/ambari/metrics/adservice/common/RangeTest.scala
 b610b97 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/scala/org/apache/ambari/metrics/adservice/common/SeasonTest.scala
 a823c73 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/scala/org/apache/ambari/metrics/adservice/model/RangeTest.scala
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/scala/org/apache/ambari/metrics/adservice/model/SeasonTest.scala
 PRE-CREATION 
  ambari-metrics/ambari-metrics-assembly/pom.xml d9875ce

Review Request 64221: AMBARI-22567 : Integrate Spark lifecycle management into AMS AD Manager.

2017-11-30 Thread Aravindan Vijayan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64221/
---

Review request for Ambari, Jayush Luniya and Sid Wagle.


Bugs: AMBARI-22567
https://issues.apache.org/jira/browse/AMBARI-22567


Repository: ambari


Description
---

Package spark tar ball along with AD Manager jar.
Implement START/STOP of embedded spark master + worker as a step in AD Manager 
start/stop.
Build ambari-metrics-admanager RPM
Verify ability to configure and deploy AD Manager using Ambari.


Diffs
-

  
ambari-metrics/ambari-metrics-anomaly-detection-service/conf/unix/ambari-metrics-admanager
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/conf/unix/ambari-metrics-admanager.sh
 f1a1ae3 
  ambari-metrics/ambari-metrics-anomaly-detection-service/conf/unix/config.yaml 
PRE-CREATION 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/conf/unix/log4j.properties
 9dba1da 
  ambari-metrics/ambari-metrics-anomaly-detection-service/pom.xml c6927dd 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/assemblies/empty.xml
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/resources/config.yaml
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/resources/config.yml
 7de06b4 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/app/AnomalyDetectionAppConfig.scala
 f9ed4b2 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/common/Range.scala
 003c18f 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/common/Season.scala
 f875e3b 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/common/SeasonType.scala
 067972c 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/common/TimeRange.scala
 50df658 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/configuration/SparkConfiguration.scala
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/db/PhoenixAnomalyStoreAccessor.scala
 147d1f7 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/model/PointInTimeAnomalyInstance.scala
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/model/Range.scala
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/model/Season.scala
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/model/SeasonType.scala
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/model/TimeRange.scala
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/model/TrendAnomalyInstance.scala
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/subsystem/pointintime/PointInTimeAnomalyInstance.scala
 56ca2c1 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/main/scala/org/apache/ambari/metrics/adservice/subsystem/trend/TrendAnomalyInstance.scala
 7392d59 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/scala/org/apache/ambari/metrics/adservice/app/AnomalyDetectionAppConfigTest.scala
 989ba21 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/scala/org/apache/ambari/metrics/adservice/app/DefaultADResourceSpecTest.scala
 e38ea40 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/scala/org/apache/ambari/metrics/adservice/common/RangeTest.scala
 b610b97 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/scala/org/apache/ambari/metrics/adservice/common/SeasonTest.scala
 a823c73 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/scala/org/apache/ambari/metrics/adservice/model/RangeTest.scala
 PRE-CREATION 
  
ambari-metrics/ambari-metrics-anomaly-detection-service/src/test/scala/org/apache/ambari/metrics/adservice/model/SeasonTest.scala
 PRE-CREATION 
  ambari-metrics/ambari-metrics-assembly/pom.xml d9875ce 
  
ambari-metrics/ambari-metrics-assembly/src/main/assembly/anomaly-detection.xml 
PRE-CREATION 
  ambari-metrics/pom.xml 4848084 
  
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/configuration/ams-ad-spark-env.xml
 PRE-CREATION