Re: Review Request 28917: Spark Backend to Resource Manager

2014-12-17 Thread Chris Mattmann 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28917/#review65466
---


This is good enough to ship, so ship it, but consider making the updates I 
requested at least after.

- Chris Mattmann


On Dec. 17, 2014, 7:13 p.m., Michael Starch wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28917/
> ---
> 
> (Updated Dec. 17, 2014, 7:13 p.m.)
> 
> 
> Review request for oodt and Chris Mattmann.
> 
> 
> Repository: oodt
> 
> 
> Description
> ---
> 
> This change allows Spark jobs to be run in the resource manager.  It 
> integrates with a generic Spark daemon, so spark can be run locally, as a 
> test spark cluster, and in mesos (not affiliated with the resouce manager 
> mesos integration).
> 
> This is a solution to: https://issues.apache.org/jira/browse/OODT-780
> 
> 
> Diffs
> -
> 
>   trunk/core/pom.xml 1644527 
>   trunk/resource/pom.xml 1644527 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/examples/NoSparkFilePalindromeExample.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/examples/PalindromeUtils.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/examples/SparkFilePalindromeExample.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/examples/StreamingPalindromeExample.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/scheduler/SparkScheduler.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/scheduler/SparkSchedulerFactory.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/structs/SparkInstance.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/structs/StreamingInstance.java
>  PRE-CREATION 
>   trunk/resource/src/main/resources/examples/jobs/exPalindrome.xml 
> PRE-CREATION 
>   trunk/resource/src/main/resources/examples/jobs/exSparkJob.xml PRE-CREATION 
>   trunk/resource/src/main/resources/examples/jobs/exSparkPalindrome.xml 
> PRE-CREATION 
>   trunk/resource/src/main/resources/examples/jobs/exStreamingPalindrome.xml 
> PRE-CREATION 
>   trunk/resource/src/main/resources/resource.properties 1644527 
>   
> trunk/resource/src/main/scala/org/apache/oodt/cas/resource/examples/ScalaHelloWorld.scala
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/28917/diff/
> 
> 
> Testing
> ---
> 
> Demonstration created, demonstration jobs run and benchmarks taken.
> 
> 
> Thanks,
> 
> Michael Starch
> 
>

Re: Review Request 28917: Spark Backend to Resource Manager

2014-12-17 Thread Chris Mattmann 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28917/#review65465
---

Ship it!


Ship It!

- Chris Mattmann


On Dec. 17, 2014, 7:13 p.m., Michael Starch wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28917/
> ---
> 
> (Updated Dec. 17, 2014, 7:13 p.m.)
> 
> 
> Review request for oodt and Chris Mattmann.
> 
> 
> Repository: oodt
> 
> 
> Description
> ---
> 
> This change allows Spark jobs to be run in the resource manager.  It 
> integrates with a generic Spark daemon, so spark can be run locally, as a 
> test spark cluster, and in mesos (not affiliated with the resouce manager 
> mesos integration).
> 
> This is a solution to: https://issues.apache.org/jira/browse/OODT-780
> 
> 
> Diffs
> -
> 
>   trunk/core/pom.xml 1644527 
>   trunk/resource/pom.xml 1644527 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/examples/NoSparkFilePalindromeExample.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/examples/PalindromeUtils.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/examples/SparkFilePalindromeExample.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/examples/StreamingPalindromeExample.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/scheduler/SparkScheduler.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/scheduler/SparkSchedulerFactory.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/structs/SparkInstance.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/structs/StreamingInstance.java
>  PRE-CREATION 
>   trunk/resource/src/main/resources/examples/jobs/exPalindrome.xml 
> PRE-CREATION 
>   trunk/resource/src/main/resources/examples/jobs/exSparkJob.xml PRE-CREATION 
>   trunk/resource/src/main/resources/examples/jobs/exSparkPalindrome.xml 
> PRE-CREATION 
>   trunk/resource/src/main/resources/examples/jobs/exStreamingPalindrome.xml 
> PRE-CREATION 
>   trunk/resource/src/main/resources/resource.properties 1644527 
>   
> trunk/resource/src/main/scala/org/apache/oodt/cas/resource/examples/ScalaHelloWorld.scala
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/28917/diff/
> 
> 
> Testing
> ---
> 
> Demonstration created, demonstration jobs run and benchmarks taken.
> 
> 
> Thanks,
> 
> Michael Starch
> 
>

Re: Review Request 28917: Spark Backend to Resource Manager

2014-12-17 Thread Chris Mattmann 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28917/#review65463
---



trunk/resource/src/main/java/org/apache/oodt/cas/resource/examples/NoSparkFilePalindromeExample.java


should read //JDK imports (and then //OODT imports above the OODT ones)



trunk/resource/src/main/java/org/apache/oodt/cas/resource/examples/SparkFilePalindromeExample.java


JDK imports



trunk/resource/src/main/java/org/apache/oodt/cas/resource/examples/StreamingPalindromeExample.java


JDK imports


- Chris Mattmann


On Dec. 17, 2014, 7:13 p.m., Michael Starch wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28917/
> ---
> 
> (Updated Dec. 17, 2014, 7:13 p.m.)
> 
> 
> Review request for oodt and Chris Mattmann.
> 
> 
> Repository: oodt
> 
> 
> Description
> ---
> 
> This change allows Spark jobs to be run in the resource manager.  It 
> integrates with a generic Spark daemon, so spark can be run locally, as a 
> test spark cluster, and in mesos (not affiliated with the resouce manager 
> mesos integration).
> 
> This is a solution to: https://issues.apache.org/jira/browse/OODT-780
> 
> 
> Diffs
> -
> 
>   trunk/core/pom.xml 1644527 
>   trunk/resource/pom.xml 1644527 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/examples/NoSparkFilePalindromeExample.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/examples/PalindromeUtils.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/examples/SparkFilePalindromeExample.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/examples/StreamingPalindromeExample.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/scheduler/SparkScheduler.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/scheduler/SparkSchedulerFactory.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/structs/SparkInstance.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/structs/StreamingInstance.java
>  PRE-CREATION 
>   trunk/resource/src/main/resources/examples/jobs/exPalindrome.xml 
> PRE-CREATION 
>   trunk/resource/src/main/resources/examples/jobs/exSparkJob.xml PRE-CREATION 
>   trunk/resource/src/main/resources/examples/jobs/exSparkPalindrome.xml 
> PRE-CREATION 
>   trunk/resource/src/main/resources/examples/jobs/exStreamingPalindrome.xml 
> PRE-CREATION 
>   trunk/resource/src/main/resources/resource.properties 1644527 
>   
> trunk/resource/src/main/scala/org/apache/oodt/cas/resource/examples/ScalaHelloWorld.scala
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/28917/diff/
> 
> 
> Testing
> ---
> 
> Demonstration created, demonstration jobs run and benchmarks taken.
> 
> 
> Thanks,
> 
> Michael Starch
> 
>

Re: Review Request 28917: Spark Backend to Resource Manager

2014-12-17 Thread Michael Starch 


> On Dec. 11, 2014, 1:39 a.m., Chris Mattmann wrote:
> > trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/StreamingInstance.java,
> >  line 1
> > 
> >
> > batchmgr pkg

Put in structs like SparkInstance.  This is where it belongs, as it is a struct.


> On Dec. 11, 2014, 1:39 a.m., Chris Mattmann wrote:
> > trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/examples/PalindromeUtils.java,
> >  line 1
> > 
> >
> > o.a.oodt.cas.resource.util

Placed in Examples, as it is shared-code for the examples.  It is only intended 
to support examples.  If the class needs to be renamed, that can be done.


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28917/#review64662
---


On Dec. 17, 2014, 7:13 p.m., Michael Starch wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28917/
> ---
> 
> (Updated Dec. 17, 2014, 7:13 p.m.)
> 
> 
> Review request for oodt and Chris Mattmann.
> 
> 
> Repository: oodt
> 
> 
> Description
> ---
> 
> This change allows Spark jobs to be run in the resource manager.  It 
> integrates with a generic Spark daemon, so spark can be run locally, as a 
> test spark cluster, and in mesos (not affiliated with the resouce manager 
> mesos integration).
> 
> This is a solution to: https://issues.apache.org/jira/browse/OODT-780
> 
> 
> Diffs
> -
> 
>   trunk/core/pom.xml 1644527 
>   trunk/resource/pom.xml 1644527 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/examples/NoSparkFilePalindromeExample.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/examples/PalindromeUtils.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/examples/SparkFilePalindromeExample.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/examples/StreamingPalindromeExample.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/scheduler/SparkScheduler.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/scheduler/SparkSchedulerFactory.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/structs/SparkInstance.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/structs/StreamingInstance.java
>  PRE-CREATION 
>   trunk/resource/src/main/resources/examples/jobs/exPalindrome.xml 
> PRE-CREATION 
>   trunk/resource/src/main/resources/examples/jobs/exSparkJob.xml PRE-CREATION 
>   trunk/resource/src/main/resources/examples/jobs/exSparkPalindrome.xml 
> PRE-CREATION 
>   trunk/resource/src/main/resources/examples/jobs/exStreamingPalindrome.xml 
> PRE-CREATION 
>   trunk/resource/src/main/resources/resource.properties 1644527 
>   
> trunk/resource/src/main/scala/org/apache/oodt/cas/resource/examples/ScalaHelloWorld.scala
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/28917/diff/
> 
> 
> Testing
> ---
> 
> Demonstration created, demonstration jobs run and benchmarks taken.
> 
> 
> Thanks,
> 
> Michael Starch
> 
>

Re: Review Request 28917: Spark Backend to Resource Manager

2014-12-17 Thread Michael Starch 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28917/
---

(Updated Dec. 17, 2014, 7:13 p.m.)


Review request for oodt and Chris Mattmann.


Changes
---

Sorted packages


Repository: oodt


Description
---

This change allows Spark jobs to be run in the resource manager.  It integrates 
with a generic Spark daemon, so spark can be run locally, as a test spark 
cluster, and in mesos (not affiliated with the resouce manager mesos 
integration).

This is a solution to: https://issues.apache.org/jira/browse/OODT-780


Diffs (updated)
-

  trunk/core/pom.xml 1644527 
  trunk/resource/pom.xml 1644527 
  
trunk/resource/src/main/java/org/apache/oodt/cas/resource/examples/NoSparkFilePalindromeExample.java
 PRE-CREATION 
  
trunk/resource/src/main/java/org/apache/oodt/cas/resource/examples/PalindromeUtils.java
 PRE-CREATION 
  
trunk/resource/src/main/java/org/apache/oodt/cas/resource/examples/SparkFilePalindromeExample.java
 PRE-CREATION 
  
trunk/resource/src/main/java/org/apache/oodt/cas/resource/examples/StreamingPalindromeExample.java
 PRE-CREATION 
  
trunk/resource/src/main/java/org/apache/oodt/cas/resource/scheduler/SparkScheduler.java
 PRE-CREATION 
  
trunk/resource/src/main/java/org/apache/oodt/cas/resource/scheduler/SparkSchedulerFactory.java
 PRE-CREATION 
  
trunk/resource/src/main/java/org/apache/oodt/cas/resource/structs/SparkInstance.java
 PRE-CREATION 
  
trunk/resource/src/main/java/org/apache/oodt/cas/resource/structs/StreamingInstance.java
 PRE-CREATION 
  trunk/resource/src/main/resources/examples/jobs/exPalindrome.xml PRE-CREATION 
  trunk/resource/src/main/resources/examples/jobs/exSparkJob.xml PRE-CREATION 
  trunk/resource/src/main/resources/examples/jobs/exSparkPalindrome.xml 
PRE-CREATION 
  trunk/resource/src/main/resources/examples/jobs/exStreamingPalindrome.xml 
PRE-CREATION 
  trunk/resource/src/main/resources/resource.properties 1644527 
  
trunk/resource/src/main/scala/org/apache/oodt/cas/resource/examples/ScalaHelloWorld.scala
 PRE-CREATION 

Diff: https://reviews.apache.org/r/28917/diff/


Testing
---

Demonstration created, demonstration jobs run and benchmarks taken.


Thanks,

Michael Starch

Re: Review Request 28917: Spark Backend to Resource Manager

2014-12-11 Thread Chris Mattmann 


> On Dec. 11, 2014, 1:39 a.m., Chris Mattmann wrote:
> >
> 
> Michael Starch wrote:
> I put all of the Spark stuff in a spark package so that it is seperated 
> from the rest of the resource manager. It is an inter-dependent set of code. 
> Do you really want me to sort its components out in to the various other 
> subdirectories?  If so, I can.

Thanks Mike, yes I understand that. And yes, in fact, I'd like you to sort them 
into the various sub directories b/c that is the architecture of the resource 
manager. It shouldn't matter that this is Spark, etc. We have an XML-RPC and 
Ganglia, etc., set of plugins that are part of the architectural sub folders 
for the resource manager. We shouldn't introduce new top level directories and 
folders unless we are actually creating new extension points, which we are not 
here. We are simply implementing plugins that rely on Spark and Mesos, and 
those plugins should fit in the extension point folders for Resource Manager as 
they are currently defined.


- Chris


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28917/#review64662
---


On Dec. 10, 2014, 9:38 p.m., Michael Starch wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28917/
> ---
> 
> (Updated Dec. 10, 2014, 9:38 p.m.)
> 
> 
> Review request for oodt and Chris Mattmann.
> 
> 
> Repository: oodt
> 
> 
> Description
> ---
> 
> This change allows Spark jobs to be run in the resource manager.  It 
> integrates with a generic Spark daemon, so spark can be run locally, as a 
> test spark cluster, and in mesos (not affiliated with the resouce manager 
> mesos integration).
> 
> This is a solution to: https://issues.apache.org/jira/browse/OODT-780
> 
> 
> Diffs
> -
> 
>   trunk/core/pom.xml 1644524 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/SparkInstance.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/SparkScheduler.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/SparkSchedulerFactory.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/StreamingInstance.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/examples/NoSparkFilePalindromeExample.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/examples/PalindromeUtils.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/examples/SparkFilePalindromeExample.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/examples/StreamingPalindromeExample.java
>  PRE-CREATION 
>   trunk/resource/src/main/resources/examples/jobs/exPalindrome.xml 
> PRE-CREATION 
>   trunk/resource/src/main/resources/examples/jobs/exSparkJob.xml PRE-CREATION 
>   trunk/resource/src/main/resources/examples/jobs/exSparkPalindrome.xml 
> PRE-CREATION 
>   trunk/resource/src/main/resources/examples/jobs/exStreamingPalindrome.xml 
> PRE-CREATION 
>   trunk/resource/src/main/resources/resource.properties 1644524 
>   
> trunk/resource/src/main/scala/org/apache/oodt/cas/resource/examples/ScalaHelloWorld.scala
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/28917/diff/
> 
> 
> Testing
> ---
> 
> Demonstration created, demonstration jobs run and benchmarks taken.
> 
> 
> Thanks,
> 
> Michael Starch
> 
>

Re: Review Request 28917: Spark Backend to Resource Manager

2014-12-11 Thread Michael Starch 


> On Dec. 11, 2014, 1:39 a.m., Chris Mattmann wrote:
> >

I put all of the Spark stuff in a spark package so that it is seperated from 
the rest of the resource manager. It is an inter-dependent set of code. Do you 
really want me to sort its components out in to the various other 
subdirectories?  If so, I can.


- Michael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28917/#review64662
---


On Dec. 10, 2014, 9:38 p.m., Michael Starch wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28917/
> ---
> 
> (Updated Dec. 10, 2014, 9:38 p.m.)
> 
> 
> Review request for oodt and Chris Mattmann.
> 
> 
> Repository: oodt
> 
> 
> Description
> ---
> 
> This change allows Spark jobs to be run in the resource manager.  It 
> integrates with a generic Spark daemon, so spark can be run locally, as a 
> test spark cluster, and in mesos (not affiliated with the resouce manager 
> mesos integration).
> 
> This is a solution to: https://issues.apache.org/jira/browse/OODT-780
> 
> 
> Diffs
> -
> 
>   trunk/core/pom.xml 1644524 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/SparkInstance.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/SparkScheduler.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/SparkSchedulerFactory.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/StreamingInstance.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/examples/NoSparkFilePalindromeExample.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/examples/PalindromeUtils.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/examples/SparkFilePalindromeExample.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/examples/StreamingPalindromeExample.java
>  PRE-CREATION 
>   trunk/resource/src/main/resources/examples/jobs/exPalindrome.xml 
> PRE-CREATION 
>   trunk/resource/src/main/resources/examples/jobs/exSparkJob.xml PRE-CREATION 
>   trunk/resource/src/main/resources/examples/jobs/exSparkPalindrome.xml 
> PRE-CREATION 
>   trunk/resource/src/main/resources/examples/jobs/exStreamingPalindrome.xml 
> PRE-CREATION 
>   trunk/resource/src/main/resources/resource.properties 1644524 
>   
> trunk/resource/src/main/scala/org/apache/oodt/cas/resource/examples/ScalaHelloWorld.scala
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/28917/diff/
> 
> 
> Testing
> ---
> 
> Demonstration created, demonstration jobs run and benchmarks taken.
> 
> 
> Thanks,
> 
> Michael Starch
> 
>

Re: Review Request 28917: Spark Backend to Resource Manager

2014-12-10 Thread Chris Mattmann 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28917/#review64662
---



trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/SparkInstance.java


o.a.oodt.cas.resource.structs pkg



trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/SparkScheduler.java


scheduler pkg



trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/SparkSchedulerFactory.java


scheduler pkg



trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/StreamingInstance.java


batchmgr pkg



trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/examples/NoSparkFilePalindromeExample.java


resmgr examples package



trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/examples/PalindromeUtils.java


o.a.oodt.cas.resource.util



trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/examples/SparkFilePalindromeExample.java


resource.examples



trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/examples/StreamingPalindromeExample.java


resource.examples


- Chris Mattmann


On Dec. 10, 2014, 9:38 p.m., Michael Starch wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28917/
> ---
> 
> (Updated Dec. 10, 2014, 9:38 p.m.)
> 
> 
> Review request for oodt and Chris Mattmann.
> 
> 
> Repository: oodt
> 
> 
> Description
> ---
> 
> This change allows Spark jobs to be run in the resource manager.  It 
> integrates with a generic Spark daemon, so spark can be run locally, as a 
> test spark cluster, and in mesos (not affiliated with the resouce manager 
> mesos integration).
> 
> This is a solution to: https://issues.apache.org/jira/browse/OODT-780
> 
> 
> Diffs
> -
> 
>   trunk/core/pom.xml 1644524 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/SparkInstance.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/SparkScheduler.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/SparkSchedulerFactory.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/StreamingInstance.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/examples/NoSparkFilePalindromeExample.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/examples/PalindromeUtils.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/examples/SparkFilePalindromeExample.java
>  PRE-CREATION 
>   
> trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/examples/StreamingPalindromeExample.java
>  PRE-CREATION 
>   trunk/resource/src/main/resources/examples/jobs/exPalindrome.xml 
> PRE-CREATION 
>   trunk/resource/src/main/resources/examples/jobs/exSparkJob.xml PRE-CREATION 
>   trunk/resource/src/main/resources/examples/jobs/exSparkPalindrome.xml 
> PRE-CREATION 
>   trunk/resource/src/main/resources/examples/jobs/exStreamingPalindrome.xml 
> PRE-CREATION 
>   trunk/resource/src/main/resources/resource.properties 1644524 
>   
> trunk/resource/src/main/scala/org/apache/oodt/cas/resource/examples/ScalaHelloWorld.scala
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/28917/diff/
> 
> 
> Testing
> ---
> 
> Demonstration created, demonstration jobs run and benchmarks taken.
> 
> 
> Thanks,
> 
> Michael Starch
> 
>

Review Request 28917: Spark Backend to Resource Manager

2014-12-10 Thread Michael Starch 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28917/
---

Review request for oodt and Chris Mattmann.


Repository: oodt


Description
---

This change allows Spark jobs to be run in the resource manager.  It integrates 
with a generic Spark daemon, so spark can be run locally, as a test spark 
cluster, and in mesos (not affiliated with the resouce manager mesos 
integration).

This is a solution to: https://issues.apache.org/jira/browse/OODT-780


Diffs
-

  trunk/core/pom.xml 1644524 
  
trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/SparkInstance.java
 PRE-CREATION 
  
trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/SparkScheduler.java
 PRE-CREATION 
  
trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/SparkSchedulerFactory.java
 PRE-CREATION 
  
trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/StreamingInstance.java
 PRE-CREATION 
  
trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/examples/NoSparkFilePalindromeExample.java
 PRE-CREATION 
  
trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/examples/PalindromeUtils.java
 PRE-CREATION 
  
trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/examples/SparkFilePalindromeExample.java
 PRE-CREATION 
  
trunk/resource/src/main/java/org/apache/oodt/cas/resource/spark/examples/StreamingPalindromeExample.java
 PRE-CREATION 
  trunk/resource/src/main/resources/examples/jobs/exPalindrome.xml PRE-CREATION 
  trunk/resource/src/main/resources/examples/jobs/exSparkJob.xml PRE-CREATION 
  trunk/resource/src/main/resources/examples/jobs/exSparkPalindrome.xml 
PRE-CREATION 
  trunk/resource/src/main/resources/examples/jobs/exStreamingPalindrome.xml 
PRE-CREATION 
  trunk/resource/src/main/resources/resource.properties 1644524 
  
trunk/resource/src/main/scala/org/apache/oodt/cas/resource/examples/ScalaHelloWorld.scala
 PRE-CREATION 

Diff: https://reviews.apache.org/r/28917/diff/


Testing
---

Demonstration created, demonstration jobs run and benchmarks taken.


Thanks,

Michael Starch