Re: Announcing the official Spark Job Server repo

2014-03-24 Thread Evan Chan
Andy, doesn't Marathon handle fault tolerance amongst its apps? ie if you say that N instances of an app are running, and one shuts off, then it spins up another one no? The tricky thing was that I was planning to use Akka Cluster to coordinate, but Mesos itself can be used to coordinate as

Re: Announcing the official Spark Job Server repo

2014-03-24 Thread andy petrella
Thx for answering! see inline for my thoughts (or misunderstanding ? ^^) Andy, doesn't Marathon handle fault tolerance amongst its apps? ie if you say that N instances of an app are running, and one shuts off, then it spins up another one no? Yes indeed, but my wonder is about how to know how

Re: Announcing the official Spark Job Server repo

2014-03-19 Thread Evan Chan
Andy, Yeah, we've thought of deploying this on Marathon ourselves, but we're not sure how much Mesos we're going to use yet. (Indeed if you look at bin/server_start.sh, I think I set up the PORT environment var specifically for Marathon.)This is also why we have deploy scripts which package

Re: Announcing the official Spark Job Server repo

2014-03-19 Thread Evan Chan
Matei, Maybe it's time to explore the spark-contrib idea again? Should I start a JIRA ticket? -Evan On Tue, Mar 18, 2014 at 4:04 PM, Matei Zaharia matei.zaha...@gmail.com wrote: Cool, glad to see this posted! I've added a link to it at

Re: Announcing the official Spark Job Server repo

2014-03-19 Thread Patrick Wendell
Evan - yep definitely open a JIRA. It would be nice to have a contrib repo set-up for the 1.0 release. On Tue, Mar 18, 2014 at 11:28 PM, Evan Chan e...@ooyala.com wrote: Matei, Maybe it's time to explore the spark-contrib idea again? Should I start a JIRA ticket? -Evan On Tue, Mar 18,

Re: Announcing the official Spark Job Server repo

2014-03-19 Thread Gerard Maas
this is cool +1 On Wed, Mar 19, 2014 at 6:54 PM, Patrick Wendell pwend...@gmail.com wrote: Evan - yep definitely open a JIRA. It would be nice to have a contrib repo set-up for the 1.0 release. On Tue, Mar 18, 2014 at 11:28 PM, Evan Chan e...@ooyala.com wrote: Matei, Maybe it's time

Re: Announcing the official Spark Job Server repo

2014-03-19 Thread Evan Chan
https://spark-project.atlassian.net/browse/SPARK-1283 On Wed, Mar 19, 2014 at 10:59 AM, Gerard Maas gerard.m...@gmail.com wrote: this is cool +1 On Wed, Mar 19, 2014 at 6:54 PM, Patrick Wendell pwend...@gmail.com wrote: Evan - yep definitely open a JIRA. It would be nice to have a contrib

Announcing the official Spark Job Server repo

2014-03-18 Thread Evan Chan
Dear Spark developers, Ooyala is happy to announce that we have pushed our official, Spark 0.9.0 / Scala 2.10-compatible, job server as a github repo: https://github.com/ooyala/spark-jobserver Complete with unit tests, deploy scripts, and examples. The original PR (#222) on incubator-spark is

Re: Announcing the official Spark Job Server repo

2014-03-18 Thread Henry Saputra
W00t! Thanks for releasing this, Evan. - Henry On Tue, Mar 18, 2014 at 1:51 PM, Evan Chan e...@ooyala.com wrote: Dear Spark developers, Ooyala is happy to announce that we have pushed our official, Spark 0.9.0 / Scala 2.10-compatible, job server as a github repo:

Re: Announcing the official Spark Job Server repo

2014-03-18 Thread andy petrella
tad! That's awesome. A quick question, does someone has insights regarding having such JobServers deployed using Marathon on Mesos? I'm thinking about an arch where Marathon would deploy and keep the Job Servers running along with part of the whole set of apps deployed on it regarding the