Re: Announcing the official Spark Job Server repo

[andy petrella]

Thx for answering!
see inline for my thoughts (or misunderstanding ? ^^)

Andy, doesn't Marathon handle fault tolerance amongst its apps?  ie if
> you say that N instances of an app are running, and one shuts off,
> then it spins up another one no?
>
Yes indeed, but my wonder is about how to know how many instances we need?
You know, it's purely dependent of the amount of resource consumed by the
drivers, so it fluctuates with the time.
In my actual thinking, the JobServer could ask mesos for resources
depending on the amount of resources of its currently managed job list (so
themselves should be able to deliver such info). Then (perhaps) marathon
can be (hot-)tuned to maintain N+M or N-M instances depending of the
load... But maybe am I crossing some boundaries, the ones with auto-scaling
:-/


>
> The tricky thing was that I was planning to use Akka Cluster to
> coordinate, but Mesos itself can be used to coordinate as well, which
> is an overlap/ but I didn't want ot make job server HA just
> reliant only on Mesos...

You mean using Akka cluster to dispatch jobs on the managed (Job Server)
nodes? That's something actually interesting as well, but I guess would
require some duplicated work with what Mesos or Yarn are doing (that is
resources management) right?


> Anyways we can discuss offline if needed.
>
Definitively, let's stop polluting the list !!!


C ya
andy

>
> On Thu, Mar 20, 2014 at 1:35 AM, andy petrella 
> wrote:
> > Heya,
> > That's cool you've already hacked something for this in the scripts!
> >
> > I have a related question, how would it work actually. I mean, to have
> this
> > Job Server fault tolerant using Marathon, I would guess that it will need
> > to be itself a Mesos framework, and able to publish its resources needs.
> > And also, for that, the Job Server has to be aware of the resources
> needed
> > by the Spark drivers that it will run, which is not as easy to guess,
> > unless it is provided by the job itself?
> >
> > I didn't checked the Job Server deep enough so it might be already the
> case
> > (or I'm expressing something completely dumb ^^).
> >
> > For sure, we'll try to share it when we'll reach this point to deploy
> using
> > marathon (should be planned for April)
> >
> > greetz and again, Nice Work Evan!
> >
> > Ndi
> >
> > On Wed, Mar 19, 2014 at 7:27 AM, Evan Chan  wrote:
> >
> >> Andy,
> >>
> >> Yeah, we've thought of deploying this on Marathon ourselves, but we're
> >> not sure how much Mesos we're going to use yet.   (Indeed if you look
> >> at bin/server_start.sh, I think I set up the PORT environment var
> >> specifically for Marathon.)This is also why we have deploy scripts
> >> which package into .tar.gz, again for Mesos deployment.
> >>
> >> If you do try this, please let us know.  :)
> >>
> >> -Evan
> >>
> >>
> >> On Tue, Mar 18, 2014 at 3:57 PM, andy petrella  >
> >> wrote:
> >> > tad! That's awesome.
> >> >
> >> > A quick question, does someone has insights regarding having such
> >> > JobServers deployed using Marathon on Mesos?
> >> >
> >> > I'm thinking about an arch where Marathon would deploy and keep the
> Job
> >> > Servers running along with part of the whole set of apps deployed on
> it
> >> > regarding the resources needed (à la Jenkins).
> >> >
> >> > Any idea is welcome.
> >> >
> >> > Back to the news, Evan + Ooyala team: Great Job again.
> >> >
> >> > andy
> >> >
> >> > On Tue, Mar 18, 2014 at 11:39 PM, Henry Saputra <
> henry.sapu...@gmail.com
> >> >wrote:
> >> >
> >> >> W00t!
> >> >>
> >> >> Thanks for releasing this, Evan.
> >> >>
> >> >> - Henry
> >> >>
> >> >> On Tue, Mar 18, 2014 at 1:51 PM, Evan Chan  wrote:
> >> >> > Dear Spark developers,
> >> >> >
> >> >> > Ooyala is happy to announce that we have pushed our official, Spark
> >> >> > 0.9.0 / Scala 2.10-compatible, job server as a github repo:
> >> >> >
> >> >> > https://github.com/ooyala/spark-jobserver
> >> >> >
> >> >> > Complete with unit tests, deploy scripts, and examples.
> >> >> >
> >> >> > The original PR (#222) on incubator-spark is now closed.
> >> >> >
> >> >> > Please have a look; pull requests are very welcome.
> >> >> > --
> >> >> > --
> >> >> > Evan Chan
> >> >> > Staff Engineer
> >> >> > e...@ooyala.com  |
> >> >>
> >>
> >>
> >>
> >> --
> >> --
> >> Evan Chan
> >> Staff Engineer
> >> e...@ooyala.com  |
> >>
>
>
>
> --
> --
> Evan Chan
> Staff Engineer
> e...@ooyala.com  |
>

Re: Announcing the official Spark Job Server repo

[Evan Chan]

Andy, doesn't Marathon handle fault tolerance amongst its apps?  ie if
you say that N instances of an app are running, and one shuts off,
then it spins up another one no?

The tricky thing was that I was planning to use Akka Cluster to
coordinate, but Mesos itself can be used to coordinate as well, which
is an overlap/ but I didn't want ot make job server HA just
reliant only on Mesos... Anyways we can discuss offline if needed.

On Thu, Mar 20, 2014 at 1:35 AM, andy petrella  wrote:
> Heya,
> That's cool you've already hacked something for this in the scripts!
>
> I have a related question, how would it work actually. I mean, to have this
> Job Server fault tolerant using Marathon, I would guess that it will need
> to be itself a Mesos framework, and able to publish its resources needs.
> And also, for that, the Job Server has to be aware of the resources needed
> by the Spark drivers that it will run, which is not as easy to guess,
> unless it is provided by the job itself?
>
> I didn't checked the Job Server deep enough so it might be already the case
> (or I'm expressing something completely dumb ^^).
>
> For sure, we'll try to share it when we'll reach this point to deploy using
> marathon (should be planned for April)
>
> greetz and again, Nice Work Evan!
>
> Ndi
>
> On Wed, Mar 19, 2014 at 7:27 AM, Evan Chan  wrote:
>
>> Andy,
>>
>> Yeah, we've thought of deploying this on Marathon ourselves, but we're
>> not sure how much Mesos we're going to use yet.   (Indeed if you look
>> at bin/server_start.sh, I think I set up the PORT environment var
>> specifically for Marathon.)This is also why we have deploy scripts
>> which package into .tar.gz, again for Mesos deployment.
>>
>> If you do try this, please let us know.  :)
>>
>> -Evan
>>
>>
>> On Tue, Mar 18, 2014 at 3:57 PM, andy petrella 
>> wrote:
>> > tad! That's awesome.
>> >
>> > A quick question, does someone has insights regarding having such
>> > JobServers deployed using Marathon on Mesos?
>> >
>> > I'm thinking about an arch where Marathon would deploy and keep the Job
>> > Servers running along with part of the whole set of apps deployed on it
>> > regarding the resources needed (à la Jenkins).
>> >
>> > Any idea is welcome.
>> >
>> > Back to the news, Evan + Ooyala team: Great Job again.
>> >
>> > andy
>> >
>> > On Tue, Mar 18, 2014 at 11:39 PM, Henry Saputra > >wrote:
>> >
>> >> W00t!
>> >>
>> >> Thanks for releasing this, Evan.
>> >>
>> >> - Henry
>> >>
>> >> On Tue, Mar 18, 2014 at 1:51 PM, Evan Chan  wrote:
>> >> > Dear Spark developers,
>> >> >
>> >> > Ooyala is happy to announce that we have pushed our official, Spark
>> >> > 0.9.0 / Scala 2.10-compatible, job server as a github repo:
>> >> >
>> >> > https://github.com/ooyala/spark-jobserver
>> >> >
>> >> > Complete with unit tests, deploy scripts, and examples.
>> >> >
>> >> > The original PR (#222) on incubator-spark is now closed.
>> >> >
>> >> > Please have a look; pull requests are very welcome.
>> >> > --
>> >> > --
>> >> > Evan Chan
>> >> > Staff Engineer
>> >> > e...@ooyala.com  |
>> >>
>>
>>
>>
>> --
>> --
>> Evan Chan
>> Staff Engineer
>> e...@ooyala.com  |
>>



-- 
--
Evan Chan
Staff Engineer
e...@ooyala.com  |

Re: Announcing the official Spark Job Server repo

[andy petrella]

Heya,
That's cool you've already hacked something for this in the scripts!

I have a related question, how would it work actually. I mean, to have this
Job Server fault tolerant using Marathon, I would guess that it will need
to be itself a Mesos framework, and able to publish its resources needs.
And also, for that, the Job Server has to be aware of the resources needed
by the Spark drivers that it will run, which is not as easy to guess,
unless it is provided by the job itself?

I didn't checked the Job Server deep enough so it might be already the case
(or I'm expressing something completely dumb ^^).

For sure, we'll try to share it when we'll reach this point to deploy using
marathon (should be planned for April)

greetz and again, Nice Work Evan!

Ndi

On Wed, Mar 19, 2014 at 7:27 AM, Evan Chan  wrote:

> Andy,
>
> Yeah, we've thought of deploying this on Marathon ourselves, but we're
> not sure how much Mesos we're going to use yet.   (Indeed if you look
> at bin/server_start.sh, I think I set up the PORT environment var
> specifically for Marathon.)This is also why we have deploy scripts
> which package into .tar.gz, again for Mesos deployment.
>
> If you do try this, please let us know.  :)
>
> -Evan
>
>
> On Tue, Mar 18, 2014 at 3:57 PM, andy petrella 
> wrote:
> > tad! That's awesome.
> >
> > A quick question, does someone has insights regarding having such
> > JobServers deployed using Marathon on Mesos?
> >
> > I'm thinking about an arch where Marathon would deploy and keep the Job
> > Servers running along with part of the whole set of apps deployed on it
> > regarding the resources needed (à la Jenkins).
> >
> > Any idea is welcome.
> >
> > Back to the news, Evan + Ooyala team: Great Job again.
> >
> > andy
> >
> > On Tue, Mar 18, 2014 at 11:39 PM, Henry Saputra  >wrote:
> >
> >> W00t!
> >>
> >> Thanks for releasing this, Evan.
> >>
> >> - Henry
> >>
> >> On Tue, Mar 18, 2014 at 1:51 PM, Evan Chan  wrote:
> >> > Dear Spark developers,
> >> >
> >> > Ooyala is happy to announce that we have pushed our official, Spark
> >> > 0.9.0 / Scala 2.10-compatible, job server as a github repo:
> >> >
> >> > https://github.com/ooyala/spark-jobserver
> >> >
> >> > Complete with unit tests, deploy scripts, and examples.
> >> >
> >> > The original PR (#222) on incubator-spark is now closed.
> >> >
> >> > Please have a look; pull requests are very welcome.
> >> > --
> >> > --
> >> > Evan Chan
> >> > Staff Engineer
> >> > e...@ooyala.com  |
> >>
>
>
>
> --
> --
> Evan Chan
> Staff Engineer
> e...@ooyala.com  |
>

Re: Announcing the official Spark Job Server repo

[Christopher Nguyen]

+1, Evan et al.

--
Christopher T. Nguyen
Co-founder & CEO, Adatao 
linkedin.com/in/ctnguyen



On Tue, Mar 18, 2014 at 1:51 PM, Evan Chan  wrote:

> Dear Spark developers,
>
> Ooyala is happy to announce that we have pushed our official, Spark
> 0.9.0 / Scala 2.10-compatible, job server as a github repo:
>
> https://github.com/ooyala/spark-jobserver
>
> Complete with unit tests, deploy scripts, and examples.
>
> The original PR (#222) on incubator-spark is now closed.
>
> Please have a look; pull requests are very welcome.
> --
> --
> Evan Chan
> Staff Engineer
> e...@ooyala.com  |
>

Re: Announcing the official Spark Job Server repo

[Evan Chan]

https://spark-project.atlassian.net/browse/SPARK-1283

On Wed, Mar 19, 2014 at 10:59 AM, Gerard Maas  wrote:
> this is cool +1
>
>
> On Wed, Mar 19, 2014 at 6:54 PM, Patrick Wendell  wrote:
>
>> Evan - yep definitely open a JIRA. It would be nice to have a contrib
>> repo set-up for the 1.0 release.
>>
>> On Tue, Mar 18, 2014 at 11:28 PM, Evan Chan  wrote:
>> > Matei,
>> >
>> > Maybe it's time to explore the spark-contrib idea again?   Should I
>> > start a JIRA ticket?
>> >
>> > -Evan
>> >
>> >
>> > On Tue, Mar 18, 2014 at 4:04 PM, Matei Zaharia 
>> wrote:
>> >> Cool, glad to see this posted! I've added a link to it at
>> https://cwiki.apache.org/confluence/display/SPARK/Powered+By+Spark.
>> >>
>> >> Matei
>> >>
>> >> On Mar 18, 2014, at 1:51 PM, Evan Chan  wrote:
>> >>
>> >>> Dear Spark developers,
>> >>>
>> >>> Ooyala is happy to announce that we have pushed our official, Spark
>> >>> 0.9.0 / Scala 2.10-compatible, job server as a github repo:
>> >>>
>> >>> https://github.com/ooyala/spark-jobserver
>> >>>
>> >>> Complete with unit tests, deploy scripts, and examples.
>> >>>
>> >>> The original PR (#222) on incubator-spark is now closed.
>> >>>
>> >>> Please have a look; pull requests are very welcome.
>> >>> --
>> >>> --
>> >>> Evan Chan
>> >>> Staff Engineer
>> >>> e...@ooyala.com  |
>> >>
>> >
>> >
>> >
>> > --
>> > --
>> > Evan Chan
>> > Staff Engineer
>> > e...@ooyala.com  |
>>



-- 
--
Evan Chan
Staff Engineer
e...@ooyala.com  |

Re: Announcing the official Spark Job Server repo

[Gerard Maas]

this is cool +1


On Wed, Mar 19, 2014 at 6:54 PM, Patrick Wendell  wrote:

> Evan - yep definitely open a JIRA. It would be nice to have a contrib
> repo set-up for the 1.0 release.
>
> On Tue, Mar 18, 2014 at 11:28 PM, Evan Chan  wrote:
> > Matei,
> >
> > Maybe it's time to explore the spark-contrib idea again?   Should I
> > start a JIRA ticket?
> >
> > -Evan
> >
> >
> > On Tue, Mar 18, 2014 at 4:04 PM, Matei Zaharia 
> wrote:
> >> Cool, glad to see this posted! I've added a link to it at
> https://cwiki.apache.org/confluence/display/SPARK/Powered+By+Spark.
> >>
> >> Matei
> >>
> >> On Mar 18, 2014, at 1:51 PM, Evan Chan  wrote:
> >>
> >>> Dear Spark developers,
> >>>
> >>> Ooyala is happy to announce that we have pushed our official, Spark
> >>> 0.9.0 / Scala 2.10-compatible, job server as a github repo:
> >>>
> >>> https://github.com/ooyala/spark-jobserver
> >>>
> >>> Complete with unit tests, deploy scripts, and examples.
> >>>
> >>> The original PR (#222) on incubator-spark is now closed.
> >>>
> >>> Please have a look; pull requests are very welcome.
> >>> --
> >>> --
> >>> Evan Chan
> >>> Staff Engineer
> >>> e...@ooyala.com  |
> >>
> >
> >
> >
> > --
> > --
> > Evan Chan
> > Staff Engineer
> > e...@ooyala.com  |
>

Re: Announcing the official Spark Job Server repo

[Patrick Wendell]

Evan - yep definitely open a JIRA. It would be nice to have a contrib
repo set-up for the 1.0 release.

On Tue, Mar 18, 2014 at 11:28 PM, Evan Chan  wrote:
> Matei,
>
> Maybe it's time to explore the spark-contrib idea again?   Should I
> start a JIRA ticket?
>
> -Evan
>
>
> On Tue, Mar 18, 2014 at 4:04 PM, Matei Zaharia  
> wrote:
>> Cool, glad to see this posted! I've added a link to it at 
>> https://cwiki.apache.org/confluence/display/SPARK/Powered+By+Spark.
>>
>> Matei
>>
>> On Mar 18, 2014, at 1:51 PM, Evan Chan  wrote:
>>
>>> Dear Spark developers,
>>>
>>> Ooyala is happy to announce that we have pushed our official, Spark
>>> 0.9.0 / Scala 2.10-compatible, job server as a github repo:
>>>
>>> https://github.com/ooyala/spark-jobserver
>>>
>>> Complete with unit tests, deploy scripts, and examples.
>>>
>>> The original PR (#222) on incubator-spark is now closed.
>>>
>>> Please have a look; pull requests are very welcome.
>>> --
>>> --
>>> Evan Chan
>>> Staff Engineer
>>> e...@ooyala.com  |
>>
>
>
>
> --
> --
> Evan Chan
> Staff Engineer
> e...@ooyala.com  |

Re: Announcing the official Spark Job Server repo

[Evan Chan]

Matei,

Maybe it's time to explore the spark-contrib idea again?   Should I
start a JIRA ticket?

-Evan


On Tue, Mar 18, 2014 at 4:04 PM, Matei Zaharia  wrote:
> Cool, glad to see this posted! I've added a link to it at 
> https://cwiki.apache.org/confluence/display/SPARK/Powered+By+Spark.
>
> Matei
>
> On Mar 18, 2014, at 1:51 PM, Evan Chan  wrote:
>
>> Dear Spark developers,
>>
>> Ooyala is happy to announce that we have pushed our official, Spark
>> 0.9.0 / Scala 2.10-compatible, job server as a github repo:
>>
>> https://github.com/ooyala/spark-jobserver
>>
>> Complete with unit tests, deploy scripts, and examples.
>>
>> The original PR (#222) on incubator-spark is now closed.
>>
>> Please have a look; pull requests are very welcome.
>> --
>> --
>> Evan Chan
>> Staff Engineer
>> e...@ooyala.com  |
>



-- 
--
Evan Chan
Staff Engineer
e...@ooyala.com  |

Re: Announcing the official Spark Job Server repo

[Evan Chan]

Andy,

Yeah, we've thought of deploying this on Marathon ourselves, but we're
not sure how much Mesos we're going to use yet.   (Indeed if you look
at bin/server_start.sh, I think I set up the PORT environment var
specifically for Marathon.)This is also why we have deploy scripts
which package into .tar.gz, again for Mesos deployment.

If you do try this, please let us know.  :)

-Evan


On Tue, Mar 18, 2014 at 3:57 PM, andy petrella  wrote:
> tad! That's awesome.
>
> A quick question, does someone has insights regarding having such
> JobServers deployed using Marathon on Mesos?
>
> I'm thinking about an arch where Marathon would deploy and keep the Job
> Servers running along with part of the whole set of apps deployed on it
> regarding the resources needed (à la Jenkins).
>
> Any idea is welcome.
>
> Back to the news, Evan + Ooyala team: Great Job again.
>
> andy
>
> On Tue, Mar 18, 2014 at 11:39 PM, Henry Saputra 
> wrote:
>
>> W00t!
>>
>> Thanks for releasing this, Evan.
>>
>> - Henry
>>
>> On Tue, Mar 18, 2014 at 1:51 PM, Evan Chan  wrote:
>> > Dear Spark developers,
>> >
>> > Ooyala is happy to announce that we have pushed our official, Spark
>> > 0.9.0 / Scala 2.10-compatible, job server as a github repo:
>> >
>> > https://github.com/ooyala/spark-jobserver
>> >
>> > Complete with unit tests, deploy scripts, and examples.
>> >
>> > The original PR (#222) on incubator-spark is now closed.
>> >
>> > Please have a look; pull requests are very welcome.
>> > --
>> > --
>> > Evan Chan
>> > Staff Engineer
>> > e...@ooyala.com  |
>>



-- 
--
Evan Chan
Staff Engineer
e...@ooyala.com  |

Re: Announcing the official Spark Job Server repo

[Matei Zaharia]

Cool, glad to see this posted! I’ve added a link to it at 
https://cwiki.apache.org/confluence/display/SPARK/Powered+By+Spark.

Matei

On Mar 18, 2014, at 1:51 PM, Evan Chan  wrote:

> Dear Spark developers,
> 
> Ooyala is happy to announce that we have pushed our official, Spark
> 0.9.0 / Scala 2.10-compatible, job server as a github repo:
> 
> https://github.com/ooyala/spark-jobserver
> 
> Complete with unit tests, deploy scripts, and examples.
> 
> The original PR (#222) on incubator-spark is now closed.
> 
> Please have a look; pull requests are very welcome.
> -- 
> --
> Evan Chan
> Staff Engineer
> e...@ooyala.com  |

Re: Announcing the official Spark Job Server repo

[andy petrella]

tad! That's awesome.

A quick question, does someone has insights regarding having such
JobServers deployed using Marathon on Mesos?

I'm thinking about an arch where Marathon would deploy and keep the Job
Servers running along with part of the whole set of apps deployed on it
regarding the resources needed (à la Jenkins).

Any idea is welcome.

Back to the news, Evan + Ooyala team: Great Job again.

andy

On Tue, Mar 18, 2014 at 11:39 PM, Henry Saputra wrote:

> W00t!
>
> Thanks for releasing this, Evan.
>
> - Henry
>
> On Tue, Mar 18, 2014 at 1:51 PM, Evan Chan  wrote:
> > Dear Spark developers,
> >
> > Ooyala is happy to announce that we have pushed our official, Spark
> > 0.9.0 / Scala 2.10-compatible, job server as a github repo:
> >
> > https://github.com/ooyala/spark-jobserver
> >
> > Complete with unit tests, deploy scripts, and examples.
> >
> > The original PR (#222) on incubator-spark is now closed.
> >
> > Please have a look; pull requests are very welcome.
> > --
> > --
> > Evan Chan
> > Staff Engineer
> > e...@ooyala.com  |
>

Re: Announcing the official Spark Job Server repo

[yao]

+1


On Tue, Mar 18, 2014 at 3:39 PM, Henry Saputra wrote:

> W00t!
>
> Thanks for releasing this, Evan.
>
> - Henry
>
> On Tue, Mar 18, 2014 at 1:51 PM, Evan Chan  wrote:
> > Dear Spark developers,
> >
> > Ooyala is happy to announce that we have pushed our official, Spark
> > 0.9.0 / Scala 2.10-compatible, job server as a github repo:
> >
> > https://github.com/ooyala/spark-jobserver
> >
> > Complete with unit tests, deploy scripts, and examples.
> >
> > The original PR (#222) on incubator-spark is now closed.
> >
> > Please have a look; pull requests are very welcome.
> > --
> > --
> > Evan Chan
> > Staff Engineer
> > e...@ooyala.com  |
>

Re: Announcing the official Spark Job Server repo

[Henry Saputra]

W00t!

Thanks for releasing this, Evan.

- Henry

On Tue, Mar 18, 2014 at 1:51 PM, Evan Chan  wrote:
> Dear Spark developers,
>
> Ooyala is happy to announce that we have pushed our official, Spark
> 0.9.0 / Scala 2.10-compatible, job server as a github repo:
>
> https://github.com/ooyala/spark-jobserver
>
> Complete with unit tests, deploy scripts, and examples.
>
> The original PR (#222) on incubator-spark is now closed.
>
> Please have a look; pull requests are very welcome.
> --
> --
> Evan Chan
> Staff Engineer
> e...@ooyala.com  |