@Tim Chen
Thank your for your comment! We use Stolos run about 200k jobs a day,
where some jobs are themselves Mesos frameworks (ie Spark jobs). The tool
seems so far to be scalable because it offloads all scaling problems to
established tools. Queue/job state is stored to a user-chosen queue
b
>
> I keep longing for folks with decades of experience in HTC&HPC to chime
> in "on-list".
FWIW, I come from that background, but, am not in that space at this time.
My prior life was in developing a (not open source) distributed job
scheduler and management system for batch and interactive job
@Tim We're running a combination of an in-house DAG scheduler we wrote a few
years back (kind of like https://github.com/spotify/luigi) in combination with
Jenkins. I'm not aware of a fully blown DAG scheduler that exists as a Mesos
framework but Jenkins might be a way to go, it works well for u
Hi Alex,
Have you by chance integrated with any of the tradition batch DAG systems?
http://pegasus.isi.edu/ , http://ccl.cse.nd.edu/software/makeflow/
I keep longing for folks with decades of experience in HTC&HPC to chime in
"on-list".
Subtle nudge ;-)
Tim
- Original Message -
Hi Alex,
Thanks for replying and looks like a really interesting framework! I was
orginally aiming the question back to Aaron as he stated he's looking for a
batch job scheduler.
Do you have some rough stats how many jobs or data you guys are computing
with Stolos?
Tim
On Wed, May 13, 2015 at
I should also mention that I run Stolos as a Mesos executor and another
tool I wrote, Relay.Mesos, as the Mesos Scheduler (that spins up Stolos
Executors).
The github links for these tools are:
https://github.com/sailthru/relay.mesos
https://github.com/sailthru/stolos
On Wed, May 13, 2015 at 4:0
Hi Tim (and everyone else!),
I am the primary author of Stolos. We use Stolos to run all of our batch
jobs on Mesos. The batch jobs are scripts we can run from the
command-line. Scripts range from bash scripts, Spark jobs and R scripts.
It's a great tool for us because, unlike Chronos, it lets
Okay my apologies perhaps I’m just not explaining this well.
So we have an application, that creates a spark context and adds all the
necessary jars, the job is submitted to the cluster
and as it runs the code it attempts to pull the data necessary for the job from
s3. We use our own version of
Hi Stephen,
I'm not quite sure what you mean by bootstrapping classes, do you have some
particular examples?
Usually to run any user jar you just need it to be reachable by your slaves
so it can be either S3 or any accessible place, then you just provide your
jar url when you run spark-submit.
T
How are you running your batch jobs? Is the batch job script/executable an
in-house app?
Tim
On Wed, May 13, 2015 at 9:46 AM, Andras Kerekes <
andras.kere...@ishisystems.com> wrote:
> You might want to have a look at stolos too:
>
>
>
> https://github.com/sailthru/stolos
>
>
>
> Andras
>
>
>
>
>
You might want to have a look at stolos too:
https://github.com/sailthru/stolos
Andras
From: Aaron Carey [mailto:aca...@ilm.com]
Sent: Wednesday, May 13, 2015 11:54 AM
To: user@mesos.apache.org
Subject: RE: Batch Scheduler with dependency support
Thanks! I hadn't come across tha
Thanks! I hadn't come across that one before :)
From: jeffschr...@gmail.com [jeffschr...@gmail.com] on behalf of Jeff Schroeder
[jeffschroe...@computer.org]
Sent: 13 May 2015 16:39
To: user@mesos.apache.org
Subject: Re: Batch Scheduler with dependency support
Loo
Lookup Hubspot's Singularity
On Wednesday, May 13, 2015, Aaron Carey wrote:
> Thanks Jeff,
>
> Any other options around as well?
>
> --
> *From:* jeffschr...@gmail.com
> [
> jeffschr...@gmail.com
> ] on behalf of
> Jeff Schroeder [jeffschroe...@computer.org
> ]
> *
Thanks Jeff,
Any other options around as well?
From: jeffschr...@gmail.com [jeffschr...@gmail.com] on behalf of Jeff Schroeder
[jeffschroe...@computer.org]
Sent: 13 May 2015 14:12
To: user@mesos.apache.org
Subject: Batch Scheduler with dependency support
It does
Hi,
We have a small mesos cluster and we’d like to be able to initialize some of
our classes, mostly we have a vfs we setup to be allow our code
to access S3, but there doesn’t seem to be any readily obvious way to bootstrap
these kind of classes so that they have the properly initialized config
It does both just as well, along with cron-like functionality. It is harder
to install and takes a bit more understanding however. The official
tutorial is a process that loops 100 times and then exits.
http://aurora.apache.org/documentation/latest/tutorial/#the-script
Aurora is pretty much a sup
I was under the impression Aurora was for long running services? Is it suitable
for scheduling one of batch processes too?
thanks,
Aaron
From: jeffschr...@gmail.com [jeffschr...@gmail.com] on behalf of Jeff Schroeder
[jeffschroe...@computer.org]
Sent: 13 May 201
Apache Aurora does this and you can be explicit about the ordering
On Wednesday, May 13, 2015, Aaron Carey wrote:
> Hi All,
>
> I was just wondering if anyone out there knew of a good mesos batch
> scheduler which supports dependencies between tasks? (ie Task B cannot run
> until Task A is comp
ah, sorry, no : i'm conflating the two ... we run all our batch jobs on
a schedule
:c
On 13 May 2015, at 12:28, Aaron Carey wrote:
Have you been using it for batch tasks? It seems great for time
dependent tasks but I wasn't aware it could do batch as well?
Thanks!
Aaron
Have you been using it for batch tasks? It seems great for time dependent tasks
but I wasn't aware it could do batch as well?
Thanks!
Aaron
From: craig mcmillan [mccraigmccr...@gmail.com]
Sent: 13 May 2015 12:09
To: user@mesos.apache.org
Subject: Re: Batch
i've been using https://github.com/mesos/chronos
:c
On 13 May 2015, at 12:06, Aaron Carey wrote:
Hi All,
I was just wondering if anyone out there knew of a good mesos batch
scheduler which supports dependencies between tasks? (ie Task B cannot
run until Task A is complete)
Thanks,
Aaron
Hi All,
I was just wondering if anyone out there knew of a good mesos batch scheduler
which supports dependencies between tasks? (ie Task B cannot run until Task A
is complete)
Thanks,
Aaron
22 matches
Mail list logo
