Hi Sharma, This sounds eerily familiar! Was this an in-house system you were working on, or a commercial product?
Thanks, Aaron ________________________________ From: Sharma Podila [spod...@netflix.com] Sent: 13 May 2015 23:49 To: user@mesos.apache.org Cc: Douglas Thain; Brian Bockelman Subject: [Junk released by User action] Re: Batch Scheduler with dependency support I keep longing for folks with decades of experience in HTC&HPC to chime in "on-list". FWIW, I come from that background, but, am not in that space at this time. My prior life was in developing a (not open source) distributed job scheduler and management system for batch and interactive jobs that handled dependencies, deadlines, preemptions, advance reservation of resources, etc. with multi-level priority and share tree hierarchy based allocation. Typically, dependencies and deadlines are handled outside of schedulers and fed into schedulers as task submission after dependencies have been met. We found it more optimal to have the scheduler resolve dependencies and deadlines inherently. This way, a high priority job dependent on another low priority job can induce higher priority on that dependent job. Similarly, a job with a deadline depending on another job's completion can induce an earlier launch of the latter job in order to meet it's deadline. Also, a dependent job can reserve its resources in advance, knowing the expected completion time of its dependent jobs. This was important because in that environment we always had more jobs to run than can run on available resources. It wasn't unusual to have 10s of 1000s of jobs waiting in queue to run during the day. Not sure if this helps the original question in this thread in any way. But, I am glad to share my learning, if that helps. Sharma On Wed, May 13, 2015 at 1:12 PM, Tim St Clair <tstcl...@redhat.com<mailto:tstcl...@redhat.com>> wrote: Hi Alex, Have you by chance integrated with any of the tradition batch DAG systems? http://pegasus.isi.edu/ , http://ccl.cse.nd.edu/software/makeflow/ I keep longing for folks with decades of experience in HTC&HPC to chime in "on-list". Subtle nudge ;-) Tim ________________________________ From: "Alex Gaudio" <adgau...@gmail.com<mailto:adgau...@gmail.com>> To: user@mesos.apache.org<mailto:user@mesos.apache.org> Sent: Wednesday, May 13, 2015 3:04:20 PM Subject: Re: Batch Scheduler with dependency support Hi Tim (and everyone else!), I am the primary author of Stolos. We use Stolos to run all of our batch jobs on Mesos. The batch jobs are scripts we can run from the command-line. Scripts range from bash scripts, Spark jobs and R scripts. It's a great tool for us because, unlike Chronos, it lets us define a script as stage in a dependency chain, where the script can run with different parameters for different dependency contexts. (The closest usage of this would be to have many Chronos servers, though this does not work in all cases). The tool is a critical component of Sailthru's data science infrastructure, but I believe we are the only people who use the tool right now. If you are interested in learning more, I'm happy to invest time to talk more about Stolos, what it does and how we use it! Alex On Wed, May 13, 2015 at 2:02 PM Tim Chen <t...@mesosphere.io<mailto:t...@mesosphere.io>> wrote: How are you running your batch jobs? Is the batch job script/executable an in-house app? Tim On Wed, May 13, 2015 at 9:46 AM, Andras Kerekes <andras.kere...@ishisystems.com<mailto:andras.kere...@ishisystems.com>> wrote: You might want to have a look at stolos too: https://github.com/sailthru/stolos Andras From: Aaron Carey [mailto:aca...@ilm.com<mailto:aca...@ilm.com>] Sent: Wednesday, May 13, 2015 11:54 AM To: user@mesos.apache.org<mailto:user@mesos.apache.org> Subject: RE: Batch Scheduler with dependency support Thanks! I hadn't come across that one before :) ________________________________ From: jeffschr...@gmail.com<mailto:jeffschr...@gmail.com> [jeffschr...@gmail.com<mailto:jeffschr...@gmail.com>] on behalf of Jeff Schroeder [jeffschroe...@computer.org<mailto:jeffschroe...@computer.org>] Sent: 13 May 2015 16:39 To: user@mesos.apache.org<mailto:user@mesos.apache.org> Subject: Re: Batch Scheduler with dependency support Lookup Hubspot's Singularity On Wednesday, May 13, 2015, Aaron Carey <aca...@ilm.com<mailto:aca...@ilm.com>> wrote: Thanks Jeff, Any other options around as well? ________________________________ From: jeffschr...@gmail.com<http://UrlBlockedError.aspx> [jeffschr...@gmail.com<http://UrlBlockedError.aspx>] on behalf of Jeff Schroeder [jeffschroe...@computer.org<http://UrlBlockedError.aspx>] Sent: 13 May 2015 14:12 To: user@mesos.apache.org<http://UrlBlockedError.aspx> Subject: Batch Scheduler with dependency support It does both just as well, along with cron-like functionality. It is harder to install and takes a bit more understanding however. The official tutorial is a process that loops 100 times and then exits. http://aurora.apache.org/documentation/latest/tutorial/#the-script Aurora is pretty much a superset of most other generic frameworks sans maybe hubspot's singularity. On Wednesday, May 13, 2015, Aaron Carey <aca...@ilm.com<http://UrlBlockedError.aspx>> wrote: I was under the impression Aurora was for long running services? Is it suitable for scheduling one of batch processes too? thanks, Aaron ________________________________ From: jeffschr...@gmail.com<mailto:jeffschr...@gmail.com> [jeffschr...@gmail.com<mailto:jeffschr...@gmail.com>] on behalf of Jeff Schroeder [jeffschroe...@computer.org<mailto:jeffschroe...@computer.org>] Sent: 13 May 2015 13:12 To: user@mesos.apache.org<mailto:user@mesos.apache.org> Subject: Re: Batch Scheduler with dependency support Apache Aurora does this and you can be explicit about the ordering On Wednesday, May 13, 2015, Aaron Carey <aca...@ilm.com<mailto:aca...@ilm.com>> wrote: Hi All, I was just wondering if anyone out there knew of a good mesos batch scheduler which supports dependencies between tasks? (ie Task B cannot run until Task A is complete) Thanks, Aaron -- Text by Jeff, typos by iPhone -- Text by Jeff, typos by iPhone -- Text by Jeff, typos by iPhone -- Cheers, Timothy St. Clair Red Hat Inc.
