Re: Mesos scheduler obeying limit of tasks / executor

Hi Charles, I am not sure I totally understand your issues, but the spark.task.cpus limit is imposed at a higher level, for all cluster managers. The code is in TaskSchedulerImpl

Re: Mesos scheduler obeying limit of tasks / executor

That answers it thanks! On Fri, Dec 11, 2015 at 6:37 AM Iulian Dragoș wrote: > Hi Charles, > > I am not sure I totally understand your issues, but the spark.task.cpus > limit is imposed at a higher level, for all cluster managers. The code is > in TaskSchedulerImpl >

Mesos scheduler obeying limit of tasks / executor

I have a spark app in development which has relatively strict cpu/mem ratios that are required. As such, I cannot arbitrarily add CPUs to a limited memory size. The general spark cluster behaves as expected, where tasks are launched with a specified memory/cpu ratio, but the mesos scheduler seems