Hi, Thodoris You can configure resources per executor and manipulate with number of executers instead using spark.max.cores. I think spark.dynamicAllocation.minExecutors and spark.dynamicAllocation.maxExecutors configuration values can help you.
On Tue, Jul 10, 2018 at 5:07 PM Thodoris Zois <z...@ics.forth.gr> wrote: > Actually after some experiments we figured out that spark.max.cores / > spark.executor.cores is the upper bound for the executors. Spark apps will > run even only if one executor can be launched. > > Is there any way to specify also the lower bound? It is a bit annoying > that seems that we can’t control the resource usage of an application. By > the way, we are not using dynamic allocation. > > - Thodoris > > > On 10 Jul 2018, at 14:35, Pavel Plotnikov <pavel.plotni...@team.wrike.com> > wrote: > > Hello Thodoris! > Have you checked this: > - does mesos cluster have available resources? > - if spark have waiting tasks in queue more than > spark.dynamicAllocation.schedulerBacklogTimeout configuration value? > - And then, have you checked that mesos send offers to spark app mesos > framework at least with 10 cores and 2GB RAM? > > If mesos have not available offers with 10 cores, for example, but have > with 8 or 9, so you can use smaller executers for better fit for available > resources on nodes for example with 4 cores and 1 GB RAM, for example > > Cheers, > Pavel > > On Mon, Jul 9, 2018 at 9:05 PM Thodoris Zois <z...@ics.forth.gr> wrote: > >> Hello list, >> >> We are running Apache Spark on a Mesos cluster and we face a weird >> behavior of executors. When we submit an app with e.g 10 cores and 2GB of >> memory and max cores 30, we expect to see 3 executors running on the >> cluster. However, sometimes there are only 2... Spark applications are not >> the only one that run on the cluster. I guess that Spark starts executors >> on the available offers even if it does not satisfy our needs. Is there any >> configuration that we can use in order to prevent Spark from starting when >> there are no resource offers for the total number of executors? >> >> Thank you >> - Thodoris >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >> >>
