Hi Susan,
This is exactly what we have used. Thank you for your interest!
- Thodoris
> On 23 Jul 2018, at 20:55, Susan X. Huynh wrote:
>
> Hi Thodoris,
>
> Maybe setting "spark.scheduler.minRegisteredResourcesRatio" to > 0 would
> help? Default value is 0 with Mesos.
>
> "The minimum
Hi Thodoris,
Maybe setting "spark.scheduler.minRegisteredResourcesRatio" to > 0 would
help? Default value is 0 with Mesos.
"The minimum ratio of registered resources (registered resources / total
expected resources) (resources are executors in yarn mode and Kubernetes
mode, CPU cores in
Oh, sorry, i missed that you use spark without dynamic allocation. Anyway,
i don't know does this parameters works without dynamic allocation.
On Wed, Jul 11, 2018 at 5:11 PM Thodoris Zois wrote:
> Hello,
>
> Yeah you are right, but I think that works only if you use Spark dynamic
> allocation.
Hello,
Yeah you are right, but I think that works only if you use Spark dynamic
allocation. Am I wrong?
-Thodoris
> On 11 Jul 2018, at 17:09, Pavel Plotnikov
> wrote:
>
> Hi, Thodoris
> You can configure resources per executor and manipulate with number of
> executers instead using
Hi, Thodoris
You can configure resources per executor and manipulate with number of
executers instead using spark.max.cores. I think
spark.dynamicAllocation.minExecutors
and spark.dynamicAllocation.maxExecutors configuration values can help you.
On Tue, Jul 10, 2018 at 5:07 PM Thodoris Zois
Actually after some experiments we figured out that spark.max.cores /
spark.executor.cores is the upper bound for the executors. Spark apps will run
even only if one executor can be launched.
Is there any way to specify also the lower bound? It is a bit annoying that
seems that we can’t
Hello Thodoris!
Have you checked this:
- does mesos cluster have available resources?
- if spark have waiting tasks in queue more than
spark.dynamicAllocation.schedulerBacklogTimeout configuration value?
- And then, have you checked that mesos send offers to spark app mesos
framework at least
Hello list,
We are running Apache Spark on a Mesos cluster and we face a weird behavior of
executors. When we submit an app with e.g 10 cores and 2GB of memory and max
cores 30, we expect to see 3 executors running on the cluster. However,
sometimes there are only 2... Spark applications are
