Hi Susan,
This is exactly what we have used. Thank you for your interest!
- Thodoris
> On 23 Jul 2018, at 20:55, Susan X. Huynh wrote:
>
> Hi Thodoris,
>
> Maybe setting "spark.scheduler.minRegisteredResourcesRatio" to > 0 would
> help? Default value is 0 with Mesos.
>
> "The minimum ratio
Hi Thodoris,
Maybe setting "spark.scheduler.minRegisteredResourcesRatio" to > 0 would
help? Default value is 0 with Mesos.
"The minimum ratio of registered resources (registered resources / total
expected resources) (resources are executors in yarn mode and Kubernetes
mode, CPU cores in standalon
Oh, sorry, i missed that you use spark without dynamic allocation. Anyway,
i don't know does this parameters works without dynamic allocation.
On Wed, Jul 11, 2018 at 5:11 PM Thodoris Zois wrote:
> Hello,
>
> Yeah you are right, but I think that works only if you use Spark dynamic
> allocation.
Hello,
Yeah you are right, but I think that works only if you use Spark dynamic
allocation. Am I wrong?
-Thodoris
> On 11 Jul 2018, at 17:09, Pavel Plotnikov
> wrote:
>
> Hi, Thodoris
> You can configure resources per executor and manipulate with number of
> executers instead using spark.ma
Hi, Thodoris
You can configure resources per executor and manipulate with number of
executers instead using spark.max.cores. I think
spark.dynamicAllocation.minExecutors
and spark.dynamicAllocation.maxExecutors configuration values can help you.
On Tue, Jul 10, 2018 at 5:07 PM Thodoris Zois wrote
Actually after some experiments we figured out that spark.max.cores /
spark.executor.cores is the upper bound for the executors. Spark apps will run
even only if one executor can be launched.
Is there any way to specify also the lower bound? It is a bit annoying that
seems that we can’t contro
Hello Thodoris!
Have you checked this:
- does mesos cluster have available resources?
- if spark have waiting tasks in queue more than
spark.dynamicAllocation.schedulerBacklogTimeout configuration value?
- And then, have you checked that mesos send offers to spark app mesos
framework at least w
