My understanding is that dynamic allocation is only enabled for Spark-on-Yarn. Those settings likely have no impact in standalone mode.
Nick On Mon, Jan 11, 2016, 5:10 AM Yiannis Gkoufas <johngou...@gmail.com> wrote: > Hi, > > I am exploring a bit the dynamic resource allocation provided by the > Standalone Cluster Mode and I was wondering whether this behavior I am > experiencing is expected. > In my configuration I have 3 slaves with 24 cores each. > I have in my spark-defaults.conf: > > spark.shuffle.service.enabled true > spark.dynamicAllocation.enabled true > spark.dynamicAllocation.minExecutors 1 > spark.dynamicAllocation.maxExecutors 6 > spark.executor.cores 4 > > When I submit a first Job it takes up all of the 72 cores of the cluster. > When I submit the second Job while the first one is running I get the > error: > > Initial job has not accepted any resources; check your cluster UI to > ensure that workers are registered and have sufficient resources > > Is this the expected behavior? > > Thanks a lot > > >
