Yes, that does appear to be the case. The documentation is very clear about the heap settings and that they can not be used with spark.executor.extraJavaOptions
spark.executor.extraJavaOptions(none)A string of extra JVM options to pass to executors. For instance, GC settings or other logging. *Note that it is illegal to set Spark properties or heap size settings with this option.* Spark properties should be set using a SparkConf object or the spark-defaults.conf file used with the spark-submit script. *Heap size settings can be set with spark.executor.memory*. So it appears to be a limitation at this time. -Todd On Thu, Jul 2, 2015 at 4:13 PM, Mulugeta Mammo <mulugeta.abe...@gmail.com> wrote: > thanks but my use case requires I specify different start and max heap > sizes. Looks like spark sets start and max sizes same value. > > On Thu, Jul 2, 2015 at 1:08 PM, Todd Nist <tsind...@gmail.com> wrote: > >> You should use: >> >> spark.executor.memory >> >> from the docs <https://spark.apache.org/docs/latest/configuration.html>: >> spark.executor.memory512mAmount of memory to use per executor process, >> in the same format as JVM memory strings (e.g.512m, 2g). >> >> -Todd >> >> >> >> On Thu, Jul 2, 2015 at 3:36 PM, Mulugeta Mammo <mulugeta.abe...@gmail.com >> > wrote: >> >>> tried that one and it throws error - extraJavaOptions is not allowed to >>> alter memory settings, use spakr.executor.memory instead. >>> >>> On Thu, Jul 2, 2015 at 12:21 PM, Benjamin Fradet < >>> benjamin.fra...@gmail.com> wrote: >>> >>>> Hi, >>>> >>>> You can set those parameters through the >>>> >>>> spark.executor.extraJavaOptions >>>> >>>> Which is documented in the configuration guide: >>>> spark.apache.org/docs/latest/configuration.htnl >>>> On 2 Jul 2015 9:06 pm, "Mulugeta Mammo" <mulugeta.abe...@gmail.com> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> I'm running Spark 1.4.0, I want to specify the start and max size >>>>> (-Xms and Xmx) of the jvm heap size for my executors, I tried: >>>>> >>>>> executor.cores.memory="-Xms1g -Xms8g" >>>>> >>>>> but doesn't work. How do I specify? >>>>> >>>>> Appreciate your help. >>>>> >>>>> Thanks, >>>>> >>>>> >>> >> >
