Hi Grzegorz,
For local mode you only have one executor, and this executor is your
driver, so you need to set the driver's memory instead. *That said, in
local mode, by the time you run spark-submit, a JVM has already been
launched with the default memory settings, so setting "spark.driver.memory"
in your conf won't actually do anything for you. Instead, you need to run
spark-submit as follows
bin/spark-submit --driver-memory 2g --class your.class.here app.jar
This will start the JVM with 2G instead of the default 512M.
-Andrew
2014-08-05 6:43 GMT-07:00 Grzegorz Białek <grzegorz.bia...@codilime.com>:
> Hi,
>
> I wanted to make simple Spark app running in local mode with 2g
> spark.executor.memory and 1g for caching. But following code:
>
> val conf = new SparkConf()
> .setMaster("local")
> .setAppName("app")
> .set("spark.executor.memory", "2g")
> .set("spark.storage.memoryFraction", "0.5")
> val sc = new SparkContext(conf)
>
> doesn't work. In spark UI this variables are set properly but memory store
> is around 0.5 * 512MB (default spark.executor.memory) not 0.5 * 2GB:
>
> 14/08/05 15:34:00 INFO MemoryStore: MemoryStore started with capacity
> 245.8 MB.
>
> I have neither spark-defaults.conf nor spark-env.sh in my $SPARK_HOME/conf
> directory. I use Spark 1.0.0
> How can I set this values properly?
>
> Thanks,
> Grzegorz
>
>
