If you're using standalone mode, you need to make sure the Spark Workers know about the extra memory. This can be configured in spark-env.sh on the workers as
export SPARK_WORKER_MEMORY=4g On Tue, May 6, 2014 at 5:29 PM, Ian Ferreira <[email protected]>wrote: > Hi there, > > Why can’t I seem to kick the executor memory higher? See below from EC2 > deployment using m1.large > > > And in the spark-env.sh > export SPARK_MEM=6154m > > > And in the spark context > sconf.setExecutorEnv("spark.executor.memory", "4g”) > > Cheers > - Ian > >
