Thanks! From: Aaron Davidson <[email protected]> Reply-To: <[email protected]> Date: Tuesday, May 6, 2014 at 5:32 PM To: <[email protected]> Subject: Re: Easy one
If you're using standalone mode, you need to make sure the Spark Workers know about the extra memory. This can be configured in spark-env.sh on the workers as export SPARK_WORKER_MEMORY=4g On Tue, May 6, 2014 at 5:29 PM, Ian Ferreira <[email protected]> wrote: > Hi there, > > Why can¹t I seem to kick the executor memory higher? See below from EC2 > deployment using m1.large > > > And in the spark-env.sh > export SPARK_MEM=6154m > > > And in the spark context > sconf.setExecutorEnv("spark.executor.memory", "4g²) > > Cheers > - Ian >
