Hi Ian, Don't use SPARK_MEM in spark-env.sh. It will get it set for all of your jobs. The better way is to use only the second option sconf.setExecutorEnv("spark.executor.memory", "4g”) i.e. set it in the driver program. In this way every job will have memory according to requirment.
For example you have 4gb memory on each worker node, you won't be able to run more than one job with 4gb given to each. Two jobs with 2GB(or slightly less) each will work. Laeeq On Wednesday, May 7, 2014 2:29 AM, Ian Ferreira <ianferre...@hotmail.com> wrote: Hi there, Why can’t I seem to kick the executor memory higher? See below from EC2 deployment using m1.large And in the spark-env.sh export SPARK_MEM=6154m And in the spark context sconf.setExecutorEnv("spark.executor.memory", "4g”) Cheers - Ian