Hi guys, Currently I am running Spark program on Amazon EC2. Each worker has around (less than but near to )2 gb memory.
By default, I can see each worker is allocated 976 mb memory as the table shows below on Spark WEB UI. I know this value is from (Total memory minus 1 GB). But I want more than 1 gb in each of my worker. AddressStateCoresMemory ALIVE1 (0 Used)976.0 MB (0.0 B Used)Based on the instruction on Spark website, I made "export SPARK_WORKER_MEMORY=1g" in spark-env.sh. But it doesn't work. BTW, I can set "SPARK_EXECUTOR_MEMORY=1g" and it works. Can anyone help me? Is there a requirement that one worker must maintain 1 gb memory for itself aside from the memory for Spark? Thanks, Jia
