Hi all, I have a doubt regarding the options in spark-env.sh. I set the following values in the file in master and 2 workers
SPARK_WORKER_MEMORY=7g SPARK_EXECUTOR_MEMORY=6g SPARK_DAEMON_JAVA_OPTS+="- Dspark.akka.timeout=300000 -Dspark.akka.frameSize=10000 -Dspark.blockManagerHeartBeatMs=800000 -Dspark.shuffle.spill=false But SPARK_EXECUTOR_MEMORY is showing 4g in web UI.Do I need to change it anywhere else to make it 4g and to reflect it in web UI. A warning is coming that blockManagerHeartBeatMs is exceeding 450000 while executing a process even though I set it to 800000. So I doubt whether it should be set as SPARK_MASTER_OPTS or SPARK_WORKER_OPTS.. Thanks & Regards, Meethu M
