Hi, Couple of questions here:
0. I modified SparkLR.scala to change the N(# of data points) and D (# of dimensions) , and ran it with: # bin/run-example -Dspark.executor.memory=40g org.apache.spark.examples.SparkLR local[23] 500 And here’s the process table: /net/home/ltsai/jdk1.7.0_51/bin/java -cp /net/home/ltsai/spark-0.9.0-incubating/examples/target/scala-2.10/spark-examples-assembly-0.9.0-incubating.jar::/net/home/ltsai/spark-0.9.0-incubating/conf:/net/home/ltsai/spark-0.9.0-incubating/assembly/target/scala-2.10/spark-assembly-0.9.0-incubating-hadoop2.2.0.jar -Djava.net.preferIPv4Stack=true -XX:+UseConcMarkSweepGC -XX:+UseNUMA -XX:ConcGCThreads=8 -Djava.library.path= -Dspark.executor.memory=40g org.apache.spark.examples.SparkLR local[23] 500 1. When using local threading local[N], does executor memory (spark.executor.memory) applies? Based on top, the process is consuming at least 15G RES memory. Shouldn’t the JVM run out of memory because of the default heap size of 512M? 2. What is this block Manager memory? Is this related to RDD? Under the Spark Web UI, it is listed under executors? How did it determine 9.3GB? 14/03/13 13:31:06 INFO BlockManagerMasterActor$BlockManagerInfo: Registering block manager xxx:57084 with 9.3 GB RAM Further down, it printed these warnings: 14/03/13 13:40:15 WARN BlockManagerMasterActor: Removing BlockManager BlockManagerId(<driver>, xxx, 57084, 0) with no recent heart beats: 52528ms exceeds 45000ms 14/03/13 13:40:15 INFO BlockManager: BlockManager reregistering with master 14/03/13 13:40:15 INFO BlockManagerMaster: Trying to register BlockManager 14/03/13 13:40:15 INFO BlockManagerMasterActor$BlockManagerInfo: Registering block manager xxx:57084 with 9.3 GB RAM 14/03/13 13:40:15 INFO BlockManagerMaster: Registered BlockManager 14/03/13 13:40:15 INFO BlockManager: Reporting 0 blocks to the master. After which, I can see the system mostly idling. The process takes only 100% CPU in top. The above messages will repeat itself again. What happened? Thanks!
