Hi, I'm trying to run the SimpleApp example ( http://spark.apache.org/docs/latest/quick-start.html#a-standalone-app-in-scala) on a larger dataset.
The input file is about 1GB, but when I run the Spark program, it says:"java.lang.OutOfMemoryError: GC overhead limit exceeded", the full error output is attached at the end of the E-mail. Then I tried multiple ways of setting the memory limit. In SimpleApp.scala file, I set the following configurations: val conf = new SparkConf() .setAppName("Simple Application") .set("spark.executor.memory", "10g") And I have also tried appending the following configuration to conf/spark-defaults.conf file: spark.executor.memory 10g But neither of them works. In the error message, it claims "(estimated size 103.8 MB, free 191.1 MB)", so the total available memory is still 300MB. Why? Thanks, Cui $ ~/spark-1.0.0-bin-hadoop1/bin/spark-submit --class "SimpleApp" --master local[4] target/scala-2.10/simple-project_2.10-1.0.jar /tmp/mdata0-10.tsd Spark assembly has been built with Hive, including Datanucleus jars on classpath 14/06/09 15:06:29 INFO SecurityManager: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 14/06/09 15:06:29 INFO SecurityManager: Changing view acls to: cuihe 14/06/09 15:06:29 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(cuihe) 14/06/09 15:06:29 INFO Slf4jLogger: Slf4jLogger started 14/06/09 15:06:29 INFO Remoting: Starting remoting 14/06/09 15:06:30 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sp...@131-1.bfc.hpl.hp.com:40779] 14/06/09 15:06:30 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sp...@131-1.bfc.hpl.hp.com:40779] 14/06/09 15:06:30 INFO SparkEnv: Registering MapOutputTracker 14/06/09 15:06:30 INFO SparkEnv: Registering BlockManagerMaster 14/06/09 15:06:30 INFO DiskBlockManager: Created local directory at /tmp/spark-local-20140609150630-eaa9 14/06/09 15:06:30 INFO MemoryStore: MemoryStore started with capacity 294.9 MB. 14/06/09 15:06:30 INFO ConnectionManager: Bound socket to port 47164 with id = ConnectionManagerId(131-1.bfc.hpl.hp.com,47164) 14/06/09 15:06:30 INFO BlockManagerMaster: Trying to register BlockManager 14/06/09 15:06:30 INFO BlockManagerInfo: Registering block manager 131-1.bfc.hpl.hp.com:47164 with 294.9 MB RAM 14/06/09 15:06:30 INFO BlockManagerMaster: Registered BlockManager 14/06/09 15:06:30 INFO HttpServer: Starting HTTP Server 14/06/09 15:06:30 INFO HttpBroadcast: Broadcast server started at http://16.106.36.131:48587 14/06/09 15:06:30 INFO HttpFileServer: HTTP File server directory is /tmp/spark-35e1c47b-bfa1-4fba-bc64-df8eee287bb7 14/06/09 15:06:30 INFO HttpServer: Starting HTTP Server 14/06/09 15:06:30 INFO SparkUI: Started SparkUI at http://131-1.bfc.hpl.hp.com:4040 14/06/09 15:06:30 INFO SparkContext: Added JAR file:/data/cuihe/spark-app/target/scala-2.10/simple-project_2.10-1.0.jar at http://16.106.36.131:35579/jars/simple-project_2.10-1.0.jar with timestamp 1402351590741 14/06/09 15:06:30 INFO MemoryStore: ensureFreeSpace(32856) called with curMem=0, maxMem=309225062 14/06/09 15:06:30 INFO MemoryStore: Block broadcast_0 stored as values to memory (estimated size 32.1 KB, free 294.9 MB) 14/06/09 15:06:30 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 14/06/09 15:06:30 WARN LoadSnappy: Snappy native library not loaded 14/06/09 15:06:30 INFO FileInputFormat: Total input paths to process : 1 14/06/09 15:06:30 INFO SparkContext: Starting job: count at SimpleApp.scala:14 14/06/09 15:06:31 INFO DAGScheduler: Got job 0 (count at SimpleApp.scala:14) with 7 output partitions (allowLocal=false) 14/06/09 15:06:31 INFO DAGScheduler: Final stage: Stage 0(count at SimpleApp.scala:14) 14/06/09 15:06:31 INFO DAGScheduler: Parents of final stage: List() 14/06/09 15:06:31 INFO DAGScheduler: Missing parents: List() 14/06/09 15:06:31 INFO DAGScheduler: Submitting Stage 0 (FilteredRDD[2] at filter at SimpleApp.scala:14), which has no missing parents 14/06/09 15:06:31 INFO DAGScheduler: Submitting 7 missing tasks from Stage 0 (FilteredRDD[2] at filter at SimpleApp.scala:14) 14/06/09 15:06:31 INFO TaskSchedulerImpl: Adding task set 0.0 with 7 tasks 14/06/09 15:06:31 INFO TaskSetManager: Starting task 0.0:0 as TID 0 on executor localhost: localhost (PROCESS_LOCAL) 14/06/09 15:06:31 INFO TaskSetManager: Serialized task 0.0:0 as 1839 bytes in 2 ms 14/06/09 15:06:31 INFO TaskSetManager: Starting task 0.0:1 as TID 1 on executor localhost: localhost (PROCESS_LOCAL) 14/06/09 15:06:31 INFO TaskSetManager: Serialized task 0.0:1 as 1839 bytes in 0 ms 14/06/09 15:06:31 INFO TaskSetManager: Starting task 0.0:2 as TID 2 on executor localhost: localhost (PROCESS_LOCAL) 14/06/09 15:06:31 INFO TaskSetManager: Serialized task 0.0:2 as 1839 bytes in 1 ms 14/06/09 15:06:31 INFO TaskSetManager: Starting task 0.0:3 as TID 3 on executor localhost: localhost (PROCESS_LOCAL) 14/06/09 15:06:31 INFO TaskSetManager: Serialized task 0.0:3 as 1839 bytes in 1 ms 14/06/09 15:06:31 INFO Executor: Running task ID 0 14/06/09 15:06:31 INFO Executor: Running task ID 1 14/06/09 15:06:31 INFO Executor: Running task ID 2 14/06/09 15:06:31 INFO Executor: Running task ID 3 14/06/09 15:06:31 INFO Executor: Fetching http://16.106.36.131:35579/jars/simple-project_2.10-1.0.jar with timestamp 1402351590741 14/06/09 15:06:31 INFO Utils: Fetching http://16.106.36.131:35579/jars/simple-project_2.10-1.0.jar to /tmp/fetchFileTemp7241193225836706654.tmp 14/06/09 15:06:31 INFO Executor: Adding file:/tmp/spark-68aa13c8-8146-4e6a-80a1-c406a4cef89f/simple-project_2.10-1.0.jar to class loader 14/06/09 15:06:31 INFO BlockManager: Found block broadcast_0 locally 14/06/09 15:06:31 INFO BlockManager: Found block broadcast_0 locally 14/06/09 15:06:31 INFO BlockManager: Found block broadcast_0 locally 14/06/09 15:06:31 INFO BlockManager: Found block broadcast_0 locally 14/06/09 15:06:31 INFO CacheManager: Partition rdd_1_2 not found, computing it 14/06/09 15:06:31 INFO CacheManager: Partition rdd_1_0 not found, computing it 14/06/09 15:06:31 INFO CacheManager: Partition rdd_1_1 not found, computing it 14/06/09 15:06:31 INFO CacheManager: Partition rdd_1_3 not found, computing it 14/06/09 15:06:31 INFO HadoopRDD: Input split: file:/tmp/mdata0-10.tsd:67108864+33554432 14/06/09 15:06:31 INFO HadoopRDD: Input split: file:/tmp/mdata0-10.tsd:33554432+33554432 14/06/09 15:06:31 INFO HadoopRDD: Input split: file:/tmp/mdata0-10.tsd:100663296+33554432 14/06/09 15:06:31 INFO HadoopRDD: Input split: file:/tmp/mdata0-10.tsd:0+33554432 14/06/09 15:06:50 INFO MemoryStore: ensureFreeSpace(108800293) called with curMem=32856, maxMem=309225062 14/06/09 15:06:50 INFO MemoryStore: Block rdd_1_2 stored as values to memory (estimated size 103.8 MB, free 191.1 MB) 14/06/09 15:06:50 INFO MemoryStore: ensureFreeSpace(108716407) called with curMem=108833149, maxMem=309225062 14/06/09 15:06:50 ERROR Executor: Exception in task ID 1 java.lang.OutOfMemoryError: GC overhead limit exceeded at java.util.Arrays.copyOfRange(Arrays.java:2694) at java.lang.String.<init>(String.java:203) at java.nio.HeapCharBuffer.toString(HeapCharBuffer.java:561) at java.nio.CharBuffer.toString(CharBuffer.java:1201) at org.apache.hadoop.io.Text.decode(Text.java:350) at org.apache.hadoop.io.Text.decode(Text.java:327) at org.apache.hadoop.io.Text.toString(Text.java:254) at org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:458) at org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:458) at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103) at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:107) at org.apache.spark.rdd.RDD.iterator(RDD.scala:227) at org.apache.spark.rdd.FilteredRDD.compute(FilteredRDD.scala:34) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262) at org.apache.spark.rdd.RDD.iterator(RDD.scala:229) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111) at org.apache.spark.scheduler.Task.run(Task.scala:51) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 14/06/09 15:06:50 INFO BlockManagerInfo: Added rdd_1_2 in memory on 131-1.bfc.hpl.hp.com:47164 (size: 103.8 MB, free: 191.1 MB) 14/06/09 15:06:50 INFO MemoryStore: Block rdd_1_3 stored as values to memory (estimated size 103.7 MB, free 87.4 MB) 14/06/09 15:06:50 INFO BlockManagerMaster: Updated info of block rdd_1_2 14/06/09 15:06:50 INFO BlockManagerInfo: Added rdd_1_3 in memory on 131-1.bfc.hpl.hp.com:47164 (size: 103.7 MB, free: 87.5 MB) 14/06/09 15:06:50 INFO BlockManagerMaster: Updated info of block rdd_1_3 14/06/09 15:06:50 ERROR ExecutorUncaughtExceptionHandler: Uncaught exception in thread Thread[Executor task launch worker-1,5,main] java.lang.OutOfMemoryError: GC overhead limit exceeded at java.util.Arrays.copyOfRange(Arrays.java:2694) at java.lang.String.<init>(String.java:203) at java.nio.HeapCharBuffer.toString(HeapCharBuffer.java:561) at java.nio.CharBuffer.toString(CharBuffer.java:1201) at org.apache.hadoop.io.Text.decode(Text.java:350) at org.apache.hadoop.io.Text.decode(Text.java:327) at org.apache.hadoop.io.Text.toString(Text.java:254) at org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:458) at org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:458) at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103) at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:107) at org.apache.spark.rdd.RDD.iterator(RDD.scala:227) at org.apache.spark.rdd.FilteredRDD.compute(FilteredRDD.scala:34) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262) at org.apache.spark.rdd.RDD.iterator(RDD.scala:229) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111) at org.apache.spark.scheduler.Task.run(Task.scala:51) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 14/06/09 15:06:50 INFO TaskSetManager: Starting task 0.0:4 as TID 4 on executor localhost: localhost (PROCESS_LOCAL)