hello everyone,
I have a problem when I run the wordcount example. I read data from hdfs , its
almost 7G.
I haven't seen the info from the web ui or sparkhome/work . This is the console
info :
.....
13/12/16 19:48:02 INFO LocalTaskSetManager: Size of task 52 is 1834 bytes
13/12/16 19:48:02 INFO LocalScheduler: Running 52
13/12/16 19:48:02 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Getting
52 non-zero-bytes blocks out of 52 blocks
13/12/16 19:48:02 INFO BlockFetcherIterator$BasicBlockFetcherIterator: Started
0 remote gets in 7 ms
13/12/16 19:48:09 INFO LocalTaskSetManager: Loss was due to
java.lang.OutOfMemoryError
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2271)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
at
java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
at
java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1857)
at
java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1766)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1185)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:346)
at
org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:27)
at
org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:47)
at
org.apache.spark.scheduler.local.LocalScheduler.runTask(LocalScheduler.scala:204)
at
org.apache.spark.scheduler.local.LocalActor$$anonfun$launchTask$1$$anon$1.run(LocalScheduler.scala:68)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
13/12/16 19:48:09 INFO LocalScheduler: Remove TaskSet 0.0 from pool
13/12/16 19:48:09 INFO DAGScheduler: Failed to run collect at <console>:17
org.apache.spark.SparkException: Job failed: Task 0.0:0 failed more than 4
times; aborting job java.lang.OutOfMemoryError: Java heap space
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:760)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:758)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:758)
at
org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:379)
at
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$run(DAGScheduler.scala:441)
at
org.apache.spark.scheduler.DAGScheduler$$anon$1.run(DAGScheduler.scala:149)
this is my spark-env.sh :
export SPARK_HOME=/home/lh1/spark_hadoopapp/spark-0.8.0-hadoop2.0.0-cdh4.2.1
export JAVA_HOME=/home/lh1/app/jdk1.7.0
export SCALA_HOME=/home/lh1/sparkapp/scala-2.9.3
export SPARK_WORKER_CORES=2
export SPARK_WORKER_MEMORY=1024m
export SPARK_WORKER_INSTANCES=2
export SPARK_DAEMON_JAVA_OPTS=9000m
I just started to use Spark , so can you give me some suggestions ?
Thanks .
Leo
leosand...@gmail.com
