Hi, I am running Spark 0.9.2 on an EC2 cluster with about 16 r3.4xlarge machines The cluster is running Spark standalone and is launched with the ec2 scripts. In my Spark job, I am using ephemeral HDFS to checkpoint some of my RDDs. I'm also reading and writing to S3. My jobs also involve a large amountf of shuffles.
I run the same job on multiple set of data and for 50-70% of these runs, the job completes with no issues. (Typically a rerun will allow the "failures" to complete as well) However on the rest of the 30%, I see a bunch of different kinds of issues pop up. (which will go away if I rerun the same job) (1) Checkpointing silently fails (I assume). the checkpoint dir exists in HDFS, but no data files are written out. And a later step in the job tries to reload these RDDs and I get a failure about not being able to read from HDFS. -- Usually a start, stop-dfs "cures" this. *Q: What could be the cause of this? Timeouts? * (2) Other times I get ... no idea who or what is causing this... in master /spark/logs: 2014-08-21 16:46:15 ERROR EndpointWriter: AssociationError [akka.tcp:// sparkmas...@ec2-54-218-216-19.us-west-2.compute.amazonaws.com:7077] -> [akka.tcp://sp...@ip-10-34-2-246.us-west-2.compute.internal:37681]: Error [Association failed with [akka.tcp://spark@ip-10 -34-2-246.us-west-2.compute.internal:37681]] [ akka.remote.EndpointAssociationException: Association failed with [akka.tcp://sp...@ip-10-34-2-246.us-west-2.compute.internal:37681] Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: ip-10-34-2-246.us-west-2.compute.internal/ 10.34.2.246:37681 ] Slave Log: 2014-08-21 16:46:47 INFO ConnectionManager: Removing SendingConnection to ConnectionManagerId(ip-10-33-7-4.us-west-2.compute.internal,33242) 2014-08-21 16:46:47 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionManagerId(ip-10-33-7-4.us-west-2.compute.internal,33242) java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295) at org.apache.spark.network.SendingConnection.read(Connection.scala:398) at org.apache.spark.network.ConnectionManager$$anon$5.run(ConnectionManager.scala:158) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) *Q: Where do I even start debugging this kind of issues? Are the machines too loaded and so timeouts are getting hit? Am I not setting some configuration number correctly? I would be grateful for some hints on where to start looking!* (3) Often (2) will be preceeded by the following in spark.logs.. 2014-08-21 16:34:10 WARN TaskSetManager: Lost TID 102135 (task 398.0:147) 2014-08-21 16:34:10 WARN TaskSetManager: Loss was due to fetch failure from BlockManagerId(0, ip-10-33-131-250.us-west-2.compute.internal, 51371, 0) 2014-08-21 16:34:10 WARN TaskSetManager: Loss was due to fetch failure from BlockManagerId(0, ip-10-33-131-250.us-west-2.compute.internal, 51371, 0) 2014-08-21 16:34:10 WARN TaskSetManager: Loss was due to fetch failure from BlockManagerId(0, ip-10-33-131-250.us-west-2.compute.internal, 51371, 0) Not sure if this is an indication... I'll be very grateful for any ideas on how to start debugging these. Is there anything I should be noting -- CPU using on Master/Slave. Number of executors/cpu, akka threads etc? Cheers, shay