Suren Hiraman created SPARK-2202:
------------------------------------
Summary: saveAsTextFile hangs on final 2 tasks
Key: SPARK-2202
URL: https://issues.apache.org/jira/browse/SPARK-2202
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 1.0.0
Environment: CentOS 5.7
16 nodes, 24 cores per node, 14g RAM per executor
Reporter: Suren Hiraman
Priority: Blocker
I have a flow that takes in about 10 GB of data and writes out about 10 GB of
data.
The final step is saveAsTextFile() to HDFS. This seems to hang on 2 remaining
tasks, always on the same node.
It seems that the 2 tasks are waiting for data from a remote task/RDD partition.
After about 2 hours or so, the stuck tasks get a closed connection exception
and you can see the remote side logging that as well. Log lines are below.
My custom settings are:
conf.set("spark.executor.memory", "14g") // TODO make this
configurable
// shuffle configs
conf.set("spark.default.parallelism", "320")
conf.set("spark.shuffle.file.buffer.kb", "200")
conf.set("spark.reducer.maxMbInFlight", "96")
conf.set("spark.rdd.compress","true")
conf.set("spark.worker.timeout","180")
// akka settings
conf.set("spark.akka.threads", "300")
conf.set("spark.akka.timeout", "180")
conf.set("spark.akka.frameSize", "100")
conf.set("spark.akka.batchSize", "30")
conf.set("spark.akka.askTimeout", "30")
// block manager
conf.set("spark.storage.blockManagerTimeoutIntervalMs", "180000")
conf.set("spark.blockManagerHeartBeatMs", "80000")
"STUCK" WORKER
14/06/18 19:41:18 WARN network.ReceivingConnection: Error reading from
connection to ConnectionManagerId(172.16.25.103,57626)
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:251)
at sun.nio.ch.IOUtil.read(IOUtil.java:224)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:254)
at org.apache.spark.network.ReceivingConnection.read(Connection.scala:496)
REMOTE WORKER
14/06/18 19:41:18 INFO network.ConnectionManager: Removing ReceivingConnection
to ConnectionManagerId(172.16.25.124,55610)
14/06/18 19:41:18 ERROR network.ConnectionManager: Corresponding
SendingConnectionManagerId not found
--
This message was sent by Atlassian JIRA
(v6.2#6252)
