[
https://issues.apache.org/jira/browse/SPARK-14209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214731#comment-15214731
]
Marcelo Vanzin commented on SPARK-14209:
----------------------------------------
Ah, my bad, I should have noticed this in the original stack trace. This seems
to be related to caching and not shuffle, right? Can you confirm that you're
caching RDDs in that app?
It seems that code should be more resilient to executors going down.
> Application failure during preemption.
> --------------------------------------
>
> Key: SPARK-14209
> URL: https://issues.apache.org/jira/browse/SPARK-14209
> Project: Spark
> Issue Type: Bug
> Components: Block Manager
> Affects Versions: 1.6.1
> Environment: Spark on YARN
> Reporter: Miles Crawford
>
> We have a fair-sharing cluster set up, including the external shuffle
> service. When a new job arrives, existing jobs are successfully preempted
> down to fit.
> A spate of these messages arrives:
> ExecutorLostFailure (executor 48 exited unrelated to the running tasks)
> Reason: Container container_1458935819920_0019_01_000143 on host:
> ip-10-12-46-235.us-west-2.compute.internal was preempted.
> This seems fine - the problem is that soon thereafter, our whole application
> fails because it is unable to fetch blocks from the pre-empted containers:
> org.apache.spark.storage.BlockFetchException: Failed to fetch block from 1
> locations. Most recent failure cause:
> Caused by: java.io.IOException: Failed to connect to
> ip-10-12-46-235.us-west-2.compute.internal/10.12.46.235:55681
> Caused by: java.net.ConnectException: Connection refused:
> ip-10-12-46-235.us-west-2.compute.internal/10.12.46.235:55681
> Full stack: https://gist.github.com/milescrawford/33a1c1e61d88cc8c6daf
> Spark does not attempt to recreate these blocks - the tasks simply fail over
> and over until the maxTaskAttempts value is reached.
> It appears to me that there is some fault in the way preempted containers are
> being handled - shouldn't these blocks be recreated on demand?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
