[ https://issues.apache.org/jira/browse/SPARK-17484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15488273#comment-15488273 ]
Apache Spark commented on SPARK-17484: -------------------------------------- User 'JoshRosen' has created a pull request for this issue: https://github.com/apache/spark/pull/15085 > Race condition when cancelling a job during a cache write can lead to block > fetch failures > ------------------------------------------------------------------------------------------ > > Key: SPARK-17484 > URL: https://issues.apache.org/jira/browse/SPARK-17484 > Project: Spark > Issue Type: Improvement > Components: Block Manager > Affects Versions: 2.0.0 > Reporter: Josh Rosen > Assignee: Josh Rosen > > On a production cluster, I observed the following weird behavior where a > block manager cached a block, the store failed due to a task being killed / > cancelled, and then a subsequent task incorrectly attempted to read the > cached block from the machine where the write failed, eventually leading to a > complete job failure. > Here's the executor log snippet from the machine performing the failed cache: > {code} > 16/09/06 16:10:31 INFO MemoryStore: Block rdd_25_1 stored as values in memory > (estimated size 976.8 MB, free 9.8 GB) > 16/09/06 16:10:31 WARN BlockManager: Putting block rdd_25_1 failed > 16/09/06 16:10:31 INFO Executor: Executor killed task 0.0 in stage 3.0 (TID > 127) > {code} > Here's the exception from the reader in the failed job: > {code} > org.apache.spark.SparkException: Job aborted due to stage failure: Task 4 in > stage 46.0 failed 4 times, most recent failure: Lost task 4.3 in stage 46.0 > (TID 1484, 10.69.255.197): org.apache.spark.storage.BlockFetchException: > Failed to fetch block after 1 fetch failures. Most recent failure cause: > at > org.apache.spark.storage.BlockManager.getRemoteBytes(BlockManager.scala:565) > at > org.apache.spark.storage.BlockManager.getRemoteValues(BlockManager.scala:522) > at org.apache.spark.storage.BlockManager.get(BlockManager.scala:609) > at > org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:661) > {code} > I believe that there's a race condition in how we handle cleanup after failed > cache stores. Here's an excerpt from {{BlockManager.doPut()}} > {code} > var blockWasSuccessfullyStored: Boolean = false > val result: Option[T] = try { > val res = putBody(putBlockInfo) > blockWasSuccessfullyStored = res.isEmpty > res > } finally { > if (blockWasSuccessfullyStored) { > if (keepReadLock) { > blockInfoManager.downgradeLock(blockId) > } else { > blockInfoManager.unlock(blockId) > } > } else { > blockInfoManager.removeBlock(blockId) > logWarning(s"Putting block $blockId failed") > } > } > {code} > The only way that I think this "successfully stored followed by immediately > failed" case could appear in our logs is if the local memory store write > succeeds and then an exception (perhaps InterruptedException) causes us to > enter the {{finally}} block's error-cleanup path. The problem is that the > {{finally}} block only cleans up the block's metadata rather than performing > the full cleanup path which would also notify the master that the block is no > longer available at this host. > The fact that the Spark task was not resilient in the face of remote block > fetches is a separate issue which I'll report and fix separately. The scope > of this JIRA, however, is the fact that Spark still attempted reads from a > machine which was missing the block. > In order to fix this problem, I think that the {{finally}} block should > perform more thorough cleanup and should send a "block removed" status update > to the master following any error during the write. This is necessary because > the body of {{doPut()}} may have already notified the master of block > availability. In addition, we can extend the block serving code path to > automatically update the master with "block deleted" statuses whenever the > block manager receives invalid requests for blocks that it doesn't have. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org