Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3582#issuecomment-65911726 To clarify a bit further: I think that `BlockInfo.waitForReady()` is designed to allow callers to block until a block write has completed. If we have many threads (readers) waiting for a block to be written, then think we should be okay because `notifyAll()` will wake all of those threads when the block becomes ready. From your description, it sounds like you're worried about a multiple writer-threads case, where we have many threads attempting to write the same block and a failed write attempt from _one_ of them wakes up the waiting threads and notifies them of a failure even though there's another write in progress which might succeed. Is your goal to wait until _all_ of the pending writes have failed before notifying a reader that the write has failed and to wait for _one_ of them to succeed before notifying the reader that the write succeeded? I'll have to dig into the BlockManager internals to see whether we can ever have multiple in-progress writes for the same block. Do you have an example of when this can happen? I'd be happy to look over the code and provide more feedback / suggestions, but I want to make sure that I understand the motivation and confirm that this is fixing an actual bug, since it seems like this adds a moderate amount of complexity.
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org