GitHub user rezasafi opened a pull request:
https://github.com/apache/spark/pull/22325
[SPARK-25318]. Add exception handling when wrapping the input stream during
the the fetch or stage retry in response to a corrupted block
SPARK-4105 provided a solution to block corruption issue by retrying the
fetch or the stage. In that solution there is a step that wraps the input
stream with compression and/or encryption. This step is prone to exceptions,
but in the current code there is no exception handling for this step and this
has caused confusion for the user. This change adds exception handling for the
wrapping step and also adds a fetch retry if we experience a corruption during
the wrapping step.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rezasafi/spark localcorruption
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22325.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22325
----
commit cc1c4cdf2bd3b77326f831212c64ede338c807b1
Author: Reza Safi <rezasafi@...>
Date: 2018-09-04T03:06:33Z
[SPARK-25318]. Add exception handling when wrapping the input stream during
the the fetch or stage retry in response to a corrupted block
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
