[
https://issues.apache.org/jira/browse/SPARK-26795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
feiwang updated SPARK-26795:
----------------------------
Description:
There is a parameter spark.maxRemoteBlockSizeFetchToMem, which means the remote
block will be fetched to disk when size of the block is above this threshold in
bytes.
So during shuffle read phase, the managedBuffer which throw IOException may be
a remote downloaded FileSegment and should be retried instead of
throwFetchFailed directly.
was:During shuffle read phase, the managedBuffer which throw IOException may
be a remote download FileSegment and should be retry instead of
throwFetchFailed directly.
> Retry remote fileSegmentManagedBuffer when creating inputStream failed during
> shuffle read phase
> ------------------------------------------------------------------------------------------------
>
> Key: SPARK-26795
> URL: https://issues.apache.org/jira/browse/SPARK-26795
> Project: Spark
> Issue Type: Bug
> Components: Shuffle
> Affects Versions: 2.3.0, 2.3.1, 2.3.2, 2.4.0
> Reporter: feiwang
> Priority: Major
> Fix For: 2.3.0, 2.3.1, 2.3.2, 2.4.0
>
>
> There is a parameter spark.maxRemoteBlockSizeFetchToMem, which means the
> remote block will be fetched to disk when size of the block is above this
> threshold in bytes.
> So during shuffle read phase, the managedBuffer which throw IOException may
> be a remote downloaded FileSegment and should be retried instead of
> throwFetchFailed directly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
