[
https://issues.apache.org/jira/browse/SPARK-54536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
angerszhu updated SPARK-54536:
------------------------------
Description:
When ShuffleBlockFetcherIterator initialize, call fetchUpToMaxBytes then call
fetchAllHostLocalBlocks, but fetchAllHostLocalBlocks requests always need to
wait fetchUpToMaxBytes's open block request return then can return. if open
block request return slow, fetchRemoteWaitTime metrics difference is very large
.
!image-2025-11-27-15-53-16-649.png|width=746,height=564!
> Shuffle FetchWaitTime missing first call fetchUpToMaxBytes() open block rpc
> response time
> ------------------------------------------------------------------------------------------
>
> Key: SPARK-54536
> URL: https://issues.apache.org/jira/browse/SPARK-54536
> Project: Spark
> Issue Type: Task
> Components: Spark Core
> Affects Versions: 3.2.4, 4.0.1
> Reporter: angerszhu
> Priority: Major
> Attachments: image-2025-11-27-15-53-16-649.png
>
>
> When ShuffleBlockFetcherIterator initialize, call fetchUpToMaxBytes then call
> fetchAllHostLocalBlocks, but fetchAllHostLocalBlocks requests always need to
> wait fetchUpToMaxBytes's open block request return then can return. if open
> block request return slow, fetchRemoteWaitTime metrics difference is very
> large .
> !image-2025-11-27-15-53-16-649.png|width=746,height=564!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
