[
https://issues.apache.org/jira/browse/SPARK-53898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
wuyi updated SPARK-53898:
-------------------------
Description: MapOutputTrackerMaster.shufflestatuses can be mistakenly
cleaned by Shuffle Cleanup feature, leading to SparkException (crashing the
SparkContext) by the subsequent access to that already removed shuffle
metadata. A real case (limited to local cluster currently) is the ongoing
subquery could access the shuffle metadata which has been already cleanedup
after the main query completes. See the detailed discussion at:
[https://github.com/apache/spark/pull/52213#discussion_r2415632474]. (was:
MapOutputTrackerMaster.shufflestatuses can be mistakenly cleaned by Shuffle
Cleanup feature, leading to SparkException (crashing the SparkContext) by the
subsequent access to that already removed shuffle metadata. A real case is the
ongoing subquery could access the shuffle metadata which has been already
cleanedup after the main query completes. See the detailed discussion at:
https://github.com/apache/spark/pull/52213#discussion_r2415632474.)
> MapOutputTrackerMaster.shufflestatuses is mistakenly cleaned by Shuffle
> cleanup
> -------------------------------------------------------------------------------
>
> Key: SPARK-53898
> URL: https://issues.apache.org/jira/browse/SPARK-53898
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 4.0.0, 4.0.1
> Reporter: wuyi
> Priority: Major
>
> MapOutputTrackerMaster.shufflestatuses can be mistakenly cleaned by Shuffle
> Cleanup feature, leading to SparkException (crashing the SparkContext) by the
> subsequent access to that already removed shuffle metadata. A real case
> (limited to local cluster currently) is the ongoing subquery could access the
> shuffle metadata which has been already cleanedup after the main query
> completes. See the detailed discussion at:
> [https://github.com/apache/spark/pull/52213#discussion_r2415632474].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
