[ https://issues.apache.org/jira/browse/SPARK-47141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
mahesh kumar behera updated SPARK-47141: ---------------------------------------- Summary: Support enabling migration of shuffle data directly to external storage using config parameter (was: Support shuffle migration to external storage) > Support enabling migration of shuffle data directly to external storage using > config parameter > ---------------------------------------------------------------------------------------------- > > Key: SPARK-47141 > URL: https://issues.apache.org/jira/browse/SPARK-47141 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 4.0.0 > Reporter: mahesh kumar behera > Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > Currently Spark supports migration of shuffle data to peer nodes during node > decommissioning. If peer nodes are not accessible, then Spark falls back to > external storage. User needs to provide the storage location path. There are > scenarios where user may want to migrate to external storage instead of peer > nodes. This may be because of unstable nodes or due to the need of > aggressive scale down. So user should be able to configure to migrate the > shuffle data directly to external storage if the use case permits. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org