holdenk commented on pull request #31493: URL: https://github.com/apache/spark/pull/31493#issuecomment-774275909
> In general, this mitigation cannot avoid the disk full situation fundamentally. Can we choose the migration target in a better way instead of this rejecting because this is due to the blind migration target selection? Yeah so this is a two part JIRA. The first part allows users to configure a maximum amount of shuffle blocks to store before rejecting remote shuffle blocks, and then longer term https://issues.apache.org/jira/browse/SPARK-34364 proposes to allow users to configure a maximum disk pressure threshold. The primary reason I split this into two is that on K8s, measuring disk pressure is pretty complicated and I haven't been able to think of a good solution. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
