mccheah opened a new pull request #28618: URL: https://github.com/apache/spark/pull/28618
### What changes were proposed in this pull request? Adds a `ShuffleOutputTracker` API that can be used for managing shuffle metadata on the driver. Accepts map output metadata returned by the map output writers. Requires https://github.com/apache/spark/pull/28616. ### Why are the changes needed? Part of the design as discussed in [this document](https://docs.google.com/document/d/1Aj6IyMsbS2sdIfHxLvIbHUNjHIWHTabfknIPoxOrTjk/edit#), and part of the wider effort of SPARK-25299. ### Does this PR introduce _any_ user-facing change? Enables additional APIs for the shuffle storage plugin tree. Usage will become more apparent when the read side of the shuffle plugin tree is introduced. ### How was this patch tested? We've added a mock implementation of the shuffle plugin tree here, to prove that a Spark job using a different implementation of the plugin can use all of the plugin points for an alternative shuffle data storage solution. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
