Hi everyone,
Earlier this year, we proposed SPARK-25299, proposing the idea of using other storage systems for persisting shuffle files. Since that time, we have been continuing to work on prototypes for this project. In the interest of increasing transparency into our work, we have created a progress report document where you may find a summary of the work we have been doing, as well as links to our prototypes on Github. We would ask that anyone who is very familiar with the inner workings of Spark’s shuffle could provide feedback and comments on our work thus far. We welcome any further discussion in this space. You may comment in this e-mail thread or by commenting on the progress report document. Looking forward to hearing from you. Thanks, -Matt Cheah
smime.p7s
Description: S/MIME cryptographic signature
