Hi All, While working on pluggable shuffle architecture, looking into interactive programming and fine-grained recovery efforts, we released that lifecycle management of intermediate result partitions needs more detailed discussion to enable envisioned use cases.
Here I prepared a design document to address this concern. The document proposes certain extensions to FLIP-31 (Design of Pluggable Shuffle Service): https://docs.google.com/document/d/13vAJJxfRXAwI4MtO8dux8hHnNMw2Biu5XRrb_hvGehA Looking forward to your feedback. Thanks, Andrey
