Hi Andrey, Thanks for concluding this valuable proposal. It really extends and enhances the details of partition lifecycle management ever mentioned in FLIP-31. I reviewed the google doc and agreed the general points in it.
One implict point in doc is that the shuffle is not limted to only serve for downstream and upstream in one job, even the data shuffle is required across jobs like interactive feature, so the ShuffleMaster is not coupled with JobMaster. These thoughts are also consitent with our previous confirmation and we could fight in this direction in long term. We would further focus on them in specific PRs considering the feasibility in next 1.9 release. Best, Zhijiang ------------------------------------------------------------------ From:Andrey Zagrebin <and...@ververica.com> Send Time:2019年3月29日(星期五) 18:38 To:dev <dev@flink.apache.org> Subject:[Proposal] Shuffle resources lifecycle management Hi All, While working on pluggable shuffle architecture, looking into interactive programming and fine-grained recovery efforts, we released that lifecycle management of intermediate result partitions needs more detailed discussion to enable envisioned use cases. Here I prepared a design document to address this concern. The document proposes certain extensions to FLIP-31 (Design of Pluggable Shuffle Service): https://docs.google.com/document/d/13vAJJxfRXAwI4MtO8dux8hHnNMw2Biu5XRrb_hvGehA Looking forward to your feedback. Thanks, Andrey
