I want to quickly bump this discussion to gather more consensus from others on the FLIP, and see if we want to aim this for the upcoming 1.9.0 release. The proposal touches binary formats of savepoints, which is a major part of Flink's public user interface, so having explicit approval from other members of the community would be nice here.
Cheers, Gordon On Wed, May 29, 2019 at 11:45 AM Tzu-Li (Gordon) Tai <tzuli...@apache.org> wrote: > I also should point out something that I forgot to mention in the initial > post: > Stefan has helped a lot in understanding the current status of state > backends and also participated a lot in design choices for the FLIP :) > > On Wed, May 29, 2019 at 5:02 AM Tzu-Li (Gordon) Tai <tzuli...@apache.org> > wrote: > >> Hi Flink devs, >> >> Congxian, Kostas, and I have recently been discussing to unify the binary >> formats for keyed state in savepoints, which would allow for more >> operational flexibility such as swapping state backends across restores. >> >> As part of this FLIP, another main proposal is to start allowing >> checkpoints and savepoints to have different formats. Savepoint formats >> should in the future be designed with interoperability in mind and >> reasonable snapshot / restore overhead is tolerable, while checkpoints are >> allowed to be backend specific for more efficient snapshots and restores. >> From recent proposals in the state backends such as disk-spilling heap >> backend [1], this flexibility seems to be reasonable. >> >> The main user-facing API this would affect is of course, the binary >> formats of savepoints, as well as the fact that we will no longer be >> guaranteeing functional parity between savepoints and full checkpoints in >> the future (w.r.t. operational features related to upgrading applications; >> so far they have equal functionality). >> >> Therefore, we would like to collect feedback on the proposal before >> continuing efforts. >> >> This is the FLIP: >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-41%3A+Unify+Keyed+State+Snapshot+Binary+Format+for+Savepoints >> . >> >> I'm happy to discuss details and looking forward to any feedback. >> >> Cheers, >> Gordon >> >> [1] >> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Proposal-to-support-disk-spilling-in-HeapKeyedStateBackend-td29109.html >> >
