@Konstantin agreed, that was a large impotence for this feature. Also I am happy to change the name to something that better describes the feature set.
@Lan Savepoints depend heavily on a number of flink internal components that may change between versions: state backends internals, type serializers, the specific hash function used to turn a UID into an OperatorID, etc. I consider it a feature of this proposal that the library depends on those internal components instead of reverse engineering the binary format. This way as those internals change, or new state features are added (think the recent addition of TTL) we will get support automatically. I do not believe anything else is maintainable. Seth > On May 31, 2019, at 5:56 AM, Jan Lukavský <je...@seznam.cz> wrote: > > Hi, > > this is awesome, and really useful feature. If I might ask for one thing to > consider - would it be possible to make the Savepoint manipulation API (at > least writing the Savepoint) less dependent on other parts of Flink internals > (e.g. |KeyedStateBootstrapFunction|) and provide something more general (e.g. > some generic Writer)? Why I'm asking for that - I can totally imagine > situation, where users might want to create bootstrapped state by some other > runner (e.g. Apache Spark), and then run Apache Flink after the state has > been created. This makes even more sense in context of Apache Beam, which > provides all the necessary work to make this happen. The question is - would > it be possible to design this feature so that writing the savepoint from > different runner would be possible? > > Cheers, > > Jan > >> On 5/30/19 1:14 AM, Seth Wiesman wrote: >> Hey Everyone! >> >> Gordon and I have been discussing adding a savepoint connector to flink for >> reading, writing and modifying savepoints. >> >> This is useful for: >> >> Analyzing state for interesting patterns >> Troubleshooting or auditing jobs by checking for discrepancies in state >> Bootstrapping state for new applications >> Modifying savepoints such as: >> Changing max parallelism >> Making breaking schema changes >> Correcting invalid state >> >> We are looking forward to your feedback! >> >> This is the FLIP: >> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-43%3A+Savepoint+Connector >> >> Seth >> >> >>
