Re: Enabling allowNonRestoredState when resuming from a savepoint causes ClassNotFoundException

2021-02-10 Thread Roman Khachatryan
Hi Dongwon, With State Processor API you should be able to create a new snapshot that doesn't reference the unused classes. Regards, Roman On Tue, Feb 9, 2021 at 3:39 AM Dongwon Kim wrote: > Hi Khachatryan, > > Thanks for the explanation and the input! > > 1. Use the State Processor API to

Re: Enabling allowNonRestoredState when resuming from a savepoint causes ClassNotFoundException

2021-02-08 Thread Dongwon Kim
Hi Khachatryan, Thanks for the explanation and the input! 1. Use the State Processor API to create a new snapshot [1] I haven't used it. but does the API prevent the class of a specific serializer from being loaded? 2. If the operator has only this state then changing uid (together with >

Re: Enabling allowNonRestoredState when resuming from a savepoint causes ClassNotFoundException

2021-02-08 Thread Khachatryan Roman
Hi, I'm pulling Yun Tang who is familiar with StateBackends and RocksDB in particular. >From what I see, the 2nd snapshot (sp2) is built using the same set of states obtained from the starting savepoint/checkpoint (sp1) to write its metadata. This metadata includes serializers snapshots,

Re: Enabling allowNonRestoredState when resuming from a savepoint causes ClassNotFoundException

2021-02-08 Thread Dongwon Kim
Hi 张静, Q1: By default, a savepoint restore will try to match all state > back to the restored job. `AllowNonRestoredState` cannot avoid > recovery all state from savepoint, but only skip match all of the > restore state back to the restored job. So `ClassNotFoundException ` > could not be

Enabling allowNonRestoredState when resuming from a savepoint causes ClassNotFoundException

2021-02-08 Thread Dongwon Kim
Hi, I have an original job (say v1) and I want to start a new job (say v2) from a savepoint of v1. An operator of v1 used to have per-key states of a POJO type, but I want to remove the states together with the definition of the POJO type. When I start v2 from a savepoint of v1, I specified