Hi,
We are trying out state schema migration for one of our stateful pipelines.
We use few Avro type states. Changes made to the job:
1. Updated the schema for one of the states (added a new 'boolean'
field with default value).
2. Modified the code by removing a couple of ValueStates.
To
I believe this should work, with a couple of caveats:
- You can't do this with unaligned checkpoints
- If you have dropped some state, you must specify --allowNonRestoredState
when you restart the job
David
On Wed, Jul 22, 2020 at 4:06 PM Sivaprasanna
wrote:
> Hi,
>
> We are trying out state s
Hi David,
Thanks for the response. I'm actually specifying --allowNonRestoredState
while I submit the job to the yarn session but it still fails with the same
error:
StateMigrationException: The new state serializer cannot be incompatible.
Maybe we cannot resume from incremental checkpoint with s
Hi Sivaprasanna
I think state schema evolution can work for incremental checkpoint. And
I tried with a simple Pojo schema, It also works. maybe you need to check
the schema, from the exception stack, the schema before and after are
incompatible.
Best,
Congxian
Sivaprasanna 于2020年7月24日周五 上午12
Thanks, Congxian & David. There was a mistake on the new schema we used.
After fixing that, we were able to migrate the state, and since we touched
important code blocks, and removed/refactored certain functionalities, we
took a savepoint instead of checkpoint. All good now. Thanks again : )
Sivap
