State migration into multiple operators

[bastien dine]

Hello,

I would like to have some advices about splitting an operator with a state
into multiple operators.
The new operators would have state containing pieces of information of the
initial state
We will "split" the state

For exemple, I have operator (process) with uid A, with a state containing
field1, field2
I would like to split it into two operator B & C with, respectively, state
field1 and state field2

How can I split my state upon multiple operators ?

Best Regards,
Bastien

--

Bastien DINE
Data Architect / Software Engineer / Sysadmin
bastiendine.io

Re: State migration into multiple operators

[Piotr Nowojski]

Hi,

Currently there is no native Flink support for modifying the state in a such 
manner. However there is an on-going effort [1] and a third party project [2] 
to address exactly this. Both allows you you to read savepoint, modify it and 
write back the new modified savepoint from which you can restore.

[1] https://github.com/sjwiesman/flink/commits/savepoint-connector 

[2] https://github.com/king/bravo 

You can take a look at them.

Piotrek

> On 14 May 2019, at 09:55, bastien dine  wrote:
> 
> Hello,
> 
> I would like to have some advices about splitting an operator with a state 
> into multiple operators.
> The new operators would have state containing pieces of information of the 
> initial state
> We will "split" the state
> 
> For exemple, I have operator (process) with uid A, with a state containing 
> field1, field2
> I would like to split it into two operator B & C with, respectively, state 
> field1 and state field2
> 
> How can I split my state upon multiple operators ?
> 
> Best Regards,
> Bastien 
> 
> --
> 
> Bastien DINE
> Data Architect / Software Engineer / Sysadmin
> bastiendine.io 

Re: State migration into multiple operators

[Tzu-Li (Gordon) Tai]

Hi,

Just to add to what Piotr already mentioned:
The community is working on adding support for this directly in Flink.
You can follow the efforts here:
https://issues.apache.org/jira/browse/FLINK-12047.

Cheers,
Gordon


On Tue, May 14, 2019 at 11:39 AM Piotr Nowojski  wrote:

> Hi,
>
> Currently there is no native Flink support for modifying the state in a
> such manner. However there is an on-going effort [1] and a third party project
> [2] to address exactly this. Both allows you you to read savepoint,
> modify it and write back the new modified savepoint from which you can
> restore.
>
> [1] https://github.com/sjwiesman/flink/commits/savepoint-connector
> [2] https://github.com/king/bravo
>
> You can take a look at them.
>
> Piotrek
>
> On 14 May 2019, at 09:55, bastien dine  wrote:
>
> Hello,
>
> I would like to have some advices about splitting an operator with a state
> into multiple operators.
> The new operators would have state containing pieces of information of the
> initial state
> We will "split" the state
>
> For exemple, I have operator (process) with uid A, with a state containing
> field1, field2
> I would like to split it into two operator B & C with, respectively, state
> field1 and state field2
>
> How can I split my state upon multiple operators ?
>
> Best Regards,
> Bastien
>
> --
>
> Bastien DINE
> Data Architect / Software Engineer / Sysadmin
> bastiendine.io
>
>
>