Re: Keyed raw state - example
Hello Congxian, Thanks for your response, Don't you have an example with an Operator extending the AbstractUdfStreamOperator? Using the context.getRawKeyedStateInputs() (& output to snapshots) TimeService is reimplementing the whole stuff :/ -- Bastien DINE Data Architect / Software Engineer / Sysadmin bastiendine.io Le lun. 18 nov. 2019 à 03:19, Congxian Qiu a écrit : > Hi >Currently, I think you can ref the implementation of timerservice[1] > which used the raw keyed state, the snapshot happens in > AbstractStreamOperator#snapshotState(), for using Raw State you need to > implement a new operator[2]. There is an issue wants to give some example > for raw state[2] > > [1] > https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/InternalTimeServiceManager.java > [2] > https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html#raw-and-managed-state > [3] https://issues.apache.org/jira/browse/FLINK-14379 > > Best, > Congxian > > > bastien dine 于2019年11月16日周六 上午5:57写道: > >> Hello everyone, >> >> I would like to know if anybody has a working example on how to declare a >> keyed raw state ( in my case a keyedprocessoperator) and how to use it in >> my UDF (keyedprocessfunction)? >> >> Basicaly we have a huge problem with a ValueState w Rocksdb, getting >> serialized for every element ( need to access it and update) so it's taking >> a crazy amount of time and we would like to have it serialized only on >> snapshot, so using Raw state is a possible good solution, >> But i cannot find anyexample of it :/ >> >> Thanks and best regards, >> >> Bastien DINE >> Freelance >> Data Architect / Software Engineer / Sysadmin >> http://bastiendine.io >> >> >> >
Re: Keyed raw state - example
Hi Currently, I think you can ref the implementation of timerservice[1] which used the raw keyed state, the snapshot happens in AbstractStreamOperator#snapshotState(), for using Raw State you need to implement a new operator[2]. There is an issue wants to give some example for raw state[2] [1] https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/InternalTimeServiceManager.java [2] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html#raw-and-managed-state [3] https://issues.apache.org/jira/browse/FLINK-14379 Best, Congxian bastien dine 于2019年11月16日周六 上午5:57写道: > Hello everyone, > > I would like to know if anybody has a working example on how to declare a > keyed raw state ( in my case a keyedprocessoperator) and how to use it in > my UDF (keyedprocessfunction)? > > Basicaly we have a huge problem with a ValueState w Rocksdb, getting > serialized for every element ( need to access it and update) so it's taking > a crazy amount of time and we would like to have it serialized only on > snapshot, so using Raw state is a possible good solution, > But i cannot find anyexample of it :/ > > Thanks and best regards, > > Bastien DINE > Freelance > Data Architect / Software Engineer / Sysadmin > http://bastiendine.io > > >
Keyed raw state - example
Hello everyone, I would like to know if anybody has a working example on how to declare a keyed raw state ( in my case a keyedprocessoperator) and how to use it in my UDF (keyedprocessfunction)? Basicaly we have a huge problem with a ValueState w Rocksdb, getting serialized for every element ( need to access it and update) so it's taking a crazy amount of time and we would like to have it serialized only on snapshot, so using Raw state is a possible good solution, But i cannot find anyexample of it :/ Thanks and best regards, Bastien DINE Freelance Data Architect / Software Engineer / Sysadmin http://bastiendine.io