Hi all First of all, I agreed with Yu that we should support to make state type pluginable.
If we take a look at current Flink implementation. Users could implement their pluginable state backend to satisfy their own meets now. However, even users could define their own state descriptor, they cannot store the customized state within their state backend. The root cause behind this is that current StateBackendFactory could accept user defined state backend factory while StateFactory (within HeapKeyedStateBackend [1] and RocksDBKeyedStateBackend [2] ) cannot. If we agreed that we should leave the right of implementing customized state backend to users, it's naturally to agree that we should also leave the right of implementing customized states to users. [1] https://github.com/apache/flink/blob/576228651382db040aaa006cf9142f6568930cb1/flink-runtime/src/main/java/org/apache/flink/runtime/state/heap/HeapKeyedStateBackend.java#L79 [2] https://github.com/apache/flink/blob/576228651382db040aaa006cf9142f6568930cb1/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java#L114 Best Yun Tang ________________________________ From: Yu Li <car...@gmail.com> Sent: Monday, September 9, 2019 2:24 To: dev <dev@flink.apache.org> Subject: Re: [DISCUSS] Support customize state in customized KeyedStateBackend Hi Shimin, Thanks for bring this discussion up. First of all, I'd like to confirm/clarify that this discussion is mainly about managed state with customized state descriptor rather than raw state, right? Asking because raw state was the very first thing came to my mind when seeing the title. And this is actually the first topic/question we need to discuss, that whether we should support user-defined state descriptor and still ask framework to manage the state life cycle. Personally I'm +1 on this since the "official" state (data-structure) types (currently mainly value, list and map) may not be optimized for customer case, but we'd better ask others' opinion. Secondly, if the result of the first question is "Yes", then it's truly a problem that "Although we can implement customized StateDescriptors for different kind of data structures, we do not really have access to such customized state in RichFunctions", and how to resolve it is the second topic/question to discuss. I've noticed your proposal of exposing "getParitionedState" method out in "RuntimeContext" and "KeyedStateStore" in JIRA (FLINK-14003), but IMO adding a specific interface like below is better than exposing the internal one: <S extends State, V> State getCustomizedState(StateDescriptor<S, V> stateProperties); Finally, I think this is a user-facing and definitely worthwhile discussion, and requires a FLIP to document the conclusion and design/implementation (if any) down. What's your opinion? Thanks. Best Regards, Yu On Fri, 6 Sep 2019 at 13:27, shimin yang <ysmcl...@gmail.com> wrote: > Hi every, > > I would like to start a discussion on supporting customize state > in customized KeyedStateBackend. > > In Flink, users can customize KeyedStateBackend to support different type > of data store. Although we can implement customized StateDescriptors for > different kind of data structrues, we do not really have access to such > customized state in RichFunctions. > > I propose to add a getOtherState method in RuntimeContext and > DefaultKeyedStateStore which directly takes StateDescriptor as parameter to > allow user to get customized state. > > What do you think? > > Thanks. > > Best, > Shimin >
