Hi Anwar! 0.10.0 was feature frozen at that time already and under testing. Key/value state on connected streams will have to go into the next release...
Stephan On Mon, Nov 16, 2015 at 3:00 PM, Anwar Rizal <anriza...@gmail.com> wrote: > Stephan, > > Having a look at the brand new 0.10 release, I noticed that OperatorState > is not implemented for ConnectedStream, which is quite the opposite of what > you said below. > > Or maybe I misunderstood your sentence here ? > > Thanks, > Anwar. > > > On Wed, Nov 11, 2015 at 10:49 AM, Stephan Ewen <se...@apache.org> wrote: > >> Hi! >> >> In general, if you can keep state in Flink, you get better >> throughput/latency/consistency and have one less system to worry about >> (external k/v store). State outside means that the Flink processes can be >> slimmer and need fewer resources and as such recover a bit faster. There >> are use cases for that as well. >> >> Storing the model in OperatorState is a good idea, if you can. On the >> roadmap is to migrate the operator state to managed memory as well, so that >> should take care of the GC issues. >> >> We are just adding functionality to make the Key/Value operator state >> usable in CoMap/CoFlatMap as well (currently it only works in windows and >> in Map/FlatMap/Filter functions over the KeyedStream). >> Until the, you should be able to use a simple Java HashMap and use the >> "Checkpointed" interface to get it persistent. >> >> Greetings, >> Stephan >> >> >> On Sun, Nov 8, 2015 at 10:11 AM, Welly Tambunan <if05...@gmail.com> >> wrote: >> >>> Thanks for the answer. >>> >>> Currently the approach that i'm using right now is creating a >>> base/marker interface to stream different type of message to the same >>> operator. Not sure about the performance hit about this compare to the >>> CoFlatMap function. >>> >>> Basically this one is providing query cache, so i'm thinking instead of >>> using in memory cache like redis, ignite etc, i can just use operator state >>> for this one. >>> >>> I just want to gauge do i need to use memory cache or operator state >>> would be just fine. >>> >>> However i'm concern about the Gen 2 Garbage Collection for caching our >>> own state without using operator state. Is there any clarification on that >>> one ? >>> >>> >>> >>> On Sat, Nov 7, 2015 at 12:38 AM, Anwar Rizal <anriza...@gmail.com> >>> wrote: >>> >>>> >>>> Let me understand your case better here. You have a stream of model and >>>> stream of data. To process the data, you will need a way to access your >>>> model from the subsequent stream operations (map, filter, flatmap, ..). >>>> I'm not sure in which case Operator State is a good choice, but I think >>>> you can also live without. >>>> >>>> val modelStream = .... // get the model stream >>>> val dataStream = >>>> >>>> modelStream.broadcast.connect(dataStream). coFlatMap( ) Then you can >>>> keep the latest model in a CoFlatMapRichFunction, not necessarily as >>>> Operator State, although maybe OperatorState is a good choice too. >>>> >>>> Does it make sense to you ? >>>> >>>> Anwar >>>> >>>> On Fri, Nov 6, 2015 at 10:21 AM, Welly Tambunan <if05...@gmail.com> >>>> wrote: >>>> >>>>> Hi All, >>>>> >>>>> We have a high density data that required a downsample. However this >>>>> downsample model is very flexible based on the client device and user >>>>> interaction. So it will be wasteful to precompute and store to db. >>>>> >>>>> So we want to use Apache Flink to do downsampling and cache the result >>>>> for subsequent query. >>>>> >>>>> We are considering using Flink Operator state for that one. >>>>> >>>>> Is that the right approach to use that for memory cache ? Or if that >>>>> preferable using memory cache like redis etc. >>>>> >>>>> Any comments will be appreciated. >>>>> >>>>> >>>>> Cheers >>>>> -- >>>>> Welly Tambunan >>>>> Triplelands >>>>> >>>>> http://weltam.wordpress.com >>>>> http://www.triplelands.com <http://www.triplelands.com/blog/> >>>>> >>>> >>>> >>> >>> >>> -- >>> Welly Tambunan >>> Triplelands >>> >>> http://weltam.wordpress.com >>> http://www.triplelands.com <http://www.triplelands.com/blog/> >>> >> >> >
