Awesome ! This is really the best weekend gift ever. :)
Cheers On Fri, Nov 13, 2015 at 3:54 PM, Robert Metzger <rmetz...@apache.org> wrote: > Hi Welly, > Flink 0.10.0 is out, its just not announced yet. > Its available on maven central and the global mirrors are currently > syncing it. This mirror for example has the update already: > http://apache.mirror.digionline.de/flink/flink-0.10.0/ > > On Fri, Nov 13, 2015 at 9:50 AM, Welly Tambunan <if05...@gmail.com> wrote: > >> Hi Aljoscha, >> >> Thanks for this one. Looking forward for 0.10 release version. >> >> Cheers >> >> On Thu, Nov 12, 2015 at 5:34 PM, Aljoscha Krettek <aljos...@apache.org> >> wrote: >> >>> Hi, >>> I don’t know yet when the operator state will be transitioned to managed >>> memory but it could happen for 1.0 (which will come after 0.10). The good >>> thing is that the interfaces won’t change, so state can be used as it is >>> now. >>> >>> For 0.10, the release vote is winding down right now, so you can expect >>> the release to happen today or tomorrow. I think the streaming is >>> production ready now, we expect to mostly to hardening and some >>> infrastructure changes (for example annotations that specify API stability) >>> for the 1.0 release. >>> >>> Let us know if you need more information. >>> >>> Cheers, >>> Aljoscha >>> > On 12 Nov 2015, at 02:42, Welly Tambunan <if05...@gmail.com> wrote: >>> > >>> > Hi Stephan, >>> > >>> > >Storing the model in OperatorState is a good idea, if you can. On the >>> roadmap is to migrate the operator state to managed memory as well, so that >>> should take care of the GC issues. >>> > Is this using off the heap memory ? Which version we expect this one >>> to be available ? >>> > >>> > Another question is when will the release version of 0.10 will be out >>> ? We would love to upgrade to that one when it's available. That version >>> will be a production ready streaming right ? >>> > >>> > >>> > >>> > >>> > >>> > On Wed, Nov 11, 2015 at 4:49 PM, Stephan Ewen <se...@apache.org> >>> wrote: >>> > Hi! >>> > >>> > In general, if you can keep state in Flink, you get better >>> throughput/latency/consistency and have one less system to worry about >>> (external k/v store). State outside means that the Flink processes can be >>> slimmer and need fewer resources and as such recover a bit faster. There >>> are use cases for that as well. >>> > >>> > Storing the model in OperatorState is a good idea, if you can. On the >>> roadmap is to migrate the operator state to managed memory as well, so that >>> should take care of the GC issues. >>> > >>> > We are just adding functionality to make the Key/Value operator state >>> usable in CoMap/CoFlatMap as well (currently it only works in windows and >>> in Map/FlatMap/Filter functions over the KeyedStream). >>> > Until the, you should be able to use a simple Java HashMap and use the >>> "Checkpointed" interface to get it persistent. >>> > >>> > Greetings, >>> > Stephan >>> > >>> > >>> > On Sun, Nov 8, 2015 at 10:11 AM, Welly Tambunan <if05...@gmail.com> >>> wrote: >>> > Thanks for the answer. >>> > >>> > Currently the approach that i'm using right now is creating a >>> base/marker interface to stream different type of message to the same >>> operator. Not sure about the performance hit about this compare to the >>> CoFlatMap function. >>> > >>> > Basically this one is providing query cache, so i'm thinking instead >>> of using in memory cache like redis, ignite etc, i can just use operator >>> state for this one. >>> > >>> > I just want to gauge do i need to use memory cache or operator state >>> would be just fine. >>> > >>> > However i'm concern about the Gen 2 Garbage Collection for caching our >>> own state without using operator state. Is there any clarification on that >>> one ? >>> > >>> > >>> > >>> > On Sat, Nov 7, 2015 at 12:38 AM, Anwar Rizal <anriza...@gmail.com> >>> wrote: >>> > >>> > Let me understand your case better here. You have a stream of model >>> and stream of data. To process the data, you will need a way to access your >>> model from the subsequent stream operations (map, filter, flatmap, ..). >>> > I'm not sure in which case Operator State is a good choice, but I >>> think you can also live without. >>> > >>> > val modelStream = .... // get the model stream >>> > val dataStream = >>> > >>> > modelStream.broadcast.connect(dataStream). coFlatMap( ) Then you can >>> keep the latest model in a CoFlatMapRichFunction, not necessarily as >>> Operator State, although maybe OperatorState is a good choice too. >>> > >>> > Does it make sense to you ? >>> > >>> > Anwar >>> > >>> > On Fri, Nov 6, 2015 at 10:21 AM, Welly Tambunan <if05...@gmail.com> >>> wrote: >>> > Hi All, >>> > >>> > We have a high density data that required a downsample. However this >>> downsample model is very flexible based on the client device and user >>> interaction. So it will be wasteful to precompute and store to db. >>> > >>> > So we want to use Apache Flink to do downsampling and cache the result >>> for subsequent query. >>> > >>> > We are considering using Flink Operator state for that one. >>> > >>> > Is that the right approach to use that for memory cache ? Or if that >>> preferable using memory cache like redis etc. >>> > >>> > Any comments will be appreciated. >>> > >>> > >>> > Cheers >>> > -- >>> > Welly Tambunan >>> > Triplelands >>> > >>> > http://weltam.wordpress.com >>> > http://www.triplelands.com >>> > >>> > >>> > >>> > >>> > -- >>> > Welly Tambunan >>> > Triplelands >>> > >>> > http://weltam.wordpress.com >>> > http://www.triplelands.com >>> > >>> > >>> > >>> > >>> > -- >>> > Welly Tambunan >>> > Triplelands >>> > >>> > http://weltam.wordpress.com >>> > http://www.triplelands.com >>> >>> >> >> >> -- >> Welly Tambunan >> Triplelands >> >> http://weltam.wordpress.com >> http://www.triplelands.com <http://www.triplelands.com/blog/> >> > > -- Welly Tambunan Triplelands http://weltam.wordpress.com http://www.triplelands.com <http://www.triplelands.com/blog/>
