Re: State in external db (dynamodb)

t; conveniently has TTL, instead of DynamoDB. >>>>> >>>>> I just wanted to check whether eviction of (inactive/quiet) state from >>>>> memory is something that I should consider implementing, or whether Flink >>>>> already had some built-in

Re: State in external db (dynamodb)

For >>>> example, if you are only executing a sum on the contents of the window, the >>>> window state doesn't need to store all the individual items in the window, >>>> it only needs to store the sum. Aggregations other than "sum" might have >&

Re: State in external db (dynamodb)

gt; window state doesn't need to store all the individual items in the window, >>> it only needs to store the sum. Aggregations other than "sum" might have >>> that characteristic too. I don't know if Flink is already that intelligent >>> or whether I should fig

Re: State in external db (dynamodb)

gt;>> characteristic too. I don't know if Flink is already that intelligent or >>> whether I should figure out how to aggregate window contents myself when >>> possible with something like a window fold? Another poster (Aljoscha) was >>> talking about adding in

Re: State in external db (dynamodb)

ontents myself when >> possible with something like a window fold? Another poster (Aljoscha) was >> talking about adding incremental snapshots, but it sounds like that would >> only improve the write throughput not the memory usage. >> >> Thanks again! >> Shannon C

Re: State in external db (dynamodb)

n Carey > > > From: Stephan Ewen <se...@apache.org> > Date: Wednesday, April 6, 2016 at 10:37 PM > To: <user@flink.apache.org> > Subject: Re: State in external db (dynamodb) > > Hi Shannon! > > Welcome to the Flink community! > > You are right, sinks need in

Re: State in external db (dynamodb)

mory usage. Thanks again! Shannon Carey From: Stephan Ewen <se...@apache.org<mailto:se...@apache.org>> Date: Wednesday, April 6, 2016 at 10:37 PM To: <user@flink.apache.org<mailto:user@flink.apache.org>> Subject: Re: State in external db (dynamodb) Hi Shannon! Welcome to t

Re: State in external db (dynamodb)

FYI Cassandra has a TTL on data: https://docs.datastax.com/en/cql/3.1/cql/cql_using/use_expire_t.html On Wed, Apr 6, 2016 at 7:55 AM, Shannon Carey wrote: > Hi, new Flink user here! > > I found a discussion on user@flink.apache.org about using DynamoDB as a > sink. However,