FYI Cassandra has a TTL on data: https://docs.datastax.com/en/cql/3.1/cql/cql_using/use_expire_t.html
On Wed, Apr 6, 2016 at 7:55 AM, Shannon Carey <sca...@expedia.com> wrote: > Hi, new Flink user here! > > I found a discussion on user@flink.apache.org about using DynamoDB as a > sink. However, as noted, sinks have an at-least-once guarantee so your > operations must idempotent. > > However, another way to go about this (and correct me if I'm wrong) is to > write the state to the external store via a custom State Backend. Since the > state participates in checkpointing, you don't have to worry about > idempotency: every time state is checkpointed, overwrite the value of that > key. > > We are starting a project with Flink, and we are interested in evicting > the state from memory once a TTL is reached during which no events have > come in for that state. Subsequently, when an event is processed, we must > be able to quickly load up any evicted state. Does this sound reasonable? > We are considering using DynamoDB for our state backend because it seems > like all we will need is a key-value store. The only weakness of this is > that if state gets older than, say, 2 years we would like to get rid of it > which might not be easy in DynamoDB. I don't suppose Flink has any > behind-the-scenes features that deal with getting rid of old state (either > evicting from memory or TTL/aging out entirely)? > > Thanks for your time! > Shannon Carey >
