[
https://issues.apache.org/jira/browse/FLINK-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15856972#comment-15856972
]
Chen Qin commented on FLINK-4266:
---------------------------------
When this jira was filed, there are multiple issues around incremental
checkpointing support as well as scaling non partitioned states of long running
jobs.
With completion of dynamic scaling of non partitioned states FLINK-4379,
scaling large states dynamically no longer becomes a blocker. Flink is also
working on a incremental checkpointing design which likely address large states
checkpointing performance
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/States-split-over-to-external-storage-td15344.html#none
In light of all these efforts, I would like to limit effort of this jira to
serve as a split over of rocksdb statebackend. updated states will be batched
writes to remote db during checkpointing phase. When jobs running in multiple
data centers in parallel, remote split over would enable source fail over
without loss states nor manual file movements which introduce latency & errors
> Remote Database Statebackend
> ----------------------------
>
> Key: FLINK-4266
> URL: https://issues.apache.org/jira/browse/FLINK-4266
> Project: Flink
> Issue Type: New Feature
> Components: State Backends, Checkpointing
> Affects Versions: 1.3.0
> Reporter: Chen Qin
> Assignee: Chen Qin
> Priority: Minor
>
> Current FileSystem statebackend limits whole state size to disk space.
> Dealing with scale out checkpoint states beyond local disk space such as long
> running task that hold window content for long period of time. Pipelines
> needs to split out states to durable remote storage even replicated to
> different data centers.
> We draft a design that leverage checkpoint id as mono incremental logic
> timestamp and perform range query to get evicited state k/v. we also
> introduce checkpoint time commit and eviction threshold that reduce "hot
> states" hitting remote db per every update between adjacent checkpoints by
> tracking update logs and merge them, do batch updates only when checkpoint;
> lastly, we are looking for eviction policy that can identify "hot keys" in
> k/v state and lazy load those "cold keys" from remote storage(e.g Cassandra).
> For now, we don't have good story regarding to data retirement. We might have
> to keep forever until manually run command and clean per job related state
> data. Some of features might related to incremental checkpointing feature, we
> hope to align with effort there also.
> Welcome comments, I will try to put a draft design doc after gathering some
> feedback.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
