[ 
https://issues.apache.org/jira/browse/FLINK-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15824927#comment-15824927
 ] 

Syinchwun Leo commented on FLINK-4266:
--------------------------------------

We also meet this problem,and plan to using incremental method. In currently 
master branch, there are four window state: ValueState, ListState, ReduingState 
and FoldingState. In ReducingState and FoldingState, window only keeps 
intermediate results calculated by UDF reducing function or folding function. 
When a element coming, the related intermediate result is recomputed again by 
applying the UDF function. In my opinion it is costly to record all the changes 
in realtime, what's more, we have to scan the pipeline reversely for the 
purpose of recovery . In current sliding window's saving method, there is a big 
drawback: an element may be kept in multiple windows, when checkpointing, all 
these backups are export to persistent stores, it's time and storage consuming. 
we have modify the saving method of sliding window (including 
SlidingProcessingTimeWindow and SlidingEventTimeWIndow), and it is testing. You 
can keep in touch with me(liu...@mail.ustc.edu.cn), and we can develop this 
feature in cooperation.

> Remote Database Statebackend
> ----------------------------
>
>                 Key: FLINK-4266
>                 URL: https://issues.apache.org/jira/browse/FLINK-4266
>             Project: Flink
>          Issue Type: New Feature
>          Components: State Backends, Checkpointing
>    Affects Versions: 1.2.0
>            Reporter: Chen Qin
>            Priority: Minor
>
> Current FileSystem statebackend limits whole state size to disk space. 
> Dealing with scale out checkpoint states beyond local disk space such as long 
> running task that hold window content for long period of time. Pipelines 
> needs to split out states to durable remote storage even replicated to 
> different data centers.
> We draft a design that leverage checkpoint id as mono incremental logic 
> timestamp and perform range query to get evicited state k/v. we also 
> introduce checkpoint time commit and eviction threshold that reduce "hot 
> states" hitting remote db per every update between adjacent checkpoints by 
> tracking update logs and merge them, do batch updates only when checkpoint; 
> lastly, we are looking for eviction policy that can identify "hot keys" in 
> k/v state and lazy load those "cold keys" from remote storage(e.g Cassandra).
> For now, we don't have good story regarding to data retirement. We might have 
> to keep forever until manually run command and clean per job related state 
> data. Some of features might related to incremental checkpointing feature, we 
> hope to align with effort there also.
> Welcome comments, I will try to put a draft design doc after gathering some 
> feedback.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to