[
https://issues.apache.org/jira/browse/FLINK-34975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17832917#comment-17832917
]
Kenneth William Krugler commented on FLINK-34975:
-------------------------------------------------
I'd previously explored using [AnyBlob|https://github.com/durner/AnyBlob] for
maintaining state in S3, though I stopped when I ran into C++/JNI build issues
on my Mac (M1 processor-related). AnyBlob is C++, so not sure it would be worth
the effort, but I think they use some standard remote storage optimization
techniques that could be interesting.
> FLIP-427: ForSt - Disaggregated state Store
> -------------------------------------------
>
> Key: FLINK-34975
> URL: https://issues.apache.org/jira/browse/FLINK-34975
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / State Backends
> Reporter: Hangxiang Yu
> Assignee: Hangxiang Yu
> Priority: Major
> Fix For: 2.0.0
>
>
> This is a sub-FLIP for the disaggregated state management and its related
> work, please read the [FLIP-423|https://cwiki.apache.org/confluence/x/R4p3EQ]
> first to know the whole story.
> As described in FLIP-423, there are some tough issues about embedded state
> backend on local file system, respecially when dealing with extremely large
> state:
> # {*}Constraints of local disk space complicate the prediction of storage
> requirements, potentially leading to job failures{*}: Especially in cloud
> native deployment mode, pre-allocated local disks typically face strict
> capacity constraints, making it challenging to forecast the size requirements
> of job states. Over-provisioning disk space results in unnecessary resource
> overhead, while under-provisioning risks job failure due to insufficient
> space.
> # *The tight coupling of compute and storage resources leads to
> underutilization and increased waste:* Jobs can generally be categorized as
> either CPU-intensive or IO-intensive. In a coupled architecture,
> CPU-intensive jobs leave a significant portion of storage resources
> underutilized, whereas IO-intensive jobs result in idle computing resources.
> By considering remote storage as the primary storage, all working states are
> maintained on the remote file system, which brings several advantages:
> # *Remote storages e.g. S3/HDFS typically offer elastic scalability,
> theoretically providing unlimited space.*
> # *The allocation of remote storage resources can be optimized by reducing
> them for CPU-intensive jobs and augmenting them for IO-intensive jobs, thus
> enhancing overall resource utilization.*
> # *This architecture facilitates a highly efficient and lightweight process
> for checkpointing, recovery, and rescaling through fast copy or simple move.*
> This FLIP aims to realize disaggregated state for our new key-value store
> named *ForSt* which evloves from RocksDB and supports remote file system.
> This makes Flink get rid of the disadvantages by coupled state architecture
> and embrace the scalable as well as flexible cloud-native storage.
> Please see [FLIP-427 |https://cwiki.apache.org/confluence/x/T4p3EQ]for more
> details.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
