[
https://issues.apache.org/jira/browse/FLINK-37608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hangxiang Yu updated FLINK-37608:
---------------------------------
Description:
File Sink closes buckets sequentially when snapshot currently, we found it may
block snapshot if there are some slow nodes for HDFS.
we'd like to introduce an option to close buckets in parallel when snapshot,
which could reduce 95% time cost for checkpointing in our environment.
was:
File Sink closes buckets sequentially when snapshot currently, we found it may
block snapshot if there are some slow nodes for HDFS.
we'd like to introduce an option to close buckets in parallel when snapshot,
which could reduce 95% time cost for checkpointing in our environment.
> File sink supports to close buckets in parallel when snapshot
> -------------------------------------------------------------
>
> Key: FLINK-37608
> URL: https://issues.apache.org/jira/browse/FLINK-37608
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / FileSystem
> Reporter: Hangxiang Yu
> Assignee: Hangxiang Yu
> Priority: Major
> Fix For: 2.1.0
>
> Attachments: After.png, Before.png
>
>
> File Sink closes buckets sequentially when snapshot currently, we found it
> may block snapshot if there are some slow nodes for HDFS.
> we'd like to introduce an option to close buckets in parallel when snapshot,
> which could reduce 95% time cost for checkpointing in our environment.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
