[
https://issues.apache.org/jira/browse/FLINK-11696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski closed FLINK-11696.
----------------------------------
Fix Version/s: 1.10.0
Resolution: Fixed
merged commit c5e5f0a into apache:master
> Avoid to send mkdir requests to DFS from task side
> --------------------------------------------------
>
> Key: FLINK-11696
> URL: https://issues.apache.org/jira/browse/FLINK-11696
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing
> Reporter: Yun Tang
> Assignee: Yun Tang
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Currently, when we create checkpoint directory in the distributed file
> system. Not only {{CheckpointCoordinator}} but also {{FsCheckpointStorage}}
> in {{StreamTask}} would create the {{checkpointsDirectory}},
> {{sharedStateDirectory}} and {{taskOwnedStateDirectory}}. These many
> {{mkdir}} RPC requests would cause a very high pressure on the distributed
> file system, especially when the parallelism is large or jobs continue to
> failover again and again.
> We could avoid these {{mkdir}} requests from task side if writing to a
> distributed file system.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
