[
https://issues.apache.org/jira/browse/FLINK-21694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17317256#comment-17317256
]
Stephan Ewen commented on FLINK-21694:
--------------------------------------
[~qinjunjerry] Can you create a separate ticket for the
{{state.backend.rocksdb.thread.num}} increasing?
[~yunta] or [~liyu] - it seems there is consensus that this value should be
increased.
I would still go for a value of 8 to give good out-of-the-box experience. Few
people will find this configuration, they will just assume restores are slower
than they need to be.
We cannot guarantee anyways that Flink wouldn't put a lot of downloads onto the
DFS, because of parallel operators. So we need to document anyways how to
reduce load, if there is a problem. On top of that, many deployments run in the
cloud, where the Object Store anyways has quotas and throttling for downloads,
if a single account put too much loads on the storage. The "put too much strain
on DFS" seems mostly relevant for self-managed HDFS setups, is that right?
> Increase default value of
> "state.backend.rocksdb.checkpoint.transfer.thread.num"
> --------------------------------------------------------------------------------
>
> Key: FLINK-21694
> URL: https://issues.apache.org/jira/browse/FLINK-21694
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
> Reporter: Stephan Ewen
> Priority: Critical
> Fix For: 1.13.0
>
>
> The default value for the number of threads used to download state artifacts
> from checkpoint storage should be increased.
> The increase should not pose risk of regression, but does in many cases speed
> up checkpoint recovery significantly.
> Something similar was reported in this blog post, item (3).
> https://engineering.contentsquare.com/2021/ten-flink-gotchas/
> A default value of 8 (eight) sounds like a good default. It should not result
> in excessive thread explosion, and already speeds up recovery.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
