[
https://issues.apache.org/jira/browse/FLINK-35232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xintong Song closed FLINK-35232.
--------------------------------
Fix Version/s: 1.20.0
Resolution: Done
master (1.20): f2c0c3ddcdd78c1e2876087139e56534fe3f8421
> Support for retry settings on GCS connector
> -------------------------------------------
>
> Key: FLINK-35232
> URL: https://issues.apache.org/jira/browse/FLINK-35232
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / FileSystem
> Reporter: Vikas M
> Assignee: Oleksandr Nitavskyi
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.20.0
>
>
> https://issues.apache.org/jira/browse/FLINK-32877 is tracking ability to
> specify transport options in GCS connector. While setting the params enabled
> here reduced read timeouts, we still see 503 errors leading to Flink job
> restarts.
> Thus, in this ticket, we want to specify additional retry settings as noted
> in
> [https://cloud.google.com/storage/docs/retry-strategy#customize-retries.|https://cloud.google.com/storage/docs/retry-strategy#customize-retries]
> We need
> [these|https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings#methods]
> methods available for Flink users so that they can customize their
> deployment. In particular next settings seems to be the minimum required to
> adjust GCS timeout with Job's checkpoint config:
> *
> [maxAttempts|https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings#com_google_api_gax_retrying_RetrySettings_getMaxAttempts__]
> *
> [initialRpcTimeout|https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings#com_google_api_gax_retrying_RetrySettings_getInitialRpcTimeout__]
> *
> [rpcTimeoutMultiplier|https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings#com_google_api_gax_retrying_RetrySettings_getRpcTimeoutMultiplier__]
> *
> [maxRpcTimeout|https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings#com_google_api_gax_retrying_RetrySettings_getMaxRpcTimeout__]
> *
> [totalTimeout|https://cloud.google.com/java/docs/reference/gax/latest/com.google.api.gax.retrying.RetrySettings#com_google_api_gax_retrying_RetrySettings_getTotalTimeout__]
>
> Basically the proposal is to be able to tune the timeout via multiplier,
> maxAttemts + totalTimeout mechanisms.
> All of the config options should be optional and the default one should be
> used in case some of configs are not provided.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
