Re: GCS FileSink Read Timeouts

2024-04-16 Thread Dylan Fontana via user
Thanks for the links! We've tried the `gs.writer.chunk.size` before and found it didn't make a meaningful difference unfortunately. The hadoop-connector link you've sent I think is actually not applicable since the gcs Filesystem connector isn't using the hadoop implementation but instead the

Re: GCS FileSink Read Timeouts

2024-04-02 Thread Asimansu Bera
Hello Dylan, I'm not an expert. There are many configuration settings(tuning) which could be setup via flink configuration. Pls refer to the second link below - specifically retry options. https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/deployment/filesystems/gcs/

GCS FileSink Read Timeouts

2024-04-02 Thread Dylan Fontana via user
Hey Flink Users, We've been facing an issue with GCS that I'm keen to hear the community's thoughts or insights on. We're using the GCS FileSystem on a FileSink to write parquets in our Flink app. We're finding sporadic instances of `com.google.cloud.storage.StorageException: Read timed out`