Re: Flink File Source: File read strategy

2023-09-24 Thread Shammon FY
hat File Source reads files directly from S3? > > > > Regards, > > Kirti Dhar > > > > *From:* Shammon FY > *Sent:* 25 September 2023 06:27 > *To:* Kirti Dhar Upadhyay K > *Cc:* user@flink.apache.org > *Subject:* Re: Flink File Source: File read strategy &

RE: Flink File Source: File read strategy

2023-09-24 Thread Kirti Dhar Upadhyay K via user
Thanks Shammon. Is there any way to verify that File Source reads files directly from S3? Regards, Kirti Dhar From: Shammon FY Sent: 25 September 2023 06:27 To: Kirti Dhar Upadhyay K Cc: user@flink.apache.org Subject: Re: Flink File Source: File read strategy Hi Kirti, I think the default

Re: Flink File Source: File read strategy

2023-09-24 Thread Shammon FY
Hi Kirti, I think the default file `Source` does not download files locally in Flink, but reads them directly from S3. However, Flink also supports configuring temporary directories through `io.tmp.dirs`. If it is a user-defined source, it can be obtained from FlinkS3FileSystem. After the Flink jo