Re: Flink application mode/S3A Exception after upgrading from to Flink 1.16.0

2023-01-27 Thread Leon Xu
Thank you Gabor. I will test with a downgraded version to see how it goes. On Fri, Jan 27, 2023 at 11:51 AM Gabor Somogyi wrote: > The min supported version was 2.8.5 but in 1.17 it's gonna be 2.10.1 so > one can downgrade. > > G > > On Fri, Jan 27, 2023, 20:42 Leon Xu wrote: > >> Thank you

Re: Flink application mode/S3A Exception after upgrading from to Flink 1.16.0

2023-01-27 Thread Gabor Somogyi
The min supported version was 2.8.5 but in 1.17 it's gonna be 2.10.1 so one can downgrade. G On Fri, Jan 27, 2023, 20:42 Leon Xu wrote: > Thank you Mate. > Yeah this looks like the root cause. A follow-up question, do you know if > Flink 1.16 will have a hard dependency on Hadoop 3.3.x? or can

Re: Flink application mode/S3A Exception after upgrading from to Flink 1.16.0

2023-01-27 Thread Leon Xu
Thank you Mate. Yeah this looks like the root cause. A follow-up question, do you know if Flink 1.16 will have a hard dependency on Hadoop 3.3.x? or can we actually use a older version of hadoop to work around this issue. Thanks Leon On Fri, Jan 27, 2023 at 10:28 AM Máté Czagány wrote: > Hi

Re: Flink application mode/S3A Exception after upgrading from to Flink 1.16.0

2023-01-27 Thread Máté Czagány
Hi Leon, It seems to me like S3 cannot be used as YARN resource storage starting with Hadoop 3.3.2 In HADOOP-17139 S3AFileSystem#copyFromLocalFile was refactored and expects the local source Hadoop Path object to have a scheme specified which the YarnClusterDescriptor uploading the local files

Flink application mode/S3A Exception after upgrading from to Flink 1.16.0

2023-01-26 Thread Leon Xu
Hi Flink Users, We are trying to upgrade Flink from 1.12.7 to 1.16.0. But we run into the following issue: We are running Flink job through application mode. After the upgrade, when we submit the job and now it gets this exception: *org.apache.flink.client.deployment.ClusterDeploymentException: