Bodong Liu created FLINK-33569: ---------------------------------- Summary: Could not deploy yarn-application when using yarn over s3a filesystem. Key: FLINK-33569 URL: https://issues.apache.org/jira/browse/FLINK-33569 Project: Flink Issue Type: Bug Components: Deployment / YARN Affects Versions: 1.17.1, 1.18.0 Environment: h1. *Env:* * OS: ArchLinux kernel:{color:#000000}6.6.1 AMD64{color} * Flink: 1.17.1 * Hadoop: 3.3.6 * Minio: 2023-11-15
h1. Settings h2. hadoop core-site.xml: {code:java} <property> <name>fs.defaultFS</name> <value>s3a://hadoop</value> </property> <property> <name>fs.s3a.path.style.access</name> <value>true</value> </property> <!-- minio username --> <property> <name>fs.s3a.access.key</name> <value>admin</value> </property> <!-- minio password --> <property> <name>fs.s3a.secret.key</name> <value>password</value> </property> <!-- minio endpoint --> <property> <name>fs.s3a.endpoint</name> <value>http://localhost:9000</value> </property> <property> <name>fs.s3a.connection.establish.timeout</name> <value>5000</value> </property> <property> <name>fs.s3a.multipart.size</name> <value>512M</value> </property> <property> <name>fs.s3a.impl</name> <value>org.apache.hadoop.fs.s3a.S3AFileSystem</value> </property> <property> <name>fs.AbstractFileSystem.s3a.impl</name> <value>org.apache.hadoop.fs.s3a.S3A</value> </property> <!-- S3 end -->{code} h1. Flink run command: ./bin/flink run-application -t yarn-application ./examples/streaming/TopSpeedWindowing.jar Reporter: Bodong Liu Attachments: 2023-11-16_16-47.png, image-2023-11-16-16-46-21-684.png, image-2023-11-16-16-48-40-223.png I now use the `yarn-application` mode to deploy Flink. I found that when I set Hadoop's storage to the s3a file system, Flink could not submit tasks to Yarn. The error is reported as follows: {code:java} ------------------------------------------------------------ The program finished with the following exception: org.apache.flink.client.deployment.ClusterDeploymentException: Couldn't deploy Yarn Application Cluster at org.apache.flink.yarn.YarnClusterDescriptor.deployApplicationCluster(YarnClusterDescriptor.java:481) at org.apache.flink.client.deployment.application.cli.ApplicationClusterDeployer.run(ApplicationClusterDeployer.java:67) at org.apache.flink.client.cli.CliFrontend.runApplication(CliFrontend.java:212) at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1098) at org.apache.flink.client.cli.CliFrontend.lambda$mainInternal$9(CliFrontend.java:1189) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899) at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) at org.apache.flink.client.cli.CliFrontend.mainInternal(CliFrontend.java:1189) at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1157) Caused by: org.apache.hadoop.fs.PathIOException: `Cannot get relative path for URI:file:///tmp/application_1700122774429_0001-flink-conf.yaml5526160496134930395.tmp': Input/output error at org.apache.hadoop.fs.s3a.impl.CopyFromLocalOperation.getFinalPath(CopyFromLocalOperation.java:360) at org.apache.hadoop.fs.s3a.impl.CopyFromLocalOperation.uploadSourceFromFS(CopyFromLocalOperation.java:222) at org.apache.hadoop.fs.s3a.impl.CopyFromLocalOperation.execute(CopyFromLocalOperation.java:169) at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$copyFromLocalFile$26(S3AFileSystem.java:3854) at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547) at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528) at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:449) at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2480) at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2499) at org.apache.hadoop.fs.s3a.S3AFileSystem.copyFromLocalFile(S3AFileSystem.java:3847) at org.apache.flink.yarn.YarnApplicationFileUploader.copyToRemoteApplicationDir(YarnApplicationFileUploader.java:397) at org.apache.flink.yarn.YarnApplicationFileUploader.uploadLocalFileToRemote(YarnApplicationFileUploader.java:202) at org.apache.flink.yarn.YarnApplicationFileUploader.registerSingleLocalResource(YarnApplicationFileUploader.java:181) at org.apache.flink.yarn.YarnClusterDescriptor.startAppMaster(YarnClusterDescriptor.java:1050) at org.apache.flink.yarn.YarnClusterDescriptor.deployInternal(YarnClusterDescriptor.java:626) at org.apache.flink.yarn.YarnClusterDescriptor.deployApplicationCluster(YarnClusterDescriptor.java:474) ... 10 more {code} I found by looking through the source code and debugging that when Hadoop uses the s3a file system, uploading and downloading files must use URIs with `scheme` to build path parameters. In the `org.apache.flink.yarn.YarnClusterDescriptor` class, when uploading a temporarily generated `yaml` configuration file, the absolute path of the file is used instead of the URI as the path construction parameter, but other file upload and download behaviors They all use URI as the path parameter. This is the reason for the error reported above. -- This message was sent by Atlassian Jira (v8.20.10#820010)