yanyan300300 commented on a change in pull request #8215: [FLINK-8801][yarn/s3] Fix jars downloading issues due to inconsistent timestamp in S3 Filesystem URL: https://github.com/apache/flink/pull/8215#discussion_r279876022
########## File path: flink-yarn/src/main/java/org/apache/flink/yarn/Utils.java ########## @@ -160,8 +170,37 @@ public static void setupYarnClassPath(Configuration conf, Map<String, String> ap fs.copyFromLocalFile(false, true, localSrcPath, dst); + // Note: If we directly used registerLocalResource(FileSystem, Path) here, we would access the remote + // file once again which has problems with eventually consistent read-after-write file + // systems. Instead, we decide to wait until the remote file be available. + + FileStatus[] fss = null; + int iter = 1; + while (iter <= REMOTE_RESOURCES_FETCH_NUM_RETRY) { Review comment: And could you clarify by "did in YarnFileStageTest#testCopyFromLocalRecursive()"? Are you suggesting testing the retry on `FileNotFoundException`? I think currently Flink is using a real S3 filesystem instead of a mock, so I am not quite sure how to do it. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services