yuruguo commented on a change in pull request #14538:
URL: https://github.com/apache/flink/pull/14538#discussion_r552508506
##########
File path:
flink-yarn/src/main/java/org/apache/flink/yarn/YarnApplicationFileUploader.java
##########
@@ -163,8 +164,15 @@ YarnLocalResourceDescriptor registerSingleLocalResource(
addToRemotePaths(whetherToAddToRemotePaths, resourcePath);
if (Utils.isRemotePath(resourcePath.toString())) {
- final FileStatus fileStatus =
fileSystem.getFileStatus(resourcePath);
- LOG.debug("Using remote file {} to register local resource",
fileStatus.getPath());
+ final FileSystem srcFs =
resourcePath.getFileSystem(fileSystem.getConf());
+ final FileStatus resourceFileStatus =
srcFs.getFileStatus(resourcePath);
+ LOG.debug("Using remote file {} to register local resource",
resourceFileStatus.getPath());
+ final Path applicationDir = getApplicationDirPath(homeDir,
applicationId);
+ final String suffix = (relativeDstPath.isEmpty() ? "" :
relativeDstPath + "/") + resourcePath.getName();
+ final Path dst = new Path(applicationDir, suffix);
+ LOG.debug("Copying from {} to {} with replication factor {}",
resourcePath, dst, fileReplication);
+ FileUtil.copy(srcFs, resourcePath, fileSystem, dst, false,
fileSystem.getConf());
+ final FileStatus fileStatus = fileSystem.getFileStatus(dst);
Review comment:
It’s possible without `isLocal `, but in this case, it will be repeated
in multiple places to judge whether the file is local or remote with
`Utils.isRemotePath(resourcePath.toString())`. In fact, we only need to judge
once and pass this result.
Whether it is a remote file is based on method `Utils.isRemotePath(String
path)`，false(w/o schema or schema is "file://") as local file.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
