becketqin commented on code in PR #23164:
URL: https://github.com/apache/flink/pull/23164#discussion_r1296581609
##########
flink-yarn/src/main/java/org/apache/flink/yarn/YarnApplicationFileUploader.java:
##########
@@ -360,11 +362,27 @@ List<String> registerProvidedLocalResources() {
envShipResourceList.add(descriptor);
if (!isFlinkDistJar(filePath.getName()) &&
!isPlugin(filePath)) {
- classPaths.add(fileName);
+ URI parentDirectoryUri = new
Path(fileName).getParent().toUri();
+ String relativeParentDirectory =
+ new Path(filePath.getName())
+ .toUri()
+ .relativize(parentDirectoryUri)
+ .toString();
+
+ if (!resourcesDir.contains(relativeParentDirectory)) {
+ resourcesDir.add(relativeParentDirectory);
+ }
+ resources.add(fileName);
Review Comment:
The main concern is the length of class path. I think there is a limit of
32K characters. For one of my simple local job, the class path is already over
10K chars. If we are including all the parent dir into the class path. This
might easily go beyond the limit. Also, in general, adding the resource dir to
class path in a more explicit and accurate way should probably be preferred.
> Also introducing a new resources dir and only supporting that feels
limiting. Ideally, users or platforms would like to logically group resources
in to their own dirs. For eg: hadoop, hive libs etc in to their own dirs.
Maybe we can have a `yarn.provided.resources.dirs` in this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
