[ https://issues.apache.org/jira/browse/SPARK-10858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14936317#comment-14936317 ]
Saisai Shao edited comment on SPARK-10858 at 9/30/15 5:49 AM: -------------------------------------------------------------- So basically I think the problem is if we specify the scheme, do we need to guarantee the preserved chars to be escaped by our own. If we rely on Spark to handle this, I think we need to fix this issue. Another interesting thing is that not sure why your result is different from mine. was (Author: jerryshao): So basically I think the problem is do we need to treat this name "xx#xx" as a legal name, if so we need to fix this behavior. Another interesting thing is that not sure why your result is different from mine. > YARN: archives/jar/files rename with # doesn't work unless scheme given > ----------------------------------------------------------------------- > > Key: SPARK-10858 > URL: https://issues.apache.org/jira/browse/SPARK-10858 > Project: Spark > Issue Type: Bug > Components: YARN > Affects Versions: 1.5.1 > Reporter: Thomas Graves > Priority: Minor > > The YARN distributed cache feature with --jars, --archives, --files where you > can rename the file/archive using a # symbol only works if you explicitly > include the scheme in the path: > works: > --jars file:///home/foo/my.jar#renamed.jar > doesn't work: > --jars /home/foo/my.jar#renamed.jar > Exception in thread "main" java.io.FileNotFoundException: File > file:/home/foo/my.jar#renamed.jar does not exist > at > org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:534) > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:747) > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:524) > at > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:416) > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337) > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289) > at > org.apache.spark.deploy.yarn.Client.copyFileToRemote(Client.scala:240) > at > org.apache.spark.deploy.yarn.Client.org$apache$spark$deploy$yarn$Client$$distribute$1(Client.scala:329) > at > org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$6$$anonfun$apply$2.apply(Client.scala:393) > at > org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$6$$anonfun$apply$2.apply(Client.scala:392) > at > scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) > at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108) -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org