[ https://issues.apache.org/jira/browse/SPARK-21917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16154097#comment-16154097 ]
Marcelo Vanzin commented on SPARK-21917: ---------------------------------------- I think the optimal way would be to try to add things to the cache and, if that fails, fallback to download + re-upload. The sucky part is that there's no reliable way to do it. The libraries available on the client side may support different file systems than the libraries available on the NM; so if you have the http fs in your classpath, but the NM does not, the container localizer would probably fail. #1 works but it also penalizes those who are running YARN 2.9 or any other future version where that support exists. So perhaps a compromise could be: - by default, assume that client and NM libraries are "in sync"; if {{FileSystem.get()}} does not complain, assume the NM can also download files from that scheme. If it does, then download the file and re-upload it to HDFS. - add a config option where users can blacklist schemes and force those to download and re-upload. > Remote http(s) resources is not supported in YARN mode > ------------------------------------------------------ > > Key: SPARK-21917 > URL: https://issues.apache.org/jira/browse/SPARK-21917 > Project: Spark > Issue Type: Bug > Components: Spark Submit, YARN > Affects Versions: 2.2.0 > Reporter: Saisai Shao > Priority: Minor > > In the current Spark, when submitting application on YARN with remote > resources {{./bin/spark-shell --jars > http://central.maven.org/maven2/com/github/swagger-akka-http/swagger-akka-http_2.11/0.10.1/swagger-akka-http_2.11-0.10.1.jar > --master yarn-client -v}}, Spark will be failed with: > {noformat} > java.io.IOException: No FileSystem for scheme: http > at > org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2586) > at > org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2593) > at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91) > at > org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2632) > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2614) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370) > at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296) > at > org.apache.spark.deploy.yarn.Client.copyFileToRemote(Client.scala:354) > at > org.apache.spark.deploy.yarn.Client.org$apache$spark$deploy$yarn$Client$$distribute$1(Client.scala:478) > at > org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$11$$anonfun$apply$6.apply(Client.scala:600) > at > org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$11$$anonfun$apply$6.apply(Client.scala:599) > at scala.collection.mutable.ArraySeq.foreach(ArraySeq.scala:74) > at > org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$11.apply(Client.scala:599) > at > org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$11.apply(Client.scala:598) > at scala.collection.immutable.List.foreach(List.scala:381) > at > org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:598) > at > org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:848) > at > org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:173) > {noformat} > This is because {{YARN#client}} assumes resources must be on the Hadoop > compatible FS, also in the NM > (https://github.com/apache/hadoop/blob/99e558b13ba4d5832aea97374e1d07b4e78e5e39/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java#L245) > it will only use Hadoop compatible FS to download resources. So this makes > Spark on YARN fail to support remote http(s) resources. > To solve this problem, there might be several options: > * Download remote http(s) resources to local and add this local downloaded > resources to dist cache. The downside of this option is that remote resources > will be uploaded again unnecessarily. > * Filter remote http(s) resources and add them with spark.jars or > spark.files, to leverage Spark's internal fileserver to distribute remote > http(s) resources. The problem of this solution is: for some resources which > require to be available before application start may not work. > * Leverage Hadoop's support http(s) file system > (https://issues.apache.org/jira/browse/HADOOP-14383). This is only worked in > Hadoop 2.9+, and I think even we implement a similar one in Spark will not be > worked. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org