GitHub user Chaos-Ju opened a pull request: https://github.com/apache/spark/pull/19206
Client and ApplicationMaster resolvePath is inappropriate when use viewfs ## What changes were proposed in this pull request? When HDFS use viewfs and spark construct Executor's and ApplicationMaster's localResource Map ( the list of localized files ) ,can't covert viewfs:// path to the real hdfs:// path . Therefore , when NodeManager download the local Resource, will throw java.io.IOException: ViewFs: Cannot initialize: Empty Mount table in config for viewfs://clusterName/ Exception stackï¼ java.io.IOException: ViewFs: Cannot initialize: Empty Mount table in config for viewfs://ns-view/ at org.apache.hadoop.fs.viewfs.InodeTree.<init>(InodeTree.java:337) at org.apache.hadoop.fs.viewfs.ViewFileSystem$1.<init>(ViewFileSystem.java:167) at org.apache.hadoop.fs.viewfs.ViewFileSystem.initialize(ViewFileSystem.java:167) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:251) at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:63) at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:361) at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1700) at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:358) at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Failing this attempt. Failing the application ## How was this patch tested? manual tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/Chaos-Ju/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/19206.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #19206 ---- commit f1fff009d32b8f7d1d2b24734e4d677c6264ec90 Author: Chaos-Ju <chao...@163.com> Date: 2017-09-12T12:45:36Z fix spark support viewfs ---- --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org