[
https://issues.apache.org/jira/browse/SPARK-23123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16328364#comment-16328364
]
Saisai Shao commented on SPARK-23123:
-------------------------------------
[~ste...@apache.org], any thought on this issue?
> Unable to run Spark Job with Hadoop NameNode Federation using ViewFS
> --------------------------------------------------------------------
>
> Key: SPARK-23123
> URL: https://issues.apache.org/jira/browse/SPARK-23123
> Project: Spark
> Issue Type: Bug
> Components: Spark Submit
> Affects Versions: 1.6.3
> Reporter: Nihar Nayak
> Priority: Major
> Labels: Hadoop, Spark
>
> Added following to core-site.xml in order to make use of ViewFS in a NameNode
> federated cluster.
> {noformat}
> <property>
> <name>fs.defaultFS</name>
> <value>viewfs:///</value>
> </property>
> <property>
> <name>fs.viewfs.mounttable.default.link./apps</name>
> <value>hdfs://nameservice1/apps</value>
> </property>
> <property>
> <name>fs.viewfs.mounttable.default.link./app-logs</name>
> <value>hdfs://nameservice2/app-logs</value>
> </property>
> <property>
> <name>fs.viewfs.mounttable.default.link./tmp</name>
> <value>hdfs://nameservice2/tmp</value>
> </property>
> <property>
> <name>fs.viewfs.mounttable.default.link./user</name>
> <value>hdfs://nameservice2/user</value>
> </property>
> <property>
> <name>fs.viewfs.mounttable.default.link./ns1/user</name>
> <value>hdfs://nameservice1/user</value>
> </property>
> <property>
> <name>fs.viewfs.mounttable.default.link./ns2/user</name>
> <value>hdfs://nameservice2/user</value>
> </property>
> {noformat}
> Got the following error .
> {noformat}
> spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client
> --num-executors 3 --driver-memory 512m --executor-memory 512m
> --executor-cores 1 ${SPARK_HOME}/lib/spark-examples*.jar 10
> 18/01/17 02:14:45 INFO spark.SparkContext: Added JAR
> file:/home/nayakxxxx/hdp26_c4000_stg/spark2/lib/spark-examples_2.11-2.1.1.2.6.2.0-205.jar
> at spark://xxxxx:35633/jars/spark-examples_2.11-2.1.1.2.6.2.0-205.jar with
> timestamp 1516155285534
> 18/01/17 02:14:46 INFO client.ConfiguredRMFailoverProxyProvider: Failing over
> to rm2
> 18/01/17 02:14:46 INFO yarn.Client: Requesting a new application from cluster
> with 26 NodeManagers
> 18/01/17 02:14:46 INFO yarn.Client: Verifying our application has not
> requested more than the maximum memory capability of the cluster (13800 MB
> per container)
> 18/01/17 02:14:46 INFO yarn.Client: Will allocate AM container, with 896 MB
> memory including 384 MB overhead
> 18/01/17 02:14:46 INFO yarn.Client: Setting up container launch context for
> our AM
> 18/01/17 02:14:46 INFO yarn.Client: Setting up the launch environment for our
> AM container
> 18/01/17 02:14:46 INFO yarn.Client: Preparing resources for our AM container
> 18/01/17 02:14:46 INFO security.HDFSCredentialProvider: getting token for
> namenode: viewfs:/user/nayakxxxx
> 18/01/17 02:14:46 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token
> 22488202 for nayakxxxx on ha-hdfs:nameservice1
> 18/01/17 02:14:46 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 50
> for nayakxxxx on ha-hdfs:nameservice2
> 18/01/17 02:14:47 INFO hive.metastore: Trying to connect to metastore with
> URI thrift://XXXX:9083
> 18/01/17 02:14:47 INFO hive.metastore: Connected to metastore.
> 18/01/17 02:14:49 INFO security.HiveCredentialProvider: Get Token from hive
> metastore: Kind: HIVE_DELEGATION_TOKEN, Service: , Ident: 00 29 6e 61 79 61
> 6b 6e 69 68 61 72 72 61 30 31 40 53 54 47 32 30 30 30 2e 48 41 44 4f 4f 50 2e
> 52 41 4b 55 54 45 4e 2e 43 4f 4d 04 68 69 76 65 00 8a 01 61 01 e5 be 03 8a 01
> 61 25 f2 42 03 8d 02 21 bb 8e 02 b7
> 18/01/17 02:14:49 WARN yarn.Client: Neither spark.yarn.jars nor
> spark.yarn.archive is set, falling back to uploading libraries under
> SPARK_HOME.
> 18/01/17 02:14:50 INFO yarn.Client: Uploading resource
> file:/tmp/spark-7498ee81-d22b-426e-9466-3a08f7c827b1/__spark_libs__6643608006679813597.zip
> ->
> viewfs:/user/nayakxxxx/.sparkStaging/application_1515035441414_275503/__spark_libs__6643608006679813597.zip
> 18/01/17 02:14:55 INFO yarn.Client: Uploading resource
> file:/tmp/spark-7498ee81-d22b-426e-9466-3a08f7c827b1/__spark_conf__405432153902988742.zip
> ->
> viewfs:/user/nayakxxxx/.sparkStaging/application_1515035441414_275503/__spark_conf__.zip
> 18/01/17 02:14:55 INFO spark.SecurityManager: Changing view acls to: nayakxxxx
> 18/01/17 02:14:55 INFO spark.SecurityManager: Changing modify acls to:
> nayakxxxx
> 18/01/17 02:14:55 INFO spark.SecurityManager: Changing view acls groups to:
> 18/01/17 02:14:55 INFO spark.SecurityManager: Changing modify acls groups to:
> 18/01/17 02:14:55 INFO spark.SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(nayakxxxx);
> groups with view permissions: Set(); users with modify permissions:
> Set(nayakxxxx); groups with modify permissions: Set()
> 18/01/17 02:14:55 INFO yarn.Client: Submitting application
> application_1515035441414_275503 to ResourceManager
> 18/01/17 02:14:55 INFO impl.YarnClientImpl: Submitted application
> application_1515035441414_275503
> 18/01/17 02:14:55 INFO cluster.SchedulerExtensionServices: Starting Yarn
> extension services with app application_1515035441414_275503 and attemptId
> None
> 18/01/17 02:14:56 INFO yarn.Client: Application report for
> application_1515035441414_275503 (state: ACCEPTED)
> 18/01/17 02:14:56 INFO yarn.Client:
> client token: Token { kind: YARN_CLIENT_TOKEN, service: }
> diagnostics: AM container is launched, waiting for AM container to
> Register with RM
> ApplicationMaster host: N/A
> ApplicationMaster RPC port: -1
> queue: common
> start time: 1516155295100
> final status: UNDEFINED
> tracking URL:
> http://XXXX:8088/proxy/application_1515035441414_275503/
> user: nayakxxxx
> 18/01/17 02:14:57 INFO yarn.Client: Application report for
> application_1515035441414_275503 (state: ACCEPTED)
> 18/01/17 02:14:58 INFO yarn.Client: Application report for
> application_1515035441414_275503 (state: ACCEPTED)
> 18/01/17 02:14:59 INFO yarn.Client: Application report for
> application_1515035441414_275503 (state: ACCEPTED)
> 18/01/17 02:15:00 INFO yarn.Client: Application report for
> application_1515035441414_275503 (state: ACCEPTED)
> 18/01/17 02:15:01 INFO yarn.Client: Application report for
> application_1515035441414_275503 (state: ACCEPTED)
> 18/01/17 02:15:02 INFO yarn.Client: Application report for
> application_1515035441414_275503 (state: ACCEPTED)
> 18/01/17 02:15:03 INFO yarn.Client: Application report for
> application_1515035441414_275503 (state: ACCEPTED)
> 18/01/17 02:15:04 INFO yarn.Client: Application report for
> application_1515035441414_275503 (state: ACCEPTED)
> 18/01/17 02:15:05 INFO yarn.Client: Application report for
> application_1515035441414_275503 (state: ACCEPTED)
> 18/01/17 02:15:06 INFO yarn.Client: Application report for
> application_1515035441414_275503 (state: ACCEPTED)
> 18/01/17 02:15:07 INFO yarn.Client: Application report for
> application_1515035441414_275503 (state: ACCEPTED)
> 18/01/17 02:15:08 INFO yarn.Client: Application report for
> application_1515035441414_275503 (state: ACCEPTED)
> 18/01/17 02:15:09 INFO yarn.Client: Application report for
> application_1515035441414_275503 (state: ACCEPTED)
> 18/01/17 02:15:10 INFO yarn.Client: Application report for
> application_1515035441414_275503 (state: ACCEPTED)
> 18/01/17 02:15:11 INFO yarn.Client: Application report for
> application_1515035441414_275503 (state: ACCEPTED)
> 18/01/17 02:15:12 INFO yarn.Client: Application report for
> application_1515035441414_275503 (state: ACCEPTED)
> 18/01/17 02:15:13 INFO yarn.Client: Application report for
> application_1515035441414_275503 (state: ACCEPTED)
> 18/01/17 02:15:14 INFO yarn.Client: Application report for
> application_1515035441414_275503 (state: ACCEPTED)
> 18/01/17 02:15:15 INFO yarn.Client: Application report for
> application_1515035441414_275503 (state: ACCEPTED)
> 18/01/17 02:15:16 INFO yarn.Client: Application report for
> application_1515035441414_275503 (state: ACCEPTED)
> 18/01/17 02:15:17 INFO yarn.Client: Application report for
> application_1515035441414_275503 (state: ACCEPTED)
> 18/01/17 02:15:18 INFO yarn.Client: Application report for
> application_1515035441414_275503 (state: ACCEPTED)
> 18/01/17 02:15:19 INFO yarn.Client: Application report for
> application_1515035441414_275503 (state: ACCEPTED)
> 18/01/17 02:15:20 INFO yarn.Client: Application report for
> application_1515035441414_275503 (state: ACCEPTED)
> 18/01/17 02:15:21 INFO yarn.Client: Application report for
> application_1515035441414_275503 (state: ACCEPTED)
> 18/01/17 02:15:22 INFO yarn.Client: Application report for
> application_1515035441414_275503 (state: ACCEPTED)
> 18/01/17 02:15:23 INFO yarn.Client: Application report for
> application_1515035441414_275503 (state: ACCEPTED)
> 18/01/17 02:15:24 INFO yarn.Client: Application report for
> application_1515035441414_275503 (state: ACCEPTED)
> 18/01/17 02:15:25 INFO yarn.Client: Application report for
> application_1515035441414_275503 (state: ACCEPTED)
> 18/01/17 02:15:26 INFO yarn.Client: Application report for
> application_1515035441414_275503 (state: ACCEPTED)
> 18/01/17 02:15:27 INFO yarn.Client: Application report for
> application_1515035441414_275503 (state: ACCEPTED)
> 18/01/17 02:15:28 INFO yarn.Client: Application report for
> application_1515035441414_275503 (state: ACCEPTED)
> 18/01/17 02:15:29 INFO yarn.Client: Application report for
> application_1515035441414_275503 (state: ACCEPTED)
> 18/01/17 02:15:30 INFO yarn.Client: Application report for
> application_1515035441414_275503 (state: ACCEPTED)
> 18/01/17 02:15:31 INFO yarn.Client: Application report for
> application_1515035441414_275503 (state: ACCEPTED)
> 18/01/17 02:15:32 INFO yarn.Client: Application report for
> application_1515035441414_275503 (state: ACCEPTED)
> 18/01/17 02:15:33 INFO yarn.Client: Application report for
> application_1515035441414_275503 (state: ACCEPTED)
> 18/01/17 02:15:34 INFO yarn.Client: Application report for
> application_1515035441414_275503 (state: FAILED)
> 18/01/17 02:15:34 INFO yarn.Client:
> client token: N/A
> diagnostics: Application application_1515035441414_275503 failed 10
> times due to AM Container for appattempt_1515035441414_275503_000010 exited
> with exitCode: -1000
> For more detailed output, check the application tracking page:
> http://xxxxx:8088/cluster/app/application_1515035441414_275503 Then click on
> links to logs of each attempt.
> Diagnostics: ViewFs: Cannot initialize: Empty Mount table in config for
> viewfs://default/
> java.io.IOException: ViewFs: Cannot initialize: Empty Mount table in config
> for viewfs://default/
> at org.apache.hadoop.fs.viewfs.InodeTree.<init>(InodeTree.java:337)
> at
> org.apache.hadoop.fs.viewfs.ViewFileSystem$1.<init>(ViewFileSystem.java:168)
> at
> org.apache.hadoop.fs.viewfs.ViewFileSystem.initialize(ViewFileSystem.java:168)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2795)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:99)
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2829)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2811)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:390)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
> at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:251)
> at
> org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:63)
> at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:361)
> at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:358)
> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Failing this attempt. Failing the application.
> ApplicationMaster host: N/A
> ApplicationMaster RPC port: -1
> queue: common
> start time: 1516155295100
> final status: FAILED
> tracking URL:
> http://xxxxxx:8088/cluster/app/application_1515035441414_275503
> user: nayakxxxx
> 18/01/17 02:15:34 INFO yarn.Client: Deleted staging directory
> viewfs:/user/nayakxxxx/.sparkStaging/application_1515035441414_275503
> 18/01/17 02:15:34 ERROR spark.SparkContext: Error initializing SparkContext.
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
