RE: Problem submitting a spark job using yarn-client as master

Sent: Tuesday, November 15, 2016 9:27 PM To: David Robison <david.robi...@psgglobal.net> Cc: user@spark.apache.org Subject: Re: Problem submitting a spark job using yarn-client as master you can set hdfs as defaults, sparksession.sparkContext().hadoopConfiguration().set("fs.defaultFS&

Re: Problem submitting a spark job using yarn-client as master

you can set hdfs as defaults, sparksession.sparkContext().hadoopConfiguration().set("fs.defaultFS", “hdfs://master_node:8020”); Regards Rohit On Nov 16, 2016, at 3:15 AM, David Robison > wrote: I am trying to submit a spark job

Problem submitting a spark job using yarn-client as master

I am trying to submit a spark job through the yarn-client master setting. The job gets created and submitted to the clients but immediately errors out. Here is the relevant portion of the log: 15:39:37,385 INFO [org.apache.spark.deploy.yarn.Client] (default task-1) Requesting a new