Re: How to run spark shell using YARN

;> restart-service-emr/ >> >> >> >> Femi >> >> >> >> *From: *kant kodali <kanth...@gmail.com> >> *Date: *Wednesday, March 14, 2018 at 6:16 AM >> *To: *Femi Anthony <femib...@gmail.com> >> *Cc: *vermanurag <anurag.ve..

Re: How to run spark shell using YARN

thlogic.com>, "user @spark" < > user@spark.apache.org> > *Subject: *Re: How to run spark shell using YARN > > > > 16GB RAM. AWS m4.xlarge. It's a three node cluster and I only have YARN > and HDFS running. Resources are barely used however I believe there

Re: How to run spark shell using YARN

l.com> Cc: vermanurag <anurag.ve...@fnmathlogic.com>, "user @spark" <user@spark.apache.org> Subject: Re: How to run spark shell using YARN 16GB RAM. AWS m4.xlarge. It's a three node cluster and I only have YARN and HDFS running. Resources are barely used however I believ

Re: How to run spark shell using YARN

16GB RAM. AWS m4.xlarge. It's a three node cluster and I only have YARN and HDFS running. Resources are barely used however I believe there is something in my config that is preventing YARN to see that I have good amount of resources I think (thats my guess I never worked with YARN before). My

Re: How to run spark shell using YARN

What's the hardware configuration of the box you're running on i.e. how much memory does it have ? Femi On Wed, Mar 14, 2018 at 5:32 AM, kant kodali wrote: > Tried this > > ./spark-shell --master yarn --deploy-mode client --executor-memory 4g > > > Same issue. Keeps going

Re: How to run spark shell using YARN

Tried this ./spark-shell --master yarn --deploy-mode client --executor-memory 4g Same issue. Keeps going forever.. 18/03/14 09:31:25 INFO Client: client token: N/A diagnostics: N/A ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: default start time: 1521019884656

Re: How to run spark shell using YARN

Make sure you have enough memory allocated for Spark workers, try specifying executor memory as follows: --executor-memory to spark-submit. On Wed, Mar 14, 2018 at 3:25 AM, kant kodali wrote: > I am using spark 2.3.0 and hadoop 2.7.3. > > Also I have done the following

Re: How to run spark shell using YARN

I am using spark 2.3.0 and hadoop 2.7.3. Also I have done the following and restarted all. But I still see ACCEPTED: waiting for AM container to be allocated, launched and register with RM. And i am unable to spawn spark-shell. editing $HADOOP_HOME/etc/hadoop/capacity-scheduler.xml and change

Re: How to run spark shell using YARN

any idea? On Wed, Mar 14, 2018 at 12:12 AM, kant kodali wrote: > I set core-site.xml, hdfs-site.xml, yarn-site.xml as per this website > and these are the > only three files I changed Do I need to set or change

Re: How to run spark shell using YARN

I set core-site.xml, hdfs-site.xml, yarn-site.xml as per this website and these are the only three files I changed Do I need to set or change anything in mapred-site.xml (As of now I have not touched mapred-site.xml)? when I do yarn -node

Re: How to run spark shell using YARN

This does not look like Spark error. Looks like yarn has not been able to allocate resources for spark driver. If you check resource manager UI you are likely to see this as spark application waiting for resources. Try reducing the driver node memory and/ or other bottlenecks based on what you see

Re: How to run spark shell using YARN

Looks like you either have a misconfigured HDFS service, or you're using the wrong configuration on the client. BTW, as I said in the previous response, the message you saw initially is *not* an error. If you're just trying things out, you don't need to do anything and Spark should still work.

Re: How to run spark shell using YARN

Hi, I read that doc several times now. I am stuck with the below error message when I run ./spark-shell --master yarn --deploy-mode client. I have my HADOOP_CONF_DIR set to /usr/local/hadoop-2.7.3/etc/hadoop and SPARK_HOME set to /usr/local/spark on all 3 machines (1 node for Resource Manager

Re: How to run spark shell using YARN

That's not an error, just a warning. The docs [1] have more info about the config options mentioned in that message. [1] http://spark.apache.org/docs/latest/running-on-yarn.html On Mon, Mar 12, 2018 at 4:42 PM, kant kodali wrote: > Hi All, > > I am trying to use YARN for the

How to run spark shell using YARN

Hi All, I am trying to use YARN for the very first time. I believe I configured all the resource manager and name node fine. And then I run the below command ./spark-shell --master yarn --deploy-mode client *I get the below output and it hangs there forever *(I had been waiting over 10 minutes)