Re: Saprk 1.6 Driver Memory Issue

Hi Karthik , You must set the value before the SparkContext (sc) is created. Also don't assign too much overhead like 20g for maxResultSize , You can set it to 2G maximum as per your error message. Also if you are using Java 1.8 , Please add the below section in your Yarn-site.xml

Submitting Job to YARN-Cluster using Spark Job Server

Hi Guys , Does any of you have tried this mechanism before? I am able to run it locally and get the output ..But how do i submit the job to the Yarn-Cluster using Spark-JobServer. Any documentation ? Regards Ashesh -- View this message in context:

RE: Unable to Access files in Hadoop HA enabled from using Spark

Are you running from eclipse ? If so add the *Hadoop_conf_dir* path to the classpath And then you can access your hdfs directory as below object sparkExample { def main(args: Array[String]){ val logname = "///user/hduser/input/sample.txt" val conf = new

Re: Unable to Access files in Hadoop HA enabled from using Spark

Try adding the following property into hdfs-site.xml dfs.client.failover.proxy.provider. org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider -- View this message in context:

Re: Running Spark on Yarn-Client/Cluster mode

I have updated all my nodes in the Cluster to have 4GB RAM memory , but still face the same error when trying to launch Spark-Shell in yarn-client mode Any suggestion ? -- View this message in context:

Re: Running Spark on Yarn-Client/Cluster mode

I have Modified my Yarn-site to include the following properties , yarn.nodemanager.resource.memory-mb 4096 yarn.scheduler.minimum-allocation-mb 256 yarn.scheduler.maximum-allocation-mb 2250

Re: Running Spark on Yarn-Client/Cluster mode

Hi Dhiraj , Thanks for the clarification , Yes i indeed checked that Both YARN related (Nodemanager & ResourceManager) daemons are running in their respective nodes and i can access HDFS directory structure from each node. I am using Hadoop version 2.7.2 and i have downloaded Pre-build version

Re: Running Spark on Yarn-Client/Cluster mode

Few more added information with Nodes Memory and Core ptfhadoop01v - 4GB ntpcam01v - 1GB ntpcam03v - 2GB Each of the VM has only 1 core CPU -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Running-Spark-on-Yarn-Client-Cluster-mode-tp26691p26714.html Sent

Re: Running Spark on Yarn-Client/Cluster mode

Hi , Just a Quick Update , After trying for a while , i rebooted all the Three machines used in the Cluster and formatted namenode and ZKFC . Then i started every Daemon in the Cluster. After all the Daemons were up and Running i tried to issue the same command as earlier

Re: Running Spark on Yarn-Client/Cluster mode

Hi , I am also attaching a screenshot of my ResourceManager UI which shows the available cores and memory allocated for each node , -- View this message in context:

Re: Running Spark on Yarn-Client/Cluster mode

Hi Guys , Thanks for your valuable inputs , I have tried few alternatives as suggested but it all leads me to same result - Unable to start Spark Context @Dhiraj Peechara I am able to start my spark SC(SparkContext) in stand-alone mode by just issuing the *$spark-shell* command from the

Running Spark on Yarn-Client/Cluster mode

Hi, I am new to the world of Hadoop and this is my first post in here. Recently i have setup a Multi-node Hadoop cluster (3 Nodes Cluster) with HA feature for Namenode & ResourceManager with Zookeeper server. *Daemons running in NN1 (ptfhadoop01v) :* 2945 JournalNode 3137