yes , when i see my yarn logs for that particular failed app_id, i got the following error.
ERROR yarn.ApplicationMaster: SparkContext did not initialize after waiting for 100000 ms. Please check earlier log output for errors. Failing the application For this error, I need to change the 'SparkContext', set the Master on yarn cluster ( SetMaster("yarn-cluster") ). Its working fine in cluster mode. Thanks for everyone. *Thanks*, <https://in.linkedin.com/in/ramkumarcs31> On Fri, Aug 21, 2015 at 6:41 AM, Jeff Zhang <zjf...@gmail.com> wrote: > AM fails to launch, could you check the yarn app logs ? You can use > command "yarn logs -<your_app_id>" to get the yarn app logs. > > > > On Thu, Aug 20, 2015 at 1:15 AM, Ramkumar V <ramkumar.c...@gmail.com> > wrote: > >> I'm getting some spark exception. Please look this log trace ( >> *http://pastebin.com/xL9jaRUa >> <http://pastebin.com/xL9jaRUa>* ). >> >> *Thanks*, >> <https://in.linkedin.com/in/ramkumarcs31> >> >> >> On Wed, Aug 19, 2015 at 10:20 PM, Hari Shreedharan < >> hshreedha...@cloudera.com> wrote: >> >>> It looks like you are having issues with the files getting distributed >>> to the cluster. What is the exception you are getting now? >>> >>> >>> On Wednesday, August 19, 2015, Ramkumar V <ramkumar.c...@gmail.com> >>> wrote: >>> >>>> Thanks a lot for your suggestion. I had modified HADOOP_CONF_DIR in >>>> spark-env.sh so that core-site.xml is under HADOOP_CONF_DIR. i can >>>> able to see the logs like that you had shown above. Now i can able to run >>>> for 3 minutes and store results between every minutes. After sometimes, >>>> there is an exception. How to fix this exception ? and Can you please >>>> explain where its going wrong ? >>>> >>>> *Log Link : http://pastebin.com/xL9jaRUa >>>> <http://pastebin.com/xL9jaRUa> * >>>> >>>> >>>> *Thanks*, >>>> <https://in.linkedin.com/in/ramkumarcs31> >>>> >>>> >>>> On Wed, Aug 19, 2015 at 1:54 PM, Jeff Zhang <zjf...@gmail.com> wrote: >>>> >>>>> HADOOP_CONF_DIR is the environment variable point to the hadoop conf >>>>> directory. Not sure how CDH organize that, make sure core-site.xml is >>>>> under HADOOP_CONF_DIR. >>>>> >>>>> On Wed, Aug 19, 2015 at 4:06 PM, Ramkumar V <ramkumar.c...@gmail.com> >>>>> wrote: >>>>> >>>>>> We are using Cloudera-5.3.1. since it is one of the earlier version >>>>>> of CDH, it doesnt supports the latest version of spark. So i installed >>>>>> spark-1.4.1 separately in my machine. I couldnt able to do spark-submit >>>>>> in >>>>>> cluster mode. How to core-site.xml under classpath ? it will be very >>>>>> helpful if you could explain in detail to solve this issue. >>>>>> >>>>>> *Thanks*, >>>>>> <https://in.linkedin.com/in/ramkumarcs31> >>>>>> >>>>>> >>>>>> On Fri, Aug 14, 2015 at 8:25 AM, Jeff Zhang <zjf...@gmail.com> wrote: >>>>>> >>>>>>> >>>>>>> 1. 15/08/12 13:24:49 INFO Client: Source and destination file >>>>>>> systems are the same. Not copying >>>>>>> >>>>>>> file:/home/hdfs/spark-1.4.1/assembly/target/scala-2.10/spark-assembly-1.4.1-hadoop2.5.0-cdh5.3.5.jar >>>>>>> 2. 15/08/12 13:24:49 INFO Client: Source and destination file >>>>>>> systems are the same. Not copying >>>>>>> >>>>>>> file:/home/hdfs/spark-1.4.1/external/kafka-assembly/target/spark-streaming-kafka-assembly_2.10-1.4.1.jar >>>>>>> 3. 15/08/12 13:24:49 INFO Client: Source and destination file >>>>>>> systems are the same. Not copying >>>>>>> file:/home/hdfs/spark-1.4.1/python/lib/pyspark.zip >>>>>>> 4. 15/08/12 13:24:49 INFO Client: Source and destination file >>>>>>> systems are the same. Not copying >>>>>>> file:/home/hdfs/spark-1.4.1/python/lib/py4j-0.8.2.1-src.zip >>>>>>> 5. 15/08/12 13:24:49 INFO Client: Source and destination file >>>>>>> systems are the same. Not copying >>>>>>> file:/home/hdfs/spark-1.4.1/examples/src/main/python/streaming/kyt.py >>>>>>> 6. >>>>>>> >>>>>>> >>>>>>> 1. diagnostics: Application application_1437639737006_3808 >>>>>>> failed 2 times due to AM Container for >>>>>>> appattempt_1437639737006_3808_000002 >>>>>>> exited with exitCode: -1000 due to: File >>>>>>> file:/home/hdfs/spark-1.4.1/python/lib/pyspark.zip does not exist >>>>>>> 2. .Failing this attempt.. Failing the application. >>>>>>> >>>>>>> >>>>>>> >>>>>>> The machine you run spark is the client machine, while the yarn AM >>>>>>> is running on another machine. And the yarn AM complains that the files >>>>>>> are >>>>>>> not found as your logs shown. >>>>>>> From the logs, its seems that these files are not copied to the HDFS >>>>>>> as local resources. I doubt that you didn't put core-site.xml under your >>>>>>> classpath, so that spark can not detect your remote file system and >>>>>>> won't >>>>>>> copy the files to hdfs as local resources. Usually in yarn-cluster mode, >>>>>>> you should be able to see the logs like following. >>>>>>> >>>>>>> > 15/08/14 10:48:49 INFO yarn.Client: Preparing resources for our AM >>>>>>> container >>>>>>> > 15/08/14 10:48:49 INFO yarn.Client: Uploading resource >>>>>>> file:/Users/abc/github/spark/assembly/target/scala-2.10/spark-assembly-1.5.0-SNAPSHOT-hadoop2.6.0.jar >>>>>>> -> hdfs:// >>>>>>> 0.0.0.0:9000/user/abc/.sparkStaging/application_1439432662178_0019/spark-assembly-1.5.0-SNAPSHOT-hadoop2.6.0.jar >>>>>>> > 15/08/14 10:48:50 INFO yarn.Client: Uploading resource >>>>>>> file:/Users/abc/github/spark/spark.py -> hdfs:// >>>>>>> 0.0.0.0:9000/user/abc/.sparkStaging/application_1439432662178_0019/spark.py >>>>>>> > 15/08/14 10:48:50 INFO yarn.Client: Uploading resource >>>>>>> file:/Users/abc/github/spark/python/lib/pyspark.zip -> hdfs:// >>>>>>> 0.0.0.0:9000/user/abc/.sparkStaging/application_1439432662178_0019/pyspark.zip >>>>>>> >>>>>>> On Thu, Aug 13, 2015 at 2:50 PM, Ramkumar V <ramkumar.c...@gmail.com >>>>>>> > wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I have a cluster of 1 master and 2 slaves. I'm running a spark >>>>>>>> streaming in master and I want to utilize all nodes in my cluster. i >>>>>>>> had >>>>>>>> specified some parameters like driver memory and executor memory in my >>>>>>>> code. when i give --deploy-mode cluster --master yarn-cluster in my >>>>>>>> spark-submit, it gives the following error. >>>>>>>> >>>>>>>> Log link : *http://pastebin.com/kfyVWDGR >>>>>>>> <http://pastebin.com/kfyVWDGR>* >>>>>>>> >>>>>>>> How to fix this issue ? Please help me if i'm doing wrong. >>>>>>>> >>>>>>>> >>>>>>>> *Thanks*, >>>>>>>> Ramkumar V >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Best Regards >>>>>>> >>>>>>> Jeff Zhang >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Best Regards >>>>> >>>>> Jeff Zhang >>>>> >>>> >>>> >>> >>> -- >>> >>> Thanks, >>> Hari >>> >>> >> > > > -- > Best Regards > > Jeff Zhang >