I'm getting some spark exception. Please look this log trace ( *http://pastebin.com/xL9jaRUa <http://pastebin.com/xL9jaRUa>* ).
*Thanks*, <https://in.linkedin.com/in/ramkumarcs31> On Wed, Aug 19, 2015 at 10:20 PM, Hari Shreedharan < hshreedha...@cloudera.com> wrote: > It looks like you are having issues with the files getting distributed to > the cluster. What is the exception you are getting now? > > > On Wednesday, August 19, 2015, Ramkumar V <ramkumar.c...@gmail.com> wrote: > >> Thanks a lot for your suggestion. I had modified HADOOP_CONF_DIR in >> spark-env.sh so that core-site.xml is under HADOOP_CONF_DIR. i can able >> to see the logs like that you had shown above. Now i can able to run for 3 >> minutes and store results between every minutes. After sometimes, there is >> an exception. How to fix this exception ? and Can you please explain where >> its going wrong ? >> >> *Log Link : http://pastebin.com/xL9jaRUa <http://pastebin.com/xL9jaRUa> * >> >> >> *Thanks*, >> <https://in.linkedin.com/in/ramkumarcs31> >> >> >> On Wed, Aug 19, 2015 at 1:54 PM, Jeff Zhang <zjf...@gmail.com> wrote: >> >>> HADOOP_CONF_DIR is the environment variable point to the hadoop conf >>> directory. Not sure how CDH organize that, make sure core-site.xml is >>> under HADOOP_CONF_DIR. >>> >>> On Wed, Aug 19, 2015 at 4:06 PM, Ramkumar V <ramkumar.c...@gmail.com> >>> wrote: >>> >>>> We are using Cloudera-5.3.1. since it is one of the earlier version of >>>> CDH, it doesnt supports the latest version of spark. So i installed >>>> spark-1.4.1 separately in my machine. I couldnt able to do spark-submit in >>>> cluster mode. How to core-site.xml under classpath ? it will be very >>>> helpful if you could explain in detail to solve this issue. >>>> >>>> *Thanks*, >>>> <https://in.linkedin.com/in/ramkumarcs31> >>>> >>>> >>>> On Fri, Aug 14, 2015 at 8:25 AM, Jeff Zhang <zjf...@gmail.com> wrote: >>>> >>>>> >>>>> 1. 15/08/12 13:24:49 INFO Client: Source and destination file >>>>> systems are the same. Not copying >>>>> >>>>> file:/home/hdfs/spark-1.4.1/assembly/target/scala-2.10/spark-assembly-1.4.1-hadoop2.5.0-cdh5.3.5.jar >>>>> 2. 15/08/12 13:24:49 INFO Client: Source and destination file >>>>> systems are the same. Not copying >>>>> >>>>> file:/home/hdfs/spark-1.4.1/external/kafka-assembly/target/spark-streaming-kafka-assembly_2.10-1.4.1.jar >>>>> 3. 15/08/12 13:24:49 INFO Client: Source and destination file >>>>> systems are the same. Not copying >>>>> file:/home/hdfs/spark-1.4.1/python/lib/pyspark.zip >>>>> 4. 15/08/12 13:24:49 INFO Client: Source and destination file >>>>> systems are the same. Not copying >>>>> file:/home/hdfs/spark-1.4.1/python/lib/py4j-0.8.2.1-src.zip >>>>> 5. 15/08/12 13:24:49 INFO Client: Source and destination file >>>>> systems are the same. Not copying >>>>> file:/home/hdfs/spark-1.4.1/examples/src/main/python/streaming/kyt.py >>>>> 6. >>>>> >>>>> >>>>> 1. diagnostics: Application application_1437639737006_3808 failed >>>>> 2 times due to AM Container for appattempt_1437639737006_3808_000002 >>>>> exited >>>>> with exitCode: -1000 due to: File >>>>> file:/home/hdfs/spark-1.4.1/python/lib/pyspark.zip does not exist >>>>> 2. .Failing this attempt.. Failing the application. >>>>> >>>>> >>>>> >>>>> The machine you run spark is the client machine, while the yarn AM is >>>>> running on another machine. And the yarn AM complains that the files are >>>>> not found as your logs shown. >>>>> From the logs, its seems that these files are not copied to the HDFS >>>>> as local resources. I doubt that you didn't put core-site.xml under your >>>>> classpath, so that spark can not detect your remote file system and won't >>>>> copy the files to hdfs as local resources. Usually in yarn-cluster mode, >>>>> you should be able to see the logs like following. >>>>> >>>>> > 15/08/14 10:48:49 INFO yarn.Client: Preparing resources for our AM >>>>> container >>>>> > 15/08/14 10:48:49 INFO yarn.Client: Uploading resource >>>>> file:/Users/abc/github/spark/assembly/target/scala-2.10/spark-assembly-1.5.0-SNAPSHOT-hadoop2.6.0.jar >>>>> -> hdfs:// >>>>> 0.0.0.0:9000/user/abc/.sparkStaging/application_1439432662178_0019/spark-assembly-1.5.0-SNAPSHOT-hadoop2.6.0.jar >>>>> > 15/08/14 10:48:50 INFO yarn.Client: Uploading resource >>>>> file:/Users/abc/github/spark/spark.py -> hdfs:// >>>>> 0.0.0.0:9000/user/abc/.sparkStaging/application_1439432662178_0019/spark.py >>>>> > 15/08/14 10:48:50 INFO yarn.Client: Uploading resource >>>>> file:/Users/abc/github/spark/python/lib/pyspark.zip -> hdfs:// >>>>> 0.0.0.0:9000/user/abc/.sparkStaging/application_1439432662178_0019/pyspark.zip >>>>> >>>>> On Thu, Aug 13, 2015 at 2:50 PM, Ramkumar V <ramkumar.c...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I have a cluster of 1 master and 2 slaves. I'm running a spark >>>>>> streaming in master and I want to utilize all nodes in my cluster. i had >>>>>> specified some parameters like driver memory and executor memory in my >>>>>> code. when i give --deploy-mode cluster --master yarn-cluster in my >>>>>> spark-submit, it gives the following error. >>>>>> >>>>>> Log link : *http://pastebin.com/kfyVWDGR >>>>>> <http://pastebin.com/kfyVWDGR>* >>>>>> >>>>>> How to fix this issue ? Please help me if i'm doing wrong. >>>>>> >>>>>> >>>>>> *Thanks*, >>>>>> Ramkumar V >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Best Regards >>>>> >>>>> Jeff Zhang >>>>> >>>> >>>> >>> >>> >>> -- >>> Best Regards >>> >>> Jeff Zhang >>> >> >> > > -- > > Thanks, > Hari > >
