Hi Dilip, I tried this option also spark-submit --master yarn --class org.spark.apache.CsvDataSource /home/cloudera/Desktop/TestMain.jar --files hdfs://quickstart.cloudera:8020/people_csv & getting similar error
Exception in thread "main" org.apache.spark.SparkException: Could not parse Master URL: '--files' at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2244) at org.apache.spark.SparkContext.<init>(SparkContext.scala:361) at org.apache.spark.SparkContext.<init>(SparkContext.scala:154) at org.spark.apache.CsvDataSource$.main(CsvDataSource.scala:10) at org.spark.apache.CsvDataSource.main(CsvDataSource.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Thanks & Regards, Giri. On Thu, Oct 15, 2015 at 5:43 PM, Dilip Biswal <dbis...@us.ibm.com> wrote: > Hi Giri, > > You are perhaps missing the "--files" option before the supplied hdfs > file name ? > > spark-submit --master yarn --class org.spark.apache.CsvDataSource > /home/cloudera/Desktop/TestMain.jar *--files* > hdfs://quickstart.cloudera:8020/people_csv > > Please refer to Ritchard's comments on why the --files option may be > redundant in > your case. > > Regards, > Dilip Biswal > Tel: 408-463-4980 > dbis...@us.ibm.com > > > > From: Giri <giridhar.madduk...@gmail.com> > To: user@spark.apache.org > Date: 10/15/2015 02:44 AM > Subject: Re: SPARK SQL Error > ------------------------------ > > > > Hi Ritchard, > > Thank you so much again for your input.This time I ran the command in the > below way > spark-submit --master yarn --class org.spark.apache.CsvDataSource > /home/cloudera/Desktop/TestMain.jar > hdfs://quickstart.cloudera:8020/people_csv > But I am facing the new error "Could not parse Master URL: > 'hdfs://quickstart.cloudera:8020/people_csv'" > file path is correct > > hadoop fs -ls hdfs://quickstart.cloudera:8020/people_csv > -rw-r--r-- 1 cloudera supergroup 29 2015-10-10 00:02 > hdfs://quickstart.cloudera:8020/people_csv > > Can you help me to fix this new error > > 15/10/15 02:24:39 INFO spark.SparkContext: Added JAR > file:/home/cloudera/Desktop/TestMain.jar at > http://10.0.2.15:40084/jars/TestMain.jarwith timestamp 1444901079484 > Exception in thread "main" org.apache.spark.SparkException: Could not parse > Master URL: 'hdfs://quickstart.cloudera:8020/people_csv' > at > > org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2244) > at > org.apache.spark.SparkContext.<init>(SparkContext.scala:361) > at > org.apache.spark.SparkContext.<init>(SparkContext.scala:154) > at > org.spark.apache.CsvDataSource$.main(CsvDataSource.scala:10) > at org.spark.apache.CsvDataSource.main(CsvDataSource.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569) > at > org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166) > at > org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189) > at > org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110) > at > org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > > > Thanks & Regards, > Giri. > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/SPARK-SQL-Error-tp25050p25075.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > > > >
