Hi Satish, The problem is that `--jars` accepts a comma-delimited list of jars! E.g.
spark-submit ... --jars lib1.jar,lib2.jar,lib3.jar main.jar where main.jar is your main application jar (the one that starts a SparkContext), and lib*.jar refer to additional libraries that your main application jar uses. -Andrew 2015-08-13 3:22 GMT-07:00 Javier Domingo Cansino <javier.domi...@fon.com>: > Please notice that 'jars: null' > > I don't know why you put ///..... but I would propose you just put normal > absolute paths. > > dse spark-submit --master spark://10.246.43.15:7077 --class HelloWorld > --jars /home/missingmerch/postgresql-9.4-1201.jdbc41.jar > /home/missingmerch/dse.jar > /home/missingmerch/spark-cassandra-connector-java_2.10-1.1.1.jar > /home/missingmerch/etl-0.0.1-SNAPSHOT.jar > > Hope this is helpful! > > [image: Fon] <http://www.fon.com/>Javier Domingo CansinoResearch & > Development Engineer+34 946545847Skype: javier.domingo.fonAll information > in this email is confidential <http://corp.fon.com/legal/email-disclaimer> > > On Tue, Aug 11, 2015 at 3:42 PM, satish chandra j < > jsatishchan...@gmail.com> wrote: > >> HI, >> >> Please find the log details below: >> >> >> dse spark-submit --verbose --master local --class HelloWorld >> etl-0.0.1-SNAPSHOT.jar --jars >> file:/home/missingmerch/postgresql-9.4-1201.jdbc41.jar >> file:/home/missingmerch/dse.jar >> file:/home/missingmerch/postgresql-9.4-1201.jdbc41.jar >> >> Using properties file: /etc/dse/spark/spark-defaults.conf >> >> Adding default property: >> spark.cassandra.connection.factory=com.datastax.bdp.spark.DseCassandraConnectionFactory >> >> Adding default property: spark.ssl.keyStore=.keystore >> >> Adding default property: spark.ssl.enabled=false >> >> Adding default property: spark.ssl.trustStore=.truststore >> >> Adding default property: >> spark.cassandra.auth.conf.factory=com.datastax.bdp.spark.DseAuthConfFactory >> >> Adding default property: spark.ssl.keyPassword=cassandra >> >> Adding default property: spark.ssl.keyStorePassword=cassandra >> >> Adding default property: spark.ssl.protocol=TLS >> >> Adding default property: spark.ssl.useNodeLocalConf=true >> >> Adding default property: spark.ssl.trustStorePassword=cassandra >> >> Adding default property: >> spark.ssl.enabledAlgorithms=TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA >> >> Parsed arguments: >> >> master local >> >> deployMode null >> >> executorMemory null >> >> executorCores null >> >> totalExecutorCores null >> >> propertiesFile /etc/dse/spark/spark-defaults.conf >> >> driverMemory 512M >> >> driverCores null >> >> driverExtraClassPath null >> >> driverExtraLibraryPath null >> >> driverExtraJavaOptions -Dcassandra.username=missingmerch >> -Dcassandra.password=STMbrjrlb -XX:MaxPermSize=256M >> >> supervise false >> >> queue null >> >> numExecutors null >> >> files null >> >> pyFiles null >> >> archives null >> >> mainClass HelloWorld >> >> primaryResource file:/home/missingmerch/etl-0.0.1-SNAPSHOT.jar >> >> name HelloWorld >> >> childArgs [--jars >> file:/home/missingmerch/postgresql-9.4-1201.jdbc41.jar >> file:/home/missingmerch/dse.jar >> file:/home/missingmerch/postgresql-9.4-1201.jdbc41.jar] >> >> jars null >> >> verbose true >> >> >> >> Spark properties used, including those specified through >> >> --conf and those from the properties file >> /etc/dse/spark/spark-defaults.conf: >> >> spark.cassandra.connection.factory -> >> com.datastax.bdp.spark.DseCassandraConnectionFactory >> >> spark.ssl.useNodeLocalConf -> true >> >> spark.ssl.enabled -> false >> >> spark.executor.extraJavaOptions -> -XX:MaxPermSize=256M >> >> spark.ssl.keyStore -> .keystore >> >> spark.ssl.trustStore -> .truststore >> >> spark.ssl.trustStorePassword -> cassandra >> >> spark.ssl.enabledAlgorithms -> >> TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA >> >> spark.cassandra.auth.conf.factory -> >> com.datastax.bdp.spark.DseAuthConfFactory >> >> spark.ssl.protocol -> TLS >> >> spark.ssl.keyPassword -> cassandra >> >> spark.ssl.keyStorePassword -> cassandra >> >> >> >> >> >> Main class: >> >> HelloWorld >> >> Arguments: >> >> --jars >> >> file:/home/missingmerch/postgresql-9.4-1201.jdbc41.jar >> >> file:/home/missingmerch/dse.jar >> >> file:/home/missingmerch/postgresql-9.4-1201.jdbc41.jar >> >> System properties: >> >> spark.cassandra.connection.factory -> >> com.datastax.bdp.spark.DseCassandraConnectionFactory >> >> spark.driver.memory -> 512M >> >> spark.ssl.useNodeLocalConf -> true >> >> spark.ssl.enabled -> false >> >> SPARK_SUBMIT -> true >> >> spark.executor.extraJavaOptions -> -XX:MaxPermSize=256M >> >> spark.app.name -> HelloWorld >> >> spark.ssl.enabledAlgorithms -> >> TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA >> >> spark.ssl.trustStorePassword -> cassandra >> >> spark.driver.extraJavaOptions -> -Dcassandra.username=missingmerch >> -Dcassandra.password=STMbrjrlb -XX:MaxPermSize=256M >> >> spark.ssl.keyStore -> .keystore >> >> spark.ssl.trustStore -> .truststore >> >> spark.jars -> file:/home/missingmerch/etl-0.0.1-SNAPSHOT.jar >> >> spark.cassandra.auth.conf.factory -> >> com.datastax.bdp.spark.DseAuthConfFactory >> >> spark.master -> local >> >> spark.ssl.protocol -> TLS >> >> spark.ssl.keyPassword -> cassandra >> >> spark.ssl.keyStorePassword -> cassandra >> >> Classpath elements: >> >> file:/home/missingmerch/etl-0.0.1-SNAPSHOT.jar >> >> >> >> >> >> WARN 2015-08-11 08:23:25 org.apache.spark.util.Utils: Service 'SparkUI' >> could not bind on port 4040. Attempting port 4041. >> >> Exception in thread "main" java.lang.ClassNotFoundException: >> org.postgresql.Driver >> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:366) >> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:355) >> >> at java.security.AccessController.doPrivileged(Native Method) >> >> at java.net.URLClassLoader.findClass(URLClassLoader.java:354) >> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:425) >> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:358) >> >> at java.lang.Class.forName0(Native Method) >> >> at java.lang.Class.forName(Class.java:190) >> >> at HelloWorld$.main(HelloWorld.scala:26) >> >> at HelloWorld.main(HelloWorld.scala) >> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >> at java.lang.reflect.Method.invoke(Method.java:606) >> >> at >> org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358) >> >> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75) >> >> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >> >> >> Regards, >> >> Satish Chandra J >> >> On Tue, Aug 11, 2015 at 6:15 PM, Javier Domingo Cansino < >> javier.domi...@fon.com> wrote: >> >>> use --verbose, it might give you some insights on what0s happening,.... >>> >>> [image: Fon] <http://www.fon.com/>Javier Domingo CansinoResearch & >>> Development Engineer+34 946545847Skype: javier.domingo.fonAll >>> information in this email is confidential >>> <http://corp.fon.com/legal/email-disclaimer> >>> >>> On Tue, Aug 11, 2015 at 2:44 PM, satish chandra j < >>> jsatishchan...@gmail.com> wrote: >>> >>>> HI , >>>> I have used --jars option as well, please find the command below >>>> >>>> *Command:* >>>> >>>> dse spark-submit --master spark://10.246.43.15:7077 --class HelloWorld >>>> *--jars* ///home/missingmerch/postgresql-9.4-1201.jdbc41.jar >>>> ///home/missingmerch/dse.jar >>>> ///home/missingmerch/spark-cassandra-connector-java_2.10-1.1.1.jar >>>> ///home/missingmerch/etl-0.0.1-SNAPSHOT.jar >>>> >>>> Regards, >>>> Satish >>>> >>>> On Tue, Aug 11, 2015 at 4:08 PM, Javier Domingo Cansino < >>>> javier.domi...@fon.com> wrote: >>>> >>>>> I have no real idea (not java user), but have you tried with the >>>>> --jars option? >>>>> >>>>> >>>>> http://spark.apache.org/docs/latest/submitting-applications.html#advanced-dependency-management >>>>> >>>>> AFAIK, you are currently submitting the jar names as arguments to the >>>>> called Class instead of the jars themselves >>>>> >>>>> [image: Fon] <http://www.fon.com/>Javier Domingo CansinoResearch & >>>>> Development Engineer+34 946545847Skype: javier.domingo.fonAll >>>>> information in this email is confidential >>>>> <http://corp.fon.com/legal/email-disclaimer> >>>>> >>>>> On Tue, Aug 11, 2015 at 12:29 PM, satish chandra j < >>>>> jsatishchan...@gmail.com> wrote: >>>>> >>>>>> >>>>>> *HI,* >>>>>> >>>>>> Please let me know if i am missing anything in the command below >>>>>> >>>>>> *Command:* >>>>>> >>>>>> dse spark-submit --master spark://10.246.43.15:7077 --class >>>>>> HelloWorld --jars ///home/missingmerch/postgresql-9.4-1201.jdbc41.jar >>>>>> ///home/missingmerch/dse.jar >>>>>> ///home/missingmerch/spark-cassandra-connector-java_2.10-1.1.1.jar >>>>>> ///home/missingmerch/etl-0.0.1-SNAPSHOT.jar >>>>>> >>>>>> >>>>>> *Error:* >>>>>> >>>>>> *java.lang.ClassNotFoundException: HelloWorld* >>>>>> >>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366) >>>>>> >>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355) >>>>>> >>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>> >>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354) >>>>>> >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425) >>>>>> >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358) >>>>>> >>>>>> at java.lang.Class.forName0(Native Method) >>>>>> >>>>>> at java.lang.Class.forName(Class.java:270) >>>>>> >>>>>> at >>>>>> org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:342) >>>>>> >>>>>> at >>>>>> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75) >>>>>> >>>>>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >>>>>> >>>>>> >>>>>> I understand the way I am giving multiple jar file paths in the >>>>>> command is an issue, please provide an appropriate format for providing >>>>>> multiple jars in the command >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Thanks for support >>>>>> >>>>>> >>>>>> Satish Chandra >>>>>> >>>>> >>>>> >>>> >>> >> >
