HI All, I have tried Commands as mentioned below but still it is errors dse spark-submit --master spark://10.246.43.15:7077 --class HelloWorld --jars /home/missingmerch/ postgresql-9.4-1201.jdbc41.jar,/home/missingmerch/dse.jar,/home/missingmerch/spark- cassandra-connector-java_2.10-1.1.1.jar /home/missingmerch/etl-0.0. 1-SNAPSHOT.jar
dse spark-submit --master spark://10.246.43.15:7077 --class HelloWorld --jars /home/missingmerch/ postgresql-9.4-1201.jdbc41.jar,/home/missingmerch/dse.jar,/home/missingmerch/spark- cassandra-connector-java_2.10-1.1.1.jar,/home/missingmerch/etl-0.0. 1-SNAPSHOT.jar I understand only problem with the way I provide list of jar file in the command, if anybody using Datastax Enterprise could please provide thier inputs to get this issue resolved Thanks for your support Satish Chandra On Mon, Aug 10, 2015 at 7:16 PM, Dean Wampler <deanwamp...@gmail.com> wrote: > I don't know if DSE changed spark-submit, but you have to use a > comma-separated list of jars to --jars. It probably looked for HelloWorld > in the second one, the dse.jar file. Do this: > > dse spark-submit --master spark://10.246.43.15:7077 --class HelloWorld > --jars /home/missingmerch/ > postgresql-9.4-1201.jdbc41.jar,/home/missingmerch/dse.jar,/home/missingmerch/spark- > cassandra-connector-java_2.10-1.1.1.jar /home/missingmerch/etl-0.0. > 1-SNAPSHOT.jar > > I also removed the extra "//". Or put "file:" in front of them so they are > proper URLs. Note the "snapshot" jar isn't in the --jars list. I assume > that's where HelloWorld is found. Confusing, yes it is... > > dean > > Dean Wampler, Ph.D. > Author: Programming Scala, 2nd Edition > <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly) > Typesafe <http://typesafe.com> > @deanwampler <http://twitter.com/deanwampler> > http://polyglotprogramming.com > > On Mon, Aug 10, 2015 at 8:23 AM, satish chandra j < > jsatishchan...@gmail.com> wrote: > >> Hi, >> Thanks for quick input, now I am getting class not found error >> >> *Command:* >> >> dse spark-submit --master spark://10.246.43.15:7077 --class HelloWorld >> --jars ///home/missingmerch/postgresql-9.4-1201.jdbc41.jar >> ///home/missingmerch/dse.jar >> ///home/missingmerch/spark-cassandra-connector-java_2.10-1.1.1.jar >> ///home/missingmerch/etl-0.0.1-SNAPSHOT.jar >> >> >> *Error:* >> >> java.lang.ClassNotFoundException: HelloWorld >> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:366) >> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:355) >> >> at java.security.AccessController.doPrivileged(Native Method) >> >> at java.net.URLClassLoader.findClass(URLClassLoader.java:354) >> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:425) >> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:358) >> >> at java.lang.Class.forName0(Native Method) >> >> at java.lang.Class.forName(Class.java:270) >> >> at >> org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:342) >> >> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75) >> >> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >> >> >> Previously I could fix the issue by changing the order of arguments >> passing in DSE command line interface but now I am not sure why the issue >> again >> >> Please let me know if still I am missing anything in my Command as >> mentioned above(as insisted I have added dse.jar and >> spark-cassandra-connector-java_2.10.1.1.1.jar) >> >> >> Thanks for support >> >> >> Satish Chandra >> >> On Mon, Aug 10, 2015 at 6:19 PM, Dean Wampler <deanwamp...@gmail.com> >> wrote: >> >>> Add the other Cassandra dependencies (dse.jar, >>> spark-cassandra-connect-java_2.10) to your --jars argument on the command >>> line. >>> >>> Dean Wampler, Ph.D. >>> Author: Programming Scala, 2nd Edition >>> <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly) >>> Typesafe <http://typesafe.com> >>> @deanwampler <http://twitter.com/deanwampler> >>> http://polyglotprogramming.com >>> >>> On Mon, Aug 10, 2015 at 7:44 AM, satish chandra j < >>> jsatishchan...@gmail.com> wrote: >>> >>>> HI All, >>>> Please help me to fix Spark Cassandra Connector issue, find the details >>>> below >>>> >>>> *Command:* >>>> >>>> dse spark-submit --master spark://10.246.43.15:7077 --class HelloWorld >>>> --jars ///home/missingmerch/postgresql-9.4-1201.jdbc41.jar >>>> ///home/missingmerch/etl-0.0.1-SNAPSHOT.jar >>>> >>>> >>>> *Error:* >>>> >>>> >>>> WARN 2015-08-10 06:33:35 org.apache.spark.util.Utils: Service >>>> 'SparkUI' could not bind on port 4040. Attempting port 4041. >>>> >>>> Exception in thread "main" java.lang.NoSuchMethodError: >>>> com.datastax.spark.connector.package$.toRDDFunctions(Lorg/apache/spark/rdd/RDD;Lscala/reflect/ClassTag;)Lcom/datastax/spark/connector/RDDFunctions; >>>> >>>> at HelloWorld$.main(HelloWorld.scala:29) >>>> >>>> at HelloWorld.main(HelloWorld.scala) >>>> >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>> >>>> at >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>>> >>>> at >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> >>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>> >>>> at >>>> org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358) >>>> >>>> at >>>> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75) >>>> >>>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >>>> >>>> >>>> *Code:* >>>> >>>> *import* *org.apache*.spark.SparkContext >>>> >>>> *import* *org.apache*.spark.SparkContext._ >>>> >>>> *import* *org.apache*.spark.SparkConf >>>> >>>> *import* *org.apache*.spark.rdd.JdbcRDD >>>> >>>> *import* *com.datastax*.spark.connector._ >>>> >>>> *import* com.datastax.spark.connector.cql.CassandraConnector >>>> >>>> *import* com.datastax.bdp.spark.DseSparkConfHelper._ >>>> >>>> *import* java.sql.{Connection, DriverManager, ResultSet, >>>> PreparedStatement, SQLException, Statement} >>>> >>>> *object* HelloWorld { >>>> >>>> *def* main(args: Array[String]) { >>>> >>>> *def* createSparkContext() = { >>>> >>>> *val** myJar = >>>> *getClass.getProtectionDomain.getCodeSource.getLocation.getPath >>>> >>>> >>>> *val* conf = *new* SparkConf().set( >>>> "spark.cassandra.connection.host", "10.246.43.15") >>>> >>>> .setAppName("First Spark App") >>>> >>>> .setMaster("local") >>>> >>>> * .s*etJars(Array(myJar)) >>>> >>>> .set("cassandra.username", "username") >>>> >>>> .set("cassandra.password", "password") >>>> >>>> .forDse >>>> >>>> *new* SparkContext(conf) >>>> >>>> } >>>> >>>> >>>> >>>> *val* sc = createSparkContext() >>>> >>>> *val* user="hkonak0" >>>> >>>> *val** pass=*"Winter18" >>>> >>>> Class.forName("org.postgresql.Driver").newInstance >>>> >>>> *val* url = "jdbc:postgresql://gptester:5432/db_test" >>>> >>>> *val* myRDD27 = *new* JdbcRDD( sc, ()=> >>>> DriverManager.getConnection(url,user,pass),"select * from >>>> wmax_vmax.arm_typ_txt LIMIT ? OFFSET ?",5,0,1,(r: ResultSet) => >>>> {(r.getInt("alarm_type_code"),r.getString("language_code"),r.getString( >>>> "alrm_type_cd_desc"))}) >>>> >>>> myRDD27.saveToCassandra("keyspace","arm_typ_txt",SomeColumns( >>>> "alarm_type_code","language_code","alrm_type_cd_desc")) >>>> >>>> println(myRDD27.count()) >>>> >>>> println(myRDD27.first) >>>> >>>> sc.stop() >>>> >>>> sys.exit() >>>> >>>> >>>> >>>> } >>>> >>>> } >>>> >>>> >>>> >>>> *POM XML:* >>>> >>>> >>>> <dependencies> >>>> >>>> <dependency> >>>> >>>> <groupId>org.apache.spark</groupId> >>>> >>>> <artifactId>spark-core_2.10</artifactId> >>>> >>>> <version>1.2.2</version> >>>> >>>> </dependency> >>>> >>>> <dependency> >>>> >>>> <groupId>org.apache.hadoop</groupId> >>>> >>>> <artifactId>*hadoop*-client</artifactId> >>>> >>>> <version>1.2.1</version> >>>> >>>> </dependency> >>>> >>>> <dependency> >>>> >>>> <groupId>org.scala-*lang*</groupId> >>>> >>>> <artifactId>*scala*-library</artifactId> >>>> >>>> <version>2.10.5</version> >>>> >>>> </dependency> >>>> >>>> <dependency> >>>> >>>> <groupId>*junit*</groupId> >>>> >>>> <artifactId>*junit*</artifactId> >>>> >>>> <version>3.8.1</version> >>>> >>>> <scope>test</scope> >>>> >>>> </dependency> >>>> >>>> <dependency> >>>> >>>> <groupId>com.datastax.dse</groupId> >>>> >>>> <artifactId>*dse*</artifactId> >>>> >>>> <version>4.7.2</version> >>>> >>>> <scope>system</scope> >>>> >>>> <systemPath>C:\workspace\*etl*\*lib*\dse.jar</ >>>> systemPath> >>>> >>>> </dependency> >>>> >>>> <dependency> >>>> >>>> <groupId>com.datastax.spark</groupId> >>>> >>>> <artifactId>spark-*cassandra*-connector-java_2.10</ >>>> artifactId> >>>> >>>> <version>1.1.1</version> >>>> >>>> </dependency> >>>> >>>> </dependencies> >>>> >>>> >>>> Please let me know if any further details required to analyze the issue >>>> >>>> >>>> Regards, >>>> >>>> Satish Chandra >>>> >>> >>> >> >
