I don't know if DSE changed spark-submit, but you have to use a
comma-separated list of jars to --jars. It probably looked for HelloWorld
in the second one, the dse.jar file. Do this:

dse spark-submit --master spark://10.246.43.15:7077 --class HelloWorld
--jars /home/missingmerch/
postgresql-9.4-1201.jdbc41.jar,/home/missingmerch/dse.jar,/home/missingmerch/spark-
cassandra-connector-java_2.10-1.1.1.jar /home/missingmerch/etl-0.0.
1-SNAPSHOT.jar

I also removed the extra "//". Or put "file:" in front of them so they are
proper URLs. Note the "snapshot" jar isn't in the --jars list. I assume
that's where HelloWorld is found. Confusing, yes it is...

dean

Dean Wampler, Ph.D.
Author: Programming Scala, 2nd Edition
<http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
Typesafe <http://typesafe.com>
@deanwampler <http://twitter.com/deanwampler>
http://polyglotprogramming.com

On Mon, Aug 10, 2015 at 8:23 AM, satish chandra j <jsatishchan...@gmail.com>
wrote:

> Hi,
> Thanks for quick input, now I am getting class not found error
>
> *Command:*
>
> dse spark-submit --master spark://10.246.43.15:7077 --class HelloWorld
> --jars ///home/missingmerch/postgresql-9.4-1201.jdbc41.jar
> ///home/missingmerch/dse.jar
> ///home/missingmerch/spark-cassandra-connector-java_2.10-1.1.1.jar
> ///home/missingmerch/etl-0.0.1-SNAPSHOT.jar
>
>
> *Error:*
>
> java.lang.ClassNotFoundException: HelloWorld
>
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>
>         at java.security.AccessController.doPrivileged(Native Method)
>
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>
>         at java.lang.Class.forName0(Native Method)
>
>         at java.lang.Class.forName(Class.java:270)
>
>         at
> org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:342)
>
>         at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
>
>         at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
>
> Previously I could fix the issue by changing the order of arguments
> passing in DSE command line interface but now I am not sure why the issue
> again
>
> Please let me know if still I am missing anything in my Command as
> mentioned above(as insisted I have added dse.jar and
> spark-cassandra-connector-java_2.10.1.1.1.jar)
>
>
> Thanks for support
>
>
> Satish Chandra
>
> On Mon, Aug 10, 2015 at 6:19 PM, Dean Wampler <deanwamp...@gmail.com>
> wrote:
>
>> Add the other Cassandra dependencies (dse.jar,
>> spark-cassandra-connect-java_2.10) to your --jars argument on the command
>> line.
>>
>> Dean Wampler, Ph.D.
>> Author: Programming Scala, 2nd Edition
>> <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
>> Typesafe <http://typesafe.com>
>> @deanwampler <http://twitter.com/deanwampler>
>> http://polyglotprogramming.com
>>
>> On Mon, Aug 10, 2015 at 7:44 AM, satish chandra j <
>> jsatishchan...@gmail.com> wrote:
>>
>>> HI All,
>>> Please help me to fix Spark Cassandra Connector issue, find the details
>>> below
>>>
>>> *Command:*
>>>
>>> dse spark-submit --master spark://10.246.43.15:7077 --class HelloWorld
>>> --jars ///home/missingmerch/postgresql-9.4-1201.jdbc41.jar
>>> ///home/missingmerch/etl-0.0.1-SNAPSHOT.jar
>>>
>>>
>>> *Error:*
>>>
>>>
>>> WARN  2015-08-10 06:33:35 org.apache.spark.util.Utils: Service 'SparkUI'
>>> could not bind on port 4040. Attempting port 4041.
>>>
>>> Exception in thread "main" java.lang.NoSuchMethodError:
>>> com.datastax.spark.connector.package$.toRDDFunctions(Lorg/apache/spark/rdd/RDD;Lscala/reflect/ClassTag;)Lcom/datastax/spark/connector/RDDFunctions;
>>>
>>>         at HelloWorld$.main(HelloWorld.scala:29)
>>>
>>>         at HelloWorld.main(HelloWorld.scala)
>>>
>>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>
>>>         at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>
>>>         at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>
>>>         at java.lang.reflect.Method.invoke(Method.java:606)
>>>
>>>         at
>>> org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358)
>>>
>>>         at
>>> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
>>>
>>>         at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>>
>>>
>>> *Code:*
>>>
>>> *import* *org.apache*.spark.SparkContext
>>>
>>> *import* *org.apache*.spark.SparkContext._
>>>
>>> *import* *org.apache*.spark.SparkConf
>>>
>>> *import* *org.apache*.spark.rdd.JdbcRDD
>>>
>>> *import* *com.datastax*.spark.connector._
>>>
>>> *import* com.datastax.spark.connector.cql.CassandraConnector
>>>
>>> *import* com.datastax.bdp.spark.DseSparkConfHelper._
>>>
>>> *import* java.sql.{Connection, DriverManager, ResultSet,
>>> PreparedStatement, SQLException, Statement}
>>>
>>> *object* HelloWorld {
>>>
>>>     *def* main(args: Array[String]) {
>>>
>>>       *def* createSparkContext() = {
>>>
>>>        *val** myJar = 
>>> *getClass.getProtectionDomain.getCodeSource.getLocation.getPath
>>>
>>>
>>>        *val* conf = *new* SparkConf().set(
>>> "spark.cassandra.connection.host", "10.246.43.15")
>>>
>>>        .setAppName("First Spark App")
>>>
>>>        .setMaster("local")
>>>
>>> *       .s*etJars(Array(myJar))
>>>
>>>        .set("cassandra.username", "username")
>>>
>>>        .set("cassandra.password", "password")
>>>
>>>        .forDse
>>>
>>>    *new* SparkContext(conf)
>>>
>>> }
>>>
>>>
>>>
>>>       *val* sc = createSparkContext()
>>>
>>>       *val* user="hkonak0"
>>>
>>>       *val** pass=*"Winter18"
>>>
>>>       Class.forName("org.postgresql.Driver").newInstance
>>>
>>>       *val* url = "jdbc:postgresql://gptester:5432/db_test"
>>>
>>>       *val* myRDD27 = *new* JdbcRDD( sc, ()=>
>>> DriverManager.getConnection(url,user,pass),"select * from
>>> wmax_vmax.arm_typ_txt LIMIT ? OFFSET ?",5,0,1,(r: ResultSet) =>
>>> {(r.getInt("alarm_type_code"),r.getString("language_code"),r.getString(
>>> "alrm_type_cd_desc"))})
>>>
>>>       myRDD27.saveToCassandra("keyspace","arm_typ_txt",SomeColumns(
>>> "alarm_type_code","language_code","alrm_type_cd_desc"))
>>>
>>>       println(myRDD27.count())
>>>
>>>       println(myRDD27.first)
>>>
>>>       sc.stop()
>>>
>>>       sys.exit()
>>>
>>>
>>>
>>>     }
>>>
>>>   }
>>>
>>>
>>>
>>> *POM XML:*
>>>
>>>
>>> <dependencies>
>>>
>>>               <dependency>
>>>
>>>                      <groupId>org.apache.spark</groupId>
>>>
>>>                      <artifactId>spark-core_2.10</artifactId>
>>>
>>>                      <version>1.2.2</version>
>>>
>>>               </dependency>
>>>
>>>               <dependency>
>>>
>>>                      <groupId>org.apache.hadoop</groupId>
>>>
>>>                      <artifactId>*hadoop*-client</artifactId>
>>>
>>>                      <version>1.2.1</version>
>>>
>>>               </dependency>
>>>
>>>               <dependency>
>>>
>>>                      <groupId>org.scala-*lang*</groupId>
>>>
>>>                      <artifactId>*scala*-library</artifactId>
>>>
>>>                      <version>2.10.5</version>
>>>
>>>               </dependency>
>>>
>>>               <dependency>
>>>
>>>                      <groupId>*junit*</groupId>
>>>
>>>                      <artifactId>*junit*</artifactId>
>>>
>>>                      <version>3.8.1</version>
>>>
>>>                      <scope>test</scope>
>>>
>>>               </dependency>
>>>
>>>               <dependency>
>>>
>>>                      <groupId>com.datastax.dse</groupId>
>>>
>>>                      <artifactId>*dse*</artifactId>
>>>
>>>                      <version>4.7.2</version>
>>>
>>>               <scope>system</scope>
>>>
>>>                      <systemPath>C:\workspace\*etl*\*lib*\dse.jar</
>>> systemPath>
>>>
>>>        </dependency>
>>>
>>>               <dependency>
>>>
>>>               <groupId>com.datastax.spark</groupId>
>>>
>>>               <artifactId>spark-*cassandra*-connector-java_2.10</
>>> artifactId>
>>>
>>>               <version>1.1.1</version>
>>>
>>>        </dependency>
>>>
>>>        </dependencies>
>>>
>>>
>>> Please let me know if any further details required to analyze the issue
>>>
>>>
>>> Regards,
>>>
>>> Satish Chandra
>>>
>>
>>
>

Reply via email to