HI,
Can we apply *saveToCassandra method to a JdbcRDD *
Code:
*Code:*
*import* *org.apache*.spark.SparkContext
*import* *org.apache*.spark.SparkContext._
*import* *org.apache*.spark.SparkConf
*import* *org.apache*.spark.rdd.JdbcRDD
*import* *com.datastax*.spark.connector._
*import* com.datastax.spark.connector.cql.CassandraConnector
*import* com.datastax.bdp.spark.DseSparkConfHelper._
*import* java.sql.{Connection, DriverManager, ResultSet, PreparedStatement,
SQLException, Statement}
*object* HelloWorld {
*def* main(args: Array[String]) {
*def* createSparkContext() = {
*val* conf = *new* SparkConf().set("spark.cassandra.connection.host"
, "10.246.43.15")
.setAppName("First Spark App")
.setMaster("local")
.set("cassandra.username", "username")
.set("cassandra.password", "password")
.forDse
*new* SparkContext(conf)
}
*val* sc = createSparkContext()
*val* user="hkonak0"
*val** pass=*"Winter18"
Class.forName("org.postgresql.Driver").newInstance
*val* url = "jdbc:postgresql://gptester:5432/db_test"
*val* myRDD27 = *new* JdbcRDD( sc, ()=>
DriverManager.getConnection(url,user,pass),"select * from
wmax_vmax.arm_typ_txt LIMIT ? OFFSET ?",5,0,1,(r: ResultSet) => {(r.getInt(
"alarm_type_code"),r.getString("language_code"),r.getString(
"alrm_type_cd_desc"))})
* myRDD27.saveToCassandra(*
*"keyspace","arm_typ_txt",SomeColumns("alarm_type_code","language_code","alrm_type_cd_desc"))*
println(myRDD27.count())
println(myRDD27.first)
sc.stop()
sys.exit()
}
}
I am getting the error as below:
*Exception in thread "main" java.lang.NoSuchMethodError:
com.datastax.spark.connector.package$.toRDDFunctions(Lorg/apache/spark/rdd/RDD;Lscala/reflect/ClassTag;)Lcom/datastax/spark/connector/RDDFunctions*
Regards,
Satish Chandra
On Tue, Aug 11, 2015 at 9:23 AM, satish chandra j <jsatishchan...@gmail.com>
wrote:
> HI All,
> I have tried Commands as mentioned below but still it is errors
>
> dse spark-submit --master spark://10.246.43.15:7077 --class HelloWorld
> --jars /home/missingmerch/
> postgresql-9.4-1201.jdbc41.jar,/home/missingmerch/dse.jar,/home/missingmerch/spark-
> cassandra-connector-java_2.10-1.1.1.jar /home/missingmerch/etl-0.0.
> 1-SNAPSHOT.jar
>
> dse spark-submit --master spark://10.246.43.15:7077 --class HelloWorld
> --jars /home/missingmerch/
> postgresql-9.4-1201.jdbc41.jar,/home/missingmerch/dse.jar,/home/missingmerch/spark-
> cassandra-connector-java_2.10-1.1.1.jar,/home/missingmerch/etl-0.0.
> 1-SNAPSHOT.jar
>
> I understand only problem with the way I provide list of jar file in the
> command, if anybody using Datastax Enterprise could please provide thier
> inputs to get this issue resolved
>
> Thanks for your support
>
> Satish Chandra
>
>
> On Mon, Aug 10, 2015 at 7:16 PM, Dean Wampler <deanwamp...@gmail.com>
> wrote:
>
>> I don't know if DSE changed spark-submit, but you have to use a
>> comma-separated list of jars to --jars. It probably looked for HelloWorld
>> in the second one, the dse.jar file. Do this:
>>
>> dse spark-submit --master spark://10.246.43.15:7077 --class HelloWorld
>> --jars /home/missingmerch/
>> postgresql-9.4-1201.jdbc41.jar,/home/missingmerch/dse.jar,/home/missingmerch/spark-
>> cassandra-connector-java_2.10-1.1.1.jar /home/missingmerch/etl-0.0.
>> 1-SNAPSHOT.jar
>>
>> I also removed the extra "//". Or put "file:" in front of them so they
>> are proper URLs. Note the "snapshot" jar isn't in the --jars list. I assume
>> that's where HelloWorld is found. Confusing, yes it is...
>>
>> dean
>>
>> Dean Wampler, Ph.D.
>> Author: Programming Scala, 2nd Edition
>> <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
>> Typesafe <http://typesafe.com>
>> @deanwampler <http://twitter.com/deanwampler>
>> http://polyglotprogramming.com
>>
>> On Mon, Aug 10, 2015 at 8:23 AM, satish chandra j <
>> jsatishchan...@gmail.com> wrote:
>>
>>> Hi,
>>> Thanks for quick input, now I am getting class not found error
>>>
>>> *Command:*
>>>
>>> dse spark-submit --master spark://10.246.43.15:7077 --class HelloWorld
>>> --jars ///home/missingmerch/postgresql-9.4-1201.jdbc41.jar
>>> ///home/missingmerch/dse.jar
>>> ///home/missingmerch/spark-cassandra-connector-java_2.10-1.1.1.jar
>>> ///home/missingmerch/etl-0.0.1-SNAPSHOT.jar
>>>
>>>
>>> *Error:*
>>>
>>> java.lang.ClassNotFoundException: HelloWorld
>>>
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>>>
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>>>
>>> at java.security.AccessController.doPrivileged(Native Method)
>>>
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>>>
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>>>
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>>>
>>> at java.lang.Class.forName0(Native Method)
>>>
>>> at java.lang.Class.forName(Class.java:270)
>>>
>>> at
>>> org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:342)
>>>
>>> at
>>> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
>>>
>>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>>
>>>
>>> Previously I could fix the issue by changing the order of arguments
>>> passing in DSE command line interface but now I am not sure why the issue
>>> again
>>>
>>> Please let me know if still I am missing anything in my Command as
>>> mentioned above(as insisted I have added dse.jar and
>>> spark-cassandra-connector-java_2.10.1.1.1.jar)
>>>
>>>
>>> Thanks for support
>>>
>>>
>>> Satish Chandra
>>>
>>> On Mon, Aug 10, 2015 at 6:19 PM, Dean Wampler <deanwamp...@gmail.com>
>>> wrote:
>>>
>>>> Add the other Cassandra dependencies (dse.jar,
>>>> spark-cassandra-connect-java_2.10) to your --jars argument on the command
>>>> line.
>>>>
>>>> Dean Wampler, Ph.D.
>>>> Author: Programming Scala, 2nd Edition
>>>> <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
>>>> Typesafe <http://typesafe.com>
>>>> @deanwampler <http://twitter.com/deanwampler>
>>>> http://polyglotprogramming.com
>>>>
>>>> On Mon, Aug 10, 2015 at 7:44 AM, satish chandra j <
>>>> jsatishchan...@gmail.com> wrote:
>>>>
>>>>> HI All,
>>>>> Please help me to fix Spark Cassandra Connector issue, find the
>>>>> details below
>>>>>
>>>>> *Command:*
>>>>>
>>>>> dse spark-submit --master spark://10.246.43.15:7077 --class
>>>>> HelloWorld --jars ///home/missingmerch/postgresql-9.4-1201.jdbc41.jar
>>>>> ///home/missingmerch/etl-0.0.1-SNAPSHOT.jar
>>>>>
>>>>>
>>>>> *Error:*
>>>>>
>>>>>
>>>>> WARN 2015-08-10 06:33:35 org.apache.spark.util.Utils: Service
>>>>> 'SparkUI' could not bind on port 4040. Attempting port 4041.
>>>>>
>>>>> Exception in thread "main" java.lang.NoSuchMethodError:
>>>>> com.datastax.spark.connector.package$.toRDDFunctions(Lorg/apache/spark/rdd/RDD;Lscala/reflect/ClassTag;)Lcom/datastax/spark/connector/RDDFunctions;
>>>>>
>>>>> at HelloWorld$.main(HelloWorld.scala:29)
>>>>>
>>>>> at HelloWorld.main(HelloWorld.scala)
>>>>>
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>
>>>>> at
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>>
>>>>> at
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>
>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>
>>>>> at
>>>>> org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358)
>>>>>
>>>>> at
>>>>> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
>>>>>
>>>>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>>>>
>>>>>
>>>>> *Code:*
>>>>>
>>>>> *import* *org.apache*.spark.SparkContext
>>>>>
>>>>> *import* *org.apache*.spark.SparkContext._
>>>>>
>>>>> *import* *org.apache*.spark.SparkConf
>>>>>
>>>>> *import* *org.apache*.spark.rdd.JdbcRDD
>>>>>
>>>>> *import* *com.datastax*.spark.connector._
>>>>>
>>>>> *import* com.datastax.spark.connector.cql.CassandraConnector
>>>>>
>>>>> *import* com.datastax.bdp.spark.DseSparkConfHelper._
>>>>>
>>>>> *import* java.sql.{Connection, DriverManager, ResultSet,
>>>>> PreparedStatement, SQLException, Statement}
>>>>>
>>>>> *object* HelloWorld {
>>>>>
>>>>> *def* main(args: Array[String]) {
>>>>>
>>>>> *def* createSparkContext() = {
>>>>>
>>>>> *val** myJar =
>>>>> *getClass.getProtectionDomain.getCodeSource.getLocation.getPath
>>>>>
>>>>>
>>>>> *val* conf = *new* SparkConf().set(
>>>>> "spark.cassandra.connection.host", "10.246.43.15")
>>>>>
>>>>> .setAppName("First Spark App")
>>>>>
>>>>> .setMaster("local")
>>>>>
>>>>> * .s*etJars(Array(myJar))
>>>>>
>>>>> .set("cassandra.username", "username")
>>>>>
>>>>> .set("cassandra.password", "password")
>>>>>
>>>>> .forDse
>>>>>
>>>>> *new* SparkContext(conf)
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> *val* sc = createSparkContext()
>>>>>
>>>>> *val* user="hkonak0"
>>>>>
>>>>> *val** pass=*"Winter18"
>>>>>
>>>>> Class.forName("org.postgresql.Driver").newInstance
>>>>>
>>>>> *val* url = "jdbc:postgresql://gptester:5432/db_test"
>>>>>
>>>>> *val* myRDD27 = *new* JdbcRDD( sc, ()=>
>>>>> DriverManager.getConnection(url,user,pass),"select * from
>>>>> wmax_vmax.arm_typ_txt LIMIT ? OFFSET ?",5,0,1,(r: ResultSet) =>
>>>>> {(r.getInt("alarm_type_code"),r.getString("language_code"
>>>>> ),r.getString("alrm_type_cd_desc"))})
>>>>>
>>>>> myRDD27.saveToCassandra("keyspace","arm_typ_txt",SomeColumns(
>>>>> "alarm_type_code","language_code","alrm_type_cd_desc"))
>>>>>
>>>>> println(myRDD27.count())
>>>>>
>>>>> println(myRDD27.first)
>>>>>
>>>>> sc.stop()
>>>>>
>>>>> sys.exit()
>>>>>
>>>>>
>>>>>
>>>>> }
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> *POM XML:*
>>>>>
>>>>>
>>>>> <dependencies>
>>>>>
>>>>> <dependency>
>>>>>
>>>>> <groupId>org.apache.spark</groupId>
>>>>>
>>>>> <artifactId>spark-core_2.10</artifactId>
>>>>>
>>>>> <version>1.2.2</version>
>>>>>
>>>>> </dependency>
>>>>>
>>>>> <dependency>
>>>>>
>>>>> <groupId>org.apache.hadoop</groupId>
>>>>>
>>>>> <artifactId>*hadoop*-client</artifactId>
>>>>>
>>>>> <version>1.2.1</version>
>>>>>
>>>>> </dependency>
>>>>>
>>>>> <dependency>
>>>>>
>>>>> <groupId>org.scala-*lang*</groupId>
>>>>>
>>>>> <artifactId>*scala*-library</artifactId>
>>>>>
>>>>> <version>2.10.5</version>
>>>>>
>>>>> </dependency>
>>>>>
>>>>> <dependency>
>>>>>
>>>>> <groupId>*junit*</groupId>
>>>>>
>>>>> <artifactId>*junit*</artifactId>
>>>>>
>>>>> <version>3.8.1</version>
>>>>>
>>>>> <scope>test</scope>
>>>>>
>>>>> </dependency>
>>>>>
>>>>> <dependency>
>>>>>
>>>>> <groupId>com.datastax.dse</groupId>
>>>>>
>>>>> <artifactId>*dse*</artifactId>
>>>>>
>>>>> <version>4.7.2</version>
>>>>>
>>>>> <scope>system</scope>
>>>>>
>>>>> <systemPath>C:\workspace\*etl*\*lib*\dse.jar</
>>>>> systemPath>
>>>>>
>>>>> </dependency>
>>>>>
>>>>> <dependency>
>>>>>
>>>>> <groupId>com.datastax.spark</groupId>
>>>>>
>>>>> <artifactId>spark-*cassandra*-connector-java_2.10</
>>>>> artifactId>
>>>>>
>>>>> <version>1.1.1</version>
>>>>>
>>>>> </dependency>
>>>>>
>>>>> </dependencies>
>>>>>
>>>>>
>>>>> Please let me know if any further details required to analyze the issue
>>>>>
>>>>>
>>>>> Regards,
>>>>>
>>>>> Satish Chandra
>>>>>
>>>>
>>>>
>>>
>>
>
