Re: Error - Spark STREAMING
Which version of spark are you using? On Tue, Sep 20, 2022, 1:57 PM Akash Vellukai wrote: > Hello, > > > py4j.protocol.Py4JJavaError: An error occurred while calling o80.load. : > java.lang.NoClassDefFoundError: > org/apache/spark/sql/internal/connector/SimpleTableProvider > > > May anyone help Me to solve this issue. > > > Thanks and regards > Akash >
Re: Error: Spark-streaming to Cassandra
I am curious why you use the 1.0.4 java artifact with the latest 1.1.0? This
might be your compilation problem - The older java version.
dependency
groupIdcom.datastax.spark/groupId
artifactIdspark-cassandra-connector_2.10/artifactId
version1.1.0/version
/dependency
dependency
groupIdcom.datastax.spark/groupId
artifactIdspark-cassandra-connector-java_2.10/artifactId
version1.0.4/version
/dependency
See:
- doc
https://github.com/datastax/spark-cassandra-connector/blob/master/doc/7_java_api.md
- mvn repo
http://mvnrepository.com/artifact/com.datastax.spark/spark-cassandra-connector-java_2.10/1.1.0
- Helena
@helenaedelson
On Dec 8, 2014, at 12:47 PM, m.sar...@accenture.com wrote:
Hi,
I am intending to save the streaming data from kafka into Cassandra,
using spark-streaming:
But there seems to be problem with line
javaFunctions(data).writerBuilder(testkeyspace, test_table,
mapToRow(TestTable.class)).saveToCassandra();
I am getting 2 errors.
the code, the error-log and POM.xml dependencies are listed below:
Please help me find the reason as to why is this happening.
public class SparkStream {
static int key=0;
public static void main(String args[]) throws Exception
{
if(args.length != 3)
{
System.out.println(SparkStream zookeeper_ip group_nm
topic1,topic2,...);
System.exit(1);
}
Logger.getLogger(org).setLevel(Level.OFF);
Logger.getLogger(akka).setLevel(Level.OFF);
MapString,Integer topicMap = new HashMapString,Integer();
String[] topic = args[2].split(,);
for(String t: topic)
{
topicMap.put(t, new Integer(3));
}
/* Connection to Spark */
SparkConf conf = new SparkConf();
JavaSparkContext sc = new JavaSparkContext(local[4],
SparkStream,conf);
JavaStreamingContext jssc = new JavaStreamingContext(sc, new
Duration(3000));
/* connection to cassandra */
/*conf.set(spark.cassandra.connection.host, 127.0.0.1:9042);
CassandraConnector connector = CassandraConnector.apply(sc.getConf());
Session session = connector.openSession();
session.execute(CREATE TABLE IF NOT EXISTS testkeyspace.test_table
(key INT PRIMARY KEY, value TEXT));
*/
/* Receive Kafka streaming inputs */
JavaPairReceiverInputDStreamString, String messages =
KafkaUtils.createStream(jssc, args[0], args[1], topicMap );
/* Create DStream */
JavaDStreamTestTable data = messages.map(new
FunctionTuple2String,String, TestTable ()
{
public TestTable call(Tuple2String, String message)
{
return new TestTable(new Integer(++key), message._2() );
}
}
);
/* Write to cassandra */
javaFunctions(data).writerBuilder(testkeyspace, test_table,
mapToRow(TestTable.class)).saveToCassandra();
// data.print();
jssc.start();
jssc.awaitTermination();
}
}
class TestTable implements Serializable
{
Integer key;
String value;
public TestTable() {}
public TestTable(Integer k, String v)
{
key=k;
value=v;
}
public Integer getKey(){
return key;
}
public void setKey(Integer k){
key=k;
}
public String getValue(){
return value;
}
public void setValue(String v){
value=v;
}
public String toString(){
return MessageFormat.format(TestTable'{'key={0},
value={1}'}', key, value);
}
}
The output log is:
[INFO] Compiling 1 source file to
/root/Documents/SparkStreamSample/target/classes
[INFO] 2 errors
[INFO] -
[ERROR] COMPILATION ERROR :
[INFO] -
[ERROR]
/root/Documents/SparkStreamSample/src/main/java/com/spark/SparkStream.java:[76,81]
cannot find symbol
symbol: method mapToRow(java.lang.Classcom.spark.TestTable)
location: class com.spark.SparkStream
[ERROR]
/root/Documents/SparkStreamSample/src/main/java/com/spark/SparkStream.java:[76,17]
no suitable method found for
javaFunctions(org.apache.spark.streaming.api.java.JavaDStreamcom.spark.TestTable)
method
com.datastax.spark.connector.CassandraJavaUtil.TjavaFunctions(org.apache.spark.streaming.api.java.JavaDStreamT,java.lang.ClassT)
is not applicable
(cannot infer type-variable(s) T
(actual and formal argument lists differ in length))
method
com.datastax.spark.connector.CassandraJavaUtil.TjavaFunctions(org.apache.spark.streaming.dstream.DStreamT,java.lang.ClassT)
is not applicable
(cannot infer type-variable(s) T
(actual and formal argument lists differ in length))
method
com.datastax.spark.connector.CassandraJavaUtil.TjavaFunctions(org.apache.spark.api.java.JavaRDDT,java.lang.ClassT)
is not applicable
Re: Error: Spark-streaming to Cassandra
This seems to be compilation errors. The second one seems to be that
you are using CassandraJavaUtil.javafunctions wrong. Look at the
documentation and set the parameter list correctly.
TD
On Mon, Dec 8, 2014 at 9:47 AM, m.sar...@accenture.com wrote:
Hi,
I am intending to save the streaming data from kafka into Cassandra,
using spark-streaming:
But there seems to be problem with line
javaFunctions(data).writerBuilder(testkeyspace, test_table,
mapToRow(TestTable.class)).saveToCassandra();
I am getting 2 errors.
the code, the error-log and POM.xml dependencies are listed below:
Please help me find the reason as to why is this happening.
public class SparkStream {
static int key=0;
public static void main(String args[]) throws Exception
{
if(args.length != 3)
{
System.out.println(SparkStream zookeeper_ip group_nm
topic1,topic2,...);
System.exit(1);
}
Logger.getLogger(org).setLevel(Level.OFF);
Logger.getLogger(akka).setLevel(Level.OFF);
MapString,Integer topicMap = new HashMapString,Integer();
String[] topic = args[2].split(,);
for(String t: topic)
{
topicMap.put(t, new Integer(3));
}
/* Connection to Spark */
SparkConf conf = new SparkConf();
JavaSparkContext sc = new JavaSparkContext(local[4],
SparkStream,conf);
JavaStreamingContext jssc = new JavaStreamingContext(sc, new
Duration(3000));
/* connection to cassandra */
/*conf.set(spark.cassandra.connection.host, 127.0.0.1:9042);
CassandraConnector connector = CassandraConnector.apply(sc.getConf());
Session session = connector.openSession();
session.execute(CREATE TABLE IF NOT EXISTS testkeyspace.test_table
(key INT PRIMARY KEY, value TEXT));
*/
/* Receive Kafka streaming inputs */
JavaPairReceiverInputDStreamString, String messages =
KafkaUtils.createStream(jssc, args[0], args[1], topicMap );
/* Create DStream */
JavaDStreamTestTable data = messages.map(new
FunctionTuple2String,String, TestTable ()
{
public TestTable call(Tuple2String, String message)
{
return new TestTable(new Integer(++key), message._2() );
}
}
);
/* Write to cassandra */
javaFunctions(data).writerBuilder(testkeyspace, test_table,
mapToRow(TestTable.class)).saveToCassandra();
// data.print();
jssc.start();
jssc.awaitTermination();
}
}
class TestTable implements Serializable
{
Integer key;
String value;
public TestTable() {}
public TestTable(Integer k, String v)
{
key=k;
value=v;
}
public Integer getKey(){
return key;
}
public void setKey(Integer k){
key=k;
}
public String getValue(){
return value;
}
public void setValue(String v){
value=v;
}
public String toString(){
return MessageFormat.format(TestTable'{'key={0},
value={1}'}', key, value);
}
}
The output log is:
[INFO] Compiling 1 source file to
/root/Documents/SparkStreamSample/target/classes
[INFO] 2 errors
[INFO] -
[ERROR] COMPILATION ERROR :
[INFO] -
[ERROR]
/root/Documents/SparkStreamSample/src/main/java/com/spark/SparkStream.java:[76,81]
cannot find symbol
symbol: method mapToRow(java.lang.Classcom.spark.TestTable)
location: class com.spark.SparkStream
[ERROR]
/root/Documents/SparkStreamSample/src/main/java/com/spark/SparkStream.java:[76,17]
no suitable method found for
javaFunctions(org.apache.spark.streaming.api.java.JavaDStreamcom.spark.TestTable)
method
com.datastax.spark.connector.CassandraJavaUtil.TjavaFunctions(org.apache.spark.streaming.api.java.JavaDStreamT,java.lang.ClassT)
is not applicable
(cannot infer type-variable(s) T
(actual and formal argument lists differ in length))
method
com.datastax.spark.connector.CassandraJavaUtil.TjavaFunctions(org.apache.spark.streaming.dstream.DStreamT,java.lang.ClassT)
is not applicable
(cannot infer type-variable(s) T
(actual and formal argument lists differ in length))
method
com.datastax.spark.connector.CassandraJavaUtil.TjavaFunctions(org.apache.spark.api.java.JavaRDDT,java.lang.ClassT)
is not applicable
(cannot infer type-variable(s) T
(actual and formal argument lists differ in length))
method
com.datastax.spark.connector.CassandraJavaUtil.TjavaFunctions(org.apache.spark.rdd.RDDT,java.lang.ClassT)
is not applicable
(cannot infer type-variable(s) T
(actual and formal argument lists differ in length))
method
