Hi,
I am integrating Kafka and Spark, using spark-streaming. I have created a topic
as a kafka producer:
bin/kafka-topics.sh --create --zookeeper localhost:2181
--replication-factor 1 --partitions 1 --topic test
I am publishing messages in kafka and trying to read them using
Hi,
The spark master is working, and I have given the same url in the code:
[cid:image001.png@01D00D82.6DC2FFF0]
The warning is gone, and the new log is:
---
Time: 141742785 ms
---
INFO
Hi Gerard/Akhil,
By how do I specify a batch I was trying to ask that when does the data in
the JavaDStream gets flushed into Cassandra table?.
I read somewhere that the streaming data in batches gets written in Cassandra.
This batch can be of some particular time, or one particular run.
Hi Akhil, Vyas, Helena,
Thank you for your suggestions.
As Akhil suggested earlier, i have implemented the batch Duration into
JavaStreamingContext and waitForTermination(Duration).
The approach Helena suggested is Scala oriented.
But the issue now is that I want to set Cassandra as my output.
Thank you Helena,
But I would like to explain my problem space:
The output is supposed to be Cassandra. To achieve that, I have to use
spark-cassandra-connecter APIs.
So going in a botton-up approach, to write to cassandra, I have to use:
javaFunctions(JavaRDD object rdd,
Hi,
I am intending to save the streaming data from kafka into Cassandra,
using spark-streaming:
But there seems to be problem with line
javaFunctions(data).writerBuilder(testkeyspace, test_table,
mapToRow(TestTable.class)).saveToCassandra();
I am getting 2 errors.
the code, the error-log and
Hi,
I am intending to save the streaming data from kafka into Cassandra, using
spark-streaming:
But there seems to be problem with line
javaFunctions(data).writerBuilder(testkeyspace, test_table,
mapToRow(TestTable.class)).saveToCassandra();
I am getting NoSuchMethodError.
The code, the
Hi,
@Gerard- Thanks, i added one more dependency for
conf.set(spark.cassandra.connection.host, localhost).
But now, i am able to create a connection, but the data is not getting inserted
into the cassandra table.
and the logs show its getting connected and the next second getting