I am using spark streaming for text files. I have feeder program which moves
files to spark streaming directory.
while Spark processing particular file at the same time if feeders puts
another file into streaming directory, sometimes spark does not pick file
for processing.
we are using
want to read file data and check if file line data is present in Cassandra
if it's present then needs to merge otherwise fresh insert to C*. File data
just contains name,address in json format, in Cassandra student table have
UUID as primary key and there is secondry index on name
Once data is
Hi,
I am trying simple file streaming example using
Sparkstreaming(spark-streaming_2.10,version:1.5.1)
public class DStreamExample {
public static void main(final String[] args) {
final SparkConf sparkConf = new SparkConf();
sparkConf.setAppName("SparkJob");
you can use mapfunction..
This is java example..
final JavaRDD rdd1 = sc.textFile("filepath").map((line) -> {
//logic for line to product converstion});
Product class might have 5 attributes like you said class Product{
String str1;
int i1;
String str2;
int i2;
String str3;
// with getter
