We are using createDirectStream api to receive messages from 48
partitioned topic. I am setting up --num-executors 48 & --executor-cores
1 in spark-submit
All partitions were parallely received and corresponding RDDs in
foreachRDD were executed in parallel. But when I join a transformed RDD
jsonDF (code below) with another RDD, i could see that they are not
executed in parallell for each partitions. There were more shuffle read
and writes and no of executors executing were less than the no of
partitions. I mean, no of executors were not equal to no of partitions when
join is executing.
How could I make sure to execute join in all executors. Can anyone provide
help.
kstream.foreachRDD { rdd =>
val jsonDF = sqlContext.read.json(rdd).toDF
.....
...
val metaDF = ssc.sparkContext.textfile("file").toDF
val join = jsonDF.join(metaDF)
join.map (....).count
}
