I think you need to dig into the custom receiver implementation. As long as the
source is distributed and partitioned, the downstream .map, .foreachXX are all
distributed as you would expect.
You could look at how the “classic” Kafka receiver is instantiated in the
streaming guide and try to
Hello,
Please could you explain me what is exactly distributed when I launch a spark
streaming job over YARN cluster ?
My code is something like :
JavaDStream customReceiverStream =
ssc.receiverStream(streamConfig.getJmsReceiver());
JavaDStream incoming_msg = customReceiverStream.map(
