Thanks Todd, that solved my problem.
Regards,
Gaurav
(please excuse spelling mistakes)
Sent from phone
On Jun 11, 2015 6:42 PM, Todd Nist tsind...@gmail.com wrote:
Hi Gaurav,
Seems like you could use a broadcast variable for this if I understand
your use case. Create it in the driver based
Hi,
I am using Kafka Spark cluster for real time aggregation analytics use case
in production.
Cluster details
6 nodes, each node running 1 Spark and kafka processes each.
Node1 - 1 Master , 1 Worker, 1 Driver,
1 Kafka process
Node 2,3,4,5,6 - 1 Worker prcocess each
Hi Gaurav,
Seems like you could use a broadcast variable for this if I understand your
use case. Create it in the driver based on the CommandLineArguments and
then use it in the workers.
https://spark.apache.org/docs/latest/programming-guide.html#broadcast-variables
So something like:
