Hi all, I have a long lived Spark Streaming cluster feeding from several Kinesis streams. There is some data that must be obtained from another data source to be used in one of the steps on the workers.
My understanding is that the best way to achieve this is by broadcasting that data. Is that correct? How can I periodically broadcast this info? Is it a good idea to use StreamingListener.onBatchStarted to achieve this? Best regards, Serafin.
