Hi, I have a streaming application which uses a broadcast variable which I populate from a database. I would like every once in a while (or even every batch) to update/replace the broadcast variable with the latest data from the database.
Only way I found online to do this is this "hackish" way ( http://stackoverflow.com/questions/28573816/periodic-broadcast-in-apache-spark-streaming) which I'm not sure gets re-executed per batch anyway: val broadcastFactory = new TorrentBroadcastFactory() broadcastFactory.unbroadcast(BroadcastId, true, true) // append some ids to initIds val broadcastcontent = broadcastFactory.newBroadcast[.Set[String]](initIds, false, BroadcastId) Is there a proper way to do that? Thank you, Daniel