Re: Updating Broadcast Variable in Spark Streaming 2.4.4

2022-07-22 Thread Sean Owen
I think you're taking the right approach, trying to create a new broadcast var. What part doesn't work? for example I wonder if comparing Map equality like that does what you think, isn't it just reference equality? debug a bit more to see whether it even destroys and recreates the broadcast in

Updating Broadcast Variable in Spark Streaming 2.4.4

2022-07-22 Thread Dipl.-Inf. Rico Bergmann
Hi folks! I'm trying to implement an update of a broadcast var in Spark Streaming. The idea is that whenever some configuration value has changed (this is periodically checked by the driver) the existing broadcast variable is unpersisted and then (re-)broadcasted. In a local test setup