Hello, I want to set data in a broadcast (Map<key,value>) variable in Spark. Sometimes there are new data so I have to update/refresh the values but I'm not sure how I could do this.
My idea is to use accumulators like a flag when a cache error occurs, in this point I could read the data and reload the broadcast variable for the next microbach and finally reset the accumulator to 0. I don't know if there are a better solution or others ideas to do this. Has anyone faced this problem?
