Does 'Stage cancelled because SparkContext was shut down' is a error

2022-09-28 Thread lk_spark
hi,all : when I try to merge a iceberg table by spark , I can see faild job on spark ui , but the spark application final state is SUCCEEDED. I submit an issue : https://github.com/apache/iceberg/issues/5876 I wonder to know is this a real error ? thanks .

Re: Updating Broadcast Variable in Spark Streaming 2.4.4

2022-09-28 Thread Sean Owen
I don't think that can work. Your BroadcastUpdater is copied to the task, with a reference to an initial broadcast. When that is later updated on the driver, this does not affect the broadcast inside the copy in the tasks. On Wed, Sep 28, 2022 at 10:11 AM Dipl.-Inf. Rico Bergmann <

Updating Broadcast Variable in Spark Streaming 2.4.4

2022-09-28 Thread Dipl.-Inf. Rico Bergmann
Hi folks! I'm trying to implement an update of a broadcast var in Spark Streaming. The idea is that whenever some configuration value has changed (this is periodically checked by the driver) the existing broadcast variable is unpersisted and then (re-)broadcasted. In a local test setup