Re: Broadcast Variable Life Cycle

Yeah, after destroy, accessing the broadcast variable results in an error. Accessing it after it's unpersisted (on an executor) causes it to be rebroadcast. On Tue, Aug 30, 2016 at 5:12 PM, Jerry Lam wrote: > Hi Sean, > > Thank you for sharing the knowledge between

Re: Broadcast Variable Life Cycle

Hi Sean, Thank you for sharing the knowledge between unpersist and destroy. Does that mean unpersist keeps the broadcast variable in the driver whereas destroy will delete everything about the broadcast variable like it has never existed? Best Regards, Jerry On Tue, Aug 30, 2016 at 11:58 AM,

Re: Broadcast Variable Life Cycle

Yes, although there's a difference between unpersist and destroy, you'll hit the same type of question either way. You do indeed have to reason about when you know the broadcast variable is no longer needed in the face of lazy evaluation, and that's hard. Sometimes it's obvious and you can take

Re: Broadcast Variable Life Cycle

Hi Sean, Thank you for the response. The only problem is that actively managing broadcast variables require to return the broadcast variables to the caller if the function that creates the broadcast variables does not contain any action. That is the scope that uses the broadcast variables cannot

Re: Broadcast Variable Life Cycle

Yes you want to actively unpersist() or destroy() broadcast variables when they're no longer needed. They can eventually be removed when the reference on the driver is garbage collected, but you usually would not want to rely on that. On Mon, Aug 29, 2016 at 4:30 PM, Jerry Lam

Re: Broadcast Variable Life Cycle

Hello spark developers, Anyone can shed some lights on the life cycle of the broadcast variables? Basically, if I have a broadcast variable defined in a loop and for each iteration, I provide a different value. // For example: for(i< 1 to 10) { val bc = sc.broadcast(i)

Broadcast Variable Life Cycle

Hello spark developers, Can someone explain to me what is the lifecycle of a broadcast variable? When a broadcast variable will be garbage-collected at the driver-side and at the executor-side? Does a spark application need to actively manage the broadcast variables to ensure that it will not run