Hi again, On Mon, Nov 17, 2014 at 8:16 AM, Tobias Pfeiffer <[email protected]> wrote: > > I have been trying to mis-use broadcast as in > - create a class with a boolean var, set to true > - query this boolean on the executors as a prerequisite to process the > next item > - when I want to shutdown, I set the boolean to false and unpersist the > broadcast variable (which will trigger re-delivery). > This is very dirty, but it works with a "local[*]" master. Unfortunately, > when deployed on YARN, the new value will never arrive at my executors. >
In fact, it seems as if "change mutable object (like mutable list) and unpersist in order to trigger redeploy" only works locally. When running on YARN, even after an unpersist, the value will always be identical to what I shipped first. Now I wonder what unpersist actually does in that case. Must I call unpersist from an executor or from the driver? Thanks Tobias
