Init is easy -- initialize them in your singleton. Shutdown is harder; a shutdown hook is probably the only reliable way to go. Global state is not ideal in Spark. Consider initializing things like connections per partition, and open/close them with the lifecycle of a computation on a partition instead.
On Wed, Oct 26, 2016 at 9:27 PM Walter rakoff <walter.rak...@gmail.com> wrote: > Hello, > > Is there a way I can add an init() call when an executor is created? I'd > like to initialize a few connections that are part of my singleton object. > Preferably this happens before it runs the first task > On the same line, how can I provide an shutdown hook that cleans up these > connections on termination. > > Thanks > Walt >
