I think this is usually done with `FileSystem.deleteOnExit`, which cleans up when an executor exits. That's not quite what you want (job finish) but is probably a reasonable fix.
rb On Thu, Jan 19, 2017 at 9:31 PM, Keith Chapman <keithgchap...@gmail.com> wrote: > Hi , > > Is it possible for an executor (or slave) to know when an actual job ends? > I'm running spark on a cluster (with yarn) and my workers create some > temporary files that I would like to clean up once the job ends. Is there a > way for the worker to detect that a job has finished? I tried doing it in > the JobProgressListener but it does not seem to work in a cluster. The > event is not triggered in the worker. > > Regards, > Keith. > > http://keith-chapman.com > -- Ryan Blue Software Engineer Netflix
