So you are using `client` mode. Right? If so, Spark cluster doesn't manage the driver for you. Did you see any error logs in driver?
On Mon, May 15, 2017 at 3:01 PM, map reduced <k3t.gi...@gmail.com> wrote: > Hi, > > Setup: Standalone cluster with 32 workers, 1 master > I am running a long running streaming spark job (read from Kafka -> > process -> send to Http endpoint) which should ideally never stop. > > I have 2 questions: > 1) I have seen some times Driver is still running but application marked > as *Finished*. *Any idea why this happens or any way to debug this?* > Sometimes after running for say 2-3 days (or 4-5 days - random timeframe) > this issue arises, not sure what is causing it. Nothing in logs suggests > failures or exceptions > > 2) Is there a way for Driver to kill itself instead of keeping on running > without any application to drive? > > Thanks, > KP >
