StreamingContext.stop(stopGracefully = true) stops the streaming context gracefully. Then you can safely terminate the Spark cluster. They are two different steps and needs to be done separately ensuring that the driver process has been completely terminated before the Spark cluster is the terminated.
On Wed, Jul 29, 2015 at 6:43 AM, Michal Čizmazia <mici...@gmail.com> wrote: > How to initiate graceful shutdown from outside of the Spark Streaming > driver process? Both for the local and cluster mode of Spark Standalone as > well as EMR. > > Does sbin/stop-all.sh stop the context gracefully? How is it done? Is > there a signal sent to the driver process? > > For EMR, is there a way how to terminate an EMR cluster with Spark > Streaming graceful shutdown? > > Thanks! > > >
