Thanks Tathagata for making this change.. Dibyendu
On Thu, May 21, 2015 at 8:24 AM, Tathagata Das <t...@databricks.com> wrote: > If you are talking about handling driver crash failures, then all bets are > off anyways! Adding a shutdown hook in the hope of handling driver process > failure, handles only a some cases (Ctrl-C), but does not handle cases like > SIGKILL (does not run JVM shutdown hooks) or driver machine crash. So its > not a good idea to rely on that. > > Nonetheless I have opened a PR to handle the shutdown of the > StreamigntContext in the same way as SparkContext. > https://github.com/apache/spark/pull/6307 > > > On Tue, May 19, 2015 at 12:51 AM, Dibyendu Bhattacharya < > dibyendu.bhattach...@gmail.com> wrote: > >> Thenka Sean . you are right. If driver program is running then I can >> handle shutdown in main exit path . But if Driver machine is crashed (if >> you just stop the application, for example killing the driver process ), >> then Shutdownhook is the only option isn't it ? What I try to say is , just >> doing ssc.stop in sys.ShutdownHookThread or >> Runtime.getRuntime().addShutdownHook ( in java) wont work anymore. I need >> to use the Utils.addShutdownHook with a priority .. So just checking if >> Spark Streaming can make graceful shutdown as default shutdown mechanism. >> >> Dibyendu >> >> On Tue, May 19, 2015 at 1:03 PM, Sean Owen <so...@cloudera.com> wrote: >> >>> I don't think you should rely on a shutdown hook. Ideally you try to >>> stop it in the main exit path of your program, even in case of an >>> exception. >>> >>> On Tue, May 19, 2015 at 7:59 AM, Dibyendu Bhattacharya >>> <dibyendu.bhattach...@gmail.com> wrote: >>> > You mean to say within Runtime.getRuntime().addShutdownHook I call >>> > ssc.stop(stopSparkContext = true, stopGracefully = true) ? >>> > >>> > This won't work anymore in 1.4. >>> > >>> > The SparkContext got stopped before Receiver processed all received >>> blocks >>> > and I see below exception in logs. But if I add the >>> Utils.addShutdownHook >>> > with the priority as I mentioned , then only graceful shutdown works . >>> In >>> > that case shutdown-hook run in priority order. >>> > >>> >> >> >
