In general, it is a little risky to put long running stuff in a shutdown
hook as it may delay shutdown of the process which may delay other things.
That said, you could try it out.
A better way to explicitly shutdown gracefully is to use an RPC to signal
the driver process to start shutting down, and then the process will
gracefully stop the context and terminate. This is more robust that than
leveraging shutdown hooks.
On Mon, Aug 10, 2015 at 11:56 AM, Shushant Arora <shushantaror...@gmail.com>
wrote:
> Any help in best recommendation for gracefully shutting down a spark
> stream application ?
> I am running it on yarn and a way to tell from externally either yarn
> application -kill command or some other way but need current batch to be
> processed completely and checkpoint to be saved before shutting down.
>
> Runtime.getRuntime().addShutdownHook does not seem to be working. Yarn
> kills the application immediately and dooes not call shutdown hook call
> back .
>
>
> On Sun, Aug 9, 2015 at 12:45 PM, Shushant Arora <shushantaror...@gmail.com
> > wrote:
>
>> Hi
>>
>> How to ensure in spark streaming 1.3 with kafka that when an application
>> is killed , last running batch is fully processed and offsets are written
>> to checkpointing dir.
>>
>> On Fri, Aug 7, 2015 at 8:56 AM, Shushant Arora <shushantaror...@gmail.com
>> > wrote:
>>
>>> Hi
>>>
>>> I am using spark stream 1.3 and using custom checkpoint to save kafka
>>> offsets.
>>>
>>> 1.Is doing
>>> Runtime.getRuntime().addShutdownHook(new Thread() {
>>> @Override
>>> public void run() {
>>> jssc.stop(true, true);
>>> System.out.println("Inside Add Shutdown Hook");
>>> }
>>> });
>>>
>>> to handle stop is safe ?
>>>
>>> 2.And I need to handle saving checkoinnt in shutdown hook also or driver
>>> will handle it automatically since it grcaefully stops stream and handle
>>> completion of foreachRDD function on stream ?
>>> directKafkaStream.foreachRDD(new Function<JavaRDD<byte[][]>, Void>() {
>>> }
>>>
>>> Thanks
>>>
>>>
>>
>
