1. RPC can be done in many ways, and a web service is one of many ways. A
even more hacky version can be the app polling a file in a file system, if
the file exists start shutting down.
2. No need to set a flag. When you get the signal from RPC, you can just
call context.stop(stopGracefully = true) . Though note that this is
blocking, so gotta be carefully about doing blocking calls on the RPC
thread.
On Mon, Aug 10, 2015 at 12:24 PM, Shushant Arora <shushantaror...@gmail.com>
wrote:
> By RPC you mean web service exposed on driver which listens and set some
> flag and driver checks that flag at end of each batch and if set then
> gracefully stop the application ?
>
> On Tue, Aug 11, 2015 at 12:43 AM, Tathagata Das <t...@databricks.com>
> wrote:
>
>> In general, it is a little risky to put long running stuff in a shutdown
>> hook as it may delay shutdown of the process which may delay other things.
>> That said, you could try it out.
>>
>> A better way to explicitly shutdown gracefully is to use an RPC to signal
>> the driver process to start shutting down, and then the process will
>> gracefully stop the context and terminate. This is more robust that than
>> leveraging shutdown hooks.
>>
>> On Mon, Aug 10, 2015 at 11:56 AM, Shushant Arora <
>> shushantaror...@gmail.com> wrote:
>>
>>> Any help in best recommendation for gracefully shutting down a spark
>>> stream application ?
>>> I am running it on yarn and a way to tell from externally either yarn
>>> application -kill command or some other way but need current batch to be
>>> processed completely and checkpoint to be saved before shutting down.
>>>
>>> Runtime.getRuntime().addShutdownHook does not seem to be working. Yarn
>>> kills the application immediately and dooes not call shutdown hook call
>>> back .
>>>
>>>
>>> On Sun, Aug 9, 2015 at 12:45 PM, Shushant Arora <
>>> shushantaror...@gmail.com> wrote:
>>>
>>>> Hi
>>>>
>>>> How to ensure in spark streaming 1.3 with kafka that when an
>>>> application is killed , last running batch is fully processed and offsets
>>>> are written to checkpointing dir.
>>>>
>>>> On Fri, Aug 7, 2015 at 8:56 AM, Shushant Arora <
>>>> shushantaror...@gmail.com> wrote:
>>>>
>>>>> Hi
>>>>>
>>>>> I am using spark stream 1.3 and using custom checkpoint to save kafka
>>>>> offsets.
>>>>>
>>>>> 1.Is doing
>>>>> Runtime.getRuntime().addShutdownHook(new Thread() {
>>>>> @Override
>>>>> public void run() {
>>>>> jssc.stop(true, true);
>>>>> System.out.println("Inside Add Shutdown Hook");
>>>>> }
>>>>> });
>>>>>
>>>>> to handle stop is safe ?
>>>>>
>>>>> 2.And I need to handle saving checkoinnt in shutdown hook also or
>>>>> driver will handle it automatically since it grcaefully stops stream and
>>>>> handle
>>>>> completion of foreachRDD function on stream ?
>>>>> directKafkaStream.foreachRDD(new Function<JavaRDD<byte[][]>, Void>() {
>>>>> }
>>>>>
>>>>> Thanks
>>>>>
>>>>>
>>>>
>>>
>>
>
