I don't have a direct answer for your question, but if I would to
investigate what may be going on, i would try running the server outside of
Kubernetes and container. I would run it on a local system, send the
SIGTERM signal, and see what happens first.
On Saturday, January 15, 2022 at 7:29:03 AM UTC+11 M T wrote:
> Pressed the wrong button when I tried to edit the code example... here's
> the correct (simplified) example:
>
>
> def handle_sigterm(*_: Any) -> None :
> """Shutdown gracefully."""
> done_event = server.stop(30)
> done_event.wait(30)
> print('Stop complete.')
>
> server = grpc.server(
> futures.ThreadPoolExecutor(max_workers = options['max_workers']),
> )
> add_Servicer_to_server(service, server)
> server.add_insecure_port(options['address'])
> server.start()
> signal(SIGTERM, handle_sigterm)
> server.wait_for_termination()
>
> I'm deploying my service through kubernetes, which stops pods by first
> sending a SIGTERM event, and, if the pod is still alive after a timeout, it
> kills it using SIGKILL.
>
>
> The behavior that I witness:
> * the time it takes for the pod to terminate looks a lot like it's using
> the full 30s that kubernetes gives it = it ends up getting killed by SIGKILL
> * the log for the stop of the service inside the SIGTERM handler never
> appears
>
>
> So I think that one of following things might be happening here (that I
> can think of):
>
>
> 1. the handler never gets called because the process gets killed before it
> gets to handle it
> 2. the server takes so long to shut down, that kubernetes ends up killing
> it before it gets to logging the message
> 3. the shutdown event doesn't get set even after the server is done
> shutting down, causing kubernetes to kill it without logging the message
>
>
> I see that the official examples contain a asyncio example for graceful
> shutdown, can we maybe get something like for the regular, non-asyncio case
> (https://github.com/lidizheng/grpc/tree/master/examples/python/helloworld
> )?
>
> And for the long run: would it be possible to get something like
> wait_for_termination that is capable of handling graceful shutdown
> natively, without the developers having to add something on top of it?
>
> Thanks for your help!
> On Friday, January 14, 2022 at 9:20:22 PM UTC+1 M T wrote:
>
>> Hi all,
>>
>> I'm currently trying to add some graceful shutdown logic into my gRPC
>> server, but it seems that my shutdown handler never gets called:
>>
>> def handle_sigterm(*_: Any) -> None :
>> """Shutdown gracefully."""
>> done_event = server.stop(30)
>> done_event.wait(30)
>> self.stdout.write('Stop complete.')
>>
>> server = grpc.server(
>> futures.ThreadPoolExecutor(max_workers = options['max_workers']),
>> )
>> add_
>> server.add_insecure_port(options['address'])
>> self.stdout.write(f'Starting gRPC server at {options["address"]}...')
>> server.start()
>> start_http_server(int(khaleesi_settings['MONITORING']['PORT']))
>> signal(SIGTERM, handle_sigterm)
>> self._log_server_state_event(
>> action = Event.Action.ActionType.START,
>> result = Event.Action.ResultType.SUCCESS,
>> details = 'Server started successfully.'
>> )
>> server.wait_for_termination()
>> except Exception as start_exception:
>> self._log_server_state_event(
>> action = Event.Action.ActionType.START,
>> result = Event.Action.ResultType.ERROR,
>> details = f'Server startup failed. {type(start_exception).__name__}:
>> {str(start_exception)}'
>> )
>> raise start_exception from None
>>
>
--
You received this message because you are subscribed to the Google Groups
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to grpc-io+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/grpc-io/bb042dab-e17c-47e7-b3cc-d2a77fe48474n%40googlegroups.com.
