You question is beyond gRPC framework, and one cannot interrupt thread has been a headache for Python (and programming languages with multi-threading) for a long time.
Alternatively, you could: 1) Have the server thread instead of sleep for a complete 5 minute, you can break it down to like 1 second and check for termination flag. The termination flag can be flipped by other threads. 2) If the job you run can be ran with "subprocess", then it will be easier to control its life cycle. 3) Wrap your job with Python one of "Future" implementation. 4̶)̶ ̶I̶n̶v̶o̶k̶e̶ ̶C̶P̶y̶t̶h̶o̶n̶ ̶C̶ ̶A̶P̶I̶ ̶P̶y̶T̶h̶r̶e̶a̶d̶S̶t̶a̶t̶e̶_̶S̶e̶t̶A̶s̶y̶n̶c̶E̶x̶c̶ <https://docs.python.org/2/c-api/init.html#c.PyThreadState_SetAsyncExc>.̶ On Thursday, February 28, 2019 at 5:14:18 PM UTC-8, Josh Liburdi wrote: > > That is a good example of using the callback! Where I get stuck is the > first example you mentioned, cancelling the running job. A simulation of my > problem would be to have the server perform a very long task (e.g. 5 minute > sleep call); in those cases, I would need the callback to interrupt/cancel > that sleep call. Usually I would handle this with signals and setting an > explicit timer in the server process, but (from what I’ve seen and read) > signals cannot be used in threads. > > On Thu, Feb 28, 2019 at 4:48 PM lidiz via grpc.io <[email protected] > <javascript:>> wrote: > >> I wrote an example about the "add_callback" API last December after >> reading this thread: https://github.com/grpc/grpc/pull/17551. But I >> haven't really push to merge that pull request. >> You can add your special logic in the server-side callback, like cancel >> the running job, log metrics, and other stuff. >> Please take a look at the example, and let me know if it failed to solve >> your question. >> >> >> On Thursday, February 28, 2019 at 4:32:07 PM UTC-8, [email protected] >> wrote: >>> >>> Out of curiosity, can anyone show an example of how add_callback can be >>> used to interrupt the server-side process? I have the same problem as the >>> OP for my application -- server-side can run for a very long time and if >>> the client times out, then I need the server to cancel immediately. I've >>> tried a variety of techniques, but I cannot get the callback function to >>> stop the server-side call. >>> >>> On Tuesday, December 18, 2018 at 12:51:23 PM UTC-8, [email protected] >>> wrote: >>>> >>>> Ah; thanks--we're having to use subprocess.Popen in a few cases >>>> anyway. I'll try that and see what we can do. Thanks for the note on >>>> "grpc within grpc"; that may simplify some things too. >>>> >>>> On Tuesday, December 18, 2018 at 1:07:00 PM UTC-6, Eric Gribkoff wrote: >>>>> >>>>> >>>>> >>>>> On Tue, Dec 18, 2018 at 10:45 AM <[email protected]> wrote: >>>>> >>>>>> Thanks, Eric. That makes some degree of sense, although there are a >>>>>> few cases we still won't be able to deal with, I suspect (and we may >>>>>> have >>>>>> trouble later anyway... in some cases our server program has to shell >>>>>> out >>>>>> to run a separate program, and if that runs into the fork trouble and >>>>>> can't >>>>>> be supported by GRPC we may be stuck with a very clanky REST >>>>>> implementation). >>>>>> >>>>>> >>>>> Sorry, I should have been more precise in my earlier response: you are >>>>> fine to use fork+exec (e.g., subprocess.Popen) to run a separate program >>>>> in >>>>> a new shell. (Caveat: we had a bug >>>>> <https://github.com/grpc/grpc/issues/17093> that may cause problems >>>>> even with fork+exec when using Python3. The fix is now merged and will be >>>>> in the next release; our nightly builds will also include the fix >>>>> ~tomorrow >>>>> if you are hitting this issue). The issues on the server-side with fork >>>>> arise when using libraries that fork and, rather than exec'ing a new >>>>> program, continue to run the original program in the child process, e.g., >>>>> Python's multiprocessing module. >>>>> >>>>> >>>>> >>>>>> Hmm, quite a pickle. I can see I'll be playing with a bunch of toy >>>>>> problems for a bit before even considering doing a migration to GRPC. >>>>>> Most >>>>>> disagreeable, but we'll see what we get. >>>>>> >>>>>> Can grpc client stubs be used from within grpc servicers? (imagining >>>>>> fracturing this whole thing into microservices even if that doesn't >>>>>> solve >>>>>> this particular problem). >>>>>> >>>>> >>>>> Absolutely, and that's an intended/common usage. >>>>> >>>>> Thanks, >>>>> >>>>> Eric >>>>> >>>>> >>>>>> >>>>>> On Tuesday, December 18, 2018 at 12:32:15 PM UTC-6, Eric Gribkoff >>>>>> wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Tue, Dec 18, 2018 at 10:17 AM <[email protected]> wrote: >>>>>>> >>>>>>>> Hmm; I'm having some luck looking at the context, which quite >>>>>>>> happily changes from is_active() to not is_active() the instant I kill >>>>>>>> the >>>>>>>> waiting client. So I thought I'd proceed with something like >>>>>>>> >>>>>>>> while not my_future.done(): >>>>>>>> if not context.is_active(): >>>>>>>> my_future.cancel() >>>>>>>> >>>>>>>> >>>>>>> Consider using add_callback >>>>>>> <https://grpc.io/grpc/python/grpc.html#grpc.RpcContext.add_callback> on >>>>>>> the RpcContext instead, so you don't have to poll. >>>>>>> >>>>>>> >>>>>>>> Terminating the worker thread/process is actually vexing me >>>>>>>> though! I tried having a ThreadPoolExecutor to give me a future for >>>>>>>> the >>>>>>>> worker task, but you can't really cancel a future from a thread, it >>>>>>>> turns >>>>>>>> out (you can only cancel it if it hasn't started running; once it's >>>>>>>> started, it still goes to completion). So I've tried having a >>>>>>>> separate >>>>>>>> ProcessPoolExecutor (maybe processes can be killed?) but that's not >>>>>>>> actually going so well either, as attempts to use that to generate >>>>>>>> futures >>>>>>>> results in some odd "Failed accept4: Invalid Argument" errors which I >>>>>>>> can't >>>>>>>> quite work through. >>>>>>>> >>>>>>>> >>>>>>> ProcessPoolExecutor will fork subprocesses, and gRPC servers (and >>>>>>> many other multi-threaded libraries) are not compatible with this. >>>>>>> There is >>>>>>> some discussion around this in >>>>>>> https://github.com/grpc/grpc/issues/16001. You could pre-fork (fork >>>>>>> before creating the gRPC server), but I don't think this will help with >>>>>>> your goal of cancelling long-running jobs. It's difficult to cleanly >>>>>>> kill >>>>>>> subprocesses, as they may be in the middle of an operation that you >>>>>>> would >>>>>>> really like to clean up gracefully. >>>>>>> >>>>>>> >>>>>>>> Most confusing. I wonder if I'll need to subclass grpc.server or >>>>>>>> if my servicer can manually run a secondary process or some such. >>>>>>>> >>>>>>>> Still, surprising to me this isn't a solved problem built into >>>>>>>> GRPC. I feel like I'm missing something really obvious. >>>>>>>> >>>>>>>> >>>>>>> I wouldn't consider cancelling long running jobs spawned by your >>>>>>> server as part of the functionality that gRPC is intended for - this is >>>>>>> a >>>>>>> task that can came up regardless of what server protocol you are using, >>>>>>> and >>>>>>> will arise often even on non-server applications. A standard approach >>>>>>> for >>>>>>> this in a multi-threaded environment would be setting a cancel boolean >>>>>>> variable (e.g., in your gRPC servicer implementation) that your task >>>>>>> (the >>>>>>> long-running subroutine) periodically checks for to exit early. This >>>>>>> should >>>>>>> be compatible with ThreadPoolExecutor. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Eric >>>>>>> >>>>>>> >>>>>>>> On Monday, December 17, 2018 at 1:35:41 PM UTC-6, robert engels >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> You don’t have to - just use the future as described - if the >>>>>>>>> stream is cancelled by the client - you can cancel the future - if >>>>>>>>> the >>>>>>>>> future completes you send the result back in the stream (if any) - >>>>>>>>> you >>>>>>>>> don’t have to keep sending messages as long as the keep alive is on. >>>>>>>>> >>>>>>>>> On Dec 17, 2018, at 1:32 PM, [email protected] wrote: >>>>>>>>> >>>>>>>>> Good idea, but the problem I have with this (if I understand you >>>>>>>>> right) is that some of the server tasks are just these big monolithic >>>>>>>>> calls >>>>>>>>> that sit there doing CPU-intensive work (sometimes in a third-party >>>>>>>>> library; it's not trivial to change them to stream back progress >>>>>>>>> reports or >>>>>>>>> anything). >>>>>>>>> >>>>>>>>> So it feels like some way of running them in a separate thread and >>>>>>>>> having an overseer method able to kill them if the client disconnects >>>>>>>>> is >>>>>>>>> the way to go. We're already using a ThreadPoolExecutor to run >>>>>>>>> worker >>>>>>>>> threads so I feel like there's something that can be done on that >>>>>>>>> side... >>>>>>>>> just seems like this ought to be a Really Common Problem, so I'm >>>>>>>>> surprised >>>>>>>>> it's either not directly addressed or at least commonly answered. >>>>>>>>> >>>>>>>>> On Monday, December 17, 2018 at 1:27:39 PM UTC-6, robert engels >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> You can do this if you use the streaming protocol - that is the >>>>>>>>>> only way I know to have any facilities to determine when a “client >>>>>>>>>> disconnects”. >>>>>>>>>> >>>>>>>>>> On Dec 17, 2018, at 1:24 PM, [email protected] wrote: >>>>>>>>>> >>>>>>>>>> I'm sure it's been answered before but I've searched for quite a >>>>>>>>>> while and not found anything, so apologies: >>>>>>>>>> >>>>>>>>>> We're using python... we've got server tasks that can last quite >>>>>>>>>> a while (minutes) and chew up lots of CPU. Right now we're using >>>>>>>>>> REST, and >>>>>>>>>> when/if the client disconnects before return, the task keeps running >>>>>>>>>> on the >>>>>>>>>> server side. This is unfortunate; it's costly (since the server may >>>>>>>>>> be >>>>>>>>>> using for-pay services remotely, leaving the task running could cost >>>>>>>>>> the >>>>>>>>>> client) and vulnerable (a malicious client could just start and >>>>>>>>>> immediately >>>>>>>>>> disconnect hundreds of tasks and lock the server up for quite a >>>>>>>>>> while). >>>>>>>>>> >>>>>>>>>> I was hoping that a move to GRPC, in addition to solving other >>>>>>>>>> problems, would provide a clean way to deal with this. But it's not >>>>>>>>>> immediately obvious how to do so. I could see maybe manually >>>>>>>>>> starting a >>>>>>>>>> thread/Future for the worker process and iterating sleeping until >>>>>>>>>> either >>>>>>>>>> the context is invalid or the thread/future returns, but I feel like >>>>>>>>>> that's >>>>>>>>>> manually hacking something that probably exists and I'm not >>>>>>>>>> understanding. >>>>>>>>>> Maybe some sort of server interceptor? >>>>>>>>>> >>>>>>>>>> How would it be best to handle this? I'd like to handle both >>>>>>>>>> very long unary calls and streaming calls in the same manner. >>>>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> Vic >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>> Google Groups "grpc.io" group. >>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>> send an email to [email protected]. >>>>>>>>>> To post to this group, send email to [email protected]. >>>>>>>>>> Visit this group at https://groups.google.com/group/grpc-io. >>>>>>>>>> To view this discussion on the web visit >>>>>>>>>> https://groups.google.com/d/msgid/grpc-io/9e84949d-139c-43df-a09e-5d8cc79022be%40googlegroups.com >>>>>>>>>> >>>>>>>>>> <https://groups.google.com/d/msgid/grpc-io/9e84949d-139c-43df-a09e-5d8cc79022be%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>>> . >>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "grpc.io" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to [email protected]. >>>>>>>>> To post to this group, send email to [email protected]. >>>>>>>>> Visit this group at https://groups.google.com/group/grpc-io. >>>>>>>>> To view this discussion on the web visit >>>>>>>>> https://groups.google.com/d/msgid/grpc-io/90ba2085-8fb9-4851-9ae7-75ad45a5021d%40googlegroups.com >>>>>>>>> >>>>>>>>> <https://groups.google.com/d/msgid/grpc-io/90ba2085-8fb9-4851-9ae7-75ad45a5021d%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>> . >>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "grpc.io" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> To post to this group, send email to [email protected]. >>>>>>>> Visit this group at https://groups.google.com/group/grpc-io. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/grpc-io/733b0293-6162-47c8-85f7-28cfa0b932b8%40googlegroups.com >>>>>>>> >>>>>>>> <https://groups.google.com/d/msgid/grpc-io/733b0293-6162-47c8-85f7-28cfa0b932b8%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "grpc.io" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To post to this group, send email to [email protected]. >>>>>> Visit this group at https://groups.google.com/group/grpc-io. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/grpc-io/e67efea6-e740-4e08-90c1-b093b85a9914%40googlegroups.com >>>>>> >>>>>> <https://groups.google.com/d/msgid/grpc-io/e67efea6-e740-4e08-90c1-b093b85a9914%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "grpc.io" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/grpc-io/3FdOOF7AK1g/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected] <javascript:>. >> To post to this group, send email to [email protected] >> <javascript:>. >> Visit this group at https://groups.google.com/group/grpc-io. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/grpc-io/9109d969-ade6-485d-b60d-792e75b18123%40googlegroups.com >> >> <https://groups.google.com/d/msgid/grpc-io/9109d969-ade6-485d-b60d-792e75b18123%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/grpc-io. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/6baa9d8e-3a7d-486d-8437-f4885908ebdb%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
