On Tue, Dec 18, 2018 at 10:45 AM <vbp...@gmail.com> wrote: > Thanks, Eric. That makes some degree of sense, although there are a few > cases we still won't be able to deal with, I suspect (and we may have > trouble later anyway... in some cases our server program has to shell out > to run a separate program, and if that runs into the fork trouble and can't > be supported by GRPC we may be stuck with a very clanky REST > implementation). > > Sorry, I should have been more precise in my earlier response: you are fine to use fork+exec (e.g., subprocess.Popen) to run a separate program in a new shell. (Caveat: we had a bug <https://github.com/grpc/grpc/issues/17093> that may cause problems even with fork+exec when using Python3. The fix is now merged and will be in the next release; our nightly builds will also include the fix ~tomorrow if you are hitting this issue). The issues on the server-side with fork arise when using libraries that fork and, rather than exec'ing a new program, continue to run the original program in the child process, e.g., Python's multiprocessing module.
> Hmm, quite a pickle. I can see I'll be playing with a bunch of toy > problems for a bit before even considering doing a migration to GRPC. Most > disagreeable, but we'll see what we get. > > Can grpc client stubs be used from within grpc servicers? (imagining > fracturing this whole thing into microservices even if that doesn't solve > this particular problem). > Absolutely, and that's an intended/common usage. Thanks, Eric > > On Tuesday, December 18, 2018 at 12:32:15 PM UTC-6, Eric Gribkoff wrote: >> >> >> >> On Tue, Dec 18, 2018 at 10:17 AM <vbp...@gmail.com> wrote: >> >>> Hmm; I'm having some luck looking at the context, which quite happily >>> changes from is_active() to not is_active() the instant I kill the waiting >>> client. So I thought I'd proceed with something like >>> >>> while not my_future.done(): >>> if not context.is_active(): >>> my_future.cancel() >>> >>> >> Consider using add_callback >> <https://grpc.io/grpc/python/grpc.html#grpc.RpcContext.add_callback> on >> the RpcContext instead, so you don't have to poll. >> >> >>> Terminating the worker thread/process is actually vexing me though! I >>> tried having a ThreadPoolExecutor to give me a future for the worker task, >>> but you can't really cancel a future from a thread, it turns out (you can >>> only cancel it if it hasn't started running; once it's started, it still >>> goes to completion). So I've tried having a separate ProcessPoolExecutor >>> (maybe processes can be killed?) but that's not actually going so well >>> either, as attempts to use that to generate futures results in some odd >>> "Failed accept4: Invalid Argument" errors which I can't quite work through. >>> >>> >> ProcessPoolExecutor will fork subprocesses, and gRPC servers (and many >> other multi-threaded libraries) are not compatible with this. There is some >> discussion around this in https://github.com/grpc/grpc/issues/16001. You >> could pre-fork (fork before creating the gRPC server), but I don't think >> this will help with your goal of cancelling long-running jobs. It's >> difficult to cleanly kill subprocesses, as they may be in the middle of an >> operation that you would really like to clean up gracefully. >> >> >>> Most confusing. I wonder if I'll need to subclass grpc.server or if my >>> servicer can manually run a secondary process or some such. >>> >>> Still, surprising to me this isn't a solved problem built into GRPC. I >>> feel like I'm missing something really obvious. >>> >>> >> I wouldn't consider cancelling long running jobs spawned by your server >> as part of the functionality that gRPC is intended for - this is a task >> that can came up regardless of what server protocol you are using, and will >> arise often even on non-server applications. A standard approach for this >> in a multi-threaded environment would be setting a cancel boolean variable >> (e.g., in your gRPC servicer implementation) that your task (the >> long-running subroutine) periodically checks for to exit early. This should >> be compatible with ThreadPoolExecutor. >> >> Thanks, >> >> Eric >> >> >>> On Monday, December 17, 2018 at 1:35:41 PM UTC-6, robert engels wrote: >>>> >>>> You don’t have to - just use the future as described - if the stream is >>>> cancelled by the client - you can cancel the future - if the future >>>> completes you send the result back in the stream (if any) - you don’t have >>>> to keep sending messages as long as the keep alive is on. >>>> >>>> On Dec 17, 2018, at 1:32 PM, vbp...@gmail.com wrote: >>>> >>>> Good idea, but the problem I have with this (if I understand you right) >>>> is that some of the server tasks are just these big monolithic calls that >>>> sit there doing CPU-intensive work (sometimes in a third-party library; >>>> it's not trivial to change them to stream back progress reports or >>>> anything). >>>> >>>> So it feels like some way of running them in a separate thread and >>>> having an overseer method able to kill them if the client disconnects is >>>> the way to go. We're already using a ThreadPoolExecutor to run worker >>>> threads so I feel like there's something that can be done on that side... >>>> just seems like this ought to be a Really Common Problem, so I'm surprised >>>> it's either not directly addressed or at least commonly answered. >>>> >>>> On Monday, December 17, 2018 at 1:27:39 PM UTC-6, robert engels wrote: >>>>> >>>>> You can do this if you use the streaming protocol - that is the only >>>>> way I know to have any facilities to determine when a “client >>>>> disconnects”. >>>>> >>>>> On Dec 17, 2018, at 1:24 PM, vbp...@gmail.com wrote: >>>>> >>>>> I'm sure it's been answered before but I've searched for quite a while >>>>> and not found anything, so apologies: >>>>> >>>>> We're using python... we've got server tasks that can last quite a >>>>> while (minutes) and chew up lots of CPU. Right now we're using REST, and >>>>> when/if the client disconnects before return, the task keeps running on >>>>> the >>>>> server side. This is unfortunate; it's costly (since the server may be >>>>> using for-pay services remotely, leaving the task running could cost the >>>>> client) and vulnerable (a malicious client could just start and >>>>> immediately >>>>> disconnect hundreds of tasks and lock the server up for quite a while). >>>>> >>>>> I was hoping that a move to GRPC, in addition to solving other >>>>> problems, would provide a clean way to deal with this. But it's not >>>>> immediately obvious how to do so. I could see maybe manually starting a >>>>> thread/Future for the worker process and iterating sleeping until either >>>>> the context is invalid or the thread/future returns, but I feel like >>>>> that's >>>>> manually hacking something that probably exists and I'm not understanding. >>>>> Maybe some sort of server interceptor? >>>>> >>>>> How would it be best to handle this? I'd like to handle both very >>>>> long unary calls and streaming calls in the same manner. >>>>> >>>>> Cheers, >>>>> Vic >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "grpc.io" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to grpc-io+u...@googlegroups.com. >>>>> To post to this group, send email to grp...@googlegroups.com. >>>>> Visit this group at https://groups.google.com/group/grpc-io. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/grpc-io/9e84949d-139c-43df-a09e-5d8cc79022be%40googlegroups.com >>>>> <https://groups.google.com/d/msgid/grpc-io/9e84949d-139c-43df-a09e-5d8cc79022be%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>>> >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "grpc.io" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to grpc-io+u...@googlegroups.com. >>>> To post to this group, send email to grp...@googlegroups.com. >>>> Visit this group at https://groups.google.com/group/grpc-io. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/grpc-io/90ba2085-8fb9-4851-9ae7-75ad45a5021d%40googlegroups.com >>>> <https://groups.google.com/d/msgid/grpc-io/90ba2085-8fb9-4851-9ae7-75ad45a5021d%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "grpc.io" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to grpc-io+u...@googlegroups.com. >>> To post to this group, send email to grp...@googlegroups.com. >>> Visit this group at https://groups.google.com/group/grpc-io. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/grpc-io/733b0293-6162-47c8-85f7-28cfa0b932b8%40googlegroups.com >>> <https://groups.google.com/d/msgid/grpc-io/733b0293-6162-47c8-85f7-28cfa0b932b8%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- > You received this message because you are subscribed to the Google Groups " > grpc.io" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to grpc-io+unsubscr...@googlegroups.com. > To post to this group, send email to grpc-io@googlegroups.com. > Visit this group at https://groups.google.com/group/grpc-io. > To view this discussion on the web visit > https://groups.google.com/d/msgid/grpc-io/e67efea6-e740-4e08-90c1-b093b85a9914%40googlegroups.com > <https://groups.google.com/d/msgid/grpc-io/e67efea6-e740-4e08-90c1-b093b85a9914%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscr...@googlegroups.com. To post to this group, send email to grpc-io@googlegroups.com. Visit this group at https://groups.google.com/group/grpc-io. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CALUXJ7jDjmTHD3KEfHc4egsKk5o29pS20Mf8YU22x1zAgAHO_A%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
