Hi Jeff and Mark, I just ran into the same issue with an async c++ GRPC server (version 1.37.1), was curious about these default-executo threads and then got this thread, did you guys figure out what these threads are for? The number seems to be about 2x of the polling worker threads.
Thanks! On Friday, January 7, 2022 at 3:47:51 PM UTC-5 Jeff Steger wrote: > Thanks Mark, I will turn on trace and see if I see anything odd. I was > reading about a function called Executor::SetThreadingDefault(bool enable) > that I think I can safely call after i create my grpc server. It is a > public function and seems to allow me to toggle between a threaded > implementation and an async one. Is that accurate? Is calling this function > safe to do and/or recommended (or at least not contra-recommended). Thanks > again for your help! > > Jeff > > > > On Fri, Jan 7, 2022 at 11:14 AM Mark D. Roth <ro...@google.com> wrote: > >> Oh, sorry, I thought you were asking about the sync server threads. The >> default-executor threads sound like threads that are spawned internally >> inside of C-core for things like synchronous DNS resolution; those should >> be completely unrelated to the sync server threads. I'm not sure what >> would cause those threads to pile up. >> >> Try running with the env vars GRPC_VERBOSITY=DEBUG GRPC_TRACE=executor >> and see if that yields any useful log information. In particular, try >> running that with a debug build, since that will add additional information >> about where in the code the closures on the executor threads are coming >> from. >> >> On Thu, Jan 6, 2022 at 7:05 PM Jeff Steger <be2...@gmail.com> wrote: >> >>> >>> Thanks for the info! It sort of sounds like the default-executor threads >>> and the sync-server threads come from the same thread pool. I know that >>> ServerBuilder::SetResourceQuota sets the max number sync-server threads. >>> Does it have any impact on the number of default-executor threads? It >>> doesn't seem to. >>> >>> Also, can you tell me under what conditions a process can expect to wind >>> up with dozens of sleeping default-executor threads? My process doesn't >>> start off like that but after a few hours thats how it winds up. It is not >>> under high load, just serving a few short-lived streaming-response queries >>> every few seconds. If I limit the max number of these threads to lets say >>> 10, do you anticipate that connections would be refused under >>> these conditions (ie serving a few short-lived streaming-response queries >>> every few seconds)? >>> >>> Last, is it safe to manually kill these sleeping threads? They seem to >>> be blocked on a condition variable. >>> >>> Thanks! >>> >>> -Jeff >>> >>> On Thu, Jan 6, 2022 at 3:39 PM Mark D. Roth <ro...@google.com> wrote: >>> >>>> The C++ sync server has one thread pool for both polling and request >>>> handlers. When a request comes in, an existing polling thread basically >>>> becomes a request handler thread, and when the request handler completes, >>>> that thread is available to become a polling thread again. The MIN_POLLERS >>>> and MAX_POLLERS options >>>> <https://grpc.github.io/grpc/cpp/classgrpc_1_1_server_builder.html#aff66bd93cba7d4240a64550fe1fca88d> >>>> >>>> (which can be set via ServerBuilder::SetSyncServerOption() >>>> <https://grpc.github.io/grpc/cpp/classgrpc_1_1_server_builder.html#acbc2a859672203e7be097de55e296d4b>) >>>> >>>> allow tuning the number of threads that are used for polling: when a >>>> polling thread becomes a request handler thread, if there are not enough >>>> polling threads remaining, a new one will be spawned, and when a request >>>> handler finishes, if there are too many polling threads, the thread will >>>> terminate. >>>> >>>> On Wed, Jan 5, 2022 at 12:54 PM Jeff Steger <be2...@gmail.com> wrote: >>>> >>>>> Ah never mind I see you answered, apologies. Let me ask you this: am I >>>>> stuck with all of these default-executor threads that my process is >>>>> spawning? Is there no way to limit them? Do they come from same pool as >>>>> grpc sync server threads? >>>>> >>>>> On Wed, Jan 5, 2022 at 3:51 PM Jeff Steger <be2...@gmail.com> wrote: >>>>> >>>>>> Can you specifically answer this: >>>>>> >>>>>> grpc-java has a method in its ServerBuilder class to set the >>>>>> Executor. Is there similar functionality for grpc-c++ ? >>>>>> >>>>>> Thanks! >>>>>> >>>>>> On Tue, Jan 4, 2022 at 11:55 AM Mark D. Roth <ro...@google.com> >>>>>> wrote: >>>>>> >>>>>>> I answered this in the other thread you posted on. >>>>>>> >>>>>>> On Sun, Jan 2, 2022 at 9:39 AM Jeff Steger <be2...@gmail.com> wrote: >>>>>>> >>>>>>>> grpc-java has a method in its ServerBuilder class to set the >>>>>>>> Executor. Is there similar functionality for grpc-c++ ? I am running a >>>>>>>> C++ >>>>>>>> grpc server and the number of executor threads it spawns is high and >>>>>>>> seems >>>>>>>> to never decrease, even when connections stop. >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "grpc.io" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to grpc-io+u...@googlegroups.com. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/grpc-io/CAA-WHunWvX5Tr6Vp3e-E6vcKgzD%3DGzsCNoZzqNNQ8Ox8BZvggA%40mail.gmail.com >>>>>>>> >>>>>>>> <https://groups.google.com/d/msgid/grpc-io/CAA-WHunWvX5Tr6Vp3e-E6vcKgzD%3DGzsCNoZzqNNQ8Ox8BZvggA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Mark D. Roth <ro...@google.com> >>>>>>> Software Engineer >>>>>>> Google, Inc. >>>>>>> >>>>>> >>>> >>>> -- >>>> Mark D. Roth <ro...@google.com> >>>> Software Engineer >>>> Google, Inc. >>>> >>> >> >> -- >> Mark D. Roth <ro...@google.com> >> Software Engineer >> Google, Inc. >> > -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/3ef87e52-f06e-492f-aecd-b6a58b3bd654n%40googlegroups.com.
