I don't see anything obviously wrong with your code. Since this test is sending RPCs serially instead of in parallel, it's possible that there are too many network round-trips happening here, each one of which would increase latency because the next operation is blocking on the previous one. Can you try running with the environment variables GRPC_VERBOSITY=DEBUG GRPC_TRACE=tcp? Or, alternatively, getting a wireshark capture of the network communication? That might help us see how many round-trips are happening here.
You might also consider whether sending a bunch of RPCs serially is actually a realistic benchmark for your production workload. You might get better performance by parallelizing the requests from the client. On Fri, Sep 3, 2021 at 2:29 AM Sureshbabu Seshadri <sureshbabu8...@gmail.com> wrote: > Thanks Mark. The below link has source code of my sample, please let me > know if you need any other information to analyze > > > https://drive.google.com/file/d/12PH65OYwflaPBpa2a-yMcBqSE3xYX9S-/view?usp=sharing > > > On Thursday, September 2, 2021 at 3:45:04 AM UTC+5:30 Mark D. Roth wrote: > >> I'm so sorry for not responding sooner! For some reason, gmail >> tagged your messages as spam, so I didn't see them. :( >> >> On Fri, Aug 27, 2021 at 10:55 PM Sureshbabu Seshadri < >> sureshb...@gmail.com> wrote: >> >>> Dear GRPC team, >>> Can any one help on this? >>> >>> On Friday, August 13, 2021 at 12:53:21 PM UTC+5:30 Sureshbabu Seshadri >>> wrote: >>> >>>> Mark, >>>> Please find the grpc ttrace logs in the following link >>>> https://drive.google.com/file/d/15y7KzyCtIeAoYSUzyPHpY4gcr7uUnIP0/view?usp=sharing >>>> >>>> I am not able to upload files directly here. Please note that the >>>> profiling is done for same API called in loop for 1000 times and let me >>>> know. >>>> >>>> On Thursday, August 12, 2021 at 11:27:16 AM UTC+5:30 Sureshbabu >>>> Seshadri wrote: >>>> >>>>> Thanks Mark, my current profile does not include channel creation >>>>> time. Profiling is only applicable for RPC calls. >>>> >>>> >> Note that when you first create a gRPC channel, it does not actually do >> any name resolution or connect to any servers until you either explicitly >> tell it to do so (such as by calling channel->WaitForConnected(gpr_ >> inf_future(GPR_CLOCK_MONOTONIC))) or send the first RPC on it. So if >> you don't proactively tell the channel to connect but start counting the >> elapsed time right before you send the first RPC, then you are actually >> including the channel connection time in your benchmark. >> >> From the trace log above, though, it seems clear that the problem you're >> seeing here is not actually channel startup time. The channel starts to >> connect on this line: >> >> I0812 21:30:17.760000000 5748 resolving_lb_policy.cc:161] >> resolving_lb=000001EBA08F00D0: starting name resolution >> >> >> And it finishes connecting here: >> >> I0812 21:30:17.903000000 44244 client_channel.cc:1362] >> chand=000001EBA08F35B0: update: state=READY picker=000001EBA0900B70 >> >> >> So it took the channel only 0.143 seconds to get connected, which means >> that's probably not the problem you're seeing here. >> >> Once it did get connected, it looks like it took about 8 seconds to >> process 1000 RPCs, which does seem quite slow. >> >> Can you share the code you're using for the client and server? >> >> >> >>> We have an existing code base for IPC which uses CORBA architecture and >>>>> we are trying to replace it with GRPC, similar sample in CORBA completes >>>>> quickly that is 1000 RPCs are completed within 2 seconds in same network. >>>>> Hence this is kind of roadblock for our migration. >>>>> >>>>> I will execute test with traces enabled and share the logs ASAP >>>>> >>>>> On Wednesday, August 11, 2021 at 10:38:48 PM UTC+5:30 Mark D. Roth >>>>> wrote: >>>>> >>>>>> You can check to see whether the problem is a channel startup problem >>>>>> or a latency problem by calling >>>>>> channel->WaitForConnected(gpr_inf_future(GPR_CLOCK_MONOTONIC)) >>>>>> before you start sending RPCs on the channel. That call won't return >>>>>> until >>>>>> the channel has completed the DNS lookup and established a connection to >>>>>> the server, so if you start timing after that, your timing will exclude >>>>>> the >>>>>> channel startup time. >>>>>> >>>>>> If you see that the channel startup time is high but the RPCs flow >>>>>> quickly once the startup time is passed, then the problem is either the >>>>>> DNS >>>>>> lookup or establishing a connection to the server. In that case, please >>>>>> try running with the environment variables GRPC_VERBOSITY=DEBUG >>>>>> GRPC_TRACE=client_channel_routing,pick_first and share the log, so >>>>>> that we can help you figure out which one is the problem. >>>>>> >>>>>> If you see that the channel startup time is not that high but that >>>>>> the RPCs are actually flowing more slowly over the network, then the >>>>>> problem might be network congestion of some sort. >>>>>> >>>>>> Also, if the problem does turn out to be channel startup time, note >>>>>> that it probably won't matter much in practice, as long as your >>>>>> application >>>>>> creates the channel once and reuses it for all of its RPCs. We do not >>>>>> recommend a pattern where you create a channel, send a bunch of RPCs, >>>>>> then >>>>>> destroy the channel, and then do that whole thing again later when you >>>>>> need >>>>>> to send more RPCs. >>>>>> >>>>>> I hope this information is helpful. >>>>>> On Sunday, August 8, 2021 at 9:34:43 AM UTC-7 sureshb...@gmail.com >>>>>> wrote: >>>>>> >>>>>>> *Environment* >>>>>>> >>>>>>> 1. Both client and server are C++ >>>>>>> 2. Server might be running either locally or in different system >>>>>>> 3. In case of remote server, it is in same network. >>>>>>> 4. Using SYNC C++ server >>>>>>> 5. Unary RPC >>>>>>> >>>>>>> Our performance numbers are very low for running 1000 RPC calls >>>>>>> (continuous calls through loop for testing) it takes about 10 seconds >>>>>>> when >>>>>>> server running in different PC. >>>>>>> >>>>>>> The client creates channel using *hostname:portnumber *method and >>>>>>> using this approach the local server were also taking similar 10 seconds >>>>>>> for 1000 calls. Later we modified channel creation for local server by >>>>>>> using *localhost:port *then it was much improved performance, all >>>>>>> the 1000 calls completed within 300 ms. >>>>>>> >>>>>>> Based on the above test, we strongly believe DNS resolution seems to >>>>>>> cause slow performance as change hostname to localhost results in huge >>>>>>> performance gain, however that is not possible for servers running on >>>>>>> different PC. >>>>>>> >>>>>>> Can someone help with this? Is DNS the real culprit or what else can >>>>>>> be changed to get good performance throughput in this case. >>>>>>> >>>>>>> Please let me know if there any other input required for this. >>>>>>> >>>>>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "grpc.io" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/grpc-io/pD3HiDvxymY/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> grpc-io+u...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/grpc-io/66ff211a-e86f-47ff-9583-b423877b8f02n%40googlegroups.com >>> <https://groups.google.com/d/msgid/grpc-io/66ff211a-e86f-47ff-9583-b423877b8f02n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> >> >> -- >> Mark D. Roth <ro...@google.com> >> Software Engineer >> Google, Inc. >> > -- > You received this message because you are subscribed to a topic in the > Google Groups "grpc.io" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/grpc-io/pD3HiDvxymY/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > grpc-io+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/grpc-io/f010b3e1-8554-4c39-a086-a0741e4f7d12n%40googlegroups.com > <https://groups.google.com/d/msgid/grpc-io/f010b3e1-8554-4c39-a086-a0741e4f7d12n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- Mark D. Roth <r...@google.com> Software Engineer Google, Inc. -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CAJgPXp7Q8HczxrwJomC77GWVzmBC8iQEUhBsUQDazi39cozJbw%40mail.gmail.com.
