thanks Mark, parallelizing is not an option for my project. As I initially mentioned this project is conversion of CORBA IPC into GRPC whereas CORBA able to complete 1000 RPCs in 1 second vs 10 seconds in GRPC.
I will send GRPC logs with TCP enabled soon; If you prefer wireshark, i need to learn the tool usage and send it later. On Friday, September 3, 2021 at 9:58:02 PM UTC+5:30 Mark D. Roth wrote: > I don't see anything obviously wrong with your code. > > Since this test is sending RPCs serially instead of in parallel, it's > possible that there are too many network round-trips happening here, each > one of which would increase latency because the next operation is blocking > on the previous one. Can you try running with the environment variables > GRPC_VERBOSITY=DEBUG > GRPC_TRACE=tcp? Or, alternatively, getting a wireshark capture of the > network communication? That might help us see how many round-trips are > happening here. > > You might also consider whether sending a bunch of RPCs serially is > actually a realistic benchmark for your production workload. You might get > better performance by parallelizing the requests from the client. > > On Fri, Sep 3, 2021 at 2:29 AM Sureshbabu Seshadri <sureshb...@gmail.com> > wrote: > >> Thanks Mark. The below link has source code of my sample, please let me >> know if you need any other information to analyze >> >> >> https://drive.google.com/file/d/12PH65OYwflaPBpa2a-yMcBqSE3xYX9S-/view?usp=sharing >> >> >> On Thursday, September 2, 2021 at 3:45:04 AM UTC+5:30 Mark D. Roth wrote: >> >>> I'm so sorry for not responding sooner! For some reason, gmail >>> tagged your messages as spam, so I didn't see them. :( >>> >>> On Fri, Aug 27, 2021 at 10:55 PM Sureshbabu Seshadri < >>> sureshb...@gmail.com> wrote: >>> >>>> Dear GRPC team, >>>> Can any one help on this? >>>> >>>> On Friday, August 13, 2021 at 12:53:21 PM UTC+5:30 Sureshbabu Seshadri >>>> wrote: >>>> >>>>> Mark, >>>>> Please find the grpc ttrace logs in the following link >>>>> https://drive.google.com/file/d/15y7KzyCtIeAoYSUzyPHpY4gcr7uUnIP0/view?usp=sharing >>>>> >>>>> I am not able to upload files directly here. Please note that the >>>>> profiling is done for same API called in loop for 1000 times and let me >>>>> know. >>>>> >>>>> On Thursday, August 12, 2021 at 11:27:16 AM UTC+5:30 Sureshbabu >>>>> Seshadri wrote: >>>>> >>>>>> Thanks Mark, my current profile does not include channel creation >>>>>> time. Profiling is only applicable for RPC calls. >>>>> >>>>> >>> Note that when you first create a gRPC channel, it does not actually do >>> any name resolution or connect to any servers until you either explicitly >>> tell it to do so (such as by calling channel->WaitForConnected(gpr_ >>> inf_future(GPR_CLOCK_MONOTONIC))) or send the first RPC on it. So if >>> you don't proactively tell the channel to connect but start counting the >>> elapsed time right before you send the first RPC, then you are actually >>> including the channel connection time in your benchmark. >>> >>> From the trace log above, though, it seems clear that the problem you're >>> seeing here is not actually channel startup time. The channel starts to >>> connect on this line: >>> >>> I0812 21:30:17.760000000 5748 resolving_lb_policy.cc:161] >>> resolving_lb=000001EBA08F00D0: starting name resolution >>> >>> >>> And it finishes connecting here: >>> >>> I0812 21:30:17.903000000 44244 client_channel.cc:1362] >>> chand=000001EBA08F35B0: update: state=READY picker=000001EBA0900B70 >>> >>> >>> So it took the channel only 0.143 seconds to get connected, which means >>> that's probably not the problem you're seeing here. >>> >>> Once it did get connected, it looks like it took about 8 seconds to >>> process 1000 RPCs, which does seem quite slow. >>> >>> Can you share the code you're using for the client and server? >>> >>> >>> >>>> We have an existing code base for IPC which uses CORBA architecture and >>>>>> we are trying to replace it with GRPC, similar sample in CORBA completes >>>>>> quickly that is 1000 RPCs are completed within 2 seconds in same >>>>>> network. >>>>>> Hence this is kind of roadblock for our migration. >>>>>> >>>>>> I will execute test with traces enabled and share the logs ASAP >>>>>> >>>>>> On Wednesday, August 11, 2021 at 10:38:48 PM UTC+5:30 Mark D. Roth >>>>>> wrote: >>>>>> >>>>>>> You can check to see whether the problem is a channel startup >>>>>>> problem or a latency problem by calling >>>>>>> channel->WaitForConnected(gpr_inf_future(GPR_CLOCK_MONOTONIC)) >>>>>>> before you start sending RPCs on the channel. That call won't return >>>>>>> until >>>>>>> the channel has completed the DNS lookup and established a connection >>>>>>> to >>>>>>> the server, so if you start timing after that, your timing will exclude >>>>>>> the >>>>>>> channel startup time. >>>>>>> >>>>>>> If you see that the channel startup time is high but the RPCs flow >>>>>>> quickly once the startup time is passed, then the problem is either the >>>>>>> DNS >>>>>>> lookup or establishing a connection to the server. In that case, >>>>>>> please >>>>>>> try running with the environment variables GRPC_VERBOSITY=DEBUG >>>>>>> GRPC_TRACE=client_channel_routing,pick_first and share the log, so >>>>>>> that we can help you figure out which one is the problem. >>>>>>> >>>>>>> If you see that the channel startup time is not that high but that >>>>>>> the RPCs are actually flowing more slowly over the network, then the >>>>>>> problem might be network congestion of some sort. >>>>>>> >>>>>>> Also, if the problem does turn out to be channel startup time, note >>>>>>> that it probably won't matter much in practice, as long as your >>>>>>> application >>>>>>> creates the channel once and reuses it for all of its RPCs. We do not >>>>>>> recommend a pattern where you create a channel, send a bunch of RPCs, >>>>>>> then >>>>>>> destroy the channel, and then do that whole thing again later when you >>>>>>> need >>>>>>> to send more RPCs. >>>>>>> >>>>>>> I hope this information is helpful. >>>>>>> On Sunday, August 8, 2021 at 9:34:43 AM UTC-7 sureshb...@gmail.com >>>>>>> wrote: >>>>>>> >>>>>>>> *Environment* >>>>>>>> >>>>>>>> 1. Both client and server are C++ >>>>>>>> 2. Server might be running either locally or in different system >>>>>>>> 3. In case of remote server, it is in same network. >>>>>>>> 4. Using SYNC C++ server >>>>>>>> 5. Unary RPC >>>>>>>> >>>>>>>> Our performance numbers are very low for running 1000 RPC calls >>>>>>>> (continuous calls through loop for testing) it takes about 10 seconds >>>>>>>> when >>>>>>>> server running in different PC. >>>>>>>> >>>>>>>> The client creates channel using *hostname:portnumber *method and >>>>>>>> using this approach the local server were also taking similar 10 >>>>>>>> seconds >>>>>>>> for 1000 calls. Later we modified channel creation for local server by >>>>>>>> using *localhost:port *then it was much improved performance, all >>>>>>>> the 1000 calls completed within 300 ms. >>>>>>>> >>>>>>>> Based on the above test, we strongly believe DNS resolution seems >>>>>>>> to cause slow performance as change hostname to localhost results in >>>>>>>> huge >>>>>>>> performance gain, however that is not possible for servers running on >>>>>>>> different PC. >>>>>>>> >>>>>>>> Can someone help with this? Is DNS the real culprit or what else >>>>>>>> can be changed to get good performance throughput in this case. >>>>>>>> >>>>>>>> Please let me know if there any other input required for this. >>>>>>>> >>>>>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "grpc.io" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/grpc-io/pD3HiDvxymY/unsubscribe. >>>> To unsubscribe from this group and all its topics, send an email to >>>> grpc-io+u...@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/grpc-io/66ff211a-e86f-47ff-9583-b423877b8f02n%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/grpc-io/66ff211a-e86f-47ff-9583-b423877b8f02n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> >>> >>> -- >>> Mark D. Roth <ro...@google.com> >>> Software Engineer >>> Google, Inc. >>> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "grpc.io" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/grpc-io/pD3HiDvxymY/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> grpc-io+u...@googlegroups.com. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/grpc-io/f010b3e1-8554-4c39-a086-a0741e4f7d12n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/grpc-io/f010b3e1-8554-4c39-a086-a0741e4f7d12n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > > > -- > Mark D. Roth <ro...@google.com> > Software Engineer > Google, Inc. > -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/50c98f17-fdee-4e33-bbc1-dbd4d77897fdn%40googlegroups.com.
