Thanks Mark. The below link has source code of my sample, please let me know if you need any other information to analyze
https://drive.google.com/file/d/12PH65OYwflaPBpa2a-yMcBqSE3xYX9S-/view?usp=sharing On Thursday, September 2, 2021 at 3:45:04 AM UTC+5:30 Mark D. Roth wrote: > I'm so sorry for not responding sooner! For some reason, gmail > tagged your messages as spam, so I didn't see them. :( > > On Fri, Aug 27, 2021 at 10:55 PM Sureshbabu Seshadri <sureshb...@gmail.com> > wrote: > >> Dear GRPC team, >> Can any one help on this? >> >> On Friday, August 13, 2021 at 12:53:21 PM UTC+5:30 Sureshbabu Seshadri >> wrote: >> >>> Mark, >>> Please find the grpc ttrace logs in the following link >>> https://drive.google.com/file/d/15y7KzyCtIeAoYSUzyPHpY4gcr7uUnIP0/view?usp=sharing >>> >>> I am not able to upload files directly here. Please note that the >>> profiling is done for same API called in loop for 1000 times and let me >>> know. >>> >>> On Thursday, August 12, 2021 at 11:27:16 AM UTC+5:30 Sureshbabu Seshadri >>> wrote: >>> >>>> Thanks Mark, my current profile does not include channel creation time. >>>> Profiling is only applicable for RPC calls. >>> >>> > Note that when you first create a gRPC channel, it does not actually do > any name resolution or connect to any servers until you either explicitly > tell it to do so (such as by calling channel->WaitForConnected(gpr_ > inf_future(GPR_CLOCK_MONOTONIC))) or send the first RPC on it. So if you > don't proactively tell the channel to connect but start counting the > elapsed time right before you send the first RPC, then you are actually > including the channel connection time in your benchmark. > > From the trace log above, though, it seems clear that the problem you're > seeing here is not actually channel startup time. The channel starts to > connect on this line: > > I0812 21:30:17.760000000 5748 resolving_lb_policy.cc:161] > resolving_lb=000001EBA08F00D0: starting name resolution > > > And it finishes connecting here: > > I0812 21:30:17.903000000 44244 client_channel.cc:1362] > chand=000001EBA08F35B0: update: state=READY picker=000001EBA0900B70 > > > So it took the channel only 0.143 seconds to get connected, which means > that's probably not the problem you're seeing here. > > Once it did get connected, it looks like it took about 8 seconds to > process 1000 RPCs, which does seem quite slow. > > Can you share the code you're using for the client and server? > > > >> We have an existing code base for IPC which uses CORBA architecture and >>>> we are trying to replace it with GRPC, similar sample in CORBA completes >>>> quickly that is 1000 RPCs are completed within 2 seconds in same network. >>>> Hence this is kind of roadblock for our migration. >>>> >>>> I will execute test with traces enabled and share the logs ASAP >>>> >>>> On Wednesday, August 11, 2021 at 10:38:48 PM UTC+5:30 Mark D. Roth >>>> wrote: >>>> >>>>> You can check to see whether the problem is a channel startup problem >>>>> or a latency problem by calling >>>>> channel->WaitForConnected(gpr_inf_future(GPR_CLOCK_MONOTONIC)) before >>>>> you start sending RPCs on the channel. That call won't return until the >>>>> channel has completed the DNS lookup and established a connection to the >>>>> server, so if you start timing after that, your timing will exclude the >>>>> channel startup time. >>>>> >>>>> If you see that the channel startup time is high but the RPCs flow >>>>> quickly once the startup time is passed, then the problem is either the >>>>> DNS >>>>> lookup or establishing a connection to the server. In that case, please >>>>> try running with the environment variables GRPC_VERBOSITY=DEBUG >>>>> GRPC_TRACE=client_channel_routing,pick_first and share the log, so >>>>> that we can help you figure out which one is the problem. >>>>> >>>>> If you see that the channel startup time is not that high but that the >>>>> RPCs are actually flowing more slowly over the network, then the problem >>>>> might be network congestion of some sort. >>>>> >>>>> Also, if the problem does turn out to be channel startup time, note >>>>> that it probably won't matter much in practice, as long as your >>>>> application >>>>> creates the channel once and reuses it for all of its RPCs. We do not >>>>> recommend a pattern where you create a channel, send a bunch of RPCs, >>>>> then >>>>> destroy the channel, and then do that whole thing again later when you >>>>> need >>>>> to send more RPCs. >>>>> >>>>> I hope this information is helpful. >>>>> On Sunday, August 8, 2021 at 9:34:43 AM UTC-7 sureshb...@gmail.com >>>>> wrote: >>>>> >>>>>> *Environment* >>>>>> >>>>>> 1. Both client and server are C++ >>>>>> 2. Server might be running either locally or in different system >>>>>> 3. In case of remote server, it is in same network. >>>>>> 4. Using SYNC C++ server >>>>>> 5. Unary RPC >>>>>> >>>>>> Our performance numbers are very low for running 1000 RPC calls >>>>>> (continuous calls through loop for testing) it takes about 10 seconds >>>>>> when >>>>>> server running in different PC. >>>>>> >>>>>> The client creates channel using *hostname:portnumber *method and >>>>>> using this approach the local server were also taking similar 10 seconds >>>>>> for 1000 calls. Later we modified channel creation for local server by >>>>>> using *localhost:port *then it was much improved performance, all >>>>>> the 1000 calls completed within 300 ms. >>>>>> >>>>>> Based on the above test, we strongly believe DNS resolution seems to >>>>>> cause slow performance as change hostname to localhost results in huge >>>>>> performance gain, however that is not possible for servers running on >>>>>> different PC. >>>>>> >>>>>> Can someone help with this? Is DNS the real culprit or what else can >>>>>> be changed to get good performance throughput in this case. >>>>>> >>>>>> Please let me know if there any other input required for this. >>>>>> >>>>> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "grpc.io" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/grpc-io/pD3HiDvxymY/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> grpc-io+u...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/grpc-io/66ff211a-e86f-47ff-9583-b423877b8f02n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/grpc-io/66ff211a-e86f-47ff-9583-b423877b8f02n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > > > -- > Mark D. Roth <ro...@google.com> > Software Engineer > Google, Inc. > -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/f010b3e1-8554-4c39-a086-a0741e4f7d12n%40googlegroups.com.
