I'm so sorry for not responding sooner! For some reason, gmail tagged your messages as spam, so I didn't see them. :(
On Fri, Aug 27, 2021 at 10:55 PM Sureshbabu Seshadri < sureshbabu8...@gmail.com> wrote: > Dear GRPC team, > Can any one help on this? > > On Friday, August 13, 2021 at 12:53:21 PM UTC+5:30 Sureshbabu Seshadri > wrote: > >> Mark, >> Please find the grpc ttrace logs in the following link >> https://drive.google.com/file/d/15y7KzyCtIeAoYSUzyPHpY4gcr7uUnIP0/view?usp=sharing >> >> I am not able to upload files directly here. Please note that the >> profiling is done for same API called in loop for 1000 times and let me >> know. >> >> On Thursday, August 12, 2021 at 11:27:16 AM UTC+5:30 Sureshbabu Seshadri >> wrote: >> >>> Thanks Mark, my current profile does not include channel creation time. >>> Profiling is only applicable for RPC calls. >> >> Note that when you first create a gRPC channel, it does not actually do any name resolution or connect to any servers until you either explicitly tell it to do so (such as by calling channel->WaitForConnected(gpr_ inf_future(GPR_CLOCK_MONOTONIC))) or send the first RPC on it. So if you don't proactively tell the channel to connect but start counting the elapsed time right before you send the first RPC, then you are actually including the channel connection time in your benchmark. >From the trace log above, though, it seems clear that the problem you're seeing here is not actually channel startup time. The channel starts to connect on this line: I0812 21:30:17.760000000 5748 resolving_lb_policy.cc:161] resolving_lb=000001EBA08F00D0: starting name resolution And it finishes connecting here: I0812 21:30:17.903000000 44244 client_channel.cc:1362] chand=000001EBA08F35B0: update: state=READY picker=000001EBA0900B70 So it took the channel only 0.143 seconds to get connected, which means that's probably not the problem you're seeing here. Once it did get connected, it looks like it took about 8 seconds to process 1000 RPCs, which does seem quite slow. Can you share the code you're using for the client and server? > We have an existing code base for IPC which uses CORBA architecture and we >>> are trying to replace it with GRPC, similar sample in CORBA completes >>> quickly that is 1000 RPCs are completed within 2 seconds in same network. >>> Hence this is kind of roadblock for our migration. >>> >>> I will execute test with traces enabled and share the logs ASAP >>> >>> On Wednesday, August 11, 2021 at 10:38:48 PM UTC+5:30 Mark D. Roth wrote: >>> >>>> You can check to see whether the problem is a channel startup problem >>>> or a latency problem by calling >>>> channel->WaitForConnected(gpr_inf_future(GPR_CLOCK_MONOTONIC)) before >>>> you start sending RPCs on the channel. That call won't return until the >>>> channel has completed the DNS lookup and established a connection to the >>>> server, so if you start timing after that, your timing will exclude the >>>> channel startup time. >>>> >>>> If you see that the channel startup time is high but the RPCs flow >>>> quickly once the startup time is passed, then the problem is either the DNS >>>> lookup or establishing a connection to the server. In that case, please >>>> try running with the environment variables GRPC_VERBOSITY=DEBUG >>>> GRPC_TRACE=client_channel_routing,pick_first and share the log, so >>>> that we can help you figure out which one is the problem. >>>> >>>> If you see that the channel startup time is not that high but that the >>>> RPCs are actually flowing more slowly over the network, then the problem >>>> might be network congestion of some sort. >>>> >>>> Also, if the problem does turn out to be channel startup time, note >>>> that it probably won't matter much in practice, as long as your application >>>> creates the channel once and reuses it for all of its RPCs. We do not >>>> recommend a pattern where you create a channel, send a bunch of RPCs, then >>>> destroy the channel, and then do that whole thing again later when you need >>>> to send more RPCs. >>>> >>>> I hope this information is helpful. >>>> On Sunday, August 8, 2021 at 9:34:43 AM UTC-7 sureshb...@gmail.com >>>> wrote: >>>> >>>>> *Environment* >>>>> >>>>> 1. Both client and server are C++ >>>>> 2. Server might be running either locally or in different system >>>>> 3. In case of remote server, it is in same network. >>>>> 4. Using SYNC C++ server >>>>> 5. Unary RPC >>>>> >>>>> Our performance numbers are very low for running 1000 RPC calls >>>>> (continuous calls through loop for testing) it takes about 10 seconds when >>>>> server running in different PC. >>>>> >>>>> The client creates channel using *hostname:portnumber *method and >>>>> using this approach the local server were also taking similar 10 seconds >>>>> for 1000 calls. Later we modified channel creation for local server by >>>>> using *localhost:port *then it was much improved performance, all the >>>>> 1000 calls completed within 300 ms. >>>>> >>>>> Based on the above test, we strongly believe DNS resolution seems to >>>>> cause slow performance as change hostname to localhost results in huge >>>>> performance gain, however that is not possible for servers running on >>>>> different PC. >>>>> >>>>> Can someone help with this? Is DNS the real culprit or what else can >>>>> be changed to get good performance throughput in this case. >>>>> >>>>> Please let me know if there any other input required for this. >>>>> >>>> -- > You received this message because you are subscribed to a topic in the > Google Groups "grpc.io" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/grpc-io/pD3HiDvxymY/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > grpc-io+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/grpc-io/66ff211a-e86f-47ff-9583-b423877b8f02n%40googlegroups.com > <https://groups.google.com/d/msgid/grpc-io/66ff211a-e86f-47ff-9583-b423877b8f02n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- Mark D. Roth <r...@google.com> Software Engineer Google, Inc. -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CAJgPXp4T5X5p25vmjmh3Mo_k2Z4S%3DDyej3_XR6MfsTzw2K1iPg%40mail.gmail.com.
