Yes Sanjay your understanding is correct. We solved this by
a) Client sends some dummy request to server every 1 min. This is a actual request defined in proto buf by passing some type like dummy request. b) On reception of every such above request, server responds back with a dummy response, which client ignores based on request type like dummy response. Earlier we were not doing part-b, only part-a was there and server was just ignoring it. Now issue is solved completely, when we implemented part-b as well. Regards, Rajat On Tue, 17 May, 2022, 8:18 am Sanjay Pujare, <sanjaypuj...@google.com> wrote: > Comments inline below: > > On Mon, May 16, 2022 at 8:35 AM Rajat Goyal <rajatgoyal247...@gmail.com> > wrote: > >> Hi Sanjay / Grpc team, >> >> I have implemented RPC based regular pings. Like I am sending >> some dummy request each minute to LB once connected from client side. >> I observed below : >> a) This method works fine if there is no request from the server >> side. This way the connection is alive for many hours without issues. >> > > "request from the server side": just want to clarify what is being said. > You mean a response message since a server can only send response messages > back. Or did you mean server sending requests on a different connection to > another server? > > > >> b) But this method doesn't work if there is some bi-directional >> request from the server. The moment it receives a response from server, the >> connections is dropped from LB after exactly 5 mins, even if I am sending a >> regular dummy request from the client side every minute. >> > > Again I would like to clarify your "bi-directional request from the > server" : do you mean a bi-di RPC into the server where server is sending > responses to the client? And in such a case the LB drops the connection > after 5 mins? > > >> >> I also checked the server logs the dummy request is being received every >> min, which means client is sending regular 1 min ping, but still LB is >> dropping the connection. >> While in case there is no response from server side, the connection is >> not dropped by LB. >> >> Can you please help me what grpc / LB might be doing in both cases ? >> > > To summarize my understanding of what you are. saying: when a connection > is established through the ALB to a gRPC backend the connection stays alive > indefinitely if the client only sends a dummy RPC every minute. This dummy > RPC has a dummy request but no response (only header(s) and status code). > As soon as you send any real RPCs where the server sends any response > messages then the LB does not keep the connection alive but drops it 5 > minutes after the last non-dummy RPC. Is this correct? > > > >> >> Regards, >> Rajat >> >> >> On Tue, Jan 11, 2022 at 4:01 AM Sanjay Pujare <sanjaypuj...@google.com> >> wrote: >> >>> Check this >>> https://stackoverflow.com/questions/66818645/http2-ping-frames-over-aws-alb-grpc-keepalive-ping >>> >>> "*ALB does not support the HTTP2 ping frames*." >>> >>> On Mon, Jan 10, 2022 at 12:16 PM Rajat Goyal <rajatgoyal247...@gmail.com> >>> wrote: >>> >>>> ALB is configured with idle-timeout - 5 minutes. >>>> I configured bi-di client with : >>>> keepAliveWithoutCalls(true).keepAliveTime(90, >>>> TimeUnit.SECONDS).keepAliveTimeout(10, TimeUnit.SECONDS) >>>> while server is configured with : >>>> permitKeepAliveWithoutCalls(true).permitKeepAliveTime(1, TimeUnit.MINUTES) >>>> >>>> But I received INTERNAL: HTTP/2 error code: PROTOCOL_ERROR Received Rst >>>> Stream after exactly 5 minutes. Which looks like ALB has dropped the >>>> connection after 5 minutes. >>>> >>>> Any idea how we can keep idle connection alive ? >>>> >>>> >>>> >>>> On Mon, Jan 10, 2022 at 10:39 PM Rajat Goyal < >>>> rajatgoyal247...@gmail.com> wrote: >>>> >>>>> Hi Sanjay, >>>>> >>>>> I see that bi-directional streamObserver object gets call back >>>>> onError() in case of any error in network. >>>>> >>>>> Isn't that done by any heartbeat mechanism already?. If so, then >>>>> connection at ALB should be active with these ping-pong packets ? >>>>> >>>>> Regards, >>>>> Rajat >>>>> >>>>> On Mon, 10 Jan, 2022, 10:33 pm Sanjay Pujare, <sanjaypuj...@google.com> >>>>> wrote: >>>>> >>>>>> This may probably help? >>>>>> https://grpc.io/blog/grpc-on-http2/#keeping-connections-alive ? >>>>>> >>>>>> On Mon, Jan 10, 2022 at 8:54 AM Rajat Goyal < >>>>>> rajatgoyal247...@gmail.com> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Gentle reminder for any resolution for above. >>>>>>> >>>>>>> Regards, >>>>>>> Rajat >>>>>>> >>>>>>> On Sun, 9 Jan, 2022, 6:50 pm Rajat Goyal, < >>>>>>> rajatgoyal247...@gmail.com> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> We have a system where clients open bi-directional grpc stream >>>>>>>> to ALB, which proxies to one of active server. So >>>>>>>> >>>>>>>> bi-di >>>>>>>> client <----------------> ALB <----------------> server >>>>>>>> >>>>>>>> In-case of any failure of connection, clients re-connects to us as >>>>>>>> we want to keep a bi-di channel open. >>>>>>>> >>>>>>>> Question is : How can we keep the channel open even if there is no >>>>>>>> activity for sometime. ALB are configured with 300 sec idle-timeout >>>>>>>> which >>>>>>>> means it will drop the connection if no packets are exchanged in 300 >>>>>>>> sec. >>>>>>>> >>>>>>>> As we want to keep the connection open as much possible ( only >>>>>>>> re-create in-case of any issue), and not let it die due to idle >>>>>>>> timeout, >>>>>>>> what properties should server can client set ? >>>>>>>> Should keep-alive setting at both client & server help out ? >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "grpc.io" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to grpc-io+unsubscr...@googlegroups.com. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/grpc-io/532f5551-e978-467e-b71c-0031a54953bfn%40googlegroups.com >>>>>>>> <https://groups.google.com/d/msgid/grpc-io/532f5551-e978-467e-b71c-0031a54953bfn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "grpc.io" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to grpc-io+unsubscr...@googlegroups.com. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/grpc-io/CALO6TtuVMUxHQmZMGj1wvD7r4qvRQuKd2gy0j%2B0z8b8EYdw7BA%40mail.gmail.com >>>>>>> <https://groups.google.com/d/msgid/grpc-io/CALO6TtuVMUxHQmZMGj1wvD7r4qvRQuKd2gy0j%2B0z8b8EYdw7BA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>> -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CALO6TtvXMpjxrfyKDDn4aX7W7WFm%3DtNWY-sy8YiNj541%2B_ucpQ%40mail.gmail.com.
