Hi grpc people! We have a setup where we're running a grpc service (written in Go) on GKE, and we're accepting traffic from outside the cluster through nginx ingresses. Our clients are all using Core GRPC libraries (mostly Ruby) to make calls to the nginx ingress, which load-balances per-call to our backend pods.
The problem we have with this setup is that whenever the nginx ingresses reload they drop all client connections, which results in spikes of Unavailable errors from our grpc clients. There are many nginx ingresses but they all have a single IP, the incoming TCP connections are routed through a google cloud L4 load balancer. Whenever an nginx . client closes a TCP connection the GRPC subchannel treats the backend as unavailable, even though there are many more nginx pods that may be available immediately to serve traffic, and it goes into backoff logic. My understanding is that with multiple subchannels even if one nginx ingress is restarted the others can continue to serve requests and we shouldn't see Unavailable errors. My question is: what is the best way to make GRPC Core establish multiple connections to a single IP, so we can have long-lived connections to multiple nginx ingresses? Possibilities we've considered: - DNS round-robin with multiple public IPs on a single A record - we've tested this and it works, but it requires us to manually administer the DNS records and run multiple L4 LBs - DNS SRV records - it seems like we could have multiple SRV records with the same hostname, but in my testing this requires us to add a look-aside load-balancer as well, and enable ares DNS which doesn't seem to be production-ready - Host a look-aside load-balancer - we could host our own LB service, but it's not clear to me how we would overcome this issue for the LB service? The LB would be behind the same nginx ingresses. I haven't found great documentation on how to set this up either. - Connection pooling in the client - wrapping the Ruby GRPC channels in a library that explicitly establishes multiple channels, each with one sub-channel. I've tried to write this but it's tricky to implement at a high level. I couldn't get it to perform as well during failures as the DNS round-robin approach. Are there options I missed? Is there any supported pattern for this? Has anyone deployed a similar architecture (many clients connecting through nginx on a single public IP)? Thanks, Alysha -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscr...@googlegroups.com. To post to this group, send email to grpc-io@googlegroups.com. Visit this group at https://groups.google.com/group/grpc-io. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/0571d7d1-d91c-417a-b1ee-5c7f2296bc38%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
