[grpc-io] Re: Connection management and load balancing

['Srini Polavarapu' via grpc.io]

Hi Vitaly,

Please see this post 
 if you 
are planning to use gRPCLB. gRPC has moved away from gRPCLB protocol. 
Instead, gRPC is adopting xDS protocol. A number of xDS features 
, 
including round robin LB, are already supported in gRPC. This project 
 might be useful to you but I 
think it is blocked on this issue 
. This 
 project might be useful too.

On Friday, February 19, 2021 at 3:47:22 PM UTC-8 vitaly@gmail.com wrote:

> Thanks Srini,
>
> I haven't tested option 2 yet, I would expect though that since client is 
> unaware of what is happening we should see some request failures/latency 
> spikes until new connection is established. That's why I would consider it 
> mostly for disaster prevention rather than for general connection balancing.
> I'm actually now more interested in exploring option 4 as it looks like we 
> can achieve safe setup if we keep proxy in front of servers and expose a 
> separate proxy port for each server.
> Can someone recommend a good opensource grpclb implementation? I've found 
> bsm/grpclb  which looks reasonable but 
> wasn't sure if there is anything else available.
>
> On Friday, February 19, 2021 at 12:50:17 PM UTC-8 Srini Polavarapu wrote:
>
>> Hi,
>>
>> Option 3 is ideal but since you don't have that as well as option 4 
>> available, option 2 is worth exploring. Are the concerns with options 2 
>> based on some experiments you have done or is it just a hunch? This 
>> comment 
>>  has 
>> some relevant info that you could use.  
>>
>> On Thursday, February 18, 2021 at 7:06:37 PM UTC-8 vitaly@gmail.com 
>> wrote:
>>
>>>
>>> Hey folks,
>>>
>>> I'm trying to solve a problem of even load (or at least connection) 
>>> distribution between  grpc clients and our backend servers.
>>>
>>> First of all let me describe our setup:
>>> We are using network load balancing (L4) in front of our grpc servers.
>>> Clients will see one endpoint (LB) and connect to it. This means that 
>>> standard client-side load balancing features like round robing wouldn't 
>>> work as there will only be one sub-channel for client-server communication.
>>>
>>> One issue with this approach can be demonstrated by the following 
>>> example:
>>> Let's say we have 2 servers running and 20 clients connect to them. At 
>>> the beginning, since we go through the network load balancer, connections 
>>> will be distributed evenly (or close to that), so we'll roughly have 50% of 
>>> connections to each server. Now let's assume these servers reboot one after 
>>> another, like in a deployment. What would happen is that server that comes 
>>> up first would get all 20 worker connections and server that comes up later 
>>> would have zero. This situation won't change unless client or server would 
>>> drop a connection periodically or more clients request connections.
>>>
>>> I've considered a few options for solving this:
>>> 1. Connection management on the client side - do something to reset the 
>>> channel (like [enterIdle](
>>> https://grpc.github.io/grpc-java/javadoc/io/grpc/ManagedChannel.html#enterIdle)
>>>  
>>> in grpc-java). Downside - it seems that this feature has been developed for 
>>> android and I can't find similar functionality in grpc-go.
>>> 2. Connection management on the server side - drop connections 
>>> periodically on the server. Downside - this approach looks less graceful 
>>> than the client side one and may impact request latency and result in 
>>> request failures on the client side.
>>> 3. Use request based grpc-aware L7 LB, this way client would connect to 
>>> the LB, which would fan out requests to the servers. Downside - I've been 
>>> told by our infra guys that it is hard to implement in our setup due to the 
>>> way we use TLS and manage certificates.
>>> 4. Expose our servers outside and use grpc-lb or client side load 
>>> balancing. Downside - it seems less secure and would make it harder to 
>>> protect against DDoS attacks if we go this route. I think this downside 
>>> makes this approach unviable.
>>>
>>> My bias is towards going with option 3 and doing request based load 
>>> balancing because it allows much more fine grained control based on load, 
>>> but since our infra can not support it at the moment, I might be forced to 
>>> use option 1 or 2 in the short to mid term. Option 2 I like the least, as 
>>> it might result in latency spikes and errors on the client side.
>>>
>>> My questions are:
>>> 1. Which approach is generally preferable? 
>>> 2. Are there other options to consider?
>>> 3. Is it possible to influence grpc channel state in grpc-go, which 
>>> would trigger 

[grpc-io] Re: Connection management and load balancing

[Vitaly]

Thanks Srini,

I haven't tested option 2 yet, I would expect though that since client is 
unaware of what is happening we should see some request failures/latency 
spikes until new connection is established. That's why I would consider it 
mostly for disaster prevention rather than for general connection balancing.
I'm actually now more interested in exploring option 4 as it looks like we 
can achieve safe setup if we keep proxy in front of servers and expose a 
separate proxy port for each server.
Can someone recommend a good opensource grpclb implementation? I've found 
bsm/grpclb  which looks reasonable but 
wasn't sure if there is anything else available.

On Friday, February 19, 2021 at 12:50:17 PM UTC-8 Srini Polavarapu wrote:

> Hi,
>
> Option 3 is ideal but since you don't have that as well as option 4 
> available, option 2 is worth exploring. Are the concerns with options 2 
> based on some experiments you have done or is it just a hunch? This 
> comment  
> has 
> some relevant info that you could use.  
>
> On Thursday, February 18, 2021 at 7:06:37 PM UTC-8 vitaly@gmail.com 
> wrote:
>
>>
>> Hey folks,
>>
>> I'm trying to solve a problem of even load (or at least connection) 
>> distribution between  grpc clients and our backend servers.
>>
>> First of all let me describe our setup:
>> We are using network load balancing (L4) in front of our grpc servers.
>> Clients will see one endpoint (LB) and connect to it. This means that 
>> standard client-side load balancing features like round robing wouldn't 
>> work as there will only be one sub-channel for client-server communication.
>>
>> One issue with this approach can be demonstrated by the following example:
>> Let's say we have 2 servers running and 20 clients connect to them. At 
>> the beginning, since we go through the network load balancer, connections 
>> will be distributed evenly (or close to that), so we'll roughly have 50% of 
>> connections to each server. Now let's assume these servers reboot one after 
>> another, like in a deployment. What would happen is that server that comes 
>> up first would get all 20 worker connections and server that comes up later 
>> would have zero. This situation won't change unless client or server would 
>> drop a connection periodically or more clients request connections.
>>
>> I've considered a few options for solving this:
>> 1. Connection management on the client side - do something to reset the 
>> channel (like [enterIdle](
>> https://grpc.github.io/grpc-java/javadoc/io/grpc/ManagedChannel.html#enterIdle)
>>  
>> in grpc-java). Downside - it seems that this feature has been developed for 
>> android and I can't find similar functionality in grpc-go.
>> 2. Connection management on the server side - drop connections 
>> periodically on the server. Downside - this approach looks less graceful 
>> than the client side one and may impact request latency and result in 
>> request failures on the client side.
>> 3. Use request based grpc-aware L7 LB, this way client would connect to 
>> the LB, which would fan out requests to the servers. Downside - I've been 
>> told by our infra guys that it is hard to implement in our setup due to the 
>> way we use TLS and manage certificates.
>> 4. Expose our servers outside and use grpc-lb or client side load 
>> balancing. Downside - it seems less secure and would make it harder to 
>> protect against DDoS attacks if we go this route. I think this downside 
>> makes this approach unviable.
>>
>> My bias is towards going with option 3 and doing request based load 
>> balancing because it allows much more fine grained control based on load, 
>> but since our infra can not support it at the moment, I might be forced to 
>> use option 1 or 2 in the short to mid term. Option 2 I like the least, as 
>> it might result in latency spikes and errors on the client side.
>>
>> My questions are:
>> 1. Which approach is generally preferable? 
>> 2. Are there other options to consider?
>> 3. Is it possible to influence grpc channel state in grpc-go, which would 
>> trigger resolver and balancer to establish a new connection similar to what 
>> enterIdle does in java? From what I see in the [clientconn.go](
>> https://github.com/grpc/grpc-go/blob/master/clientconn.go) there is no 
>> option to change the channel state to idle or trigger a reconnect in some 
>> other way.
>> 4. Is there a way to implement server side connection management cleanly 
>> without impacting client-side severely?
>>
>> Here are links that I find useful for some context:
>> grpc/load-balancing.md at master · grpc/grpc 
>> 
>> proposal/A9-server-side-conn-mgt.md at master · grpc/proposal 
>> 

[grpc-io] Re: Connection management and load balancing

['Srini Polavarapu' via grpc.io]

Hi,

Option 3 is ideal but since you don't have that as well as option 4 
available, option 2 is worth exploring. Are the concerns with options 2 
based on some experiments you have done or is it just a hunch? This comment 
 has some 
relevant info that you could use.  

On Thursday, February 18, 2021 at 7:06:37 PM UTC-8 vitaly@gmail.com 
wrote:

>
> Hey folks,
>
> I'm trying to solve a problem of even load (or at least connection) 
> distribution between  grpc clients and our backend servers.
>
> First of all let me describe our setup:
> We are using network load balancing (L4) in front of our grpc servers.
> Clients will see one endpoint (LB) and connect to it. This means that 
> standard client-side load balancing features like round robing wouldn't 
> work as there will only be one sub-channel for client-server communication.
>
> One issue with this approach can be demonstrated by the following example:
> Let's say we have 2 servers running and 20 clients connect to them. At the 
> beginning, since we go through the network load balancer, connections will 
> be distributed evenly (or close to that), so we'll roughly have 50% of 
> connections to each server. Now let's assume these servers reboot one after 
> another, like in a deployment. What would happen is that server that comes 
> up first would get all 20 worker connections and server that comes up later 
> would have zero. This situation won't change unless client or server would 
> drop a connection periodically or more clients request connections.
>
> I've considered a few options for solving this:
> 1. Connection management on the client side - do something to reset the 
> channel (like [enterIdle](
> https://grpc.github.io/grpc-java/javadoc/io/grpc/ManagedChannel.html#enterIdle)
>  
> in grpc-java). Downside - it seems that this feature has been developed for 
> android and I can't find similar functionality in grpc-go.
> 2. Connection management on the server side - drop connections 
> periodically on the server. Downside - this approach looks less graceful 
> than the client side one and may impact request latency and result in 
> request failures on the client side.
> 3. Use request based grpc-aware L7 LB, this way client would connect to 
> the LB, which would fan out requests to the servers. Downside - I've been 
> told by our infra guys that it is hard to implement in our setup due to the 
> way we use TLS and manage certificates.
> 4. Expose our servers outside and use grpc-lb or client side load 
> balancing. Downside - it seems less secure and would make it harder to 
> protect against DDoS attacks if we go this route. I think this downside 
> makes this approach unviable.
>
> My bias is towards going with option 3 and doing request based load 
> balancing because it allows much more fine grained control based on load, 
> but since our infra can not support it at the moment, I might be forced to 
> use option 1 or 2 in the short to mid term. Option 2 I like the least, as 
> it might result in latency spikes and errors on the client side.
>
> My questions are:
> 1. Which approach is generally preferable? 
> 2. Are there other options to consider?
> 3. Is it possible to influence grpc channel state in grpc-go, which would 
> trigger resolver and balancer to establish a new connection similar to what 
> enterIdle does in java? From what I see in the [clientconn.go](
> https://github.com/grpc/grpc-go/blob/master/clientconn.go) there is no 
> option to change the channel state to idle or trigger a reconnect in some 
> other way.
> 4. Is there a way to implement server side connection management cleanly 
> without impacting client-side severely?
>
> Here are links that I find useful for some context:
> grpc/load-balancing.md at master · grpc/grpc 
> 
> proposal/A9-server-side-conn-mgt.md at master · grpc/proposal 
> 
>  
> proposal/A8-client-side-keepalive.md at master · grpc/proposal 
> 
>   
> grpc/keepalive.md at master · grpc/grpc 
> 
>
>
> Sorry for the long read,
> Vitaly
>

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To view this discussion on the web visit