Re: [grpc-io] Re: gRPC A6: Retries

['Rishabh Mor' via grpc.io]

How can we know the status of retries in go-grpc ? I see hedging is not 
implemented but apart from that pretty much everything else is there. Are 
the go-grpc retries being used internally within Google ?

Thanks,
Rishabh

On Wednesday, September 30, 2020 at 10:50:58 PM UTC+5:30 zda...@google.com 
wrote:

> Agree with Eric. I'll also note that if connection is broken in the middle 
> of an RPC after the client receives partial data from the server, say, only 
> the response headers, then although the channel will be reconnecting 
> automatically by the library, that individual RPC is not retried 
> automatically by the library, see the definition of committed 
> 
>  in 
> the retry design for details.
>
> On Wednesday, September 30, 2020 at 9:37:24 AM UTC-7 Eric Anderson wrote:
>
>> You need to call `enableRetry()` on the channel builder. See the retry 
>> example 
>> 
>>  and 
>> example config 
>> 
>> .
>>
>> I think your methodConfig may not be selected because there is no 'name' 
>> list for methods to match. Now that we support wildcard service names, you 
>> could probably use methodConfig.put("name", 
>> Arrays.asList(Collections.emptyMap())).
>>
>> I'll note that reconnect attempts are completely separate from RPC 
>> retries. gRPC always has reconnect behavior enabled.
>>
>> On Wed, Sep 30, 2020 at 8:39 AM 'Mark D. Roth' via grpc.io <
>> grp...@googlegroups.com> wrote:
>>
>>> As per discussion earlier in this thread, we haven't yet finished 
>>> implementing the retry functionality, so it's not yet enabled by default.  
>>> I believe that in Java, you may be able to use it, albeit with some 
>>> caveats.  Penn (CC'ed) can tell you what the current status is in Java.
>>>
>>> On Wed, Sep 30, 2020 at 8:35 AM Guillermo Romero  
>>> wrote:
>>>
 Thanks Mark: 

 So.
what's level this retry policy works?. 


 final Map retryPolicy = new HashMap<>();
 retryPolicy.put("maxAttempts", 10D);
 retryPolicy.put("initialBackoff", "10s");
 retryPolicy.put("maxBackoff", "30s");
 retryPolicy.put("backoffMultiplier", 2D);
 retryPolicy.put("retryableStatusCodes", Arrays.asList(
 "UNAVAILABLE" , "RESOURCE_EXHAUSTED" , "INTERNAL"));
 final Map methodConfig = new HashMap<>();
 methodConfig.put("retryPolicy", retryPolicy);

 final Map serviceConfig = new HashMap<>();
 serviceConfig.put("methodConfig", Collections.singletonList(
 methodConfig));

 I'm having a problem with netty client, it thows an exception  when tcp 
 breaks an not try to reconnect N times (MaxAttemps) - 



 El miércoles, 30 de septiembre de 2020 a las 12:09:25 UTC-3, Mark D. 
 Roth escribió:

> gRPC client channels will automatically reconnect to the server when 
> the TCP connection fails.  That has nothing to do with the retry feature, 
> and it's not something you need to configure -- it will happen 
> automatically.
>
> Now, if an individual request is already in-flight when the TCP 
> connection fails, that will cause the request to fail.  And in that case, 
> retrying the request would be what you want.
>
> On Wed, Sep 30, 2020 at 6:01 AM Guillermo Romero <
> guillerm...@gmail.com> wrote:
>
>> Hi: 
>> I'm using Jboss Netty as Grpc client, and my doubts are related 
>> to the Retry Policy. My understanding is that the Retry Policy is 
>> related 
>> to the internal message transport between the client and the server 
>> using 
>> the gRPC protocol.
>>  But my problem is related to the TCP breaks, there is a way of write 
>> a TCP retry policy?
>>
>>
>> El viernes, 10 de febrero de 2017 a las 21:31:01 UTC-3, 
>> ncte...@google.com escribió:
>>
>>> I've created a gRFC describing the design and implementation plan 
>>> for gRPC Retries.
>>>
>>> Take a look at the gRPC on Github 
>>> .
>>>
>> -- 
>>
> You received this message because you are subscribed to the Google 
>> Groups "grpc.io" group.
>> To unsubscribe from this group and stop receiving emails from it, 
>> send an email to grpc-io+u...@googlegroups.com.
>>
> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/grpc-io/33fc1235-073a-48ca-b387-b964b74e366fn%40googlegroups.com
>>  
>> 
>

Re: [grpc-io] Re: gRPC A6: Retries

['zda...@google.com' via grpc.io]

Agree with Eric. I'll also note that if connection is broken in the middle 
of an RPC after the client receives partial data from the server, say, only 
the response headers, then although the channel will be reconnecting 
automatically by the library, that individual RPC is not retried 
automatically by the library, see the definition of committed 

 in 
the retry design for details.

On Wednesday, September 30, 2020 at 9:37:24 AM UTC-7 Eric Anderson wrote:

> You need to call `enableRetry()` on the channel builder. See the retry 
> example 
> 
>  and 
> example config 
> 
> .
>
> I think your methodConfig may not be selected because there is no 'name' 
> list for methods to match. Now that we support wildcard service names, you 
> could probably use methodConfig.put("name", 
> Arrays.asList(Collections.emptyMap())).
>
> I'll note that reconnect attempts are completely separate from RPC 
> retries. gRPC always has reconnect behavior enabled.
>
> On Wed, Sep 30, 2020 at 8:39 AM 'Mark D. Roth' via grpc.io <
> grp...@googlegroups.com> wrote:
>
>> As per discussion earlier in this thread, we haven't yet finished 
>> implementing the retry functionality, so it's not yet enabled by default.  
>> I believe that in Java, you may be able to use it, albeit with some 
>> caveats.  Penn (CC'ed) can tell you what the current status is in Java.
>>
>> On Wed, Sep 30, 2020 at 8:35 AM Guillermo Romero  
>> wrote:
>>
>>> Thanks Mark: 
>>>
>>> So.
>>>what's level this retry policy works?. 
>>>
>>>
>>> final Map retryPolicy = new HashMap<>();
>>> retryPolicy.put("maxAttempts", 10D);
>>> retryPolicy.put("initialBackoff", "10s");
>>> retryPolicy.put("maxBackoff", "30s");
>>> retryPolicy.put("backoffMultiplier", 2D);
>>> retryPolicy.put("retryableStatusCodes", Arrays.asList(
>>> "UNAVAILABLE" , "RESOURCE_EXHAUSTED" , "INTERNAL"));
>>> final Map methodConfig = new HashMap<>();
>>> methodConfig.put("retryPolicy", retryPolicy);
>>>
>>> final Map serviceConfig = new HashMap<>();
>>> serviceConfig.put("methodConfig", Collections.singletonList(
>>> methodConfig));
>>>
>>> I'm having a problem with netty client, it thows an exception  when tcp 
>>> breaks an not try to reconnect N times (MaxAttemps) - 
>>>
>>>
>>>
>>> El miércoles, 30 de septiembre de 2020 a las 12:09:25 UTC-3, Mark D. 
>>> Roth escribió:
>>>
 gRPC client channels will automatically reconnect to the server when 
 the TCP connection fails.  That has nothing to do with the retry feature, 
 and it's not something you need to configure -- it will happen 
 automatically.

 Now, if an individual request is already in-flight when the TCP 
 connection fails, that will cause the request to fail.  And in that case, 
 retrying the request would be what you want.

 On Wed, Sep 30, 2020 at 6:01 AM Guillermo Romero  
 wrote:

> Hi: 
> I'm using Jboss Netty as Grpc client, and my doubts are related to 
> the Retry Policy. My understanding is that the Retry Policy is related to 
> the internal message transport between the client and the server using 
> the 
> gRPC protocol.
>  But my problem is related to the TCP breaks, there is a way of write 
> a TCP retry policy?
>
>
> El viernes, 10 de febrero de 2017 a las 21:31:01 UTC-3, 
> ncte...@google.com escribió:
>
>> I've created a gRFC describing the design and implementation plan for 
>> gRPC Retries.
>>
>> Take a look at the gRPC on Github 
>> .
>>
> -- 
>
 You received this message because you are subscribed to the Google 
> Groups "grpc.io" group.
> To unsubscribe from this group and stop receiving emails from it, send 
> an email to grpc-io+u...@googlegroups.com.
>
 To view this discussion on the web visit 
> https://groups.google.com/d/msgid/grpc-io/33fc1235-073a-48ca-b387-b964b74e366fn%40googlegroups.com
>  
> 
> .
>


 -- 
 Mark D. Roth 
 Software Engineer
 Google, Inc.

>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "grpc.io" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to grpc-io+u...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/grpc-io/6319a287-42c4-424d-8f40-3dfd97a8be5dn%40

Re: [grpc-io] Re: gRPC A6: Retries

['Mark D. Roth' via grpc.io]

It's definitely something that we want to finish.  I personally spent
almost a year working on the C-core implementation, and it's mostly
complete, but not quite enough to actually use yet -- there's still a bit
of missing functionality to implement, and there are some design issues
related to stats that we need to resolve.

Unfortunately, we've had other higher priority items come up that have
required us to set this aside.  I hope to be able to get back to finishing
this up in Q2 next year.

On Wed, Sep 30, 2020 at 9:12 AM Nathan Roberson 
wrote:

> I would advocate for finishing this implementation and releasing for C++
> as a high priority item. :)
>
>
>
> On Wednesday, September 30, 2020 at 8:39:51 AM UTC-7 Mark D. Roth wrote:
>
>> As per discussion earlier in this thread, we haven't yet finished
>> implementing the retry functionality, so it's not yet enabled by default.
>> I believe that in Java, you may be able to use it, albeit with some
>> caveats.  Penn (CC'ed) can tell you what the current status is in Java.
>>
>> On Wed, Sep 30, 2020 at 8:35 AM Guillermo Romero 
>> wrote:
>>
>>> Thanks Mark:
>>>
>>> So.
>>>what's level this retry policy works?.
>>>
>>>
>>> final Map retryPolicy = new HashMap<>();
>>> retryPolicy.put("maxAttempts", 10D);
>>> retryPolicy.put("initialBackoff", "10s");
>>> retryPolicy.put("maxBackoff", "30s");
>>> retryPolicy.put("backoffMultiplier", 2D);
>>> retryPolicy.put("retryableStatusCodes", Arrays.asList(
>>> "UNAVAILABLE" , "RESOURCE_EXHAUSTED" , "INTERNAL"));
>>> final Map methodConfig = new HashMap<>();
>>> methodConfig.put("retryPolicy", retryPolicy);
>>>
>>> final Map serviceConfig = new HashMap<>();
>>> serviceConfig.put("methodConfig", Collections.singletonList(
>>> methodConfig));
>>>
>>> I'm having a problem with netty client, it thows an exception  when tcp
>>> breaks an not try to reconnect N times (MaxAttemps) -
>>>
>>>
>>>
>>> El miércoles, 30 de septiembre de 2020 a las 12:09:25 UTC-3, Mark D.
>>> Roth escribió:
>>>
 gRPC client channels will automatically reconnect to the server when
 the TCP connection fails.  That has nothing to do with the retry feature,
 and it's not something you need to configure -- it will happen
 automatically.

 Now, if an individual request is already in-flight when the TCP
 connection fails, that will cause the request to fail.  And in that case,
 retrying the request would be what you want.

 On Wed, Sep 30, 2020 at 6:01 AM Guillermo Romero 
 wrote:

> Hi:
> I'm using Jboss Netty as Grpc client, and my doubts are related to
> the Retry Policy. My understanding is that the Retry Policy is related to
> the internal message transport between the client and the server using the
> gRPC protocol.
>  But my problem is related to the TCP breaks, there is a way of write
> a TCP retry policy?
>
>
> El viernes, 10 de febrero de 2017 a las 21:31:01 UTC-3,
> ncte...@google.com escribió:
>
>> I've created a gRFC describing the design and implementation plan for
>> gRPC Retries.
>>
>> Take a look at the gRPC on Github
>> .
>>
> --
>
 You received this message because you are subscribed to the Google
> Groups "grpc.io" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to grpc-io+u...@googlegroups.com.
>
 To view this discussion on the web visit
> https://groups.google.com/d/msgid/grpc-io/33fc1235-073a-48ca-b387-b964b74e366fn%40googlegroups.com
> 
> .
>


 --
 Mark D. Roth 
 Software Engineer
 Google, Inc.

>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "grpc.io" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to grpc-io+u...@googlegroups.com.
>>>
>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/grpc-io/6319a287-42c4-424d-8f40-3dfd97a8be5dn%40googlegroups.com
>>> 
>>> .
>>>
>>
>>
>> --
>> Mark D. Roth 
>> Software Engineer
>> Google, Inc.
>>
> --
> You received this message because you are subscribed to the Google Groups "
> grpc.io" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to grpc-io+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/grpc-io/4c3a51ea-4609-4ac5-b3e7-5dea86fee824n%40googlegroups.com
> 
> .
>


-- 
Mark D. Roth 
Softwar

Re: [grpc-io] Re: gRPC A6: Retries

['Eric Anderson' via grpc.io]

You need to call `enableRetry()` on the channel builder. See the retry
example

and
example config

.

I think your methodConfig may not be selected because there is no 'name'
list for methods to match. Now that we support wildcard service names, you
could probably use methodConfig.put("name",
Arrays.asList(Collections.emptyMap())).

I'll note that reconnect attempts are completely separate from RPC retries.
gRPC always has reconnect behavior enabled.

On Wed, Sep 30, 2020 at 8:39 AM 'Mark D. Roth' via grpc.io <
grpc-io@googlegroups.com> wrote:

> As per discussion earlier in this thread, we haven't yet finished
> implementing the retry functionality, so it's not yet enabled by default.
> I believe that in Java, you may be able to use it, albeit with some
> caveats.  Penn (CC'ed) can tell you what the current status is in Java.
>
> On Wed, Sep 30, 2020 at 8:35 AM Guillermo Romero <
> guillermo.rom...@gmail.com> wrote:
>
>> Thanks Mark:
>>
>> So.
>>what's level this retry policy works?.
>>
>>
>> final Map retryPolicy = new HashMap<>();
>> retryPolicy.put("maxAttempts", 10D);
>> retryPolicy.put("initialBackoff", "10s");
>> retryPolicy.put("maxBackoff", "30s");
>> retryPolicy.put("backoffMultiplier", 2D);
>> retryPolicy.put("retryableStatusCodes", Arrays.asList(
>> "UNAVAILABLE" , "RESOURCE_EXHAUSTED" , "INTERNAL"));
>> final Map methodConfig = new HashMap<>();
>> methodConfig.put("retryPolicy", retryPolicy);
>>
>> final Map serviceConfig = new HashMap<>();
>> serviceConfig.put("methodConfig", Collections.singletonList(
>> methodConfig));
>>
>> I'm having a problem with netty client, it thows an exception  when tcp
>> breaks an not try to reconnect N times (MaxAttemps) -
>>
>>
>>
>> El miércoles, 30 de septiembre de 2020 a las 12:09:25 UTC-3, Mark D. Roth
>> escribió:
>>
>>> gRPC client channels will automatically reconnect to the server when the
>>> TCP connection fails.  That has nothing to do with the retry feature, and
>>> it's not something you need to configure -- it will happen automatically.
>>>
>>> Now, if an individual request is already in-flight when the TCP
>>> connection fails, that will cause the request to fail.  And in that case,
>>> retrying the request would be what you want.
>>>
>>> On Wed, Sep 30, 2020 at 6:01 AM Guillermo Romero 
>>> wrote:
>>>
 Hi:
 I'm using Jboss Netty as Grpc client, and my doubts are related to
 the Retry Policy. My understanding is that the Retry Policy is related to
 the internal message transport between the client and the server using the
 gRPC protocol.
  But my problem is related to the TCP breaks, there is a way of write a
 TCP retry policy?


 El viernes, 10 de febrero de 2017 a las 21:31:01 UTC-3,
 ncte...@google.com escribió:

> I've created a gRFC describing the design and implementation plan for
> gRPC Retries.
>
> Take a look at the gRPC on Github
> .
>
 --

>>> You received this message because you are subscribed to the Google
 Groups "grpc.io" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to grpc-io+u...@googlegroups.com.

>>> To view this discussion on the web visit
 https://groups.google.com/d/msgid/grpc-io/33fc1235-073a-48ca-b387-b964b74e366fn%40googlegroups.com
 
 .

>>>
>>>
>>> --
>>> Mark D. Roth 
>>> Software Engineer
>>> Google, Inc.
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "grpc.io" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to grpc-io+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/grpc-io/6319a287-42c4-424d-8f40-3dfd97a8be5dn%40googlegroups.com
>> 
>> .
>>
>
>
> --
> Mark D. Roth 
> Software Engineer
> Google, Inc.
>
> --
> You received this message because you are subscribed to the Google Groups "
> grpc.io" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to grpc-io+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/grpc-io/CAJgPXp44%3DNF95XfmGCk4nzBCuO89HaR886XSKeYZMCC-i6m3Fg%40mail.gmail.com
> 

Re: [grpc-io] Re: gRPC A6: Retries

[Nathan Roberson]

I would advocate for finishing this implementation and releasing for C++ as 
a high priority item. :)



On Wednesday, September 30, 2020 at 8:39:51 AM UTC-7 Mark D. Roth wrote:

> As per discussion earlier in this thread, we haven't yet finished 
> implementing the retry functionality, so it's not yet enabled by default.  
> I believe that in Java, you may be able to use it, albeit with some 
> caveats.  Penn (CC'ed) can tell you what the current status is in Java.
>
> On Wed, Sep 30, 2020 at 8:35 AM Guillermo Romero  
> wrote:
>
>> Thanks Mark: 
>>
>> So.
>>what's level this retry policy works?. 
>>
>>
>> final Map retryPolicy = new HashMap<>();
>> retryPolicy.put("maxAttempts", 10D);
>> retryPolicy.put("initialBackoff", "10s");
>> retryPolicy.put("maxBackoff", "30s");
>> retryPolicy.put("backoffMultiplier", 2D);
>> retryPolicy.put("retryableStatusCodes", Arrays.asList(
>> "UNAVAILABLE" , "RESOURCE_EXHAUSTED" , "INTERNAL"));
>> final Map methodConfig = new HashMap<>();
>> methodConfig.put("retryPolicy", retryPolicy);
>>
>> final Map serviceConfig = new HashMap<>();
>> serviceConfig.put("methodConfig", Collections.singletonList(
>> methodConfig));
>>
>> I'm having a problem with netty client, it thows an exception  when tcp 
>> breaks an not try to reconnect N times (MaxAttemps) - 
>>
>>
>>
>> El miércoles, 30 de septiembre de 2020 a las 12:09:25 UTC-3, Mark D. Roth 
>> escribió:
>>
>>> gRPC client channels will automatically reconnect to the server when the 
>>> TCP connection fails.  That has nothing to do with the retry feature, and 
>>> it's not something you need to configure -- it will happen automatically.
>>>
>>> Now, if an individual request is already in-flight when the TCP 
>>> connection fails, that will cause the request to fail.  And in that case, 
>>> retrying the request would be what you want.
>>>
>>> On Wed, Sep 30, 2020 at 6:01 AM Guillermo Romero  
>>> wrote:
>>>
 Hi: 
 I'm using Jboss Netty as Grpc client, and my doubts are related to 
 the Retry Policy. My understanding is that the Retry Policy is related to 
 the internal message transport between the client and the server using the 
 gRPC protocol.
  But my problem is related to the TCP breaks, there is a way of write a 
 TCP retry policy?


 El viernes, 10 de febrero de 2017 a las 21:31:01 UTC-3, 
 ncte...@google.com escribió:

> I've created a gRFC describing the design and implementation plan for 
> gRPC Retries.
>
> Take a look at the gRPC on Github 
> .
>
 -- 

>>> You received this message because you are subscribed to the Google 
 Groups "grpc.io" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to grpc-io+u...@googlegroups.com.

>>> To view this discussion on the web visit 
 https://groups.google.com/d/msgid/grpc-io/33fc1235-073a-48ca-b387-b964b74e366fn%40googlegroups.com
  
 
 .

>>>
>>>
>>> -- 
>>> Mark D. Roth 
>>> Software Engineer
>>> Google, Inc.
>>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "grpc.io" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to grpc-io+u...@googlegroups.com.
>>
> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/grpc-io/6319a287-42c4-424d-8f40-3dfd97a8be5dn%40googlegroups.com
>>  
>> 
>> .
>>
>
>
> -- 
> Mark D. Roth 
> Software Engineer
> Google, Inc.
>

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/4c3a51ea-4609-4ac5-b3e7-5dea86fee824n%40googlegroups.com.

Re: [grpc-io] Re: gRPC A6: Retries

['Mark D. Roth' via grpc.io]

As per discussion earlier in this thread, we haven't yet finished
implementing the retry functionality, so it's not yet enabled by default.
I believe that in Java, you may be able to use it, albeit with some
caveats.  Penn (CC'ed) can tell you what the current status is in Java.

On Wed, Sep 30, 2020 at 8:35 AM Guillermo Romero 
wrote:

> Thanks Mark:
>
> So.
>what's level this retry policy works?.
>
>
> final Map retryPolicy = new HashMap<>();
> retryPolicy.put("maxAttempts", 10D);
> retryPolicy.put("initialBackoff", "10s");
> retryPolicy.put("maxBackoff", "30s");
> retryPolicy.put("backoffMultiplier", 2D);
> retryPolicy.put("retryableStatusCodes", Arrays.asList(
> "UNAVAILABLE" , "RESOURCE_EXHAUSTED" , "INTERNAL"));
> final Map methodConfig = new HashMap<>();
> methodConfig.put("retryPolicy", retryPolicy);
>
> final Map serviceConfig = new HashMap<>();
> serviceConfig.put("methodConfig", Collections.singletonList(
> methodConfig));
>
> I'm having a problem with netty client, it thows an exception  when tcp
> breaks an not try to reconnect N times (MaxAttemps) -
>
>
>
> El miércoles, 30 de septiembre de 2020 a las 12:09:25 UTC-3, Mark D. Roth
> escribió:
>
>> gRPC client channels will automatically reconnect to the server when the
>> TCP connection fails.  That has nothing to do with the retry feature, and
>> it's not something you need to configure -- it will happen automatically.
>>
>> Now, if an individual request is already in-flight when the TCP
>> connection fails, that will cause the request to fail.  And in that case,
>> retrying the request would be what you want.
>>
>> On Wed, Sep 30, 2020 at 6:01 AM Guillermo Romero 
>> wrote:
>>
>>> Hi:
>>> I'm using Jboss Netty as Grpc client, and my doubts are related to
>>> the Retry Policy. My understanding is that the Retry Policy is related to
>>> the internal message transport between the client and the server using the
>>> gRPC protocol.
>>>  But my problem is related to the TCP breaks, there is a way of write a
>>> TCP retry policy?
>>>
>>>
>>> El viernes, 10 de febrero de 2017 a las 21:31:01 UTC-3,
>>> ncte...@google.com escribió:
>>>
 I've created a gRFC describing the design and implementation plan for
 gRPC Retries.

 Take a look at the gRPC on Github
 .

>>> --
>>>
>> You received this message because you are subscribed to the Google Groups
>>> "grpc.io" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to grpc-io+u...@googlegroups.com.
>>>
>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/grpc-io/33fc1235-073a-48ca-b387-b964b74e366fn%40googlegroups.com
>>> 
>>> .
>>>
>>
>>
>> --
>> Mark D. Roth 
>> Software Engineer
>> Google, Inc.
>>
> --
> You received this message because you are subscribed to the Google Groups "
> grpc.io" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to grpc-io+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/grpc-io/6319a287-42c4-424d-8f40-3dfd97a8be5dn%40googlegroups.com
> 
> .
>


-- 
Mark D. Roth 
Software Engineer
Google, Inc.

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/CAJgPXp44%3DNF95XfmGCk4nzBCuO89HaR886XSKeYZMCC-i6m3Fg%40mail.gmail.com.

Re: [grpc-io] Re: gRPC A6: Retries

[Guillermo Romero]

Thanks Mark: 

So.
   what's level this retry policy works?. 


final Map retryPolicy = new HashMap<>();
retryPolicy.put("maxAttempts", 10D);
retryPolicy.put("initialBackoff", "10s");
retryPolicy.put("maxBackoff", "30s");
retryPolicy.put("backoffMultiplier", 2D);
retryPolicy.put("retryableStatusCodes", Arrays.asList("UNAVAILABLE" 
, "RESOURCE_EXHAUSTED" , "INTERNAL"));
final Map methodConfig = new HashMap<>();
methodConfig.put("retryPolicy", retryPolicy);

final Map serviceConfig = new HashMap<>();
serviceConfig.put("methodConfig", Collections.singletonList(
methodConfig));

I'm having a problem with netty client, it thows an exception  when tcp 
breaks an not try to reconnect N times (MaxAttemps) - 



El miércoles, 30 de septiembre de 2020 a las 12:09:25 UTC-3, Mark D. Roth 
escribió:

> gRPC client channels will automatically reconnect to the server when the 
> TCP connection fails.  That has nothing to do with the retry feature, and 
> it's not something you need to configure -- it will happen automatically.
>
> Now, if an individual request is already in-flight when the TCP connection 
> fails, that will cause the request to fail.  And in that case, retrying the 
> request would be what you want.
>
> On Wed, Sep 30, 2020 at 6:01 AM Guillermo Romero  
> wrote:
>
>> Hi: 
>> I'm using Jboss Netty as Grpc client, and my doubts are related to 
>> the Retry Policy. My understanding is that the Retry Policy is related to 
>> the internal message transport between the client and the server using the 
>> gRPC protocol.
>>  But my problem is related to the TCP breaks, there is a way of write a 
>> TCP retry policy?
>>
>>
>> El viernes, 10 de febrero de 2017 a las 21:31:01 UTC-3, 
>> ncte...@google.com escribió:
>>
>>> I've created a gRFC describing the design and implementation plan for 
>>> gRPC Retries.
>>>
>>> Take a look at the gRPC on Github 
>>> .
>>>
>> -- 
>>
> You received this message because you are subscribed to the Google Groups "
>> grpc.io" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to grpc-io+u...@googlegroups.com.
>>
> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/grpc-io/33fc1235-073a-48ca-b387-b964b74e366fn%40googlegroups.com
>>  
>> 
>> .
>>
>
>
> -- 
> Mark D. Roth 
> Software Engineer
> Google, Inc.
>

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/6319a287-42c4-424d-8f40-3dfd97a8be5dn%40googlegroups.com.

Re: [grpc-io] Re: gRPC A6: Retries

['Mark D. Roth' via grpc.io]

gRPC client channels will automatically reconnect to the server when the
TCP connection fails.  That has nothing to do with the retry feature, and
it's not something you need to configure -- it will happen automatically.

Now, if an individual request is already in-flight when the TCP connection
fails, that will cause the request to fail.  And in that case, retrying the
request would be what you want.

On Wed, Sep 30, 2020 at 6:01 AM Guillermo Romero 
wrote:

> Hi:
> I'm using Jboss Netty as Grpc client, and my doubts are related to the
> Retry Policy. My understanding is that the Retry Policy is related to the
> internal message transport between the client and the server using the gRPC
> protocol.
>  But my problem is related to the TCP breaks, there is a way of write a
> TCP retry policy?
>
>
> El viernes, 10 de febrero de 2017 a las 21:31:01 UTC-3, ncte...@google.com
> escribió:
>
>> I've created a gRFC describing the design and implementation plan for
>> gRPC Retries.
>>
>> Take a look at the gRPC on Github
>> .
>>
> --
> You received this message because you are subscribed to the Google Groups "
> grpc.io" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to grpc-io+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/grpc-io/33fc1235-073a-48ca-b387-b964b74e366fn%40googlegroups.com
> 
> .
>


-- 
Mark D. Roth 
Software Engineer
Google, Inc.

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/CAJgPXp7T%3DKzz1J2vb9NaT6qwE92cYmuB47H92%2B5Dan-KXETXMg%40mail.gmail.com.

[grpc-io] Re: gRPC A6: Retries

[Guillermo Romero]

Hi: 
I'm using Jboss Netty as Grpc client, and my doubts are related to the 
Retry Policy. My understanding is that the Retry Policy is related to the 
internal message transport between the client and the server using the gRPC 
protocol.
 But my problem is related to the TCP breaks, there is a way of write a TCP 
retry policy?


El viernes, 10 de febrero de 2017 a las 21:31:01 UTC-3, ncte...@google.com 
escribió:

> I've created a gRFC describing the design and implementation plan for gRPC 
> Retries.
>
> Take a look at the gRPC on Github 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/33fc1235-073a-48ca-b387-b964b74e366fn%40googlegroups.com.

Re: [grpc-io] Re: gRPC A6: Retries

['Mark D. Roth' via grpc.io]

Unfortunately, nothing has changed here.  At this point, the soonest we
could get back to this would probably be sometime in Q2 next year.

On Fri, Jul 24, 2020 at 7:54 AM  wrote:

> Mark,
>
> Are there any updates to this or does the latest post still stand?
>
> Thanks,
> Nathan
>
> On Thursday, June 27, 2019 at 7:06:00 AM UTC-7, Mark D. Roth wrote:
>>
>> On Thu, Jun 27, 2019 at 6:36 AM Pau Freixes  wrote:
>>
>>> Hi Mark,
>>>
>>> Thanks for the update
>>>
>>> > The current implementation in C-core is only partially complete.  The
>>> basic retry code is there, but there is still an outstanding design
>>> question of how we handle stats for retries, and there is not yet any
>>> support for transparent retries nor for hedging.  And even the basic retry
>>> code is extremely invasive and has not yet received any production testing,
>>> so there are probably numerous bugs waiting to be found.
>>>
>>> What do you mean by "there is not yet any support for transparent
>>> retries"? I thought hat retries were done under the hood transparently
>>> when some conditions are met - like using the response codes.
>>>
>>
>> Transparent retries are the ones described in this section of the spec:
>>
>>
>> https://github.com/grpc/proposal/blob/master/A6-client-retries.md#transparent-retries
>>
>> We have not yet implemented that functionality in C-core.
>>
>>
>>>
>>> Regarding the stats thing is it related to having the capacity of
>>> retrying when the number of "errors" is lower than a specific
>>> threshold?
>>>
>>
>> No, it's a more basic problem than that.  Whenever there are multiple
>> attempts on a given RPC, the additional attempts don't show up at all in
>> stats recorded via systems like census.
>>
>>
>>>
>>> Thanks for the advice on not using this in production. Does it mean
>>> that gRPC community still believe that the way to go for having a all
>>> of the needs covered for calling external dependencies - retrying,
>>> circuit breakers, etc - it's by implementing their own wrappers on top
>>> of the gRPC clients? What is being done within google right now?
>>>
>>
>> At the moment, the best way to do this is probably to write an
>> interceptor.
>>
>>
>>>
>>>
>>> Thanks!
>>>
>>> --
>>> --pau
>>>
>>
>>
>> --
>> Mark D. Roth 
>> Software Engineer
>> Google, Inc.
>>
> --
> You received this message because you are subscribed to the Google Groups "
> grpc.io" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to grpc-io+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/grpc-io/4c444986-66c1-4a6a-b551-b7cfdc3935deo%40googlegroups.com
> 
> .
>


-- 
Mark D. Roth 
Software Engineer
Google, Inc.

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/CAJgPXp7v%3DCCKJc1PfDf6j%2BayGiqafnxFz43x9nkokXmAk2AjUg%40mail.gmail.com.

Re: [grpc-io] Re: gRPC A6: Retries

[nathaniroberson]

Mark,

Are there any updates to this or does the latest post still stand?

Thanks,
Nathan

On Thursday, June 27, 2019 at 7:06:00 AM UTC-7, Mark D. Roth wrote:
>
> On Thu, Jun 27, 2019 at 6:36 AM Pau Freixes  > wrote:
>
>> Hi Mark,
>>
>> Thanks for the update
>>
>> > The current implementation in C-core is only partially complete.  The 
>> basic retry code is there, but there is still an outstanding design 
>> question of how we handle stats for retries, and there is not yet any 
>> support for transparent retries nor for hedging.  And even the basic retry 
>> code is extremely invasive and has not yet received any production testing, 
>> so there are probably numerous bugs waiting to be found.
>>
>> What do you mean by "there is not yet any support for transparent
>> retries"? I thought hat retries were done under the hood transparently
>> when some conditions are met - like using the response codes.
>>
>
> Transparent retries are the ones described in this section of the spec:
>
>
> https://github.com/grpc/proposal/blob/master/A6-client-retries.md#transparent-retries
>
> We have not yet implemented that functionality in C-core.
>  
>
>>
>> Regarding the stats thing is it related to having the capacity of
>> retrying when the number of "errors" is lower than a specific
>> threshold?
>>
>
> No, it's a more basic problem than that.  Whenever there are multiple 
> attempts on a given RPC, the additional attempts don't show up at all in 
> stats recorded via systems like census.
>  
>
>>
>> Thanks for the advice on not using this in production. Does it mean
>> that gRPC community still believe that the way to go for having a all
>> of the needs covered for calling external dependencies - retrying,
>> circuit breakers, etc - it's by implementing their own wrappers on top
>> of the gRPC clients? What is being done within google right now?
>>
>
> At the moment, the best way to do this is probably to write an interceptor.
>  
>
>>
>>
>> Thanks!
>>
>> -- 
>> --pau
>>
>
>
> -- 
> Mark D. Roth >
> Software Engineer
> Google, Inc.
>

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/4c444986-66c1-4a6a-b551-b7cfdc3935deo%40googlegroups.com.

Re: [grpc-io] Re: gRPC A6: Retries

['Mark D. Roth' via grpc.io]

On Thu, Jun 27, 2019 at 6:36 AM Pau Freixes  wrote:

> Hi Mark,
>
> Thanks for the update
>
> > The current implementation in C-core is only partially complete.  The
> basic retry code is there, but there is still an outstanding design
> question of how we handle stats for retries, and there is not yet any
> support for transparent retries nor for hedging.  And even the basic retry
> code is extremely invasive and has not yet received any production testing,
> so there are probably numerous bugs waiting to be found.
>
> What do you mean by "there is not yet any support for transparent
> retries"? I thought hat retries were done under the hood transparently
> when some conditions are met - like using the response codes.
>

Transparent retries are the ones described in this section of the spec:

https://github.com/grpc/proposal/blob/master/A6-client-retries.md#transparent-retries

We have not yet implemented that functionality in C-core.


>
> Regarding the stats thing is it related to having the capacity of
> retrying when the number of "errors" is lower than a specific
> threshold?
>

No, it's a more basic problem than that.  Whenever there are multiple
attempts on a given RPC, the additional attempts don't show up at all in
stats recorded via systems like census.


>
> Thanks for the advice on not using this in production. Does it mean
> that gRPC community still believe that the way to go for having a all
> of the needs covered for calling external dependencies - retrying,
> circuit breakers, etc - it's by implementing their own wrappers on top
> of the gRPC clients? What is being done within google right now?
>

At the moment, the best way to do this is probably to write an interceptor.


>
>
> Thanks!
>
> --
> --pau
>


-- 
Mark D. Roth 
Software Engineer
Google, Inc.

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/CAJgPXp6L95movuXVZfDcQyCFqek6KSQNCKLmhAUoLChHCeK%3DwQ%40mail.gmail.com.

Re: [grpc-io] Re: gRPC A6: Retries

[Pau Freixes]

Hi Mark,

Thanks for the update

> The current implementation in C-core is only partially complete.  The basic 
> retry code is there, but there is still an outstanding design question of how 
> we handle stats for retries, and there is not yet any support for transparent 
> retries nor for hedging.  And even the basic retry code is extremely invasive 
> and has not yet received any production testing, so there are probably 
> numerous bugs waiting to be found.

What do you mean by "there is not yet any support for transparent
retries"? I thought hat retries were done under the hood transparently
when some conditions are met - like using the response codes.

Regarding the stats thing is it related to having the capacity of
retrying when the number of "errors" is lower than a specific
threshold?

Thanks for the advice on not using this in production. Does it mean
that gRPC community still believe that the way to go for having a all
of the needs covered for calling external dependencies - retrying,
circuit breakers, etc - it's by implementing their own wrappers on top
of the gRPC clients? What is being done within google right now?


Thanks!

-- 
--pau

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/CA%2BULCcFG-iQjn_dVsD6VCfBiJLFTFzcpPTJgeZEPcdLQ636qQQ%40mail.gmail.com.

Re: [grpc-io] Re: gRPC A6: Retries

['Mark D. Roth' via grpc.io]

Unfortunately, the status of this hasn't changed recently.  We will
definitely get back to this at some point, but we've got other higher
priority work that will be occupying our attention for the next few
quarters, so the soonest we might be able to get back to this would be
toward the end of the year.

The current implementation in C-core is only partially complete.  The basic
retry code is there, but there is still an outstanding design question of
how we handle stats for retries, and there is not yet any support for
transparent retries nor for hedging.  And even the basic retry code is
extremely invasive and has not yet received any production testing, so
there are probably numerous bugs waiting to be found.

I would not recommend using this code in its current state.

On Wed, Jun 26, 2019 at 8:48 AM Pau Freixes  wrote:

> Hi,
>
> Can I ask what's the status of the implementation? From what I can see in
> the code [1] this might be already implemented. I'm wondering if the retry
> feature is still experimental and if it implements everything that is
> stated in the gRFC [2] document.
>
> [1]
> https://github.com/grpc/grpc/blob/master/src/core/ext/filters/client_channel/client_channel.cc#L2193
> [2] https://github.com/grpc/proposal/blob/master/A6-client-retries.md
>
> On Saturday, February 11, 2017 at 1:31:01 AM UTC+1, ncte...@google.com
> wrote:
>>
>> I've created a gRFC describing the design and implementation plan for
>> gRPC Retries.
>>
>> Take a look at the gRPC on Github
>> .
>>
> --
> You received this message because you are subscribed to the Google Groups "
> grpc.io" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to grpc-io+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/grpc-io/d39a80df-619b-41f2-bf83-37844ad08f19%40googlegroups.com
> 
> .
>


-- 
Mark D. Roth 
Software Engineer
Google, Inc.

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/CAJgPXp61m-EQeJGFZ9LTF9rCqY0g-4s2-vkhcox%3D32NAKqi74A%40mail.gmail.com.

[grpc-io] Re: gRPC A6: Retries

[Pau Freixes]

Hi, 

Can I ask what's the status of the implementation? From what I can see in 
the code [1] this might be already implemented. I'm wondering if the retry 
feature is still experimental and if it implements everything that is 
stated in the gRFC [2] document.

[1] 
https://github.com/grpc/grpc/blob/master/src/core/ext/filters/client_channel/client_channel.cc#L2193
[2] https://github.com/grpc/proposal/blob/master/A6-client-retries.md

On Saturday, February 11, 2017 at 1:31:01 AM UTC+1, ncte...@google.com 
wrote:
>
> I've created a gRFC describing the design and implementation plan for gRPC 
> Retries.
>
> Take a look at the gRPC on Github 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/d39a80df-619b-41f2-bf83-37844ad08f19%40googlegroups.com.

Re: [grpc-io] Re: gRPC A6: Retries

['Alun Evans' via grpc.io]

Michael, Mark,

thanks for the feedback, sounds fair.

We've sometimes had the opposite experience where we have a long tail of
clients using older versions, which makes it hard to upgrade the server
side to emit a new error.

All praise the mono-repo I guess.


thanks,


A.

On Wed 20 Mar '19 at 08:40 Michael Rose  wrote:
> 
> For some more color, we (internally) have made outages worse by retrying on
> status codes we shouldn't, sometimes through multiple layers of services
> resulting in essentially DDoSing our own services. For instance if you
> retry 3 times at each client, and your service passes through N layers,
> then you have 3^N retries. A service I worked on ended up 4 layers deep
> with misconfigured retry behavior that resulted in 81 retries per top-level
> request. That was fun, attempting to slough off ~82x our normal traffic. :)
>
> Also as Mark said, it may not always be correct to retry: not all RPCs are
> idempotent and  may have state implications, so this really should be a
> case-by-case (and a code-by-code) decision. No sense in retrying something
> that isn't transient.
>
> On Wed, Mar 20, 2019, 08:06 'Mark D. Roth' via grpc.io <
> grpc-io@googlegroups.com> wrote:
>
>> In general, unless an application is explicitly designed to allow an RPC
>> to be retried, it's not safe to do so.  As a result, we wanted service
>> owners to make an explicit choice about which ones they deem safe to retry,
>> rather than accidentally configuring retries in a case where it's not safe.
>>
>> On Tue, Mar 19, 2019 at 7:05 PM alun via grpc.io 
>> wrote:
>>
>>> I have a query about:
>>>
>>> When gRPC receives a non-OK response status from a server, this status is
 checked against the set of retryable status codes in retryableStatusCodes
 to determine if a retry attempt should be made.
>>>
>>>
>>> I was wondering why it wasn't chosen to have a set of fatalStatusCodes,
>>> to determine if a retry attempt should not be made ?
>>>
>>> - Especially with respect to Postel's law.
>>>
>>> thanks,
>>>
>>> A.
>>>
>>>
>>> On Friday, February 10, 2017 at 4:31:01 PM UTC-8, ncte...@google.com
>>> wrote:

 I've created a gRFC describing the design and implementation plan for
 gRPC Retries.

 Take a look at the gRPC on Github
 .

>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "grpc.io" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to grpc-io+unsubscr...@googlegroups.com.
>>> To post to this group, send email to grpc-io@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/grpc-io.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/grpc-io/b78c2861-49ea-4fe3-a0dd-70e5ed199432%40googlegroups.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>> --
>> Mark D. Roth 
>> Software Engineer
>> Google, Inc.
>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "grpc.io" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/grpc-io/zzHIICbwTZE/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> grpc-io+unsubscr...@googlegroups.com.
>> To post to this group, send email to grpc-io@googlegroups.com.
>> Visit this group at https://groups.google.com/group/grpc-io.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/grpc-io/CAJgPXp7_9vhjoJEy%2Bb-t%2B70ooZwbZ8FWZte2wiL93M1LAAN6hg%40mail.gmail.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>

-- 
Alun Evans

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To post to this group, send email to grpc-io@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/m2bm25mp33.fsf%40uber.com.
For more options, visit https://groups.google.com/d/optout.

Re: [grpc-io] Re: gRPC A6: Retries

[Michael Rose]

For some more color, we (internally) have made outages worse by retrying on
status codes we shouldn't, sometimes through multiple layers of services
resulting in essentially DDoSing our own services. For instance if you
retry 3 times at each client, and your service passes through N layers,
then you have 3^N retries. A service I worked on ended up 4 layers deep
with misconfigured retry behavior that resulted in 81 retries per top-level
request. That was fun, attempting to slough off ~82x our normal traffic. :)

Also as Mark said, it may not always be correct to retry: not all RPCs are
idempotent and  may have state implications, so this really should be a
case-by-case (and a code-by-code) decision. No sense in retrying something
that isn't transient.

On Wed, Mar 20, 2019, 08:06 'Mark D. Roth' via grpc.io <
grpc-io@googlegroups.com> wrote:

> In general, unless an application is explicitly designed to allow an RPC
> to be retried, it's not safe to do so.  As a result, we wanted service
> owners to make an explicit choice about which ones they deem safe to retry,
> rather than accidentally configuring retries in a case where it's not safe.
>
> On Tue, Mar 19, 2019 at 7:05 PM alun via grpc.io 
> wrote:
>
>> I have a query about:
>>
>> When gRPC receives a non-OK response status from a server, this status is
>>> checked against the set of retryable status codes in retryableStatusCodes
>>> to determine if a retry attempt should be made.
>>
>>
>> I was wondering why it wasn't chosen to have a set of fatalStatusCodes,
>> to determine if a retry attempt should not be made ?
>>
>> - Especially with respect to Postel's law.
>>
>> thanks,
>>
>> A.
>>
>>
>> On Friday, February 10, 2017 at 4:31:01 PM UTC-8, ncte...@google.com
>> wrote:
>>>
>>> I've created a gRFC describing the design and implementation plan for
>>> gRPC Retries.
>>>
>>> Take a look at the gRPC on Github
>>> .
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "grpc.io" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to grpc-io+unsubscr...@googlegroups.com.
>> To post to this group, send email to grpc-io@googlegroups.com.
>> Visit this group at https://groups.google.com/group/grpc-io.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/grpc-io/b78c2861-49ea-4fe3-a0dd-70e5ed199432%40googlegroups.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
> --
> Mark D. Roth 
> Software Engineer
> Google, Inc.
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "grpc.io" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/grpc-io/zzHIICbwTZE/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> grpc-io+unsubscr...@googlegroups.com.
> To post to this group, send email to grpc-io@googlegroups.com.
> Visit this group at https://groups.google.com/group/grpc-io.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/grpc-io/CAJgPXp7_9vhjoJEy%2Bb-t%2B70ooZwbZ8FWZte2wiL93M1LAAN6hg%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To post to this group, send email to grpc-io@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/CAOMdnbKUhY3TQ8J6CgfieNiW9JW9nTZxAXXe1O%2BDSJBoh6TZsw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[grpc-io] Re: gRPC A6: Retries

[asatur440]

суббота, 11 февраля 2017 г., 3:31:01 UTC+3 пользователь ncte...@google.com 
написал:
> I've created a gRFC describing the design and implementation plan for gRPC 
> Retries.
> 
> 
> Take a look at the gRPC on Github.

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To post to this group, send email to grpc-io@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/3131ad56-9b95-40f5-a506-5c0b521d9367%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [grpc-io] Re: gRPC A6: Retries

['Mark D. Roth' via grpc.io]

In general, unless an application is explicitly designed to allow an RPC to
be retried, it's not safe to do so.  As a result, we wanted service owners
to make an explicit choice about which ones they deem safe to retry, rather
than accidentally configuring retries in a case where it's not safe.

On Tue, Mar 19, 2019 at 7:05 PM alun via grpc.io 
wrote:

> I have a query about:
>
> When gRPC receives a non-OK response status from a server, this status is
>> checked against the set of retryable status codes in retryableStatusCodes
>> to determine if a retry attempt should be made.
>
>
> I was wondering why it wasn't chosen to have a set of fatalStatusCodes, to
> determine if a retry attempt should not be made ?
>
> - Especially with respect to Postel's law.
>
> thanks,
>
> A.
>
>
> On Friday, February 10, 2017 at 4:31:01 PM UTC-8, ncte...@google.com
> wrote:
>>
>> I've created a gRFC describing the design and implementation plan for
>> gRPC Retries.
>>
>> Take a look at the gRPC on Github
>> .
>>
> --
> You received this message because you are subscribed to the Google Groups "
> grpc.io" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to grpc-io+unsubscr...@googlegroups.com.
> To post to this group, send email to grpc-io@googlegroups.com.
> Visit this group at https://groups.google.com/group/grpc-io.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/grpc-io/b78c2861-49ea-4fe3-a0dd-70e5ed199432%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Mark D. Roth 
Software Engineer
Google, Inc.

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To post to this group, send email to grpc-io@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/CAJgPXp7_9vhjoJEy%2Bb-t%2B70ooZwbZ8FWZte2wiL93M1LAAN6hg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[grpc-io] Re: gRPC A6: Retries

[alun via grpc.io]

I have a query about:

When gRPC receives a non-OK response status from a server, this status is 
> checked against the set of retryable status codes in retryableStatusCodes 
> to determine if a retry attempt should be made.


I was wondering why it wasn't chosen to have a set of fatalStatusCodes, to 
determine if a retry attempt should not be made ?

- Especially with respect to Postel's law.

thanks,

A.


On Friday, February 10, 2017 at 4:31:01 PM UTC-8, ncte...@google.com wrote:
>
> I've created a gRFC describing the design and implementation plan for gRPC 
> Retries.
>
> Take a look at the gRPC on Github 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To post to this group, send email to grpc-io@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/b78c2861-49ea-4fe3-a0dd-70e5ed199432%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [grpc-io] Re: gRPC A6: Retries

['Noah Eisen' via grpc.io]

The only use case we can think of so far would be an alternative solution
to this routing affinity and hedging interaction. We initially discussed
putting the previously tried addresses in the metadata of an RPC, and then
the actual load balancing service would have access to it. But as
mentioned, this was written off because of the extra overhead.

On Wed, Mar 1, 2017 at 3:54 PM, 'Michael Rose' via grpc.io <
grpc-io@googlegroups.com> wrote:

> > To address your comments, we will be making a small change to the load
> balancing policy with respect to hedging RPCs. The change will support
> passing the local lb_policy a list of previously used addresses. The list
> will essentially be, "if possible, don't choose one of these addresses."
> For most cases this will solve your concern about the relation between
> affinity routing and hedging.
>
> It does! Thank you for your consideration, I definitely look forward to
> testing it out.
>
> > These changes will only occur in the local lb_policy. We do not want to
> send any extra data over the wire due to performance concerns.
>
> Seems reasonable to me. Out of curiosity, are there any use cases for
> doing so (other than perhaps server-aided hedge canceling)?
>
> *Michael Rose*
> Team Lead, Identity Resolution
> *Full*Contact | fullcontact.com
> 
> m: +1.720.837.1357 | t: @xorlev
> [image: Inline image 1]
>
> We’re hiring awesome people!
> See our open positions
> 
>
> On Wed, Mar 1, 2017 at 4:51 PM, Noah Eisen  wrote:
>
>> Hi Michael,
>>
>> To address your comments, we will be making a small change to the load
>> balancing policy with respect to hedging RPCs. The change will support
>> passing the local lb_policy a list of previously used addresses. The list
>> will essentially be, "if possible, don't choose one of these addresses."
>> For most cases this will solve your concern about the relation between
>> affinity routing and hedging.
>>
>> These changes will only occur in the local lb_policy. We do not want to
>> send any extra data over the wire due to performance concerns.
>>
>> gRPC support for affinity routing is ongoing, but this change to the
>> existing policy will make it easier to have hedging and affinity routing
>> work together in the future.
>>
>> On Sun, Feb 12, 2017 at 7:26 PM,  wrote:
>>
>>> > We are not supporting explicit load balancing constraints for retries.
>>> The retry attempt or hedged RPC will be re-resolved through the
>>> load-balancer, so it's up to the service owner to ensure that this has a
>>> low-likelihood of issuing the request to the same backend.
>>>
>>> That seems fairly difficult for any service with request-dependent
>>> routing semantics. Lets use a DFS as an example: many DFSes maintain N
>>> replicas of a given file block. In the case where you send a hedged request
>>> for a block, your likelihood is 1/N of requerying the same DFS node which
>>> might well have a slow disk. At least for us using HDFS, N=3 most of the
>>> time; a therefore 33% chance of requerying the same node. Even assuming a
>>> smart load balancing service which intelligently removes poorly performing
>>> storage nodes from service, it still seems desirable to ensure hedged
>>> requests go to a different node. Not having a story for more informed load
>>> balancing seems like it makes a lot of use cases more difficult than they
>>> need to be.
>>>
>>> Regards,
>>> Michael
>>>
>>> On Sunday, February 12, 2017 at 7:24:59 PM UTC-7, Eric Gribkoff wrote:

 Hi Michael,

 Thanks for the feedback. Responses to your questions (and Josh's
 follow-up question on retry backoff times) are inline below.

 On Sat, Feb 11, 2017 at 1:57 PM, 'Michael Rose' via grpc.io <
 grp...@googlegroups.com> wrote:

> A few questions:
>
> 1) Under this design, is it possible to add a load balancing
> constraints for retried/hedged requests? Especially during hedging, I'd
> like to be able to try a different server since the original server might
> be garbage collecting or have otherwise collected a queue of requests such
> that a retry/hedge to this server will not be very useful. Or, perhaps the
> key I'm looking up lives on a specific subset of storage servers and
> therefore should be balanced to that specific subset. While that's the
> domain of a LB policy, what information will hedging/retries provide to 
> the
> LB policy?
>
>
 We are not supporting explicit load balancing constraints for retries.
 The retry attempt or hedged RPC will be re-resolved through the
 load-balancer, so it's up to the service owner t

Re: [grpc-io] Re: gRPC A6: Retries

['Michael Rose' via grpc.io]

> To address your comments, we will be making a small change to the load
balancing policy with respect to hedging RPCs. The change will support
passing the local lb_policy a list of previously used addresses. The list
will essentially be, "if possible, don't choose one of these addresses."
For most cases this will solve your concern about the relation between
affinity routing and hedging.

It does! Thank you for your consideration, I definitely look forward to
testing it out.

> These changes will only occur in the local lb_policy. We do not want to
send any extra data over the wire due to performance concerns.

Seems reasonable to me. Out of curiosity, are there any use cases for doing
so (other than perhaps server-aided hedge canceling)?

*Michael Rose*
Team Lead, Identity Resolution
*Full*Contact | fullcontact.com

m: +1.720.837.1357 | t: @xorlev
[image: Inline image 1]

We’re hiring awesome people!
See our open positions


On Wed, Mar 1, 2017 at 4:51 PM, Noah Eisen  wrote:

> Hi Michael,
>
> To address your comments, we will be making a small change to the load
> balancing policy with respect to hedging RPCs. The change will support
> passing the local lb_policy a list of previously used addresses. The list
> will essentially be, "if possible, don't choose one of these addresses."
> For most cases this will solve your concern about the relation between
> affinity routing and hedging.
>
> These changes will only occur in the local lb_policy. We do not want to
> send any extra data over the wire due to performance concerns.
>
> gRPC support for affinity routing is ongoing, but this change to the
> existing policy will make it easier to have hedging and affinity routing
> work together in the future.
>
> On Sun, Feb 12, 2017 at 7:26 PM,  wrote:
>
>> > We are not supporting explicit load balancing constraints for retries.
>> The retry attempt or hedged RPC will be re-resolved through the
>> load-balancer, so it's up to the service owner to ensure that this has a
>> low-likelihood of issuing the request to the same backend.
>>
>> That seems fairly difficult for any service with request-dependent
>> routing semantics. Lets use a DFS as an example: many DFSes maintain N
>> replicas of a given file block. In the case where you send a hedged request
>> for a block, your likelihood is 1/N of requerying the same DFS node which
>> might well have a slow disk. At least for us using HDFS, N=3 most of the
>> time; a therefore 33% chance of requerying the same node. Even assuming a
>> smart load balancing service which intelligently removes poorly performing
>> storage nodes from service, it still seems desirable to ensure hedged
>> requests go to a different node. Not having a story for more informed load
>> balancing seems like it makes a lot of use cases more difficult than they
>> need to be.
>>
>> Regards,
>> Michael
>>
>> On Sunday, February 12, 2017 at 7:24:59 PM UTC-7, Eric Gribkoff wrote:
>>>
>>> Hi Michael,
>>>
>>> Thanks for the feedback. Responses to your questions (and Josh's
>>> follow-up question on retry backoff times) are inline below.
>>>
>>> On Sat, Feb 11, 2017 at 1:57 PM, 'Michael Rose' via grpc.io <
>>> grp...@googlegroups.com> wrote:
>>>
 A few questions:

 1) Under this design, is it possible to add a load balancing
 constraints for retried/hedged requests? Especially during hedging, I'd
 like to be able to try a different server since the original server might
 be garbage collecting or have otherwise collected a queue of requests such
 that a retry/hedge to this server will not be very useful. Or, perhaps the
 key I'm looking up lives on a specific subset of storage servers and
 therefore should be balanced to that specific subset. While that's the
 domain of a LB policy, what information will hedging/retries provide to the
 LB policy?


>>> We are not supporting explicit load balancing constraints for retries.
>>> The retry attempt or hedged RPC will be re-resolved through the
>>> load-balancer, so it's up to the service owner to ensure that this has a
>>> low-likelihood of issuing the request to the same backend. This is part of
>>> a decision to keep the retry design as simple as possible while satisfying
>>> the majority of use cases. If your load-balancing policy has a high
>>> likelihood of sending requests to the same server each time, hedging (and
>>> to some extent retries) will be less useful regardless. There will be
>>> metadata attached to the call indicating that it's a retry, but it won't
>>> include information about which servers the previous requests went to.
>>>
>>>
>>

Re: [grpc-io] Re: gRPC A6: Retries

['Noah Eisen' via grpc.io]

Hi Michael,

To address your comments, we will be making a small change to the load
balancing policy with respect to hedging RPCs. The change will support
passing the local lb_policy a list of previously used addresses. The list
will essentially be, "if possible, don't choose one of these addresses."
For most cases this will solve your concern about the relation between
affinity routing and hedging.

These changes will only occur in the local lb_policy. We do not want to
send any extra data over the wire due to performance concerns.

gRPC support for affinity routing is ongoing, but this change to the
existing policy will make it easier to have hedging and affinity routing
work together in the future.

On Sun, Feb 12, 2017 at 7:26 PM,  wrote:

> > We are not supporting explicit load balancing constraints for retries.
> The retry attempt or hedged RPC will be re-resolved through the
> load-balancer, so it's up to the service owner to ensure that this has a
> low-likelihood of issuing the request to the same backend.
>
> That seems fairly difficult for any service with request-dependent routing
> semantics. Lets use a DFS as an example: many DFSes maintain N replicas of
> a given file block. In the case where you send a hedged request for a
> block, your likelihood is 1/N of requerying the same DFS node which might
> well have a slow disk. At least for us using HDFS, N=3 most of the time; a
> therefore 33% chance of requerying the same node. Even assuming a smart
> load balancing service which intelligently removes poorly performing
> storage nodes from service, it still seems desirable to ensure hedged
> requests go to a different node. Not having a story for more informed load
> balancing seems like it makes a lot of use cases more difficult than they
> need to be.
>
> Regards,
> Michael
>
> On Sunday, February 12, 2017 at 7:24:59 PM UTC-7, Eric Gribkoff wrote:
>>
>> Hi Michael,
>>
>> Thanks for the feedback. Responses to your questions (and Josh's
>> follow-up question on retry backoff times) are inline below.
>>
>> On Sat, Feb 11, 2017 at 1:57 PM, 'Michael Rose' via grpc.io <
>> grp...@googlegroups.com> wrote:
>>
>>> A few questions:
>>>
>>> 1) Under this design, is it possible to add a load balancing constraints
>>> for retried/hedged requests? Especially during hedging, I'd like to be able
>>> to try a different server since the original server might be garbage
>>> collecting or have otherwise collected a queue of requests such that a
>>> retry/hedge to this server will not be very useful. Or, perhaps the key I'm
>>> looking up lives on a specific subset of storage servers and therefore
>>> should be balanced to that specific subset. While that's the domain of a LB
>>> policy, what information will hedging/retries provide to the LB policy?
>>>
>>>
>> We are not supporting explicit load balancing constraints for retries.
>> The retry attempt or hedged RPC will be re-resolved through the
>> load-balancer, so it's up to the service owner to ensure that this has a
>> low-likelihood of issuing the request to the same backend. This is part of
>> a decision to keep the retry design as simple as possible while satisfying
>> the majority of use cases. If your load-balancing policy has a high
>> likelihood of sending requests to the same server each time, hedging (and
>> to some extent retries) will be less useful regardless. There will be
>> metadata attached to the call indicating that it's a retry, but it won't
>> include information about which servers the previous requests went to.
>>
>>
>>
>>> 2) "Clients cannot override retry policy set by the service config." --
>>> is this intended for inside Google? How about gRPC users outside of Google
>>> which don't use the DNS mechanism to push configuration? It seems like
>>> having a client override for retry/hedging policy is pragmatic.
>>>
>>>
>> In general, we don't want to support client specification of retry
>> policies. The necessary information about what methods are safe to retry or
>> hedge, the potential for increased load, etc., are really decisions that
>> should be left to the service owner. The retry policy will definitely be a
>> part of the service config. While there are still some security-related
>> discussions about the exact delivery mechanism for the service config and
>> retry policies, I think your concern here  should be part of the service
>> config design discussion rather than something specific to retry support.
>>
>>
>>> 3) Retry backoff time -- if I'm reading it right, it will always retry
>>> in random(0, current_backoff) milliseconds. What's your feeling on this vs.
>>> a retry w/ configurable jitter parameter (e.x. linear 1000ms increase w/
>>> 10% jitter). Is it OK if there's no minimum backoff?
>>>
>>>
>> You are reading the backoff time correctly. There are a number of ways of
>> doing this, (see https://www.awsarchitectureblog.com/2015/03/backoff.html)
>> but choosing between random(0, current_backoff) is done intentionally

Re: [grpc-io] Re: gRPC A6: Retries

['Mark D. Roth' via grpc.io]

(+dgq)

I think this is actually a question to be addressed in the load-balancing
affinity design, which David is working on.  I suspect that the main thing
we need to do is to expose the request metadata that indicates that a
request is a retry to the LB policy, so that it can use that information to
make its decision.  Then it's up to the LB policy to notice that a request
is a retry and apply any necessary logic for that case.


On Sun, Feb 12, 2017 at 7:26 PM,  wrote:

> > We are not supporting explicit load balancing constraints for retries.
> The retry attempt or hedged RPC will be re-resolved through the
> load-balancer, so it's up to the service owner to ensure that this has a
> low-likelihood of issuing the request to the same backend.
>
> That seems fairly difficult for any service with request-dependent routing
> semantics. Lets use a DFS as an example: many DFSes maintain N replicas of
> a given file block. In the case where you send a hedged request for a
> block, your likelihood is 1/N of requerying the same DFS node which might
> well have a slow disk. At least for us using HDFS, N=3 most of the time; a
> therefore 33% chance of requerying the same node. Even assuming a smart
> load balancing service which intelligently removes poorly performing
> storage nodes from service, it still seems desirable to ensure hedged
> requests go to a different node. Not having a story for more informed load
> balancing seems like it makes a lot of use cases more difficult than they
> need to be.
>
> Regards,
> Michael
>
> On Sunday, February 12, 2017 at 7:24:59 PM UTC-7, Eric Gribkoff wrote:
>>
>> Hi Michael,
>>
>> Thanks for the feedback. Responses to your questions (and Josh's
>> follow-up question on retry backoff times) are inline below.
>>
>> On Sat, Feb 11, 2017 at 1:57 PM, 'Michael Rose' via grpc.io <
>> grp...@googlegroups.com> wrote:
>>
>>> A few questions:
>>>
>>> 1) Under this design, is it possible to add a load balancing constraints
>>> for retried/hedged requests? Especially during hedging, I'd like to be able
>>> to try a different server since the original server might be garbage
>>> collecting or have otherwise collected a queue of requests such that a
>>> retry/hedge to this server will not be very useful. Or, perhaps the key I'm
>>> looking up lives on a specific subset of storage servers and therefore
>>> should be balanced to that specific subset. While that's the domain of a LB
>>> policy, what information will hedging/retries provide to the LB policy?
>>>
>>>
>> We are not supporting explicit load balancing constraints for retries.
>> The retry attempt or hedged RPC will be re-resolved through the
>> load-balancer, so it's up to the service owner to ensure that this has a
>> low-likelihood of issuing the request to the same backend. This is part of
>> a decision to keep the retry design as simple as possible while satisfying
>> the majority of use cases. If your load-balancing policy has a high
>> likelihood of sending requests to the same server each time, hedging (and
>> to some extent retries) will be less useful regardless. There will be
>> metadata attached to the call indicating that it's a retry, but it won't
>> include information about which servers the previous requests went to.
>>
>>
>>
>>> 2) "Clients cannot override retry policy set by the service config." --
>>> is this intended for inside Google? How about gRPC users outside of Google
>>> which don't use the DNS mechanism to push configuration? It seems like
>>> having a client override for retry/hedging policy is pragmatic.
>>>
>>>
>> In general, we don't want to support client specification of retry
>> policies. The necessary information about what methods are safe to retry or
>> hedge, the potential for increased load, etc., are really decisions that
>> should be left to the service owner. The retry policy will definitely be a
>> part of the service config. While there are still some security-related
>> discussions about the exact delivery mechanism for the service config and
>> retry policies, I think your concern here  should be part of the service
>> config design discussion rather than something specific to retry support.
>>
>>
>>> 3) Retry backoff time -- if I'm reading it right, it will always retry
>>> in random(0, current_backoff) milliseconds. What's your feeling on this vs.
>>> a retry w/ configurable jitter parameter (e.x. linear 1000ms increase w/
>>> 10% jitter). Is it OK if there's no minimum backoff?
>>>
>>>
>> You are reading the backoff time correctly. There are a number of ways of
>> doing this, (see https://www.awsarchitectureblog.com/2015/03/backoff.html)
>> but choosing between random(0, current_backoff) is done intentionally and
>> should generally give the best results. We do not want a configurable
>> "jitter" parameter. Empirically, the retries should have more varied
>> backoff time, and we also do not want to let service owners specify very
>> low values for jitter (e.g., 1% or even 0), as 

Re: [grpc-io] Re: gRPC A6: Retries

[Josh Humphries]

On Sun, Feb 12, 2017 at 9:24 PM, 'Eric Gribkoff' via grpc.io <
grpc-io@googlegroups.com> wrote:

> Hi Michael,
>
> Thanks for the feedback. Responses to your questions (and Josh's follow-up
> question on retry backoff times) are inline below.
>
> On Sat, Feb 11, 2017 at 1:57 PM, 'Michael Rose' via grpc.io <
> grpc-io@googlegroups.com> wrote:
>
>> A few questions:
>>
>> 1) Under this design, is it possible to add a load balancing constraints
>> for retried/hedged requests? Especially during hedging, I'd like to be able
>> to try a different server since the original server might be garbage
>> collecting or have otherwise collected a queue of requests such that a
>> retry/hedge to this server will not be very useful. Or, perhaps the key I'm
>> looking up lives on a specific subset of storage servers and therefore
>> should be balanced to that specific subset. While that's the domain of a LB
>> policy, what information will hedging/retries provide to the LB policy?
>>
>>
> We are not supporting explicit load balancing constraints for retries. The
> retry attempt or hedged RPC will be re-resolved through the load-balancer,
> so it's up to the service owner to ensure that this has a low-likelihood of
> issuing the request to the same backend. This is part of a decision to keep
> the retry design as simple as possible while satisfying the majority of use
> cases. If your load-balancing policy has a high likelihood of sending
> requests to the same server each time, hedging (and to some extent retries)
> will be less useful regardless. There will be metadata attached to the call
> indicating that it's a retry, but it won't include information about which
> servers the previous requests went to.
>
>
>
>> 2) "Clients cannot override retry policy set by the service config." --
>> is this intended for inside Google? How about gRPC users outside of Google
>> which don't use the DNS mechanism to push configuration? It seems like
>> having a client override for retry/hedging policy is pragmatic.
>>
>>
> In general, we don't want to support client specification of retry
> policies. The necessary information about what methods are safe to retry or
> hedge, the potential for increased load, etc., are really decisions that
> should be left to the service owner. The retry policy will definitely be a
> part of the service config. While there are still some security-related
> discussions about the exact delivery mechanism for the service config and
> retry policies, I think your concern here  should be part of the service
> config design discussion rather than something specific to retry support.
>
>
>> 3) Retry backoff time -- if I'm reading it right, it will always retry in
>> random(0, current_backoff) milliseconds. What's your feeling on this vs. a
>> retry w/ configurable jitter parameter (e.x. linear 1000ms increase w/ 10%
>> jitter). Is it OK if there's no minimum backoff?
>>
>>
> You are reading the backoff time correctly. There are a number of ways of
> doing this, (see https://www.awsarchitectureblog.com/2015/03/backoff.html)
> but choosing between random(0, current_backoff) is done intentionally and
> should generally give the best results. We do not want a configurable
> "jitter" parameter. Empirically, the retries should have more varied
> backoff time, and we also do not want to let service owners specify very
> low values for jitter (e.g., 1% or even 0), as this would cluster all
> retries tightly together and further contribute to server overloading.
>

In that case, perhaps it should be random(0, 2*current_backoff) so that the
mean is the targeted backoff (with effectively 100% jitter). Otherwise,
documentation will need to be very clear that the actual expected value for
backoff is 1/2 of any configured values.


>
> Best,
>
> Eric Gribkoff
>
>
> Regards,
>> Michael
>>
>> On Friday, February 10, 2017 at 5:31:01 PM UTC-7, ncte...@google.com
>> wrote:
>>>
>>> I've created a gRFC describing the design and implementation plan for
>>> gRPC Retries.
>>>
>>> Take a look at the gRPC on Github
>>> .
>>>
>>
>> *CONFIDENTIALITY NOTICE: This email message, and any documents, files or
>> previous e-mail messages attached to it is for the sole use of the intended
>> recipient(s) and may contain confidential and privileged information. Any
>> unauthorized review, use, disclosure or distribution is prohibited. If you
>> are not the intended recipient, please contact the sender by reply email
>> and destroy all copies of the original message.*
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "grpc.io" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to grpc-io+unsubscr...@googlegroups.com.
>> To post to this group, send email to grpc-io@googlegroups.com.
>> Visit this group at https://groups.google.com/group/grpc-io.
>> To view this discussion on the web visit https://groups.google.com/d/ms
>> gid/grpc

Re: [grpc-io] Re: gRPC A6: Retries

[elementation]

> We are not supporting explicit load balancing constraints for retries. 
The retry attempt or hedged RPC will be re-resolved through the 
load-balancer, so it's up to the service owner to ensure that this has a 
low-likelihood of issuing the request to the same backend.

That seems fairly difficult for any service with request-dependent routing 
semantics. Lets use a DFS as an example: many DFSes maintain N replicas of 
a given file block. In the case where you send a hedged request for a 
block, your likelihood is 1/N of requerying the same DFS node which might 
well have a slow disk. At least for us using HDFS, N=3 most of the time; a 
therefore 33% chance of requerying the same node. Even assuming a smart 
load balancing service which intelligently removes poorly performing 
storage nodes from service, it still seems desirable to ensure hedged 
requests go to a different node. Not having a story for more informed load 
balancing seems like it makes a lot of use cases more difficult than they 
need to be.

Regards,
Michael

On Sunday, February 12, 2017 at 7:24:59 PM UTC-7, Eric Gribkoff wrote:
>
> Hi Michael,
>
> Thanks for the feedback. Responses to your questions (and Josh's follow-up 
> question on retry backoff times) are inline below.
>
> On Sat, Feb 11, 2017 at 1:57 PM, 'Michael Rose' via grpc.io <
> grp...@googlegroups.com > wrote:
>
>> A few questions:
>>
>> 1) Under this design, is it possible to add a load balancing constraints 
>> for retried/hedged requests? Especially during hedging, I'd like to be able 
>> to try a different server since the original server might be garbage 
>> collecting or have otherwise collected a queue of requests such that a 
>> retry/hedge to this server will not be very useful. Or, perhaps the key I'm 
>> looking up lives on a specific subset of storage servers and therefore 
>> should be balanced to that specific subset. While that's the domain of a LB 
>> policy, what information will hedging/retries provide to the LB policy?
>>
>>
> We are not supporting explicit load balancing constraints for retries. The 
> retry attempt or hedged RPC will be re-resolved through the load-balancer, 
> so it's up to the service owner to ensure that this has a low-likelihood of 
> issuing the request to the same backend. This is part of a decision to keep 
> the retry design as simple as possible while satisfying the majority of use 
> cases. If your load-balancing policy has a high likelihood of sending 
> requests to the same server each time, hedging (and to some extent retries) 
> will be less useful regardless. There will be metadata attached to the call 
> indicating that it's a retry, but it won't include information about which 
> servers the previous requests went to.
>
>  
>
>> 2) "Clients cannot override retry policy set by the service config." -- 
>> is this intended for inside Google? How about gRPC users outside of Google 
>> which don't use the DNS mechanism to push configuration? It seems like 
>> having a client override for retry/hedging policy is pragmatic.
>>
>>
> In general, we don't want to support client specification of retry 
> policies. The necessary information about what methods are safe to retry or 
> hedge, the potential for increased load, etc., are really decisions that 
> should be left to the service owner. The retry policy will definitely be a 
> part of the service config. While there are still some security-related 
> discussions about the exact delivery mechanism for the service config and 
> retry policies, I think your concern here  should be part of the service 
> config design discussion rather than something specific to retry support.
>  
>
>> 3) Retry backoff time -- if I'm reading it right, it will always retry in 
>> random(0, current_backoff) milliseconds. What's your feeling on this vs. a 
>> retry w/ configurable jitter parameter (e.x. linear 1000ms increase w/ 10% 
>> jitter). Is it OK if there's no minimum backoff?
>>
>>
> You are reading the backoff time correctly. There are a number of ways of 
> doing this, (see https://www.awsarchitectureblog.com/2015/03/backoff.html) 
> but choosing between random(0, current_backoff) is done intentionally and 
> should generally give the best results. We do not want a configurable 
> "jitter" parameter. Empirically, the retries should have more varied 
> backoff time, and we also do not want to let service owners specify very 
> low values for jitter (e.g., 1% or even 0), as this would cluster all 
> retries tightly together and further contribute to server overloading.
>
> Best,
>
> Eric Gribkoff
>  
>
> Regards,
>> Michael
>>
>> On Friday, February 10, 2017 at 5:31:01 PM UTC-7, ncte...@google.com 
>> wrote:
>>>
>>> I've created a gRFC describing the design and implementation plan for 
>>> gRPC Retries.
>>>
>>> Take a look at the gRPC on Github 
>>> .
>>>
>>
>> *CONFIDENTIALITY NOTICE: This email message, and any documents, files or 
>> previous e-m

Re: [grpc-io] Re: gRPC A6: Retries

['Eric Gribkoff' via grpc.io]

Hi Michael,

Thanks for the feedback. Responses to your questions (and Josh's follow-up
question on retry backoff times) are inline below.

On Sat, Feb 11, 2017 at 1:57 PM, 'Michael Rose' via grpc.io <
grpc-io@googlegroups.com> wrote:

> A few questions:
>
> 1) Under this design, is it possible to add a load balancing constraints
> for retried/hedged requests? Especially during hedging, I'd like to be able
> to try a different server since the original server might be garbage
> collecting or have otherwise collected a queue of requests such that a
> retry/hedge to this server will not be very useful. Or, perhaps the key I'm
> looking up lives on a specific subset of storage servers and therefore
> should be balanced to that specific subset. While that's the domain of a LB
> policy, what information will hedging/retries provide to the LB policy?
>
>
We are not supporting explicit load balancing constraints for retries. The
retry attempt or hedged RPC will be re-resolved through the load-balancer,
so it's up to the service owner to ensure that this has a low-likelihood of
issuing the request to the same backend. This is part of a decision to keep
the retry design as simple as possible while satisfying the majority of use
cases. If your load-balancing policy has a high likelihood of sending
requests to the same server each time, hedging (and to some extent retries)
will be less useful regardless. There will be metadata attached to the call
indicating that it's a retry, but it won't include information about which
servers the previous requests went to.



> 2) "Clients cannot override retry policy set by the service config." -- is
> this intended for inside Google? How about gRPC users outside of Google
> which don't use the DNS mechanism to push configuration? It seems like
> having a client override for retry/hedging policy is pragmatic.
>
>
In general, we don't want to support client specification of retry
policies. The necessary information about what methods are safe to retry or
hedge, the potential for increased load, etc., are really decisions that
should be left to the service owner. The retry policy will definitely be a
part of the service config. While there are still some security-related
discussions about the exact delivery mechanism for the service config and
retry policies, I think your concern here  should be part of the service
config design discussion rather than something specific to retry support.


> 3) Retry backoff time -- if I'm reading it right, it will always retry in
> random(0, current_backoff) milliseconds. What's your feeling on this vs. a
> retry w/ configurable jitter parameter (e.x. linear 1000ms increase w/ 10%
> jitter). Is it OK if there's no minimum backoff?
>
>
You are reading the backoff time correctly. There are a number of ways of
doing this, (see https://www.awsarchitectureblog.com/2015/03/backoff.html)
but choosing between random(0, current_backoff) is done intentionally and
should generally give the best results. We do not want a configurable
"jitter" parameter. Empirically, the retries should have more varied
backoff time, and we also do not want to let service owners specify very
low values for jitter (e.g., 1% or even 0), as this would cluster all
retries tightly together and further contribute to server overloading.

Best,

Eric Gribkoff


Regards,
> Michael
>
> On Friday, February 10, 2017 at 5:31:01 PM UTC-7, ncte...@google.com
> wrote:
>>
>> I've created a gRFC describing the design and implementation plan for
>> gRPC Retries.
>>
>> Take a look at the gRPC on Github
>> .
>>
>
> *CONFIDENTIALITY NOTICE: This email message, and any documents, files or
> previous e-mail messages attached to it is for the sole use of the intended
> recipient(s) and may contain confidential and privileged information. Any
> unauthorized review, use, disclosure or distribution is prohibited. If you
> are not the intended recipient, please contact the sender by reply email
> and destroy all copies of the original message.*
>
> --
> You received this message because you are subscribed to the Google Groups "
> grpc.io" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to grpc-io+unsubscr...@googlegroups.com.
> To post to this group, send email to grpc-io@googlegroups.com.
> Visit this group at https://groups.google.com/group/grpc-io.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/grpc-io/62809dba-3349-4a60-9aa9-ccc044d27f53%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To post to t

Re: [grpc-io] Re: gRPC A6: Retries

[Josh Humphries]

On Sat, Feb 11, 2017 at 4:57 PM, 'Michael Rose' via grpc.io <
grpc-io@googlegroups.com> wrote:

> A few questions:
>
> 1) Under this design, is it possible to add a load balancing constraints
> for retried/hedged requests? Especially during hedging, I'd like to be able
> to try a different server since the original server might be garbage
> collecting or have otherwise collected a queue of requests such that a
> retry/hedge to this server will not be very useful. Or, perhaps the key I'm
> looking up lives on a specific subset of storage servers and therefore
> should be balanced to that specific subset. While that's the domain of a LB
> policy, what information will hedging/retries provide to the LB policy?
>
> 2) "Clients cannot override retry policy set by the service config." -- is
> this intended for inside Google? How about gRPC users outside of Google
> which don't use the DNS mechanism to push configuration? It seems like
> having a client override for retry/hedging policy is pragmatic.
>
> 3) Retry backoff time -- if I'm reading it right, it will always retry in
> random(0, current_backoff) milliseconds. What's your feeling on this vs. a
> retry w/ configurable jitter parameter (e.x. linear 1000ms increase w/ 10%
> jitter). Is it OK if there's no minimum backoff?
>


I was about to ask the same thing :)

The current text:
The failed RPCs will be retried after *x* seconds, where *x* is
defined as random(0,
current_backoff).

What I would instead expect:
The failed RPCs will be retried after *x* seconds, where *x* is
defined as random(current_backoff*(1-jitter),
current_backoff*(1+jitter)) where jitter would be 0.25, for example, to
indicate 25% jitter.



> Regards,
> Michael
>
> On Friday, February 10, 2017 at 5:31:01 PM UTC-7, ncte...@google.com
> wrote:
>>
>> I've created a gRFC describing the design and implementation plan for
>> gRPC Retries.
>>
>> Take a look at the gRPC on Github
>> .
>>
>
> *CONFIDENTIALITY NOTICE: This email message, and any documents, files or
> previous e-mail messages attached to it is for the sole use of the intended
> recipient(s) and may contain confidential and privileged information. Any
> unauthorized review, use, disclosure or distribution is prohibited. If you
> are not the intended recipient, please contact the sender by reply email
> and destroy all copies of the original message.*
>
> --
> You received this message because you are subscribed to the Google Groups "
> grpc.io" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to grpc-io+unsubscr...@googlegroups.com.
> To post to this group, send email to grpc-io@googlegroups.com.
> Visit this group at https://groups.google.com/group/grpc-io.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/grpc-io/62809dba-3349-4a60-9aa9-ccc044d27f53%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To post to this group, send email to grpc-io@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/CAO78j%2BJDAH1LQ5-OjdtUwKaE%2BJGg2y6p5-Gte3BMtV0nSj2dZg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[grpc-io] Re: gRPC A6: Retries

['Michael Rose' via grpc.io]

A few questions:

1) Under this design, is it possible to add a load balancing constraints 
for retried/hedged requests? Especially during hedging, I'd like to be able 
to try a different server since the original server might be garbage 
collecting or have otherwise collected a queue of requests such that a 
retry/hedge to this server will not be very useful. Or, perhaps the key I'm 
looking up lives on a specific subset of storage servers and therefore 
should be balanced to that specific subset. While that's the domain of a LB 
policy, what information will hedging/retries provide to the LB policy?

2) "Clients cannot override retry policy set by the service config." -- is 
this intended for inside Google? How about gRPC users outside of Google 
which don't use the DNS mechanism to push configuration? It seems like 
having a client override for retry/hedging policy is pragmatic.

3) Retry backoff time -- if I'm reading it right, it will always retry in 
random(0, current_backoff) milliseconds. What's your feeling on this vs. a 
retry w/ configurable jitter parameter (e.x. linear 1000ms increase w/ 10% 
jitter). Is it OK if there's no minimum backoff?

Regards,
Michael

On Friday, February 10, 2017 at 5:31:01 PM UTC-7, ncte...@google.com wrote:
>
> I've created a gRFC describing the design and implementation plan for gRPC 
> Retries.
>
> Take a look at the gRPC on Github 
> .
>

-- 
*CONFIDENTIALITY NOTICE: This email message, and any documents, files or 
previous e-mail messages attached to it is for the sole use of the intended 
recipient(s) and may contain confidential and privileged information. Any 
unauthorized review, use, disclosure or distribution is prohibited. If you 
are not the intended recipient, please contact the sender by reply email 
and destroy all copies of the original message.*

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To post to this group, send email to grpc-io@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/62809dba-3349-4a60-9aa9-ccc044d27f53%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.