Re: [grpc-io] gRPC not dropping disconnected channel

['Lidi Zheng' via grpc.io]

AFAIK, the transparent retries is not fully implemented.

The feature you want probably is "wait_for_ready
". In
case of TRANSIENT_FAILURE (server not available temporarily), it will
automatically wait for the channel become READY again without failing. Read
more

about
wait for ready.

Lidi Zheng

On Mon, Apr 15, 2019 at 11:41 AM Manatsawin Hanmongkolchai <
manatsa...@gmail.com> wrote:

> Thank you for your answer Lidi and Sidhartha,
>
> I've updated the code in GitHub repository to gracefully stop server.
> Unfortunately, the same problem still occurs.
> Previously I also have tested the same issue using Python client and a
> Go server (stopping by manually ctrl+c the server) which also has the
> same problem.
>
> In proposal A6 section "Transparent Retries" mentions that in case of
> RPC failure which RPC never leaves the client, it will be retried
> automatically.
> I've confirmed using Wireshark that the RPC never got sent out in the
> wire, and there is only 2 outgoing calls (no retry).
> I'm not sure whether the proposal is implemented or I have to set a
> settings somewhere?
>
> On Tue, Apr 16, 2019 at 1:17 AM Lidi Zheng  wrote:
> >
> > gRPC Python server can be closed by "server.stop()".
> >
> > Underlying, the gRPC Python server has a background polling thread. The
> deallocation of the server object is not directly terminating the polling
> thread since there is no way to interrupt thread in Python. So, during the
> server deallocation, it will set a destruction flag, and if the polling
> thread sees the flag it will terminate itself. It's possible that there are
> some lag before the server completely shuts down. To eliminate this issue,
> we recommend you close the server explicitly by "server.stop()" which will
> block until the server is completely shut down.
> >
> > Lidi Zheng
> >
> > On Mon, Apr 15, 2019 at 11:09 AM Sidhartha Thota 
> wrote:
> >>
> >> This is known issue afaik. May be your issue is related to this one:
> https://github.com/grpc/grpc/issues/9767
> >>
> >> If the connection is down the session will not be re-useful but channel
> is always useful. Here session is individual RPC/Streaming RPC call. So,
> when session is in progress and connection is down then session cannot be
> restored. Channel is different and maintains connection with other end
> (here server) and retries to connect periodically so it takes time to
> reconnect to to server.
> >>
> >>
> >>
> >> On Mon, Apr 15, 2019 at 8:49 AM  wrote:
> >>>
> >>> Steps to reproduce
> >>>
> >>> 1. Start server
> >>> 2. Send a client RPC to server
> >>> 3. Restart server
> >>> 4. Using the same client, send another RPC. The call will fail
> >>> 5. Send another RPC, this call will success
> >>>
> >>> Also I found that if the server is leave stopped for a long time
> before starting up again, the call in step 5 will return "channel is in
> state TRANSIENT_FAILURE" as well.
> >>>
> >>> Example code: https://github.com/whs/grpc-repro
> >>> (Install from requirements.txt then run main.py)
> >>>
> >>> Expected result
> >>>
> >>> All call should success.
> >>>
> >>> Tested with Python grpcio==1.19.0 server/client and with go-grpc
> server. I tried setting grpc.max_connection_age_grace_ms,
> grpc.max_connection_age_ms, grpc.max_connection_idle_ms,
> grpc.keepalive_time_ms, grpc.keepalive_permit_without_calls but they
> doesn't seems to help.
> >>>
> >>> From my own investigation, I believe that gRPC channels doesn't drop
> closed channels and tries to send calls in closed channel. There is also
> TCP RST sent.
> >>>
> >>> --
> >>> You received this message because you are subscribed to the Google
> Groups "grpc.io" group.
> >>> To unsubscribe from this group and stop receiving emails from it, send
> an email to grpc-io+unsubscr...@googlegroups.com.
> >>> To post to this group, send email to grpc-io@googlegroups.com.
> >>> Visit this group at https://groups.google.com/group/grpc-io.
> >>> To view this discussion on the web visit
> https://groups.google.com/d/msgid/grpc-io/e8465f2b-96f1-4ad1-816f-ee6521595179%40googlegroups.com
> .
> >>> For more options, visit https://groups.google.com/d/optout.
> >>
> >>
> >>
> >> --
> >> Thanks,
> >> Sidhartha Thota.
> >>
> >> --
> >> You received this message because you are subscribed to the Google
> Groups "grpc.io" group.
> >> To unsubscribe from this group and stop receiving emails from it, send
> an email to grpc-io+unsubscr...@googlegroups.com.
> >> To post to this group, send email to grpc-io@googlegroups.com.
> >> Visit this group at https://groups.google.com/group/grpc-io.
> >> To view this discussion on the web visit
> https://groups.google.com/d/msgid/grpc-io/CALguN-Gg0iiDSEUrO-VEOuGu5Mhemc9bZVqxqi_BcY7-pc%3DQWg%40mail.gmail.com
> .
> >> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are sub

Re: [grpc-io] gRPC not dropping disconnected channel

[Manatsawin Hanmongkolchai]

Thank you for your answer Lidi and Sidhartha,

I've updated the code in GitHub repository to gracefully stop server.
Unfortunately, the same problem still occurs.
Previously I also have tested the same issue using Python client and a
Go server (stopping by manually ctrl+c the server) which also has the
same problem.

In proposal A6 section "Transparent Retries" mentions that in case of
RPC failure which RPC never leaves the client, it will be retried
automatically.
I've confirmed using Wireshark that the RPC never got sent out in the
wire, and there is only 2 outgoing calls (no retry).
I'm not sure whether the proposal is implemented or I have to set a
settings somewhere?

On Tue, Apr 16, 2019 at 1:17 AM Lidi Zheng  wrote:
>
> gRPC Python server can be closed by "server.stop()".
>
> Underlying, the gRPC Python server has a background polling thread. The 
> deallocation of the server object is not directly terminating the polling 
> thread since there is no way to interrupt thread in Python. So, during the 
> server deallocation, it will set a destruction flag, and if the polling 
> thread sees the flag it will terminate itself. It's possible that there are 
> some lag before the server completely shuts down. To eliminate this issue, we 
> recommend you close the server explicitly by "server.stop()" which will block 
> until the server is completely shut down.
>
> Lidi Zheng
>
> On Mon, Apr 15, 2019 at 11:09 AM Sidhartha Thota  
> wrote:
>>
>> This is known issue afaik. May be your issue is related to this one: 
>> https://github.com/grpc/grpc/issues/9767
>>
>> If the connection is down the session will not be re-useful but channel is 
>> always useful. Here session is individual RPC/Streaming RPC call. So, when 
>> session is in progress and connection is down then session cannot be 
>> restored. Channel is different and maintains connection with other end (here 
>> server) and retries to connect periodically so it takes time to reconnect to 
>> to server.
>>
>>
>>
>> On Mon, Apr 15, 2019 at 8:49 AM  wrote:
>>>
>>> Steps to reproduce
>>>
>>> 1. Start server
>>> 2. Send a client RPC to server
>>> 3. Restart server
>>> 4. Using the same client, send another RPC. The call will fail
>>> 5. Send another RPC, this call will success
>>>
>>> Also I found that if the server is leave stopped for a long time before 
>>> starting up again, the call in step 5 will return "channel is in state 
>>> TRANSIENT_FAILURE" as well.
>>>
>>> Example code: https://github.com/whs/grpc-repro
>>> (Install from requirements.txt then run main.py)
>>>
>>> Expected result
>>>
>>> All call should success.
>>>
>>> Tested with Python grpcio==1.19.0 server/client and with go-grpc server. I 
>>> tried setting grpc.max_connection_age_grace_ms, grpc.max_connection_age_ms, 
>>> grpc.max_connection_idle_ms, grpc.keepalive_time_ms, 
>>> grpc.keepalive_permit_without_calls but they doesn't seems to help.
>>>
>>> From my own investigation, I believe that gRPC channels doesn't drop closed 
>>> channels and tries to send calls in closed channel. There is also TCP RST 
>>> sent.
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups 
>>> "grpc.io" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to grpc-io+unsubscr...@googlegroups.com.
>>> To post to this group, send email to grpc-io@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/grpc-io.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/grpc-io/e8465f2b-96f1-4ad1-816f-ee6521595179%40googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>>
>> --
>> Thanks,
>> Sidhartha Thota.
>>
>> --
>> You received this message because you are subscribed to the Google Groups 
>> "grpc.io" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to grpc-io+unsubscr...@googlegroups.com.
>> To post to this group, send email to grpc-io@googlegroups.com.
>> Visit this group at https://groups.google.com/group/grpc-io.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/grpc-io/CALguN-Gg0iiDSEUrO-VEOuGu5Mhemc9bZVqxqi_BcY7-pc%3DQWg%40mail.gmail.com.
>> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To post to this group, send email to grpc-io@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/CAJPLQCCvchQ28wJRA0ZuaEyktYP5iQ12xW-XTM9P%2BWzQmxLxfA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [grpc-io] gRPC not dropping disconnected channel

['Lidi Zheng' via grpc.io]

gRPC Python server can be closed by "server.stop()".

Underlying, the gRPC Python server has a background polling thread. The
deallocation of the server object is not directly terminating the polling
thread since there is no way to interrupt thread in Python. So, during the
server deallocation, it will set a destruction flag, and if the polling
thread sees the flag it will terminate itself. It's possible that there are
some lag before the server completely shuts down. To eliminate this issue,
we recommend you close the server explicitly by "server.stop()" which will
block until the server is completely shut down.

Lidi Zheng

On Mon, Apr 15, 2019 at 11:09 AM Sidhartha Thota 
wrote:

> This is known issue afaik. May be your issue is related to this one:
> https://github.com/grpc/grpc/issues/9767
>
> If the connection is down the session will not be re-useful but channel is
> always useful. Here session is individual RPC/Streaming RPC call. So, when
> session is in progress and connection is down then session cannot be
> restored. Channel is different and maintains connection with other end
> (here server) and retries to connect periodically so it takes time to
> reconnect to to server.
>
>
>
> On Mon, Apr 15, 2019 at 8:49 AM  wrote:
>
>> Steps to reproduce
>>
>> 1. Start server
>> 2. Send a client RPC to server
>> 3. Restart server
>> 4. Using the same client, send another RPC. The call will fail
>> 5. Send another RPC, this call will success
>>
>> Also I found that if the server is leave stopped for a long time before
>> starting up again, the call in step 5 will return "channel is in state
>> TRANSIENT_FAILURE" as well.
>>
>> Example code: https://github.com/whs/grpc-repro
>> (Install from requirements.txt then run main.py)
>>
>> Expected result
>>
>> All call should success.
>>
>> Tested with Python grpcio==1.19.0 server/client and with go-grpc server.
>> I tried
>> setting grpc.max_connection_age_grace_ms, grpc.max_connection_age_ms, 
>> grpc.max_connection_idle_ms, grpc.keepalive_time_ms, 
>> grpc.keepalive_permit_without_calls
>> but they doesn't seems to help.
>>
>> From my own investigation, I believe that gRPC channels doesn't drop
>> closed channels and tries to send calls in closed channel. There is also
>> TCP RST sent.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "grpc.io" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to grpc-io+unsubscr...@googlegroups.com.
>> To post to this group, send email to grpc-io@googlegroups.com.
>> Visit this group at https://groups.google.com/group/grpc-io.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/grpc-io/e8465f2b-96f1-4ad1-816f-ee6521595179%40googlegroups.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
> --
> Thanks,
> Sidhartha Thota.
>
> --
> You received this message because you are subscribed to the Google Groups "
> grpc.io" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to grpc-io+unsubscr...@googlegroups.com.
> To post to this group, send email to grpc-io@googlegroups.com.
> Visit this group at https://groups.google.com/group/grpc-io.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/grpc-io/CALguN-Gg0iiDSEUrO-VEOuGu5Mhemc9bZVqxqi_BcY7-pc%3DQWg%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To post to this group, send email to grpc-io@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/CAMC1%3DjdAO13aORdiXSv2R%3DVYRmTHQHkd9Di3nXztKwYSPvQfwQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [grpc-io] gRPC not dropping disconnected channel

[Sidhartha Thota]

This is known issue afaik. May be your issue is related to this one:
https://github.com/grpc/grpc/issues/9767

If the connection is down the session will not be re-useful but channel is
always useful. Here session is individual RPC/Streaming RPC call. So, when
session is in progress and connection is down then session cannot be
restored. Channel is different and maintains connection with other end
(here server) and retries to connect periodically so it takes time to
reconnect to to server.



On Mon, Apr 15, 2019 at 8:49 AM  wrote:

> Steps to reproduce
>
> 1. Start server
> 2. Send a client RPC to server
> 3. Restart server
> 4. Using the same client, send another RPC. The call will fail
> 5. Send another RPC, this call will success
>
> Also I found that if the server is leave stopped for a long time before
> starting up again, the call in step 5 will return "channel is in state
> TRANSIENT_FAILURE" as well.
>
> Example code: https://github.com/whs/grpc-repro
> (Install from requirements.txt then run main.py)
>
> Expected result
>
> All call should success.
>
> Tested with Python grpcio==1.19.0 server/client and with go-grpc server. I
> tried
> setting grpc.max_connection_age_grace_ms, grpc.max_connection_age_ms, 
> grpc.max_connection_idle_ms, grpc.keepalive_time_ms, 
> grpc.keepalive_permit_without_calls
> but they doesn't seems to help.
>
> From my own investigation, I believe that gRPC channels doesn't drop
> closed channels and tries to send calls in closed channel. There is also
> TCP RST sent.
>
> --
> You received this message because you are subscribed to the Google Groups "
> grpc.io" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to grpc-io+unsubscr...@googlegroups.com.
> To post to this group, send email to grpc-io@googlegroups.com.
> Visit this group at https://groups.google.com/group/grpc-io.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/grpc-io/e8465f2b-96f1-4ad1-816f-ee6521595179%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Thanks,
Sidhartha Thota.

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To post to this group, send email to grpc-io@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/CALguN-Gg0iiDSEUrO-VEOuGu5Mhemc9bZVqxqi_BcY7-pc%3DQWg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[grpc-io] gRPC not dropping disconnected channel

[manatsawin]

Steps to reproduce

1. Start server
2. Send a client RPC to server
3. Restart server
4. Using the same client, send another RPC. The call will fail
5. Send another RPC, this call will success

Also I found that if the server is leave stopped for a long time before 
starting up again, the call in step 5 will return "channel is in state 
TRANSIENT_FAILURE" as well.

Example code: https://github.com/whs/grpc-repro
(Install from requirements.txt then run main.py)

Expected result

All call should success.

Tested with Python grpcio==1.19.0 server/client and with go-grpc server. I 
tried 
setting grpc.max_connection_age_grace_ms, grpc.max_connection_age_ms, 
grpc.max_connection_idle_ms, grpc.keepalive_time_ms, 
grpc.keepalive_permit_without_calls 
but they doesn't seems to help.

>From my own investigation, I believe that gRPC channels doesn't drop closed 
channels and tries to send calls in closed channel. There is also TCP RST 
sent.

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To post to this group, send email to grpc-io@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/e8465f2b-96f1-4ad1-816f-ee6521595179%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.