Re: [grpc-io] Re: Python Rendezvous exception when running python server, python client and C++ client on the same PC

['Lidi Zheng' via grpc.io]

Hi Alex,

>From the log, I found the error is located at client-side. The gRPC client
sends reset stream frame with error code 2 indicating internal error, and
its channel state somehow becomes SHUTDOWN.
Also, I tried your code in Linux, it works fine without any error. I had
another failed attempt to build the VS solution with 400+ errors...

Although the root cause still remained unclear, the scope is reduced to the
gRPC client behavior on Windows. To dig deeper into the bug, I think one
have to trace if the "subprocess" finishes and the life cycle of channel.

Lidi Zheng

On Mon, Apr 1, 2019 at 7:55 AM Alejandro Villagrán 
wrote:

> Hi Lidi,
>
> I've set those environment variables you mentioned and I've attached the
> log files (serverlog.txt and clientlog.txt).
> Since I reported this issue, I've upgrade the Python version of gRPC, so
> the attached log files were created with grpcio 1.19.0
>
> Do they give you any hint as to what could be wrong?
>
> Thanks,
> Alex.
>
> On Thu, 28 Mar 2019 at 17:07, Alejandro Villagrán 
> wrote:
>
>> Hi Lidi,
>>
>> In the last version of the code I attached, I don't use CherryPy anymore
>> so the issue cannot come from there. Eric said that fork handlers are not
>> registered on Windows so he ruled out that option too.
>> Unfortunately I haven't got access to a Linux machine and I'm not
>> familiar enough with Linux to be able to recreate the issue there.
>>
>> I did use turn on the debug trace a few weeks ago but I couldn't see
>> anything obvious. I'll find time to enable them again and share the logs
>> with you.
>>
>> Thanks,
>> Alex.
>>
>> On Thu, 28 Mar 2019 at 16:55, Lidi Zheng  wrote:
>>
>>> My apologies Alex. There are other stuff keep come up and consumed my
>>> time. I'm not a Windows expert, it will take me a long time to setup the
>>> compilation environment for gRPC in Windows with debugger. And I failed to
>>> find a Windows expert to debug your issue.
>>> Eric has mentioned that the breakage can be caused by either fork
>>> handlers registration or CherryPy. Do you think you can migrate the
>>> reproduce case to Linux, if the root cause is the software? It would be
>>> much easier to debug.
>>> Also, have you tried to turn on the debug trace in gRPC by setting 
>>> environmental
>>> variables
>>> 
>>> "*GRPC_VERBOSITY*" to "*DEBUG*", and "*GRPC_TRACE*" to "
>>> *api,channel,connectivity_state*"? They might produce useful
>>> information for us to identify the problem.
>>>
>>> Thanks,
>>> Lidi Zheng
>>>
>>> On Thu, Mar 28, 2019 at 3:47 AM Alejandro Villagrán 
>>> wrote:
>>>
 Hi Lidi,

 Did you manage to reproduce the issue?

 Thanks,
 Alex.

 On Tue, 19 Mar 2019 at 01:06, Lidi Zheng  wrote:

> Hi Alex,
>
> Thank you for providing the reproduce code. I will spin up a Windows
> machine to investigate this error.
> If I'm able to find something useful, I'll let you know.
>
> Lidi Zheng
>
> On Mon, Mar 18, 2019 at 4:09 AM Alejandro Villagrán <
> negral...@gmail.com> wrote:
>
>> Hi Eric/Lidi,
>>
>> Yes, I'm running on Windows. I have now removed the CherryPy code and
>> I still get the exception.
>>
>> Please follow these steps to reproduce the issue:
>> - Unzip ReproduceGRPCIssue.zip
>> - Go to the BreakGRPC folder and compile BreakGRPC.sln. Make sure
>> BreakGRPC.exe is saved in BreakGRPC/x64/Release.
>> - Go to the Services/src folder and open two command prompts there.
>> - Run "python GRPCserver.py" in one command prompt.
>> - Run "python GRPCclient.py" in the other command prompt.
>>
>> You should see the exception on the client command prompt.
>>
>> Please let me know if you are still unable to reproduce the issue
>> with this version of the code.
>>
>> Thanks,
>> Alex.
>>
>> On Mon, 4 Mar 2019 at 17:04, Eric Gribkoff 
>> wrote:
>>
>>> +Lidi Zheng , who will be available for any
>>> follow-up questions (it will be easier for him to notice your questions 
>>> if
>>> you include his email address on the "to:" line)
>>>
>>> Hi Alex,
>>>
>>> Sorry for the delay. I was not able to reproduce the problem; it
>>> looks like you are running on Windows, in which case gRPC's fork 
>>> handlers
>>> are not registered/run, so those shouldn't be the cause here . Since the
>>> reproduction example also uses CherryPy websockets, it's quite possible 
>>> the
>>> issue stems from that software rather than the gRPC stack - we'd likely
>>> need a reproduction case that only uses gRPC, without the websockets, 
>>> to be
>>> able to help debug this further.
>>>
>>> Thanks,
>>>
>>> Eric
>>>
>>> On Mon, Mar 4, 2019 at 2:36 AM Alex  wrote:
>>>
 Hi Eric,

 Just wondering if you had time to run my attached example and

[grpc-io] Re: C++: Unable to create new client stream if server restarted

[yashkt via grpc.io]

The ClientContext object needs to be alive for the entire duration of the 
RPC. The scope of your ClientContext is too limited. I suggest 
restructuring your code such that RPC invoking function is separated from 
the channel state checking. Also, gRPC will automatically try to connect 
the channel if there are pending calls, so the state checking is 
unnecessary.

On Wednesday, March 6, 2019 at 6:18:42 PM UTC-8, Sidhartha Thota wrote:
>
> HI,
>
> I am new to GRPC and using GRPC C++ for my project. I am using streaming 
> RPC where client will stream the messages it has and server listens to the 
> messages. When streaming is in progress from client side (lets say after 
> couple of messages sent from client end), I restart server (Ctrl+c and 
> start server again) then client is unable to establish new session with 
> server.
>
> If I am not wrong, writer is created per session so I am trying to create 
> new session by creating new writer instance from Stub. I see following 
> error at client
>
> E0307 02:09:43.854424957   22379 sync_posix.cc:85]   assertion 
> failed: pthread_mutex_lock(mu) == 0 [Client output attached below]
>
> GRPC version using : v1.18
>
> -
> Client code:
> ---
>
> Message getMessage(int i) {
>   Message mesg;
>   mesg.set_protocol_version(i);
>   return mesg;
> }
>
> class Client {
>  public:
>   Client(){
>   }
>
>   void sendMessages(std::shared_ptr channel,
> std::unique_ptr &stub_,
> std::unique_ptr > &writer) {
> Message message;
> const int kPoints = 10;
>
> for (int i = 0; i < kPoints; i++) {
>   Message message = getMessage(i);
>   std::cout << "Waiting to write " << std::endl;
>   if (!writer->Write(message)) {
> std::cout << "broken stream. Will create new session" << "\n";
> while (channel->GetState(true) != GRPC_CHANNEL_READY) {
>   std::cout << "Retry connection... after 1 sec" << std::endl;
>   std::this_thread::sleep_for(std::chrono::milliseconds(1000));
> }
> ClientContext context_retry;
> ReplyMessage reply_message_retry;
> writer = std::move(stub_->receive_resources(&context_retry, 
> &reply_message_retry));
> std::cout << "created new session successfully" << std::endl;
>   }
>   std::this_thread::sleep_for(std::chrono::milliseconds(1000));
> }
>   }
> };
>
> int main(int argc, char** argv) {
>   std::shared_ptr channel;
>   channel = grpc::CreateChannel("localhost:50051",
>   grpc::InsecureChannelCredentials());
>   std::unique_ptr stub_(Listener::NewStub(channel));
>   ReplyMessage reply_message;
>   ClientContext context;
>   std::unique_ptr > 
> writer(stub_->receive_resources(&context, &reply_message));
>   Client client;
>
>   std::cout << "-- SendMessages --" << std::endl;
>   client.sendMessages(channel, stub_, writer);
>   writer->WritesDone();
>   Status status = writer->Finish();
>   if (status.ok()) {
> std::cout << "Close message " << reply_message.close_message()
>   << std::endl;
>   } else {
> std::cout << "RecordRoute rpc failed." << std::endl;
>   }
>
>   return 0;
> }
>
> -
> RPC proto file:
> ---
>
> package listener;
>
> // Interface exported by the server.
> service Listener {
>   rpc receive_resources(stream Message) returns (ReplyMessage) {}
> }
>
> // Message and ReplyMessage are protos not defined here (Let me know if 
> they are applicable, I can add).
>
> --
> Server code
> -
>
> class ListenerImpl final: public Listener::Service {
>   Status receive_resources(ServerContext* context, ServerReader* 
> reader,
>   ReplyMessage* message) override {
> Message gMessage;
> unsigned int message_count = 0;
>
> while (reader->Read(&gMessage)) {
>   message_count++;
>   std::cout << "received" << std::endl;
>   std::cout << gMessage.protocol_version() << std::endl;
> }
> std::cout << message_count << std::endl;
> message->set_close_message("close");
> std::cout << "sent close message" << std::endl;
> return Status::OK;
>   }
> };
>
> void RunServer() {
>   std::string server_address("0.0.0.0:50051");
>   ListenerImpl service;
>
>   ServerBuilder builder;
>   builder.AddListeningPort(server_address, 
> grpc::InsecureServerCredentials());
>   builder.RegisterService(&service);
>   std::unique_ptr server(builder.BuildAndStart());
>   std::cout << "Server listening on " << server_address << std::endl;
>   server->Wait();
> }
>
> int main(int argc, char** argv) {
>   RunServer();
>   return 0;
> }
>
> ---
> Client side output
> ---
>
> -- SendMessages --
> Waiting to write 
> Waiting to write 
> Waiting to write 
> Waiting to write 
> Waiting to write 
> Waiting to write 
> Waiting to write 
> Waiting to w

[grpc-io] Re: How to create Asynch streaming server to serve multiple clients (chat bot)

[yashkt via grpc.io]

Hi, 

There would be a unique stream per call. The threading model you use will 
depend a lot on the application and the performance requirements. 

I think there is a slight confusion here about the sync and async API. In 
the async API, reads and writes do not block, i.e., the calls should return 
immediately. To make sure that the reads and writes actually complete, the 
async API requires completion_queue.Next() to be called.

Yes, it is possible to have concurrent reads and writes. The transport 
layer in gRPC would take care of the synchronization.

On Sunday, March 17, 2019 at 2:31:58 AM UTC-7, x.pa...@gmail.com wrote:
>
> Hello,
>
> I want to create in C++ a chat server: One serve to serve X clients at the 
> same time, when a client sends a message, the server broadcast.
> I found this example: https://github.com/grpc/grpc/pull/8934/files which 
> is close to what I want, but it contains only one client connecting (If I'm 
> not mistaken).
>
> If I want multiple clients, do I need to create one stream per client (in 
> a thread ?) or all the clients can use the same stream ?
> If multiple streams it means I will have all threads waiting to read a 
> message on their stream, and when someone receives one, the "main thread" 
> after the call to "Next" will loop on all streams to write the message.
>
> Is it possible to write a message on a stream while another thread is 
> blocking on "Read" ?
>
> If you have any advice to at least not let me go on a wrong direction, 
> shout :)
>
> Thanks.
>

-- 
You received this message because you are subscribed to the Google Groups 
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to grpc-io+unsubscr...@googlegroups.com.
To post to this group, send email to grpc-io@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/grpc-io/fa99c51d-0483-4e6a-b9d2-3ddf5aaf745e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[grpc-io] Client channel unusable after a network reset

[antonsavelyev]

grpc github issue: https://github.com/grpc/grpc/issues/18554
StackOverflow post: 
https://stackoverflow.com/questions/55460086/client-channel-unusable-after-a-network-reset

Summary: If a client channel is in a READY state and the network is 
disconnected, the channel becomes unusable and the client will not attempt 
to reconnect to the server once the network connection is re-established. *The 
channel does not transition from a READY state to a TRANSIENT_FAILURE on a 
DEADLINE_EXCEEDED error (deadline set by my client application).*

What version of gRPC and what language are you using?

1.17.2
Same issue experience in version 1.11.x
C++


What operating system (Linux, Windows,...) and version?

Client running on Ubuntu 16.04.
Server running Windows Enterprise.


What did you do?

Server and client are both started on a connected network. I can 
successfully make calls and receive responses from the server. When the 
network is turned off, the server receives a "Disconnected client - 
Endpoint read failed" error. Some other relevant fields in this debug 
message - "grpc_status":14 (UNAVAILABLE), "occured_during_write":0, 
"description":"An established connection was aborted by the software in 
your host machine".

At the time of network disconnect, the client does not print out any logs 
at all (using 
GRPC_TRACE=connectivity_state,call_error,op_failure,server_channel,client_channel,channel
 
GRPC_VERBOSITY=DEBUG).

Once the network is turned on again there are no logs experienced on 
neither the server nor the client. Attempting to make a call using the 
client (send a launch request) results in a repeated DEADLINE_EXCEEDED 
error. Turning off the network connection at this time does not result in a 
server side "Disconnected client" error.

The client context is set to use a deadline (tested with 2 and 10 seconds). 
Synchronous calls are used in this case.


*Code sniplets:*
*/rpc_service.proto*
syntax = "proto3";

import "google/rpc/status.proto";

message RpcRequest {
}

message RpcResponse {
}

service RpcService{
rpc Call(RpcRequest) returns (RpcResponse);
}


*/client.cc*
Initialization:

std::unique_ptrRpcService::Stub stub_ = RpcService::NewStub(::grpc::
CreateChannel(
server_endpoint, ::grpc::InsecureChannelCredentials()));

*Sending a rpc request:*

::grpc::ClientContext context;
context.set_deadline(
gpr_time_from_micros(call_timeout_.InMicroseconds(), GPR_TIMESPAN));
RpcRequest request;
RpcResponse response;
::grpc::Status grpc_status = stub_->Call(&context, request, &response);

*/server.cc*
grpc::ServerBuilder builder;
builder.AddListeningPort(endpoint, ::grpc::InsecureServerCredentials());
builder.RegisterService(&rpc_service);
std::unique_ptrgrpc::Server grpc_server_ = builder.BuildAndStart();

What did you expect to see?

Client should make a successful call after a network reset.


What did you see instead?

Client fails to receive a response from the server.


Anything else we should know about your project / environment?

When the network connection is re-established and the client fails to 
receive a response from the server, tcpdump captures the client sending out 
some packets.
Starting up both client and server with network ON, and then unplugging the 
network does not result in any error messages until a call is attempted. 
This is the same result as when starting both client and server with the 
network disconnected. Once a call is attempted the client will transition 
from IDLE to CONNECTING and then begin to bounce back and forth between 
CONNECTING and TRANSIENT_FAILURE states (attempting to reconnect with 
exponential back-off) until the connection is re-established.
If the client is started with the network connected, but doesn't send a 
request and the network is disconnected the server doesn't get a 
disconnected client error. Until a call is made, the client stays in a 
"IDLE".
If a client is initialized and a call is made on a disconnected network, 
then the client will enter a CONNECTING state (with exponential backoff up 
to a max of 2 min where the client will be in a TRANSIENT_FAILURE state). 
Once the network is connected, the connection will be re-established the 
next time the channel will enter a CONNECTING state and the client will 
enter the READY state. After this, each call will succeed until the network 
is reset.
Disconnecting the network after the client is in a READY state will not 
transition the client out of a READY state.
In summary: Until a call is made, the client will stay in an "IDLE" state 
no matter the network status. Once a call is made, the client will attempt 
to make a connection by entering the CONNECTING state. If no connection is 
found, it will transition bounce in-between CONNECTING and 
TRANSIENT_FAILURE states. Once a connection is found, the client will go 
into a READY state. From here, if a connection is lost, the client will not 
attempt to enter a CONNECTING state again.


*Similar issue (closed) to the one I’m having:*
https:

Re: [grpc-io] Re: Python Rendezvous exception when running python server, python client and C++ client on the same PC

[Alejandro Villagrán]

Hi Lidi,

I've set those environment variables you mentioned and I've attached the
log files (serverlog.txt and clientlog.txt).
Since I reported this issue, I've upgrade the Python version of gRPC, so
the attached log files were created with grpcio 1.19.0

Do they give you any hint as to what could be wrong?

Thanks,
Alex.

On Thu, 28 Mar 2019 at 17:07, Alejandro Villagrán 
wrote:

> Hi Lidi,
>
> In the last version of the code I attached, I don't use CherryPy anymore
> so the issue cannot come from there. Eric said that fork handlers are not
> registered on Windows so he ruled out that option too.
> Unfortunately I haven't got access to a Linux machine and I'm not familiar
> enough with Linux to be able to recreate the issue there.
>
> I did use turn on the debug trace a few weeks ago but I couldn't see
> anything obvious. I'll find time to enable them again and share the logs
> with you.
>
> Thanks,
> Alex.
>
> On Thu, 28 Mar 2019 at 16:55, Lidi Zheng  wrote:
>
>> My apologies Alex. There are other stuff keep come up and consumed my
>> time. I'm not a Windows expert, it will take me a long time to setup the
>> compilation environment for gRPC in Windows with debugger. And I failed to
>> find a Windows expert to debug your issue.
>> Eric has mentioned that the breakage can be caused by either fork
>> handlers registration or CherryPy. Do you think you can migrate the
>> reproduce case to Linux, if the root cause is the software? It would be
>> much easier to debug.
>> Also, have you tried to turn on the debug trace in gRPC by setting 
>> environmental
>> variables
>>  "
>> *GRPC_VERBOSITY*" to "*DEBUG*", and "*GRPC_TRACE*" to "
>> *api,channel,connectivity_state*"? They might produce useful information
>> for us to identify the problem.
>>
>> Thanks,
>> Lidi Zheng
>>
>> On Thu, Mar 28, 2019 at 3:47 AM Alejandro Villagrán 
>> wrote:
>>
>>> Hi Lidi,
>>>
>>> Did you manage to reproduce the issue?
>>>
>>> Thanks,
>>> Alex.
>>>
>>> On Tue, 19 Mar 2019 at 01:06, Lidi Zheng  wrote:
>>>
 Hi Alex,

 Thank you for providing the reproduce code. I will spin up a Windows
 machine to investigate this error.
 If I'm able to find something useful, I'll let you know.

 Lidi Zheng

 On Mon, Mar 18, 2019 at 4:09 AM Alejandro Villagrán <
 negral...@gmail.com> wrote:

> Hi Eric/Lidi,
>
> Yes, I'm running on Windows. I have now removed the CherryPy code and
> I still get the exception.
>
> Please follow these steps to reproduce the issue:
> - Unzip ReproduceGRPCIssue.zip
> - Go to the BreakGRPC folder and compile BreakGRPC.sln. Make sure
> BreakGRPC.exe is saved in BreakGRPC/x64/Release.
> - Go to the Services/src folder and open two command prompts there.
> - Run "python GRPCserver.py" in one command prompt.
> - Run "python GRPCclient.py" in the other command prompt.
>
> You should see the exception on the client command prompt.
>
> Please let me know if you are still unable to reproduce the issue with
> this version of the code.
>
> Thanks,
> Alex.
>
> On Mon, 4 Mar 2019 at 17:04, Eric Gribkoff 
> wrote:
>
>> +Lidi Zheng , who will be available for any
>> follow-up questions (it will be easier for him to notice your questions 
>> if
>> you include his email address on the "to:" line)
>>
>> Hi Alex,
>>
>> Sorry for the delay. I was not able to reproduce the problem; it
>> looks like you are running on Windows, in which case gRPC's fork handlers
>> are not registered/run, so those shouldn't be the cause here . Since the
>> reproduction example also uses CherryPy websockets, it's quite possible 
>> the
>> issue stems from that software rather than the gRPC stack - we'd likely
>> need a reproduction case that only uses gRPC, without the websockets, to 
>> be
>> able to help debug this further.
>>
>> Thanks,
>>
>> Eric
>>
>> On Mon, Mar 4, 2019 at 2:36 AM Alex  wrote:
>>
>>> Hi Eric,
>>>
>>> Just wondering if you had time to run my attached example and
>>> managed to reproduce the problem?
>>>
>>> Thanks,
>>> Alex.
>>>
>>> On Wednesday, February 20, 2019 at 7:04:51 PM UTC, Eric Gribkoff
>>> wrote:

 Can you post the code you're using to reproduce this error? If
 you're using subprocess.Popen (or otherwise using fork+exec) to start 
 the
 C++ grpc client process, the C++ client itself cannot be interfering 
 with
 the Python process. Something could be going wrong in the gRPC core 
 fork
 handlers, however - you can try running with the environment variable
 `GRPC_ENABLE_FORK_SUPPORT=0` to disable this feature and see if it 
 fixes
 the issue.

 Also, in your step 5 you note that the