guocanwen commented on issue #962: brpc block timeout URL: https://github.com/apache/incubator-brpc/issues/962#issuecomment-551366875 谢谢回复，具体的一些情况如下： 具体的操作是： 2019-11-07 20:17:56 远端服务10.12.145.207:12306 core掉（grpc server） 开始出现大量的超时 2019-11-07 20:18:05 名字服务（自己实现的）发现10.12.145.207:12306心跳超时，本地关掉连接（代码实现的时候，stub和channel 都是调用析构函数进行对象回收，没有close之类的接口）。 错误一直持续 在server端，请求是正常的（server端有其他服务也在使用），而且出问题的机器也不是全部失败，就是失败率很高，20%左右 其中一台机器（一半的机器有问题）出问题时的日志： [2019-11-07 20:17:00.865998][17804][connection_pool.cc:54] Sync connection pool size: 1 [2019-11-07 20:17:01.633376][ 1152][forward_table_data_manager.h:389] Current consume msg_count=5250000 [2019-11-07 20:17:14.881232][20310][connection_pool.cc:54] Sync connection pool size: 1 [2019-11-07 20:17:19.816668][ 1234][forward_table_data_manager.h:229] Message is HEART_BEAT, timestamp=1573129039 [2019-11-07 20:17:24.024580][ 1260][http2_rpc_protocol.cpp:206] Unknown setting, id=65027 value=1 [2019-11-07 20:17:25.304358][ 1152][forward_table_data_manager.h:389] Current consume msg_count=5260000 [2019-11-07 20:17:26.629755][ 1288][http2_rpc_protocol.cpp:206] Unknown setting, id=65027 value=1 [2019-11-07 20:17:28.404240][22809][connection_pool.cc:54] Sync connection pool size: 1 [2019-11-07 20:17:41.649389][25284][connection_pool.cc:54] Sync connection pool size: 1 [2019-11-07 20:17:49.494298][ 1152][forward_table_data_manager.h:389] Current consume msg_count=5270000 [2019-11-07 20:17:55.607590][27801][connection_pool.cc:54] Sync connection pool size: 1 [2019-11-07 20:17:57.247980][ 876][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.145.207:12306 [2019-11-07 20:17:57.468590][ 401][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.145.207:12306 [2019-11-07 20:17:58.866565][ 1273][creative_service.cc:190] wait future in FetchCreative timeout [2019-11-07 20:17:58.851737][ 957][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.170.229:12306 [2019-11-07 20:17:58.851717][ 524][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.144.29:12306 [2019-11-07 20:17:58.618955][ 458][remote_call.cc:107] Call failed. error code=[E111]Fail to connect Socket{id=8590041217 addr=10.12.145.207:12306} (0x0x7fabd6180840): Connection refused [R1][E112]Not connected to 10.12.145.207:12306 yet, server_id=8589934890 [R2][E112]Not connected to 10.12.145.207:12306 yet, server_id=8589934890 [R3][E112]Not connected to 10.12.145.207:12306 yet, server_id=8589934890 [2019-11-07 20:17:58.851699][ 453][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.144.183:12306 [2019-11-07 20:17:58.719017][ 1260][socket.cpp:2260] Checking Socket{id=8589934890 addr=10.12.145.207:12306} (0x7fb17e6f5fc0) [2019-11-07 20:17:58.501103][ 1277][input_messenger.cpp:212] Fail to read from Socket{id=8589934884 fd=351 addr=10.12.145.207:12306:42632} (0x7fb17e6f53c0): Connection reset by peer [104] [2019-11-07 20:17:58.982826][ 878][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.145.113:12306 [2019-11-07 20:17:58.851709][ 914][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.171.13:12306 [2019-11-07 20:17:58.984495][ 953][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.203.127:12306 [2019-11-07 20:17:58.859853][ 710][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.171.219:12306 [2019-11-07 20:17:58.984462][ 1110][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.247.59:12306 [2019-11-07 20:17:58.982813][ 623][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.145.97:12306 [2019-11-07 20:17:58.325278][ 1023][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.145.207:12306 [2019-11-07 20:17:58.993119][ 534][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.220.121:12306 [2019-11-07 20:17:58.853020][ 647][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.171.1:12306 [2019-11-07 20:17:58.860050][ 715][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.170.105:12306 [2019-11-07 20:17:58.011910][ 330][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.145.207:12306 [2019-11-07 20:17:58.851717][ 723][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.247.49:12306 [2019-11-07 20:17:58.984458][ 1024][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.224.99:12306 [2019-11-07 20:17:59.258399][ 1025][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.144.29:12306 [2019-11-07 20:17:59.889668][ 1039][remote_call.cc:107] Call failed. error code=[E112]Not connected to 10.12.145.207:12306 yet, server_id=8589934890 [R1][E112]Not connected to 10.12.145.207:12306 yet, server_id=8589934890 [R2][E112]Not connected to 10.12.145.207:12306 yet, server_id=8589934890 [R3][E112]Not connected to 10.12.145.207:12306 yet, server_id=8589934890 [2019-11-07 20:17:59.632350][ 978][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.234.55:12306 [2019-11-07 20:17:59.632324][ 849][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.170.145:12306 [2019-11-07 20:17:59.254508][ 328][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.227.123:12306 [2019-11-07 20:17:59.258391][ 382][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.170.105:12306 [2019-11-07 20:17:59.258401][ 928][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.170.229:12306 [2019-11-07 20:17:59.885679][ 658][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.170.103:12306 [2019-11-07 20:17:59.885681][ 1031][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.246.125:12306 [2019-11-07 20:17:59.632336][ 959][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.247.131:12306 [2019-11-07 20:17:59.254518][ 569][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.168.119:12306 [2019-11-07 20:17:59.258436][ 868][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.171.13:12306 [2019-11-07 20:17:59.259838][ 1047][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.171.219:12306 [2019-11-07 20:17:59.258409][ 753][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.247.49:12306 [2019-11-07 20:17:59.632362][ 654][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.199.108:12306 [2019-11-07 20:17:59.258385][ 1018][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.144.183:12306 [2019-11-07 20:17:59.635424][ 1088][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.225.185:12306 [2019-11-07 20:17:59.933835][ 994][remote_call.cc:107] Call failed. error code=[E112]Not connected to 10.12.145.207:12306 yet, server_id=8589934890 [R1][E112]Not connected to 10.12.145.207:12306 yet, server_id=8589934890 [R2][E112]Not connected to 10.12.145.207:12306 yet, server_id=8589934890 [R3][E112]Not connected to 10.12.145.207:12306 yet, server_id=8589934890 [2019-11-07 20:17:59.632350][ 571][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.233.3:12306 [2019-11-07 20:17:59.885679][ 973][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.171.177:12306 [2019-11-07 20:17:59.885661][ 549][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.233.165:12306 [2019-11-07 20:17:59.632359][ 611][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.144.189:12306 [2019-11-07 20:17:59.012722][ 738][remote_call.cc:107] Call failed. error code=[E112]Not connected to 10.12.145.207:12306 yet, server_id=8589934890 [R1][E112]Not connected to 10.12.145.207:12306 yet, server_id=8589934890 [R2][E112]Not connected to 10.12.145.207:12306 yet, server_id=8589934890 [R3][E112]Not connected to 10.12.145.207:12306 yet, server_id=8589934890 [2019-11-07 20:17:59.258451][ 1114][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.171.1:12306 [2019-11-07 20:17:59.254487][ 843][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.147.131:12306 [2019-11-07 20:17:59.254508][ 701][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.168.85:12306 ... [2019-11-07 20:26:36.016777][ 659][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.160.237:12306 [2019-11-07 20:26:36.016753][ 1057][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.171.217:12306 [2019-11-07 20:26:36.002652][ 761][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.161.223:12306 [2019-11-07 20:26:36.284142][ 949][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.224.96:12306 [2019-11-07 20:26:36.002652][ 824][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.161.235:12306 [2019-11-07 20:26:36.284214][ 984][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.169.173:12306 [2019-11-07 20:26:36.019434][ 668][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.180.37:12306 [2019-11-07 20:26:36.005793][ 605][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.144.21:12306 [2019-11-07 20:26:36.284135][ 733][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.199.73:12306 [2019-11-07 20:26:36.016758][ 578][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.199.34:12306 [2019-11-07 20:26:35.855804][ 908][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.171.131:12306 [2019-11-07 20:26:36.285341][ 988][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.144.95:12306 [2019-11-07 20:26:36.024909][ 580][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.246.147:12306 [2019-11-07 20:26:36.016813][ 461][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.160.89:12306 [2019-11-07 20:26:36.285405][ 690][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.246.195:12306 [2019-11-07 20:26:36.284118][ 757][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.145.243:12306 [2019-11-07 20:26:36.549294][ 1015][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.170.103:12306 [2019-11-07 20:26:36.549294][ 870][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.171.13:12306 [2019-11-07 20:26:36.549311][ 398][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.247.49:12306 [2019-11-07 20:26:36.858644][ 1042][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.145.205:12306 [2019-11-07 20:26:37.267210][ 322][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.226.47:12306 [2019-11-07 20:26:36.861842][ 758][remote_call.cc:107] Call failed. error code=[E1008]Reached timeout=120ms @10.12.169.91:12306 ------------------ 原始邮件 ------------------ 发件人: "Ge Jun"<notificati...@github.com>; 发送时间: 2019年11月8日(星期五) 上午10:59 收件人: "apache/incubator-brpc"<incubator-b...@noreply.github.com>; 抄送: "郭灿文"<326756...@qq.com>;"Author"<aut...@noreply.github.com>; 主题: Re: [apache/incubator-brpc] brpc block timeout (#962) 有具体的错误日志么？ — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For additional commands, e-mail: dev-h...@brpc.apache.org
