Hi!
You are talking about messages like below, right?
INFO [epollEventLoopGroup-2-8] 2018-05-05 16:57:45,537 Message.java:623 -
Unexpected exception during request; channel = [id: 0xa4879fdd,
L:/10.175.20.112:9042 - R:/10.175.20.73:2508]
io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed:
Connection reset by peer
at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown
Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
As far as I understand, these messages mean that some C* client (10.175.20.73
in your case) closed connection abnormally and the issue is on client side. So,
this should be investigated on client side. I'd say if your app works fine it's
up to you to investigate the issue or not ("due diligence").
At the same time there are some messages in the log related to hinted-handoff,
which could mean your cluster suffers from write timeouts (unless these
messages are caused by temporary unavailability of 10.175.20.114 because of
node reboot etc). This definitely worth investigation. It can be caused by:
non-optimal GC setting (btw messages like "ParNew GC in 309ms" may imply this,
delay of 309 ms is too long), hw performance issues, networking issues etc.
Regards,
Kyrill
________________________________
From: Xiangfei Ni <xiangfei...@cm-dt.com>
Sent: Monday, May 7, 2018 5:44:21 AM
To: user@cassandra.apache.org
Subject: 3.11.2 io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown
Source)
Hi Expert,
I have this error in the Cassandra logs,the nodes are ok and behave as normal,
INFO [epollEventLoopGroup-2-8] 2018-05-05 16:57:45,537 Message.java:623 -
Unexpected exception during request; channel = [id: 0xa4879fdd,
L:/10.175.20.112:9042 - R:/10.175.20.73:2508]
io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed:
Connection reset by peer
at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown
Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
INFO [epollEventLoopGroup-2-9] 2018-05-05 16:57:45,537 Message.java:623 -
Unexpected exception during request; channel = [id: 0x86e1a3d9,
L:/10.175.20.112:9042 - R:/10.175.20.37:44006]
io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed:
Connection reset by peer
at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown
Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
INFO [HintsDispatcher:43] 2018-05-05 16:57:49,991 HintsStore.java:126 -
Deleted hint file 5696cbef-edd1-42a6-b4f1-3c2f0881b8cb-1525510648347-1.hints
INFO [HintsDispatcher:43] 2018-05-05 16:57:49,991
HintsDispatchExecutor.java:282 - Finished hinted handoff of file
5696cbef-edd1-42a6-b4f1-3c2f0881b8cb-1525510648347-1.hints to endpoint
/10.175.20.114: 5696cbef-edd1-42a6-b4f1-3c2f0881b8cb
INFO [Service Thread] 2018-05-05 16:58:01,159 GCInspector.java:284 - ParNew GC
in 309ms. CMS Old Gen: 1258291920 -> 1258482424; Par Eden Space: 503316480 ->
0; Par Survivor Space: 700080 -> 782256
WARN [GossipTasks:1] 2018-05-05 16:58:23,273 Gossiper.java:791 - Gossip stage
has 2 pending tasks; skipping status check (no nodes will be marked down)
INFO [HintsDispatcher:44] 2018-05-05 16:58:33,913 HintsStore.java:126 -
Deleted hint file 21033fe2-dca9-42a5-8e68-496dd3517f53-1525510690633-1.hints
INFO [HintsDispatcher:44] 2018-05-05 16:58:33,913
HintsDispatchExecutor.java:282 - Finished hinted handoff of file
21033fe2-dca9-42a5-8e68-496dd3517f53-1525510690633-1.hints to endpoint
/10.175.20.113: 21033fe2-dca9-42a5-8e68-496dd3517f53
INFO [epollEventLoopGroup-2-10] 2018-05-05 16:58:41,750 Message.java:623 -
Unexpected exception during request; channel = [id: 0xc640c7a2,
L:/10.175.20.112:9042 - R:/10.175.20.37:56784]
io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed:
Connection reset by peer
at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown
Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
the limit setting in CentOS 6.8 is setting as below :
* hard memlock unlimited
* hard nofile 100000
* soft nofile 100000
* soft memlock unlimited
* hard nproc 32768
* hard as unlimited
* soft nproc 32768
* soft as unlimited
root soft nofile 32768
root hard nofile 32768
can I ignore this error or do you have any suggestions about this error?
Best Regards,
倪项菲/ David Ni
中移德电网络科技有限公司
Virtue Intelligent Network Ltd, co.
Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei
Mob: +86 13797007811|Tel: + 86 27 5024 2516
