Hi! You are talking about messages like below, right?
INFO [epollEventLoopGroup-2-8] 2018-05-05 16:57:45,537 Message.java:623 - Unexpected exception during request; channel = [id: 0xa4879fdd, L:/10.175.20.112:9042 - R:/10.175.20.73:2508] io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: Connection reset by peer at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final] As far as I understand, these messages mean that some C* client (10.175.20.73 in your case) closed connection abnormally and the issue is on client side. So, this should be investigated on client side. I'd say if your app works fine it's up to you to investigate the issue or not ("due diligence"). At the same time there are some messages in the log related to hinted-handoff, which could mean your cluster suffers from write timeouts (unless these messages are caused by temporary unavailability of 10.175.20.114 because of node reboot etc). This definitely worth investigation. It can be caused by: non-optimal GC setting (btw messages like "ParNew GC in 309ms" may imply this, delay of 309 ms is too long), hw performance issues, networking issues etc. Regards, Kyrill ________________________________ From: Xiangfei Ni <xiangfei...@cm-dt.com> Sent: Monday, May 7, 2018 5:44:21 AM To: user@cassandra.apache.org Subject: 3.11.2 io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source) Hi Expert, I have this error in the Cassandra logs,the nodes are ok and behave as normal, INFO [epollEventLoopGroup-2-8] 2018-05-05 16:57:45,537 Message.java:623 - Unexpected exception during request; channel = [id: 0xa4879fdd, L:/10.175.20.112:9042 - R:/10.175.20.73:2508] io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: Connection reset by peer at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final] INFO [epollEventLoopGroup-2-9] 2018-05-05 16:57:45,537 Message.java:623 - Unexpected exception during request; channel = [id: 0x86e1a3d9, L:/10.175.20.112:9042 - R:/10.175.20.37:44006] io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: Connection reset by peer at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final] INFO [HintsDispatcher:43] 2018-05-05 16:57:49,991 HintsStore.java:126 - Deleted hint file 5696cbef-edd1-42a6-b4f1-3c2f0881b8cb-1525510648347-1.hints INFO [HintsDispatcher:43] 2018-05-05 16:57:49,991 HintsDispatchExecutor.java:282 - Finished hinted handoff of file 5696cbef-edd1-42a6-b4f1-3c2f0881b8cb-1525510648347-1.hints to endpoint /10.175.20.114: 5696cbef-edd1-42a6-b4f1-3c2f0881b8cb INFO [Service Thread] 2018-05-05 16:58:01,159 GCInspector.java:284 - ParNew GC in 309ms. CMS Old Gen: 1258291920 -> 1258482424; Par Eden Space: 503316480 -> 0; Par Survivor Space: 700080 -> 782256 WARN [GossipTasks:1] 2018-05-05 16:58:23,273 Gossiper.java:791 - Gossip stage has 2 pending tasks; skipping status check (no nodes will be marked down) INFO [HintsDispatcher:44] 2018-05-05 16:58:33,913 HintsStore.java:126 - Deleted hint file 21033fe2-dca9-42a5-8e68-496dd3517f53-1525510690633-1.hints INFO [HintsDispatcher:44] 2018-05-05 16:58:33,913 HintsDispatchExecutor.java:282 - Finished hinted handoff of file 21033fe2-dca9-42a5-8e68-496dd3517f53-1525510690633-1.hints to endpoint /10.175.20.113: 21033fe2-dca9-42a5-8e68-496dd3517f53 INFO [epollEventLoopGroup-2-10] 2018-05-05 16:58:41,750 Message.java:623 - Unexpected exception during request; channel = [id: 0xc640c7a2, L:/10.175.20.112:9042 - R:/10.175.20.37:56784] io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: Connection reset by peer at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final] the limit setting in CentOS 6.8 is setting as below : * hard memlock unlimited * hard nofile 100000 * soft nofile 100000 * soft memlock unlimited * hard nproc 32768 * hard as unlimited * soft nproc 32768 * soft as unlimited root soft nofile 32768 root hard nofile 32768 can I ignore this error or do you have any suggestions about this error? Best Regards, 倪项菲/ David Ni 中移德电网络科技有限公司 Virtue Intelligent Network Ltd, co. Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei Mob: +86 13797007811|Tel: + 86 27 5024 2516