Re: one node down and cluster works better
Thanks Mehmet and Erick, I don't have any monitoring other than nodetool but I manage to see some disk errors cause exceptions. I changed faulty disk and performance ok now. Regards, Osman On Sun, 5 Apr 2020 at 03:17, Erick Ramirez wrote: > > With only 2 replicas per DC, it means you're likely writing with a > consistency level of either ONE or LOCAL_ONE. Everytime you hit the > problematic node, the write performance drops. All other configurations being > equal, this indicates an issue with the commitlog disk on the node. > > Get your sysadmin to check for any issues with the disk, perhaps there's a > problem with the hardware and failure is impending. As a side note, it's > probably a good time to ensure that cluster repairs are in order in case the > node fails. Cheers! > > GOT QUESTIONS? Apache Cassandra experts from the community and DataStax have > answers! Share your expertise on https://community.datastax.com/. > - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Re: one node down and cluster works better
With only 2 replicas per DC, it means you're likely writing with a consistency level of either ONE or LOCAL_ONE. Everytime you hit the problematic node, the write performance drops. All other configurations being equal, this indicates an issue with the commitlog disk on the node. Get your sysadmin to check for any issues with the disk, perhaps there's a problem with the hardware and failure is impending. As a side note, it's probably a good time to ensure that cluster repairs are in order in case the node fails. Cheers! GOT QUESTIONS? Apache Cassandra experts from the community and DataStax have answers! Share your expertise on https://community.datastax.com/.
Re: one node down and cluster works better
Hi Osman,Do you use any monitoring solution such as prometheus on your cluster? If yes, you should install and use cassandra exporter from the link below and examine some detailed metrics.https://github.com/criteo/cassandra_exporter ndroid’de Yahoo Postadan gönderildi 15:53’’4e’ 4 Nis 2020 Cmt tarihinde, Osman Yozgatlıoğlu şunu yazdı: Hello, I manage one cluster with 2 dc, 7 nodes each and replication factor is 2:2 My insertion performance dropped somehow. I restarted nodes one by one and found one node degrades performance. Verified this node after problem occurs a couple of times. How can I continue to investigate? Regards, Osman - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
