Re: Hints files are not getting truncated
Hi, First of all, I don’t know why they get delivered so slowly. However, if your gc grace seconds is the default 10 days then those hints from May are not needed and could/should be truncated. If the hint delivery is causing problems, then one option is that you could just disable it and rely on periodic repair doing its job and getting the data synchronized. Now my question to you: Do you have repairs running periodically across the cluster and actually succeeding if the connection is flaky? Hannu On 27 June 2017 at 19:17:26, Meg Mara (mm...@digitalriver.com) wrote: Hello, I am facing an issue with Hinted Handoff files in Cassandra v3.0.10. A DC1 node is storing large number of hints for DC2 nodes (we are facing connection timeout issues). The problem is that the hint files which are created on DC1 are not getting deleted after the 3 hour window. Hints are now being stored as flat files in the Cassandra home directory and I can see that old hints are being deleted but at a very slow pace. It still contains hints from May. max_hint_window_in_ms: 1080 max_hints_delivery_threads: 2 Why do you suppose this is happening? Any suggestions or recommendations would be much appreciated. Thanks for your time. Meg Mara
Re: Hints files are not getting truncated
Hi Meg, max_hint_window_in_ms =3 hrs means that if a node is down/unresponsive for more than 3 hrs, hints would not be stored for it any further until it becomes responsive again. It should not mean that already stored hints would be truncated after 3 hours. Regarding connection timeouts between DCs, please check your firewall settings and tcp settings on node. Firewall between the DC must not kill an idle connection which is still considered to be usable by Cassandra. Please see http://docs.datastax.com/en/archived/cassandra/2.0/cassandra/troubleshooting/trblshootIdleFirewall.html . In multi dc setup, documentation recommends to increase number of threads. You can try increasing it and check whether it improves the situation. ThanksAnuj On Tue, Jun 27, 2017 at 9:47 PM, Meg Mara wrote: Hello, I am facing an issue with Hinted Handoff files in Cassandra v3.0.10. A DC1 node is storing large number of hints for DC2 nodes (we are facing connection timeout issues). The problem is that the hint files which are created on DC1 are not getting deleted after the 3 hour window. Hints are now being stored as flat files in the Cassandra home directory and I can see that old hints are being deleted but at a very slow pace. It still contains hints from May. max_hint_window_in_ms: 1080 max_hints_delivery_threads: 2 Why do you suppose this is happening? Any suggestions or recommendations would be much appreciated. Thanks for your time. Meg Mara
RE: Hints files are not getting truncated
Hannu, Thanks for your response. Yes our gc grace period is at default setting, so you are right, I do not need hints which are a month old. Especially since I have scheduled repairs running on the cluster almost every day and cross DC repair running every week. The network connection is slow but we have never faced any major issues apart from some dropped mutation messages from time to time. I will consider your advice about turning off hints. Thank you, Meg Mara From: Hannu Kröger [mailto:hkro...@gmail.com] Sent: Tuesday, June 27, 2017 11:37 AM To: Meg Mara ; user@cassandra.apache.org Subject: Re: Hints files are not getting truncated Hi, First of all, I don’t know why they get delivered so slowly. However, if your gc grace seconds is the default 10 days then those hints from May are not needed and could/should be truncated. If the hint delivery is causing problems, then one option is that you could just disable it and rely on periodic repair doing its job and getting the data synchronized. Now my question to you: Do you have repairs running periodically across the cluster and actually succeeding if the connection is flaky? Hannu On 27 June 2017 at 19:17:26, Meg Mara (mm...@digitalriver.com<mailto:mm...@digitalriver.com>) wrote: Hello, I am facing an issue with Hinted Handoff files in Cassandra v3.0.10. A DC1 node is storing large number of hints for DC2 nodes (we are facing connection timeout issues). The problem is that the hint files which are created on DC1 are not getting deleted after the 3 hour window. Hints are now being stored as flat files in the Cassandra home directory and I can see that old hints are being deleted but at a very slow pace. It still contains hints from May. max_hint_window_in_ms: 1080 max_hints_delivery_threads: 2 Why do you suppose this is happening? Any suggestions or recommendations would be much appreciated. Thanks for your time. Meg Mara
RE: Hints files are not getting truncated
Anuj, Thank you for your explanation. I had misread the documentation and was under the impression all these days that Hints_window refers to ‘time until deletion’. Also, thanks for the information about TCP settings, I will have to recheck the existing configuration. I will also test the performance improvement of increasing the threads for the hint delivery. Thank you, Meg Mara From: Anuj Wadehra [mailto:anujw_2...@yahoo.co.in.INVALID] Sent: Tuesday, June 27, 2017 11:56 AM To: user@cassandra.apache.org Subject: Re: Hints files are not getting truncated Hi Meg, max_hint_window_in_ms =3 hrs means that if a node is down/unresponsive for more than 3 hrs, hints would not be stored for it any further until it becomes responsive again. It should not mean that already stored hints would be truncated after 3 hours. Regarding connection timeouts between DCs, please check your firewall settings and tcp settings on node. Firewall between the DC must not kill an idle connection which is still considered to be usable by Cassandra. Please see http://docs.datastax.com/en/archived/cassandra/2.0/cassandra/troubleshooting/trblshootIdleFirewall.html . In multi dc setup, documentation recommends to increase number of threads. You can try increasing it and check whether it improves the situation. Thanks Anuj On Tue, Jun 27, 2017 at 9:47 PM, Meg Mara mailto:mm...@digitalriver.com>> wrote: Hello, I am facing an issue with Hinted Handoff files in Cassandra v3.0.10. A DC1 node is storing large number of hints for DC2 nodes (we are facing connection timeout issues). The problem is that the hint files which are created on DC1 are not getting deleted after the 3 hour window. Hints are now being stored as flat files in the Cassandra home directory and I can see that old hints are being deleted but at a very slow pace. It still contains hints from May. max_hint_window_in_ms: 1080 max_hints_delivery_threads: 2 Why do you suppose this is happening? Any suggestions or recommendations would be much appreciated. Thanks for your time. Meg Mara
