Re: Open File Handles for Deleted sstables
Restarting may be a temporary workaround but cant be a permanent solution. After some days, the problem will come back again. ThanksAnuj Sent from Yahoo Mail on Android On Thu, 29 Sep, 2016 at 12:54 AM, sai krishnam raju potturi wrote: restarting the cassandra service helped get rid of those files in our situation. thanksSai On Wed, Sep 28, 2016 at 3:15 PM, Anuj Wadehra wrote: Hi, We are facing an issue where Cassandra has open file handles for deleted sstable files. These open file handles keep on increasing with time and eventually lead to disk crisis. This is visible via lsof command. There are no Exceptions in logs.We are suspecting a race condition where compactions/repairs and reads are done on same sstable. I have gone through few JIRAs but somehow not able to coorelate the issue with those tickets. We are using 2.0.14. OS is Red Hat Linux. Any suggestions? ThanksAnuj
Re: Open File Handles for Deleted sstables
There have been a history of leaks where repairs are multiple repairs were run on the same node at the same time ( e.g: https://issues.apache.org/jira/browse/CASSANDRA-11215 ) You’re running a very old version of Cassandra. If you’re able to upgrade to newest 2.1 or 2.2, it’s likely that at least SOME similar bugs are addressed. From: Anuj Wadehra Reply-To: "user@cassandra.apache.org" Date: Wednesday, September 28, 2016 at 12:15 PM To: User Subject: Open File Handles for Deleted sstables Hi, We are facing an issue where Cassandra has open file handles for deleted sstable files. These open file handles keep on increasing with time and eventually lead to disk crisis. This is visible via lsof command. There are no Exceptions in logs.We are suspecting a race condition where compactions/repairs and reads are done on same sstable. I have gone through few JIRAs but somehow not able to coorelate the issue with those tickets. We are using 2.0.14. OS is Red Hat Linux. Any suggestions? Thanks Anuj CONFIDENTIALITY NOTE: This e-mail and any attachments are confidential and may be legally privileged. If you are not the intended recipient, do not disclose, copy, distribute, or use this email or any attachments. If you have received this in error please let the sender know and then delete the email and all attachments. smime.p7s Description: S/MIME cryptographic signature
Re: Open File Handles for Deleted sstables
restarting the cassandra service helped get rid of those files in our situation. thanks Sai On Wed, Sep 28, 2016 at 3:15 PM, Anuj Wadehra wrote: > Hi, > > We are facing an issue where Cassandra has open file handles for deleted > sstable files. These open file handles keep on increasing with time and > eventually lead to disk crisis. This is visible via lsof command. > > There are no Exceptions in logs.We are suspecting a race condition where > compactions/repairs and reads are done on same sstable. I have gone through > few JIRAs but somehow not able to coorelate the issue with those tickets. > > We are using 2.0.14. OS is Red Hat Linux. > > Any suggestions? > > Thanks > Anuj > > >
Open File Handles for Deleted sstables
Hi, We are facing an issue where Cassandra has open file handles for deleted sstable files. These open file handles keep on increasing with time and eventually lead to disk crisis. This is visible via lsof command. There are no Exceptions in logs.We are suspecting a race condition where compactions/repairs and reads are done on same sstable. I have gone through few JIRAs but somehow not able to coorelate the issue with those tickets. We are using 2.0.14. OS is Red Hat Linux. Any suggestions? ThanksAnuj
