Hi Dipan, Your node failure trace said : java.io.FileNotFoundException: /home/install/cassandra-3.11.0/data/data/hhahistory/history-065e0c90d9be11e7afbcdfeb48785ac5/mc-19095-big-Filter.db (Too many open files) You are probably crossing the max number of opened files set at OS level for Cassandra login.
On linux boxes you can get the number of file handles currently opened by Cassandra and compare it to the max number set at OS level. Can you, please, do the following as Cassandra login : $ ps -efa | grep -i Cassandra # to get the Cassandra process id; let say 1234 $ lsof -n -p 1234 # change 1234 to your current Cassandra process id $ ulimit -Hn ; ulimit -Sn What are your OS and its version ? Thanks, Adama De : Dipan Shah [mailto:dipan....@hotmail.com] Envoyé : mercredi 20 décembre 2017 07:34 À : User Objet : Re: Error during select query - Found other issues with cluster too Hello Kurt, We are using V 3.11.0 and I think this might a part of a bigger problem. I can see that nodes are failing in my cluster unexpectedly and also repair commands are failing. Repair command failure error: INFO [Native-Transport-Requests-2] 2017-12-19 17:06:02,332 Message.java:619 - Unexpected exception during request; channel = [id: 0xacc9a54a, L:/10.10.52.17:9042 ! R:/10.10.55.229:58712] io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: Connection reset by peer at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final] INFO [Native-Transport-Requests-2] 2017-12-19 17:06:11,056 Message.java:619 - Unexpected exception during request; channel = [id: 0xeebf628d, L:/10.10.52.17:9042 ! R:/10.10.55.229:58130] io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: Connection reset by peer Node failure error: ERROR [STREAM-IN-/10.10.52.22:7000] 2017-12-20 01:17:17,691 JVMStabilityInspector.java:142 - JVM state determined to be unstable. Exiting forcefully due to: java.io.FileNotFoundException: /home/install/cassandra-3.11.0/data/data/hhahistory/history-065e0c90d9be11e7afbcdfeb48785ac5/mc-19095-big-Filter.db (Too many open files) at java.io.FileOutputStream.open0(Native Method) ~[na:1.8.0_131] at java.io.FileOutputStream.open(FileOutputStream.java:270) ~[na:1.8.0_131] at java.io.FileOutputStream.<init>(FileOutputStream.java:213) ~[na:1.8.0_131] at java.io.FileOutputStream.<init>(FileOutputStream.java:101) ~[na:1.8.0_131] at org.apache.cassandra.io.sstable.format.big.BigTableWriter$IndexWriter.flushBf(BigTableWriter.java:486) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.io.sstable.format.big.BigTableWriter$IndexWriter.doPrepare(BigTableWriter.java:516) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.io.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:364) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.io.sstable.format.SSTableWriter.finish(SSTableWriter.java:264) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.finish(SimpleSSTableMultiWriter.java:59) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.io.sstable.format.RangeAwareSSTableWriter.finish(RangeAwareSSTableWriter.java:129) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.streaming.StreamReceiveTask.received(StreamReceiveTask.java:110) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:656) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:523) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:317) ~[apache-cassandra-3.11.0.jar:3.11.0] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131] Thanks, Dipan Shah ________________________________ From: kurt greaves <k...@instaclustr.com<mailto:k...@instaclustr.com>> Sent: Wednesday, December 20, 2017 2:23 AM To: User Subject: Re: Error during select query Can you send through the full stack trace as reported in the Cassandra logs? Also, what version are you running? On 19 Dec. 2017 9:23 pm, "Dipan Shah" <dipan....@hotmail.com<mailto:dipan....@hotmail.com>> wrote: Hello, I am getting an error message when I'm running a select query from 1 particular node. The error is "ServerError: java.lang.IllegalStateException: Unable to compute ceiling for max when histogram overflowed". Has anyone faced this error earlier? I tried to search for this but did not get anything that matches my scenario. Please note, I do not get this error when I run the same query from any other node. And I'm connecting to the node using cqlsh. Thanks, Dipan Shah _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.
