[jira] [Comment Edited] (CASSANDRA-4571) Strange permament socket descriptors increasing leads to "Too many open files"
[ https://issues.apache.org/jira/browse/CASSANDRA-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13446020#comment-13446020 ] Steven Willcox edited comment on CASSANDRA-4571 at 9/1/12 3:21 AM: --- We can reliably reproduce this issue in our test environment every day. Start the servers up in the morning and by the end of the workday the number of open file descriptors reaches from 40-60K and the nodes stop responding. We have turned row caching off and it still has this problem. You can contact me if you think remotely debugging this issue will help in determining what is causing this. We have reproduced this using just one node. was (Author: swillcox): We can reliably reproduce this issue in our test environment every day. Start the servers up in the morning and by the end of the workday the number of open file descriptors reaches from 40-60K and the nodes stop responding. We have turned row caching off and it still has this problem. You can contact me if you think remotely debugging this issue will help in determining what is causing this. > Strange permament socket descriptors increasing leads to "Too many open files" > -- > > Key: CASSANDRA-4571 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4571 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.1.1, 1.1.2, 1.1.3 > Environment: CentOS 5.8 Linux 2.6.18-308.13.1.el5 #1 SMP Tue Aug 21 > 17:10:18 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux. > java version "1.6.0_33" > Java(TM) SE Runtime Environment (build 1.6.0_33-b03) > Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode) >Reporter: Serg Shnerson >Priority: Critical > > On the two-node cluster there was found strange socket descriptors > increasing. lsof -n | grep java shows many rows like" > java 8380 cassandra 113r unix 0x8101a374a080 > 938348482 socket > java 8380 cassandra 114r unix 0x8101a374a080 > 938348482 socket > java 8380 cassandra 115r unix 0x8101a374a080 > 938348482 socket > java 8380 cassandra 116r unix 0x8101a374a080 > 938348482 socket > java 8380 cassandra 117r unix 0x8101a374a080 > 938348482 socket > java 8380 cassandra 118r unix 0x8101a374a080 > 938348482 socket > java 8380 cassandra 119r unix 0x8101a374a080 > 938348482 socket > java 8380 cassandra 120r unix 0x8101a374a080 > 938348482 socket > " And number of this rows constantly increasing. After about 24 hours this > situation leads to error. > We use PHPCassa client. Load is not so high (aroud ~50kb/s on write). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4571) Strange permament socket descriptors increasing leads to "Too many open files"
[ https://issues.apache.org/jira/browse/CASSANDRA-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13444558#comment-13444558 ] Serg Shnerson edited comment on CASSANDRA-4571 at 8/30/12 11:08 AM: bq.Are you sure you can't reproduce on a single-node cluster? My mistake. I've checked it again. Bug also was reproduced with one-node cluster. was (Author: sergshne): bq.Are you sure you can't reproduce on a single-node cluster? My mistake. Bug also was reproduced with one-node cluster. > Strange permament socket descriptors increasing leads to "Too many open files" > -- > > Key: CASSANDRA-4571 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4571 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.1.1, 1.1.2, 1.1.3 > Environment: CentOS 5.8 Linux 2.6.18-308.13.1.el5 #1 SMP Tue Aug 21 > 17:10:18 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux. > java version "1.6.0_33" > Java(TM) SE Runtime Environment (build 1.6.0_33-b03) > Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode) >Reporter: Serg Shnerson >Priority: Critical > > On the two-node cluster there was found strange socket descriptors > increasing. lsof -n | grep java shows many rows like" > java 8380 cassandra 113r unix 0x8101a374a080 > 938348482 socket > java 8380 cassandra 114r unix 0x8101a374a080 > 938348482 socket > java 8380 cassandra 115r unix 0x8101a374a080 > 938348482 socket > java 8380 cassandra 116r unix 0x8101a374a080 > 938348482 socket > java 8380 cassandra 117r unix 0x8101a374a080 > 938348482 socket > java 8380 cassandra 118r unix 0x8101a374a080 > 938348482 socket > java 8380 cassandra 119r unix 0x8101a374a080 > 938348482 socket > java 8380 cassandra 120r unix 0x8101a374a080 > 938348482 socket > " And number of this rows constantly increasing. After about 24 hours this > situation leads to error. > We use PHPCassa client. Load is not so high (aroud ~50kb/s on write). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4571) Strange permament socket descriptors increasing leads to "Too many open files"
[ https://issues.apache.org/jira/browse/CASSANDRA-4571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13444161#comment-13444161 ] Steven Willcox edited comment on CASSANDRA-4571 at 8/30/12 3:22 AM: We are also seeing this bug and all nodes eventually run out of file descriptors and crash. It is a blocker for us. was (Author: swillcox): We are also seeing this bug and all nodes eventually run out of file descriptors and crash. > Strange permament socket descriptors increasing leads to "Too many open files" > -- > > Key: CASSANDRA-4571 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4571 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 1.1.1, 1.1.2, 1.1.3 > Environment: CentOS 5.8 Linux 2.6.18-308.13.1.el5 #1 SMP Tue Aug 21 > 17:10:18 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux. > java version "1.6.0_33" > Java(TM) SE Runtime Environment (build 1.6.0_33-b03) > Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode) >Reporter: Serg Shnerson >Priority: Critical > > On the two-node cluster there was found strange socket descriptors > increasing. lsof -n | grep java shows many rows like" > java 8380 cassandra 113r unix 0x8101a374a080 > 938348482 socket > java 8380 cassandra 114r unix 0x8101a374a080 > 938348482 socket > java 8380 cassandra 115r unix 0x8101a374a080 > 938348482 socket > java 8380 cassandra 116r unix 0x8101a374a080 > 938348482 socket > java 8380 cassandra 117r unix 0x8101a374a080 > 938348482 socket > java 8380 cassandra 118r unix 0x8101a374a080 > 938348482 socket > java 8380 cassandra 119r unix 0x8101a374a080 > 938348482 socket > java 8380 cassandra 120r unix 0x8101a374a080 > 938348482 socket > " And number of this rows constantly increasing. After about 24 hours this > situation leads to error. > We use PHPCassa client. Load is not so high (aroud ~50kb/s on write). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
