[ https://issues.apache.org/jira/browse/CASSANDRA-11594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alex Petrov updated CASSANDRA-11594: ------------------------------------ Description: I have a 6 nodes cluster in prod in 3 racks. each node : - 4Gb commitlogs on 343 files - 275Gb data on 504 files On saturday, 1 node in each rack crash with with too many open files (seems to be the similar node in each rack). {code} lsof -n -p $PID give me 66899 out of 65826 max {code} it contains 64527 open directories (2371 uniq) a part of the list : {code} java 19076 root 2140r DIR 8,17 143360 4386718705 /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 java 19076 root 2141r DIR 8,17 143360 4386718705 /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 java 19076 root 2142r DIR 8,17 143360 4386718705 /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 java 19076 root 2143r DIR 8,17 143360 4386718705 /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 java 19076 root 2144r DIR 8,17 143360 4386718705 /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 java 19076 root 2145r DIR 8,17 143360 4386718705 /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 java 19076 root 2146r DIR 8,17 143360 4386718705 /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 java 19076 root 2147r DIR 8,17 143360 4386718705 /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 java 19076 root 2148r DIR 8,17 143360 4386718705 /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 java 19076 root 2149r DIR 8,17 143360 4386718705 /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 java 19076 root 2150r DIR 8,17 143360 4386718705 /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 java 19076 root 2151r DIR 8,17 143360 4386718705 /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 java 19076 root 2152r DIR 8,17 143360 4386718705 /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 java 19076 root 2153r DIR 8,17 143360 4386718705 /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 java 19076 root 2154r DIR 8,17 143360 4386718705 /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 java 19076 root 2155r DIR 8,17 143360 4386718705 /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 {code} The 3 others nodes crashes 4 hours later was: I have a 6 nodes cluster in prod in 3 racks. each node : - 4Gb commitlogs on 343 files - 275Gb data on 504 files On saturday, 1 node in each rack crash with with too many open files (seems to be the similar node in each rack). lsof -n -p $PID give me 66899 out of 65826 max it contains 64527 open directories (2371 uniq) a part of the list : ``` ... java 19076 root 2140r DIR 8,17 143360 4386718705 /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 java 19076 root 2141r DIR 8,17 143360 4386718705 /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 java 19076 root 2142r DIR 8,17 143360 4386718705 /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 java 19076 root 2143r DIR 8,17 143360 4386718705 /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 java 19076 root 2144r DIR 8,17 143360 4386718705 /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 java 19076 root 2145r DIR 8,17 143360 4386718705 /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 java 19076 root 2146r DIR 8,17 143360 4386718705 /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 java 19076 root 2147r DIR 8,17 143360 4386718705 /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 java 19076 root 2148r DIR 8,17 143360 4386718705 /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 java 19076 root 2149r DIR 8,17 143360 4386718705 /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 java 19076 root 2150r DIR 8,17 143360 4386718705 /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 java 19076 root 2151r DIR 8,17 143360 4386718705 /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 java 19076 root 2152r DIR 8,17 143360 4386718705 /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 java 19076 root 2153r DIR 8,17 143360 4386718705 /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 java 19076 root 2154r DIR 8,17 143360 4386718705 /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 java 19076 root 2155r DIR 8,17 143360 4386718705 /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 ... ``` The 3 others nodes crashes 4 hours later > Too many open files on directories > ---------------------------------- > > Key: CASSANDRA-11594 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11594 > Project: Cassandra > Issue Type: Bug > Components: Core > Reporter: n0rad > Priority: Critical > > I have a 6 nodes cluster in prod in 3 racks. > each node : > - 4Gb commitlogs on 343 files > - 275Gb data on 504 files > On saturday, 1 node in each rack crash with with too many open files (seems > to be the similar node in each rack). > {code} > lsof -n -p $PID give me 66899 out of 65826 max > {code} > it contains 64527 open directories (2371 uniq) > a part of the list : > {code} > java 19076 root 2140r DIR 8,17 143360 4386718705 > /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 > java 19076 root 2141r DIR 8,17 143360 4386718705 > /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 > java 19076 root 2142r DIR 8,17 143360 4386718705 > /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 > java 19076 root 2143r DIR 8,17 143360 4386718705 > /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 > java 19076 root 2144r DIR 8,17 143360 4386718705 > /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 > java 19076 root 2145r DIR 8,17 143360 4386718705 > /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 > java 19076 root 2146r DIR 8,17 143360 4386718705 > /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 > java 19076 root 2147r DIR 8,17 143360 4386718705 > /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 > java 19076 root 2148r DIR 8,17 143360 4386718705 > /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 > java 19076 root 2149r DIR 8,17 143360 4386718705 > /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 > java 19076 root 2150r DIR 8,17 143360 4386718705 > /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 > java 19076 root 2151r DIR 8,17 143360 4386718705 > /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 > java 19076 root 2152r DIR 8,17 143360 4386718705 > /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 > java 19076 root 2153r DIR 8,17 143360 4386718705 > /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 > java 19076 root 2154r DIR 8,17 143360 4386718705 > /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 > java 19076 root 2155r DIR 8,17 143360 4386718705 > /opt/stage2/pod-cassandra-aci-cassandra/rootfs/data/keyspaces/email_logs_query/emails-2d4abd00e9ea11e591199d740e07bd95 > {code} > The 3 others nodes crashes 4 hours later -- This message was sent by Atlassian JIRA (v6.3.4#6332)