Cassandra Daemon not coming up
I am using Cassandra 3.0.9 version on a 12 node cluster. I have multiple node down after a restart. The cassandra VM is not coming up with an asset error as below. On running in debug mode it is failing while doing operation on " resource_role_permissons_index" in system_auth keyspace. Please let me know how to bring the cassandra running from this state. Logs from system.log INFO [main] 2018-02-27 15:43:24,005 ColumnFamilyStore.java:389 - Initializing system_schema.columns INFO [main] 2018-02-27 15:43:24,012 ColumnFamilyStore.java:389 - Initializing system_schema.triggers INFO [main] 2018-02-27 15:43:24,019 ColumnFamilyStore.java:389 - Initializing system_schema.dropped_columns INFO [main] 2018-02-27 15:43:24,029 ColumnFamilyStore.java:389 - Initializing system_schema.views INFO [main] 2018-02-27 15:43:24,038 ColumnFamilyStore.java:389 - Initializing system_schema.types INFO [main] 2018-02-27 15:43:24,049 ColumnFamilyStore.java:389 - Initializing system_schema.functions INFO [main] 2018-02-27 15:43:24,061 ColumnFamilyStore.java:389 - Initializing system_schema.aggregates INFO [main] 2018-02-27 15:43:24,072 ColumnFamilyStore.java:389 - Initializing system_schema.indexes ERROR [main] 2018-02-27 15:43:24,127 CassandraDaemon.java:709 - Exception encountered during startup java.lang.AssertionError: null at org.apache.cassandra.db.marshal.CompositeType.getInstance(CompositeType.java:103) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.config.CFMetaData.rebuild(CFMetaData.java:311) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.config.CFMetaData.(CFMetaData.java:288) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.config.CFMetaData.create(CFMetaData.java:366) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:954) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:928) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:891) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:868) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:856) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:136) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:126) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:239) [apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:568) [apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:696) [apache-cassandra-3.0.9.jar:3.0.9] -- Regards, Mahesh Rajamani
RE: Cassandra Daemon not coming up
Can you check if you’ve enough disk space available ? ~Asad From: mahesh rajamani [mailto:rajamani.mah...@gmail.com] Sent: Tuesday, February 27, 2018 10:11 AM To: user@cassandra.apache.org Subject: Cassandra Daemon not coming up I am using Cassandra 3.0.9 version on a 12 node cluster. I have multiple node down after a restart. The cassandra VM is not coming up with an asset error as below. On running in debug mode it is failing while doing operation on " resource_role_permissons_index" in system_auth keyspace. Please let me know how to bring the cassandra running from this state. Logs from system.log INFO [main] 2018-02-27 15:43:24,005 ColumnFamilyStore.java:389 - Initializing system_schema.columns INFO [main] 2018-02-27 15:43:24,012 ColumnFamilyStore.java:389 - Initializing system_schema.triggers INFO [main] 2018-02-27 15:43:24,019 ColumnFamilyStore.java:389 - Initializing system_schema.dropped_columns INFO [main] 2018-02-27 15:43:24,029 ColumnFamilyStore.java:389 - Initializing system_schema.views INFO [main] 2018-02-27 15:43:24,038 ColumnFamilyStore.java:389 - Initializing system_schema.types INFO [main] 2018-02-27 15:43:24,049 ColumnFamilyStore.java:389 - Initializing system_schema.functions INFO [main] 2018-02-27 15:43:24,061 ColumnFamilyStore.java:389 - Initializing system_schema.aggregates INFO [main] 2018-02-27 15:43:24,072 ColumnFamilyStore.java:389 - Initializing system_schema.indexes ERROR [main] 2018-02-27 15:43:24,127 CassandraDaemon.java:709 - Exception encountered during startup java.lang.AssertionError: null at org.apache.cassandra.db.marshal.CompositeType.getInstance(CompositeType.java:103) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.config.CFMetaData.rebuild(CFMetaData.java:311) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.config.CFMetaData.(CFMetaData.java:288) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.config.CFMetaData.create(CFMetaData.java:366) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:954) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:928) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:891) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:868) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:856) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:136) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:126) ~[apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:239) [apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:568) [apache-cassandra-3.0.9.jar:3.0.9] at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:696) [apache-cassandra-3.0.9.jar:3.0.9] -- Regards, Mahesh Rajamani
RE: Cassandra Daemon not coming up
Were there any changes to the system such as permissions, etc. Did you add users / change auth scheme? On Feb 27, 2018, 10:27 AM -0600, ZAIDI, ASAD A , wrote: > Can you check if you’ve enough disk space available ? > ~Asad > > From: mahesh rajamani [mailto:rajamani.mah...@gmail.com] > Sent: Tuesday, February 27, 2018 10:11 AM > To: user@cassandra.apache.org > Subject: Cassandra Daemon not coming up > > I am using Cassandra 3.0.9 version on a 12 node cluster. I have multiple node > down after a restart. The cassandra VM is not coming up with an asset error > as below. On running in debug mode it is failing while doing operation on " > resource_role_permissons_index" in system_auth keyspace. Please let me know > how to bring the cassandra running from this state. > > Logs from system.log > > INFO [main] 2018-02-27 15:43:24,005 ColumnFamilyStore.java:389 - > Initializing system_schema.columns > INFO [main] 2018-02-27 15:43:24,012 ColumnFamilyStore.java:389 - > Initializing system_schema.triggers > INFO [main] 2018-02-27 15:43:24,019 ColumnFamilyStore.java:389 - > Initializing system_schema.dropped_columns > INFO [main] 2018-02-27 15:43:24,029 ColumnFamilyStore.java:389 - > Initializing system_schema.views > INFO [main] 2018-02-27 15:43:24,038 ColumnFamilyStore.java:389 - > Initializing system_schema.types > INFO [main] 2018-02-27 15:43:24,049 ColumnFamilyStore.java:389 - > Initializing system_schema.functions > INFO [main] 2018-02-27 15:43:24,061 ColumnFamilyStore.java:389 - > Initializing system_schema.aggregates > INFO [main] 2018-02-27 15:43:24,072 ColumnFamilyStore.java:389 - > Initializing system_schema.indexes > ERROR [main] 2018-02-27 15:43:24,127 CassandraDaemon.java:709 - Exception > encountered during startup > java.lang.AssertionError: null > at > org.apache.cassandra.db.marshal.CompositeType.getInstance(CompositeType.java:103) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.config.CFMetaData.rebuild(CFMetaData.java:311) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at org.apache.cassandra.config.CFMetaData.(CFMetaData.java:288) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at org.apache.cassandra.config.CFMetaData.create(CFMetaData.java:366) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:954) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:928) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:891) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:868) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:856) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:136) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:126) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:239) > [apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:568) > [apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:696) > [apache-cassandra-3.0.9.jar:3.0.9] > > -- > Regards, > Mahesh Rajamani
Re: Cassandra Daemon not coming up
I did not add any user and disk space was fine. On Tue, Feb 27, 2018, 11:33 Rahul Singh wrote: > Were there any changes to the system such as permissions, etc. Did you add > users / change auth scheme? > > On Feb 27, 2018, 10:27 AM -0600, ZAIDI, ASAD A , wrote: > > Can you check if you’ve enough disk space available ? > > ~Asad > > > > *From:* mahesh rajamani [mailto:rajamani.mah...@gmail.com] > *Sent:* Tuesday, February 27, 2018 10:11 AM > *To:* user@cassandra.apache.org > *Subject:* Cassandra Daemon not coming up > > > > I am using Cassandra 3.0.9 version on a 12 node cluster. I have multiple > node down after a restart. The cassandra VM is not coming up with an asset > error as below. On running in debug mode it is failing while doing > operation on " resource_role_permissons_index" in system_auth keyspace. > Please let me know how to bring the cassandra running from this state. > > > > Logs from system.log > > > > INFO [main] 2018-02-27 15:43:24,005 ColumnFamilyStore.java:389 - > Initializing system_schema.columns > > > INFO [main] 2018-02-27 15:43:24,012 ColumnFamilyStore.java:389 - > Initializing system_schema.triggers > > > INFO [main] 2018-02-27 15:43:24,019 ColumnFamilyStore.java:389 - > Initializing system_schema.dropped_columns > > > INFO [main] 2018-02-27 15:43:24,029 ColumnFamilyStore.java:389 - > Initializing system_schema.views > > > INFO [main] 2018-02-27 15:43:24,038 ColumnFamilyStore.java:389 - > Initializing system_schema.types > > > INFO [main] 2018-02-27 15:43:24,049 ColumnFamilyStore.java:389 - > Initializing system_schema.functions > > > INFO [main] 2018-02-27 15:43:24,061 ColumnFamilyStore.java:389 - > Initializing system_schema.aggregates > > > INFO [main] 2018-02-27 15:43:24,072 ColumnFamilyStore.java:389 - > Initializing system_schema.indexes > > > ERROR [main] 2018-02-27 15:43:24,127 CassandraDaemon.java:709 - Exception > encountered during startup > > > java.lang.AssertionError: null > > > > at > org.apache.cassandra.db.marshal.CompositeType.getInstance(CompositeType.java:103) > ~[apache-cassandra-3.0.9.jar:3.0.9] > > > at > org.apache.cassandra.config.CFMetaData.rebuild(CFMetaData.java:311) > ~[apache-cassandra-3.0.9.jar:3.0.9] > > > at > org.apache.cassandra.config.CFMetaData.(CFMetaData.java:288) > ~[apache-cassandra-3.0.9.jar:3.0.9] > > > at > org.apache.cassandra.config.CFMetaData.create(CFMetaData.java:366) > ~[apache-cassandra-3.0.9.jar:3.0.9] > > > at > org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:954) > ~[apache-cassandra-3.0.9.jar:3.0.9] > > > at > org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:928) > ~[apache-cassandra-3.0.9.jar:3.0.9] > > > at > org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:891) > ~[apache-cassandra-3.0.9.jar:3.0.9] > > > at > org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:868) > ~[apache-cassandra-3.0.9.jar:3.0.9] > > > at > org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:856) > ~[apache-cassandra-3.0.9.jar:3.0.9] > > > at > org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:136) > ~[apache-cassandra-3.0.9.jar:3.0.9] > > > at > org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:126) > ~[apache-cassandra-3.0.9.jar:3.0.9] > > > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:239) > [apache-cassandra-3.0.9.jar:3.0.9] > > > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:568) > [apache-cassandra-3.0.9.jar:3.0.9] > > > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:696) > [apache-cassandra-3.0.9.jar:3.0.9] > > > > -- > > Regards, > Mahesh Rajamani > >
Re: Cassandra Daemon not coming up
It’s possible that the schema supporting roles and users is corrupted. Do you have a backup of it? Another quick fix would be to potentially reset permissions on your data dirs and restart. You can also inspect using the offline Cassandra sstable reader to see if it’s unaffected. -- Rahul Singh rahul.si...@anant.us Anant Corporation On Mar 5, 2018, 6:26 PM -0500, mahesh rajamani , wrote: > I did not add any user and disk space was fine. > > > > > On Tue, Feb 27, 2018, 11:33 Rahul Singh > > wrote: > > > Were there any changes to the system such as permissions, etc. Did you > > > add users / change auth scheme? > > > > > > On Feb 27, 2018, 10:27 AM -0600, ZAIDI, ASAD A , wrote: > > > > Can you check if you’ve enough disk space available ? > > > > ~Asad > > > > > > > > From: mahesh rajamani [mailto:rajamani.mah...@gmail.com] > > > > Sent: Tuesday, February 27, 2018 10:11 AM > > > > To: user@cassandra.apache.org > > > > Subject: Cassandra Daemon not coming up > > > > > > > > I am using Cassandra 3.0.9 version on a 12 node cluster. I have > > > > multiple node down after a restart. The cassandra VM is not coming up > > > > with an asset error as below. On running in debug mode it is failing > > > > while doing operation on " resource_role_permissons_index" in > > > > system_auth keyspace. Please let me know how to bring the cassandra > > > > running from this state. > > > > > > > > Logs from system.log > > > > > > > > INFO [main] 2018-02-27 15:43:24,005 ColumnFamilyStore.java:389 - > > > > Initializing system_schema.columns > > > > INFO [main] 2018-02-27 15:43:24,012 ColumnFamilyStore.java:389 - > > > > Initializing system_schema.triggers > > > > INFO [main] 2018-02-27 15:43:24,019 ColumnFamilyStore.java:389 - > > > > Initializing system_schema.dropped_columns > > > > INFO [main] 2018-02-27 15:43:24,029 ColumnFamilyStore.java:389 - > > > > Initializing system_schema.views > > > > INFO [main] 2018-02-27 15:43:24,038 ColumnFamilyStore.java:389 - > > > > Initializing system_schema.types > > > > INFO [main] 2018-02-27 15:43:24,049 ColumnFamilyStore.java:389 - > > > > Initializing system_schema.functions > > > > INFO [main] 2018-02-27 15:43:24,061 ColumnFamilyStore.java:389 - > > > > Initializing system_schema.aggregates > > > > INFO [main] 2018-02-27 15:43:24,072 ColumnFamilyStore.java:389 - > > > > Initializing system_schema.indexes > > > > ERROR [main] 2018-02-27 15:43:24,127 CassandraDaemon.java:709 - > > > > Exception encountered during startup > > > > java.lang.AssertionError: null > > > > at > > > > org.apache.cassandra.db.marshal.CompositeType.getInstance(CompositeType.java:103) > > > > ~[apache-cassandra-3.0.9.jar:3.0.9] > > > > at > > > > org.apache.cassandra.config.CFMetaData.rebuild(CFMetaData.java:311) > > > > ~[apache-cassandra-3.0.9.jar:3.0.9] > > > > at > > > > org.apache.cassandra.config.CFMetaData.(CFMetaData.java:288) > > > > ~[apache-cassandra-3.0.9.jar:3.0.9] > > > > at > > > > org.apache.cassandra.config.CFMetaData.create(CFMetaData.java:366) > > > > ~[apache-cassandra-3.0.9.jar:3.0.9] > > > > at > > > > org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:954) > > > > ~[apache-cassandra-3.0.9.jar:3.0.9] > > > > at > > > > org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:928) > > > > ~[apache-cassandra-3.0.9.jar:3.0.9] > > > > at > > > > org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:891) > > > > ~[apache-cassandra-3.0.9.jar:3.0.9] > > > > at > > > > org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:868) > > > > ~[apache-cassandra-3.0.9.jar:3.0.9] > > > > at > > > > org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:856) > > > > ~[apache-cassandra-3.0.9.jar:3.0.9] > > > > at > > > > org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:136) > > > > ~[apache-cassandra-3.0.9.jar:3.0.9] > > > > at > > > > org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:126) > > > > ~[apache-cassandra-3.0.9.jar:3.0.9] > > > > at > > > > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:239) > > > > [apache-cassandra-3.0.9.jar:3.0.9] > > > > at > > > > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:568) > > > > [apache-cassandra-3.0.9.jar:3.0.9] > > > > at > > > > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:696) > > > > [apache-cassandra-3.0.9.jar:3.0.9] > > > > > > > > -- > > > > Regards, > > > > Mahesh Rajamani