Re: Cassandra Out of Memory on startup while reading cache
As a work around remove the key / row caches before startup. Cheers - Aaron Morton Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 23/07/2013, at 6:44 AM, Janne Jalkanen wrote: > > Sounds like this: https://issues.apache.org/jira/browse/CASSANDRA-5706, which > is fixed in 1.2.7. > > /Janne > > On 22 Jul 2013, at 20:40, Jason Tyler wrote: > >> Hello, >> >> Since upgrading from 1.1.9 to 1.2.6 over the last week, we've had two >> instances where cassandra was unable, but kept trying to restart: >> >> SNIP >> INFO [main] 2013-07-19 16:12:36,769 AutoSavingCache.java (line 140) reading >> saved cache /var/cassandra/caches/SyncCore-CommEvents-KeyCache-b.db >> ERROR [main] 2013-07-19 16:12:36,966 CassandraDaemon.java (line 458) >> Exception encountered during startup >> java.lang.OutOfMemoryError: Java heap space >> at >> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394) >> at >> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:355) >> at >> org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:379) >> at >> org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:145) >> at >> org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:266) >> at >> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:382) >> at >> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:354) >> at org.apache.cassandra.db.Table.initCf(Table.java:329) >> at org.apache.cassandra.db.Table.(Table.java:272) >> at org.apache.cassandra.db.Table.open(Table.java:109) >> at org.apache.cassandra.db.Table.open(Table.java:87) >> at >> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:271) >> at >> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:441) >> at >> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:484) >> INFO [main] 2013-07-19 16:12:43,288 CassandraDaemon.java (line 118) Logging >> initialized >> SNIP >> >> This is new behavior with 1.2.6. >> >> Stopping cassandra, moving the offending file, then starting cassandra does >> succeed. >> >> Any config suggestions (key cache config?) to prevent this from happening? >> >> THX >> >> >> Cheers, >> >> ~Jason >
Re: Cassandra Out of Memory on startup while reading cache
Sounds like this: https://issues.apache.org/jira/browse/CASSANDRA-5706, which is fixed in 1.2.7. /Janne On 22 Jul 2013, at 20:40, Jason Tyler wrote: > Hello, > > Since upgrading from 1.1.9 to 1.2.6 over the last week, we've had two > instances where cassandra was unable, but kept trying to restart: > > SNIP > INFO [main] 2013-07-19 16:12:36,769 AutoSavingCache.java (line 140) reading > saved cache /var/cassandra/caches/SyncCore-CommEvents-KeyCache-b.db > ERROR [main] 2013-07-19 16:12:36,966 CassandraDaemon.java (line 458) > Exception encountered during startup > java.lang.OutOfMemoryError: Java heap space > at > org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394) > at > org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:355) > at > org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:379) > at > org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:145) > at > org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:266) > at > org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:382) > at > org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:354) > at org.apache.cassandra.db.Table.initCf(Table.java:329) > at org.apache.cassandra.db.Table.(Table.java:272) > at org.apache.cassandra.db.Table.open(Table.java:109) > at org.apache.cassandra.db.Table.open(Table.java:87) > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:271) > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:441) > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:484) > INFO [main] 2013-07-19 16:12:43,288 CassandraDaemon.java (line 118) Logging > initialized > SNIP > > This is new behavior with 1.2.6. > > Stopping cassandra, moving the offending file, then starting cassandra does > succeed. > > Any config suggestions (key cache config?) to prevent this from happening? > > THX > > > Cheers, > > ~Jason
Cassandra Out of Memory on startup while reading cache
Hello, Since upgrading from 1.1.9 to 1.2.6 over the last week, we've had two instances where cassandra was unable, but kept trying to restart: SNIP INFO [main] 2013-07-19 16:12:36,769 AutoSavingCache.java (line 140) reading saved cache /var/cassandra/caches/SyncCore-CommEvents-KeyCache-b.db ERROR [main] 2013-07-19 16:12:36,966 CassandraDaemon.java (line 458) Exception encountered during startup java.lang.OutOfMemoryError: Java heap space at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394) at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:355) at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:379) at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:145) at org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:266) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:382) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:354) at org.apache.cassandra.db.Table.initCf(Table.java:329) at org.apache.cassandra.db.Table.(Table.java:272) at org.apache.cassandra.db.Table.open(Table.java:109) at org.apache.cassandra.db.Table.open(Table.java:87) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:271) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:441) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:484) INFO [main] 2013-07-19 16:12:43,288 CassandraDaemon.java (line 118) Logging initialized SNIP This is new behavior with 1.2.6. Stopping cassandra, moving the offending file, then starting cassandra does succeed. Any config suggestions (key cache config?) to prevent this from happening? THX Cheers, ~Jason
