this looks like compaction is OOMing while processing a large row. On Fri, Oct 1, 2010 at 1:36 AM, Sudhir j <[email protected]> wrote: > Hi, > We are using cassandra 0.6.5 to store some analysis data. Recently we added > a timer task to remove data older than a month and deployed on a test > setup. > The removal worked fine but immediately after the data was removed cassandra > node got OutOfMemory in compaction thread. Heap size is configured as 1G on > the test machine and 8G on prod machine. > > Is slowing down the deletes from the timer task necessary (and recommended > practice) to avoid this (even after increasing heap size)? > Even after the uncaught exception GC Inspection thread kept running. (any > possible reasons for this) > > ERROR [COMPACTION-POOL:1] 2010-09-30 01:05:26,457 CassandraDaemon.java (line > 86) Uncaught exception in thread Thread[COMPACTION-POOL:1,5,main] > java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError > at > java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) > at java.util.concurrent.FutureTask.get(FutureTask.java:83) > at > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86) > at > org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:577) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > Caused by: java.lang.OutOfMemoryError > at java.io.RandomAccessFile.readBytes(Native Method) > at java.io.RandomAccessFile.read(RandomAccessFile.java:322) > at > org.apache.cassandra.io.util.BufferedRandomAccessFile.fillBuffer(BufferedRandomAccessFile.java:210) > at > org.apache.cassandra.io.util.BufferedRandomAccessFile.seek(BufferedRandomAccessFile.java:247) > at > org.apache.cassandra.io.util.BufferedRandomAccessFile.writeAtMost(BufferedRandomAccessFile.java:390) > at > org.apache.cassandra.io.util.BufferedRandomAccessFile.write(BufferedRandomAccessFile.java:366) > at java.io.DataOutputStream.writeUTF(DataOutputStream.java:384) > at java.io.RandomAccessFile.writeUTF(RandomAccessFile.java:1064) > at > org.apache.cassandra.io.SSTableWriter.append(SSTableWriter.java:96) > at > org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:300) > at > org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:102) > at > org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:83) > at > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > ... 2 more > INFO [GC inspection] 2010-09-30 01:05:26,462 GCInspector.java (line 156) > FLUSH-WRITER-POOL 0 0 > INFO [GC inspection] 2010-09-30 01:05:26,523 GCInspector.java (line 161) > CompactionManager n/a 1 > INFO [GC inspection] 2010-09-30 03:48:54,613 GCInspector.java (line 129) GC > for ParNew: 609 ms, 34948216 reclaimed leaving 819991944 used; max is > 1207828480 > INFO [GC inspection] 2010-09-30 04:49:46,435 GCInspector.java (line 129) GC > for ParNew: 4561 ms, 34929320 reclaimed leaving 819993312 used; max is > 1207828480 > > -- > Regards, > > Sudhir Jorwekar > > > >
-- Jonathan Ellis Project Chair, Apache Cassandra co-founder of Riptano, the source for professional Cassandra support http://riptano.com
