With 248G per box, you probably have slightly more than 1/2 billion items? One current implementation detail in Cassandra is that it loads 128th of the index into memory for faster lookups. This means you might have something like 4.5 million keys in memory at the moment.
The '128' value is a constant at SSTable.INDEX_INTERVAL. You should be able to recompile with '1024' to allow for an 8 times larger database, but understand that this will have a negative effect on your read performance. Thanks, Stu -----Original Message----- From: "Dan Di Spaltro" <dan.dispal...@gmail.com> Sent: Sunday, December 13, 2009 12:06pm To: cassandra-user@incubator.apache.org Subject: Re: OOM Exception What consistencyLevel are you inserting the elements? If you do ./bin/nodeprobe -host localhost tpstats on each machine do you see one metric that has a lot of pending items? On Sun, Dec 13, 2009 at 8:14 AM, Brian Burruss <bburr...@real.com> wrote: > another OOM exception. the only thing interesting about my testing is that > there are 2 servers, RF=2, W=1, R=1 ... there is 248G of data on each > server. I have -Xmx3G assigned to each server > > 2009-12-12 22:04:37,436 ERROR [pool-1-thread-309] [Cassandra.java:734] > Internal error processing get > java.lang.RuntimeException: java.util.concurrent.ExecutionException: > java.lang.OutOfMemoryError: Java heap space > at > org.apache.cassandra.service.StorageProxy.weakReadLocal(StorageProxy.java:523) > at > org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:373) > at > org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:92) > at > org.apache.cassandra.service.CassandraServer.multigetColumns(CassandraServer.java:265) > at > org.apache.cassandra.service.CassandraServer.multigetInternal(CassandraServer.java:320) > at > org.apache.cassandra.service.CassandraServer.get(CassandraServer.java:253) > at > org.apache.cassandra.service.Cassandra$Processor$get.process(Cassandra.java:724) > at > org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:712) > at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:636) > > ________________________________________ > From: Brian Burruss > Sent: Saturday, December 12, 2009 7:45 AM > To: cassandra-user@incubator.apache.org > Subject: OOM Exception > > this happened after cassandra was running for a couple of days. I have > -Xmx3G on JVM. > > is there any other info you need so this makes sense? > > thx! > > > 2009-12-11 21:38:37,216 ERROR [HINTED-HANDOFF-POOL:1] > [DebuggableThreadPoolExecutor.java:157] Error in ThreadPoolExecutor > java.lang.OutOfMemoryError: Java heap space > at > org.apache.cassandra.io.BufferedRandomAccessFile.init(BufferedRandomAccessFile.java:151) > at > org.apache.cassandra.io.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:144) > at > org.apache.cassandra.io.SSTableWriter.<init>(SSTableWriter.java:53) > at > org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:911) > at > org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:855) > at > org.apache.cassandra.db.ColumnFamilyStore.doMajorCompactionInternal(ColumnFamilyStore.java:698) > at > org.apache.cassandra.db.ColumnFamilyStore.doMajorCompaction(ColumnFamilyStore.java:670) > at > org.apache.cassandra.db.HintedHandOffManager.deliverAllHints(HintedHandOffManager.java:190) > at > org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOffManager.java:75) > at > org.apache.cassandra.db.HintedHandOffManager$1.run(HintedHandOffManager.java:249) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:636) > -- Dan Di Spaltro
