Sorry to create a new thread about Out of Memory problem, but I
checked all other threads and did not find the answer.
We have a running cluster of 2 cassandra nodes replication factor = 2
on red hat 4.8 32 bits with 4 G of memory where we run periodicaly
out of memory (every 2 weeks) and both nodes are crashing (trace at
the end of the file).
We are running 0.6.12 (do not had the time to upgrade to 0.7.3) and we
followed the memory configuration from the wiki :
memtable_throughput_in_mb * 3 * number of hot CFs + 1G + internal caches
memtable_throughput_in_mb = 16
We have 11 column families
So if my calculation is right, I need 16*3*11 + 1G + internal caches
: 528 Mbytes + 1G + internal caches, so more than 1,5 G of heap memory
We start with 2G
Here is the cassandra.in.sh parameters
JVM_OPTS=" \
-ea \
-Xms2000M \
-Xmx2000M \
-XX:+UseParNewGC \
-XX:+UseConcMarkSweepGC \
-XX:+CMSParallelRemarkEnabled \
-XX:SurvivorRatio=8 \
-XX:MaxTenuringThreshold=1 \
-XX:CMSInitiatingOccupancyFraction=75 \
-XX:+UseCMSInitiatingOccupancyOnly \
-XX:+HeapDumpOnOutOfMemoryError \
The question is I don't really understand the configuration problem,
if some body have any clue of what we could do, or what we should
monitor to avoid the problem, as it is very difficult to reproduce the
problem as it does not happen very often.
Thanks for any help.
Jean-Yves
<---------------------below the stack trace
------------------------------------------>
INFO [FLUSH-WRITER-POOL:1] 2011-03-04 17:59:45,952 Memtable.java (line 152)
Writing Memtable-HintsColumnFamily@15830055(2280 bytes, 240 operations)
INFO [FLUSH-WRITER-POOL:1] 2011-03-04 17:59:45,972 Memtable.java (line 166)
Completed flushing
/opt/database/data/system/HintsColumnFamily-164-Data.db (255 bytes)
INFO [FLUSH-TIMER] 2011-03-04 18:17:47,802 ColumnFamilyStore.java (line 478)
VoiceMail has reached its threshold; switching in a fresh Memtable at
CommitLogContext(file='/opt/database/commitlog/CommitLog-1297775500410.log',
position=10407973)
INFO [FLUSH-TIMER] 2011-03-04 18:17:47,803 ColumnFamilyStore.java (line 748)
Enqueuing flush of Memtable-VoiceMail@29468188(3000 bytes, 120 operations)
INFO [FLUSH-WRITER-POOL:1] 2011-03-04 18:17:47,803 Memtable.java (line 152)
Writing Memtable-VoiceMail@29468188(3000 bytes, 120 operations)
ERROR [FLUSH-WRITER-POOL:1] 2011-03-04 18:17:47,824 CassandraDaemon.java (line
87) Uncaught exception in thread Thread[FLUSH-WRITER-POOL:1,5,main]
java.lang.OutOfMemoryError
at java.io.RandomAccessFile.readBytes(Native Method)
at java.io.RandomAccessFile.read(RandomAccessFile.java:322)
at
org.apache.cassandra.io.util.BufferedRandomAccessFile.fillBuffer(BufferedRandomAccessFile.java:209)
at
org.apache.cassandra.io.util.BufferedRandomAccessFile.seek(BufferedRandomAccessFile.java:246)
at
org.apache.cassandra.io.util.BufferedRandomAccessFile.writeAtMost(BufferedRandomAccessFile.java:389)
at
org.apache.cassandra.io.util.BufferedRandomAccessFile.write(BufferedRandomAccessFile.java:365)
at java.io.DataOutputStream.writeUTF(DataOutputStream.java:384)
at java.io.RandomAccessFile.writeUTF(RandomAccessFile.java:1064)
at org.apache.cassandra.io.SSTableWriter.append(SSTableWriter.java:94)
at
org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:162)
at org.apache.cassandra.db.Memtable.access$000(Memtable.java:46)
at org.apache.cassandra.db.Memtable$1.runMayThrow(Memtable.java:178)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
INFO [main] 2011-03-07 17:46:09,454 CLibrary.java (line 47) JNA not found.
Native methods will be disabled.
INFO [main] 2011-03-07 17:46:09,768 DatabaseDescriptor.java (line 277)
DiskAccessMode 'auto' determined to be standard, indexAccessMode is standard
