I upgraded to 0.6.1 and was able to bring up all the nodes and make queries.
After adding some new data, the java vm ran out of memory on three of the nodes. Cassandra continues to run for about 20 minutes before it exits completely: DEBUG [ROW-MUTATION-STAGE:2] 2010-04-30 16:02:27,298 RowMutationVerbHandler.java (line 73) RowMutation(table='Tracking', key='5f1c96b3', modifications=[ColumnFamily(IpTag [323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:7...@1272668502,])]) applied. Sending response to 749432@/10.1.29.20 DEBUG [ROW-MUTATION-STAGE:19] 2010-04-30 16:02:27,298 RowMutationVerbHandler.java (line 73) RowMutation(table='Tracking', key='4dfdfc26', modifications=[ColumnFamily(IpTag [323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8...@1272668513,])]) applied. Sending response to 1744234@/10.1.29.18 DEBUG [ROW-MUTATION-STAGE:91] 2010-04-30 16:02:27,449 RowMutationVerbHandler.java (line 73) RowMutation(table='Tracking', key='5efd19a7', modifications=[ColumnFamily(IpTag [323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8...@1272668511,])]) applied. Sending response to 969767@/10.1.29.22 ERROR [COMPACTION-POOL:1] 2010-04-30 16:02:27,295 DebuggableThreadPoolExecutor.java (line 94) Error in executor futuretask java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source) at java.util.concurrent.FutureTask.get(Unknown Source) at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.jav a:86) at org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:582) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Unknown Source) at java.io.ByteArrayOutputStream.write(Unknown Source) at java.io.DataOutputStream.write(Unknown Source) at java.io.FilterOutputStream.write(Unknown Source) at org.apache.cassandra.db.ColumnSerializer.writeName(ColumnSerializer.java:39) at org.apache.cassandra.db.ColumnSerializer.serialize(ColumnSerializer.java:61) at org.apache.cassandra.db.ColumnSerializer.serialize(ColumnSerializer.java:29) at org.apache.cassandra.db.ColumnFamilySerializer.serializeForSSTable(ColumnFamilySerializer.java:87) at org.apache.cassandra.db.ColumnFamilySerializer.serializeWithIndexes(ColumnFamilySerializer.java:99) at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:131) at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:41) at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:135) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:130) at org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183) at org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94) at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:299) at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:102) at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:83) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) ... 3 more DEBUG [ROW-MUTATION-STAGE:64] 2010-04-30 16:02:27,247 RowMutationVerbHandler.java (line 47) Applying RowMutation(table='Tracking', key='298dff6c', modifications=[ColumnFamily(IpTag [323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:7...@1272668521,])]) DEBUG [ROW-MUTATION-STAGE:20] 2010-04-30 16:02:27,247 RowMutationVerbHandler.java (line 47) Applying RowMutation(table='Tracking', key='5ce38012', modifications=[ColumnFamily(IpTag [323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8...@1272668520,])]) DEBUG [WRITE-/10.1.29.12] 2010-04-30 16:02:27,247 OutboundTcpConnection.java (line 142) attempting to connect to /10.1.29.12 . . . DEBUG [ROW-MUTATION-STAGE:22] 2010-04-30 16:33:50,235 RowMutationVerbHandler.java (line 47) Applying RowMutation(table='Tracking', key='29c4eeeb', modifications=[ColumnFamily(IpTag [323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8...@1272669788,])]) DEBUG [ROW-MUTATION-STAGE:22] 2010-04-30 16:33:50,235 RowMutationVerbHandler.java (line 73) RowMutation(table='Tracking', key='29c4eeeb', modifications=[ColumnFamily(IpTag [323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8...@1272669788,])]) applied. Sending response to 756622@/10.1.29.20 DEBUG [ROW-MUTATION-STAGE:22] 2010-04-30 16:33:50,236 RowMutationVerbHandler.java (line 47) Applying RowMutation(table='Tracking', key='5f25be67', modifications=[ColumnFamily(IpTag [323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8...@1272669788,])]) DEBUG [ROW-MUTATION-STAGE:22] 2010-04-30 16:33:50,236 RowMutationVerbHandler.java (line 73) RowMutation(table='Tracking', key='5f25be67', modifications=[ColumnFamily(IpTag [323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8...@1272669788,])]) applied. Sending response to 756627@/10.1.29.20 DEBUG [ROW-MUTATION-STAGE:34] 2010-04-30 16:24:01,230 RowMutationVerbHandler.java (line 47) Applying RowMutation(table='Tracking', key='51f2f208', modifications=[ColumnFamily(IpTag [323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:7...@1272669785,])]) DEBUG [ROW-MUTATION-STAGE:98] 2010-04-30 16:24:01,230 RowMutationVerbHandler.java (line 73) RowMutation(table='Tracking', key='29ed2f93', modifications=[ColumnFamily(IpTag [323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:7...@1272669778,])]) applied. Sending response to 1017012@/10.1.29.16 DEBUG [ROW-MUTATION-STAGE:8] 2010-04-30 16:24:01,230 RowMutationVerbHandler.java (line 47) Applying RowMutation(table='Tracking', key='3d40ecfc', modifications=[ColumnFamily(IpTag [323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8...@1272669781,])]) DEBUG [ROW-MUTATION-STAGE:67] 2010-04-30 16:24:01,228 RowMutationVerbHandler.java (line 47) Applying RowMutation(table='Tracking', key='4d1e2910', modifications=[ColumnFamily(IpTag [323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8...@1272669780,])]) On 04/26/2010 06:48 PM, Jonathan Ellis wrote: > 0.5 has a bug that allows it to OOM itself from replaying the log too > fast. You should upgrade to 0.6.1. > >