Re: Java heap space on Cassandra start up version 1.0.10
Thanks Jonathan that did the trick. I deleted the Statistics.db files for the offending column family and was able to get Cassandra to start. Thank you, Jason
Re: Java heap space on Cassandra start up version 1.0.10
You may have a corrupt metadata/statistics sstable component. You can try deleting those and restarting. Cassandra can rebuild that component if it is missing. On Fri, Jul 6, 2012 at 6:00 PM, Jason Hill jasonhill...@gmail.com wrote: Hello friends, I'm getting a: ERROR 22:50:29,695 Fatal exception in thread Thread[SSTableBatchOpen:2,5,main] java.lang.OutOfMemoryError: Java heap space error when I start Cassandra. This node was running fine and after some server work/upgrades it started throwing this error when I start the Cassandra service. I was on 0.8.? and have upgraded to 1.0.10 to see if it would help, but I get the same error. I've removed some of the column families from my keyspace directory to see if I can get it to start without the heap space error and with some combinations it will run. However, I'd like to get it running with all my colFams and wonder if someone could give me some advice on what might be causing my error. It doesn't seem to be related to compaction, if I am reading the log correctly, and most of the help I've found on this topic deals with compaction. I'm thinking that my 2 column families should not be enough to fill my heap, but I am at a loss as to what I should try next? Thanks for your consideration. output.log: INFO 22:50:26,319 JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.6.0_26 INFO 22:50:26,322 Heap size: 5905580032/5905580032 INFO 22:50:26,322 Classpath: /usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.4.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.2.jar:/usr/share/cassandra/lib/guava-r08.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.4.0.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.4.0.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jline-0.9.94.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.6.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.6.1.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.6.1.jar:/usr/share/cassandra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.4.1.jar:/usr/share/cassandra/apache-cassandra-1.0.10.jar:/usr/share/cassandra/apache-cassandra-thrift-1.0.10.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar INFO 22:50:28,586 JNA mlockall successful INFO 22:50:28,593 Loading settings from file:/etc/cassandra/cassandra.yaml DEBUG 22:50:28,677 Syncing log with a period of 1 INFO 22:50:28,677 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap INFO 22:50:28,686 Global memtable threshold is enabled at 1877MB DEBUG 22:50:28,761 setting auto_bootstrap to true snip DEBUG 22:50:28,797 Checking directory /var/lib/cassandra/data DEBUG 22:50:28,798 Checking directory /var/lib/cassandra/commitlog DEBUG 22:50:28,798 Checking directory /var/lib/cassandra/saved_caches DEBUG 22:50:28,806 Removing compacted SSTable files from NodeIdInfo (see http://wiki.apache.org/cassandra/MemtableSSTable) DEBUG 22:50:28,808 Removing compacted SSTable files from Versions (see http://wiki.apache.org/cassandra/MemtableSSTable) DEBUG 22:50:28,818 Removing compacted SSTable files from Versions.76657273696f6e (see http://wiki.apache.org/cassandra/MemtableSSTable) DEBUG 22:50:28,819 Removing compacted SSTable files from IndexInfo (see http://wiki.apache.org/cassandra/MemtableSSTable) DEBUG 22:50:28,821 Removing compacted SSTable files from Schema (see http://wiki.apache.org/cassandra/MemtableSSTable) DEBUG 22:50:28,823 Removing compacted SSTable files from Migrations (see http://wiki.apache.org/cassandra/MemtableSSTable) DEBUG 22:50:28,825 Removing compacted SSTable files from LocationInfo (see http://wiki.apache.org/cassandra/MemtableSSTable) DEBUG 22:50:28,827 Removing compacted SSTable files from HintsColumnFamily (see http://wiki.apache.org/cassandra/MemtableSSTable) DEBUG 22:50:28,833 Initializing system.NodeIdInfo DEBUG 22:50:28,839 Starting CFS NodeIdInfo DEBUG 22:50:28,868 Creating IntervalNode from [] DEBUG 22:50:28,869 KeyCache capacity for NodeIdInfo is 1 DEBUG 22:50:28,871 Initializing system.Versions DEBUG 22:50:28,873 Starting CFS Versions INFO 22:50:28,877 Opening /var/lib/cassandra/data/system/Versions-hd-5 (248 bytes) DEBUG 22:50:28,879 Load metadata for /var/lib/cassandra/data/system/Versions-hd-5 INFO 22:50:28,880 Opening /var/lib/cassandra/data/system/Versions-hd-6 (248 bytes) DEBUG 22:50:28,880 Load metadata for
Re: Java heap space on Cassandra start up version 1.0.10
That's a good point Tyler. I watched Top during this process and even though the heap dump is small, I can see all of my memory resources consumed while Cassandra tries to start. I have the heap dump and can run the Memory Analyzer Tool in Eclipse on it, but I will confess, I'm not sure what I am looking for that would be interesting. Can you help me out with that? Jason On Sat, Jul 7, 2012 at 8:20 PM, Tyler Hobbs ty...@datastax.com wrote: The heap dump is only 47mb, so something strange is going on. Is there anything interesting in the heap dump?
Re: Java heap space on Cassandra start up version 1.0.10
The heap dump is only 47mb, so something strange is going on. Is there anything interesting in the heap dump? On Fri, Jul 6, 2012 at 6:00 PM, Jason Hill jasonhill...@gmail.com wrote: Hello friends, I'm getting a: ERROR 22:50:29,695 Fatal exception in thread Thread[SSTableBatchOpen:2,5,main] java.lang.OutOfMemoryError: Java heap space error when I start Cassandra. This node was running fine and after some server work/upgrades it started throwing this error when I start the Cassandra service. I was on 0.8.? and have upgraded to 1.0.10 to see if it would help, but I get the same error. I've removed some of the column families from my keyspace directory to see if I can get it to start without the heap space error and with some combinations it will run. However, I'd like to get it running with all my colFams and wonder if someone could give me some advice on what might be causing my error. It doesn't seem to be related to compaction, if I am reading the log correctly, and most of the help I've found on this topic deals with compaction. I'm thinking that my 2 column families should not be enough to fill my heap, but I am at a loss as to what I should try next? Thanks for your consideration. output.log: INFO 22:50:26,319 JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.6.0_26 INFO 22:50:26,322 Heap size: 5905580032/5905580032 INFO 22:50:26,322 Classpath: /usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.4.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.2.jar:/usr/share/cassandra/lib/guava-r08.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.4.0.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.4.0.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jline-0.9.94.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.6.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.6.1.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.6.1.jar:/usr/share/cassandra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.4.1.jar:/usr/share/cassandra/apache-cassandra-1.0.10.jar:/usr/share/cassandra/apache-cassandra-thrift-1.0.10.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar INFO 22:50:28,586 JNA mlockall successful INFO 22:50:28,593 Loading settings from file:/etc/cassandra/cassandra.yaml DEBUG 22:50:28,677 Syncing log with a period of 1 INFO 22:50:28,677 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap INFO 22:50:28,686 Global memtable threshold is enabled at 1877MB DEBUG 22:50:28,761 setting auto_bootstrap to true snip DEBUG 22:50:28,797 Checking directory /var/lib/cassandra/data DEBUG 22:50:28,798 Checking directory /var/lib/cassandra/commitlog DEBUG 22:50:28,798 Checking directory /var/lib/cassandra/saved_caches DEBUG 22:50:28,806 Removing compacted SSTable files from NodeIdInfo (see http://wiki.apache.org/cassandra/MemtableSSTable) DEBUG 22:50:28,808 Removing compacted SSTable files from Versions (see http://wiki.apache.org/cassandra/MemtableSSTable) DEBUG 22:50:28,818 Removing compacted SSTable files from Versions.76657273696f6e (see http://wiki.apache.org/cassandra/MemtableSSTable) DEBUG 22:50:28,819 Removing compacted SSTable files from IndexInfo (see http://wiki.apache.org/cassandra/MemtableSSTable) DEBUG 22:50:28,821 Removing compacted SSTable files from Schema (see http://wiki.apache.org/cassandra/MemtableSSTable) DEBUG 22:50:28,823 Removing compacted SSTable files from Migrations (see http://wiki.apache.org/cassandra/MemtableSSTable) DEBUG 22:50:28,825 Removing compacted SSTable files from LocationInfo (see http://wiki.apache.org/cassandra/MemtableSSTable) DEBUG 22:50:28,827 Removing compacted SSTable files from HintsColumnFamily (see http://wiki.apache.org/cassandra/MemtableSSTable) DEBUG 22:50:28,833 Initializing system.NodeIdInfo DEBUG 22:50:28,839 Starting CFS NodeIdInfo DEBUG 22:50:28,868 Creating IntervalNode from [] DEBUG 22:50:28,869 KeyCache capacity for NodeIdInfo is 1 DEBUG 22:50:28,871 Initializing system.Versions DEBUG 22:50:28,873 Starting CFS Versions INFO 22:50:28,877 Opening /var/lib/cassandra/data/system/Versions-hd-5 (248 bytes) DEBUG 22:50:28,879 Load metadata for /var/lib/cassandra/data/system/Versions-hd-5 INFO 22:50:28,880 Opening /var/lib/cassandra/data/system/Versions-hd-6 (248 bytes) DEBUG 22:50:28,880 Load metadata for /var/lib/cassandra/data/system/Versions-hd-6 DEBUG 22:50:28,897 INDEX LOAD TIME for
Java heap space on Cassandra start up version 1.0.10
Hello friends, I'm getting a: ERROR 22:50:29,695 Fatal exception in thread Thread[SSTableBatchOpen:2,5,main] java.lang.OutOfMemoryError: Java heap space error when I start Cassandra. This node was running fine and after some server work/upgrades it started throwing this error when I start the Cassandra service. I was on 0.8.? and have upgraded to 1.0.10 to see if it would help, but I get the same error. I've removed some of the column families from my keyspace directory to see if I can get it to start without the heap space error and with some combinations it will run. However, I'd like to get it running with all my colFams and wonder if someone could give me some advice on what might be causing my error. It doesn't seem to be related to compaction, if I am reading the log correctly, and most of the help I've found on this topic deals with compaction. I'm thinking that my 2 column families should not be enough to fill my heap, but I am at a loss as to what I should try next? Thanks for your consideration. output.log: INFO 22:50:26,319 JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.6.0_26 INFO 22:50:26,322 Heap size: 5905580032/5905580032 INFO 22:50:26,322 Classpath: /usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.4.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.2.jar:/usr/share/cassandra/lib/guava-r08.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.4.0.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.4.0.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jline-0.9.94.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.6.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.6.1.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.6.1.jar:/usr/share/cassandra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.4.1.jar:/usr/share/cassandra/apache-cassandra-1.0.10.jar:/usr/share/cassandra/apache-cassandra-thrift-1.0.10.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar INFO 22:50:28,586 JNA mlockall successful INFO 22:50:28,593 Loading settings from file:/etc/cassandra/cassandra.yaml DEBUG 22:50:28,677 Syncing log with a period of 1 INFO 22:50:28,677 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap INFO 22:50:28,686 Global memtable threshold is enabled at 1877MB DEBUG 22:50:28,761 setting auto_bootstrap to true snip DEBUG 22:50:28,797 Checking directory /var/lib/cassandra/data DEBUG 22:50:28,798 Checking directory /var/lib/cassandra/commitlog DEBUG 22:50:28,798 Checking directory /var/lib/cassandra/saved_caches DEBUG 22:50:28,806 Removing compacted SSTable files from NodeIdInfo (see http://wiki.apache.org/cassandra/MemtableSSTable) DEBUG 22:50:28,808 Removing compacted SSTable files from Versions (see http://wiki.apache.org/cassandra/MemtableSSTable) DEBUG 22:50:28,818 Removing compacted SSTable files from Versions.76657273696f6e (see http://wiki.apache.org/cassandra/MemtableSSTable) DEBUG 22:50:28,819 Removing compacted SSTable files from IndexInfo (see http://wiki.apache.org/cassandra/MemtableSSTable) DEBUG 22:50:28,821 Removing compacted SSTable files from Schema (see http://wiki.apache.org/cassandra/MemtableSSTable) DEBUG 22:50:28,823 Removing compacted SSTable files from Migrations (see http://wiki.apache.org/cassandra/MemtableSSTable) DEBUG 22:50:28,825 Removing compacted SSTable files from LocationInfo (see http://wiki.apache.org/cassandra/MemtableSSTable) DEBUG 22:50:28,827 Removing compacted SSTable files from HintsColumnFamily (see http://wiki.apache.org/cassandra/MemtableSSTable) DEBUG 22:50:28,833 Initializing system.NodeIdInfo DEBUG 22:50:28,839 Starting CFS NodeIdInfo DEBUG 22:50:28,868 Creating IntervalNode from [] DEBUG 22:50:28,869 KeyCache capacity for NodeIdInfo is 1 DEBUG 22:50:28,871 Initializing system.Versions DEBUG 22:50:28,873 Starting CFS Versions INFO 22:50:28,877 Opening /var/lib/cassandra/data/system/Versions-hd-5 (248 bytes) DEBUG 22:50:28,879 Load metadata for /var/lib/cassandra/data/system/Versions-hd-5 INFO 22:50:28,880 Opening /var/lib/cassandra/data/system/Versions-hd-6 (248 bytes) DEBUG 22:50:28,880 Load metadata for /var/lib/cassandra/data/system/Versions-hd-6 DEBUG 22:50:28,897 INDEX LOAD TIME for /var/lib/cassandra/data/system/Versions-hd-5: 20 ms. DEBUG 22:50:28,898 INDEX LOAD TIME for /var/lib/cassandra/data/system/Versions-hd-6: 18 ms. DEBUG 22:50:28,899 key cache contains 0/0 keys DEBUG 22:50:28,899 key cache contains 0/0 keys INFO 22:50:28,899
