Hello friends, I'm getting a:
ERROR 22:50:29,695 Fatal exception in thread Thread[SSTableBatchOpen:2,5,main] java.lang.OutOfMemoryError: Java heap space error when I start Cassandra. This node was running fine and after some server work/upgrades it started throwing this error when I start the Cassandra service. I was on 0.8.? and have upgraded to 1.0.10 to see if it would help, but I get the same error. I've removed some of the column families from my keyspace directory to see if I can get it to start without the heap space error and with some combinations it will run. However, I'd like to get it running with all my colFams and wonder if someone could give me some advice on what might be causing my error. It doesn't seem to be related to compaction, if I am reading the log correctly, and most of the help I've found on this topic deals with compaction. I'm thinking that my 2 column families should not be enough to fill my heap, but I am at a loss as to what I should try next? Thanks for your consideration. output.log: INFO 22:50:26,319 JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.6.0_26 INFO 22:50:26,322 Heap size: 5905580032/5905580032 INFO 22:50:26,322 Classpath: /usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.4.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.2.jar:/usr/share/cassandra/lib/guava-r08.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.4.0.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.4.0.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jline-0.9.94.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.6.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.6.1.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.6.1.jar:/usr/share/cassandra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.4.1.jar:/usr/share/cassandra/apache-cassandra-1.0.10.jar:/usr/share/cassandra/apache-cassandra-thrift-1.0.10.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar INFO 22:50:28,586 JNA mlockall successful INFO 22:50:28,593 Loading settings from file:/etc/cassandra/cassandra.yaml DEBUG 22:50:28,677 Syncing log with a period of 10000 INFO 22:50:28,677 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap INFO 22:50:28,686 Global memtable threshold is enabled at 1877MB DEBUG 22:50:28,761 setting auto_bootstrap to true <snip> DEBUG 22:50:28,797 Checking directory /var/lib/cassandra/data DEBUG 22:50:28,798 Checking directory /var/lib/cassandra/commitlog DEBUG 22:50:28,798 Checking directory /var/lib/cassandra/saved_caches DEBUG 22:50:28,806 Removing compacted SSTable files from NodeIdInfo (see http://wiki.apache.org/cassandra/MemtableSSTable) DEBUG 22:50:28,808 Removing compacted SSTable files from Versions (see http://wiki.apache.org/cassandra/MemtableSSTable) DEBUG 22:50:28,818 Removing compacted SSTable files from Versions.76657273696f6e (see http://wiki.apache.org/cassandra/MemtableSSTable) DEBUG 22:50:28,819 Removing compacted SSTable files from IndexInfo (see http://wiki.apache.org/cassandra/MemtableSSTable) DEBUG 22:50:28,821 Removing compacted SSTable files from Schema (see http://wiki.apache.org/cassandra/MemtableSSTable) DEBUG 22:50:28,823 Removing compacted SSTable files from Migrations (see http://wiki.apache.org/cassandra/MemtableSSTable) DEBUG 22:50:28,825 Removing compacted SSTable files from LocationInfo (see http://wiki.apache.org/cassandra/MemtableSSTable) DEBUG 22:50:28,827 Removing compacted SSTable files from HintsColumnFamily (see http://wiki.apache.org/cassandra/MemtableSSTable) DEBUG 22:50:28,833 Initializing system.NodeIdInfo DEBUG 22:50:28,839 Starting CFS NodeIdInfo DEBUG 22:50:28,868 Creating IntervalNode from [] DEBUG 22:50:28,869 KeyCache capacity for NodeIdInfo is 1 DEBUG 22:50:28,871 Initializing system.Versions DEBUG 22:50:28,873 Starting CFS Versions INFO 22:50:28,877 Opening /var/lib/cassandra/data/system/Versions-hd-5 (248 bytes) DEBUG 22:50:28,879 Load metadata for /var/lib/cassandra/data/system/Versions-hd-5 INFO 22:50:28,880 Opening /var/lib/cassandra/data/system/Versions-hd-6 (248 bytes) DEBUG 22:50:28,880 Load metadata for /var/lib/cassandra/data/system/Versions-hd-6 DEBUG 22:50:28,897 INDEX LOAD TIME for /var/lib/cassandra/data/system/Versions-hd-5: 20 ms. DEBUG 22:50:28,898 INDEX LOAD TIME for /var/lib/cassandra/data/system/Versions-hd-6: 18 ms. DEBUG 22:50:28,899 key cache contains 0/0 keys DEBUG 22:50:28,899 key cache contains 0/0 keys INFO 22:50:28,899 Opening /var/lib/cassandra/data/system/Versions-hd-7 (248 bytes) DEBUG 22:50:28,900 Load metadata for /var/lib/cassandra/data/system/Versions-hd-7 DEBUG 22:50:28,901 INDEX LOAD TIME for /var/lib/cassandra/data/system/Versions-hd-7: 2 ms. DEBUG 22:50:28,901 key cache contains 0/0 keys DEBUG 22:50:28,903 Creating IntervalNode from [Interval(DecoratedKey(93220794208128599841715671226150005828, 746872696674), DecoratedKey(135285944860343992175601105924967452217, 63716c)), Interval(DecoratedKey(93220794208128599841715671226150005828, 746872696674), DecoratedKey(135285944860343992175601105924967452217, 63716c)), Interval(DecoratedKey(93220794208128599841715671226150005828, 746872696674), DecoratedKey(135285944860343992175601105924967452217, 63716c))] DEBUG 22:50:28,905 adding /var/lib/cassandra/data/system/Versions-hd-5 to list of files tracked for system.Versions DEBUG 22:50:28,905 adding /var/lib/cassandra/data/system/Versions-hd-6 to list of files tracked for system.Versions DEBUG 22:50:28,906 adding /var/lib/cassandra/data/system/Versions-hd-7 to list of files tracked for system.Versions DEBUG 22:50:28,906 KeyCache capacity for Versions is 3 DEBUG 22:50:28,906 Initializing system.IndexInfo DEBUG 22:50:28,908 Starting CFS IndexInfo DEBUG 22:50:28,910 Creating IntervalNode from [] DEBUG 22:50:28,910 KeyCache capacity for IndexInfo is 1 DEBUG 22:50:28,910 Initializing system.Schema DEBUG 22:50:28,912 Starting CFS Schema INFO 22:50:28,915 Opening /var/lib/cassandra/data/system/Schema-hd-46 (3831 bytes) DEBUG 22:50:28,916 Load metadata for /var/lib/cassandra/data/system/Schema-hd-46 DEBUG 22:50:28,918 INDEX LOAD TIME for /var/lib/cassandra/data/system/Schema-hd-46: 3 ms. INFO 22:50:28,915 Opening /var/lib/cassandra/data/system/Schema-hd-45 (187275 bytes) DEBUG 22:50:28,918 Load metadata for /var/lib/cassandra/data/system/Schema-hd-45 DEBUG 22:50:28,918 key cache contains 0/0 keys DEBUG 22:50:28,920 INDEX LOAD TIME for /var/lib/cassandra/data/system/Schema-hd-45: 5 ms. DEBUG 22:50:28,921 key cache contains 0/0 keys DEBUG 22:50:28,922 Creating IntervalNode from [Interval(DecoratedKey(124804735337540159479107746638263794797, 4c617374204d6967726174696f6e), DecoratedKey(168749730613100185197104044460479972233, 62626139343461302d633734632d313165312d303030302d353662363634353961646666)), Interval(DecoratedKey(2957058266746302108707016693211230853, 64323438663637302d633438662d313165312d303030302d366637313036623962396666), DecoratedKey(167756402517845603382936953195099512584, 65306532376534302d633734622d313165312d303030302d353662363634353961646666))] DEBUG 22:50:28,922 adding /var/lib/cassandra/data/system/Schema-hd-46 to list of files tracked for system.Schema DEBUG 22:50:28,923 adding /var/lib/cassandra/data/system/Schema-hd-45 to list of files tracked for system.Schema DEBUG 22:50:28,923 KeyCache capacity for Schema is 2 DEBUG 22:50:28,924 Initializing system.Migrations DEBUG 22:50:28,926 Starting CFS Migrations INFO 22:50:28,929 Opening /var/lib/cassandra/data/system/Migrations-hd-45 (323713 bytes) DEBUG 22:50:28,930 Load metadata for /var/lib/cassandra/data/system/Migrations-hd-45 INFO 22:50:28,931 Opening /var/lib/cassandra/data/system/Migrations-hd-46 (7945 bytes) DEBUG 22:50:28,931 Load metadata for /var/lib/cassandra/data/system/Migrations-hd-46 DEBUG 22:50:28,932 INDEX LOAD TIME for /var/lib/cassandra/data/system/Migrations-hd-45: 3 ms. DEBUG 22:50:28,933 INDEX LOAD TIME for /var/lib/cassandra/data/system/Migrations-hd-46: 2 ms. DEBUG 22:50:28,933 key cache contains 0/0 keys DEBUG 22:50:28,933 key cache contains 0/0 keys DEBUG 22:50:28,935 Creating IntervalNode from [Interval(DecoratedKey(159559397954378837828954138596956659794, 4d6967726174696f6e73204b6579), DecoratedKey(159559397954378837828954138596956659794, 4d6967726174696f6e73204b6579)), Interval(DecoratedKey(159559397954378837828954138596956659794, 4d6967726174696f6e73204b6579), DecoratedKey(159559397954378837828954138596956659794, 4d6967726174696f6e73204b6579))] DEBUG 22:50:28,936 adding /var/lib/cassandra/data/system/Migrations-hd-45 to list of files tracked for system.Migrations DEBUG 22:50:28,937 adding /var/lib/cassandra/data/system/Migrations-hd-46 to list of files tracked for system.Migrations DEBUG 22:50:28,937 KeyCache capacity for Migrations is 2 DEBUG 22:50:28,938 Initializing system.LocationInfo DEBUG 22:50:28,940 Starting CFS LocationInfo INFO 22:50:28,943 Opening /var/lib/cassandra/data/system/LocationInfo-hd-17 (357 bytes) DEBUG 22:50:28,944 Load metadata for /var/lib/cassandra/data/system/LocationInfo-hd-17 DEBUG 22:50:28,947 INDEX LOAD TIME for /var/lib/cassandra/data/system/LocationInfo-hd-17: 4 ms. DEBUG 22:50:28,948 key cache contains 0/0 keys DEBUG 22:50:28,948 Creating IntervalNode from [Interval(DecoratedKey(61078635599166706937511052402724559481, 4c), DecoratedKey(136380791207214638541884810421800860241, 426f6f747374726170))] DEBUG 22:50:28,949 adding /var/lib/cassandra/data/system/LocationInfo-hd-17 to list of files tracked for system.LocationInfo DEBUG 22:50:28,949 KeyCache capacity for LocationInfo is 1 DEBUG 22:50:28,949 Initializing system.HintsColumnFamily DEBUG 22:50:28,951 Starting CFS HintsColumnFamily DEBUG 22:50:28,953 Creating IntervalNode from [] DEBUG 22:50:28,954 KeyCache capacity for HintsColumnFamily is 1 DEBUG 22:50:28,959 collectTimeOrderedData DEBUG 22:50:28,975 collectTimeOrderedData INFO 22:50:28,977 Loading schema version bba944a0-c74c-11e1-0000-56b66459adff DEBUG 22:50:28,978 collectAllData DEBUG 22:50:28,983 collecting 0 of 2147483647: Avro/Schema:false:2799@1341566817771 DEBUG 22:50:28,983 collecting 1 of 2147483647: Innovari:false:788@1341566817771 DEBUG 22:50:29,143 Adding org.apache.cassandra.config.CFMetaData@37e55794[cfId=1017,ksName=Innovari,cfName=Temps,cfType=Standard,comparator=org.apache.cassandra.db.marshal.LongType,subcolumncomparator=<null>,comment=,rowCacheSize=0.0,keyCacheSize=200000.0,readRepairChance=1.0,replicateOnWrite=true,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.BytesType,minCompactionThreshold=4,maxCompactionThreshold=32,rowCacheSavePeriodInSeconds=0,keyCacheSavePeriodInSeconds=14400,rowCacheKeysToSave=2147483647,rowCacheProvider=org.apache.cassandra.cache.SerializingCacheProvider@31d8d237,mergeShardsChance=0.1,keyAlias=<null>,column_metadata={},compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionOptions={},bloomFilterFpChance=<null>] to cfIdMap DEBUG 22:50:29,144 Adding org.apache.cassandra.config.CFMetaData@3d46e381[cfId=1004,ksName=Innovari,cfName=TICStatusCurrent,cfType=Standard,comparator=org.apache.cassandra.db.marshal.AsciiType,subcolumncomparator=<null>,comment=,rowCacheSize=0.0,keyCacheSize=200000.0,readRepairChance=1.0,replicateOnWrite=true,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.BytesType,minCompactionThreshold=4,maxCompactionThreshold=32,rowCacheSavePeriodInSeconds=0,keyCacheSavePeriodInSeconds=14400,rowCacheKeysToSave=2147483647,rowCacheProvider=org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider@79024994,mergeShardsChance=0.1,keyAlias=<null>,column_metadata={},compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionOptions={},bloomFilterFpChance=<null>] to cfIdMap DEBUG 22:50:29,145 Removing compacted SSTable files from NodeIdInfo (see http://wiki.apache.org/cassandra/MemtableSSTable) DEBUG 22:50:29,147 Removing compacted SSTable files from Versions (see http://wiki.apache.org/cassandra/MemtableSSTable) DEBUG 22:50:29,149 Removing compacted SSTable files from Versions.76657273696f6e (see http://wiki.apache.org/cassandra/MemtableSSTable) DEBUG 22:50:29,150 Removing compacted SSTable files from IndexInfo (see http://wiki.apache.org/cassandra/MemtableSSTable) DEBUG 22:50:29,151 Removing compacted SSTable files from Schema (see http://wiki.apache.org/cassandra/MemtableSSTable) DEBUG 22:50:29,153 Removing compacted SSTable files from Migrations (see http://wiki.apache.org/cassandra/MemtableSSTable) DEBUG 22:50:29,155 Removing compacted SSTable files from LocationInfo (see http://wiki.apache.org/cassandra/MemtableSSTable) DEBUG 22:50:29,157 Removing compacted SSTable files from HintsColumnFamily (see http://wiki.apache.org/cassandra/MemtableSSTable) DEBUG 22:50:29,158 Removing compacted SSTable files from Temps (see http://wiki.apache.org/cassandra/MemtableSSTable) DEBUG 22:50:29,160 Removing compacted SSTable files from TICStatusCurrent (see http://wiki.apache.org/cassandra/MemtableSSTable) DEBUG 22:50:29,161 opening keyspace system DEBUG 22:50:29,161 opening keyspace Innovari DEBUG 22:50:29,162 Configured datacenter replicas are datacenter1:1 DEBUG 22:50:29,162 Initializing Innovari.Temps DEBUG 22:50:29,164 Starting CFS Temps INFO 22:50:29,166 Opening /var/lib/cassandra/data/Innovari/Temps-g-117 (156 bytes) DEBUG 22:50:29,166 Load metadata for /var/lib/cassandra/data/Innovari/Temps-g-117 INFO 22:50:29,166 Opening /var/lib/cassandra/data/Innovari/Temps-g-118 (156 bytes) DEBUG 22:50:29,167 Load metadata for /var/lib/cassandra/data/Innovari/Temps-g-118 java.lang.OutOfMemoryError: Java heap space Dumping heap to /var/lib/cassandra/java_1341615025.hprof ... Heap dump file created [47294429 bytes in 0.418 secs] ERROR 22:50:29,695 Fatal exception in thread Thread[SSTableBatchOpen:2,5,main] java.lang.OutOfMemoryError: Java heap space at org.apache.cassandra.utils.EstimatedHistogram$EstimatedHistogramSerializer.deserialize(EstimatedHistogram.java:222) at org.apache.cassandra.io.sstable.SSTableMetadata$SSTableMetadataSerializer.deserialize(SSTableMetadata.java:204) at org.apache.cassandra.io.sstable.SSTableMetadata$SSTableMetadataSerializer.deserialize(SSTableMetadata.java:194) at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:155) at org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:224) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Service exit with a return value of 100
