Looks like someone beat me to it, https://issues.apache.org/jira/browse/CASSANDRA-4321
On Fri, Jun 8, 2012 at 9:06 AM, Javier Sotelo <javier.a.sot...@gmail.com>wrote: > Different node same hardware now gets the stack overflow error but I found > the part of the stack trace that is more interesting: > > > at > com.google.common.collect.Iterators$5.hasNext(Iterators.java:517) > at > com.google.common.collect.Iterators$3.hasNext(Iterators.java:114) > at > com.google.common.collect.Iterators$5.hasNext(Iterators.java:517) > at > com.google.common.collect.Iterators$3.hasNext(Iterators.java:114) > at > com.google.common.collect.Iterators$7.computeNext(Iterators.java:614) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135) > at com.google.common.collect.Iterators.size(Iterators.java:129) > at com.google.common.collect.Sets$3.size(Sets.java:670) > at com.google.common.collect.Iterables.size(Iterables.java:80) > at > org.apache.cassandra.db.DataTracker.buildIntervalTree(DataTracker.java:557) > at > org.apache.cassandra.db.compaction.CompactionController.<init>(CompactionController.java:79) > at > org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:105) > at > org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:50) > > Is it time for a JIRA ticket? > > > On Thu, Jun 7, 2012 at 7:03 AM, Javier Sotelo > <javier.a.sot...@gmail.com>wrote: > >> nodetool ring showed 34.89GB load. Upgrading from 1.1.0. One small >> keyspace with no compression, about 250MB. The rest taken by the second >> keyspace with leveled compaction and snappy compressed. >> >> The blade is an Intel(R) Xeon(R) CPU E5620 @ 2.40GHz with 6GB of RAM. >> >> >> On Thu, Jun 7, 2012 at 2:52 AM, aaron morton <aa...@thelastpickle.com>wrote: >> >>> How much data do you have on the node ? >>> Was this a previously running system that was upgraded ? >>> >>> > with disk_access_mode mmap_index_only and mmap I see OOM map failed >>> error on SSTableBatchOpen thread >>> Do you have the stack trace from the log ? >>> >>> > ERROR [CompactionExecutor:6] 2012-06-06 20:24:19,772 >>> AbstractCassandraDaemon.java (line 134) Exception in thread >>> Thread[CompactionExecutor:6,1,main] >>> > java.lang.StackOverflowError >>> > at com.google.common.collect.Sets$1.iterator(Sets.java:578) >>> > at com.google.common.collect.Sets$1.iterator(Sets.java:578) >>> > at com.google.common.collect.Sets$1.iterator(Sets.java:578) >>> Was there more to this stack trace ? >>> What were the log messages before this error ? >>> >>> >>> > INFO [main] 2012-06-06 20:17:10,267 AbstractCassandraDaemon.java >>> (line 122) Heap size: 1525415936/1525415936 >>> The JVM only has 1.5 G of ram, this is at the lower limit. If you have >>> some data to load I would not be surprised if it failed to start. >>> >>> Cheers >>> >>> ----------------- >>> Aaron Morton >>> Freelance Developer >>> @aaronmorton >>> http://www.thelastpickle.com >>> >>> On 7/06/2012, at 8:41 AM, Javier Sotelo wrote: >>> >>> > Hi All, >>> > >>> > On SuSe Linux blade with 6GB of RAM. >>> > >>> > with disk_access_mode mmap_index_only and mmap I see OOM map failed >>> error on SSTableBatchOpen thread. cat /proc/<pid>/maps shows a peak of >>> 53521 right before it dies. vm.max_map_count = 1966080 and >>> /proc/<pid>/limits shows unlimited locked memory. >>> > >>> > with disk_access_mode standard, the node does start up but I see the >>> repeated error: >>> > ERROR [CompactionExecutor:6] 2012-06-06 20:24:19,772 >>> AbstractCassandraDaemon.java (line 134) Exception in thread >>> Thread[CompactionExecutor:6,1,main] >>> > java.lang.StackOverflowError >>> > at com.google.common.collect.Sets$1.iterator(Sets.java:578) >>> > at com.google.common.collect.Sets$1.iterator(Sets.java:578) >>> > at com.google.common.collect.Sets$1.iterator(Sets.java:578) >>> > ... >>> > >>> > I'm not sure the second error is related to the first. I prefer to run >>> with full mmap but I have run out of ideas. Is there anything else I can do >>> to debug this? >>> > >>> > Here's startup settings from debug log: >>> > INFO [main] 2012-06-06 20:17:10,267 AbstractCassandraDaemon.java >>> (line 121) JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.6.0_31 >>> > INFO [main] 2012-06-06 20:17:10,267 AbstractCassandraDaemon.java >>> (line 122) Heap size: 1525415936/1525415936 >>> > ... >>> > INFO [main] 2012-06-06 20:17:10,946 CLibrary.java (line 111) JNA >>> mlockall successful >>> > ... >>> > INFO [main] 2012-06-06 20:17:11,055 DatabaseDescriptor.java (line >>> 191) DiskAccessMode is standard, indexAccessMode is standard >>> > INFO [main] 2012-06-06 20:17:11,213 DatabaseDescriptor.java (line >>> 247) Global memtable threshold is enabled at 484MB >>> > INFO [main] 2012-06-06 20:17:11,499 CacheService.java (line 96) >>> Initializing key cache with capacity of 72 MBs. >>> > INFO [main] 2012-06-06 20:17:11,509 CacheService.java (line 107) >>> Scheduling key cache save to each 14400 seconds (going to save all keys). >>> > INFO [main] 2012-06-06 20:17:11,510 CacheService.java (line 121) >>> Initializing row cache with capacity of 0 MBs and provider >>> org.apache.cassandra.cache.SerializingCacheProvider >>> > INFO [main] 2012-06-06 20:17:11,513 CacheService.java (line 133) >>> Scheduling row cache save to each 0 seconds (going to save all keys). >>> > >>> > Thanks In Advance, >>> > Javier >>> >>> >> >
