Re: Cassandra 1.1.1 Fails to Start
Looks like someone beat me to it, https://issues.apache.org/jira/browse/CASSANDRA-4321 On Fri, Jun 8, 2012 at 9:06 AM, Javier Sotelo wrote: > Different node same hardware now gets the stack overflow error but I found > the part of the stack trace that is more interesting: > > > at > com.google.common.collect.Iterators$5.hasNext(Iterators.java:517) > at > com.google.common.collect.Iterators$3.hasNext(Iterators.java:114) > at > com.google.common.collect.Iterators$5.hasNext(Iterators.java:517) > at > com.google.common.collect.Iterators$3.hasNext(Iterators.java:114) > at > com.google.common.collect.Iterators$7.computeNext(Iterators.java:614) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135) > at com.google.common.collect.Iterators.size(Iterators.java:129) > at com.google.common.collect.Sets$3.size(Sets.java:670) > at com.google.common.collect.Iterables.size(Iterables.java:80) > at > org.apache.cassandra.db.DataTracker.buildIntervalTree(DataTracker.java:557) > at > org.apache.cassandra.db.compaction.CompactionController.(CompactionController.java:79) > at > org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:105) > at > org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:50) > > Is it time for a JIRA ticket? > > > On Thu, Jun 7, 2012 at 7:03 AM, Javier Sotelo > wrote: > >> nodetool ring showed 34.89GB load. Upgrading from 1.1.0. One small >> keyspace with no compression, about 250MB. The rest taken by the second >> keyspace with leveled compaction and snappy compressed. >> >> The blade is an Intel(R) Xeon(R) CPU E5620 @ 2.40GHz with 6GB of RAM. >> >> >> On Thu, Jun 7, 2012 at 2:52 AM, aaron morton wrote: >> >>> How much data do you have on the node ? >>> Was this a previously running system that was upgraded ? >>> >>> > with disk_access_mode mmap_index_only and mmap I see OOM map failed >>> error on SSTableBatchOpen thread >>> Do you have the stack trace from the log ? >>> >>> > ERROR [CompactionExecutor:6] 2012-06-06 20:24:19,772 >>> AbstractCassandraDaemon.java (line 134) Exception in thread >>> Thread[CompactionExecutor:6,1,main] >>> > java.lang.StackOverflowError >>> > at com.google.common.collect.Sets$1.iterator(Sets.java:578) >>> > at com.google.common.collect.Sets$1.iterator(Sets.java:578) >>> > at com.google.common.collect.Sets$1.iterator(Sets.java:578) >>> Was there more to this stack trace ? >>> What were the log messages before this error ? >>> >>> >>> > INFO [main] 2012-06-06 20:17:10,267 AbstractCassandraDaemon.java >>> (line 122) Heap size: 1525415936/1525415936 >>> The JVM only has 1.5 G of ram, this is at the lower limit. If you have >>> some data to load I would not be surprised if it failed to start. >>> >>> Cheers >>> >>> - >>> Aaron Morton >>> Freelance Developer >>> @aaronmorton >>> http://www.thelastpickle.com >>> >>> On 7/06/2012, at 8:41 AM, Javier Sotelo wrote: >>> >>> > Hi All, >>> > >>> > On SuSe Linux blade with 6GB of RAM. >>> > >>> > with disk_access_mode mmap_index_only and mmap I see OOM map failed >>> error on SSTableBatchOpen thread. cat /proc//maps shows a peak of >>> 53521 right before it dies. vm.max_map_count = 1966080 and >>> /proc//limits shows unlimited locked memory. >>> > >>> > with disk_access_mode standard, the node does start up but I see the >>> repeated error: >>> > ERROR [CompactionExecutor:6] 2012-06-06 20:24:19,772 >>> AbstractCassandraDaemon.java (line 134) Exception in thread >>> Thread[CompactionExecutor:6,1,main] >>> > java.lang.StackOverflowError >>> > at com.google.common.collect.Sets$1.iterator(Sets.java:578) >>> > at com.google.common.collect.Sets$1.iterator(Sets.java:578) >>> > at com.google.common.collect.Sets$1.iterator(Sets.java:578) >>> > ... >>> > >>> > I'm not sure the second error is related to the first. I prefer to run >>> with full mmap but I have run out of ideas. Is there anything else I can do >>> to debug this? >>> > >>> > Here's startup settings from debug log: >>> > INFO [main] 2012-06-06 20:17:10,267 AbstractCassandraDaemon.java >>> (line 121) JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.6.0_31 >>> > INFO [main] 2012-06-06 20:17:10,267 AbstractCassandraDaemon.java >>> (line 122) Heap size: 1525415936/1525415936 >>> > ... >>> > INFO [main] 2012-06-06 20:17:10,946 CLibrary.java (line 111) JNA >>> mlockall successful >>> > ... >>> > INFO [main] 2012-06-06 20:17:11,055 DatabaseDescriptor.java (line >>> 191) DiskAccessMode is standard, indexAccessMode is standard >>> > INFO [main] 2012-06-06 20:17:11,213 DatabaseDescriptor.java (line >>> 247) Global memtable threshold is enabled at 484MB >>> > INFO [main] 2012-06-06 20:17:11,499 CacheService.java (line 9
Re: Cassandra 1.1.1 Fails to Start
Different node same hardware now gets the stack overflow error but I found the part of the stack trace that is more interesting: at com.google.common.collect.Iterators$5.hasNext(Iterators.java:517) at com.google.common.collect.Iterators$3.hasNext(Iterators.java:114) at com.google.common.collect.Iterators$5.hasNext(Iterators.java:517) at com.google.common.collect.Iterators$3.hasNext(Iterators.java:114) at com.google.common.collect.Iterators$7.computeNext(Iterators.java:614) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135) at com.google.common.collect.Iterators.size(Iterators.java:129) at com.google.common.collect.Sets$3.size(Sets.java:670) at com.google.common.collect.Iterables.size(Iterables.java:80) at org.apache.cassandra.db.DataTracker.buildIntervalTree(DataTracker.java:557) at org.apache.cassandra.db.compaction.CompactionController.(CompactionController.java:79) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:105) at org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:50) Is it time for a JIRA ticket? On Thu, Jun 7, 2012 at 7:03 AM, Javier Sotelo wrote: > nodetool ring showed 34.89GB load. Upgrading from 1.1.0. One small > keyspace with no compression, about 250MB. The rest taken by the second > keyspace with leveled compaction and snappy compressed. > > The blade is an Intel(R) Xeon(R) CPU E5620 @ 2.40GHz with 6GB of RAM. > > > On Thu, Jun 7, 2012 at 2:52 AM, aaron morton wrote: > >> How much data do you have on the node ? >> Was this a previously running system that was upgraded ? >> >> > with disk_access_mode mmap_index_only and mmap I see OOM map failed >> error on SSTableBatchOpen thread >> Do you have the stack trace from the log ? >> >> > ERROR [CompactionExecutor:6] 2012-06-06 20:24:19,772 >> AbstractCassandraDaemon.java (line 134) Exception in thread >> Thread[CompactionExecutor:6,1,main] >> > java.lang.StackOverflowError >> > at com.google.common.collect.Sets$1.iterator(Sets.java:578) >> > at com.google.common.collect.Sets$1.iterator(Sets.java:578) >> > at com.google.common.collect.Sets$1.iterator(Sets.java:578) >> Was there more to this stack trace ? >> What were the log messages before this error ? >> >> >> > INFO [main] 2012-06-06 20:17:10,267 AbstractCassandraDaemon.java (line >> 122) Heap size: 1525415936/1525415936 >> The JVM only has 1.5 G of ram, this is at the lower limit. If you have >> some data to load I would not be surprised if it failed to start. >> >> Cheers >> >> - >> Aaron Morton >> Freelance Developer >> @aaronmorton >> http://www.thelastpickle.com >> >> On 7/06/2012, at 8:41 AM, Javier Sotelo wrote: >> >> > Hi All, >> > >> > On SuSe Linux blade with 6GB of RAM. >> > >> > with disk_access_mode mmap_index_only and mmap I see OOM map failed >> error on SSTableBatchOpen thread. cat /proc//maps shows a peak of >> 53521 right before it dies. vm.max_map_count = 1966080 and >> /proc//limits shows unlimited locked memory. >> > >> > with disk_access_mode standard, the node does start up but I see the >> repeated error: >> > ERROR [CompactionExecutor:6] 2012-06-06 20:24:19,772 >> AbstractCassandraDaemon.java (line 134) Exception in thread >> Thread[CompactionExecutor:6,1,main] >> > java.lang.StackOverflowError >> > at com.google.common.collect.Sets$1.iterator(Sets.java:578) >> > at com.google.common.collect.Sets$1.iterator(Sets.java:578) >> > at com.google.common.collect.Sets$1.iterator(Sets.java:578) >> > ... >> > >> > I'm not sure the second error is related to the first. I prefer to run >> with full mmap but I have run out of ideas. Is there anything else I can do >> to debug this? >> > >> > Here's startup settings from debug log: >> > INFO [main] 2012-06-06 20:17:10,267 AbstractCassandraDaemon.java (line >> 121) JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.6.0_31 >> > INFO [main] 2012-06-06 20:17:10,267 AbstractCassandraDaemon.java (line >> 122) Heap size: 1525415936/1525415936 >> > ... >> > INFO [main] 2012-06-06 20:17:10,946 CLibrary.java (line 111) JNA >> mlockall successful >> > ... >> > INFO [main] 2012-06-06 20:17:11,055 DatabaseDescriptor.java (line 191) >> DiskAccessMode is standard, indexAccessMode is standard >> > INFO [main] 2012-06-06 20:17:11,213 DatabaseDescriptor.java (line 247) >> Global memtable threshold is enabled at 484MB >> > INFO [main] 2012-06-06 20:17:11,499 CacheService.java (line 96) >> Initializing key cache with capacity of 72 MBs. >> > INFO [main] 2012-06-06 20:17:11,509 CacheService.java (line 107) >> Scheduling key cache save to each 14400 seconds (going to save all keys). >> > INFO [main] 2012-06-06 20:17:11,510 CacheService.java (line 121) >> Initializing
Re: Cassandra 1.1.1 Fails to Start
nodetool ring showed 34.89GB load. Upgrading from 1.1.0. One small keyspace with no compression, about 250MB. The rest taken by the second keyspace with leveled compaction and snappy compressed. The blade is an Intel(R) Xeon(R) CPU E5620 @ 2.40GHz with 6GB of RAM. On Thu, Jun 7, 2012 at 2:52 AM, aaron morton wrote: > How much data do you have on the node ? > Was this a previously running system that was upgraded ? > > > with disk_access_mode mmap_index_only and mmap I see OOM map failed > error on SSTableBatchOpen thread > Do you have the stack trace from the log ? > > > ERROR [CompactionExecutor:6] 2012-06-06 20:24:19,772 > AbstractCassandraDaemon.java (line 134) Exception in thread > Thread[CompactionExecutor:6,1,main] > > java.lang.StackOverflowError > > at com.google.common.collect.Sets$1.iterator(Sets.java:578) > > at com.google.common.collect.Sets$1.iterator(Sets.java:578) > > at com.google.common.collect.Sets$1.iterator(Sets.java:578) > Was there more to this stack trace ? > What were the log messages before this error ? > > > > INFO [main] 2012-06-06 20:17:10,267 AbstractCassandraDaemon.java (line > 122) Heap size: 1525415936/1525415936 > The JVM only has 1.5 G of ram, this is at the lower limit. If you have > some data to load I would not be surprised if it failed to start. > > Cheers > > - > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com > > On 7/06/2012, at 8:41 AM, Javier Sotelo wrote: > > > Hi All, > > > > On SuSe Linux blade with 6GB of RAM. > > > > with disk_access_mode mmap_index_only and mmap I see OOM map failed > error on SSTableBatchOpen thread. cat /proc//maps shows a peak of > 53521 right before it dies. vm.max_map_count = 1966080 and > /proc//limits shows unlimited locked memory. > > > > with disk_access_mode standard, the node does start up but I see the > repeated error: > > ERROR [CompactionExecutor:6] 2012-06-06 20:24:19,772 > AbstractCassandraDaemon.java (line 134) Exception in thread > Thread[CompactionExecutor:6,1,main] > > java.lang.StackOverflowError > > at com.google.common.collect.Sets$1.iterator(Sets.java:578) > > at com.google.common.collect.Sets$1.iterator(Sets.java:578) > > at com.google.common.collect.Sets$1.iterator(Sets.java:578) > > ... > > > > I'm not sure the second error is related to the first. I prefer to run > with full mmap but I have run out of ideas. Is there anything else I can do > to debug this? > > > > Here's startup settings from debug log: > > INFO [main] 2012-06-06 20:17:10,267 AbstractCassandraDaemon.java (line > 121) JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.6.0_31 > > INFO [main] 2012-06-06 20:17:10,267 AbstractCassandraDaemon.java (line > 122) Heap size: 1525415936/1525415936 > > ... > > INFO [main] 2012-06-06 20:17:10,946 CLibrary.java (line 111) JNA > mlockall successful > > ... > > INFO [main] 2012-06-06 20:17:11,055 DatabaseDescriptor.java (line 191) > DiskAccessMode is standard, indexAccessMode is standard > > INFO [main] 2012-06-06 20:17:11,213 DatabaseDescriptor.java (line 247) > Global memtable threshold is enabled at 484MB > > INFO [main] 2012-06-06 20:17:11,499 CacheService.java (line 96) > Initializing key cache with capacity of 72 MBs. > > INFO [main] 2012-06-06 20:17:11,509 CacheService.java (line 107) > Scheduling key cache save to each 14400 seconds (going to save all keys). > > INFO [main] 2012-06-06 20:17:11,510 CacheService.java (line 121) > Initializing row cache with capacity of 0 MBs and provider > org.apache.cassandra.cache.SerializingCacheProvider > > INFO [main] 2012-06-06 20:17:11,513 CacheService.java (line 133) > Scheduling row cache save to each 0 seconds (going to save all keys). > > > > Thanks In Advance, > > Javier > >
Re: Cassandra 1.1.1 Fails to Start
How much data do you have on the node ? Was this a previously running system that was upgraded ? > with disk_access_mode mmap_index_only and mmap I see OOM map failed error on > SSTableBatchOpen thread Do you have the stack trace from the log ? > ERROR [CompactionExecutor:6] 2012-06-06 20:24:19,772 > AbstractCassandraDaemon.java (line 134) Exception in thread > Thread[CompactionExecutor:6,1,main] > java.lang.StackOverflowError > at com.google.common.collect.Sets$1.iterator(Sets.java:578) > at com.google.common.collect.Sets$1.iterator(Sets.java:578) > at com.google.common.collect.Sets$1.iterator(Sets.java:578) Was there more to this stack trace ? What were the log messages before this error ? > INFO [main] 2012-06-06 20:17:10,267 AbstractCassandraDaemon.java (line 122) > Heap size: 1525415936/1525415936 The JVM only has 1.5 G of ram, this is at the lower limit. If you have some data to load I would not be surprised if it failed to start. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 7/06/2012, at 8:41 AM, Javier Sotelo wrote: > Hi All, > > On SuSe Linux blade with 6GB of RAM. > > with disk_access_mode mmap_index_only and mmap I see OOM map failed error on > SSTableBatchOpen thread. cat /proc//maps shows a peak of 53521 right > before it dies. vm.max_map_count = 1966080 and /proc//limits shows > unlimited locked memory. > > with disk_access_mode standard, the node does start up but I see the repeated > error: > ERROR [CompactionExecutor:6] 2012-06-06 20:24:19,772 > AbstractCassandraDaemon.java (line 134) Exception in thread > Thread[CompactionExecutor:6,1,main] > java.lang.StackOverflowError > at com.google.common.collect.Sets$1.iterator(Sets.java:578) > at com.google.common.collect.Sets$1.iterator(Sets.java:578) > at com.google.common.collect.Sets$1.iterator(Sets.java:578) > ... > > I'm not sure the second error is related to the first. I prefer to run with > full mmap but I have run out of ideas. Is there anything else I can do to > debug this? > > Here's startup settings from debug log: > INFO [main] 2012-06-06 20:17:10,267 AbstractCassandraDaemon.java (line 121) > JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.6.0_31 > INFO [main] 2012-06-06 20:17:10,267 AbstractCassandraDaemon.java (line 122) > Heap size: 1525415936/1525415936 > ... > INFO [main] 2012-06-06 20:17:10,946 CLibrary.java (line 111) JNA mlockall > successful > ... > INFO [main] 2012-06-06 20:17:11,055 DatabaseDescriptor.java (line 191) > DiskAccessMode is standard, indexAccessMode is standard > INFO [main] 2012-06-06 20:17:11,213 DatabaseDescriptor.java (line 247) > Global memtable threshold is enabled at 484MB > INFO [main] 2012-06-06 20:17:11,499 CacheService.java (line 96) Initializing > key cache with capacity of 72 MBs. > INFO [main] 2012-06-06 20:17:11,509 CacheService.java (line 107) Scheduling > key cache save to each 14400 seconds (going to save all keys). > INFO [main] 2012-06-06 20:17:11,510 CacheService.java (line 121) > Initializing row cache with capacity of 0 MBs and provider > org.apache.cassandra.cache.SerializingCacheProvider > INFO [main] 2012-06-06 20:17:11,513 CacheService.java (line 133) Scheduling > row cache save to each 0 seconds (going to save all keys). > > Thanks In Advance, > Javier
Cassandra 1.1.1 Fails to Start
Hi All, On SuSe Linux blade with 6GB of RAM. with disk_access_mode mmap_index_only and mmap I see OOM map failed error on SSTableBatchOpen thread. cat /proc//maps shows a peak of 53521 right before it dies. vm.max_map_count = 1966080 and /proc//limits shows unlimited locked memory. with disk_access_mode standard, the node does start up but I see the repeated error: ERROR [CompactionExecutor:6] 2012-06-06 20:24:19,772 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[CompactionExecutor:6,1,main] java.lang.StackOverflowError at com.google.common.collect.Sets$1.iterator(Sets.java:578) at com.google.common.collect.Sets$1.iterator(Sets.java:578) at com.google.common.collect.Sets$1.iterator(Sets.java:578) ... I'm not sure the second error is related to the first. I prefer to run with full mmap but I have run out of ideas. Is there anything else I can do to debug this? Here's startup settings from debug log: INFO [main] 2012-06-06 20:17:10,267 AbstractCassandraDaemon.java (line 121) JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.6.0_31 INFO [main] 2012-06-06 20:17:10,267 AbstractCassandraDaemon.java (line 122) Heap size: 1525415936/1525415936 ... INFO [main] 2012-06-06 20:17:10,946 CLibrary.java (line 111) JNA mlockall successful ... INFO [main] 2012-06-06 20:17:11,055 DatabaseDescriptor.java (line 191) DiskAccessMode is standard, indexAccessMode is standard INFO [main] 2012-06-06 20:17:11,213 DatabaseDescriptor.java (line 247) Global memtable threshold is enabled at 484MB INFO [main] 2012-06-06 20:17:11,499 CacheService.java (line 96) Initializing key cache with capacity of 72 MBs. INFO [main] 2012-06-06 20:17:11,509 CacheService.java (line 107) Scheduling key cache save to each 14400 seconds (going to save all keys). INFO [main] 2012-06-06 20:17:11,510 CacheService.java (line 121) Initializing row cache with capacity of 0 MBs and provider org.apache.cassandra.cache.SerializingCacheProvider INFO [main] 2012-06-06 20:17:11,513 CacheService.java (line 133) Scheduling row cache save to each 0 seconds (going to save all keys). Thanks In Advance, Javier
