[
https://issues.apache.org/jira/browse/CASSANDRA-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15330782#comment-15330782
]
Jeremy Hanna edited comment on CASSANDRA-12009 at 6/14/16 10:34 PM:
--------------------------------------------------------------------
This is a limits problem. Often OOMs are not due to limited memory but
insufficient OS limits for the process.
Check out the output from running
{code}
ulimit -a
{code}
or
{code}
cat /proc/<cassandra pid>/limits
{code}
What you'll likely need is to make sure your memlock setting is set to
unlimited. Also recommended is to set nofiles to something like 100000 and
nproc to 32768.
was (Author: jeromatron):
This is a limits problem.
Check out the output from running
{code}
ulimit -a
{code}
or
{code}
cat /proc/<cassandra pid>/limits
{code}
What you'll likely need is to make sure your memlock setting is set to
unlimited. Also recommended is to set nofiles to something like 100000 and
nproc to 32768.
> Nodetool scrub crashes: java.lang.OutOfMemoryError: Map failed
> --------------------------------------------------------------
>
> Key: CASSANDRA-12009
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12009
> Project: Cassandra
> Issue Type: Bug
> Environment: # java -version
> openjdk version "1.8.0_91"
> OpenJDK Runtime Environment (build 1.8.0_91-b14)
> OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)
> # uname -a
> Linux cassandra-dfs-10-10-160-19 4.4.11-23.53.amzn1.x86_64 #1 SMP Wed Jun 1
> 22:22:50 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
> # rpm -qa datastax-ddc
> datastax-ddc-3.5.0-1.noarch
> Reporter: Victor Trac
>
> Running nodetool scrub fails after a day or two. Originally, the instance had
> 8GB of RAM with 4GB of Xmx/Xms. After seeing the OOM error several times, I
> upgraded the box to a 24GB RAM box, giving 20GB to Xms/Xmx to see if it would
> finish the scrub operation. Both cases failed with the same error as below
> (tried on both the CMS and G1 GC collectors):
> {code}
> INFO [Service Thread] 2016-06-14 15:01:17,968 StatusLogger.java:106 -
> system_traces.events 0,0
> DEBUG [GossipTasks:1] 2016-06-14 15:01:18,019 Gossiper.java:336 - Convicting
> /10.240.0.4 with status removed - alive false
> WARN [CompactionExecutor:69] 2016-06-14 15:01:18,151 OutputHandler.java:57 -
> Error reading row (stacktrace follows):
> org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
> at
> org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:156)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:173)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:70)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:58)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:96)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.CompressedSegmentedFile.<init>(CompressedSegmentedFile.java:47)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:188)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:249)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:176)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:128)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.tryAppend(SSTableRewriter.java:153)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.Scrubber.scrub(Scrubber.java:224)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:873)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager.access$300(CompactionManager.java:81)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager$3.execute(CompactionManager.java:366)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:314)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [na:1.8.0_91]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_91]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_91]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940)
> ~[na:1.8.0_91]
> at
> org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> ... 23 common frames omitted
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_91]
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937)
> ~[na:1.8.0_91]
> ... 24 common frames omitted
> WARN [CompactionExecutor:69] 2016-06-14 15:01:18,152 OutputHandler.java:52 -
> Row starting at position 31397042159 is unreadable; skipping to next
> WARN [CompactionExecutor:69] 2016-06-14 15:01:18,257 OutputHandler.java:57 -
> Error reading row (stacktrace follows):
> org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
> at
> org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:156)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:162)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:70)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:58)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:96)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.CompressedSegmentedFile.<init>(CompressedSegmentedFile.java:47)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:188)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:249)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:176)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:128)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.tryAppend(SSTableRewriter.java:153)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.Scrubber.scrub(Scrubber.java:224)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:873)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager.access$300(CompactionManager.java:81)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager$3.execute(CompactionManager.java:366)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:314)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [na:1.8.0_91]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_91]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_91]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940)
> ~[na:1.8.0_91]
> at
> org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> ... 23 common frames omitted
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_91]
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937)
> ~[na:1.8.0_91]
> ... 24 common frames omitted
> WARN [CompactionExecutor:69] 2016-06-14 15:01:18,258 OutputHandler.java:52 -
> Row starting at position 31397303324 is unreadable; skipping to next
> WARN [CompactionExecutor:69] 2016-06-14 15:01:18,363 OutputHandler.java:57 -
> Error reading row (stacktrace follows):
> org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
> at
> org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:156)
> ~[apache-cassandra-3.5.0.jar:3.5Gq
> at
> org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:162)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:70)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:58)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:96)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.CompressedSegmentedFile.<init>(CompressedSegmentedFile.java:47)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:188)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:249)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:176)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:128)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.tryAppend(SSTableRewriter.java:153)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.Scrubber.scrub(Scrubber.java:224)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:873)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager.access$300(CompactionManager.java:81)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager$3.execute(CompactionManager.java:366)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:314)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [na:1.8.0_91]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_91]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_91]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940)
> ~[na:1.8.0_91]
> at
> org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> ... 23 common frames omitted
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_91]
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937)
> ~[na:1.8.0_91]
> ... 24 common frames omitted
> WARN [CompactionExecutor:69] 2016-06-14 15:01:18,363 OutputHandler.java:52 -
> Row starting at position 31397313498 is unreadable; skipping to next
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
