[
https://issues.apache.org/jira/browse/CASSANDRA-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15330782#comment-15330782
]
Jeremy Hanna commented on CASSANDRA-12009:
------------------------------------------
This is a limits problem.
Check out the output from running
{code}
ulimit -a
{code}
or
{code}
cat /proc/<cassandra pid>/limits
{code}
What you'll likely need is to make sure your memlock setting is set to
unlimited. Also recommended is to set nofiles to something like 100000 and
nproc to 32768.
> Nodetool scrub crashes: java.lang.OutOfMemoryError: Map failed
> --------------------------------------------------------------
>
> Key: CASSANDRA-12009
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12009
> Project: Cassandra
> Issue Type: Bug
> Environment: # java -version
> openjdk version "1.8.0_91"
> OpenJDK Runtime Environment (build 1.8.0_91-b14)
> OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)
> # uname -a
> Linux cassandra-dfs-10-10-160-19 4.4.11-23.53.amzn1.x86_64 #1 SMP Wed Jun 1
> 22:22:50 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
> # rpm -qa datastax-ddc
> datastax-ddc-3.5.0-1.noarch
> Reporter: Victor Trac
>
> Running nodetool scrub fails after a day or two. Originally, the instance had
> 8GB of RAM with 4GB of Xmx/Xms. After seeing the OOM error several times, I
> upgraded the box to a 24GB RAM box, giving 20GB to Xms/Xmx to see if it would
> finish the scrub operation. Both cases failed with the same error as below
> (tried on both the CMS and G1 GC collectors):
> {code}
> INFO [Service Thread] 2016-06-14 15:01:17,968 StatusLogger.java:106 -
> system_traces.events 0,0
> DEBUG [GossipTasks:1] 2016-06-14 15:01:18,019 Gossiper.java:336 - Convicting
> /10.240.0.4 with status removed - alive false
> WARN [CompactionExecutor:69] 2016-06-14 15:01:18,151 OutputHandler.java:57 -
> Error reading row (stacktrace follows):
> org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
> at
> org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:156)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:173)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:70)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:58)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:96)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.CompressedSegmentedFile.<init>(CompressedSegmentedFile.java:47)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:188)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:249)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:176)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:128)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.tryAppend(SSTableRewriter.java:153)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.Scrubber.scrub(Scrubber.java:224)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:873)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager.access$300(CompactionManager.java:81)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager$3.execute(CompactionManager.java:366)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:314)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [na:1.8.0_91]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_91]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_91]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940)
> ~[na:1.8.0_91]
> at
> org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> ... 23 common frames omitted
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_91]
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937)
> ~[na:1.8.0_91]
> ... 24 common frames omitted
> WARN [CompactionExecutor:69] 2016-06-14 15:01:18,152 OutputHandler.java:52 -
> Row starting at position 31397042159 is unreadable; skipping to next
> WARN [CompactionExecutor:69] 2016-06-14 15:01:18,257 OutputHandler.java:57 -
> Error reading row (stacktrace follows):
> org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
> at
> org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:156)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:162)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:70)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:58)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:96)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.CompressedSegmentedFile.<init>(CompressedSegmentedFile.java:47)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:188)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:249)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:176)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:128)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.tryAppend(SSTableRewriter.java:153)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.Scrubber.scrub(Scrubber.java:224)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:873)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager.access$300(CompactionManager.java:81)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager$3.execute(CompactionManager.java:366)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:314)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [na:1.8.0_91]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_91]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_91]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940)
> ~[na:1.8.0_91]
> at
> org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> ... 23 common frames omitted
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_91]
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937)
> ~[na:1.8.0_91]
> ... 24 common frames omitted
> WARN [CompactionExecutor:69] 2016-06-14 15:01:18,258 OutputHandler.java:52 -
> Row starting at position 31397303324 is unreadable; skipping to next
> WARN [CompactionExecutor:69] 2016-06-14 15:01:18,363 OutputHandler.java:57 -
> Error reading row (stacktrace follows):
> org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
> at
> org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:156)
> ~[apache-cassandra-3.5.0.jar:3.5Gq
> at
> org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:162)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:70)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:58)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:96)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.CompressedSegmentedFile.<init>(CompressedSegmentedFile.java:47)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:188)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:249)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:176)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:128)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.tryAppend(SSTableRewriter.java:153)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.Scrubber.scrub(Scrubber.java:224)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:873)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager.access$300(CompactionManager.java:81)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager$3.execute(CompactionManager.java:366)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:314)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [na:1.8.0_91]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_91]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_91]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940)
> ~[na:1.8.0_91]
> at
> org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> ... 23 common frames omitted
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_91]
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937)
> ~[na:1.8.0_91]
> ... 24 common frames omitted
> WARN [CompactionExecutor:69] 2016-06-14 15:01:18,363 OutputHandler.java:52 -
> Row starting at position 31397313498 is unreadable; skipping to next
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
