[
https://issues.apache.org/jira/browse/CASSANDRA-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15330825#comment-15330825
]
Victor Trac commented on CASSANDRA-12009:
-----------------------------------------
I don't think our problem is due to limits:
{code}
# cat /proc/16854/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 32768 32768 processes
Max open files 100000 100000 files
Max locked memory unlimited unlimited bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 122664 122664 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 122664
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 900000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 122664
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
{code}
> Nodetool scrub crashes: java.lang.OutOfMemoryError: Map failed
> --------------------------------------------------------------
>
> Key: CASSANDRA-12009
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12009
> Project: Cassandra
> Issue Type: Bug
> Environment: # java -version
> openjdk version "1.8.0_91"
> OpenJDK Runtime Environment (build 1.8.0_91-b14)
> OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)
> # uname -a
> Linux cassandra-dfs-10-10-160-19 4.4.11-23.53.amzn1.x86_64 #1 SMP Wed Jun 1
> 22:22:50 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
> # rpm -qa datastax-ddc
> datastax-ddc-3.5.0-1.noarch
> Reporter: Victor Trac
>
> Running nodetool scrub fails after a day or two. Originally, the instance had
> 8GB of RAM with 4GB of Xmx/Xms. After seeing the OOM error several times, I
> upgraded the box to a 24GB RAM box, giving 20GB to Xms/Xmx to see if it would
> finish the scrub operation. Both cases failed with the same error as below
> (tried on both the CMS and G1 GC collectors):
> {code}
> INFO [Service Thread] 2016-06-14 15:01:17,968 StatusLogger.java:106 -
> system_traces.events 0,0
> DEBUG [GossipTasks:1] 2016-06-14 15:01:18,019 Gossiper.java:336 - Convicting
> /10.240.0.4 with status removed - alive false
> WARN [CompactionExecutor:69] 2016-06-14 15:01:18,151 OutputHandler.java:57 -
> Error reading row (stacktrace follows):
> org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
> at
> org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:156)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:173)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:70)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:58)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:96)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.CompressedSegmentedFile.<init>(CompressedSegmentedFile.java:47)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:188)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:249)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:176)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:128)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.tryAppend(SSTableRewriter.java:153)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.Scrubber.scrub(Scrubber.java:224)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:873)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager.access$300(CompactionManager.java:81)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager$3.execute(CompactionManager.java:366)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:314)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [na:1.8.0_91]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_91]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_91]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940)
> ~[na:1.8.0_91]
> at
> org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> ... 23 common frames omitted
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_91]
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937)
> ~[na:1.8.0_91]
> ... 24 common frames omitted
> WARN [CompactionExecutor:69] 2016-06-14 15:01:18,152 OutputHandler.java:52 -
> Row starting at position 31397042159 is unreadable; skipping to next
> WARN [CompactionExecutor:69] 2016-06-14 15:01:18,257 OutputHandler.java:57 -
> Error reading row (stacktrace follows):
> org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
> at
> org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:156)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:162)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:70)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:58)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:96)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.CompressedSegmentedFile.<init>(CompressedSegmentedFile.java:47)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:188)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:249)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:176)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:128)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.tryAppend(SSTableRewriter.java:153)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.Scrubber.scrub(Scrubber.java:224)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:873)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager.access$300(CompactionManager.java:81)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager$3.execute(CompactionManager.java:366)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:314)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [na:1.8.0_91]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_91]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_91]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940)
> ~[na:1.8.0_91]
> at
> org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> ... 23 common frames omitted
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_91]
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937)
> ~[na:1.8.0_91]
> ... 24 common frames omitted
> WARN [CompactionExecutor:69] 2016-06-14 15:01:18,258 OutputHandler.java:52 -
> Row starting at position 31397303324 is unreadable; skipping to next
> WARN [CompactionExecutor:69] 2016-06-14 15:01:18,363 OutputHandler.java:57 -
> Error reading row (stacktrace follows):
> org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
> at
> org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:156)
> ~[apache-cassandra-3.5.0.jar:3.5Gq
> at
> org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:162)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:70)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:58)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:96)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.CompressedSegmentedFile.<init>(CompressedSegmentedFile.java:47)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:188)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:249)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:176)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:128)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.tryAppend(SSTableRewriter.java:153)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.Scrubber.scrub(Scrubber.java:224)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:873)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager.access$300(CompactionManager.java:81)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager$3.execute(CompactionManager.java:366)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:314)
> [apache-cassandra-3.5.0.jar:3.5.0]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [na:1.8.0_91]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_91]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_91]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> Caused by: java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940)
> ~[na:1.8.0_91]
> at
> org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152)
> ~[apache-cassandra-3.5.0.jar:3.5.0]
> ... 23 common frames omitted
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_91]
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937)
> ~[na:1.8.0_91]
> ... 24 common frames omitted
> WARN [CompactionExecutor:69] 2016-06-14 15:01:18,363 OutputHandler.java:52 -
> Row starting at position 31397313498 is unreadable; skipping to next
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
