[jira] [Commented] (CASSANDRA-13008) Add vm.max_map_count StartupCheck
[ https://issues.apache.org/jira/browse/CASSANDRA-13008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15862512#comment-15862512 ] Jay Zhuang commented on CASSANDRA-13008: Thanks [~Stefania] for the review. You're right, no need to check if it's not {{mmap}}: https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/util/CompressedSegmentedFile.java#L46 > Add vm.max_map_count StartupCheck > - > > Key: CASSANDRA-13008 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13008 > Project: Cassandra > Issue Type: Improvement > Components: Configuration >Reporter: Jay Zhuang >Assignee: Jay Zhuang >Priority: Minor > Fix For: 3.0.x > > Attachments: 13008-3.0.txt > > > It's recommended to set {{vm.max_map_count}} to 1048575 (CASSANDRA-3563) > When the max_map_count is low, it throws OOM exception, which is hard to link > to the real issue of vm.max_map_count. > The problem happened when we tried to remove one node, all the other nodes in > cluster crashed. As each node was trying to load more local SSTable files for > streaming. > I would suggest to add a StartupCheck for max_map_count, at least it could > give a warning message to help the debug. > {code} > ERROR [STREAM-IN-] JVMStabilityInspector.java:140 - JVM state determined to > be unstable. Exiting forcefully due to: > java.lang.OutOfMemoryError: Map failed > at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_112] > at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937) > ~[na:1.8.0_112] > at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:173) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:70) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:58) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:96) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.CompressedSegmentedFile.(CompressedSegmentedFile.java:47) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:193) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:276) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.sstable.format.big.BigTableWriter.access$600(BigTableWriter.java:50) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:313) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.sstable.format.SSTableWriter.finish(SSTableWriter.java:213) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.finish(SimpleSSTableMultiWriter.java:56) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.streaming.StreamReceiveTask.received(StreamReceiveTask.java:109) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:599) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:482) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:296) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112] > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13008) Add vm.max_map_count StartupCheck
[ https://issues.apache.org/jira/browse/CASSANDRA-13008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15861194#comment-15861194 ] Stefania commented on CASSANDRA-13008: -- On second thought, I don't think we need this warning if the disk access mode is not mmap, can you [check|https://github.com/stef1927/cassandra/commit/7d4139fb50e8ef11b824138dd72bde036b9ee152#diff-a5df240149285ae528cdd3c41aa59360R240] [~jay.zhuang]? > Add vm.max_map_count StartupCheck > - > > Key: CASSANDRA-13008 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13008 > Project: Cassandra > Issue Type: Improvement > Components: Configuration >Reporter: Jay Zhuang >Assignee: Jay Zhuang >Priority: Minor > Fix For: 3.0.x > > Attachments: 13008-3.0.txt > > > It's recommended to set {{vm.max_map_count}} to 1048575 (CASSANDRA-3563) > When the max_map_count is low, it throws OOM exception, which is hard to link > to the real issue of vm.max_map_count. > The problem happened when we tried to remove one node, all the other nodes in > cluster crashed. As each node was trying to load more local SSTable files for > streaming. > I would suggest to add a StartupCheck for max_map_count, at least it could > give a warning message to help the debug. > {code} > ERROR [STREAM-IN-] JVMStabilityInspector.java:140 - JVM state determined to > be unstable. Exiting forcefully due to: > java.lang.OutOfMemoryError: Map failed > at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_112] > at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937) > ~[na:1.8.0_112] > at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:173) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:70) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:58) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:96) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.CompressedSegmentedFile.(CompressedSegmentedFile.java:47) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:193) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:276) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.sstable.format.big.BigTableWriter.access$600(BigTableWriter.java:50) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:313) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.sstable.format.SSTableWriter.finish(SSTableWriter.java:213) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.finish(SimpleSSTableMultiWriter.java:56) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.streaming.StreamReceiveTask.received(StreamReceiveTask.java:109) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:599) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:482) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:296) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112] > {code} -- This message was sent b
[jira] [Commented] (CASSANDRA-13008) Add vm.max_map_count StartupCheck
[ https://issues.apache.org/jira/browse/CASSANDRA-13008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15859854#comment-15859854 ] Stefania commented on CASSANDRA-13008: -- Thanks for the patch and sorry for the delay with the review. The patch LGTM - I've merely fixed the brackets positions and replaced {{FBUtilities.isWindows}} with {{FBUtilities.hasProcFS()}}. I've started running the tests on 3.0 only: ||3.0||3.11||trunk|| |[patch|https://github.com/stef1927/cassandra/tree/13008-3.0]|[patch|https://github.com/stef1927/cassandra/tree/13008-3.11]|[patch|https://github.com/stef1927/cassandra/tree/13008]| |[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-13008-3.0-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-13008-3.11-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-13008-testall/]| |[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-13008-3.0-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-13008-3.11-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-13008-dtest/]| The 3.0 patch applies cleanly to 3.11 and trunk. > Add vm.max_map_count StartupCheck > - > > Key: CASSANDRA-13008 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13008 > Project: Cassandra > Issue Type: Improvement > Components: Configuration >Reporter: Jay Zhuang >Assignee: Jay Zhuang >Priority: Minor > Fix For: 3.0.x > > Attachments: 13008-3.0.txt > > > It's recommended to set {{vm.max_map_count}} to 1048575 (CASSANDRA-3563) > When the max_map_count is low, it throws OOM exception, which is hard to link > to the real issue of vm.max_map_count. > The problem happened when we tried to remove one node, all the other nodes in > cluster crashed. As each node was trying to load more local SSTable files for > streaming. > I would suggest to add a StartupCheck for max_map_count, at least it could > give a warning message to help the debug. > {code} > ERROR [STREAM-IN-] JVMStabilityInspector.java:140 - JVM state determined to > be unstable. Exiting forcefully due to: > java.lang.OutOfMemoryError: Map failed > at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_112] > at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937) > ~[na:1.8.0_112] > at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:173) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:70) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:58) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:96) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.CompressedSegmentedFile.(CompressedSegmentedFile.java:47) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:193) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:276) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.sstable.format.big.BigTableWriter.access$600(BigTableWriter.java:50) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:313) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.sstable.format.SSTableWriter.finish(SSTableWriter.java:213) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io
[jira] [Commented] (CASSANDRA-13008) Add vm.max_map_count StartupCheck
[ https://issues.apache.org/jira/browse/CASSANDRA-13008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15854684#comment-15854684 ] Jay Zhuang commented on CASSANDRA-13008: Thanks. > Add vm.max_map_count StartupCheck > - > > Key: CASSANDRA-13008 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13008 > Project: Cassandra > Issue Type: Improvement > Components: Configuration >Reporter: Jay Zhuang >Assignee: Jay Zhuang >Priority: Minor > Fix For: 3.0.x > > Attachments: 13008-3.0.txt > > > It's recommended to set {{vm.max_map_count}} to 1048575 (CASSANDRA-3563) > When the max_map_count is low, it throws OOM exception, which is hard to link > to the real issue of vm.max_map_count. > The problem happened when we tried to remove one node, all the other nodes in > cluster crashed. As each node was trying to load more local SSTable files for > streaming. > I would suggest to add a StartupCheck for max_map_count, at least it could > give a warning message to help the debug. > {code} > ERROR [STREAM-IN-] JVMStabilityInspector.java:140 - JVM state determined to > be unstable. Exiting forcefully due to: > java.lang.OutOfMemoryError: Map failed > at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_112] > at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937) > ~[na:1.8.0_112] > at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:173) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:70) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:58) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:96) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.CompressedSegmentedFile.(CompressedSegmentedFile.java:47) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:193) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:276) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.sstable.format.big.BigTableWriter.access$600(BigTableWriter.java:50) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:313) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.sstable.format.SSTableWriter.finish(SSTableWriter.java:213) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.finish(SimpleSSTableMultiWriter.java:56) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.streaming.StreamReceiveTask.received(StreamReceiveTask.java:109) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:599) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:482) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:296) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112] > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13008) Add vm.max_map_count StartupCheck
[ https://issues.apache.org/jira/browse/CASSANDRA-13008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15853687#comment-15853687 ] Stefania commented on CASSANDRA-13008: -- Sure, I'll try to review it in the next few days. > Add vm.max_map_count StartupCheck > - > > Key: CASSANDRA-13008 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13008 > Project: Cassandra > Issue Type: Improvement > Components: Configuration >Reporter: Jay Zhuang >Assignee: Jay Zhuang >Priority: Minor > Fix For: 3.0.x > > Attachments: 13008-3.0.txt > > > It's recommended to set {{vm.max_map_count}} to 1048575 (CASSANDRA-3563) > When the max_map_count is low, it throws OOM exception, which is hard to link > to the real issue of vm.max_map_count. > The problem happened when we tried to remove one node, all the other nodes in > cluster crashed. As each node was trying to load more local SSTable files for > streaming. > I would suggest to add a StartupCheck for max_map_count, at least it could > give a warning message to help the debug. > {code} > ERROR [STREAM-IN-] JVMStabilityInspector.java:140 - JVM state determined to > be unstable. Exiting forcefully due to: > java.lang.OutOfMemoryError: Map failed > at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_112] > at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937) > ~[na:1.8.0_112] > at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:173) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:70) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:58) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:96) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.CompressedSegmentedFile.(CompressedSegmentedFile.java:47) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:193) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:276) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.sstable.format.big.BigTableWriter.access$600(BigTableWriter.java:50) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:313) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.sstable.format.SSTableWriter.finish(SSTableWriter.java:213) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.finish(SimpleSSTableMultiWriter.java:56) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.streaming.StreamReceiveTask.received(StreamReceiveTask.java:109) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:599) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:482) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:296) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112] > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13008) Add vm.max_map_count StartupCheck
[ https://issues.apache.org/jira/browse/CASSANDRA-13008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15853296#comment-15853296 ] Jay Zhuang commented on CASSANDRA-13008: [~Stefania] would you please review this patch? > Add vm.max_map_count StartupCheck > - > > Key: CASSANDRA-13008 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13008 > Project: Cassandra > Issue Type: Improvement > Components: Configuration >Reporter: Jay Zhuang >Assignee: Jay Zhuang >Priority: Minor > Fix For: 3.0.x > > Attachments: 13008-3.0.txt > > > It's recommended to set {{vm.max_map_count}} to 1048575 (CASSANDRA-3563) > When the max_map_count is low, it throws OOM exception, which is hard to link > to the real issue of vm.max_map_count. > The problem happened when we tried to remove one node, all the other nodes in > cluster crashed. As each node was trying to load more local SSTable files for > streaming. > I would suggest to add a StartupCheck for max_map_count, at least it could > give a warning message to help the debug. > {code} > ERROR [STREAM-IN-] JVMStabilityInspector.java:140 - JVM state determined to > be unstable. Exiting forcefully due to: > java.lang.OutOfMemoryError: Map failed > at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_112] > at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937) > ~[na:1.8.0_112] > at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:173) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:70) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:58) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:96) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.CompressedSegmentedFile.(CompressedSegmentedFile.java:47) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:193) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:276) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.sstable.format.big.BigTableWriter.access$600(BigTableWriter.java:50) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:313) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.sstable.format.SSTableWriter.finish(SSTableWriter.java:213) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.finish(SimpleSSTableMultiWriter.java:56) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.streaming.StreamReceiveTask.received(StreamReceiveTask.java:109) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:599) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:482) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:296) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112] > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (CASSANDRA-13008) Add vm.max_map_count StartupCheck
[ https://issues.apache.org/jira/browse/CASSANDRA-13008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15761849#comment-15761849 ] Jay Zhuang commented on CASSANDRA-13008: Could someone please review this change? > Add vm.max_map_count StartupCheck > - > > Key: CASSANDRA-13008 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13008 > Project: Cassandra > Issue Type: Improvement > Components: Configuration >Reporter: Jay Zhuang >Assignee: Jay Zhuang >Priority: Minor > Fix For: 3.0.x > > Attachments: 13008-3.0.txt > > > It's recommended to set {{vm.max_map_count}} to 1048575 (CASSANDRA-3563) > When the max_map_count is low, it throws OOM exception, which is hard to link > to the real issue of vm.max_map_count. > The problem happened when we tried to remove one node, all the other nodes in > cluster crashed. As each node was trying to load more local SSTable files for > streaming. > I would suggest to add a StartupCheck for max_map_count, at least it could > give a warning message to help the debug. > {code} > ERROR [STREAM-IN-] JVMStabilityInspector.java:140 - JVM state determined to > be unstable. Exiting forcefully due to: > java.lang.OutOfMemoryError: Map failed > at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_112] > at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937) > ~[na:1.8.0_112] > at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:173) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:70) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:58) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:96) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.CompressedSegmentedFile.(CompressedSegmentedFile.java:47) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:193) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:276) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.sstable.format.big.BigTableWriter.access$600(BigTableWriter.java:50) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:313) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.sstable.format.SSTableWriter.finish(SSTableWriter.java:213) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.finish(SimpleSSTableMultiWriter.java:56) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.streaming.StreamReceiveTask.received(StreamReceiveTask.java:109) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:599) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:482) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:296) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112] > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)