[jira] [Updated] (CASSANDRA-13008) Add vm.max_map_count StartupCheck

2017-02-13 Thread Stefania (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefania updated CASSANDRA-13008:
-
   Resolution: Fixed
Fix Version/s: (was: 3.0.x)
   3.11.0
   3.0.11
   Status: Resolved  (was: Patch Available)

Committed to 3.0 as 3a570d744e938077784d8a24b1334c7b567543ed and merged upwards.

> Add vm.max_map_count StartupCheck
> -
>
> Key: CASSANDRA-13008
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13008
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Configuration
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>Priority: Minor
> Fix For: 3.0.11, 3.11.0
>
> Attachments: 13008-3.0.txt
>
>
> It's recommended to set {{vm.max_map_count}} to 1048575 (CASSANDRA-3563)
> When the max_map_count is low, it throws OOM exception, which is hard to link 
> to the real issue of vm.max_map_count.
> The problem happened when we tried to remove one node, all the other nodes in 
> cluster crashed. As each node was trying to load more local SSTable files for 
> streaming.
> I would suggest to add a StartupCheck for max_map_count, at least it could 
> give a warning message to help the debug.
> {code}
> ERROR [STREAM-IN-] JVMStabilityInspector.java:140 - JVM state determined to 
> be unstable.  Exiting forcefully due to:
> java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_112]
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937) 
> ~[na:1.8.0_112]
> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152) 
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:173)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:70) 
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:58) 
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:96) 
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.CompressedSegmentedFile.(CompressedSegmentedFile.java:47)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:193)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:276)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.access$600(BigTableWriter.java:50)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:313)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.sstable.format.SSTableWriter.finish(SSTableWriter.java:213)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.finish(SimpleSSTableMultiWriter.java:56)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.streaming.StreamReceiveTask.received(StreamReceiveTask.java:109)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:599) 
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:482)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:296)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-13008) Add vm.max_map_count StartupCheck

2017-02-06 Thread Stefania (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefania updated CASSANDRA-13008:
-
Reviewer: Stefania

> Add vm.max_map_count StartupCheck
> -
>
> Key: CASSANDRA-13008
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13008
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Configuration
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>Priority: Minor
> Fix For: 3.0.x
>
> Attachments: 13008-3.0.txt
>
>
> It's recommended to set {{vm.max_map_count}} to 1048575 (CASSANDRA-3563)
> When the max_map_count is low, it throws OOM exception, which is hard to link 
> to the real issue of vm.max_map_count.
> The problem happened when we tried to remove one node, all the other nodes in 
> cluster crashed. As each node was trying to load more local SSTable files for 
> streaming.
> I would suggest to add a StartupCheck for max_map_count, at least it could 
> give a warning message to help the debug.
> {code}
> ERROR [STREAM-IN-] JVMStabilityInspector.java:140 - JVM state determined to 
> be unstable.  Exiting forcefully due to:
> java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_112]
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937) 
> ~[na:1.8.0_112]
> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152) 
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:173)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:70) 
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:58) 
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:96) 
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.CompressedSegmentedFile.(CompressedSegmentedFile.java:47)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:193)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:276)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.access$600(BigTableWriter.java:50)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:313)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.sstable.format.SSTableWriter.finish(SSTableWriter.java:213)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.finish(SimpleSSTableMultiWriter.java:56)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.streaming.StreamReceiveTask.received(StreamReceiveTask.java:109)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:599) 
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:482)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:296)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-13008) Add vm.max_map_count StartupCheck

2016-12-06 Thread Jay Zhuang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jay Zhuang updated CASSANDRA-13008:
---
Summary: Add vm.max_map_count StartupCheck  (was: Add vm.max_map_count 
startup StartupCheck)

> Add vm.max_map_count StartupCheck
> -
>
> Key: CASSANDRA-13008
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13008
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Configuration
>Reporter: Jay Zhuang
>Priority: Minor
> Fix For: 3.0.x
>
> Attachments: 13008-3.0.txt
>
>
> It's recommended to set {{vm.max_map_count}} to 1048575 (CASSANDRA-3563)
> When the max_map_count is low, it throws OOM exception, which is hard to link 
> to the real issue of vm.max_map_count.
> The problem happened when we tried to remove one node, all the other nodes in 
> cluster crashed. As each node was trying to load more local SSTable files for 
> streaming.
> I would suggest to add a StartupCheck for max_map_count, at least it could 
> give a warning message to help the debug.
> {code}
> ERROR [STREAM-IN-] JVMStabilityInspector.java:140 - JVM state determined to 
> be unstable.  Exiting forcefully due to:
> java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_112]
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937) 
> ~[na:1.8.0_112]
> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152) 
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:173)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:70) 
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:58) 
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:96) 
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.CompressedSegmentedFile.(CompressedSegmentedFile.java:47)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:193)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:276)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.access$600(BigTableWriter.java:50)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:313)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.sstable.format.SSTableWriter.finish(SSTableWriter.java:213)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.finish(SimpleSSTableMultiWriter.java:56)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.streaming.StreamReceiveTask.received(StreamReceiveTask.java:109)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:599) 
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:482)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:296)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)