[jira] [Commented] (CASSANDRA-11710) Cassandra dies with OOM when running stress

2016-05-06 Thread T Jake Luciani (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15274283#comment-15274283
 ] 

T Jake Luciani commented on CASSANDRA-11710:


Will re-tag thanks

> Cassandra dies with OOM when running stress
> ---
>
> Key: CASSANDRA-11710
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11710
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Marcus Eriksson
>Assignee: Branimir Lambov
> Fix For: 3.6
>
>
> Running stress on trunk dies with OOM after about 3.5M ops:
> {code}
> ERROR [CompactionExecutor:1] 2016-05-04 15:01:31,231 
> JVMStabilityInspector.java:137 - JVM state determined to be unstable.  
> Exiting forcefully due to:
> java.lang.OutOfMemoryError: Direct buffer memory
> at java.nio.Bits.reserveMemory(Bits.java:693) ~[na:1.8.0_91]
> at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123) 
> ~[na:1.8.0_91]
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) 
> ~[na:1.8.0_91]
> at 
> org.apache.cassandra.utils.memory.BufferPool.allocateDirectAligned(BufferPool.java:519)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool.access$600(BufferPool.java:46) 
> ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool$GlobalPool.allocateMoreChunks(BufferPool.java:276)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool$GlobalPool.get(BufferPool.java:249)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool$LocalPool.addChunkFromGlobalPool(BufferPool.java:338)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool$LocalPool.get(BufferPool.java:381)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool.maybeTakeFromPool(BufferPool.java:142)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool.takeFromPool(BufferPool.java:114)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool.get(BufferPool.java:84) 
> ~[main/:na]
> at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:135) 
> ~[main/:na]
> at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:19) 
> ~[main/:na]
> at 
> com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalLoadingCache.lambda$new$0(BoundedLocalCache.java:2949)
>  ~[caffeine-2.2.6.jar:na]
> at 
> com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$15(BoundedLocalCache.java:1807)
>  ~[caffeine-2.2.6.jar:na]
> at 
> java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853) 
> ~[na:1.8.0_91]
> at 
> com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:1805)
>  ~[caffeine-2.2.6.jar:na]
> at 
> com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:1788)
>  ~[caffeine-2.2.6.jar:na]
> at 
> com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:97)
>  ~[caffeine-2.2.6.jar:na]
> at 
> com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:66)
>  ~[caffeine-2.2.6.jar:na]
> at 
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:215)
>  ~[main/:na]
> at 
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:193)
>  ~[main/:na]
> at 
> org.apache.cassandra.io.util.LimitingRebufferer.rebuffer(LimitingRebufferer.java:34)
>  ~[main/:na]
> at 
> org.apache.cassandra.io.util.RandomAccessReader.reBufferAt(RandomAccessReader.java:78)
>  ~[main/:na]
> at 
> org.apache.cassandra.io.util.RandomAccessReader.reBuffer(RandomAccessReader.java:72)
>  ~[main/:na]
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.read(RebufferingInputStream.java:88)
>  ~[main/:na]
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:66)
>  ~[main/:na]
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:60)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:400) 
> ~[main/:na]
> at 
> org.apache.cassandra.utils.ByteBufferUtil.readWithVIntLength(ByteBufferUtil.java:338)
>  ~[main/:na]
> at 
> org.apache.cassandra.db.marshal.AbstractType.readValue(AbstractType.java:414) 
> ~[main/:na]
> at 
> org.apache.cassandra.db.rows.Cell$Serializer.deserialize(Cell.java:243) 
> ~[main/:na]
> at 
> org.apache.cassandra.db.rows.UnfilteredSerializer.readSimpleColumn(UnfilteredSerializer.java:473)
>  ~[main/:na]
> at 
> org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.ja

[jira] [Commented] (CASSANDRA-11710) Cassandra dies with OOM when running stress

2016-05-06 Thread Sylvain Lebresne (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15274226#comment-15274226
 ] 

Sylvain Lebresne commented on CASSANDRA-11710:
--

[~tjake] I believe you may want to check what's above (I think we need to 
change the cassandra-3.6-tentative flag)

> Cassandra dies with OOM when running stress
> ---
>
> Key: CASSANDRA-11710
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11710
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Marcus Eriksson
>Assignee: Branimir Lambov
> Fix For: 3.6
>
>
> Running stress on trunk dies with OOM after about 3.5M ops:
> {code}
> ERROR [CompactionExecutor:1] 2016-05-04 15:01:31,231 
> JVMStabilityInspector.java:137 - JVM state determined to be unstable.  
> Exiting forcefully due to:
> java.lang.OutOfMemoryError: Direct buffer memory
> at java.nio.Bits.reserveMemory(Bits.java:693) ~[na:1.8.0_91]
> at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123) 
> ~[na:1.8.0_91]
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) 
> ~[na:1.8.0_91]
> at 
> org.apache.cassandra.utils.memory.BufferPool.allocateDirectAligned(BufferPool.java:519)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool.access$600(BufferPool.java:46) 
> ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool$GlobalPool.allocateMoreChunks(BufferPool.java:276)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool$GlobalPool.get(BufferPool.java:249)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool$LocalPool.addChunkFromGlobalPool(BufferPool.java:338)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool$LocalPool.get(BufferPool.java:381)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool.maybeTakeFromPool(BufferPool.java:142)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool.takeFromPool(BufferPool.java:114)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool.get(BufferPool.java:84) 
> ~[main/:na]
> at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:135) 
> ~[main/:na]
> at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:19) 
> ~[main/:na]
> at 
> com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalLoadingCache.lambda$new$0(BoundedLocalCache.java:2949)
>  ~[caffeine-2.2.6.jar:na]
> at 
> com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$15(BoundedLocalCache.java:1807)
>  ~[caffeine-2.2.6.jar:na]
> at 
> java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853) 
> ~[na:1.8.0_91]
> at 
> com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:1805)
>  ~[caffeine-2.2.6.jar:na]
> at 
> com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:1788)
>  ~[caffeine-2.2.6.jar:na]
> at 
> com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:97)
>  ~[caffeine-2.2.6.jar:na]
> at 
> com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:66)
>  ~[caffeine-2.2.6.jar:na]
> at 
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:215)
>  ~[main/:na]
> at 
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:193)
>  ~[main/:na]
> at 
> org.apache.cassandra.io.util.LimitingRebufferer.rebuffer(LimitingRebufferer.java:34)
>  ~[main/:na]
> at 
> org.apache.cassandra.io.util.RandomAccessReader.reBufferAt(RandomAccessReader.java:78)
>  ~[main/:na]
> at 
> org.apache.cassandra.io.util.RandomAccessReader.reBuffer(RandomAccessReader.java:72)
>  ~[main/:na]
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.read(RebufferingInputStream.java:88)
>  ~[main/:na]
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:66)
>  ~[main/:na]
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:60)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:400) 
> ~[main/:na]
> at 
> org.apache.cassandra.utils.ByteBufferUtil.readWithVIntLength(ByteBufferUtil.java:338)
>  ~[main/:na]
> at 
> org.apache.cassandra.db.marshal.AbstractType.readValue(AbstractType.java:414) 
> ~[main/:na]
> at 
> org.apache.cassandra.db.rows.Cell$Serializer.deserialize(Cell.java:243) 
> ~[main/:na]
> at 
> org.apache.cassandra.db.rows.UnfilteredSerializer.readSimpleColumn(UnfilteredSerializer.java:473)
>  ~[main/:na]
>

[jira] [Commented] (CASSANDRA-11710) Cassandra dies with OOM when running stress

2016-05-06 Thread Branimir Lambov (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15274219#comment-15274219
 ] 

Branimir Lambov commented on CASSANDRA-11710:
-

Yes, it is. If we don't include it, nodes spawned with default ccm settings 
will oom in 3.6.

> Cassandra dies with OOM when running stress
> ---
>
> Key: CASSANDRA-11710
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11710
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Marcus Eriksson
>Assignee: Branimir Lambov
> Fix For: 3.6
>
>
> Running stress on trunk dies with OOM after about 3.5M ops:
> {code}
> ERROR [CompactionExecutor:1] 2016-05-04 15:01:31,231 
> JVMStabilityInspector.java:137 - JVM state determined to be unstable.  
> Exiting forcefully due to:
> java.lang.OutOfMemoryError: Direct buffer memory
> at java.nio.Bits.reserveMemory(Bits.java:693) ~[na:1.8.0_91]
> at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123) 
> ~[na:1.8.0_91]
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) 
> ~[na:1.8.0_91]
> at 
> org.apache.cassandra.utils.memory.BufferPool.allocateDirectAligned(BufferPool.java:519)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool.access$600(BufferPool.java:46) 
> ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool$GlobalPool.allocateMoreChunks(BufferPool.java:276)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool$GlobalPool.get(BufferPool.java:249)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool$LocalPool.addChunkFromGlobalPool(BufferPool.java:338)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool$LocalPool.get(BufferPool.java:381)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool.maybeTakeFromPool(BufferPool.java:142)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool.takeFromPool(BufferPool.java:114)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool.get(BufferPool.java:84) 
> ~[main/:na]
> at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:135) 
> ~[main/:na]
> at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:19) 
> ~[main/:na]
> at 
> com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalLoadingCache.lambda$new$0(BoundedLocalCache.java:2949)
>  ~[caffeine-2.2.6.jar:na]
> at 
> com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$15(BoundedLocalCache.java:1807)
>  ~[caffeine-2.2.6.jar:na]
> at 
> java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853) 
> ~[na:1.8.0_91]
> at 
> com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:1805)
>  ~[caffeine-2.2.6.jar:na]
> at 
> com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:1788)
>  ~[caffeine-2.2.6.jar:na]
> at 
> com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:97)
>  ~[caffeine-2.2.6.jar:na]
> at 
> com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:66)
>  ~[caffeine-2.2.6.jar:na]
> at 
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:215)
>  ~[main/:na]
> at 
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:193)
>  ~[main/:na]
> at 
> org.apache.cassandra.io.util.LimitingRebufferer.rebuffer(LimitingRebufferer.java:34)
>  ~[main/:na]
> at 
> org.apache.cassandra.io.util.RandomAccessReader.reBufferAt(RandomAccessReader.java:78)
>  ~[main/:na]
> at 
> org.apache.cassandra.io.util.RandomAccessReader.reBuffer(RandomAccessReader.java:72)
>  ~[main/:na]
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.read(RebufferingInputStream.java:88)
>  ~[main/:na]
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:66)
>  ~[main/:na]
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:60)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:400) 
> ~[main/:na]
> at 
> org.apache.cassandra.utils.ByteBufferUtil.readWithVIntLength(ByteBufferUtil.java:338)
>  ~[main/:na]
> at 
> org.apache.cassandra.db.marshal.AbstractType.readValue(AbstractType.java:414) 
> ~[main/:na]
> at 
> org.apache.cassandra.db.rows.Cell$Serializer.deserialize(Cell.java:243) 
> ~[main/:na]
> at 
> org.apache.cassandra.db.rows.UnfilteredSerializer.readSimpleColumn(UnfilteredSerializer.java:473)
>  ~[main/:na]
> at 
> org.apache.cass

[jira] [Commented] (CASSANDRA-11710) Cassandra dies with OOM when running stress

2016-05-06 Thread Sylvain Lebresne (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15274180#comment-15274180
 ] 

Sylvain Lebresne commented on CASSANDRA-11710:
--

Is this a problem for the currently frozen 3.6? That is, should we make sure 
this get included or we have a regression?

> Cassandra dies with OOM when running stress
> ---
>
> Key: CASSANDRA-11710
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11710
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Marcus Eriksson
>Assignee: Branimir Lambov
> Fix For: 3.6
>
>
> Running stress on trunk dies with OOM after about 3.5M ops:
> {code}
> ERROR [CompactionExecutor:1] 2016-05-04 15:01:31,231 
> JVMStabilityInspector.java:137 - JVM state determined to be unstable.  
> Exiting forcefully due to:
> java.lang.OutOfMemoryError: Direct buffer memory
> at java.nio.Bits.reserveMemory(Bits.java:693) ~[na:1.8.0_91]
> at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123) 
> ~[na:1.8.0_91]
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) 
> ~[na:1.8.0_91]
> at 
> org.apache.cassandra.utils.memory.BufferPool.allocateDirectAligned(BufferPool.java:519)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool.access$600(BufferPool.java:46) 
> ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool$GlobalPool.allocateMoreChunks(BufferPool.java:276)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool$GlobalPool.get(BufferPool.java:249)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool$LocalPool.addChunkFromGlobalPool(BufferPool.java:338)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool$LocalPool.get(BufferPool.java:381)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool.maybeTakeFromPool(BufferPool.java:142)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool.takeFromPool(BufferPool.java:114)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool.get(BufferPool.java:84) 
> ~[main/:na]
> at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:135) 
> ~[main/:na]
> at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:19) 
> ~[main/:na]
> at 
> com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalLoadingCache.lambda$new$0(BoundedLocalCache.java:2949)
>  ~[caffeine-2.2.6.jar:na]
> at 
> com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$15(BoundedLocalCache.java:1807)
>  ~[caffeine-2.2.6.jar:na]
> at 
> java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853) 
> ~[na:1.8.0_91]
> at 
> com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:1805)
>  ~[caffeine-2.2.6.jar:na]
> at 
> com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:1788)
>  ~[caffeine-2.2.6.jar:na]
> at 
> com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:97)
>  ~[caffeine-2.2.6.jar:na]
> at 
> com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:66)
>  ~[caffeine-2.2.6.jar:na]
> at 
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:215)
>  ~[main/:na]
> at 
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:193)
>  ~[main/:na]
> at 
> org.apache.cassandra.io.util.LimitingRebufferer.rebuffer(LimitingRebufferer.java:34)
>  ~[main/:na]
> at 
> org.apache.cassandra.io.util.RandomAccessReader.reBufferAt(RandomAccessReader.java:78)
>  ~[main/:na]
> at 
> org.apache.cassandra.io.util.RandomAccessReader.reBuffer(RandomAccessReader.java:72)
>  ~[main/:na]
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.read(RebufferingInputStream.java:88)
>  ~[main/:na]
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:66)
>  ~[main/:na]
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:60)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:400) 
> ~[main/:na]
> at 
> org.apache.cassandra.utils.ByteBufferUtil.readWithVIntLength(ByteBufferUtil.java:338)
>  ~[main/:na]
> at 
> org.apache.cassandra.db.marshal.AbstractType.readValue(AbstractType.java:414) 
> ~[main/:na]
> at 
> org.apache.cassandra.db.rows.Cell$Serializer.deserialize(Cell.java:243) 
> ~[main/:na]
> at 
> org.apache.cassandra.db.rows.UnfilteredSerializer.readSimpleColumn(UnfilteredSerializer.java:473)
>  ~[main/:na]
>

[jira] [Commented] (CASSANDRA-11710) Cassandra dies with OOM when running stress

2016-05-05 Thread Branimir Lambov (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15272073#comment-15272073
 ] 

Branimir Lambov commented on CASSANDRA-11710:
-

Patch here:
|[code|https://github.com/blambov/cassandra/tree/11710]|[utest|http://cassci.datastax.com/job/blambov-11710-testall/]|[dtest|http://cassci.datastax.com/job/blambov-11710-dtest/]|

Fixes incorrect default value for {{file_cache_size_in_mb}} (was always 512MB 
while {{cassandra.yaml}} specifies 1/4 Xmx value), now it should not cause 
problems when max direct memory size is left at default (= Xmx). Also attempts 
to continue after allocation failures from the buffer pool -- such failures are 
not fatal as we can treat them in the same manner as reaching the size limit.


> Cassandra dies with OOM when running stress
> ---
>
> Key: CASSANDRA-11710
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11710
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Marcus Eriksson
>Assignee: Branimir Lambov
>
> Running stress on trunk dies with OOM after about 3.5M ops:
> {code}
> ERROR [CompactionExecutor:1] 2016-05-04 15:01:31,231 
> JVMStabilityInspector.java:137 - JVM state determined to be unstable.  
> Exiting forcefully due to:
> java.lang.OutOfMemoryError: Direct buffer memory
> at java.nio.Bits.reserveMemory(Bits.java:693) ~[na:1.8.0_91]
> at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123) 
> ~[na:1.8.0_91]
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) 
> ~[na:1.8.0_91]
> at 
> org.apache.cassandra.utils.memory.BufferPool.allocateDirectAligned(BufferPool.java:519)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool.access$600(BufferPool.java:46) 
> ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool$GlobalPool.allocateMoreChunks(BufferPool.java:276)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool$GlobalPool.get(BufferPool.java:249)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool$LocalPool.addChunkFromGlobalPool(BufferPool.java:338)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool$LocalPool.get(BufferPool.java:381)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool.maybeTakeFromPool(BufferPool.java:142)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool.takeFromPool(BufferPool.java:114)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.memory.BufferPool.get(BufferPool.java:84) 
> ~[main/:na]
> at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:135) 
> ~[main/:na]
> at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:19) 
> ~[main/:na]
> at 
> com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalLoadingCache.lambda$new$0(BoundedLocalCache.java:2949)
>  ~[caffeine-2.2.6.jar:na]
> at 
> com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$15(BoundedLocalCache.java:1807)
>  ~[caffeine-2.2.6.jar:na]
> at 
> java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853) 
> ~[na:1.8.0_91]
> at 
> com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:1805)
>  ~[caffeine-2.2.6.jar:na]
> at 
> com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:1788)
>  ~[caffeine-2.2.6.jar:na]
> at 
> com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:97)
>  ~[caffeine-2.2.6.jar:na]
> at 
> com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:66)
>  ~[caffeine-2.2.6.jar:na]
> at 
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:215)
>  ~[main/:na]
> at 
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:193)
>  ~[main/:na]
> at 
> org.apache.cassandra.io.util.LimitingRebufferer.rebuffer(LimitingRebufferer.java:34)
>  ~[main/:na]
> at 
> org.apache.cassandra.io.util.RandomAccessReader.reBufferAt(RandomAccessReader.java:78)
>  ~[main/:na]
> at 
> org.apache.cassandra.io.util.RandomAccessReader.reBuffer(RandomAccessReader.java:72)
>  ~[main/:na]
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.read(RebufferingInputStream.java:88)
>  ~[main/:na]
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:66)
>  ~[main/:na]
> at 
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:60)
>  ~[main/:na]
> at 
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:400) 
> ~[main/:na]
> at 
> org.apach