[jira] [Commented] (CASSANDRA-6107) CQL3 Batch statement memory leak

2013-11-03 Thread Michael Oczkowski (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-6107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13812415#comment-13812415
 ] 

Michael Oczkowski commented on CASSANDRA-6107:
--

This change appears to break code that uses EmbeddedCassandraService and 
PreparedStatements in version 1.2.11.  Please see CASSANDRA-6293 for details.

 CQL3 Batch statement memory leak
 

 Key: CASSANDRA-6107
 URL: https://issues.apache.org/jira/browse/CASSANDRA-6107
 Project: Cassandra
  Issue Type: Bug
  Components: API, Core
 Environment: - CASS version: 1.2.8 or 2.0.1, same issue seen in both
 - Running on OSX MacbookPro
 - Sun JVM 1.7
 - Single local cassandra node
 - both CMS and G1 GC used
 - we are using the cass-JDBC driver to submit our batches
Reporter: Constance Eustace
Assignee: Lyuben Todorov
Priority: Minor
 Fix For: 1.2.11

 Attachments: 6107-v4.txt, 6107.patch, 6107_v2.patch, 6107_v3.patch, 
 Screen Shot 2013-10-03 at 17.59.37.png


 We are doing large volume insert/update tests on a CASS via CQL3. 
 Using 4GB heap, after roughly 750,000 updates create/update 75,000 row keys, 
 we run out of heap, and it never dissipates, and we begin getting this 
 infamous error which many people seem to be encountering:
 WARN [ScheduledTasks:1] 2013-09-26 16:17:10,752 GCInspector.java (line 142) 
 Heap is 0.9383457210434385 full.  You may need to reduce memtable and/or 
 cache sizes.  Cassandra will now flush up to the two largest memtables to 
 free up memory.  Adjust flush_largest_memtables_at threshold in 
 cassandra.yaml if you don't want Cassandra to do this automatically
  INFO [ScheduledTasks:1] 2013-09-26 16:17:10,753 StorageService.java (line 
 3614) Unable to reduce heap usage since there are no dirty column families
 8 and 12 GB heaps appear to delay the problem by roughly proportionate 
 amounts of 75,000 - 100,000 rowkeys per 4GB. Each run of 50,000 row key 
 creations sees the heap grow and never shrink again. 
 We have attempted to no effect:
 - removing all secondary indexes to see if that alleviates overuse of bloom 
 filters 
 - adjusted parameters for compaction throughput
 - adjusted memtable flush thresholds and other parameters 
 By examining heapdumps, it seems apparent that the problem is perpetual 
 retention of CQL3 BATCH statements. We have even tried dropping the keyspaces 
 after the updates and the CQL3 statement are still visible in the heapdump, 
 and after many many many CMS GC runs. G1 also showed this issue.
 The 750,000 statements are broken into batches of roughly 200 statements.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Created] (CASSANDRA-6293) CASSANDRA-6107 causes EmbeddedCassandraService to fail

2013-11-03 Thread Michael Oczkowski (JIRA) 
Michael Oczkowski created CASSANDRA-6293:


 Summary: CASSANDRA-6107 causes EmbeddedCassandraService to fail
 Key: CASSANDRA-6293
 URL: https://issues.apache.org/jira/browse/CASSANDRA-6293
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Reporter: Michael Oczkowski


Starting with 1.2.11 calling statement.prepare() when running 
EmbeddedCassandraService causes an exception due to missing Java agent.  Prior 
to 1.2.11 this never caused an exception, though there were warnings in the 
logs about the missing agent.

2013-11-01 11:14:06,508 ERROR Native-Transport-Requests:109 
org.apache.cassandra.transport.messages.ErrorMessage - Unexpected exception 
during request
java.lang.IllegalStateException: Instrumentation is not set; Jamm must be set 
as -javaagent
at org.github.jamm.MemoryMeter.measure(MemoryMeter.java:70)
at org.github.jamm.MemoryMeter.measureDeep(MemoryMeter.java:102)
at 
org.apache.cassandra.cql3.QueryProcessor.storePreparedStatement(QueryProcessor.java:214)
at 
org.apache.cassandra.cql3.QueryProcessor.prepare(QueryProcessor.java:202)
at 
org.apache.cassandra.transport.messages.PrepareMessage.execute(PrepareMessage.java:77)
at 
org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:287)
at 
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at 
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at 
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at 
org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
at 
org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)




--
This message was sent by Atlassian JIRA
(v6.1#6144)