[jira] [Commented] (CASSANDRA-8066) High Heap Consumption due to high number of SSTableReader

2014-10-13 Thread Benoit Lacelle (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-8066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14169722#comment-14169722
 ] 

Benoit Lacelle commented on CASSANDRA-8066:
---

I have a single node. The schema holds around 8 tables. A few of them are 
counter tables.

I regularly run nearly-full scan on on a not-counter table. Else, it is mainly 
an equivalent of reads and writes on 2 given tables. These tables have a few 
million of rows.

Do you need more details?

 High Heap Consumption due to high number of SSTableReader
 -

 Key: CASSANDRA-8066
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8066
 Project: Cassandra
  Issue Type: Bug
  Components: Core
 Environment: Cassandra 2.1.0
Reporter: Benoit Lacelle
Assignee: T Jake Luciani
 Fix For: 2.1.1


 Given a workload with quite a lot of reads, I recently encountered high heap 
 memory consumption. Given 2GB of Heap, it appears I have 750.000+ tasks in 
 SSTableReader.syncExecutor, consuming more than 1.2GB. These tasks have type 
 SSTableReader$5, which I guess corresponds to :
 {code}
 readMeterSyncFuture = syncExecutor.scheduleAtFixedRate(new Runnable()
 {
 public void run()
 {
 if (!isCompacted.get())
 {
 meterSyncThrottle.acquire();
 SystemKeyspace.persistSSTableReadMeter(desc.ksname, desc.cfname, 
 desc.generation, readMeter);
 }
 }
 }, 1, 5, TimeUnit.MINUTES);
 {code}
 I do not have have to the environment right now, but I could provide a 
 threaddump later if necessary.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8066) High Heap Consumption due to high number of SSTableReader

2014-10-13 Thread Benoit Lacelle (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-8066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14169775#comment-14169775
 ] 

Benoit Lacelle commented on CASSANDRA-8066:
---

Here is the output for nodetool cfstats

{code}
Keyspace: prod_7
Read Count: 1424130
Read Latency: 3.122707294980093 ms.
Write Count: 8808265
Write Latency: 0.03234213866181365 ms.
Pending Flushes: 0
Table: alerts
SSTable count: 2
Space used (live), bytes: 10237
Space used (total), bytes: 10237
Space used by snapshots (total), bytes: 0
SSTable Compression Ratio: 0.6354418105671432
Memtable cell count: 0
Memtable data size, bytes: 0
Memtable switch count: 0
Local read count: 0
Local read latency: NaN ms
Local write count: 0
Local write latency: NaN ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.0
Bloom filter space used, bytes: 32
Compacted partition minimum bytes: 259
Compacted partition maximum bytes: 372
Compacted partition mean bytes: 341
Average live cells per slice (last five minutes): 0.0
Average tombstones per slice (last five minutes): 0.0

Table: details
SSTable count: 103
Space used (live), bytes: 578266489
Space used (total), bytes: 578266489
Space used by snapshots (total), bytes: 0
SSTable Compression Ratio: 0.724988344517149
Memtable cell count: 67212
Memtable data size, bytes: 18468770
Memtable switch count: 23
Local read count: 6
Local read latency: 10.742 ms
Local write count: 2036971
Local write latency: 0.017 ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.0
Bloom filter space used, bytes: 5136
Compacted partition minimum bytes: 87
Compacted partition maximum bytes: 129557750
Compacted partition mean bytes: 2595076
Average live cells per slice (last five minutes): 
0.
Average tombstones per slice (last five minutes): 0.0

Table: domains
SSTable count: 21
Space used (live), bytes: 122407
Space used (total), bytes: 122407
Space used by snapshots (total), bytes: 0
SSTable Compression Ratio: 0.5848437821906775
Memtable cell count: 238238
Memtable data size, bytes: 2793
Memtable switch count: 1
Local read count: 60281
Local read latency: 0.162 ms
Local write count: 402903
Local write latency: 0.012 ms
Pending flushes: 0
Bloom filter false positives: 25
Bloom filter false ratio: 0.08929
Bloom filter space used, bytes: 664
Compacted partition minimum bytes: 87
Compacted partition maximum bytes: 372
Compacted partition mean bytes: 171
Average live cells per slice (last five minutes): 
0.9985401459854014
Average tombstones per slice (last five minutes): 0.0

Table: domains_statistics
SSTable count: 7
Space used (live), bytes: 302413
Space used (total), bytes: 302413
Space used by snapshots (total), bytes: 0
SSTable Compression Ratio: 0.6144160676068052
Memtable cell count: 849893
Memtable data size, bytes: 42569
Memtable switch count: 1
Local read count: 60511
Local read latency: 0.141 ms
Local write count: 1055892
Local write latency: 0.013 ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.0
Bloom filter space used, bytes: 416
Compacted partition minimum bytes: 87
Compacted partition maximum bytes: 24601
Compacted partition mean bytes: 3795
Average live cells per slice (last five minutes): 
0.9894160583941606
Average tombstones per slice (last five minutes): 0.0

Table: latest_urls
SSTable count: 3
Space used (live), bytes: 10239518

[jira] [Created] (CASSANDRA-8066) High Heap Consumption due to high number of SSTableReader

2014-10-07 Thread Benoit Lacelle (JIRA) 
Benoit Lacelle created CASSANDRA-8066:
-

 Summary: High Heap Consumption due to high number of SSTableReader
 Key: CASSANDRA-8066
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8066
 Project: Cassandra
  Issue Type: Bug
  Components: Core
 Environment: Cassandra 2.1.0
Reporter: Benoit Lacelle


Given a workload with quite a lot of reads, I recently encountered high heap 
memory consumption. Given 2GB of Heap, it appears I have 750.000+ tasks in 
SSTableReader.syncExecutor, consuming more than 1.2GB. These tasks have type 
SSTableReader$5, which I guess corresponds to :

{code}
readMeterSyncFuture = syncExecutor.scheduleAtFixedRate(new Runnable()
{
public void run()
{
if (!isCompacted.get())
{
meterSyncThrottle.acquire();
SystemKeyspace.persistSSTableReadMeter(desc.ksname, desc.cfname, 
desc.generation, readMeter);
}
}
}, 1, 5, TimeUnit.MINUTES);
{code}

I do not have have to the environment right now, but I could provide a 
threaddump later if necessary.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (CASSANDRA-7823) IllegalStateException with DROP KEYSPACE IF EXISTS XXX

2014-09-11 Thread Benoit Lacelle (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-7823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benoit Lacelle resolved CASSANDRA-7823.
---
Resolution: Won't Fix

It was related to a large number of removal/creation of keyspaces with 
different names (for tests purposes) in a small time window.

 IllegalStateException with DROP KEYSPACE IF EXISTS XXX
 --

 Key: CASSANDRA-7823
 URL: https://issues.apache.org/jira/browse/CASSANDRA-7823
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Reporter: Benoit Lacelle

 It happens on Cassandra 2.0.9 (production server) but not in my local 
 envirnment (2.0.6).
 When executing DROP KEYSPACE IF EXISTS XXX on an unexisting keyspace, I get 
 :
 In CQLSH:
 {code}
 Traceback (most recent call last):
   File /usr/bin/cqlsh, line 930, in perform_statement_untraced
 self.cursor.execute(statement, decoder=decoder)
   File 
 /usr/share/cassandra/lib/cql-internal-only-1.4.1.zip/cql-1.4.1/cql/cursor.py,
  line 80, in execute
 response = self.get_response(prepared_q, cl)
   File 
 /usr/share/cassandra/lib/cql-internal-only-1.4.1.zip/cql-1.4.1/cql/thrifteries.py,
  line 77, in get_response
 return self.handle_cql_execution_errors(doquery, compressed_q, compress, 
 cl)
   File 
 /usr/share/cassandra/lib/cql-internal-only-1.4.1.zip/cql-1.4.1/cql/thrifteries.py,
  line 96, in handle_cql_execution_errors
 return executor(*args, **kwargs)
   File 
 /usr/share/cassandra/lib/cql-internal-only-1.4.1.zip/cql-1.4.1/cql/cassandra/Cassandra.py,
  line 1830, in execute_cql3_query
 self.send_execute_cql3_query(query, compression, consistency)
   File 
 /usr/share/cassandra/lib/cql-internal-only-1.4.1.zip/cql-1.4.1/cql/cassandra/Cassandra.py,
  line 1841, in send_execute_cql3_query
 self._oprot.trans.flush()
   File 
 /usr/share/cassandra/lib/thrift-python-internal-only-0.9.1.zip/thrift/transport/TTransport.py,
  line 292, in flush
 self.__trans.write(buf)
   File 
 /usr/share/cassandra/lib/thrift-python-internal-only-0.9.1.zip/thrift/transport/TSocket.py,
  line 128, in write
 plus = self.handle.send(buff)
 error: [Errno 32] Broken pipe
 {code}
 In system.log:
 {code}
 ERROR [Thrift:3] 2014-08-24 22:53:02,242 CustomTThreadPoolServer.java (line 
 219) Error occurred during processing of message.
 java.lang.RuntimeException: java.util.concurrent.ExecutionException: 
 java.lang.IllegalStateException: One row required, 0 found
 at 
 org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:413)
 at 
 org.apache.cassandra.service.MigrationManager.announce(MigrationManager.java:285)
 at 
 org.apache.cassandra.service.MigrationManager.announceKeyspaceDrop(MigrationManager.java:259)
 at 
 org.apache.cassandra.cql3.statements.DropKeyspaceStatement.announceMigration(DropKeyspaceStatement.java:62)
 at 
 org.apache.cassandra.cql3.statements.SchemaAlteringStatement.execute(SchemaAlteringStatement.java:71)
 at 
 org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:158)
 at 
 org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:175)
 at 
 org.apache.cassandra.thrift.CassandraServer.execute_cql3_query(CassandraServer.java:1958)
 at 
 org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4486)
 at 
 org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4470)
 at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
 at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
 at 
 org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:201)
 at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 at java.lang.Thread.run(Thread.java:744)
 Caused by: java.util.concurrent.ExecutionException: 
 java.lang.IllegalStateException: One row required, 0 found
 at java.util.concurrent.FutureTask.report(FutureTask.java:122)
 at java.util.concurrent.FutureTask.get(FutureTask.java:188)
 at 
 org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:409)
 ... 15 more
 Caused by: java.lang.IllegalStateException: One row required, 0 found
 at 
 org.apache.cassandra.cql3.UntypedResultSet.one(UntypedResultSet.java:53)
 at 
 org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:263)
 at 
 org.apache.cassandra.db.DefsTables.mergeKeyspaces(DefsTables.java:227)
 at org.apache.cassandra.db.DefsTables.mergeSchema(DefsTables.java:182)
 at

[jira] [Created] (CASSANDRA-7823) IllegalStateException with DROP KEYSPACE IF EXISTS XXX

2014-08-24 Thread Benoit Lacelle (JIRA) 
Benoit Lacelle created CASSANDRA-7823:
-

 Summary: IllegalStateException with DROP KEYSPACE IF EXISTS XXX
 Key: CASSANDRA-7823
 URL: https://issues.apache.org/jira/browse/CASSANDRA-7823
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Reporter: Benoit Lacelle


It happens on Cassandra 2.0.9 (production server) but not in my local 
envirnment (2.0.6).

When executing DROP KEYSPACE IF EXISTS XXX on an unexisting keyspace, I get :

In CQLSH:
{code}
Traceback (most recent call last):
  File /usr/bin/cqlsh, line 930, in perform_statement_untraced
self.cursor.execute(statement, decoder=decoder)
  File 
/usr/share/cassandra/lib/cql-internal-only-1.4.1.zip/cql-1.4.1/cql/cursor.py, 
line 80, in execute
response = self.get_response(prepared_q, cl)
  File 
/usr/share/cassandra/lib/cql-internal-only-1.4.1.zip/cql-1.4.1/cql/thrifteries.py,
 line 77, in get_response
return self.handle_cql_execution_errors(doquery, compressed_q, compress, cl)
  File 
/usr/share/cassandra/lib/cql-internal-only-1.4.1.zip/cql-1.4.1/cql/thrifteries.py,
 line 96, in handle_cql_execution_errors
return executor(*args, **kwargs)
  File 
/usr/share/cassandra/lib/cql-internal-only-1.4.1.zip/cql-1.4.1/cql/cassandra/Cassandra.py,
 line 1830, in execute_cql3_query
self.send_execute_cql3_query(query, compression, consistency)
  File 
/usr/share/cassandra/lib/cql-internal-only-1.4.1.zip/cql-1.4.1/cql/cassandra/Cassandra.py,
 line 1841, in send_execute_cql3_query
self._oprot.trans.flush()
  File 
/usr/share/cassandra/lib/thrift-python-internal-only-0.9.1.zip/thrift/transport/TTransport.py,
 line 292, in flush
self.__trans.write(buf)
  File 
/usr/share/cassandra/lib/thrift-python-internal-only-0.9.1.zip/thrift/transport/TSocket.py,
 line 128, in write
plus = self.handle.send(buff)
error: [Errno 32] Broken pipe
{code}

In system.log:
{code}
ERROR [Thrift:3] 2014-08-24 22:53:02,242 CustomTThreadPoolServer.java (line 
219) Error occurred during processing of message.
java.lang.RuntimeException: java.util.concurrent.ExecutionException: 
java.lang.IllegalStateException: One row required, 0 found
at 
org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:413)
at 
org.apache.cassandra.service.MigrationManager.announce(MigrationManager.java:285)
at 
org.apache.cassandra.service.MigrationManager.announceKeyspaceDrop(MigrationManager.java:259)
at 
org.apache.cassandra.cql3.statements.DropKeyspaceStatement.announceMigration(DropKeyspaceStatement.java:62)
at 
org.apache.cassandra.cql3.statements.SchemaAlteringStatement.execute(SchemaAlteringStatement.java:71)
at 
org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:158)
at 
org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:175)
at 
org.apache.cassandra.thrift.CassandraServer.execute_cql3_query(CassandraServer.java:1958)
at 
org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4486)
at 
org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4470)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at 
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:201)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.util.concurrent.ExecutionException: 
java.lang.IllegalStateException: One row required, 0 found
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:188)
at 
org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:409)
... 15 more
Caused by: java.lang.IllegalStateException: One row required, 0 found
at 
org.apache.cassandra.cql3.UntypedResultSet.one(UntypedResultSet.java:53)
at 
org.apache.cassandra.config.KSMetaData.fromSchema(KSMetaData.java:263)
at 
org.apache.cassandra.db.DefsTables.mergeKeyspaces(DefsTables.java:227)
at org.apache.cassandra.db.DefsTables.mergeSchema(DefsTables.java:182)
at 
org.apache.cassandra.service.MigrationManager$2.runMayThrow(MigrationManager.java:303)
at 
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
... 3 more
{code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)