[jira] [Updated] (CASSANDRA-7983) nodetool repair triggers OOM

2014-11-14 Thread Jose Martinez Poblete (JIRA)

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-7983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jose Martinez Poblete updated CASSANDRA-7983:
-
Environment: 
 DSE version: 4.5.0 Cassandra 2.0.8


  was:
 
{noformat}
 INFO [main] 2014-09-16 14:23:14,621 DseDaemon.java (line 368) DSE version: 
4.5.0
 INFO [main] 2014-09-16 14:23:14,622 DseDaemon.java (line 369) Hadoop version: 
1.0.4.13
 INFO [main] 2014-09-16 14:23:14,627 DseDaemon.java (line 370) Hive version: 
0.12.0.3
 INFO [main] 2014-09-16 14:23:14,628 DseDaemon.java (line 371) Pig version: 
0.10.1
 INFO [main] 2014-09-16 14:23:14,629 DseDaemon.java (line 372) Solr version: 
4.6.0.2.4
 INFO [main] 2014-09-16 14:23:14,630 DseDaemon.java (line 373) Sqoop version: 
1.4.4.14.1
 INFO [main] 2014-09-16 14:23:14,630 DseDaemon.java (line 374) Mahout version: 
0.8
 INFO [main] 2014-09-16 14:23:14,631 DseDaemon.java (line 375) Appender 
version: 3.0.2
 INFO [main] 2014-09-16 14:23:14,632 DseDaemon.java (line 376) Spark version: 
0.9.1
 INFO [main] 2014-09-16 14:23:14,632 DseDaemon.java (line 377) Shark version: 
0.9.1.1
 INFO [main] 2014-09-16 14:23:20,270 CassandraDaemon.java (line 160) JVM 
vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.7.0_51
 INFO [main] 2014-09-16 14:23:20,270 CassandraDaemon.java (line 188) Heap size: 
6316621824/6316621824
{noformat}


 nodetool repair triggers OOM
 

 Key: CASSANDRA-7983
 URL: https://issues.apache.org/jira/browse/CASSANDRA-7983
 Project: Cassandra
  Issue Type: Bug
  Components: Tools
 Environment:  DSE version: 4.5.0 Cassandra 2.0.8
Reporter: Jose Martinez Poblete
Assignee: Jimmy Mårdell
 Fix For: 2.0.11, 2.1.1

 Attachments: 7983-v1.patch, gc.log.0, nbcqa-chc-a01_systemlog.tar.Z, 
 nbcqa-chc-a03_systemlog.tar.Z, system.log


 Customer has a 3 node cluster with 500Mb data on each node
 {noformat}
 [cassandra@nbcqa-chc-a02 ~]$ nodetool status
 Note: Ownership information does not include topology; for complete 
 information, specify a keyspace
 Datacenter: CH2
 ===
 Status=Up/Down
 |/ State=Normal/Leaving/Joining/Moving
 --  AddressLoad   Tokens  Owns   Host ID  
  Rack
 UN  162.150.4.234  255.26 MB  256 33.2%  
 4ad1b6a8-8759-4920-b54a-f059126900df  RAC1
 UN  162.150.4.235  318.37 MB  256 32.6%  
 3eb0ec58-4b81-442e-bee5-4c91da447f38  RAC1
 UN  162.150.4.167  243.7 MB   256 34.2%  
 5b2c1900-bf03-41c1-bb4e-82df1655b8d8  RAC1
 [cassandra@nbcqa-chc-a02 ~]$
 {noformat}
 After we run repair command, system runs into OOM after some 45 minutes
 Nothing else is running
 {noformat}
 [cassandra@nbcqa-chc-a02 ~]$ date
 Fri Sep 19 15:55:33 UTC 2014
 [cassandra@nbcqa-chc-a02 ~]$ nodetool repair -st -9220354588320251877 -et 
 -9220354588320251873
 Sep 19, 2014 4:06:08 PM ClientCommunicatorAdmin Checker-run
 WARNING: Failed to check the connection: java.net.SocketTimeoutException: 
 Read timed out
 {noformat}
 Here is when we run OOM
 {noformat}
 ERROR [ReadStage:28914] 2014-09-19 16:34:50,381 CassandraDaemon.java (line 
 199) Exception in thread Thread[ReadStage:28914,5,main]
 java.lang.OutOfMemoryError: Java heap space
 at 
 org.apache.cassandra.io.util.RandomAccessReader.init(RandomAccessReader.java:69)
 at 
 org.apache.cassandra.io.compress.CompressedRandomAccessReader.init(CompressedRandomAccessReader.java:76)
 at 
 org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:43)
 at 
 org.apache.cassandra.io.util.CompressedPoolingSegmentedFile.createReader(CompressedPoolingSegmentedFile.java:48)
 at 
 org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:39)
 at 
 org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:1195)
 at 
 org.apache.cassandra.db.columniterator.SimpleSliceReader.init(SimpleSliceReader.java:57)
 at 
 org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:65)
 at 
 org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:42)
 at 
 org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:167)
 at 
 org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:62)
 at 
 org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:250)
 at 
 org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1547)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1376)
 at 

[jira] [Updated] (CASSANDRA-7983) nodetool repair triggers OOM

2014-10-30 Thread Jose Martinez Poblete (JIRA)

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-7983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jose Martinez Poblete updated CASSANDRA-7983:
-
Description: 
Customer has a 3 node cluster with 500Mb data on each node

{noformat}
[cassandra@nbcqa-chc-a02 ~]$ nodetool status
Note: Ownership information does not include topology; for complete 
information, specify a keyspace
Datacenter: CH2
===
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  AddressLoad   Tokens  Owns   Host ID
   Rack
UN  162.150.4.234  255.26 MB  256 33.2%  
4ad1b6a8-8759-4920-b54a-f059126900df  RAC1
UN  162.150.4.235  318.37 MB  256 32.6%  
3eb0ec58-4b81-442e-bee5-4c91da447f38  RAC1
UN  162.150.4.167  243.7 MB   256 34.2%  
5b2c1900-bf03-41c1-bb4e-82df1655b8d8  RAC1
[cassandra@nbcqa-chc-a02 ~]$
{noformat}

After we run repair command, system runs into OOM after some 45 minutes
Nothing else is running

{noformat}
[cassandra@nbcqa-chc-a02 ~]$ date
Fri Sep 19 15:55:33 UTC 2014
[cassandra@nbcqa-chc-a02 ~]$ nodetool repair -st -9220354588320251877 -et 
-9220354588320251873
Sep 19, 2014 4:06:08 PM ClientCommunicatorAdmin Checker-run
WARNING: Failed to check the connection: java.net.SocketTimeoutException: Read 
timed out
{noformat}

Here is when we run OOM

{noformat}
ERROR [ReadStage:28914] 2014-09-19 16:34:50,381 CassandraDaemon.java (line 199) 
Exception in thread Thread[ReadStage:28914,5,main]
java.lang.OutOfMemoryError: Java heap space
at 
org.apache.cassandra.io.util.RandomAccessReader.init(RandomAccessReader.java:69)
at 
org.apache.cassandra.io.compress.CompressedRandomAccessReader.init(CompressedRandomAccessReader.java:76)
at 
org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:43)
at 
org.apache.cassandra.io.util.CompressedPoolingSegmentedFile.createReader(CompressedPoolingSegmentedFile.java:48)
at 
org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:39)
at 
org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:1195)
at 
org.apache.cassandra.db.columniterator.SimpleSliceReader.init(SimpleSliceReader.java:57)
at 
org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:65)
at 
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:42)
at 
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:167)
at 
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:62)
at 
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:250)
at 
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
at 
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1547)
at 
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1376)
at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:333)
at 
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65)
at 
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1363)
at 
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1927)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
{noformat}

Cassandra process pegs 1 of the 8 CPU's 100% 

{noformat}
top - 16:50:07 up 11 days,  2:01,  2 users,  load average: 0.54, 0.60, 0.65
Tasks: 175 total,   1 running, 174 sleeping,   0 stopped,   0 zombie
Cpu0  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu2  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu3  :  0.7%us,  0.3%sy,  0.0%ni, 99.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu4  :  0.3%us,  0.3%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu5  :  0.3%us,  0.3%sy,  0.0%ni, 99.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.3%st
Cpu6  :  0.0%us,  0.3%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu7  :  0.3%us,  0.3%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  16332056k total, 16167212k used,   164844k free,   149956k buffers
Swap:0k total,0k used,0k free,  8360056k cached

  PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND
 2161 cassandr  20   0 11.5g 6.9g 227m S 107.8 44.0 281:59.49 java
 9942 root  20   0  106m 2320 1344 S  1.0  0.0   0:00.03 dhclient-script
28969 opscente  20   0 4479m 188m  

[jira] [Updated] (CASSANDRA-7983) nodetool repair triggers OOM

2014-09-29 Thread JIRA

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-7983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jimmy Mårdell updated CASSANDRA-7983:
-
Attachment: 7983-v1.patch

 nodetool repair triggers OOM
 

 Key: CASSANDRA-7983
 URL: https://issues.apache.org/jira/browse/CASSANDRA-7983
 Project: Cassandra
  Issue Type: Bug
  Components: Tools
 Environment:  
 {noformat}
  INFO [main] 2014-09-16 14:23:14,621 DseDaemon.java (line 368) DSE version: 
 4.5.0
  INFO [main] 2014-09-16 14:23:14,622 DseDaemon.java (line 369) Hadoop 
 version: 1.0.4.13
  INFO [main] 2014-09-16 14:23:14,627 DseDaemon.java (line 370) Hive version: 
 0.12.0.3
  INFO [main] 2014-09-16 14:23:14,628 DseDaemon.java (line 371) Pig version: 
 0.10.1
  INFO [main] 2014-09-16 14:23:14,629 DseDaemon.java (line 372) Solr version: 
 4.6.0.2.4
  INFO [main] 2014-09-16 14:23:14,630 DseDaemon.java (line 373) Sqoop version: 
 1.4.4.14.1
  INFO [main] 2014-09-16 14:23:14,630 DseDaemon.java (line 374) Mahout 
 version: 0.8
  INFO [main] 2014-09-16 14:23:14,631 DseDaemon.java (line 375) Appender 
 version: 3.0.2
  INFO [main] 2014-09-16 14:23:14,632 DseDaemon.java (line 376) Spark version: 
 0.9.1
  INFO [main] 2014-09-16 14:23:14,632 DseDaemon.java (line 377) Shark version: 
 0.9.1.1
  INFO [main] 2014-09-16 14:23:20,270 CassandraDaemon.java (line 160) JVM 
 vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.7.0_51
  INFO [main] 2014-09-16 14:23:20,270 CassandraDaemon.java (line 188) Heap 
 size: 6316621824/6316621824
 {noformat}
Reporter: Jose Martinez Poblete
Assignee: Yuki Morishita
 Fix For: 2.0.11

 Attachments: 7983-v1.patch, gc.log.0, nbcqa-chc-a01_systemlog.tar.Z, 
 nbcqa-chc-a03_systemlog.tar.Z, system.log


 Customer has a 3 node cluster with 500Mb data on each node
 {noformat}
 [cassandra@nbcqa-chc-a02 ~]$ nodetool status
 Note: Ownership information does not include topology; for complete 
 information, specify a keyspace
 Datacenter: CH2
 ===
 Status=Up/Down
 |/ State=Normal/Leaving/Joining/Moving
 --  AddressLoad   Tokens  Owns   Host ID  
  Rack
 UN  162.150.4.234  255.26 MB  256 33.2%  
 4ad1b6a8-8759-4920-b54a-f059126900df  RAC1
 UN  162.150.4.235  318.37 MB  256 32.6%  
 3eb0ec58-4b81-442e-bee5-4c91da447f38  RAC1
 UN  162.150.4.167  243.7 MB   256 34.2%  
 5b2c1900-bf03-41c1-bb4e-82df1655b8d8  RAC1
 [cassandra@nbcqa-chc-a02 ~]$
 {noformat}
 After we run repair command, system runs into OOM after some 45 minutes
 Nothing else is running
 {noformat}
 [cassandra@nbcqa-chc-a02 ~]$ date
 Fri Sep 19 15:55:33 UTC 2014
 [cassandra@nbcqa-chc-a02 ~]$ nodetool repair -st -9220354588320251877 -et 
 -9220354588320251873
 Sep 19, 2014 4:06:08 PM ClientCommunicatorAdmin Checker-run
 WARNING: Failed to check the connection: java.net.SocketTimeoutException: 
 Read timed out
 {noformat}
 Herer is when we run OOM
 {noformat}
 ERROR [ReadStage:28914] 2014-09-19 16:34:50,381 CassandraDaemon.java (line 
 199) Exception in thread Thread[ReadStage:28914,5,main]
 java.lang.OutOfMemoryError: Java heap space
 at 
 org.apache.cassandra.io.util.RandomAccessReader.init(RandomAccessReader.java:69)
 at 
 org.apache.cassandra.io.compress.CompressedRandomAccessReader.init(CompressedRandomAccessReader.java:76)
 at 
 org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:43)
 at 
 org.apache.cassandra.io.util.CompressedPoolingSegmentedFile.createReader(CompressedPoolingSegmentedFile.java:48)
 at 
 org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:39)
 at 
 org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:1195)
 at 
 org.apache.cassandra.db.columniterator.SimpleSliceReader.init(SimpleSliceReader.java:57)
 at 
 org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:65)
 at 
 org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:42)
 at 
 org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:167)
 at 
 org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:62)
 at 
 org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:250)
 at 
 org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1547)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1376)
 at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:333)
 at 
 

[jira] [Updated] (CASSANDRA-7983) nodetool repair triggers OOM

2014-09-23 Thread Jonathan Ellis (JIRA)

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-7983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Ellis updated CASSANDRA-7983:
--
  Component/s: (was: Core)
   Tools
 Reviewer: Jimmy Mårdell
Fix Version/s: 2.0.11
 Assignee: Yuki Morishita

Good detective work, [~yarin].

 nodetool repair triggers OOM
 

 Key: CASSANDRA-7983
 URL: https://issues.apache.org/jira/browse/CASSANDRA-7983
 Project: Cassandra
  Issue Type: Bug
  Components: Tools
 Environment:  
 {noformat}
  INFO [main] 2014-09-16 14:23:14,621 DseDaemon.java (line 368) DSE version: 
 4.5.0
  INFO [main] 2014-09-16 14:23:14,622 DseDaemon.java (line 369) Hadoop 
 version: 1.0.4.13
  INFO [main] 2014-09-16 14:23:14,627 DseDaemon.java (line 370) Hive version: 
 0.12.0.3
  INFO [main] 2014-09-16 14:23:14,628 DseDaemon.java (line 371) Pig version: 
 0.10.1
  INFO [main] 2014-09-16 14:23:14,629 DseDaemon.java (line 372) Solr version: 
 4.6.0.2.4
  INFO [main] 2014-09-16 14:23:14,630 DseDaemon.java (line 373) Sqoop version: 
 1.4.4.14.1
  INFO [main] 2014-09-16 14:23:14,630 DseDaemon.java (line 374) Mahout 
 version: 0.8
  INFO [main] 2014-09-16 14:23:14,631 DseDaemon.java (line 375) Appender 
 version: 3.0.2
  INFO [main] 2014-09-16 14:23:14,632 DseDaemon.java (line 376) Spark version: 
 0.9.1
  INFO [main] 2014-09-16 14:23:14,632 DseDaemon.java (line 377) Shark version: 
 0.9.1.1
  INFO [main] 2014-09-16 14:23:20,270 CassandraDaemon.java (line 160) JVM 
 vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.7.0_51
  INFO [main] 2014-09-16 14:23:20,270 CassandraDaemon.java (line 188) Heap 
 size: 6316621824/6316621824
 {noformat}
Reporter: Jose Martinez Poblete
Assignee: Yuki Morishita
 Fix For: 2.0.11

 Attachments: gc.log.0, nbcqa-chc-a01_systemlog.tar.Z, 
 nbcqa-chc-a03_systemlog.tar.Z, system.log


 Customer has a 3 node cluster with 500Mb data on each node
 {noformat}
 [cassandra@nbcqa-chc-a02 ~]$ nodetool status
 Note: Ownership information does not include topology; for complete 
 information, specify a keyspace
 Datacenter: CH2
 ===
 Status=Up/Down
 |/ State=Normal/Leaving/Joining/Moving
 --  AddressLoad   Tokens  Owns   Host ID  
  Rack
 UN  162.150.4.234  255.26 MB  256 33.2%  
 4ad1b6a8-8759-4920-b54a-f059126900df  RAC1
 UN  162.150.4.235  318.37 MB  256 32.6%  
 3eb0ec58-4b81-442e-bee5-4c91da447f38  RAC1
 UN  162.150.4.167  243.7 MB   256 34.2%  
 5b2c1900-bf03-41c1-bb4e-82df1655b8d8  RAC1
 [cassandra@nbcqa-chc-a02 ~]$
 {noformat}
 After we run repair command, system runs into OOM after some 45 minutes
 Nothing else is running
 {noformat}
 [cassandra@nbcqa-chc-a02 ~]$ date
 Fri Sep 19 15:55:33 UTC 2014
 [cassandra@nbcqa-chc-a02 ~]$ nodetool repair -st -9220354588320251877 -et 
 -9220354588320251873
 Sep 19, 2014 4:06:08 PM ClientCommunicatorAdmin Checker-run
 WARNING: Failed to check the connection: java.net.SocketTimeoutException: 
 Read timed out
 {noformat}
 Herer is when we run OOM
 {noformat}
 ERROR [ReadStage:28914] 2014-09-19 16:34:50,381 CassandraDaemon.java (line 
 199) Exception in thread Thread[ReadStage:28914,5,main]
 java.lang.OutOfMemoryError: Java heap space
 at 
 org.apache.cassandra.io.util.RandomAccessReader.init(RandomAccessReader.java:69)
 at 
 org.apache.cassandra.io.compress.CompressedRandomAccessReader.init(CompressedRandomAccessReader.java:76)
 at 
 org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:43)
 at 
 org.apache.cassandra.io.util.CompressedPoolingSegmentedFile.createReader(CompressedPoolingSegmentedFile.java:48)
 at 
 org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:39)
 at 
 org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:1195)
 at 
 org.apache.cassandra.db.columniterator.SimpleSliceReader.init(SimpleSliceReader.java:57)
 at 
 org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:65)
 at 
 org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:42)
 at 
 org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:167)
 at 
 org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:62)
 at 
 org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:250)
 at 
 org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1547)
 at 
 

[jira] [Updated] (CASSANDRA-7983) nodetool repair triggers OOM

2014-09-22 Thread Jose Martinez Poblete (JIRA)

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-7983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jose Martinez Poblete updated CASSANDRA-7983:
-
Attachment: nbcqa-chc-a03_systemlog.tar.Z
nbcqa-chc-a01_systemlog.tar.Z

The rest of the system logs

 nodetool repair triggers OOM
 

 Key: CASSANDRA-7983
 URL: https://issues.apache.org/jira/browse/CASSANDRA-7983
 Project: Cassandra
  Issue Type: Bug
  Components: Core
 Environment:  
 {noformat}
  INFO [main] 2014-09-16 14:23:14,621 DseDaemon.java (line 368) DSE version: 
 4.5.0
  INFO [main] 2014-09-16 14:23:14,622 DseDaemon.java (line 369) Hadoop 
 version: 1.0.4.13
  INFO [main] 2014-09-16 14:23:14,627 DseDaemon.java (line 370) Hive version: 
 0.12.0.3
  INFO [main] 2014-09-16 14:23:14,628 DseDaemon.java (line 371) Pig version: 
 0.10.1
  INFO [main] 2014-09-16 14:23:14,629 DseDaemon.java (line 372) Solr version: 
 4.6.0.2.4
  INFO [main] 2014-09-16 14:23:14,630 DseDaemon.java (line 373) Sqoop version: 
 1.4.4.14.1
  INFO [main] 2014-09-16 14:23:14,630 DseDaemon.java (line 374) Mahout 
 version: 0.8
  INFO [main] 2014-09-16 14:23:14,631 DseDaemon.java (line 375) Appender 
 version: 3.0.2
  INFO [main] 2014-09-16 14:23:14,632 DseDaemon.java (line 376) Spark version: 
 0.9.1
  INFO [main] 2014-09-16 14:23:14,632 DseDaemon.java (line 377) Shark version: 
 0.9.1.1
  INFO [main] 2014-09-16 14:23:20,270 CassandraDaemon.java (line 160) JVM 
 vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.7.0_51
  INFO [main] 2014-09-16 14:23:20,270 CassandraDaemon.java (line 188) Heap 
 size: 6316621824/6316621824
 {noformat}
Reporter: Jose Martinez Poblete
 Attachments: gc.log.0, nbcqa-chc-a01_systemlog.tar.Z, 
 nbcqa-chc-a03_systemlog.tar.Z, system.log


 Customer has a 3 node cluster with 500Mb data on each node
 {noformat}
 [cassandra@nbcqa-chc-a02 ~]$ nodetool status
 Note: Ownership information does not include topology; for complete 
 information, specify a keyspace
 Datacenter: CH2
 ===
 Status=Up/Down
 |/ State=Normal/Leaving/Joining/Moving
 --  AddressLoad   Tokens  Owns   Host ID  
  Rack
 UN  162.150.4.234  255.26 MB  256 33.2%  
 4ad1b6a8-8759-4920-b54a-f059126900df  RAC1
 UN  162.150.4.235  318.37 MB  256 32.6%  
 3eb0ec58-4b81-442e-bee5-4c91da447f38  RAC1
 UN  162.150.4.167  243.7 MB   256 34.2%  
 5b2c1900-bf03-41c1-bb4e-82df1655b8d8  RAC1
 [cassandra@nbcqa-chc-a02 ~]$
 {noformat}
 After we run repair command, system runs into OOM after some 45 minutes
 Nothing else is running
 {noformat}
 [cassandra@nbcqa-chc-a02 ~]$ date
 Fri Sep 19 15:55:33 UTC 2014
 [cassandra@nbcqa-chc-a02 ~]$ nodetool repair -st -9220354588320251877 -et 
 -9220354588320251873
 Sep 19, 2014 4:06:08 PM ClientCommunicatorAdmin Checker-run
 WARNING: Failed to check the connection: java.net.SocketTimeoutException: 
 Read timed out
 {noformat}
 Herer is when we run OOM
 {noformat}
 ERROR [ReadStage:28914] 2014-09-19 16:34:50,381 CassandraDaemon.java (line 
 199) Exception in thread Thread[ReadStage:28914,5,main]
 java.lang.OutOfMemoryError: Java heap space
 at 
 org.apache.cassandra.io.util.RandomAccessReader.init(RandomAccessReader.java:69)
 at 
 org.apache.cassandra.io.compress.CompressedRandomAccessReader.init(CompressedRandomAccessReader.java:76)
 at 
 org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:43)
 at 
 org.apache.cassandra.io.util.CompressedPoolingSegmentedFile.createReader(CompressedPoolingSegmentedFile.java:48)
 at 
 org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:39)
 at 
 org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:1195)
 at 
 org.apache.cassandra.db.columniterator.SimpleSliceReader.init(SimpleSliceReader.java:57)
 at 
 org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:65)
 at 
 org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:42)
 at 
 org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:167)
 at 
 org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:62)
 at 
 org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:250)
 at 
 org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1547)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1376)
 at 

[jira] [Updated] (CASSANDRA-7983) nodetool repair triggers OOM

2014-09-22 Thread Jose Martinez Poblete (JIRA)

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-7983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jose Martinez Poblete updated CASSANDRA-7983:
-
Attachment: (was: nbcqa-chc-a01_systemlog.tar.Z)

 nodetool repair triggers OOM
 

 Key: CASSANDRA-7983
 URL: https://issues.apache.org/jira/browse/CASSANDRA-7983
 Project: Cassandra
  Issue Type: Bug
  Components: Core
 Environment:  
 {noformat}
  INFO [main] 2014-09-16 14:23:14,621 DseDaemon.java (line 368) DSE version: 
 4.5.0
  INFO [main] 2014-09-16 14:23:14,622 DseDaemon.java (line 369) Hadoop 
 version: 1.0.4.13
  INFO [main] 2014-09-16 14:23:14,627 DseDaemon.java (line 370) Hive version: 
 0.12.0.3
  INFO [main] 2014-09-16 14:23:14,628 DseDaemon.java (line 371) Pig version: 
 0.10.1
  INFO [main] 2014-09-16 14:23:14,629 DseDaemon.java (line 372) Solr version: 
 4.6.0.2.4
  INFO [main] 2014-09-16 14:23:14,630 DseDaemon.java (line 373) Sqoop version: 
 1.4.4.14.1
  INFO [main] 2014-09-16 14:23:14,630 DseDaemon.java (line 374) Mahout 
 version: 0.8
  INFO [main] 2014-09-16 14:23:14,631 DseDaemon.java (line 375) Appender 
 version: 3.0.2
  INFO [main] 2014-09-16 14:23:14,632 DseDaemon.java (line 376) Spark version: 
 0.9.1
  INFO [main] 2014-09-16 14:23:14,632 DseDaemon.java (line 377) Shark version: 
 0.9.1.1
  INFO [main] 2014-09-16 14:23:20,270 CassandraDaemon.java (line 160) JVM 
 vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.7.0_51
  INFO [main] 2014-09-16 14:23:20,270 CassandraDaemon.java (line 188) Heap 
 size: 6316621824/6316621824
 {noformat}
Reporter: Jose Martinez Poblete
 Attachments: gc.log.0, nbcqa-chc-a01_systemlog.tar.Z, 
 nbcqa-chc-a03_systemlog.tar.Z, system.log


 Customer has a 3 node cluster with 500Mb data on each node
 {noformat}
 [cassandra@nbcqa-chc-a02 ~]$ nodetool status
 Note: Ownership information does not include topology; for complete 
 information, specify a keyspace
 Datacenter: CH2
 ===
 Status=Up/Down
 |/ State=Normal/Leaving/Joining/Moving
 --  AddressLoad   Tokens  Owns   Host ID  
  Rack
 UN  162.150.4.234  255.26 MB  256 33.2%  
 4ad1b6a8-8759-4920-b54a-f059126900df  RAC1
 UN  162.150.4.235  318.37 MB  256 32.6%  
 3eb0ec58-4b81-442e-bee5-4c91da447f38  RAC1
 UN  162.150.4.167  243.7 MB   256 34.2%  
 5b2c1900-bf03-41c1-bb4e-82df1655b8d8  RAC1
 [cassandra@nbcqa-chc-a02 ~]$
 {noformat}
 After we run repair command, system runs into OOM after some 45 minutes
 Nothing else is running
 {noformat}
 [cassandra@nbcqa-chc-a02 ~]$ date
 Fri Sep 19 15:55:33 UTC 2014
 [cassandra@nbcqa-chc-a02 ~]$ nodetool repair -st -9220354588320251877 -et 
 -9220354588320251873
 Sep 19, 2014 4:06:08 PM ClientCommunicatorAdmin Checker-run
 WARNING: Failed to check the connection: java.net.SocketTimeoutException: 
 Read timed out
 {noformat}
 Herer is when we run OOM
 {noformat}
 ERROR [ReadStage:28914] 2014-09-19 16:34:50,381 CassandraDaemon.java (line 
 199) Exception in thread Thread[ReadStage:28914,5,main]
 java.lang.OutOfMemoryError: Java heap space
 at 
 org.apache.cassandra.io.util.RandomAccessReader.init(RandomAccessReader.java:69)
 at 
 org.apache.cassandra.io.compress.CompressedRandomAccessReader.init(CompressedRandomAccessReader.java:76)
 at 
 org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:43)
 at 
 org.apache.cassandra.io.util.CompressedPoolingSegmentedFile.createReader(CompressedPoolingSegmentedFile.java:48)
 at 
 org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:39)
 at 
 org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:1195)
 at 
 org.apache.cassandra.db.columniterator.SimpleSliceReader.init(SimpleSliceReader.java:57)
 at 
 org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:65)
 at 
 org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:42)
 at 
 org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:167)
 at 
 org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:62)
 at 
 org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:250)
 at 
 org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1547)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1376)
 at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:333)
 at 
 

[jira] [Updated] (CASSANDRA-7983) nodetool repair triggers OOM

2014-09-22 Thread Jose Martinez Poblete (JIRA)

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-7983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jose Martinez Poblete updated CASSANDRA-7983:
-
Attachment: (was: nbcqa-chc-a03_systemlog.tar.Z)

 nodetool repair triggers OOM
 

 Key: CASSANDRA-7983
 URL: https://issues.apache.org/jira/browse/CASSANDRA-7983
 Project: Cassandra
  Issue Type: Bug
  Components: Core
 Environment:  
 {noformat}
  INFO [main] 2014-09-16 14:23:14,621 DseDaemon.java (line 368) DSE version: 
 4.5.0
  INFO [main] 2014-09-16 14:23:14,622 DseDaemon.java (line 369) Hadoop 
 version: 1.0.4.13
  INFO [main] 2014-09-16 14:23:14,627 DseDaemon.java (line 370) Hive version: 
 0.12.0.3
  INFO [main] 2014-09-16 14:23:14,628 DseDaemon.java (line 371) Pig version: 
 0.10.1
  INFO [main] 2014-09-16 14:23:14,629 DseDaemon.java (line 372) Solr version: 
 4.6.0.2.4
  INFO [main] 2014-09-16 14:23:14,630 DseDaemon.java (line 373) Sqoop version: 
 1.4.4.14.1
  INFO [main] 2014-09-16 14:23:14,630 DseDaemon.java (line 374) Mahout 
 version: 0.8
  INFO [main] 2014-09-16 14:23:14,631 DseDaemon.java (line 375) Appender 
 version: 3.0.2
  INFO [main] 2014-09-16 14:23:14,632 DseDaemon.java (line 376) Spark version: 
 0.9.1
  INFO [main] 2014-09-16 14:23:14,632 DseDaemon.java (line 377) Shark version: 
 0.9.1.1
  INFO [main] 2014-09-16 14:23:20,270 CassandraDaemon.java (line 160) JVM 
 vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.7.0_51
  INFO [main] 2014-09-16 14:23:20,270 CassandraDaemon.java (line 188) Heap 
 size: 6316621824/6316621824
 {noformat}
Reporter: Jose Martinez Poblete
 Attachments: gc.log.0, nbcqa-chc-a01_systemlog.tar.Z, 
 nbcqa-chc-a03_systemlog.tar.Z, system.log


 Customer has a 3 node cluster with 500Mb data on each node
 {noformat}
 [cassandra@nbcqa-chc-a02 ~]$ nodetool status
 Note: Ownership information does not include topology; for complete 
 information, specify a keyspace
 Datacenter: CH2
 ===
 Status=Up/Down
 |/ State=Normal/Leaving/Joining/Moving
 --  AddressLoad   Tokens  Owns   Host ID  
  Rack
 UN  162.150.4.234  255.26 MB  256 33.2%  
 4ad1b6a8-8759-4920-b54a-f059126900df  RAC1
 UN  162.150.4.235  318.37 MB  256 32.6%  
 3eb0ec58-4b81-442e-bee5-4c91da447f38  RAC1
 UN  162.150.4.167  243.7 MB   256 34.2%  
 5b2c1900-bf03-41c1-bb4e-82df1655b8d8  RAC1
 [cassandra@nbcqa-chc-a02 ~]$
 {noformat}
 After we run repair command, system runs into OOM after some 45 minutes
 Nothing else is running
 {noformat}
 [cassandra@nbcqa-chc-a02 ~]$ date
 Fri Sep 19 15:55:33 UTC 2014
 [cassandra@nbcqa-chc-a02 ~]$ nodetool repair -st -9220354588320251877 -et 
 -9220354588320251873
 Sep 19, 2014 4:06:08 PM ClientCommunicatorAdmin Checker-run
 WARNING: Failed to check the connection: java.net.SocketTimeoutException: 
 Read timed out
 {noformat}
 Herer is when we run OOM
 {noformat}
 ERROR [ReadStage:28914] 2014-09-19 16:34:50,381 CassandraDaemon.java (line 
 199) Exception in thread Thread[ReadStage:28914,5,main]
 java.lang.OutOfMemoryError: Java heap space
 at 
 org.apache.cassandra.io.util.RandomAccessReader.init(RandomAccessReader.java:69)
 at 
 org.apache.cassandra.io.compress.CompressedRandomAccessReader.init(CompressedRandomAccessReader.java:76)
 at 
 org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:43)
 at 
 org.apache.cassandra.io.util.CompressedPoolingSegmentedFile.createReader(CompressedPoolingSegmentedFile.java:48)
 at 
 org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:39)
 at 
 org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:1195)
 at 
 org.apache.cassandra.db.columniterator.SimpleSliceReader.init(SimpleSliceReader.java:57)
 at 
 org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:65)
 at 
 org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:42)
 at 
 org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:167)
 at 
 org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:62)
 at 
 org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:250)
 at 
 org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1547)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1376)
 at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:333)
 at 
 

[jira] [Updated] (CASSANDRA-7983) nodetool repair triggers OOM

2014-09-21 Thread Jose Martinez Poblete (JIRA)

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-7983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jose Martinez Poblete updated CASSANDRA-7983:
-
Attachment: nbcqa-chc-a01_systemlog.tar.Z
nbcqa-chc-a03_systemlog.tar.Z

system logs from other 2 nodes

 nodetool repair triggers OOM
 

 Key: CASSANDRA-7983
 URL: https://issues.apache.org/jira/browse/CASSANDRA-7983
 Project: Cassandra
  Issue Type: Bug
  Components: Core
 Environment:  
 {noformat}
  INFO [main] 2014-09-16 14:23:14,621 DseDaemon.java (line 368) DSE version: 
 4.5.0
  INFO [main] 2014-09-16 14:23:14,622 DseDaemon.java (line 369) Hadoop 
 version: 1.0.4.13
  INFO [main] 2014-09-16 14:23:14,627 DseDaemon.java (line 370) Hive version: 
 0.12.0.3
  INFO [main] 2014-09-16 14:23:14,628 DseDaemon.java (line 371) Pig version: 
 0.10.1
  INFO [main] 2014-09-16 14:23:14,629 DseDaemon.java (line 372) Solr version: 
 4.6.0.2.4
  INFO [main] 2014-09-16 14:23:14,630 DseDaemon.java (line 373) Sqoop version: 
 1.4.4.14.1
  INFO [main] 2014-09-16 14:23:14,630 DseDaemon.java (line 374) Mahout 
 version: 0.8
  INFO [main] 2014-09-16 14:23:14,631 DseDaemon.java (line 375) Appender 
 version: 3.0.2
  INFO [main] 2014-09-16 14:23:14,632 DseDaemon.java (line 376) Spark version: 
 0.9.1
  INFO [main] 2014-09-16 14:23:14,632 DseDaemon.java (line 377) Shark version: 
 0.9.1.1
  INFO [main] 2014-09-16 14:23:20,270 CassandraDaemon.java (line 160) JVM 
 vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.7.0_51
  INFO [main] 2014-09-16 14:23:20,270 CassandraDaemon.java (line 188) Heap 
 size: 6316621824/6316621824
 {noformat}
Reporter: Jose Martinez Poblete
 Attachments: gc.log.0, nbcqa-chc-a01_systemlog.tar.Z, 
 nbcqa-chc-a03_systemlog.tar.Z, system.log


 Customer has a 3 node cluster with 500Mb data on each node
 {noformat}
 [cassandra@nbcqa-chc-a02 ~]$ nodetool status
 Note: Ownership information does not include topology; for complete 
 information, specify a keyspace
 Datacenter: CH2
 ===
 Status=Up/Down
 |/ State=Normal/Leaving/Joining/Moving
 --  AddressLoad   Tokens  Owns   Host ID  
  Rack
 UN  162.150.4.234  255.26 MB  256 33.2%  
 4ad1b6a8-8759-4920-b54a-f059126900df  RAC1
 UN  162.150.4.235  318.37 MB  256 32.6%  
 3eb0ec58-4b81-442e-bee5-4c91da447f38  RAC1
 UN  162.150.4.167  243.7 MB   256 34.2%  
 5b2c1900-bf03-41c1-bb4e-82df1655b8d8  RAC1
 [cassandra@nbcqa-chc-a02 ~]$
 {noformat}
 After we run repair command, system runs into OOM after some 45 minutes
 Nothing else is running
 {noformat}
 [cassandra@nbcqa-chc-a02 ~]$ date
 Fri Sep 19 15:55:33 UTC 2014
 [cassandra@nbcqa-chc-a02 ~]$ nodetool repair -st -9220354588320251877 -et 
 -9220354588320251873
 Sep 19, 2014 4:06:08 PM ClientCommunicatorAdmin Checker-run
 WARNING: Failed to check the connection: java.net.SocketTimeoutException: 
 Read timed out
 {noformat}
 Herer is when we run OOM
 {noformat}
 ERROR [ReadStage:28914] 2014-09-19 16:34:50,381 CassandraDaemon.java (line 
 199) Exception in thread Thread[ReadStage:28914,5,main]
 java.lang.OutOfMemoryError: Java heap space
 at 
 org.apache.cassandra.io.util.RandomAccessReader.init(RandomAccessReader.java:69)
 at 
 org.apache.cassandra.io.compress.CompressedRandomAccessReader.init(CompressedRandomAccessReader.java:76)
 at 
 org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:43)
 at 
 org.apache.cassandra.io.util.CompressedPoolingSegmentedFile.createReader(CompressedPoolingSegmentedFile.java:48)
 at 
 org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:39)
 at 
 org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:1195)
 at 
 org.apache.cassandra.db.columniterator.SimpleSliceReader.init(SimpleSliceReader.java:57)
 at 
 org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:65)
 at 
 org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:42)
 at 
 org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:167)
 at 
 org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:62)
 at 
 org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:250)
 at 
 org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1547)
 at 
 org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1376)
 at