[jira] [Commented] (CASSANDRA-4973) Secondary Index stops returning rows when caching=ALL

2013-10-04 Thread J.B. Langston (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-4973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13786389#comment-13786389
 ] 

J.B. Langston commented on CASSANDRA-4973:
--

Reproducible test case attached to CASSANDRA-4785, of which this appears to be 
a duplicate.

 Secondary Index stops returning rows when caching=ALL
 -

 Key: CASSANDRA-4973
 URL: https://issues.apache.org/jira/browse/CASSANDRA-4973
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 1.1.2, 1.1.6
 Environment: Centos 6.3, Java 1.6.0_35, cass. 1.1.2 upgraded to 1.1.6
Reporter: Daniel Strawson
 Attachments: secondary_index_rowcache_restart_test.py


 I've been using cassandra on a project for a little while in development and 
 have recently suddenly started having an issue where the secondary index 
 stops working, this is happening on my new production system, we are not yet 
 live.   Things work ok one moment, then suddenly queries to the cf through 
 the secondary index stop returning data.  I've seen it happen on 3 CFs. I've 
 tried:
 - various nodetools repair / scrub / rebuild_indexes options, none seem to 
 make a difference.
 - Doing a 'update column family whatever with column_metadata=[]' then 
 repeating with my correct column_metadata definition.  This seems to fix the 
 problem (temporarily) until it comes back.
 The last time it happened I had just restarted cassandra, so it could be that 
 which is causing the issue, I've got the production system ok at the moment, 
 I will try restarting a bit later when its not being used and if I can get 
 the issue to reoccur I will add more information.
 The problem first manifested itself in 1.1.2, so I upgraded to 1.1.6, this 
 has not fixed it.
 Here is an example of the create column family I'm using for one of the CFs 
 that affected:
 create column family region
   with column_type = 'Standard'
   and comparator = 'UTF8Type'
   and default_validation_class = 'BytesType'
   and key_validation_class = 'UTF8Type'
   and read_repair_chance = 0.1
   and dclocal_read_repair_chance = 0.0
   and gc_grace = 864000
   and min_compaction_threshold = 4
   and max_compaction_threshold = 32
   and replicate_on_write = true
   and compaction_strategy = 
 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
   and caching = 'KEYS_ONLY'
   and column_metadata = [
 
 {column_name : 'label',
 validation_class : UTF8Type},
 
 {column_name : 'countryCode',
 validation_class : UTF8Type,
 index_name : 'region_countryCode_idx',
 index_type : 0},
 
 ]
   and compression_options = {'sstable_compression' : 
 'org.apache.cassandra.io.compress.SnappyCompressor'};
 I've noticed that CASSANDRA-4785 looks similar, in my case once the system 
 has the problem, it doesn't go away until I fix it.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (CASSANDRA-4973) Secondary Index stops returning rows when caching=ALL

2013-08-15 Thread David Strauss (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-4973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13741739#comment-13741739
 ] 

David Strauss commented on CASSANDRA-4973:
--

I'm seeing this when I toggle a node from running without a row cache to having 
it enabled.

We made a utility to drop all indexes [1]. We follow up by re-adding the 
indexes using our schema scripts. Once this is done, indexes work properly 
again.

[1] https://gist.github.com/davidstrauss/6246289

 Secondary Index stops returning rows when caching=ALL
 -

 Key: CASSANDRA-4973
 URL: https://issues.apache.org/jira/browse/CASSANDRA-4973
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 1.1.2, 1.1.6
 Environment: Centos 6.3, Java 1.6.0_35, cass. 1.1.2 upgraded to 1.1.6
Reporter: Daniel Strawson
 Attachments: secondary_index_rowcache_restart_test.py


 I've been using cassandra on a project for a little while in development and 
 have recently suddenly started having an issue where the secondary index 
 stops working, this is happening on my new production system, we are not yet 
 live.   Things work ok one moment, then suddenly queries to the cf through 
 the secondary index stop returning data.  I've seen it happen on 3 CFs. I've 
 tried:
 - various nodetools repair / scrub / rebuild_indexes options, none seem to 
 make a difference.
 - Doing a 'update column family whatever with column_metadata=[]' then 
 repeating with my correct column_metadata definition.  This seems to fix the 
 problem (temporarily) until it comes back.
 The last time it happened I had just restarted cassandra, so it could be that 
 which is causing the issue, I've got the production system ok at the moment, 
 I will try restarting a bit later when its not being used and if I can get 
 the issue to reoccur I will add more information.
 The problem first manifested itself in 1.1.2, so I upgraded to 1.1.6, this 
 has not fixed it.
 Here is an example of the create column family I'm using for one of the CFs 
 that affected:
 create column family region
   with column_type = 'Standard'
   and comparator = 'UTF8Type'
   and default_validation_class = 'BytesType'
   and key_validation_class = 'UTF8Type'
   and read_repair_chance = 0.1
   and dclocal_read_repair_chance = 0.0
   and gc_grace = 864000
   and min_compaction_threshold = 4
   and max_compaction_threshold = 32
   and replicate_on_write = true
   and compaction_strategy = 
 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
   and caching = 'KEYS_ONLY'
   and column_metadata = [
 
 {column_name : 'label',
 validation_class : UTF8Type},
 
 {column_name : 'countryCode',
 validation_class : UTF8Type,
 index_name : 'region_countryCode_idx',
 index_type : 0},
 
 ]
   and compression_options = {'sstable_compression' : 
 'org.apache.cassandra.io.compress.SnappyCompressor'};
 I've noticed that CASSANDRA-4785 looks similar, in my case once the system 
 has the problem, it doesn't go away until I fix it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-4973) Secondary Index stops returning rows when caching=ALL

2013-04-02 Thread Alexei Bakanov (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-4973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619781#comment-13619781
 ] 

Alexei Bakanov commented on CASSANDRA-4973:
---

Enabling JNA didn't make any difference to the test. What made the difference 
was running my test with latest C* like this:

 Secondary Index stops returning rows when caching=ALL
 -

 Key: CASSANDRA-4973
 URL: https://issues.apache.org/jira/browse/CASSANDRA-4973
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 1.1.2, 1.1.6
 Environment: Centos 6.3, Java 1.6.0_35, cass. 1.1.2 upgraded to 1.1.6
Reporter: Daniel Strawson
 Attachments: secondary_index_rowcache_restart_test.py


 I've been using cassandra on a project for a little while in development and 
 have recently suddenly started having an issue where the secondary index 
 stops working, this is happening on my new production system, we are not yet 
 live.   Things work ok one moment, then suddenly queries to the cf through 
 the secondary index stop returning data.  I've seen it happen on 3 CFs. I've 
 tried:
 - various nodetools repair / scrub / rebuild_indexes options, none seem to 
 make a difference.
 - Doing a 'update column family whatever with column_metadata=[]' then 
 repeating with my correct column_metadata definition.  This seems to fix the 
 problem (temporarily) until it comes back.
 The last time it happened I had just restarted cassandra, so it could be that 
 which is causing the issue, I've got the production system ok at the moment, 
 I will try restarting a bit later when its not being used and if I can get 
 the issue to reoccur I will add more information.
 The problem first manifested itself in 1.1.2, so I upgraded to 1.1.6, this 
 has not fixed it.
 Here is an example of the create column family I'm using for one of the CFs 
 that affected:
 create column family region
   with column_type = 'Standard'
   and comparator = 'UTF8Type'
   and default_validation_class = 'BytesType'
   and key_validation_class = 'UTF8Type'
   and read_repair_chance = 0.1
   and dclocal_read_repair_chance = 0.0
   and gc_grace = 864000
   and min_compaction_threshold = 4
   and max_compaction_threshold = 32
   and replicate_on_write = true
   and compaction_strategy = 
 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
   and caching = 'KEYS_ONLY'
   and column_metadata = [
 
 {column_name : 'label',
 validation_class : UTF8Type},
 
 {column_name : 'countryCode',
 validation_class : UTF8Type,
 index_name : 'region_countryCode_idx',
 index_type : 0},
 
 ]
   and compression_options = {'sstable_compression' : 
 'org.apache.cassandra.io.compress.SnappyCompressor'};
 I've noticed that CASSANDRA-4785 looks similar, in my case once the system 
 has the problem, it doesn't go away until I fix it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-4973) Secondary Index stops returning rows when caching=ALL

2013-04-02 Thread Alexei Bakanov (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-4973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13619782#comment-13619782
 ] 

Alexei Bakanov commented on CASSANDRA-4973:
---

Enabling JNA didn't make any difference to the test. What made the difference 
was running my test with latest C* like this:

 Secondary Index stops returning rows when caching=ALL
 -

 Key: CASSANDRA-4973
 URL: https://issues.apache.org/jira/browse/CASSANDRA-4973
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 1.1.2, 1.1.6
 Environment: Centos 6.3, Java 1.6.0_35, cass. 1.1.2 upgraded to 1.1.6
Reporter: Daniel Strawson
 Attachments: secondary_index_rowcache_restart_test.py


 I've been using cassandra on a project for a little while in development and 
 have recently suddenly started having an issue where the secondary index 
 stops working, this is happening on my new production system, we are not yet 
 live.   Things work ok one moment, then suddenly queries to the cf through 
 the secondary index stop returning data.  I've seen it happen on 3 CFs. I've 
 tried:
 - various nodetools repair / scrub / rebuild_indexes options, none seem to 
 make a difference.
 - Doing a 'update column family whatever with column_metadata=[]' then 
 repeating with my correct column_metadata definition.  This seems to fix the 
 problem (temporarily) until it comes back.
 The last time it happened I had just restarted cassandra, so it could be that 
 which is causing the issue, I've got the production system ok at the moment, 
 I will try restarting a bit later when its not being used and if I can get 
 the issue to reoccur I will add more information.
 The problem first manifested itself in 1.1.2, so I upgraded to 1.1.6, this 
 has not fixed it.
 Here is an example of the create column family I'm using for one of the CFs 
 that affected:
 create column family region
   with column_type = 'Standard'
   and comparator = 'UTF8Type'
   and default_validation_class = 'BytesType'
   and key_validation_class = 'UTF8Type'
   and read_repair_chance = 0.1
   and dclocal_read_repair_chance = 0.0
   and gc_grace = 864000
   and min_compaction_threshold = 4
   and max_compaction_threshold = 32
   and replicate_on_write = true
   and compaction_strategy = 
 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
   and caching = 'KEYS_ONLY'
   and column_metadata = [
 
 {column_name : 'label',
 validation_class : UTF8Type},
 
 {column_name : 'countryCode',
 validation_class : UTF8Type,
 index_name : 'region_countryCode_idx',
 index_type : 0},
 
 ]
   and compression_options = {'sstable_compression' : 
 'org.apache.cassandra.io.compress.SnappyCompressor'};
 I've noticed that CASSANDRA-4785 looks similar, in my case once the system 
 has the problem, it doesn't go away until I fix it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-4973) Secondary Index stops returning rows when caching=ALL

2013-03-29 Thread Jeremy Hanna (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-4973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13617740#comment-13617740
 ] 

Jeremy Hanna commented on CASSANDRA-4973:
-

It's worth checking to see if JNA is getting picked up in your startup log.  
For one user, getting the JNA working resolved this problem.

 Secondary Index stops returning rows when caching=ALL
 -

 Key: CASSANDRA-4973
 URL: https://issues.apache.org/jira/browse/CASSANDRA-4973
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 1.1.2, 1.1.6
 Environment: Centos 6.3, Java 1.6.0_35, cass. 1.1.2 upgraded to 1.1.6
Reporter: Daniel Strawson
 Attachments: secondary_index_rowcache_restart_test.py


 I've been using cassandra on a project for a little while in development and 
 have recently suddenly started having an issue where the secondary index 
 stops working, this is happening on my new production system, we are not yet 
 live.   Things work ok one moment, then suddenly queries to the cf through 
 the secondary index stop returning data.  I've seen it happen on 3 CFs. I've 
 tried:
 - various nodetools repair / scrub / rebuild_indexes options, none seem to 
 make a difference.
 - Doing a 'update column family whatever with column_metadata=[]' then 
 repeating with my correct column_metadata definition.  This seems to fix the 
 problem (temporarily) until it comes back.
 The last time it happened I had just restarted cassandra, so it could be that 
 which is causing the issue, I've got the production system ok at the moment, 
 I will try restarting a bit later when its not being used and if I can get 
 the issue to reoccur I will add more information.
 The problem first manifested itself in 1.1.2, so I upgraded to 1.1.6, this 
 has not fixed it.
 Here is an example of the create column family I'm using for one of the CFs 
 that affected:
 create column family region
   with column_type = 'Standard'
   and comparator = 'UTF8Type'
   and default_validation_class = 'BytesType'
   and key_validation_class = 'UTF8Type'
   and read_repair_chance = 0.1
   and dclocal_read_repair_chance = 0.0
   and gc_grace = 864000
   and min_compaction_threshold = 4
   and max_compaction_threshold = 32
   and replicate_on_write = true
   and compaction_strategy = 
 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
   and caching = 'KEYS_ONLY'
   and column_metadata = [
 
 {column_name : 'label',
 validation_class : UTF8Type},
 
 {column_name : 'countryCode',
 validation_class : UTF8Type,
 index_name : 'region_countryCode_idx',
 index_type : 0},
 
 ]
   and compression_options = {'sstable_compression' : 
 'org.apache.cassandra.io.compress.SnappyCompressor'};
 I've noticed that CASSANDRA-4785 looks similar, in my case once the system 
 has the problem, it doesn't go away until I fix it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-4973) Secondary Index stops returning rows when caching=ALL

2013-02-26 Thread JIRA 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-4973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13587241#comment-13587241
 ] 

André Cruz commented on CASSANDRA-4973:
---

I also have this problem. Secondary index stops working on CFs with 
caching=ALL. Dropping the index and re-creating fixes it temporarily.

Running 1.1.5.

 Secondary Index stops returning rows when caching=ALL
 -

 Key: CASSANDRA-4973
 URL: https://issues.apache.org/jira/browse/CASSANDRA-4973
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 1.1.2, 1.1.6
 Environment: Centos 6.3, Java 1.6.0_35, cass. 1.1.2 upgraded to 1.1.6
Reporter: Daniel Strawson
 Attachments: secondary_index_rowcache_restart_test.py


 I've been using cassandra on a project for a little while in development and 
 have recently suddenly started having an issue where the secondary index 
 stops working, this is happening on my new production system, we are not yet 
 live.   Things work ok one moment, then suddenly queries to the cf through 
 the secondary index stop returning data.  I've seen it happen on 3 CFs. I've 
 tried:
 - various nodetools repair / scrub / rebuild_indexes options, none seem to 
 make a difference.
 - Doing a 'update column family whatever with column_metadata=[]' then 
 repeating with my correct column_metadata definition.  This seems to fix the 
 problem (temporarily) until it comes back.
 The last time it happened I had just restarted cassandra, so it could be that 
 which is causing the issue, I've got the production system ok at the moment, 
 I will try restarting a bit later when its not being used and if I can get 
 the issue to reoccur I will add more information.
 The problem first manifested itself in 1.1.2, so I upgraded to 1.1.6, this 
 has not fixed it.
 Here is an example of the create column family I'm using for one of the CFs 
 that affected:
 create column family region
   with column_type = 'Standard'
   and comparator = 'UTF8Type'
   and default_validation_class = 'BytesType'
   and key_validation_class = 'UTF8Type'
   and read_repair_chance = 0.1
   and dclocal_read_repair_chance = 0.0
   and gc_grace = 864000
   and min_compaction_threshold = 4
   and max_compaction_threshold = 32
   and replicate_on_write = true
   and compaction_strategy = 
 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
   and caching = 'KEYS_ONLY'
   and column_metadata = [
 
 {column_name : 'label',
 validation_class : UTF8Type},
 
 {column_name : 'countryCode',
 validation_class : UTF8Type,
 index_name : 'region_countryCode_idx',
 index_type : 0},
 
 ]
   and compression_options = {'sstable_compression' : 
 'org.apache.cassandra.io.compress.SnappyCompressor'};
 I've noticed that CASSANDRA-4785 looks similar, in my case once the system 
 has the problem, it doesn't go away until I fix it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-4973) Secondary Index stops returning rows when caching=ALL

2013-02-14 Thread Matthew F. Dennis (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-4973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13578823#comment-13578823
 ] 

Matthew F. Dennis commented on CASSANDRA-4973:
--

duplicate of CASSANDRA-4785 ?

 Secondary Index stops returning rows when caching=ALL
 -

 Key: CASSANDRA-4973
 URL: https://issues.apache.org/jira/browse/CASSANDRA-4973
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 1.1.2, 1.1.6
 Environment: Centos 6.3, Java 1.6.0_35, cass. 1.1.2 upgraded to 1.1.6
Reporter: Daniel Strawson
 Attachments: secondary_index_rowcache_restart_test.py


 I've been using cassandra on a project for a little while in development and 
 have recently suddenly started having an issue where the secondary index 
 stops working, this is happening on my new production system, we are not yet 
 live.   Things work ok one moment, then suddenly queries to the cf through 
 the secondary index stop returning data.  I've seen it happen on 3 CFs. I've 
 tried:
 - various nodetools repair / scrub / rebuild_indexes options, none seem to 
 make a difference.
 - Doing a 'update column family whatever with column_metadata=[]' then 
 repeating with my correct column_metadata definition.  This seems to fix the 
 problem (temporarily) until it comes back.
 The last time it happened I had just restarted cassandra, so it could be that 
 which is causing the issue, I've got the production system ok at the moment, 
 I will try restarting a bit later when its not being used and if I can get 
 the issue to reoccur I will add more information.
 The problem first manifested itself in 1.1.2, so I upgraded to 1.1.6, this 
 has not fixed it.
 Here is an example of the create column family I'm using for one of the CFs 
 that affected:
 create column family region
   with column_type = 'Standard'
   and comparator = 'UTF8Type'
   and default_validation_class = 'BytesType'
   and key_validation_class = 'UTF8Type'
   and read_repair_chance = 0.1
   and dclocal_read_repair_chance = 0.0
   and gc_grace = 864000
   and min_compaction_threshold = 4
   and max_compaction_threshold = 32
   and replicate_on_write = true
   and compaction_strategy = 
 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
   and caching = 'KEYS_ONLY'
   and column_metadata = [
 
 {column_name : 'label',
 validation_class : UTF8Type},
 
 {column_name : 'countryCode',
 validation_class : UTF8Type,
 index_name : 'region_countryCode_idx',
 index_type : 0},
 
 ]
   and compression_options = {'sstable_compression' : 
 'org.apache.cassandra.io.compress.SnappyCompressor'};
 I've noticed that CASSANDRA-4785 looks similar, in my case once the system 
 has the problem, it doesn't go away until I fix it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-4973) Secondary Index stops returning rows

2012-11-20 Thread Daniel Strawson (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-4973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13500994#comment-13500994
 ] 

Daniel Strawson commented on CASSANDRA-4973:


Hi sorry about this - the column family definition I gave in the description 
was wrong - I've just checked and it was off a different machine, I'd made a 
change to caching to turn on row caching.  Could this be something to do with 
the issue?  Looking through the current schema, the column families where I've 
noticed the index disappear all are onces where I have turned caching on.  I've 
just checked a CF that I hadn't looked at before but has caching on and as 
indexes and it's giving the same issue, so whilst this isn't proof, the 
evidence points this way.

Here is the current output from 'show schema' for the CF 'region' :

create column family region
  with column_type = 'Standard'
  and comparator = 'UTF8Type'
  and default_validation_class = 'BytesType'
  and key_validation_class = 'UTF8Type'
  and read_repair_chance = 0.1
  and dclocal_read_repair_chance = 0.0
  and gc_grace = 864000
  and min_compaction_threshold = 4
  and max_compaction_threshold = 32
  and replicate_on_write = true
  and compaction_strategy = 
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
  and caching = 'ALL'
  and column_metadata = [
{column_name : 'countryCode',
validation_class : UTF8Type,
index_name : 'region_countryCode_idx',
index_type : 0},
{column_name : 'label',
validation_class : UTF8Type}]
  and compression_options = {'sstable_compression' : 
'org.apache.cassandra.io.compress.SnappyCompressor'};


 Secondary Index stops returning rows
 

 Key: CASSANDRA-4973
 URL: https://issues.apache.org/jira/browse/CASSANDRA-4973
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 1.1.2, 1.1.6
 Environment: Centos 6.3, Java 1.6.0_35, cass. 1.1.2 upgraded to 1.1.6
Reporter: Daniel Strawson

 I've been using cassandra on a project for a little while in development and 
 have recently suddenly started having an issue where the secondary index 
 stops working, this is happening on my new production system, we are not yet 
 live.   Things work ok one moment, then suddenly queries to the cf through 
 the secondary index stop returning data.  I've seen it happen on 3 CFs. I've 
 tried:
 - various nodetools repair / scrub / rebuild_indexes options, none seem to 
 make a difference.
 - Doing a 'update column family whatever with column_metadata=[]' then 
 repeating with my correct column_metadata definition.  This seems to fix the 
 problem (temporarily) until it comes back.
 The last time it happened I had just restarted cassandra, so it could be that 
 which is causing the issue, I've got the production system ok at the moment, 
 I will try restarting a bit later when its not being used and if I can get 
 the issue to reoccur I will add more information.
 The problem first manifested itself in 1.1.2, so I upgraded to 1.1.6, this 
 has not fixed it.
 Here is an example of the create column family I'm using for one of the CFs 
 that affected:
 create column family region
   with column_type = 'Standard'
   and comparator = 'UTF8Type'
   and default_validation_class = 'BytesType'
   and key_validation_class = 'UTF8Type'
   and read_repair_chance = 0.1
   and dclocal_read_repair_chance = 0.0
   and gc_grace = 864000
   and min_compaction_threshold = 4
   and max_compaction_threshold = 32
   and replicate_on_write = true
   and compaction_strategy = 
 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
   and caching = 'KEYS_ONLY'
   and column_metadata = [
 
 {column_name : 'label',
 validation_class : UTF8Type},
 
 {column_name : 'countryCode',
 validation_class : UTF8Type,
 index_name : 'region_countryCode_idx',
 index_type : 0},
 
 ]
   and compression_options = {'sstable_compression' : 
 'org.apache.cassandra.io.compress.SnappyCompressor'};
 I've noticed that CASSANDRA-4785 looks similar, in my case once the system 
 has the problem, it doesn't go away until I fix it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-4973) Secondary Index stops returning rows

2012-11-20 Thread Daniel Strawson (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-4973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13501077#comment-13501077
 ] 

Daniel Strawson commented on CASSANDRA-4973:


I've tried setting caching back to 'KEYS_ONLY' on the region CF, restarted and 
the secondary indexes continue working, so I'm fairly happy that this issue 
relates to the caching setting.  I will revert my other CFs back to the default 
setting, which is a good enough work around for me for the moment.

 Secondary Index stops returning rows
 

 Key: CASSANDRA-4973
 URL: https://issues.apache.org/jira/browse/CASSANDRA-4973
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 1.1.2, 1.1.6
 Environment: Centos 6.3, Java 1.6.0_35, cass. 1.1.2 upgraded to 1.1.6
Reporter: Daniel Strawson

 I've been using cassandra on a project for a little while in development and 
 have recently suddenly started having an issue where the secondary index 
 stops working, this is happening on my new production system, we are not yet 
 live.   Things work ok one moment, then suddenly queries to the cf through 
 the secondary index stop returning data.  I've seen it happen on 3 CFs. I've 
 tried:
 - various nodetools repair / scrub / rebuild_indexes options, none seem to 
 make a difference.
 - Doing a 'update column family whatever with column_metadata=[]' then 
 repeating with my correct column_metadata definition.  This seems to fix the 
 problem (temporarily) until it comes back.
 The last time it happened I had just restarted cassandra, so it could be that 
 which is causing the issue, I've got the production system ok at the moment, 
 I will try restarting a bit later when its not being used and if I can get 
 the issue to reoccur I will add more information.
 The problem first manifested itself in 1.1.2, so I upgraded to 1.1.6, this 
 has not fixed it.
 Here is an example of the create column family I'm using for one of the CFs 
 that affected:
 create column family region
   with column_type = 'Standard'
   and comparator = 'UTF8Type'
   and default_validation_class = 'BytesType'
   and key_validation_class = 'UTF8Type'
   and read_repair_chance = 0.1
   and dclocal_read_repair_chance = 0.0
   and gc_grace = 864000
   and min_compaction_threshold = 4
   and max_compaction_threshold = 32
   and replicate_on_write = true
   and compaction_strategy = 
 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
   and caching = 'KEYS_ONLY'
   and column_metadata = [
 
 {column_name : 'label',
 validation_class : UTF8Type},
 
 {column_name : 'countryCode',
 validation_class : UTF8Type,
 index_name : 'region_countryCode_idx',
 index_type : 0},
 
 ]
   and compression_options = {'sstable_compression' : 
 'org.apache.cassandra.io.compress.SnappyCompressor'};
 I've noticed that CASSANDRA-4785 looks similar, in my case once the system 
 has the problem, it doesn't go away until I fix it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-4973) Secondary Index stops returning rows

2012-11-19 Thread Jonathan Ellis (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-4973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13500285#comment-13500285
 ] 

Jonathan Ellis commented on CASSANDRA-4973:
---

Are there errors in your system log?

 Secondary Index stops returning rows
 

 Key: CASSANDRA-4973
 URL: https://issues.apache.org/jira/browse/CASSANDRA-4973
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 1.1.2, 1.1.6
 Environment: Centos 6.3, Java 1.6.0_35, cass. 1.1.2 upgraded to 1.1.6
Reporter: Daniel Strawson

 I've been using cassandra on a project for a little while in development and 
 have recently suddenly started having an issue where the secondary index 
 stops working, this is happening on my new production system, we are not yet 
 live.   Things work ok one moment, then suddenly queries to the cf through 
 the secondary index stop returning data.  I've seen it happen on 3 CFs. I've 
 tried:
 - various nodetools repair / scrub / rebuild_indexes options, none seem to 
 make a difference.
 - Doing a 'update column family whatever with column_metadata=[]' then 
 repeating with my correct column_metadata definition.  This seems to fix the 
 problem (temporarily) until it comes back.
 The last time it happened I had just restarted cassandra, so it could be that 
 which is causing the issue, I've got the production system ok at the moment, 
 I will try restarting a bit later when its not being used and if I can get 
 the issue to reoccur I will add more information.
 The problem first manifested itself in 1.1.2, so I upgraded to 1.1.6, this 
 has not fixed it.
 Here is an example of the create column family I'm using for one of the CFs 
 that affected:
 create column family region
   with column_type = 'Standard'
   and comparator = 'UTF8Type'
   and default_validation_class = 'BytesType'
   and key_validation_class = 'UTF8Type'
   and read_repair_chance = 0.1
   and dclocal_read_repair_chance = 0.0
   and gc_grace = 864000
   and min_compaction_threshold = 4
   and max_compaction_threshold = 32
   and replicate_on_write = true
   and compaction_strategy = 
 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
   and caching = 'KEYS_ONLY'
   and column_metadata = [
 
 {column_name : 'label',
 validation_class : UTF8Type},
 
 {column_name : 'countryCode',
 validation_class : UTF8Type,
 index_name : 'region_countryCode_idx',
 index_type : 0},
 
 ]
   and compression_options = {'sstable_compression' : 
 'org.apache.cassandra.io.compress.SnappyCompressor'};
 I've noticed that CASSANDRA-4785 looks similar, in my case once the system 
 has the problem, it doesn't go away until I fix it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-4973) Secondary Index stops returning rows

2012-11-19 Thread Daniel Strawson (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-4973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13500353#comment-13500353
 ] 

Daniel Strawson commented on CASSANDRA-4973:


Sorry I should have added something about that, I did get an error, but don't 
recall how I produced it or if it was at the point of querying the sec. index, 
so was hoping to put something clearer up when the issue comes back.  At one 
point the system got into a state where queries were failing, but a restart 
fixed it.  That and the upgrade meant that I didn't really want to put up   In 
the mean time, this is the last exception in my system.log and I'm pretty sure 
would have been there  when the issue was happening or arounds abouts.

ERROR [Thrift:1] 2012-11-19 11:15:41,172 AbstractCassandraDaemon.java (line 
134) Exception in thread Thread[Thrift:1,5,main]java.io.IOError: 
java.io.EOFException
at 
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:64)
at 
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66)
at 
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:78)
at 
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at 
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:63)
at 
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1331)
at 
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1193)
at 
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1123)
at 
org.apache.cassandra.db.index.keys.KeysSearcher$1.computeNext(KeysSearcher.java:144)
at 
org.apache.cassandra.db.index.keys.KeysSearcher$1.computeNext(KeysSearcher.java:113)
at 
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
at 
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
at 
org.apache.cassandra.db.ColumnFamilyStore.filter(ColumnFamilyStore.java:1440)
at 
org.apache.cassandra.db.index.keys.KeysSearcher.search(KeysSearcher.java:88)
at 
org.apache.cassandra.db.index.SecondaryIndexManager.search(SecondaryIndexManager.java:595)
at 
org.apache.cassandra.db.ColumnFamilyStore.search(ColumnFamilyStore.java:1429)
at 
org.apache.cassandra.service.RangeSliceVerbHandler.executeLocally(RangeSliceVerbHandler.java:48)
at 
org.apache.cassandra.service.StorageProxy.getRangeSlice(StorageProxy.java:867)
at 
org.apache.cassandra.thrift.CassandraServer.get_indexed_slices(CassandraServer.java:811)
at 
org.apache.cassandra.thrift.Cassandra$Processor$get_indexed_slices.getResult(Cassandra.java:3056)
at 
org.apache.cassandra.thrift.Cassandra$Processor$get_indexed_slices.getResult(Cassandra.java:3044)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
at 
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.EOFException
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:399)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:377)
at 
org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:324)
at 
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:398)
at 
org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:380)
at 
org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:54)
... 26 more



 Secondary Index stops returning rows
 

 Key: CASSANDRA-4973
 URL: https://issues.apache.org/jira/browse/CASSANDRA-4973
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 1.1.2, 1.1.6
 Environment: Centos 6.3, Java 1.6.0_35, cass. 1.1.2 upgraded to 1.1.6
Reporter: Daniel Strawson

 I've been using cassandra on a project for a little while in development and 
 have recently suddenly started having an issue where the secondary index 
 stops working, this is happening on my new production system, we are not yet 
 live.   Things work ok one moment, then suddenly queries to the cf through 
 the secondary index stop returning data.  I've seen it happen on 3

[jira] [Commented] (CASSANDRA-4973) Secondary Index stops returning rows

2012-11-19 Thread Daniel Strawson (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-4973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13500405#comment-13500405
 ] 

Daniel Strawson commented on CASSANDRA-4973:


ok - I've just done the following:

1. run the query get region where countryCode=GB.  This returned 13 rows OK.
2. stopped and started cassandra (kill pid).
3. ran the same query.  This returned no rows.

At this point there are no errors in the log file -

Next I'm doing the following:

4. run update column family region with column_metadata=[];
5. run update column family region with column_metadata=[ ...] ; (see original 
schema above)
6. ran the same query, it returned 13 rows OK.

here is the log file from the point where the restart happened:

{code}
 INFO [Thread-2] 2012-11-19 17:06:07,208 CassandraDaemon.java (line 212) 
Listening for thrift clients...
 INFO [MemoryMeter:1] 2012-11-19 17:11:59,162 Memtable.java (line 213) 
CFS(Keyspace='Twickets', ColumnFamily='tweet') liveRatio is 4.0713678844519965 
(just-counted was 4.0713678844519965).  calculation took 8ms for 103 columns
 INFO [MemoryMeter:1] 2012-11-19 17:11:59,164 Memtable.java (line 213) 
CFS(Keyspace='Twickets', ColumnFamily='tweet.tweet_737461747573_idx') liveRatio 
is 25.840086759341418 (just-counted was 21.256038647342994).  calculation took 
1ms for 9 columns
 INFO [MemoryMeter:1] 2012-11-19 17:11:59,164 Memtable.java (line 213) 
CFS(Keyspace='Twickets', ColumnFamily='tweet.tweet_status_idx') liveRatio is 
9.66183574879227 (just-counted was 9.66183574879227).  calculation took 0ms for 
9 columns
 INFO [MemoryMeter:1] 2012-11-19 17:11:59,171 Memtable.java (line 213) 
CFS(Keyspace='Twickets', ColumnFamily='tweet.tweet_tweetTwitterId_idx') 
liveRatio is 40.57971014492754 (just-counted was 40.57971014492754).  
calculation took 6ms for 9 columns
 INFO [RPC-Thread:95] 2012-11-19 17:13:06,280 MigrationManager.java (line 164) 
Update ColumnFamily 'Twickets/region' From 
org.apache.cassandra.config.CFMetaData@5922da6a[cfId=1048,ksName=Twickets,cfName=region,cfType=Standard,comparator=org.apache.cassandra.db.marshal.UTF8Type,subcolumncomparator=null,comment=,readRepairChance=0.1,dclocalReadRepairChance=0.0,replicateOnWrite=true,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.UTF8Type,minCompactionThreshold=4,maxCompactionThreshold=32,keyAlias=null,columnAliases=[],valueAlias=null,column_metadata={java.nio.HeapByteBuffer[pos=0
 lim=11 cap=11]=ColumnDefinition{name=636f756e747279436f6465, 
validator=org.apache.cassandra.db.marshal.UTF8Type, index_type=KEYS, 
index_name='region_countryCode_idx'}, java.nio.HeapByteBuffer[pos=0 lim=5 
cap=5]=ColumnDefinition{name=6c6162656c, 
validator=org.apache.cassandra.db.marshal.UTF8Type, index_type=null, 
index_name='null'}},compactionStrategyClass=class 
org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionOptions={sstable_compression=org.apache.cassandra.io.compress.SnappyCompressor},bloomFilterFpChance=null,caching=ALL]
 To 
org.apache.cassandra.config.CFMetaData@7f3192dd[cfId=1048,ksName=Twickets,cfName=region,cfType=Standard,comparator=org.apache.cassandra.db.marshal.UTF8Type,subcolumncomparator=null,comment=,readRepairChance=0.1,dclocalReadRepairChance=0.0,replicateOnWrite=true,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.UTF8Type,minCompactionThreshold=4,maxCompactionThreshold=32,keyAlias=null,columnAliases=[],valueAlias=null,column_metadata={},compactionStrategyClass=class
 
org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionOptions={sstable_compression=org.apache.cassandra.io.compress.SnappyCompressor},bloomFilterFpChance=null,caching=ALL]
 INFO [MigrationStage:1] 2012-11-19 17:13:06,329 ColumnFamilyStore.java (line 
659) Enqueuing flush of Memtable-schema_columnfamilies@216957991(1170/1462 
serialized/live bytes, 20 ops)
 INFO [FlushWriter:2] 2012-11-19 17:13:06,330 Memtable.java (line 264) Writing 
Memtable-schema_columnfamilies@216957991(1170/1462 serialized/live bytes, 20 
ops)
 INFO [FlushWriter:2] 2012-11-19 17:13:06,547 Memtable.java (line 305) 
Completed flushing 
/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hf-100-Data.db
 (1236 bytes) for commitlog position ReplayPosition(segmentId=1353344762859, 
position=26876)
 INFO [MigrationStage:1] 2012-11-19 17:13:06,548 ColumnFamilyStore.java (line 
659) Enqueuing flush of Memtable-schema_columns@284615223(534/667 
serialized/live bytes, 10 ops)
 INFO [FlushWriter:2] 2012-11-19 17:13:06,549 Memtable.java (line 264) Writing 
Memtable-schema_columns@284615223(534/667 serialized/live bytes, 10 ops)
 INFO [CompactionExecutor:9] 2012-11-19 17:13:06,559 CompactionTask.java (line 
109) Compacting