date:20141201

[
https://issues.apache.org/jira/browse/CASSANDRA-7438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14229284#comment-14229284
]

Robert Stupp edited comment on CASSANDRA-7438 at 12/1/14 9:46 AM:
--

Have pushed the latest changes of OHC to https://github.com/snazy/ohc. It has
been nearly completely rewritten.

Architecture (in brief):
* OHC consists of multiple segments (default: 2 x #CPUs). Less segments leads
to more contention, more segments gives no measurable improvement.
* Each segment consists of an off-heap-hash-map (defaults: table-size=8192,
load-factor=.75). (The hash table requires 8 bytes per bucket)
* Hash entries in a bucket are organized in a double-linked-list
* LRU replacement policy is built-in via its own double-linked-list
* Critical sections that mutually lock a segment are pretty short (code + CPU)
- just a 'synchronized' keyword, no StampedLock/ReentrantLock
* Capacity for the cache is configured globally and managed locally in each
segment
* Eviction (or replacement or cleanup) is triggered when free capacity goes
below a trigger value and cleans up to a target free capacity
* Uses murmur hash on serialized key. Most significant bits are used to find
the segment, least significant bits for the segment's hash map.

Non-production relevant stuff:
* Allows to start off-heap access in debug mode, that checks for accesses
outside of allocated region and produces exceptions instead of SIGSEGV or
jemalloc errors
* ohc-benchmark updated to reflect changes

But replacement of LRU with something else is out of scope of this ticket and
should be done with real workloads in C* - although the last one is just a
additional config parameter.

IMO we should add a per-table option that configures whether the row cache
receives data on reads+writes or just on reads. Might prevent garbage in the
cache caused by write heavy tables.

{{Unsafe.allocateMemory()}} gives about 5-10% performance improvement compared
to jemalloc. Reason fot it might be that JNA library (which has some
synchronized blocks in it).

IMO OHC is ready to be merged into C* code base.

Edit: the fact that there are two double-linked lists is a left-over of several
experiments and it will be merged into one double-linked-list. It needs to be
and will be fixed.

was (Author: snazy):
Have pushed the latest changes of OHC to https://github.com/snazy/ohc. It has
been nearly completely rewritten.

About replacement policy: Currently LRU is built in - but I'm not really sold
on LRU as is. Alternatives could be
* timestamp (not sold on this either - basically the same as LRU)
* LIRS (https://en.wikipedia.org/wiki/LIRS_caching_algorithm), big overhead
(space)
* 2Q (counts accesses, divides counter regularly)
* LRU+random (50/50) (may give the same result than LIRS, but without LIRS'
overhead)
But replacement of LRU with something else is out of scope of this ticket and
should be done with real workloads in C* - although the last one is just a
additional config parameter.

IMO we should add a per-table option that configures whether the row cache
receives data on reads+writes or just on reads. Might prevent garbage in the
cache caused by write heavy tables.

{{Unsafe.allocateMemory()}} gives about 5-10% performance improvement compared

[jira] [Commented] (CASSANDRA-8387) Schema inconsistency (cached vs schema_columnfamilies)

2014-12-01 Thread Marcus Olsson (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14229613#comment-14229613
 ] 

Marcus Olsson commented on CASSANDRA-8387:
--

{quote}
Time is not the problem. Your issue is a consequence of CASSANDRA-5202, that 
made table uuids non-deterministic.

I don't see a good way to fix this in 2.1. I will try to handle this scenario 
in CASSANDRA-6038, with the new schema change protocol, but even then, I don't 
see an immediate solution - yet.
{quote}
Could one solution be to add some way for CREATE to use LWT?

{quote}
For now, you should wait for schema agreement before issuing any CREATE 
requests like that. I believe the java driver has a method for it.
{quote}
Ok, is that an exposed method of the driver or is it a part of the CREATE query 
itself? I can't seem to find it.

Also, we are using multiple clients that might execute the same CREATE 
query(with IF NOT EXISTS).

 Schema inconsistency (cached vs schema_columnfamilies)
 --

 Key: CASSANDRA-8387
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8387
 Project: Cassandra
  Issue Type: Bug
  Components: Core
 Environment: C* 2.1.1 3-node cluster
Reporter: Marcus Olsson

 While running some tests on a 3-node cluster running C* 2.1.1 we encountered 
 a problem creating the same table schema twice(on different nodes). One thing 
 to note is that one of the nodes clock was ~4 seconds behind the others, but 
 I don't think that's the problem since the exception was reproduced here 
 aswell: http://www.mail-archive.com/user@cassandra.apache.org/msg39560.html.
 While running the same create table statement more than once(on different 
 clients) the logs outputted this on one of the nodes:
 {noformat}
 (node x.x.x.1):
 2014-11-25T16:11:44.651+0100  INFO [SharedPool-Worker-2] 
 MigrationManager.java:248 Create new ColumnFamily: 
 org.apache.cassandra.config.CFMetaData@45c290de[cfId=5e334b40-74b5-11e4-b1b6-017ad0689f5d,ksName=test,cfName=test,cfType=Standard,comparator=org.apache.cassandra.db.marshal.UTF8Type,comment=,readRepairChance=0.0,dcLocalReadRepairChance=0.1,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.UTF8Type,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=id,
  type=org.apache.cassandra.db.marshal.UTF8Type, kind=CLUSTERING_COLUMN, 
 componentIndex=null, indexName=null, indexType=null}, 
 ColumnDefinition{name=key, type=org.apache.cassandra.db.marshal.UTF8Type, 
 kind=PARTITION_KEY, componentIndex=null, indexName=null, indexType=null}, 
 ColumnDefinition{name=value, type=org.apache.cassandra.db.marshal.BytesType, 
 kind=COMPACT_VALUE, componentIndex=null, indexName=null, 
 indexType=null}],compactionStrategyClass=class 
 org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=0,caching={keys:ALL,
  
 rows_per_partition:NONE},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,droppedColumns={},triggers=[],isDense=true]
 ...
 2014-11-25T16:11:44.667+0100  INFO [MigrationStage:1] DefsTables.java:373 
 Loading 
 org.apache.cassandra.config.CFMetaData@40a1ee90[cfId=5bc7c980-74b5-11e4-9131-d9b94a3d8927,ksName=test,cfName=test,cfType=Standard,comparator=org.apache.cassandra.db.marshal.UTF8Type,comment=,readRepairChance=0.0,dcLocalReadRepairChance=0.1,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.UTF8Type,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=id,
  type=org.apache.cassandra.db.marshal.UTF8Type, kind=CLUSTERING_COLUMN, 
 componentIndex=null, indexName=null, indexType=null}, 
 ColumnDefinition{name=key, type=org.apache.cassandra.db.marshal.UTF8Type, 
 kind=PARTITION_KEY, componentIndex=null, indexName=null, indexType=null}, 
 ColumnDefinition{name=value, type=org.apache.cassandra.db.marshal.BytesType, 
 kind=COMPACT_VALUE, componentIndex=null, indexName=null, 
 indexType=null}],compactionStrategyClass=class 
 org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=0,caching={keys:ALL,
  
 rows_per_partition:NONE},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,droppedColumns={},triggers=[],isDense=true]
 ...
 java.lang.RuntimeException: 
 org.apache.cassandra.exceptions.ConfigurationException: Column family ID

[jira] [Comment Edited] (CASSANDRA-7438) Serializing Row cache alternative (Fully off heap)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-7438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14229284#comment-14229284
 ] 

Robert Stupp edited comment on CASSANDRA-7438 at 12/1/14 10:14 AM:
---

Have pushed the latest changes of OHC to https://github.com/snazy/ohc. It has 
been nearly completely rewritten.

Architecture (in brief):
* OHC consists of multiple segments (default: 2 x #CPUs). Less segments leads 
to more contention, more segments gives no measurable improvement.
* Each segment consists of an off-heap-hash-map (defaults: table-size=8192, 
load-factor=.75). (The hash table requires 8 bytes per bucket)
* Hash entries in a bucket are organized in a double-linked-list
* LRU replacement policy is built-in via its own double-linked-list
* Critical sections that mutually lock a segment are pretty short (code + CPU) 
- just a 'synchronized' keyword, no StampedLock/ReentrantLock
* Capacity for the cache is configured globally and managed locally in each 
segment
* Eviction (or replacement or cleanup) is triggered when free capacity goes 
below a trigger value and cleans up to a target free capacity
* Uses murmur hash on serialized key. Most significant bits are used to find 
the segment, least significant bits for the segment's hash map. 

Non-production relevant stuff:
* Allows to start off-heap access in debug mode, that checks for accesses 
outside of allocated region and produces exceptions instead of SIGSEGV or 
jemalloc errors
* ohc-benchmark updated to reflect changes

About replacement policy: Currently LRU is built in - but I'm not really sold 
on LRU as is. Alternatives could be
* timestamp (not sold on this either - basically the same as LRU)
* LIRS (https://en.wikipedia.org/wiki/LIRS_caching_algorithm), big overhead 
(space)
* 2Q (counts accesses, divides counter regularly)
* LRU+random (50/50) (may give the same result than LIRS, but without LIRS' 
overhead)

But replacement of LRU with something else is out of scope of this ticket and 
should be done with real workloads in C* - although the last one is just a 
additional config parameter.

IMO we should add a per-table option that configures whether the row cache 
receives data on reads+writes or just on reads. Might prevent garbage in the 
cache caused by write heavy tables.

{{Unsafe.allocateMemory()}} gives about 5-10% performance improvement compared 
to jemalloc. Reason fot it might be that JNA library (which has some 
synchronized blocks in it).

IMO OHC is ready to be merged into C* code base.

Edit2: (remove edit1)


was (Author: snazy):
Have pushed the latest changes of OHC to https://github.com/snazy/ohc. It has 
been nearly completely rewritten.

Architecture (in brief):
* OHC consists of multiple segments (default: 2 x #CPUs). Less segments leads 
to more contention, more segments gives no measurable improvement.
* Each segment consists of an off-heap-hash-map (defaults: table-size=8192, 
load-factor=.75). (The hash table requires 8 bytes per bucket)
* Hash entries in a bucket are organized in a double-linked-list
* LRU replacement policy is built-in via its own double-linked-list
* Critical sections that mutually lock a segment are pretty short (code + CPU) 
- just a 'synchronized' keyword, no StampedLock/ReentrantLock
* Capacity for the cache is configured globally and managed locally in each 
segment
* Eviction (or replacement or cleanup) is triggered when free capacity goes 
below a trigger value and cleans up to a target free capacity
* Uses murmur hash on serialized key. Most significant bits are used to find 
the segment, least significant bits for the segment's hash map. 

Non-production relevant stuff:
* Allows to start off-heap access in debug mode, that checks for accesses 
outside of allocated region and produces exceptions instead of SIGSEGV or 
jemalloc errors
* ohc-benchmark updated to reflect changes

About replacement policy: Currently LRU is built in - but I'm not really sold 
on LRU as is. Alternatives could be
* timestamp (not sold on this either - basically the same as LRU)
* LIRS (https://en.wikipedia.org/wiki/LIRS_caching_algorithm), big overhead 
(space)
* 2Q (counts accesses, divides counter regularly)
* LRU+random (50/50) (may give the same result than LIRS, but without LIRS' 
overhead)

But replacement of LRU with something else is out of scope of this ticket and 
should be done with real workloads in C* - although the last one is just a 
additional config parameter.

IMO we should add a per-table option that configures whether the row cache 
receives data on reads+writes or just on reads. Might prevent garbage in the 
cache caused by write heavy tables.

{{Unsafe.allocateMemory()}} gives about 5-10% performance improvement compared 
to jemalloc. Reason fot it might be that JNA library (which has some 
synchronized blocks in it).

IMO OHC is ready to be merged into C* code base.

[jira] [Commented] (CASSANDRA-8387) Schema inconsistency (cached vs schema_columnfamilies)

2014-12-01 Thread Jens-U. Mozdzen (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14229636#comment-14229636
 ] 

Jens-U. Mozdzen commented on CASSANDRA-8387:


 Ok, is that an exposed method of the driver or is it a part of the CREATE 
 query itself? I can't seem to find it.

it seems the method was introduced with cassandra-driver-core 2.1.3:

--- cut here ---
Builder newClusterBuilder = Cluster.builder();

newClusterBuilder.withMaxSchemaAgreementWaitSeconds( 30);
--- cut here ---

 Schema inconsistency (cached vs schema_columnfamilies)
 --

 Key: CASSANDRA-8387
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8387
 Project: Cassandra
  Issue Type: Bug
  Components: Core
 Environment: C* 2.1.1 3-node cluster
Reporter: Marcus Olsson

 While running some tests on a 3-node cluster running C* 2.1.1 we encountered 
 a problem creating the same table schema twice(on different nodes). One thing 
 to note is that one of the nodes clock was ~4 seconds behind the others, but 
 I don't think that's the problem since the exception was reproduced here 
 aswell: http://www.mail-archive.com/user@cassandra.apache.org/msg39560.html.
 While running the same create table statement more than once(on different 
 clients) the logs outputted this on one of the nodes:
 {noformat}
 (node x.x.x.1):
 2014-11-25T16:11:44.651+0100  INFO [SharedPool-Worker-2] 
 MigrationManager.java:248 Create new ColumnFamily: 
 org.apache.cassandra.config.CFMetaData@45c290de[cfId=5e334b40-74b5-11e4-b1b6-017ad0689f5d,ksName=test,cfName=test,cfType=Standard,comparator=org.apache.cassandra.db.marshal.UTF8Type,comment=,readRepairChance=0.0,dcLocalReadRepairChance=0.1,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.UTF8Type,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=id,
  type=org.apache.cassandra.db.marshal.UTF8Type, kind=CLUSTERING_COLUMN, 
 componentIndex=null, indexName=null, indexType=null}, 
 ColumnDefinition{name=key, type=org.apache.cassandra.db.marshal.UTF8Type, 
 kind=PARTITION_KEY, componentIndex=null, indexName=null, indexType=null}, 
 ColumnDefinition{name=value, type=org.apache.cassandra.db.marshal.BytesType, 
 kind=COMPACT_VALUE, componentIndex=null, indexName=null, 
 indexType=null}],compactionStrategyClass=class 
 org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=0,caching={keys:ALL,
  
 rows_per_partition:NONE},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,droppedColumns={},triggers=[],isDense=true]
 ...
 2014-11-25T16:11:44.667+0100  INFO [MigrationStage:1] DefsTables.java:373 
 Loading 
 org.apache.cassandra.config.CFMetaData@40a1ee90[cfId=5bc7c980-74b5-11e4-9131-d9b94a3d8927,ksName=test,cfName=test,cfType=Standard,comparator=org.apache.cassandra.db.marshal.UTF8Type,comment=,readRepairChance=0.0,dcLocalReadRepairChance=0.1,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.UTF8Type,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=id,
  type=org.apache.cassandra.db.marshal.UTF8Type, kind=CLUSTERING_COLUMN, 
 componentIndex=null, indexName=null, indexType=null}, 
 ColumnDefinition{name=key, type=org.apache.cassandra.db.marshal.UTF8Type, 
 kind=PARTITION_KEY, componentIndex=null, indexName=null, indexType=null}, 
 ColumnDefinition{name=value, type=org.apache.cassandra.db.marshal.BytesType, 
 kind=COMPACT_VALUE, componentIndex=null, indexName=null, 
 indexType=null}],compactionStrategyClass=class 
 org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=0,caching={keys:ALL,
  
 rows_per_partition:NONE},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,droppedColumns={},triggers=[],isDense=true]
 ...
 java.lang.RuntimeException: 
 org.apache.cassandra.exceptions.ConfigurationException: Column family ID 
 mismatch (found 5e334b40-74b5-11e4-b1b6-017ad0689f5d; expected 
 5bc7c980-74b5-11e4-9131-d9b94a3d8927)
 at 
 org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:1171) 
 ~[apache-cassandra-2.1.1.jar:2.1.1]
 at 
 org.apache.cassandra.db.DefsTables.updateColumnFamily(DefsTables.java:422) 
 ~[apache-cassandra-2.1.1.jar:2.1.1]
 at

[jira] [Comment Edited] (CASSANDRA-7438) Serializing Row cache alternative (Fully off heap)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-7438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14229284#comment-14229284
 ] 

Robert Stupp edited comment on CASSANDRA-7438 at 12/1/14 11:00 AM:
---

Have pushed the latest changes of OHC to https://github.com/snazy/ohc. It has 
been nearly completely rewritten.

Architecture (in brief):
* OHC consists of multiple segments (default: 2 x #CPUs). Less segments leads 
to more contention, more segments gives no measurable improvement.
* Each segment consists of an off-heap-hash-map (defaults: table-size=8192, 
load-factor=.75). (The hash table requires 8 bytes per bucket)
* Hash entries in a bucket are organized in a single-linked-list
* LRU replacement policy is built-in via its own double-linked-list
* Critical sections that mutually lock a segment are pretty short (code + CPU) 
- just a 'synchronized' keyword, no StampedLock/ReentrantLock
* Capacity for the cache is configured globally and managed locally in each 
segment
* Eviction (or replacement or cleanup) is triggered when free capacity goes 
below a trigger value and cleans up to a target free capacity
* Uses murmur hash on serialized key. Most significant bits are used to find 
the segment, least significant bits for the segment's hash map. 

Non-production relevant stuff:
* Allows to start off-heap access in debug mode, that checks for accesses 
outside of allocated region and produces exceptions instead of SIGSEGV or 
jemalloc errors
* ohc-benchmark updated to reflect changes

About replacement policy: Currently LRU is built in - but I'm not really sold 
on LRU as is. Alternatives could be
* timestamp (not sold on this either - basically the same as LRU)
* LIRS (https://en.wikipedia.org/wiki/LIRS_caching_algorithm), big overhead 
(space)
* 2Q (counts accesses, divides counter regularly)
* LRU+random (50/50) (may give the same result than LIRS, but without LIRS' 
overhead)

But replacement of LRU with something else is out of scope of this ticket and 
should be done with real workloads in C* - although the last one is just a 
additional config parameter.

IMO we should add a per-table option that configures whether the row cache 
receives data on reads+writes or just on reads. Might prevent garbage in the 
cache caused by write heavy tables.

{{Unsafe.allocateMemory()}} gives about 5-10% performance improvement compared 
to jemalloc. Reason fot it might be that JNA library (which has some 
synchronized blocks in it).

IMO OHC is ready to be merged into C* code base.

Edit3: (sorry for the JIRA noise) - bucket linked list is only a 
single-linked-list - LRU linked list needs to be doubly linked


was (Author: snazy):
Have pushed the latest changes of OHC to https://github.com/snazy/ohc. It has 
been nearly completely rewritten.

Architecture (in brief):
* OHC consists of multiple segments (default: 2 x #CPUs). Less segments leads 
to more contention, more segments gives no measurable improvement.
* Each segment consists of an off-heap-hash-map (defaults: table-size=8192, 
load-factor=.75). (The hash table requires 8 bytes per bucket)
* Hash entries in a bucket are organized in a double-linked-list
* LRU replacement policy is built-in via its own double-linked-list
* Critical sections that mutually lock a segment are pretty short (code + CPU) 
- just a 'synchronized' keyword, no StampedLock/ReentrantLock
* Capacity for the cache is configured globally and managed locally in each 
segment
* Eviction (or replacement or cleanup) is triggered when free capacity goes 
below a trigger value and cleans up to a target free capacity
* Uses murmur hash on serialized key. Most significant bits are used to find 
the segment, least significant bits for the segment's hash map. 

Non-production relevant stuff:
* Allows to start off-heap access in debug mode, that checks for accesses 
outside of allocated region and produces exceptions instead of SIGSEGV or 
jemalloc errors
* ohc-benchmark updated to reflect changes

About replacement policy: Currently LRU is built in - but I'm not really sold 
on LRU as is. Alternatives could be
* timestamp (not sold on this either - basically the same as LRU)
* LIRS (https://en.wikipedia.org/wiki/LIRS_caching_algorithm), big overhead 
(space)
* 2Q (counts accesses, divides counter regularly)
* LRU+random (50/50) (may give the same result than LIRS, but without LIRS' 
overhead)

But replacement of LRU with something else is out of scope of this ticket and 
should be done with real workloads in C* - although the last one is just a 
additional config parameter.

IMO we should add a per-table option that configures whether the row cache 
receives data on reads+writes or just on reads. Might prevent garbage in the 
cache caused by write heavy tables.

{{Unsafe.allocateMemory()}} gives about 5-10% performance improvement compared 
to jemalloc. Reason fot it might be that JNA

[jira] [Commented] (CASSANDRA-8387) Schema inconsistency (cached vs schema_columnfamilies)

2014-12-01 Thread Marcus Olsson (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14229667#comment-14229667
 ] 

Marcus Olsson commented on CASSANDRA-8387:
--

{quote}
it seems the method was introduced with cassandra-driver-core 2.1.3:

— cut here —
Builder newClusterBuilder = Cluster.builder();

newClusterBuilder.withMaxSchemaAgreementWaitSeconds( 30);
— cut here —
{quote}
It seems that method is for setting the time the client is waiting for schema 
agreement after the CREATE request is executed which I belive won't help when 
there are multiple clients trying to create the same table?

 Schema inconsistency (cached vs schema_columnfamilies)
 --

 Key: CASSANDRA-8387
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8387
 Project: Cassandra
  Issue Type: Bug
  Components: Core
 Environment: C* 2.1.1 3-node cluster
Reporter: Marcus Olsson

 While running some tests on a 3-node cluster running C* 2.1.1 we encountered 
 a problem creating the same table schema twice(on different nodes). One thing 
 to note is that one of the nodes clock was ~4 seconds behind the others, but 
 I don't think that's the problem since the exception was reproduced here 
 aswell: http://www.mail-archive.com/user@cassandra.apache.org/msg39560.html.
 While running the same create table statement more than once(on different 
 clients) the logs outputted this on one of the nodes:
 {noformat}
 (node x.x.x.1):
 2014-11-25T16:11:44.651+0100  INFO [SharedPool-Worker-2] 
 MigrationManager.java:248 Create new ColumnFamily: 
 org.apache.cassandra.config.CFMetaData@45c290de[cfId=5e334b40-74b5-11e4-b1b6-017ad0689f5d,ksName=test,cfName=test,cfType=Standard,comparator=org.apache.cassandra.db.marshal.UTF8Type,comment=,readRepairChance=0.0,dcLocalReadRepairChance=0.1,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.UTF8Type,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=id,
  type=org.apache.cassandra.db.marshal.UTF8Type, kind=CLUSTERING_COLUMN, 
 componentIndex=null, indexName=null, indexType=null}, 
 ColumnDefinition{name=key, type=org.apache.cassandra.db.marshal.UTF8Type, 
 kind=PARTITION_KEY, componentIndex=null, indexName=null, indexType=null}, 
 ColumnDefinition{name=value, type=org.apache.cassandra.db.marshal.BytesType, 
 kind=COMPACT_VALUE, componentIndex=null, indexName=null, 
 indexType=null}],compactionStrategyClass=class 
 org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=0,caching={keys:ALL,
  
 rows_per_partition:NONE},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,droppedColumns={},triggers=[],isDense=true]
 ...
 2014-11-25T16:11:44.667+0100  INFO [MigrationStage:1] DefsTables.java:373 
 Loading 
 org.apache.cassandra.config.CFMetaData@40a1ee90[cfId=5bc7c980-74b5-11e4-9131-d9b94a3d8927,ksName=test,cfName=test,cfType=Standard,comparator=org.apache.cassandra.db.marshal.UTF8Type,comment=,readRepairChance=0.0,dcLocalReadRepairChance=0.1,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.UTF8Type,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=id,
  type=org.apache.cassandra.db.marshal.UTF8Type, kind=CLUSTERING_COLUMN, 
 componentIndex=null, indexName=null, indexType=null}, 
 ColumnDefinition{name=key, type=org.apache.cassandra.db.marshal.UTF8Type, 
 kind=PARTITION_KEY, componentIndex=null, indexName=null, indexType=null}, 
 ColumnDefinition{name=value, type=org.apache.cassandra.db.marshal.BytesType, 
 kind=COMPACT_VALUE, componentIndex=null, indexName=null, 
 indexType=null}],compactionStrategyClass=class 
 org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=0,caching={keys:ALL,
  
 rows_per_partition:NONE},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,droppedColumns={},triggers=[],isDense=true]
 ...
 java.lang.RuntimeException: 
 org.apache.cassandra.exceptions.ConfigurationException: Column family ID 
 mismatch (found 5e334b40-74b5-11e4-b1b6-017ad0689f5d; expected 
 5bc7c980-74b5-11e4-9131-d9b94a3d8927)
 at 
 org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:1171) 
 ~[apache-cassandra-2.1.1.jar:2.1.1]
 at 
 org.apache.cassandra.db.DefsTables.updateColumnFamily(DefsTables.java:422)

[jira] [Commented] (CASSANDRA-7688) Add data sizing to a system table

[
https://issues.apache.org/jira/browse/CASSANDRA-7688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14229701#comment-14229701
]

Piotr Kołaczkowski commented on CASSANDRA-7688:
---

It would be nice to know also the average partition size in the given table,
both in bytes and in number of CQL rows. This would be useful to set
appropriate fetch.size. Additionally, current split generation API does not
allow to set split size in terms of data size in bytes or number of CQL rows,
but only by number of partitions. Number of partitions doesn't make a nice
default, as partitions can vary greatly in size and are extremely use-case
dependent. So please, don't just copy current describe_splits_ex functionality
to the new driver, but *improve this*.

We really don't need the driver / Cassandra to do the splitting for us. Instead
we need to know:

1. estimate of total amount of data in the table in bytes
2. estimate of total number of CQL rows in the table
3. estimate of total number of partitions in the table

We're interested both in totals (whole cluster; logical sizes; i.e. without
replicas), and split by token-ranges by node (physical; incuding replicas).

Add data sizing to a system table
-

Key: CASSANDRA-7688
URL: https://issues.apache.org/jira/browse/CASSANDRA-7688
Project: Cassandra
Issue Type: New Feature
Reporter: Jeremiah Jordan
Fix For: 2.1.3

Currently you can't implement something similar to describe_splits_ex purely
from the a native protocol driver.
https://datastax-oss.atlassian.net/browse/JAVA-312 is open to expose easily
getting ownership information to a client in the java-driver. But you still
need the data sizing part to get splits of a given size. We should add the
sizing information to a system table so that native clients can get to it.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-7688) Add data sizing to a system table

[
https://issues.apache.org/jira/browse/CASSANDRA-7688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14229701#comment-14229701
]

Piotr Kołaczkowski edited comment on CASSANDRA-7688 at 12/1/14 12:01 PM:
-

We really don't need the driver / Cassandra to do the splitting for us. Instead
we need to know:

1. estimate of total amount of data in the table in bytes
2. estimate of total number of CQL rows in the table
3. estimate of total number of partitions in the table

We're interested both in totals (whole cluster; logical sizes; i.e. without
replicas), and split by token-ranges by node (physical; incuding replicas).

Note that this information is useful not just for Spark/Hadoop split
generation, but also things like e.g. SparkSQL optimizer so it knows how much
data will it have to process.

The next step would be providing column data histograms to guide predicate
selectivity.

was (Author: pkolaczk):
It would be nice to know also the average partition size in the given table,
both in bytes and in number of CQL rows. This would be useful to set
appropriate fetch.size. Additionally, current split generation API does not
allow to set split size in terms of data size in bytes or number of CQL rows,
but only by number of partitions. Number of partitions doesn't make a nice
default, as partitions can vary greatly in size and are extremely use-case
dependent. So please, don't just copy current describe_splits_ex functionality
to the new driver, but *improve this*.

We really don't need the driver / Cassandra to do the splitting for us. Instead
we need to know:

1. estimate of total amount of data in the table in bytes
2. estimate of total number of CQL rows in the table
3. estimate of total number of partitions in the table

We're interested both in totals (whole cluster; logical sizes; i.e. without
replicas), and split by token-ranges by node (physical; incuding replicas).

Add data sizing to a system table
-

Key: CASSANDRA-7688
URL: https://issues.apache.org/jira/browse/CASSANDRA-7688
Project: Cassandra
Issue Type: New Feature
Reporter: Jeremiah Jordan
Fix For: 2.1.3

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-7688) Add data sizing to a system table

[
https://issues.apache.org/jira/browse/CASSANDRA-7688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14229701#comment-14229701
]

Piotr Kołaczkowski edited comment on CASSANDRA-7688 at 12/1/14 12:03 PM:
-

We really don't need the driver / Cassandra to do the splitting for us. Instead
we need to know:

1. estimate of total amount of data in the table in bytes
2. estimate of total number of CQL rows in the table
3. estimate of total number of partitions in the table

We're interested both in totals (whole cluster; logical sizes; i.e. without
replicas), and split by token-ranges by node (physical; incuding replicas).

Note that this information is useful not just for Spark/Hadoop split
generation, but also things like e.g. SparkSQL optimizer so it knows how much
data will it have to process or to set appropriate fetch sizes when getting
data, etc.

The next step would be providing column data histograms to guide predicate
selectivity.

We really don't need the driver / Cassandra to do the splitting for us. Instead
we need to know:

1. estimate of total amount of data in the table in bytes
2. estimate of total number of CQL rows in the table
3. estimate of total number of partitions in the table

We're interested both in totals (whole cluster; logical sizes; i.e. without
replicas), and split by token-ranges by node (physical; incuding replicas).

Note that this information is useful not just for Spark/Hadoop split
generation, but also things like e.g. SparkSQL optimizer so it knows how much
data will it have to process.

The next step would be providing column data histograms to guide predicate
selectivity.

Add data sizing to a system table
-

Key: CASSANDRA-7688
URL: https://issues.apache.org/jira/browse/CASSANDRA-7688
Project: Cassandra
Issue Type: New Feature
Reporter: Jeremiah Jordan
Fix For: 2.1.3

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8341) Expose time spent in each thread pool


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14229712#comment-14229712
 ] 

Robert Stupp commented on CASSANDRA-8341:
-

Note: {{TheadMXBean.getCurrentThread...()}} definitely performs better than 
{{System.nanoTime()}} - at least on OSX with a single 8-core CPU.

 Expose time spent in each thread pool
 -

 Key: CASSANDRA-8341
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8341
 Project: Cassandra
  Issue Type: New Feature
  Components: Core
Reporter: Chris Lohfink
Priority: Minor
  Labels: metrics
 Attachments: 8341.patch, 8341v2.txt


 Can increment a counter with time spent in each queue.  This can provide 
 context on how much time is spent percentage wise in each stage.  
 Additionally can be used with littles law in future if ever want to try to 
 tune the size of the pools.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-8397) Support UPDATE with IN requirement for clustering key

2014-12-01 Thread Jens Rantil (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-8397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jens Rantil updated CASSANDRA-8397:
---
Summary: Support UPDATE with IN requirement for clustering key  (was: 
Support UPDATE with IN for clustering key)

 Support UPDATE with IN requirement for clustering key
 -

 Key: CASSANDRA-8397
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8397
 Project: Cassandra
  Issue Type: Wish
Reporter: Jens Rantil
Priority: Minor

 {noformat}
 CREATE TABLE tink.events (
 userid uuid,
 id timeuuid,
 content text,
 type text,
 PRIMARY KEY (userid, id)
 )
 # Add data
 cqlsh:tink UPDATE events SET content='Hello' WHERE 
 userid=57b47f85-56c4-4968-83cf-4c4e533944e9 AND id IN 
 (046e9da0-7945-11e4-a76f-770773bbbf7e, 046e0160-7945-11e4-a76f-770773bbbf7e);
 code=2200 [Invalid query] message=Invalid operator IN for PRIMARY KEY part 
 id
 {noformat}
 I was surprised this doesn't work.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-8397) Support UPDATE with IN for clustering key

2014-12-01 Thread Jens Rantil (JIRA)

Jens Rantil created CASSANDRA-8397:
--

 Summary: Support UPDATE with IN for clustering key
 Key: CASSANDRA-8397
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8397
 Project: Cassandra
  Issue Type: Wish
Reporter: Jens Rantil
Priority: Minor


{noformat}
CREATE TABLE tink.events (
userid uuid,
id timeuuid,
content text,
type text,
PRIMARY KEY (userid, id)
)

# Add data

cqlsh:tink UPDATE events SET content='Hello' WHERE 
userid=57b47f85-56c4-4968-83cf-4c4e533944e9 AND id IN 
(046e9da0-7945-11e4-a76f-770773bbbf7e, 046e0160-7945-11e4-a76f-770773bbbf7e);
code=2200 [Invalid query] message=Invalid operator IN for PRIMARY KEY part id
{noformat}

I was surprised this doesn't work.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-8397) Support UPDATE with IN requirement for clustering key

2014-12-01 Thread Jens Rantil (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-8397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jens Rantil updated CASSANDRA-8397:
---
Description: 
{noformat}
CREATE TABLE events (
userid uuid,
id timeuuid,
content text,
type text,
PRIMARY KEY (userid, id)
)

# Add data

cqlsh:mykeyspace UPDATE events SET content='Hello' WHERE 
userid=57b47f85-56c4-4968-83cf-4c4e533944e9 AND id IN 
(046e9da0-7945-11e4-a76f-770773bbbf7e, 046e0160-7945-11e4-a76f-770773bbbf7e);
code=2200 [Invalid query] message=Invalid operator IN for PRIMARY KEY part id
{noformat}

I was surprised this doesn't work.

  was:
{noformat}
CREATE TABLE tink.events (
userid uuid,
id timeuuid,
content text,
type text,
PRIMARY KEY (userid, id)
)

# Add data

cqlsh:tink UPDATE events SET content='Hello' WHERE 
userid=57b47f85-56c4-4968-83cf-4c4e533944e9 AND id IN 
(046e9da0-7945-11e4-a76f-770773bbbf7e, 046e0160-7945-11e4-a76f-770773bbbf7e);
code=2200 [Invalid query] message=Invalid operator IN for PRIMARY KEY part id
{noformat}

I was surprised this doesn't work.


 Support UPDATE with IN requirement for clustering key
 -

 Key: CASSANDRA-8397
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8397
 Project: Cassandra
  Issue Type: Wish
Reporter: Jens Rantil
Priority: Minor

 {noformat}
 CREATE TABLE events (
 userid uuid,
 id timeuuid,
 content text,
 type text,
 PRIMARY KEY (userid, id)
 )
 # Add data
 cqlsh:mykeyspace UPDATE events SET content='Hello' WHERE 
 userid=57b47f85-56c4-4968-83cf-4c4e533944e9 AND id IN 
 (046e9da0-7945-11e4-a76f-770773bbbf7e, 046e0160-7945-11e4-a76f-770773bbbf7e);
 code=2200 [Invalid query] message=Invalid operator IN for PRIMARY KEY part 
 id
 {noformat}
 I was surprised this doesn't work.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8341) Expose time spent in each thread pool

[
https://issues.apache.org/jira/browse/CASSANDRA-8341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14229780#comment-14229780
]

Benedict commented on CASSANDRA-8341:
-

SEPWorker already grabs the nanoTime on exiting and entering its spin phase, so
tracking this would be pretty much free (we'd need to check it once if we
swapped the executor we're working on without entering a spinning state).
Flushing pent up data is pretty trivial; you can set a max time to buffer, so
it ensures it's never more than a few seconds (or millis) out of date, say.
Enough to keep the cost too small to measure.

I'm a little dubious about tracking two completely different properties as the
same thing though. CPUTime cannot be composed with nanoTime sensibly, so we
either want to track one or the other across all executors. Since the other
executors are all the ones that do infrequent expensive work (which is
explicitly why they haven't been transitioned to SEP), tracking nanoTime on
them won't be an appreciable cost.

Expose time spent in each thread pool
-

Key: CASSANDRA-8341
URL: https://issues.apache.org/jira/browse/CASSANDRA-8341
Project: Cassandra
Issue Type: New Feature
Components: Core
Reporter: Chris Lohfink
Priority: Minor
Labels: metrics
Attachments: 8341.patch, 8341v2.txt

Can increment a counter with time spent in each queue. This can provide
context on how much time is spent percentage wise in each stage.
Additionally can be used with littles law in future if ever want to try to
tune the size of the pools.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8341) Expose time spent in each thread pool


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14229796#comment-14229796
 ] 

T Jake Luciani commented on CASSANDRA-8341:
---

I added some code to do this in a branch and have been meaning to break it out. 
 It adds LatencyMetrics to AbstractTracingAwareExecutorService and tracks time 
spent in queue for each FutureTask.

The code is here 
https://github.com/tjake/cassandra/compare/new-executor#diff-9860b2ae7a7e9e05e2165fd319f1398eL26

The SEPExecutor would need something similar added.

 Expose time spent in each thread pool
 -

 Key: CASSANDRA-8341
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8341
 Project: Cassandra
  Issue Type: New Feature
  Components: Core
Reporter: Chris Lohfink
Priority: Minor
  Labels: metrics
 Attachments: 8341.patch, 8341v2.txt


 Can increment a counter with time spent in each queue.  This can provide 
 context on how much time is spent percentage wise in each stage.  
 Additionally can be used with littles law in future if ever want to try to 
 tune the size of the pools.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8341) Expose time spent in each thread pool


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14229805#comment-14229805
 ] 

Benedict commented on CASSANDRA-8341:
-

Ah, that's a good question: are we talking about queue latency or time spent 
processing each queue? The two are very different, and it sounded like we were 
discussing the latter, but the ticket description does sound more like the 
former.

 Expose time spent in each thread pool
 -

 Key: CASSANDRA-8341
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8341
 Project: Cassandra
  Issue Type: New Feature
  Components: Core
Reporter: Chris Lohfink
Priority: Minor
  Labels: metrics
 Attachments: 8341.patch, 8341v2.txt


 Can increment a counter with time spent in each queue.  This can provide 
 context on how much time is spent percentage wise in each stage.  
 Additionally can be used with littles law in future if ever want to try to 
 tune the size of the pools.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-7827) Work around for output name restriction when using MultipleOutputs with CqlBulkOutputFormat


[ 
https://issues.apache.org/jira/browse/CASSANDRA-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14229809#comment-14229809
 ] 

Piotr Kołaczkowski commented on CASSANDRA-7827:
---

+1

 Work around for output name restriction when using MultipleOutputs with 
 CqlBulkOutputFormat
 ---

 Key: CASSANDRA-7827
 URL: https://issues.apache.org/jira/browse/CASSANDRA-7827
 Project: Cassandra
  Issue Type: Improvement
  Components: Hadoop
Reporter: Paul Pak
Assignee: Paul Pak
Priority: Minor
  Labels: cql3, hadoop
 Attachments: trunk-7827-v1.txt


 When using MultipleOutputs with CqlBulkOutputFormat, the column family names 
 to output to are restricted to only alphanumeric characters due to the logic 
 found in MultipleOutputs.checkNamedOutputName(). This will provide a way to 
 alias any column family name to a MultipleOutputs compatible output name, so 
 that column family names won't be artificially restricted when using 
 MultipleOutputs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-2388) ColumnFamilyRecordReader fails for a given split because a host is down, even if records could reasonably be read from other replica.


[ 
https://issues.apache.org/jira/browse/CASSANDRA-2388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14229812#comment-14229812
 ] 

Piotr Kołaczkowski commented on CASSANDRA-2388:
---

+1

 ColumnFamilyRecordReader fails for a given split because a host is down, even 
 if records could reasonably be read from other replica.
 -

 Key: CASSANDRA-2388
 URL: https://issues.apache.org/jira/browse/CASSANDRA-2388
 Project: Cassandra
  Issue Type: Bug
  Components: Hadoop
Affects Versions: 0.6
Reporter: Eldon Stegall
Assignee: Paulo Motta
Priority: Minor
  Labels: hadoop, inputformat
 Fix For: 2.0.12

 Attachments: 0002_On_TException_try_next_split.patch, 
 1.2-CASSANDRA-2388.patch, 2.0-CASSANDRA-2388-v2.patch, 
 2.0-CASSANDRA-2388.patch, CASSANDRA-2388-addition1.patch, 
 CASSANDRA-2388-extended.patch, CASSANDRA-2388.patch, CASSANDRA-2388.patch, 
 CASSANDRA-2388.patch, CASSANDRA-2388.patch


 ColumnFamilyRecordReader only tries the first location for a given split. We 
 should try multiple locations for a given split.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-7688) Add data sizing to a system table

[
https://issues.apache.org/jira/browse/CASSANDRA-7688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14229818#comment-14229818
]

Benedict commented on CASSANDRA-7688:
-

This is a fundamentally difficult problem, and to be answered accurately
basically requires a full compaction. We can track or estimate this data for
any given sstable easily, and we can estimate the number of overlapping
partitions between two sstables (though the accuracy I'm unsure of if we
composed this data across many sstables), but we cannot say how many rows
within each overlapping partition overlap. The best we could do is probably
sample some overlapping partitions to see what proportion of row overlap tends
to prevail, and hope it is representative; if we assume a normal distribution
of overlap ratio we could return error bounds.

I don't think it's likely this data could be maintained live, at least not
accurately, or not without significant cost. It would be an on-demand
calculation that would be moderately expensive.

Add data sizing to a system table
-

Key: CASSANDRA-7688
URL: https://issues.apache.org/jira/browse/CASSANDRA-7688
Project: Cassandra
Issue Type: New Feature
Reporter: Jeremiah Jordan
Fix For: 2.1.3

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-7688) Add data sizing to a system table

[
https://issues.apache.org/jira/browse/CASSANDRA-7688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14229828#comment-14229828
]

Piotr Kołaczkowski commented on CASSANDRA-7688:
---

We only need estimates, not exact values. Factor 1.5x error is considered an
awesome estimate, factor 3x is still fairly good.
Also note that Spark/Hadoop does many token range scans. Maybe collecting some
statistics on the fly, during the scans (or during the compaction) would be
viable? And running a full compaction to get statistics more accurate - why
not? You need to do it anyway to get top speed when scanning data in Spark,
because a full table scan is doing kind-of implicit compaction anyway, isn't
it?

Also, one more thing - it would be good to have those values per column (sorry
for making it even harder, I know it is not an easy task). At least to know
that a column is responsible for xx% of data in the table - knowing such thing
would make a huge difference when estimating data size, because we're not
always fetching all columns and they may vary in size a lot (e.g.
collections!). Some sampling on insert would probably be enough.

Add data sizing to a system table
-

Key: CASSANDRA-7688
URL: https://issues.apache.org/jira/browse/CASSANDRA-7688
Project: Cassandra
Issue Type: New Feature
Reporter: Jeremiah Jordan
Fix For: 2.1.3

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-7688) Add data sizing to a system table


[ 
https://issues.apache.org/jira/browse/CASSANDRA-7688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14229831#comment-14229831
 ] 

Benedict commented on CASSANDRA-7688:
-

I'm talking about estimates. We cannot likely even estimate without pretty 
significant cost. Sampling column counts is pretty easy, but knowing how many 
cql rows there are for any merged row is not. There are tricks to make it 
easier, but there are datasets for which the tricks will not work, and any 
estimate would be complete guesswork without sampling the data.

 Add data sizing to a system table
 -

 Key: CASSANDRA-7688
 URL: https://issues.apache.org/jira/browse/CASSANDRA-7688
 Project: Cassandra
  Issue Type: New Feature
Reporter: Jeremiah Jordan
 Fix For: 2.1.3


 Currently you can't implement something similar to describe_splits_ex purely 
 from the a native protocol driver.  
 https://datastax-oss.atlassian.net/browse/JAVA-312 is open to expose easily 
 getting ownership information to a client in the java-driver.  But you still 
 need the data sizing part to get splits of a given size.  We should add the 
 sizing information to a system table so that native clients can get to it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8341) Expose time spent in each thread pool

2014-12-01 Thread Chris Lohfink (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14229859#comment-14229859
 ] 

Chris Lohfink commented on CASSANDRA-8341:
--

I want to see time spent processing per pool (user time may be more appropriate 
than walltime(nanotime) or cpu's system time).  This way we can track where cpu 
burn is occurring, and display a % of cpu in tpstats/opscenter by pool.  So 
while latencyutils or a histogram would certainly be interesting its more than 
necessary for this task.  A simple counter or meter would be sufficient.

 Expose time spent in each thread pool
 -

 Key: CASSANDRA-8341
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8341
 Project: Cassandra
  Issue Type: New Feature
  Components: Core
Reporter: Chris Lohfink
Priority: Minor
  Labels: metrics
 Attachments: 8341.patch, 8341v2.txt


 Can increment a counter with time spent in each queue.  This can provide 
 context on how much time is spent percentage wise in each stage.  
 Additionally can be used with littles law in future if ever want to try to 
 tune the size of the pools.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8341) Expose time spent in each thread pool


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14229861#comment-14229861
 ] 

T Jake Luciani commented on CASSANDRA-8341:
---

Ah, I was confused by the summary then.  I'll open a separate ticket for 
showing time waiting in queue.

 Expose time spent in each thread pool
 -

 Key: CASSANDRA-8341
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8341
 Project: Cassandra
  Issue Type: New Feature
  Components: Core
Reporter: Chris Lohfink
Priority: Minor
  Labels: metrics
 Attachments: 8341.patch, 8341v2.txt


 Can increment a counter with time spent in each queue.  This can provide 
 context on how much time is spent percentage wise in each stage.  
 Additionally can be used with littles law in future if ever want to try to 
 tune the size of the pools.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-8396) repairs creates sstable per each num tokens range


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-8396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Philip Thompson updated CASSANDRA-8396:
---
Reproduced In: 2.1.2, 2.1.1
Fix Version/s: 2.1.3

 repairs creates sstable per each num tokens range
 -

 Key: CASSANDRA-8396
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8396
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Reporter: Alexander Piavlo
Priority: Critical
 Fix For: 2.1.3


 I have num tokens set to 256
 then i run 
 `nodetool repair -pr someKeyspace someCF` 
 it creates 256 new small sstables - per each range afaiu
 on all replica nodes, this is major overkill for read performance.
 This happens with 2.1.2 and 2.1.1
 I have never anything like that with cassandra 1.0.x



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (CASSANDRA-8398) Expose time spent waiting in thread pool queue

T Jake Luciani created CASSANDRA-8398:
-

 Summary: Expose time spent waiting in thread pool queue 
 Key: CASSANDRA-8398
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8398
 Project: Cassandra
  Issue Type: Improvement
Reporter: T Jake Luciani
Priority: Minor
 Fix For: 2.1.3


We are missing an important source of latency in our system, the time waiting 
to be processed by thread pools.  We should add a metric for this so someone 
can easily see how much time is spent just waiting to be processed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-8341) Expose time spent in each thread pool


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14229861#comment-14229861
 ] 

T Jake Luciani edited comment on CASSANDRA-8341 at 12/1/14 2:52 PM:


Ah, I was confused by the summary then.  I'll open a separate ticket for 
showing time waiting in queue. CASSANDRA-8398


was (Author: tjake):
Ah, I was confused by the summary then.  I'll open a separate ticket for 
showing time waiting in queue.

 Expose time spent in each thread pool
 -

 Key: CASSANDRA-8341
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8341
 Project: Cassandra
  Issue Type: New Feature
  Components: Core
Reporter: Chris Lohfink
Priority: Minor
  Labels: metrics
 Attachments: 8341.patch, 8341v2.txt


 Can increment a counter with time spent in each queue.  This can provide 
 context on how much time is spent percentage wise in each stage.  
 Additionally can be used with littles law in future if ever want to try to 
 tune the size of the pools.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-8393) support quoted identifiers for index names


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-8393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Philip Thompson updated CASSANDRA-8393:
---
Reproduced In: 2.1.2
Fix Version/s: 2.1.3

 support quoted identifiers for index names
 --

 Key: CASSANDRA-8393
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8393
 Project: Cassandra
  Issue Type: Bug
 Environment: v2.1.2
Reporter: Jonathan Halliday
 Fix For: 2.1.3


 CREATE TABLE quoted_ident ...
 is valid in cql, whilst
 CREATE INDEX quoted_ident ...
 is not.
 This is inconsistent and troublesome for frameworks or tooling that needs to 
 sling around case sensitive identifiers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8390) The process cannot access the file because it is being used by another process


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14229870#comment-14229870
 ] 

Philip Thompson commented on CASSANDRA-8390:


Are you having this problem running Apache Cassandra on Windows?

 The process cannot access the file because it is being used by another process
 --

 Key: CASSANDRA-8390
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8390
 Project: Cassandra
  Issue Type: Bug
Reporter: Ilya Komolkin

 21:46:27.810 [NonPeriodicTasks:1] ERROR o.a.c.service.CassandraDaemon - 
 Exception in thread Thread[NonPeriodicTasks:1,5,main]
 org.apache.cassandra.io.FSWriteError: java.nio.file.FileSystemException: 
 E:\Upsource_12391\data\cassandra\data\kernel\filechangehistory_t-a277b560764611e48c8e4915424c75fe\kernel-filechangehistory_t-ka-33-Index.db:
  The process cannot access the file because it is being used by another 
 process.
  
 at 
 org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:135) 
 ~[cassandra-all-2.1.1.jar:2.1.1]
 at 
 org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:121) 
 ~[cassandra-all-2.1.1.jar:2.1.1]
 at 
 org.apache.cassandra.io.sstable.SSTable.delete(SSTable.java:113) 
 ~[cassandra-all-2.1.1.jar:2.1.1]
 at 
 org.apache.cassandra.io.sstable.SSTableDeletingTask.run(SSTableDeletingTask.java:94)
  ~[cassandra-all-2.1.1.jar:2.1.1]
 at 
 org.apache.cassandra.io.sstable.SSTableReader$6.run(SSTableReader.java:664) 
 ~[cassandra-all-2.1.1.jar:2.1.1]
 at 
 java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) 
 ~[na:1.7.0_71]
 at java.util.concurrent.FutureTask.run(FutureTask.java:262) 
 ~[na:1.7.0_71]
 at 
 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
  ~[na:1.7.0_71]
 at 
 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
  ~[na:1.7.0_71]
 at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  ~[na:1.7.0_71]
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  [na:1.7.0_71]
 at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
 Caused by: java.nio.file.FileSystemException: 
 E:\Upsource_12391\data\cassandra\data\kernel\filechangehistory_t-a277b560764611e48c8e4915424c75fe\kernel-filechangehistory_t-ka-33-Index.db:
  The process cannot access the file because it is being used by another 
 process.
  
 at 
 sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86) 
 ~[na:1.7.0_71]
 at 
 sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) 
 ~[na:1.7.0_71]
 at 
 sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102) 
 ~[na:1.7.0_71]
 at 
 sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
  ~[na:1.7.0_71]
 at 
 sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
  ~[na:1.7.0_71]
 at java.nio.file.Files.delete(Files.java:1079) ~[na:1.7.0_71]
 at 
 org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:131) 
 ~[cassandra-all-2.1.1.jar:2.1.1]
 ... 11 common frames omitted



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8341) Expose time spent in each thread pool


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14229872#comment-14229872
 ] 

Benedict commented on CASSANDRA-8341:
-

That is difficult, since we have stages that perform work that does not consume 
CPU. The RPC stage (for thrift or cql) both spend the majority of their time 
_waiting_ for the relevant work stage to complete. The proposed approaches 
would count this as busy time. The read and write stages also can block on IO, 
the former more often than the latter, but in either case we would count 
erroneously.


 Expose time spent in each thread pool
 -

 Key: CASSANDRA-8341
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8341
 Project: Cassandra
  Issue Type: New Feature
  Components: Core
Reporter: Chris Lohfink
Priority: Minor
  Labels: metrics
 Attachments: 8341.patch, 8341v2.txt


 Can increment a counter with time spent in each queue.  This can provide 
 context on how much time is spent percentage wise in each stage.  
 Additionally can be used with littles law in future if ever want to try to 
 tune the size of the pools.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8390) The process cannot access the file because it is being used by another process

2014-12-01 Thread Ilya Komolkin (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14229877#comment-14229877
 ] 

Ilya Komolkin commented on CASSANDRA-8390:
--

Yes. Windows 8.1

 The process cannot access the file because it is being used by another process
 --

 Key: CASSANDRA-8390
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8390
 Project: Cassandra
  Issue Type: Bug
Reporter: Ilya Komolkin

 21:46:27.810 [NonPeriodicTasks:1] ERROR o.a.c.service.CassandraDaemon - 
 Exception in thread Thread[NonPeriodicTasks:1,5,main]
 org.apache.cassandra.io.FSWriteError: java.nio.file.FileSystemException: 
 E:\Upsource_12391\data\cassandra\data\kernel\filechangehistory_t-a277b560764611e48c8e4915424c75fe\kernel-filechangehistory_t-ka-33-Index.db:
  The process cannot access the file because it is being used by another 
 process.
  
 at 
 org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:135) 
 ~[cassandra-all-2.1.1.jar:2.1.1]
 at 
 org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:121) 
 ~[cassandra-all-2.1.1.jar:2.1.1]
 at 
 org.apache.cassandra.io.sstable.SSTable.delete(SSTable.java:113) 
 ~[cassandra-all-2.1.1.jar:2.1.1]
 at 
 org.apache.cassandra.io.sstable.SSTableDeletingTask.run(SSTableDeletingTask.java:94)
  ~[cassandra-all-2.1.1.jar:2.1.1]
 at 
 org.apache.cassandra.io.sstable.SSTableReader$6.run(SSTableReader.java:664) 
 ~[cassandra-all-2.1.1.jar:2.1.1]
 at 
 java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) 
 ~[na:1.7.0_71]
 at java.util.concurrent.FutureTask.run(FutureTask.java:262) 
 ~[na:1.7.0_71]
 at 
 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
  ~[na:1.7.0_71]
 at 
 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
  ~[na:1.7.0_71]
 at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  ~[na:1.7.0_71]
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  [na:1.7.0_71]
 at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
 Caused by: java.nio.file.FileSystemException: 
 E:\Upsource_12391\data\cassandra\data\kernel\filechangehistory_t-a277b560764611e48c8e4915424c75fe\kernel-filechangehistory_t-ka-33-Index.db:
  The process cannot access the file because it is being used by another 
 process.
  
 at 
 sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86) 
 ~[na:1.7.0_71]
 at 
 sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) 
 ~[na:1.7.0_71]
 at 
 sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102) 
 ~[na:1.7.0_71]
 at 
 sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
  ~[na:1.7.0_71]
 at 
 sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
  ~[na:1.7.0_71]
 at java.nio.file.Files.delete(Files.java:1079) ~[na:1.7.0_71]
 at 
 org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:131) 
 ~[cassandra-all-2.1.1.jar:2.1.1]
 ... 11 common frames omitted



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-8390) The process cannot access the file because it is being used by another process


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-8390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Philip Thompson updated CASSANDRA-8390:
---
Reproduced In: 2.1.1

 The process cannot access the file because it is being used by another process
 --

 Key: CASSANDRA-8390
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8390
 Project: Cassandra
  Issue Type: Bug
Reporter: Ilya Komolkin
Assignee: Joshua McKenzie

 21:46:27.810 [NonPeriodicTasks:1] ERROR o.a.c.service.CassandraDaemon - 
 Exception in thread Thread[NonPeriodicTasks:1,5,main]
 org.apache.cassandra.io.FSWriteError: java.nio.file.FileSystemException: 
 E:\Upsource_12391\data\cassandra\data\kernel\filechangehistory_t-a277b560764611e48c8e4915424c75fe\kernel-filechangehistory_t-ka-33-Index.db:
  The process cannot access the file because it is being used by another 
 process.
  
 at 
 org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:135) 
 ~[cassandra-all-2.1.1.jar:2.1.1]
 at 
 org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:121) 
 ~[cassandra-all-2.1.1.jar:2.1.1]
 at 
 org.apache.cassandra.io.sstable.SSTable.delete(SSTable.java:113) 
 ~[cassandra-all-2.1.1.jar:2.1.1]
 at 
 org.apache.cassandra.io.sstable.SSTableDeletingTask.run(SSTableDeletingTask.java:94)
  ~[cassandra-all-2.1.1.jar:2.1.1]
 at 
 org.apache.cassandra.io.sstable.SSTableReader$6.run(SSTableReader.java:664) 
 ~[cassandra-all-2.1.1.jar:2.1.1]
 at 
 java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) 
 ~[na:1.7.0_71]
 at java.util.concurrent.FutureTask.run(FutureTask.java:262) 
 ~[na:1.7.0_71]
 at 
 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
  ~[na:1.7.0_71]
 at 
 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
  ~[na:1.7.0_71]
 at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  ~[na:1.7.0_71]
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  [na:1.7.0_71]
 at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
 Caused by: java.nio.file.FileSystemException: 
 E:\Upsource_12391\data\cassandra\data\kernel\filechangehistory_t-a277b560764611e48c8e4915424c75fe\kernel-filechangehistory_t-ka-33-Index.db:
  The process cannot access the file because it is being used by another 
 process.
  
 at 
 sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86) 
 ~[na:1.7.0_71]
 at 
 sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) 
 ~[na:1.7.0_71]
 at 
 sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102) 
 ~[na:1.7.0_71]
 at 
 sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
  ~[na:1.7.0_71]
 at 
 sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
  ~[na:1.7.0_71]
 at java.nio.file.Files.delete(Files.java:1079) ~[na:1.7.0_71]
 at 
 org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:131) 
 ~[cassandra-all-2.1.1.jar:2.1.1]
 ... 11 common frames omitted



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-8390) The process cannot access the file because it is being used by another process

2014-12-01 Thread Rajanarayanan Thottuvaikkatumana (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-8390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Philip Thompson updated CASSANDRA-8390:
---
Assignee: Joshua McKenzie

 The process cannot access the file because it is being used by another process
 --

 Key: CASSANDRA-8390
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8390
 Project: Cassandra
  Issue Type: Bug
Reporter: Ilya Komolkin
Assignee: Joshua McKenzie

 21:46:27.810 [NonPeriodicTasks:1] ERROR o.a.c.service.CassandraDaemon - 
 Exception in thread Thread[NonPeriodicTasks:1,5,main]
 org.apache.cassandra.io.FSWriteError: java.nio.file.FileSystemException: 
 E:\Upsource_12391\data\cassandra\data\kernel\filechangehistory_t-a277b560764611e48c8e4915424c75fe\kernel-filechangehistory_t-ka-33-Index.db:
  The process cannot access the file because it is being used by another 
 process.
  
 at 
 org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:135) 
 ~[cassandra-all-2.1.1.jar:2.1.1]
 at 
 org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:121) 
 ~[cassandra-all-2.1.1.jar:2.1.1]
 at 
 org.apache.cassandra.io.sstable.SSTable.delete(SSTable.java:113) 
 ~[cassandra-all-2.1.1.jar:2.1.1]
 at 
 org.apache.cassandra.io.sstable.SSTableDeletingTask.run(SSTableDeletingTask.java:94)
  ~[cassandra-all-2.1.1.jar:2.1.1]
 at 
 org.apache.cassandra.io.sstable.SSTableReader$6.run(SSTableReader.java:664) 
 ~[cassandra-all-2.1.1.jar:2.1.1]
 at 
 java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) 
 ~[na:1.7.0_71]
 at java.util.concurrent.FutureTask.run(FutureTask.java:262) 
 ~[na:1.7.0_71]
 at 
 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
  ~[na:1.7.0_71]
 at 
 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
  ~[na:1.7.0_71]
 at 
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  ~[na:1.7.0_71]
 at 
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  [na:1.7.0_71]
 at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
 Caused by: java.nio.file.FileSystemException: 
 E:\Upsource_12391\data\cassandra\data\kernel\filechangehistory_t-a277b560764611e48c8e4915424c75fe\kernel-filechangehistory_t-ka-33-Index.db:
  The process cannot access the file because it is being used by another 
 process.
  
 at 
 sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86) 
 ~[na:1.7.0_71]
 at 
 sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) 
 ~[na:1.7.0_71]
 at 
 sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102) 
 ~[na:1.7.0_71]
 at 
 sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
  ~[na:1.7.0_71]
 at 
 sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
  ~[na:1.7.0_71]
 at java.nio.file.Files.delete(Files.java:1079) ~[na:1.7.0_71]
 at 
 org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:131) 
 ~[cassandra-all-2.1.1.jar:2.1.1]
 ... 11 common frames omitted



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8267) Only stream from unrepaired sstables during incremental repair

2014-12-01 Thread Marcus Eriksson (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14229884#comment-14229884
 ] 

Marcus Eriksson commented on CASSANDRA-8267:


Did part of 8110 (just the version negotiation), pretty simple but, we have to 
bump messaging version which affects quite a few things, for example, we can't 
migrate schemas between different messaging versions (this could be special 
cased and fixed for this issue of course)

https://github.com/krummas/cassandra/commits/marcuse/8110 - simply splits out 
the header part of StreamInitMessage into its own message and replies with a 
'maxVersion' to that message when setting up the connection to agree on a 
version to use.

I think we should go with the new message solution above though, less risky for 
a minor release

 Only stream from unrepaired sstables during incremental repair
 --

 Key: CASSANDRA-8267
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8267
 Project: Cassandra
  Issue Type: Bug
Reporter: Marcus Eriksson
Assignee: Marcus Eriksson
 Fix For: 2.1.3


 Seems we stream from all sstables even if we do incremental repair, we should 
 limit this to only stream from the unrepaired sstables if we do incremental 
 repair



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (CASSANDRA-8267) Only stream from unrepaired sstables during incremental repair

2014-12-01 Thread Marcus Eriksson (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14229884#comment-14229884
 ] 

Marcus Eriksson edited comment on CASSANDRA-8267 at 12/1/14 3:03 PM:
-

Did part of 8110 (just the version negotiation), pretty simple, but we have to 
bump messaging version which affects quite a few things, for example, we can't 
migrate schemas between different messaging versions (this could be special 
cased and fixed for this issue of course)

https://github.com/krummas/cassandra/commits/marcuse/8110 - simply splits out 
the header part of StreamInitMessage into its own message and replies with a 
'maxVersion' to that message when setting up the connection to agree on a 
version to use.

I think we should go with the new message solution above though, less risky for 
a minor release


was (Author: krummas):
Did part of 8110 (just the version negotiation), pretty simple but, we have to 
bump messaging version which affects quite a few things, for example, we can't 
migrate schemas between different messaging versions (this could be special 
cased and fixed for this issue of course)

https://github.com/krummas/cassandra/commits/marcuse/8110 - simply splits out 
the header part of StreamInitMessage into its own message and replies with a 
'maxVersion' to that message when setting up the connection to agree on a 
version to use.

I think we should go with the new message solution above though, less risky for 
a minor release

 Only stream from unrepaired sstables during incremental repair
 --

 Key: CASSANDRA-8267
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8267
 Project: Cassandra
  Issue Type: Bug
Reporter: Marcus Eriksson
Assignee: Marcus Eriksson
 Fix For: 2.1.3


 Seems we stream from all sstables even if we do incremental repair, we should 
 limit this to only stream from the unrepaired sstables if we do incremental 
 repair



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8267) Only stream from unrepaired sstables during incremental repair

2014-12-01 Thread Aleksey Yeschenko (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14229885#comment-14229885
 ] 

Aleksey Yeschenko commented on CASSANDRA-8267:
--

bq. but we have to bump messaging version which affects quite a few things, for 
example, we can't migrate schemas between different messaging versions (this 
could be special cased and fixed for this issue of course)

That's unfortunate :\ I agree, then, with a new message - so long as we do it 
properly for 3.0 in CASSANDRA-8110.

 Only stream from unrepaired sstables during incremental repair
 --

 Key: CASSANDRA-8267
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8267
 Project: Cassandra
  Issue Type: Bug
Reporter: Marcus Eriksson
Assignee: Marcus Eriksson
 Fix For: 2.1.3


 Seems we stream from all sstables even if we do incremental repair, we should 
 limit this to only stream from the unrepaired sstables if we do incremental 
 repair



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8371) DateTieredCompactionStrategy is always compacting

2014-12-01 Thread Aleksey Yeschenko (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14229912#comment-14229912
 ] 

Aleksey Yeschenko commented on CASSANDRA-8371:
--

Nothing stopping us from adding another options (w _seconds), preferring it, if 
set, and slowly deprecating _days.

 DateTieredCompactionStrategy is always compacting 
 --

 Key: CASSANDRA-8371
 URL: https://issues.apache.org/jira/browse/CASSANDRA-8371
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Reporter: mck
Assignee: Björn Hegerfors
  Labels: compaction, performance
 Attachments: java_gc_counts_rate-month.png, 
 read-latency-recommenders-adview.png, read-latency.png, 
 sstables-recommenders-adviews.png, sstables.png, vg2_iad-month.png


 Running 2.0.11 and having switched a table to 
 [DTCS|https://issues.apache.org/jira/browse/CASSANDRA-6602] we've seen that 
 disk IO and gc count increase, along with the number of reads happening in 
 the compaction hump of cfhistograms.
 Data, and generally performance, looks good, but compactions are always 
 happening, and pending compactions are building up.
 The schema for this is 
 {code}CREATE TABLE search (
   loginid text,
   searchid timeuuid,
   description text,
   searchkey text,
   searchurl text,
   PRIMARY KEY ((loginid), searchid)
 );{code}
 We're sitting on about 82G (per replica) across 6 nodes in 4 DCs.
 CQL executed against this keyspace, and traffic patterns, can be seen in 
 slides 7+8 of https://prezi.com/b9-aj6p2esft/
 Attached are sstables-per-read and read-latency graphs from cfhistograms, and 
 screenshots of our munin graphs as we have gone from STCS, to LCS (week ~44), 
 to DTCS (week ~46).
 These screenshots are also found in the prezi on slides 9-11.
 [~pmcfadin], [~Bj0rn], 
 Can this be a consequence of occasional deleted rows, as is described under 
 (3) in the description of CASSANDRA-6602 ?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-4987) Support more queries when ALLOW FILTERING is used.


[ 
https://issues.apache.org/jira/browse/CASSANDRA-4987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14229918#comment-14229918
 ] 

Rajanarayanan Thottuvaikkatumana commented on CASSANDRA-4987:
-

[~ slebresne] In the Video table example of CASSANDRA-4915, if we issue a 
command {{SELECT * FROM videos WHERE tags = 'Cassandra';}}, then it will give 
error {{code=2200 [Invalid query] message=No secondary indexes on the 
restricted columns support the provided operators: }}. Is the idea to have 
{{ALLOW FILTERING}} option in the SELECT statements to support these kinds of 
WHERE clauses and the like otherwise NOT supported by Cassandra now? Thanks

 Support more queries when ALLOW FILTERING is used.
 --

 Key: CASSANDRA-4987
 URL: https://issues.apache.org/jira/browse/CASSANDRA-4987
 Project: Cassandra
  Issue Type: Improvement
Reporter: Sylvain Lebresne
  Labels: cql
 Fix For: 3.0


 Even after CASSANDRA-4915, there is still a bunch of queries that we don't 
 support even if {{ALLOW FILTERING}} is used. Typically, pretty much any 
 queries with restriction on a non-primary-key column unless we have one of 
 those restriction that is an EQ on an indexed column.
 If {{ALLOW FILTERING}} is used, we could allow those queries out of 
 convenience.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (CASSANDRA-8312) Use live sstables in snapshot repair if possible