[jira] [Created] (CASSANDRA-5267) Index organized table
Srdjan Mitrovic created CASSANDRA-5267:
--
Summary: Index organized table
Key: CASSANDRA-5267
URL: https://issues.apache.org/jira/browse/CASSANDRA-5267
Project: Cassandra
Issue Type: Improvement
Reporter: Srdjan Mitrovic
Priority: Minor
The purpose is to enable very fast scans for queries which use WHERE
indexed_column_value='foo';
We could borrow syntax from Oracle (with a small difference).
{noformat}CREATE TABLE blog_entries (
posted_at timestamp,
blog_id int
author text,
content text,
PRIMARY KEY (posted_at)
)
ORGANIZATION INDEX ON (blog_id);
{noformat}
In the background we could have a CF having only (key, indexed_value), in this
case (posted_at, blog_id) so that we can maintain our index when we delete a
row or change blog_id, and we would store other values within the index.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5267) Index organized table
[
https://issues.apache.org/jira/browse/CASSANDRA-5267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13580981#comment-13580981
]
Srdjan Mitrovic commented on CASSANDRA-5267:
I was probably misunderstood, this is just a secondary index with some extra
info stored (beside keys) so partitions are still ordered by partitioner and
data by comparator.
Benefit is that we don't need to read original CF for other data when we query
by index. If this is not possible can you tell me if there is a wiki explaining
internal architecture so that I can read that before creating issues.
Thanks.
Index organized table
-
Key: CASSANDRA-5267
URL: https://issues.apache.org/jira/browse/CASSANDRA-5267
Project: Cassandra
Issue Type: Improvement
Reporter: Srdjan Mitrovic
Priority: Minor
The purpose is to enable very fast scans for queries which use WHERE
indexed_column_value='foo';
We could borrow syntax from Oracle (with a small difference).
{noformat}CREATE TABLE blog_entries (
posted_at timestamp,
blog_id int
author text,
content text,
PRIMARY KEY (posted_at)
)
ORGANIZATION INDEX ON (blog_id);
{noformat}
In the background we could have a CF having only (key, indexed_value), in
this case (posted_at, blog_id) so that we can maintain our index when we
delete a row or change blog_id, and we would store other values within the
index.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4775) Counters 2.0
[ https://issues.apache.org/jira/browse/CASSANDRA-4775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13577055#comment-13577055 ] Srdjan Mitrovic commented on CASSANDRA-4775: bq. Not sure we'd want to support avg (since it requires extra information to be stored, as you point out) If we record every incr operation we will have extra info (until compaction :( ) I will propose a way you can make idempotent counters work and have all these features. 1. Create a CF with columns replayID, counterName, value, cnt and optional columns customField1, customField2, (Random partitioner on replayID or if we want to be sure it is unique we can use ComposityType replayID:counterName 2. Create a secondary index on counterName that we use to find sum(value) on each node separately because secondary index is distributed. 3. on compaction we delete old replayID, find total of value*cnt and sum(cnt) and store a new row (replayId, counterName, total, new cnt) We can use increment operation with some count (this will affect avg). For example incr(counters, myCounter, replayId, 3, 5) which will increment counter by 15 but it will be stored as value 3, cnt 5 so that it affects average. We can create custom fields for some reduce(IterableColumn so that we can support min, max, AND/OR/XOR... It would be ideal if a secondary index could also store values of the columns so that we can read counters in one go on each node. There is another jira issue for this. After that issue is resolved we can only keep secondary index without original CF, we just pretend it exists :) Counters 2.0 Key: CASSANDRA-4775 URL: https://issues.apache.org/jira/browse/CASSANDRA-4775 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Arya Goudarzi Labels: counters Fix For: 2.0 The existing partitioned counters remain a source of frustration for most users almost two years after being introduced. The remaining problems are inherent in the design, not something that can be fixed given enough time/eyeballs. Ideally a solution would give us - similar performance - less special cases in the code - potential for a retry mechanism -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4775) Counters 2.0
[ https://issues.apache.org/jira/browse/CASSANDRA-4775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13577055#comment-13577055 ] Srdjan Mitrovic edited comment on CASSANDRA-4775 at 2/12/13 9:58 PM: - bq. Not sure we'd want to support avg (since it requires extra information to be stored, as you point out) If we record every incr operation we will have extra info (until compaction :( ) I will propose a way you can make idempotent counters work and have all these features. 1. Create a CF with columns replayID, counterName, value, cnt and optional columns customField1, customField2, (Random partitioner on replayID or if we want to be sure it is unique we can use ComposityType replayID:counterName 2. Create a secondary index on counterName that we use to find sum(value) on each node separately because secondary index is distributed. 3. on compaction we delete old replayID, find total of value*cnt and sum(cnt) and store a new row (replayId, counterName, total, new cnt) We can use increment operation with some count (this will affect avg). For example incr(counters, myCounter, replayId, 3, 5) which will increment counter by 15 but it will be stored as value 3, cnt 5 so that it affects average in a different way than incrementing by value 15, count 1. We can create custom fields for some reduce(IterableColumn so that we can support min, max, AND/OR/XOR... It would be ideal if a secondary index could also store values of the columns so that we can read counters in one go on each node. There is another jira issue for this. After that issue is resolved we can only keep secondary index without original CF, we just pretend it exists :) was (Author: stecak): bq. Not sure we'd want to support avg (since it requires extra information to be stored, as you point out) If we record every incr operation we will have extra info (until compaction :( ) I will propose a way you can make idempotent counters work and have all these features. 1. Create a CF with columns replayID, counterName, value, cnt and optional columns customField1, customField2, (Random partitioner on replayID or if we want to be sure it is unique we can use ComposityType replayID:counterName 2. Create a secondary index on counterName that we use to find sum(value) on each node separately because secondary index is distributed. 3. on compaction we delete old replayID, find total of value*cnt and sum(cnt) and store a new row (replayId, counterName, total, new cnt) We can use increment operation with some count (this will affect avg). For example incr(counters, myCounter, replayId, 3, 5) which will increment counter by 15 but it will be stored as value 3, cnt 5 so that it affects average. We can create custom fields for some reduce(IterableColumn so that we can support min, max, AND/OR/XOR... It would be ideal if a secondary index could also store values of the columns so that we can read counters in one go on each node. There is another jira issue for this. After that issue is resolved we can only keep secondary index without original CF, we just pretend it exists :) Counters 2.0 Key: CASSANDRA-4775 URL: https://issues.apache.org/jira/browse/CASSANDRA-4775 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Arya Goudarzi Labels: counters Fix For: 2.0 The existing partitioned counters remain a source of frustration for most users almost two years after being introduced. The remaining problems are inherent in the design, not something that can be fixed given enough time/eyeballs. Ideally a solution would give us - similar performance - less special cases in the code - potential for a retry mechanism -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4775) Counters 2.0
[ https://issues.apache.org/jira/browse/CASSANDRA-4775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13577055#comment-13577055 ] Srdjan Mitrovic edited comment on CASSANDRA-4775 at 2/12/13 10:39 PM: -- bq. Not sure we'd want to support avg (since it requires extra information to be stored, as you point out) If we record every incr operation we will have extra info (until compaction :( ) I will propose a way you can make idempotent counters work and have all these features. 1. Create a CF with columns replayID, counterName, value, cnt and optional columns customField1, customField2, (Random partitioner on replayID or if we want to be sure it is unique we can use ComposityType replayID:counterName 2. Create a secondary index on counterName that we use to find sum(value) on each node separately because secondary index is distributed. 3. On compaction we delete old replayID, find total of value*cnt and sum(cnt) and store a new row (replayId, counterName, total, new cnt) We can use increment operation with some count (this will affect avg). For example incr(counters, myCounter, replayId, 3, 5) which will increment counter by 15 but it will be stored as value 3, cnt 5 so that it affects average in a different way than incrementing by value 15, count 1. We can create custom fields for some reduce(IterableColumn so that we can support min, max, AND/OR/XOR...For examoke on compaction we would store reduced max in that custom field. It would be ideal if a secondary index could also store values of the columns so that we can read counters in one go on each node. There is another jira issue for this. After that issue is resolved we can only keep secondary index without original CF, we just pretend it exists :) I guess that this approach could be achieved by clients if we have a pluggable compaction strategy but it would still be much easier if secondary indexes could also store other column values, not only keys. was (Author: stecak): bq. Not sure we'd want to support avg (since it requires extra information to be stored, as you point out) If we record every incr operation we will have extra info (until compaction :( ) I will propose a way you can make idempotent counters work and have all these features. 1. Create a CF with columns replayID, counterName, value, cnt and optional columns customField1, customField2, (Random partitioner on replayID or if we want to be sure it is unique we can use ComposityType replayID:counterName 2. Create a secondary index on counterName that we use to find sum(value) on each node separately because secondary index is distributed. 3. on compaction we delete old replayID, find total of value*cnt and sum(cnt) and store a new row (replayId, counterName, total, new cnt) We can use increment operation with some count (this will affect avg). For example incr(counters, myCounter, replayId, 3, 5) which will increment counter by 15 but it will be stored as value 3, cnt 5 so that it affects average in a different way than incrementing by value 15, count 1. We can create custom fields for some reduce(IterableColumn so that we can support min, max, AND/OR/XOR... It would be ideal if a secondary index could also store values of the columns so that we can read counters in one go on each node. There is another jira issue for this. After that issue is resolved we can only keep secondary index without original CF, we just pretend it exists :) Counters 2.0 Key: CASSANDRA-4775 URL: https://issues.apache.org/jira/browse/CASSANDRA-4775 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Arya Goudarzi Labels: counters Fix For: 2.0 The existing partitioned counters remain a source of frustration for most users almost two years after being introduced. The remaining problems are inherent in the design, not something that can be fixed given enough time/eyeballs. Ideally a solution would give us - similar performance - less special cases in the code - potential for a retry mechanism -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4775) Counters 2.0
[ https://issues.apache.org/jira/browse/CASSANDRA-4775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13577055#comment-13577055 ] Srdjan Mitrovic edited comment on CASSANDRA-4775 at 2/12/13 10:43 PM: -- bq. Not sure we'd want to support avg (since it requires extra information to be stored, as you point out) If we record every incr operation we will have extra info (until compaction :( ) I will propose a way you can make idempotent counters work and have all these features. 1. Create a CF with columns replayID, counterName, value, cnt and optional columns customField1, customField2, (Random partitioner on replayID or if we want to be sure it is unique we can use ComposityType replayID:counterName 2. Create a secondary index on counterName that we use to find sum(value) on each node separately because secondary index is distributed. 3. On compaction we delete old replayID, find total of value*cnt and sum(cnt) and store a new row (replayId, counterName, total, new cnt) We can use increment operation with some count (this will affect avg). For example incr(counters, myCounter, replayId, 3, 5) which will increment counter by 15 but it will be stored as value 3, cnt 5 so that it affects average in a different way than incrementing by value 15, count 1. We can create custom fields for some reduce(IterableColumn so that we can support min, max, AND/OR/XOR...For examoke on compaction we would store reduced max in that custom field. These counters with replayID could be used in atomic batches. It would be ideal if a secondary index could also store values of the columns so that we can read counters in one go on each node. There is another jira issue for this. After that issue is resolved we can only keep secondary index without original CF, we just pretend it exists :) I guess that this approach could be achieved by clients if we have a pluggable compaction strategy but it would still be much easier if secondary indexes could also store other column values, not only keys. was (Author: stecak): bq. Not sure we'd want to support avg (since it requires extra information to be stored, as you point out) If we record every incr operation we will have extra info (until compaction :( ) I will propose a way you can make idempotent counters work and have all these features. 1. Create a CF with columns replayID, counterName, value, cnt and optional columns customField1, customField2, (Random partitioner on replayID or if we want to be sure it is unique we can use ComposityType replayID:counterName 2. Create a secondary index on counterName that we use to find sum(value) on each node separately because secondary index is distributed. 3. On compaction we delete old replayID, find total of value*cnt and sum(cnt) and store a new row (replayId, counterName, total, new cnt) We can use increment operation with some count (this will affect avg). For example incr(counters, myCounter, replayId, 3, 5) which will increment counter by 15 but it will be stored as value 3, cnt 5 so that it affects average in a different way than incrementing by value 15, count 1. We can create custom fields for some reduce(IterableColumn so that we can support min, max, AND/OR/XOR...For examoke on compaction we would store reduced max in that custom field. It would be ideal if a secondary index could also store values of the columns so that we can read counters in one go on each node. There is another jira issue for this. After that issue is resolved we can only keep secondary index without original CF, we just pretend it exists :) I guess that this approach could be achieved by clients if we have a pluggable compaction strategy but it would still be much easier if secondary indexes could also store other column values, not only keys. Counters 2.0 Key: CASSANDRA-4775 URL: https://issues.apache.org/jira/browse/CASSANDRA-4775 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Arya Goudarzi Labels: counters Fix For: 2.0 The existing partitioned counters remain a source of frustration for most users almost two years after being introduced. The remaining problems are inherent in the design, not something that can be fixed given enough time/eyeballs. Ideally a solution would give us - similar performance - less special cases in the code - potential for a retry mechanism -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-5237) Add consistency level EACH_ONCE
Srdjan Mitrovic created CASSANDRA-5237: -- Summary: Add consistency level EACH_ONCE Key: CASSANDRA-5237 URL: https://issues.apache.org/jira/browse/CASSANDRA-5237 Project: Cassandra Issue Type: Improvement Reporter: Srdjan Mitrovic Priority: Minor Use case: If we have 2 datacenters and 2 replicas per datacenter a good fault tolerant consistency strategy would be to use level EACH_ONCE for writes as in that case it is fault tolerant even if 2 nodes die (one in each dc). If we assume that a node from local dc will always reply faster then remote dc we could use CL_TWO for reads and it would work fast for reads in normal conditions and be fault tolerant if a node or 2 nodes die. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-5219) get_indexed_keys
Srdjan Mitrovic created CASSANDRA-5219: -- Summary: get_indexed_keys Key: CASSANDRA-5219 URL: https://issues.apache.org/jira/browse/CASSANDRA-5219 Project: Cassandra Issue Type: Improvement Reporter: Srdjan Mitrovic We would like to get the list of keys for some indexed value by using only one disk seek per node. Unfortunately get_indexed_slices would also read CF where keys can be in many different places so it would take many disk seeks just to get the list of keys. Something like get_indexed_keys(keyRange) would be very beneficial to us. Much better idea is to use existing listKeySlice get_range_slices(ColumnParent column_parent, SlicePredicate predicate, KeyRange range, ConsistencyLevel consistency_level) without breaking any compatibility. If SlicePredicate has empty list of columns and KeyRange has an IndexExpression then only index should be fetched to return the desired keys without touching indexed CF. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4775) Counters 2.0
[ https://issues.apache.org/jira/browse/CASSANDRA-4775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546035#comment-13546035 ] Srdjan Mitrovic commented on CASSANDRA-4775: Don't atomic batched have potential for a retry mechanism? Now that CASSANDRA-4285 has been resolved can't you use Atomic batches to have idempotent counters? If we put in the same batch both incr and adding a wide row cell we could later see if we have a cell and not execute that batch again. I apologize if there is something fundamentally wrong in my understanding of how cassandra batches work so in that case please delete my message so that it doesn't derail the issue. Counters 2.0 Key: CASSANDRA-4775 URL: https://issues.apache.org/jira/browse/CASSANDRA-4775 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Arya Goudarzi Labels: counters Fix For: 2.0 The existing partitioned counters remain a source of frustration for most users almost two years after being introduced. The remaining problems are inherent in the design, not something that can be fixed given enough time/eyeballs. Ideally a solution would give us - similar performance - less special cases in the code - potential for a retry mechanism -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4775) Counters 2.0
[ https://issues.apache.org/jira/browse/CASSANDRA-4775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546035#comment-13546035 ] Srdjan Mitrovic edited comment on CASSANDRA-4775 at 1/7/13 9:27 PM: Don't atomic batches have potential for a retry mechanism? Now that CASSANDRA-4285 has been resolved can't you use Atomic batches to have idempotent counters? If we put in the same batch both incr and adding a wide row cell we could later see if we have a cell and not execute that batch again. I apologize if there is something fundamentally wrong in my understanding of how cassandra batches work so in that case please delete my message so that it doesn't derail the issue. was (Author: stecak): Don't atomic batched have potential for a retry mechanism? Now that CASSANDRA-4285 has been resolved can't you use Atomic batches to have idempotent counters? If we put in the same batch both incr and adding a wide row cell we could later see if we have a cell and not execute that batch again. I apologize if there is something fundamentally wrong in my understanding of how cassandra batches work so in that case please delete my message so that it doesn't derail the issue. Counters 2.0 Key: CASSANDRA-4775 URL: https://issues.apache.org/jira/browse/CASSANDRA-4775 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Arya Goudarzi Labels: counters Fix For: 2.0 The existing partitioned counters remain a source of frustration for most users almost two years after being introduced. The remaining problems are inherent in the design, not something that can be fixed given enough time/eyeballs. Ideally a solution would give us - similar performance - less special cases in the code - potential for a retry mechanism -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
