[jira] [Resolved] (CASSANDRA-11023) Restrict mutation consistency level to QUORUM+ for LWT
[ https://issues.apache.org/jira/browse/CASSANDRA-11023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sylvain Lebresne resolved CASSANDRA-11023. -- Resolution: Invalid Except that this is not how Paxos work. The non-serial consistency for LWT only matter for client visibility, not for paxos internally. More precisely a given round of Paxos will always [wait for a quorum of participants to have seen the previous commit before processing a new one|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/service/StorageProxy.java#L435-L449]. > Restrict mutation consistency level to QUORUM+ for LWT > -- > > Key: CASSANDRA-11023 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11023 > Project: Cassandra > Issue Type: Improvement > Components: Coordination >Reporter: DOAN DuyHai > > 5 nodes cluster, RF = 3 > → LWT consistency = SERIAL > → Mutation consistency = ONE > 1. Client 1 sends a INSERT INTO test (partition_key, value) VALUES(*1*, > *‘ONE’*) IF NOT EXISTS > 2. Paxos round successfully gets ballot1 > 3. The IF NOT EXISTS condition is validated (reading with QUORUM on all > replicas) > 4. Paxos round validated > 5. Mutation applied using consistency level ONE. Mutation pushed to replicas > A, B and C > 6. Replica A sends back acks. *Replicas B and C did not receive the mutation* > (temporary network issue). The LWT operation is successful since we do not > wait for acks from B and C > This LWT is considered successful > 7. Client 2 starts a LWT : INSERT INTO test (partition_key, value) > VALUES(*1*, *‘TWO’*) IF NOT EXISTS > 8. Paxos round successfully gets ballot2 (ballot2 > ballot1) > 9. The IF NOT EXISTS condition is validated (*reading with QUORUM and replica > B and C reply that partition 1 does not exist*) > 10. The semantics of LWT is violated because indeed the partition already > exists (because first LWT succeeded) with value = ‘ONE’ > I'm not saying that there is a bug in our LWT implementation, it works as > designed. The problem here is that I can't see any legit/sensible use-case > for LWT with mutation CL=ONE since it opens the door for edge cases like the > one described above and it defeats the purpose of Compare And Swap. > Furthermore it can confuse a lot of people not familiar of how LWT is > implemented internally. > Ideally, we should remove/ignore mutation CL for LWT and always use > QUORUM/LOCAL_QUORUM. But for the sake of not breaking the existing API, it > would be sufficient enough to validate and reject mutation CL < QUORUM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11024) Unexpected exception during request; java.lang.StackOverflowError: null
[
https://issues.apache.org/jira/browse/CASSANDRA-11024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15104965#comment-15104965
]
Sylvain Lebresne commented on CASSANDRA-11024:
--
Unfortunately the stack trace you attached seems to miss its beginning (the
part at the very end), which is the part we would need to know where this
happen. Are you sure you don't have anything more in the log?
> Unexpected exception during request; java.lang.StackOverflowError: null
> ---
>
> Key: CASSANDRA-11024
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11024
> Project: Cassandra
> Issue Type: Bug
> Environment: CentOS 7, Java x64 1.8.0_65
>Reporter: Kai Wang
>Priority: Minor
>
> This happened when I run a "SELECT *" query on a very wide table. The table
> has over 1000 columns and a lot of nulls. If I run "SELECT * ... LIMIT 10" or
> "SELECT a,b,c FROM ...", then it's fine. The data is being actively inserted
> when I run the query. Will try later when compaction (LCS) catches up.
> {noformat}
> ERROR [SharedPool-Worker-5] 2016-01-15 20:49:08,212 Message.java:611 -
> Unexpected exception during request; channel = [id: 0x8e11d570,
> /192.168.0.3:50332 => /192.168.0.11:9042]
> java.lang.StackOverflowError: null
> at
> com.google.common.base.Preconditions.checkPositionIndex(Preconditions.java:339)
> ~[guava-16.0.jar:na]
> at
> com.google.common.collect.AbstractIndexedListIterator.(AbstractIndexedListIterator.java:69)
> ~[guava-16.0.jar:na]
> at com.google.common.collect.Iterators$11.(Iterators.java:1048)
> ~[guava-16.0.jar:na]
> at com.google.common.collect.Iterators.forArray(Iterators.java:1048)
> ~[guava-16.0.jar:na]
> at
> com.google.common.collect.RegularImmutableList.listIterator(RegularImmutableList.java:106)
> ~[guava-16.0.jar:na]
> at
> com.google.common.collect.ImmutableList.listIterator(ImmutableList.java:344)
> ~[guava-16.0.jar:na]
> at
> com.google.common.collect.ImmutableList.iterator(ImmutableList.java:340)
> ~[guava-16.0.jar:na]
> at
> com.google.common.collect.ImmutableList.iterator(ImmutableList.java:61)
> ~[guava-16.0.jar:na]
> at com.google.common.collect.Iterables.iterators(Iterables.java:504)
> ~[guava-16.0.jar:na]
> at com.google.common.collect.Iterables.access$100(Iterables.java:60)
> ~[guava-16.0.jar:na]
> at com.google.common.collect.Iterables$2.iterator(Iterables.java:494)
> ~[guava-16.0.jar:na]
> at com.google.common.collect.Iterables$3.transform(Iterables.java:508)
> ~[guava-16.0.jar:na]
> at com.google.common.collect.Iterables$3.transform(Iterables.java:505)
> ~[guava-16.0.jar:na]
> at
> com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
> ~[guava-16.0.jar:na]
> at com.google.common.collect.Iterators$5.hasNext(Iterators.java:543)
> ~[guava-16.0.jar:na]
> at com.google.common.collect.Iterators$5.hasNext(Iterators.java:542)
> ~[guava-16.0.jar:na]
> at com.google.common.collect.Iterators$5.hasNext(Iterators.java:542)
> ~[guava-16.0.jar:na]
> at com.google.common.collect.Iterators$5.hasNext(Iterators.java:542)
> ~[guava-16.0.jar:na]
> at com.google.common.collect.Iterators$5.hasNext(Iterators.java:542)
> ~[guava-16.0.jar:na]
> at com.google.common.collect.Iterators$5.hasNext(Iterators.java:542)
> ~[guava-16.0.jar:na]
> at com.google.common.collect.Iterators$5.hasNext(Iterators.java:542)
> ~[guava-16.0.jar:na]
> at com.google.common.collect.Iterators$5.hasNext(Iterators.java:542)
> ~[guava-16.0.jar:na]
> at com.google.common.collect.Iterators$5.hasNext(Iterators.java:542)
> ~[guava-16.0.jar:na]
> at com.google.common.collect.Iterators$5.hasNext(Iterators.java:542)
> ~[guava-16.0.jar:na]
> at com.google.common.collect.Iterators$5.hasNext(Iterators.java:542)
> ~[guava-16.0.jar:na]
> at com.google.common.collect.Iterators$5.hasNext(Iterators.java:542)
> ~[guava-16.0.jar:na]
> at com.google.common.collect.Iterators$5.hasNext(Iterators.java:542)
> ~[guava-16.0.jar:na]
> at com.google.common.collect.Iterators$5.hasNext(Iterators.java:542)
> ~[guava-16.0.jar:na]
> at com.google.common.collect.Iterators$5.hasNext(Iterators.java:542)
> ~[guava-16.0.jar:na]
> at com.google.common.collect.Iterators$5.hasNext(Iterators.java:542)
> ~[guava-16.0.jar:na]
> at com.google.common.collect.Iterators$5.hasNext(Iterators.java:542)
> ~[guava-16.0.jar:na]
> at com.google.common.collect.Iterators$5.hasNext(Iterators.java:542)
> ~[guava-16.0.jar:na]
> at com.google.common.collect.Iterators$5.hasNext(Iterators.java:542)
> ~[guava-16.0.jar:na]
> at com.google.common.collect.Iterators$5.hasNext(Iterato
[jira] [Commented] (CASSANDRA-10379) Consider using -XX:+TrustFinalNonStaticFields
[
https://issues.apache.org/jira/browse/CASSANDRA-10379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15104977#comment-15104977
]
Sylvain Lebresne commented on CASSANDRA-10379:
--
For what it's worth, I'm not in favor of adding that flag as it sounds
potentially dangerous and the benefits don't sound that big. That is, I can
kind of trust us to not modify final field by reflection, but we depend on a
lot of libraries and as much as we would hope they don't do it either, it's
hard to be sure without spending a lot of investigative time and even then we
can't be absolutely guaranteed this won't change behind our backs.
Hopefully the JVM can indeed do a better safe job at this, but in the meantime,
I'd suggest closing this as "won't fix".
> Consider using -XX:+TrustFinalNonStaticFields
> -
>
> Key: CASSANDRA-10379
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10379
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Robert Stupp
>Assignee: Robert Stupp
> Fix For: 3.x
>
> Attachments: 10379.txt
>
>
> The JVM option {{-XX:+TrustFinalNonStaticFields}}, although experimental,
> seems to improve performance a bit without any code change. Therefore I
> propose to include it in {{cassandra-env.sh/psl}}.
> [cstar perf
> benchmark|http://cstar.datastax.com/graph?stats=a6e75018-5ff4-11e5-bf84-42010af0688f&metric=op_rate&operation=1_user&smoothing=1&show_aggregates=true&xmin=0&xmax=865.59&ymin=0&ymax=145568.5]
> The cstar test was run with 8u45.
> {noformat}
> JVM_OPTS="$JVM_OPTS -XX:+UnlockExperimentalVMOptions"
> JVM_OPTS="$JVM_OPTS -XX:+TrustFinalNonStaticFields"
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11023) Restrict mutation consistency level to QUORUM+ for LWT
[ https://issues.apache.org/jira/browse/CASSANDRA-11023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15104978#comment-15104978 ] DOAN DuyHai commented on CASSANDRA-11023: - You're right Sylvain, I missed this code block in the source. For once I'm happy to be wrong, our Paxos impl is pretty rock-solid > Restrict mutation consistency level to QUORUM+ for LWT > -- > > Key: CASSANDRA-11023 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11023 > Project: Cassandra > Issue Type: Improvement > Components: Coordination >Reporter: DOAN DuyHai > > 5 nodes cluster, RF = 3 > → LWT consistency = SERIAL > → Mutation consistency = ONE > 1. Client 1 sends a INSERT INTO test (partition_key, value) VALUES(*1*, > *‘ONE’*) IF NOT EXISTS > 2. Paxos round successfully gets ballot1 > 3. The IF NOT EXISTS condition is validated (reading with QUORUM on all > replicas) > 4. Paxos round validated > 5. Mutation applied using consistency level ONE. Mutation pushed to replicas > A, B and C > 6. Replica A sends back acks. *Replicas B and C did not receive the mutation* > (temporary network issue). The LWT operation is successful since we do not > wait for acks from B and C > This LWT is considered successful > 7. Client 2 starts a LWT : INSERT INTO test (partition_key, value) > VALUES(*1*, *‘TWO’*) IF NOT EXISTS > 8. Paxos round successfully gets ballot2 (ballot2 > ballot1) > 9. The IF NOT EXISTS condition is validated (*reading with QUORUM and replica > B and C reply that partition 1 does not exist*) > 10. The semantics of LWT is violated because indeed the partition already > exists (because first LWT succeeded) with value = ‘ONE’ > I'm not saying that there is a bug in our LWT implementation, it works as > designed. The problem here is that I can't see any legit/sensible use-case > for LWT with mutation CL=ONE since it opens the door for edge cases like the > one described above and it defeats the purpose of Compare And Swap. > Furthermore it can confuse a lot of people not familiar of how LWT is > implemented internally. > Ideally, we should remove/ignore mutation CL for LWT and always use > QUORUM/LOCAL_QUORUM. But for the sake of not breaking the existing API, it > would be sufficient enough to validate and reject mutation CL < QUORUM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11004) LWT results '[applied]' column name collision
[
https://issues.apache.org/jira/browse/CASSANDRA-11004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15105059#comment-15105059
]
Sylvain Lebresne commented on CASSANDRA-11004:
--
To clarify, I'm talking of result-set metadata in general, not just LWT. But
you'll get {{\[json\]}} when you query JSON ({{INSERT INTO foo (...) JSON ?}})
and you'll get {{\[timestamp\]}} and {{\[ttl\]]}} when you do for instance
{{UPDATE foo USING TIMESTAMP ? AND TTL ? ...}}.
And you're really looking for trouble, you can get name collisions when using
functions since you can declare a column name like {{"blobAsInt( x)"}} if you
so wish.
Anyway, I'll also remark that even in all those case, even though there is a
name collision, you can still access each value by position properly. In
particular, I'd argue that result of the description is a bug of cqlsh/the
python driver as the {{\[applied\]}} flag _is_ available in the result-set sent
by the server, it's cqlsh/the driver that tangle them.
Overall, I'm not sure it's worth bothering with complicating the protocol for
this. I'll add some quick note to the CQL documentation about the kind of
column names users should avoid and I think this'll be good enough.
We should however probably fix cqlsh so that the example in the description
display:
{noformat}
cassandra@cqlsh:test> INSERT INTO test (k, "[applied]") VALUES (2, 3) IF NOT
EXISTS ;
[applied] | k | [applied]
---+---+---
False | 2 | 3
{noformat}
as that's what the server sends and is more correct.
> LWT results '[applied]' column name collision
> -
>
> Key: CASSANDRA-11004
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11004
> Project: Cassandra
> Issue Type: Bug
>Reporter: Adam Holmberg
>Priority: Minor
> Fix For: 4.x
>
>
> LWT requests return a not-well-documented single row result with a boolean
> {{\[applied]}} column and optional column states.
> If the table happens to have a column named {{\[applied]}}, this causes a
> name collision. There is no error, but the {{\[applied]}} flag is not
> available.
> {code}
> cassandra@cqlsh:test> CREATE TABLE test (k int PRIMARY KEY , "[applied]" int);
> cassandra@cqlsh:test> INSERT INTO test (k, "[applied]") VALUES (2, 3) IF NOT
> EXISTS ;
> [applied]
> ---
> True
> cassandra@cqlsh:test> INSERT INTO test (k, "[applied]") VALUES (2, 3) IF NOT
> EXISTS ;
> [applied] | k
> ---+---
> 3 | 2
> {code}
> I doubt this comes up much (at all) in practice, but thought I'd mention it.
> One alternative approach might be to add a LWT result type
> ([flag|https://github.com/apache/cassandra/blob/cassandra-3.0/doc/native_protocol_v4.spec#L518-L522])
> that segregates the "applied" flag information optional row results.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[10/10] cassandra git commit: Merge branch 'cassandra-3.3' into trunk
Merge branch 'cassandra-3.3' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/22f02fe6 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/22f02fe6 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/22f02fe6 Branch: refs/heads/trunk Commit: 22f02fe602361ee93c20d5743c43a53ad65e1d45 Parents: 8f88015 a6170ef Author: Sylvain Lebresne Authored: Mon Jan 18 11:28:05 2016 +0100 Committer: Sylvain Lebresne Committed: Mon Jan 18 11:28:05 2016 +0100 -- doc/cql3/CQL.textile | 2 ++ 1 file changed, 2 insertions(+) --
[07/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0
Merge branch 'cassandra-2.2' into cassandra-3.0 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a3360afa Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a3360afa Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a3360afa Branch: refs/heads/trunk Commit: a3360afa81675f7db86d502d5950ec11ab1103ea Parents: 6cd3bef 79b9492 Author: Sylvain Lebresne Authored: Mon Jan 18 11:27:40 2016 +0100 Committer: Sylvain Lebresne Committed: Mon Jan 18 11:27:40 2016 +0100 -- doc/cql3/CQL.textile | 2 ++ 1 file changed, 2 insertions(+) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/a3360afa/doc/cql3/CQL.textile --
[05/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0
Merge branch 'cassandra-2.2' into cassandra-3.0 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a3360afa Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a3360afa Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a3360afa Branch: refs/heads/cassandra-3.3 Commit: a3360afa81675f7db86d502d5950ec11ab1103ea Parents: 6cd3bef 79b9492 Author: Sylvain Lebresne Authored: Mon Jan 18 11:27:40 2016 +0100 Committer: Sylvain Lebresne Committed: Mon Jan 18 11:27:40 2016 +0100 -- doc/cql3/CQL.textile | 2 ++ 1 file changed, 2 insertions(+) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/a3360afa/doc/cql3/CQL.textile --
[01/10] cassandra git commit: Add warning regarding quoted identifiers in CQL doc
Repository: cassandra Updated Branches: refs/heads/cassandra-2.2 1b5935736 -> 79b949236 refs/heads/cassandra-3.0 6cd3bef61 -> a3360afa8 refs/heads/cassandra-3.3 b7356dd8e -> a6170ef80 refs/heads/trunk 8f8801515 -> 22f02fe60 Add warning regarding quoted identifiers in CQL doc Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/79b94923 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/79b94923 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/79b94923 Branch: refs/heads/cassandra-2.2 Commit: 79b9492363d60863c717a9dfe4ce8c1769071c85 Parents: 1b59357 Author: Sylvain Lebresne Authored: Mon Jan 18 11:27:24 2016 +0100 Committer: Sylvain Lebresne Committed: Mon Jan 18 11:27:24 2016 +0100 -- doc/cql3/CQL.textile | 2 ++ 1 file changed, 2 insertions(+) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/79b94923/doc/cql3/CQL.textile -- diff --git a/doc/cql3/CQL.textile b/doc/cql3/CQL.textile index c2ecbd7..2dd96e7 100644 --- a/doc/cql3/CQL.textile +++ b/doc/cql3/CQL.textile @@ -46,6 +46,8 @@ p. Identifiers and (unquoted) keywords are case insensitive. Thus @SELECT@ is th p. There is a second kind of identifiers called _quoted identifiers_ defined by enclosing an arbitrary sequence of characters in double-quotes(@"@). Quoted identifiers are never keywords. Thus @"select"@ is not a reserved keyword and can be used to refer to a column, while @select@ would raise a parse error. Also, contrarily to unquoted identifiers and keywords, quoted identifiers are case sensitive (@"My Quoted Id"@ is _different_ from @"my quoted id"@). A fully lowercase quoted identifier that matches @[a-zA-Z]@@[a-zA-Z0-9_]@@*@ is equivalent to the unquoted identifier obtained by removing the double-quote (so @"myid"@ is equivalent to @myid@ and to @myId@ but different from @"myId"@). Inside a quoted identifier, the double-quote character can be repeated to escape it, so @"foo "" bar"@ is a valid identifier. +p. *Warning*: _quoted identifiers_ allows to declare columns with arbitrary names, and those can sometime clash with specific names used by the server. For instance, when using conditional update, the server will respond with a result-set containing a special result named @"[applied]"@. If you've declared a column with such a name, this could potentially confuse some tools and should be avoided. In general, unquoted identifiers should be preferred but if you use quoted identifiers, it is strongly advised to avoid any name enclosed by squared brackets (like @"[applied]"@) and any name that looks like a function call (like @"f(x)"@). + h3(#constants). Constants CQL defines the following kind of _constants_: strings, integers, floats, booleans, uuids and blobs:
[03/10] cassandra git commit: Add warning regarding quoted identifiers in CQL doc
Add warning regarding quoted identifiers in CQL doc Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/79b94923 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/79b94923 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/79b94923 Branch: refs/heads/cassandra-3.3 Commit: 79b9492363d60863c717a9dfe4ce8c1769071c85 Parents: 1b59357 Author: Sylvain Lebresne Authored: Mon Jan 18 11:27:24 2016 +0100 Committer: Sylvain Lebresne Committed: Mon Jan 18 11:27:24 2016 +0100 -- doc/cql3/CQL.textile | 2 ++ 1 file changed, 2 insertions(+) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/79b94923/doc/cql3/CQL.textile -- diff --git a/doc/cql3/CQL.textile b/doc/cql3/CQL.textile index c2ecbd7..2dd96e7 100644 --- a/doc/cql3/CQL.textile +++ b/doc/cql3/CQL.textile @@ -46,6 +46,8 @@ p. Identifiers and (unquoted) keywords are case insensitive. Thus @SELECT@ is th p. There is a second kind of identifiers called _quoted identifiers_ defined by enclosing an arbitrary sequence of characters in double-quotes(@"@). Quoted identifiers are never keywords. Thus @"select"@ is not a reserved keyword and can be used to refer to a column, while @select@ would raise a parse error. Also, contrarily to unquoted identifiers and keywords, quoted identifiers are case sensitive (@"My Quoted Id"@ is _different_ from @"my quoted id"@). A fully lowercase quoted identifier that matches @[a-zA-Z]@@[a-zA-Z0-9_]@@*@ is equivalent to the unquoted identifier obtained by removing the double-quote (so @"myid"@ is equivalent to @myid@ and to @myId@ but different from @"myId"@). Inside a quoted identifier, the double-quote character can be repeated to escape it, so @"foo "" bar"@ is a valid identifier. +p. *Warning*: _quoted identifiers_ allows to declare columns with arbitrary names, and those can sometime clash with specific names used by the server. For instance, when using conditional update, the server will respond with a result-set containing a special result named @"[applied]"@. If you've declared a column with such a name, this could potentially confuse some tools and should be avoided. In general, unquoted identifiers should be preferred but if you use quoted identifiers, it is strongly advised to avoid any name enclosed by squared brackets (like @"[applied]"@) and any name that looks like a function call (like @"f(x)"@). + h3(#constants). Constants CQL defines the following kind of _constants_: strings, integers, floats, booleans, uuids and blobs:
[04/10] cassandra git commit: Add warning regarding quoted identifiers in CQL doc
Add warning regarding quoted identifiers in CQL doc Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/79b94923 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/79b94923 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/79b94923 Branch: refs/heads/trunk Commit: 79b9492363d60863c717a9dfe4ce8c1769071c85 Parents: 1b59357 Author: Sylvain Lebresne Authored: Mon Jan 18 11:27:24 2016 +0100 Committer: Sylvain Lebresne Committed: Mon Jan 18 11:27:24 2016 +0100 -- doc/cql3/CQL.textile | 2 ++ 1 file changed, 2 insertions(+) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/79b94923/doc/cql3/CQL.textile -- diff --git a/doc/cql3/CQL.textile b/doc/cql3/CQL.textile index c2ecbd7..2dd96e7 100644 --- a/doc/cql3/CQL.textile +++ b/doc/cql3/CQL.textile @@ -46,6 +46,8 @@ p. Identifiers and (unquoted) keywords are case insensitive. Thus @SELECT@ is th p. There is a second kind of identifiers called _quoted identifiers_ defined by enclosing an arbitrary sequence of characters in double-quotes(@"@). Quoted identifiers are never keywords. Thus @"select"@ is not a reserved keyword and can be used to refer to a column, while @select@ would raise a parse error. Also, contrarily to unquoted identifiers and keywords, quoted identifiers are case sensitive (@"My Quoted Id"@ is _different_ from @"my quoted id"@). A fully lowercase quoted identifier that matches @[a-zA-Z]@@[a-zA-Z0-9_]@@*@ is equivalent to the unquoted identifier obtained by removing the double-quote (so @"myid"@ is equivalent to @myid@ and to @myId@ but different from @"myId"@). Inside a quoted identifier, the double-quote character can be repeated to escape it, so @"foo "" bar"@ is a valid identifier. +p. *Warning*: _quoted identifiers_ allows to declare columns with arbitrary names, and those can sometime clash with specific names used by the server. For instance, when using conditional update, the server will respond with a result-set containing a special result named @"[applied]"@. If you've declared a column with such a name, this could potentially confuse some tools and should be avoided. In general, unquoted identifiers should be preferred but if you use quoted identifiers, it is strongly advised to avoid any name enclosed by squared brackets (like @"[applied]"@) and any name that looks like a function call (like @"f(x)"@). + h3(#constants). Constants CQL defines the following kind of _constants_: strings, integers, floats, booleans, uuids and blobs:
[09/10] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.3
Merge branch 'cassandra-3.0' into cassandra-3.3 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a6170ef8 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a6170ef8 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a6170ef8 Branch: refs/heads/trunk Commit: a6170ef80212f003895c964fd400674d9933cc6c Parents: b7356dd a3360af Author: Sylvain Lebresne Authored: Mon Jan 18 11:27:55 2016 +0100 Committer: Sylvain Lebresne Committed: Mon Jan 18 11:27:55 2016 +0100 -- doc/cql3/CQL.textile | 2 ++ 1 file changed, 2 insertions(+) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/a6170ef8/doc/cql3/CQL.textile --
[06/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0
Merge branch 'cassandra-2.2' into cassandra-3.0 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a3360afa Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a3360afa Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a3360afa Branch: refs/heads/cassandra-3.0 Commit: a3360afa81675f7db86d502d5950ec11ab1103ea Parents: 6cd3bef 79b9492 Author: Sylvain Lebresne Authored: Mon Jan 18 11:27:40 2016 +0100 Committer: Sylvain Lebresne Committed: Mon Jan 18 11:27:40 2016 +0100 -- doc/cql3/CQL.textile | 2 ++ 1 file changed, 2 insertions(+) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/a3360afa/doc/cql3/CQL.textile --
[02/10] cassandra git commit: Add warning regarding quoted identifiers in CQL doc
Add warning regarding quoted identifiers in CQL doc Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/79b94923 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/79b94923 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/79b94923 Branch: refs/heads/cassandra-3.0 Commit: 79b9492363d60863c717a9dfe4ce8c1769071c85 Parents: 1b59357 Author: Sylvain Lebresne Authored: Mon Jan 18 11:27:24 2016 +0100 Committer: Sylvain Lebresne Committed: Mon Jan 18 11:27:24 2016 +0100 -- doc/cql3/CQL.textile | 2 ++ 1 file changed, 2 insertions(+) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/79b94923/doc/cql3/CQL.textile -- diff --git a/doc/cql3/CQL.textile b/doc/cql3/CQL.textile index c2ecbd7..2dd96e7 100644 --- a/doc/cql3/CQL.textile +++ b/doc/cql3/CQL.textile @@ -46,6 +46,8 @@ p. Identifiers and (unquoted) keywords are case insensitive. Thus @SELECT@ is th p. There is a second kind of identifiers called _quoted identifiers_ defined by enclosing an arbitrary sequence of characters in double-quotes(@"@). Quoted identifiers are never keywords. Thus @"select"@ is not a reserved keyword and can be used to refer to a column, while @select@ would raise a parse error. Also, contrarily to unquoted identifiers and keywords, quoted identifiers are case sensitive (@"My Quoted Id"@ is _different_ from @"my quoted id"@). A fully lowercase quoted identifier that matches @[a-zA-Z]@@[a-zA-Z0-9_]@@*@ is equivalent to the unquoted identifier obtained by removing the double-quote (so @"myid"@ is equivalent to @myid@ and to @myId@ but different from @"myId"@). Inside a quoted identifier, the double-quote character can be repeated to escape it, so @"foo "" bar"@ is a valid identifier. +p. *Warning*: _quoted identifiers_ allows to declare columns with arbitrary names, and those can sometime clash with specific names used by the server. For instance, when using conditional update, the server will respond with a result-set containing a special result named @"[applied]"@. If you've declared a column with such a name, this could potentially confuse some tools and should be avoided. In general, unquoted identifiers should be preferred but if you use quoted identifiers, it is strongly advised to avoid any name enclosed by squared brackets (like @"[applied]"@) and any name that looks like a function call (like @"f(x)"@). + h3(#constants). Constants CQL defines the following kind of _constants_: strings, integers, floats, booleans, uuids and blobs:
[08/10] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.3
Merge branch 'cassandra-3.0' into cassandra-3.3 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a6170ef8 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a6170ef8 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a6170ef8 Branch: refs/heads/cassandra-3.3 Commit: a6170ef80212f003895c964fd400674d9933cc6c Parents: b7356dd a3360af Author: Sylvain Lebresne Authored: Mon Jan 18 11:27:55 2016 +0100 Committer: Sylvain Lebresne Committed: Mon Jan 18 11:27:55 2016 +0100 -- doc/cql3/CQL.textile | 2 ++ 1 file changed, 2 insertions(+) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/a6170ef8/doc/cql3/CQL.textile --
[jira] [Commented] (CASSANDRA-7925) TimeUUID LSB should be unique per process, not just per machine
[ https://issues.apache.org/jira/browse/CASSANDRA-7925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15105117#comment-15105117 ] Benedict commented on CASSANDRA-7925: - Mostly +1, but not convinced by the extraction of SigarLibrary to an instance variable; it's used exactly twice, so is there any point cluttering up the place with it? > TimeUUID LSB should be unique per process, not just per machine > --- > > Key: CASSANDRA-7925 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7925 > Project: Cassandra > Issue Type: Improvement >Reporter: Peter Mädel >Assignee: Sylvain Lebresne > Fix For: 2.2.x > > Attachments: cassandra-uuidgen.patch > > > as pointed out in > [CASSANDRA-7919|https://issues.apache.org/jira/browse/CASSANDRA-7919?focusedCommentId=14132529&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14132529] > lsb collisions are also possible serverside. > a sufficient solution would be to include references to pid and classloader > within lsb. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9472) Reintroduce off heap memtables
[
https://issues.apache.org/jira/browse/CASSANDRA-9472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15105169#comment-15105169
]
Benedict commented on CASSANDRA-9472:
-
Thanks for finishing up finding the work [~Stefania].
It looks mostly good, but I would probably modify it a little from its current
representation:
* Storing an {{offheap}} variable in {{BTreeRow}} looks error-prone to
maintain, and forwarding all calls in {{AbstractBTreeRow}} to a virtual
{{onHeapCopy}} is unnecessarily costly for objects that never are offheap. I
think it makes more sense for the {{AtomicBTreePartition}} to check f it stores
on/offheap data when answering a query, and to return a suitable iterator that
knows to copy on-heap when necessary - much as it used to, but moved one step
outwards so that it can capture {{DecoratedKey}} and {{staticRow}}
* As far as segmentation faults on {{toString}} when {{peer == 0}}, this is
only a problem for the debugger, because we don't call {{toString()}}
ourselves. I don't mind adding guards, but they should really throw exceptions
since any access to these objects when they've been freed is unequivocally a
bug.
Let me know your thoughts.
> Reintroduce off heap memtables
> --
>
> Key: CASSANDRA-9472
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9472
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Benedict
>Assignee: Stefania
> Fix For: 3.x
>
>
> CASSANDRA-8099 removes off heap memtables. We should reintroduce them ASAP.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-9472) Reintroduce off heap memtables
[
https://issues.apache.org/jira/browse/CASSANDRA-9472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15105169#comment-15105169
]
Benedict edited comment on CASSANDRA-9472 at 1/18/16 11:27 AM:
---
Thanks for finishing up finding the work [~Stefania].
It looks mostly good, but I would probably modify it a little from its current
representation:
* Storing an {{offheap}} variable in {{BTreeRow}} looks error-prone to
maintain, and forwarding all calls in {{AbstractBTreeRow}} to a virtual
{{onHeapCopy}} is unnecessarily costly for objects that never are offheap. I
think it makes more sense for the {{AtomicBTreePartition}} to check f it stores
on/offheap data when answering a query, and to return a suitable iterator that
knows to copy on-heap when necessary - much as it used to, but moved one step
outwards so that it can capture {{DecoratedKey}} and {{staticRow}}
* As far as segmentation faults on {{toString}} when {{peer == 0}}, this is
only a problem for the debugger, because we don't call {{toString()}}
ourselves. I don't mind adding guards, but they should really throw exceptions
since any access to these objects when they've been freed is unequivocally a
bug.
Let me know your thoughts (and I'll finish this up, if we're in agreement).
was (Author: benedict):
Thanks for finishing up finding the work [~Stefania].
It looks mostly good, but I would probably modify it a little from its current
representation:
* Storing an {{offheap}} variable in {{BTreeRow}} looks error-prone to
maintain, and forwarding all calls in {{AbstractBTreeRow}} to a virtual
{{onHeapCopy}} is unnecessarily costly for objects that never are offheap. I
think it makes more sense for the {{AtomicBTreePartition}} to check f it stores
on/offheap data when answering a query, and to return a suitable iterator that
knows to copy on-heap when necessary - much as it used to, but moved one step
outwards so that it can capture {{DecoratedKey}} and {{staticRow}}
* As far as segmentation faults on {{toString}} when {{peer == 0}}, this is
only a problem for the debugger, because we don't call {{toString()}}
ourselves. I don't mind adding guards, but they should really throw exceptions
since any access to these objects when they've been freed is unequivocally a
bug.
Let me know your thoughts.
> Reintroduce off heap memtables
> --
>
> Key: CASSANDRA-9472
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9472
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Benedict
>Assignee: Stefania
> Fix For: 3.x
>
>
> CASSANDRA-8099 removes off heap memtables. We should reintroduce them ASAP.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-11026) OOM
Maxim Podkolzine created CASSANDRA-11026: Summary: OOM Key: CASSANDRA-11026 URL: https://issues.apache.org/jira/browse/CASSANDRA-11026 Project: Cassandra Issue Type: Bug Reporter: Maxim Podkolzine Attachments: dump.png Cassandra 3.0.2 fails with OOM. The heapdump shows large number of HeapByteBuffer instances, each retaining 1Gb (see the details on the screenshot). Overall retained size is ~2Gb. We can provide the additional info and the whole heapdump if necessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-11026) OOM
[ https://issues.apache.org/jira/browse/CASSANDRA-11026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Podkolzine updated CASSANDRA-11026: - Description: Cassandra 3.0.2 fails with OOM. The heapdump shows large number of HeapByteBuffer instances, each retaining 1Mb (see the details on the screenshot). Overall retained size is ~2Gb. We can provide the additional info and the whole heapdump if necessary. was: Cassandra 3.0.2 fails with OOM. The heapdump shows large number of HeapByteBuffer instances, each retaining 1Gb (see the details on the screenshot). Overall retained size is ~2Gb. We can provide the additional info and the whole heapdump if necessary. > OOM > --- > > Key: CASSANDRA-11026 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11026 > Project: Cassandra > Issue Type: Bug >Reporter: Maxim Podkolzine > Attachments: dump.png > > > Cassandra 3.0.2 fails with OOM. The heapdump shows large number of > HeapByteBuffer instances, each retaining 1Mb (see the details on the > screenshot). Overall retained size is ~2Gb. > We can provide the additional info and the whole heapdump if necessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-6018) Add option to encrypt commitlog
[
https://issues.apache.org/jira/browse/CASSANDRA-6018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15105206#comment-15105206
]
Branimir Lambov commented on CASSANDRA-6018:
Two of the points above don't appear to be addressed and I believe they cause
unwanted behaviour changes:
- please use {{tolerateTruncation}} in {{CommitLogReplayer}} instead of the
[AND-ed
value|https://github.com/jasobrown/cassandra/blob/7526af73ebc461d2db50ef52b89985212826e6d7/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java#L390],
which makes us less tolerant than we want to be for pre-2.1 recycled segments;
- [OR in catching
{{SegmentReadException}}|https://github.com/jasobrown/cassandra/commit/7526af73ebc461d2db50ef52b89985212826e6d7#diff-4c3a8240a441cef90e680246ee64R105]
should be an AND.
> Add option to encrypt commitlog
>
>
> Key: CASSANDRA-6018
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6018
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Jason Brown
>Assignee: Jason Brown
> Labels: commit_log, encryption, security
> Fix For: 3.x
>
>
> We are going to start using cassandra for a billing system, and while I can
> encrypt sstables at rest (via Datastax Enterprise), commit logs are more or
> less plain text. Thus, an attacker would be able to easily read, for example,
> credit card numbers in the clear text commit log (if the calling app does not
> encrypt the data itself before sending it to cassandra).
> I want to allow the option of encrypting the commit logs, most likely
> controlled by a property in the yaml.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-11027) Duplicate column familiy initialization
Sergio Bossa created CASSANDRA-11027: Summary: Duplicate column familiy initialization Key: CASSANDRA-11027 URL: https://issues.apache.org/jira/browse/CASSANDRA-11027 Project: Cassandra Issue Type: Bug Components: Distributed Metadata Reporter: Sergio Bossa Assignee: Sam Tunnicliffe Schema#addTable initializes the column family twice, as can be seen in: https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/config/Schema.java#L605 This doesn't seem to be right, with the second one probably being the one to keep to avoid race conditions with looking up the cf from the schema (as originally intended by CASSANDRA-10761?). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10969) long-running cluster sees bad gossip generation when a node restarts
[ https://issues.apache.org/jira/browse/CASSANDRA-10969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15105256#comment-15105256 ] adil chabaq commented on CASSANDRA-10969: - yes, i confirm the problem is resolved by restarting the whole cluster. thanks > long-running cluster sees bad gossip generation when a node restarts > > > Key: CASSANDRA-10969 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10969 > Project: Cassandra > Issue Type: Bug > Components: Coordination > Environment: 4-node Cassandra 2.1.1 cluster, each node running on a > Linux 2.6.32-431.20.3.dl6.x86_64 VM >Reporter: T. David Hudson >Assignee: Joel Knighton > Fix For: 3.3, 2.1.x, 2.2.x, 3.0.x > > > One of the nodes in a long-running Cassandra 2.1.1 cluster (not under my > control) restarted. The remaining nodes are logging errors like this: > "received an invalid gossip generation for peer xxx.xxx.xxx.xxx; local > generation = 1414613355, received generation = 1450978722" > The gap between the local and received generation numbers exceeds the > one-year threshold added for CASSANDRA-8113. The system clocks are > up-to-date for all nodes. > If this is a bug, the latest released Gossiper.java code in 2.1.x, 2.2.x, and > 3.0.x seems not to have changed the behavior that I'm seeing. > I presume that restarting the remaining nodes will clear up the problem, > whence the minor priority. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7925) TimeUUID LSB should be unique per process, not just per machine
[
https://issues.apache.org/jira/browse/CASSANDRA-7925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15105328#comment-15105328
]
Sylvain Lebresne commented on CASSANDRA-7925:
-
bq. not convinced by the extraction of SigarLibrary to an instance variable
That's mostly a simple way to avoid logging {{"Initializing SIGAR library"}}
(and its potential error if it can't be initialized) twice at each startup. I
agree it's not really justified otherwise, but double logging would be really
ugly. There is obviously other options but wonder if its worth bothering.
> TimeUUID LSB should be unique per process, not just per machine
> ---
>
> Key: CASSANDRA-7925
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7925
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Peter Mädel
>Assignee: Sylvain Lebresne
> Fix For: 2.2.x
>
> Attachments: cassandra-uuidgen.patch
>
>
> as pointed out in
> [CASSANDRA-7919|https://issues.apache.org/jira/browse/CASSANDRA-7919?focusedCommentId=14132529&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14132529]
> lsb collisions are also possible serverside.
> a sufficient solution would be to include references to pid and classloader
> within lsb.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7925) TimeUUID LSB should be unique per process, not just per machine
[ https://issues.apache.org/jira/browse/CASSANDRA-7925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15105335#comment-15105335 ] Benedict commented on CASSANDRA-7925: - I'm not sure how much is occupied by Sigar if we keep it around, but I would guess it's pretty unlikely to be meaningful. Just wanted to raise it for reconsideration; +1 > TimeUUID LSB should be unique per process, not just per machine > --- > > Key: CASSANDRA-7925 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7925 > Project: Cassandra > Issue Type: Improvement >Reporter: Peter Mädel >Assignee: Sylvain Lebresne > Fix For: 2.2.x > > Attachments: cassandra-uuidgen.patch > > > as pointed out in > [CASSANDRA-7919|https://issues.apache.org/jira/browse/CASSANDRA-7919?focusedCommentId=14132529&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14132529] > lsb collisions are also possible serverside. > a sufficient solution would be to include references to pid and classloader > within lsb. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[08/10] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.3
Merge branch 'cassandra-3.0' into cassandra-3.3 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/bf917cbd Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/bf917cbd Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/bf917cbd Branch: refs/heads/cassandra-3.3 Commit: bf917cbd90bcaa5be8b76e857e58561a00a82c8d Parents: a6170ef 2445ea7 Author: Sylvain Lebresne Authored: Mon Jan 18 15:25:51 2016 +0100 Committer: Sylvain Lebresne Committed: Mon Jan 18 15:25:51 2016 +0100 -- CHANGES.txt | 1 + .../org/apache/cassandra/service/StartupChecks.java | 2 +- src/java/org/apache/cassandra/utils/SigarLibrary.java | 9 - src/java/org/apache/cassandra/utils/UUIDGen.java| 12 4 files changed, 22 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/bf917cbd/CHANGES.txt -- diff --cc CHANGES.txt index c7a5c23,41978ba..840c795 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -11,7 -7,18 +11,8 @@@ Merged from 3.0 tombstone (CASSANDRA-10743) * MV should use the maximum timestamp of the primary key (CASSANDRA-10910) * Fix potential assertion error during compaction (CASSANDRA-10944) - * Fix counting of received sstables in streaming (CASSANDRA-10949) - * Implement hints compression (CASSANDRA-9428) - * Fix potential assertion error when reading static columns (CASSANDRA-10903) - * Avoid NoSuchElementException when executing empty batch (CASSANDRA-10711) - * Avoid building PartitionUpdate in toString (CASSANDRA-10897) - * Reduce heap spent when receiving many SSTables (CASSANDRA-10797) - * Add back support for 3rd party auth providers to bulk loader (CASSANDRA-10873) - * Eliminate the dependency on jgrapht for UDT resolution (CASSANDRA-10653) - * (Hadoop) Close Clusters and Sessions in Hadoop Input/Output classes (CASSANDRA-10837) - * Fix sstableloader not working with upper case keyspace name (CASSANDRA-10806) Merged from 2.2: + * Make UUID LSB unique per process (CASSANDRA-7925) * Avoid NPE when performing sstable tasks (scrub etc.) (CASSANDRA-10980) * Make sure client gets tombstone overwhelmed warning (CASSANDRA-9465) * Fix error streaming section more than 2GB (CASSANDRA-10961) http://git-wip-us.apache.org/repos/asf/cassandra/blob/bf917cbd/src/java/org/apache/cassandra/service/StartupChecks.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/bf917cbd/src/java/org/apache/cassandra/utils/UUIDGen.java --
[02/10] cassandra git commit: Make UUID LSB unique per-process
Make UUID LSB unique per-process
patch by slebresne; reviewed by benedict for CASSANDRA-7925
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/47e8ef9e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/47e8ef9e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/47e8ef9e
Branch: refs/heads/cassandra-3.0
Commit: 47e8ef9e9ce70c54115681f854f483a53992c988
Parents: 79b9492
Author: Sylvain Lebresne
Authored: Fri Jan 15 15:25:03 2016 +0100
Committer: Sylvain Lebresne
Committed: Mon Jan 18 15:24:50 2016 +0100
--
CHANGES.txt | 1 +
.../org/apache/cassandra/service/StartupChecks.java | 2 +-
src/java/org/apache/cassandra/utils/SigarLibrary.java | 9 -
src/java/org/apache/cassandra/utils/UUIDGen.java| 12
4 files changed, 22 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/47e8ef9e/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 2bfba80..f571c29 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.2.5
+ * Make UUID LSB unique per process (CASSANDRA-7925)
* Avoid NPE when performing sstable tasks (scrub etc.) (CASSANDRA-10980)
* Make sure client gets tombstone overwhelmed warning (CASSANDRA-9465)
* Fix error streaming section more than 2GB (CASSANDRA-10961)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/47e8ef9e/src/java/org/apache/cassandra/service/StartupChecks.java
--
diff --git a/src/java/org/apache/cassandra/service/StartupChecks.java
b/src/java/org/apache/cassandra/service/StartupChecks.java
index 19f32b6..34bc824 100644
--- a/src/java/org/apache/cassandra/service/StartupChecks.java
+++ b/src/java/org/apache/cassandra/service/StartupChecks.java
@@ -195,7 +195,7 @@ public class StartupChecks
{
public void execute()
{
-new SigarLibrary().warnIfRunningInDegradedMode();
+SigarLibrary.instance.warnIfRunningInDegradedMode();
}
};
http://git-wip-us.apache.org/repos/asf/cassandra/blob/47e8ef9e/src/java/org/apache/cassandra/utils/SigarLibrary.java
--
diff --git a/src/java/org/apache/cassandra/utils/SigarLibrary.java
b/src/java/org/apache/cassandra/utils/SigarLibrary.java
index 7cf4d71..0312204 100644
--- a/src/java/org/apache/cassandra/utils/SigarLibrary.java
+++ b/src/java/org/apache/cassandra/utils/SigarLibrary.java
@@ -25,6 +25,8 @@ public class SigarLibrary
{
private Logger logger = LoggerFactory.getLogger(SigarLibrary.class);
+public static final SigarLibrary instance = new SigarLibrary();
+
private Sigar sigar;
private FileSystemMap mounts = null;
private boolean initialized = false;
@@ -37,7 +39,7 @@ public class SigarLibrary
// TODO: Determine if file system is remote or local
// TODO: Determine if disk latency is within acceptable limits
-public SigarLibrary()
+private SigarLibrary()
{
logger.info("Initializing SIGAR library");
try
@@ -154,6 +156,11 @@ public class SigarLibrary
}
}
+public long getPid()
+{
+return initialized ? sigar.getPid() : -1;
+}
+
public void warnIfRunningInDegradedMode()
{
if (initialized)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/47e8ef9e/src/java/org/apache/cassandra/utils/UUIDGen.java
--
diff --git a/src/java/org/apache/cassandra/utils/UUIDGen.java
b/src/java/org/apache/cassandra/utils/UUIDGen.java
index a777a50..df07e1f 100644
--- a/src/java/org/apache/cassandra/utils/UUIDGen.java
+++ b/src/java/org/apache/cassandra/utils/UUIDGen.java
@@ -26,6 +26,7 @@ import java.util.Random;
import java.util.UUID;
import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Charsets;
/**
@@ -309,10 +310,21 @@ public class UUIDGen
{
try
{
+// Identify the host.
MessageDigest messageDigest = MessageDigest.getInstance("MD5");
for(InetAddress addr : data)
messageDigest.update(addr.getAddress());
+// Identify the process on the load: we use both the PID and class
loader hash.
+long pid = SigarLibrary.instance.getPid();
+if (pid < 0)
+pid = new Random(System.currentTimeMillis()).nextLong();
+FBUtilities.updateWithLong(messageDigest, pid);
+
+ClassLoader loader = UUIDGen.class.getClassLoader();
+int loaderId = loader != null ? System.identityHashCode
[06/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0
Merge branch 'cassandra-2.2' into cassandra-3.0 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/2445ea77 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/2445ea77 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/2445ea77 Branch: refs/heads/cassandra-3.0 Commit: 2445ea77ec7951cf738db16be9b04fb793de73be Parents: a3360af 47e8ef9 Author: Sylvain Lebresne Authored: Mon Jan 18 15:25:38 2016 +0100 Committer: Sylvain Lebresne Committed: Mon Jan 18 15:25:38 2016 +0100 -- CHANGES.txt | 1 + .../org/apache/cassandra/service/StartupChecks.java | 2 +- src/java/org/apache/cassandra/utils/SigarLibrary.java | 9 - src/java/org/apache/cassandra/utils/UUIDGen.java| 12 4 files changed, 22 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/2445ea77/CHANGES.txt -- diff --cc CHANGES.txt index 478fe08,f571c29..41978ba --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,23 -1,5 +1,24 @@@ -2.2.5 +3.0.3 + * Remove checksum files after replaying hints (CASSANDRA-10947) + * Support passing base table metadata to custom 2i validation (CASSANDRA-10924) + * Ensure stale index entries are purged during reads (CASSANDRA-11013) + * Fix AssertionError when removing from list using UPDATE (CASSANDRA-10954) + * Fix UnsupportedOperationException when reading old sstable with range + tombstone (CASSANDRA-10743) + * MV should use the maximum timestamp of the primary key (CASSANDRA-10910) + * Fix potential assertion error during compaction (CASSANDRA-10944) + * Fix counting of received sstables in streaming (CASSANDRA-10949) + * Implement hints compression (CASSANDRA-9428) + * Fix potential assertion error when reading static columns (CASSANDRA-10903) + * Avoid NoSuchElementException when executing empty batch (CASSANDRA-10711) + * Avoid building PartitionUpdate in toString (CASSANDRA-10897) + * Reduce heap spent when receiving many SSTables (CASSANDRA-10797) + * Add back support for 3rd party auth providers to bulk loader (CASSANDRA-10873) + * Eliminate the dependency on jgrapht for UDT resolution (CASSANDRA-10653) + * (Hadoop) Close Clusters and Sessions in Hadoop Input/Output classes (CASSANDRA-10837) + * Fix sstableloader not working with upper case keyspace name (CASSANDRA-10806) +Merged from 2.2: + * Make UUID LSB unique per process (CASSANDRA-7925) * Avoid NPE when performing sstable tasks (scrub etc.) (CASSANDRA-10980) * Make sure client gets tombstone overwhelmed warning (CASSANDRA-9465) * Fix error streaming section more than 2GB (CASSANDRA-10961) http://git-wip-us.apache.org/repos/asf/cassandra/blob/2445ea77/src/java/org/apache/cassandra/service/StartupChecks.java --
[04/10] cassandra git commit: Make UUID LSB unique per-process
Make UUID LSB unique per-process
patch by slebresne; reviewed by benedict for CASSANDRA-7925
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/47e8ef9e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/47e8ef9e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/47e8ef9e
Branch: refs/heads/trunk
Commit: 47e8ef9e9ce70c54115681f854f483a53992c988
Parents: 79b9492
Author: Sylvain Lebresne
Authored: Fri Jan 15 15:25:03 2016 +0100
Committer: Sylvain Lebresne
Committed: Mon Jan 18 15:24:50 2016 +0100
--
CHANGES.txt | 1 +
.../org/apache/cassandra/service/StartupChecks.java | 2 +-
src/java/org/apache/cassandra/utils/SigarLibrary.java | 9 -
src/java/org/apache/cassandra/utils/UUIDGen.java| 12
4 files changed, 22 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/47e8ef9e/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 2bfba80..f571c29 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.2.5
+ * Make UUID LSB unique per process (CASSANDRA-7925)
* Avoid NPE when performing sstable tasks (scrub etc.) (CASSANDRA-10980)
* Make sure client gets tombstone overwhelmed warning (CASSANDRA-9465)
* Fix error streaming section more than 2GB (CASSANDRA-10961)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/47e8ef9e/src/java/org/apache/cassandra/service/StartupChecks.java
--
diff --git a/src/java/org/apache/cassandra/service/StartupChecks.java
b/src/java/org/apache/cassandra/service/StartupChecks.java
index 19f32b6..34bc824 100644
--- a/src/java/org/apache/cassandra/service/StartupChecks.java
+++ b/src/java/org/apache/cassandra/service/StartupChecks.java
@@ -195,7 +195,7 @@ public class StartupChecks
{
public void execute()
{
-new SigarLibrary().warnIfRunningInDegradedMode();
+SigarLibrary.instance.warnIfRunningInDegradedMode();
}
};
http://git-wip-us.apache.org/repos/asf/cassandra/blob/47e8ef9e/src/java/org/apache/cassandra/utils/SigarLibrary.java
--
diff --git a/src/java/org/apache/cassandra/utils/SigarLibrary.java
b/src/java/org/apache/cassandra/utils/SigarLibrary.java
index 7cf4d71..0312204 100644
--- a/src/java/org/apache/cassandra/utils/SigarLibrary.java
+++ b/src/java/org/apache/cassandra/utils/SigarLibrary.java
@@ -25,6 +25,8 @@ public class SigarLibrary
{
private Logger logger = LoggerFactory.getLogger(SigarLibrary.class);
+public static final SigarLibrary instance = new SigarLibrary();
+
private Sigar sigar;
private FileSystemMap mounts = null;
private boolean initialized = false;
@@ -37,7 +39,7 @@ public class SigarLibrary
// TODO: Determine if file system is remote or local
// TODO: Determine if disk latency is within acceptable limits
-public SigarLibrary()
+private SigarLibrary()
{
logger.info("Initializing SIGAR library");
try
@@ -154,6 +156,11 @@ public class SigarLibrary
}
}
+public long getPid()
+{
+return initialized ? sigar.getPid() : -1;
+}
+
public void warnIfRunningInDegradedMode()
{
if (initialized)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/47e8ef9e/src/java/org/apache/cassandra/utils/UUIDGen.java
--
diff --git a/src/java/org/apache/cassandra/utils/UUIDGen.java
b/src/java/org/apache/cassandra/utils/UUIDGen.java
index a777a50..df07e1f 100644
--- a/src/java/org/apache/cassandra/utils/UUIDGen.java
+++ b/src/java/org/apache/cassandra/utils/UUIDGen.java
@@ -26,6 +26,7 @@ import java.util.Random;
import java.util.UUID;
import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Charsets;
/**
@@ -309,10 +310,21 @@ public class UUIDGen
{
try
{
+// Identify the host.
MessageDigest messageDigest = MessageDigest.getInstance("MD5");
for(InetAddress addr : data)
messageDigest.update(addr.getAddress());
+// Identify the process on the load: we use both the PID and class
loader hash.
+long pid = SigarLibrary.instance.getPid();
+if (pid < 0)
+pid = new Random(System.currentTimeMillis()).nextLong();
+FBUtilities.updateWithLong(messageDigest, pid);
+
+ClassLoader loader = UUIDGen.class.getClassLoader();
+int loaderId = loader != null ? System.identityHashCode(loader)
[05/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0
Merge branch 'cassandra-2.2' into cassandra-3.0 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/2445ea77 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/2445ea77 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/2445ea77 Branch: refs/heads/cassandra-3.3 Commit: 2445ea77ec7951cf738db16be9b04fb793de73be Parents: a3360af 47e8ef9 Author: Sylvain Lebresne Authored: Mon Jan 18 15:25:38 2016 +0100 Committer: Sylvain Lebresne Committed: Mon Jan 18 15:25:38 2016 +0100 -- CHANGES.txt | 1 + .../org/apache/cassandra/service/StartupChecks.java | 2 +- src/java/org/apache/cassandra/utils/SigarLibrary.java | 9 - src/java/org/apache/cassandra/utils/UUIDGen.java| 12 4 files changed, 22 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/2445ea77/CHANGES.txt -- diff --cc CHANGES.txt index 478fe08,f571c29..41978ba --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,23 -1,5 +1,24 @@@ -2.2.5 +3.0.3 + * Remove checksum files after replaying hints (CASSANDRA-10947) + * Support passing base table metadata to custom 2i validation (CASSANDRA-10924) + * Ensure stale index entries are purged during reads (CASSANDRA-11013) + * Fix AssertionError when removing from list using UPDATE (CASSANDRA-10954) + * Fix UnsupportedOperationException when reading old sstable with range + tombstone (CASSANDRA-10743) + * MV should use the maximum timestamp of the primary key (CASSANDRA-10910) + * Fix potential assertion error during compaction (CASSANDRA-10944) + * Fix counting of received sstables in streaming (CASSANDRA-10949) + * Implement hints compression (CASSANDRA-9428) + * Fix potential assertion error when reading static columns (CASSANDRA-10903) + * Avoid NoSuchElementException when executing empty batch (CASSANDRA-10711) + * Avoid building PartitionUpdate in toString (CASSANDRA-10897) + * Reduce heap spent when receiving many SSTables (CASSANDRA-10797) + * Add back support for 3rd party auth providers to bulk loader (CASSANDRA-10873) + * Eliminate the dependency on jgrapht for UDT resolution (CASSANDRA-10653) + * (Hadoop) Close Clusters and Sessions in Hadoop Input/Output classes (CASSANDRA-10837) + * Fix sstableloader not working with upper case keyspace name (CASSANDRA-10806) +Merged from 2.2: + * Make UUID LSB unique per process (CASSANDRA-7925) * Avoid NPE when performing sstable tasks (scrub etc.) (CASSANDRA-10980) * Make sure client gets tombstone overwhelmed warning (CASSANDRA-9465) * Fix error streaming section more than 2GB (CASSANDRA-10961) http://git-wip-us.apache.org/repos/asf/cassandra/blob/2445ea77/src/java/org/apache/cassandra/service/StartupChecks.java --
[01/10] cassandra git commit: Make UUID LSB unique per-process
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.2 79b949236 -> 47e8ef9e9
refs/heads/cassandra-3.0 a3360afa8 -> 2445ea77e
refs/heads/cassandra-3.3 a6170ef80 -> bf917cbd9
refs/heads/trunk 22f02fe60 -> ee5aec2d2
Make UUID LSB unique per-process
patch by slebresne; reviewed by benedict for CASSANDRA-7925
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/47e8ef9e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/47e8ef9e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/47e8ef9e
Branch: refs/heads/cassandra-2.2
Commit: 47e8ef9e9ce70c54115681f854f483a53992c988
Parents: 79b9492
Author: Sylvain Lebresne
Authored: Fri Jan 15 15:25:03 2016 +0100
Committer: Sylvain Lebresne
Committed: Mon Jan 18 15:24:50 2016 +0100
--
CHANGES.txt | 1 +
.../org/apache/cassandra/service/StartupChecks.java | 2 +-
src/java/org/apache/cassandra/utils/SigarLibrary.java | 9 -
src/java/org/apache/cassandra/utils/UUIDGen.java| 12
4 files changed, 22 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/47e8ef9e/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 2bfba80..f571c29 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.2.5
+ * Make UUID LSB unique per process (CASSANDRA-7925)
* Avoid NPE when performing sstable tasks (scrub etc.) (CASSANDRA-10980)
* Make sure client gets tombstone overwhelmed warning (CASSANDRA-9465)
* Fix error streaming section more than 2GB (CASSANDRA-10961)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/47e8ef9e/src/java/org/apache/cassandra/service/StartupChecks.java
--
diff --git a/src/java/org/apache/cassandra/service/StartupChecks.java
b/src/java/org/apache/cassandra/service/StartupChecks.java
index 19f32b6..34bc824 100644
--- a/src/java/org/apache/cassandra/service/StartupChecks.java
+++ b/src/java/org/apache/cassandra/service/StartupChecks.java
@@ -195,7 +195,7 @@ public class StartupChecks
{
public void execute()
{
-new SigarLibrary().warnIfRunningInDegradedMode();
+SigarLibrary.instance.warnIfRunningInDegradedMode();
}
};
http://git-wip-us.apache.org/repos/asf/cassandra/blob/47e8ef9e/src/java/org/apache/cassandra/utils/SigarLibrary.java
--
diff --git a/src/java/org/apache/cassandra/utils/SigarLibrary.java
b/src/java/org/apache/cassandra/utils/SigarLibrary.java
index 7cf4d71..0312204 100644
--- a/src/java/org/apache/cassandra/utils/SigarLibrary.java
+++ b/src/java/org/apache/cassandra/utils/SigarLibrary.java
@@ -25,6 +25,8 @@ public class SigarLibrary
{
private Logger logger = LoggerFactory.getLogger(SigarLibrary.class);
+public static final SigarLibrary instance = new SigarLibrary();
+
private Sigar sigar;
private FileSystemMap mounts = null;
private boolean initialized = false;
@@ -37,7 +39,7 @@ public class SigarLibrary
// TODO: Determine if file system is remote or local
// TODO: Determine if disk latency is within acceptable limits
-public SigarLibrary()
+private SigarLibrary()
{
logger.info("Initializing SIGAR library");
try
@@ -154,6 +156,11 @@ public class SigarLibrary
}
}
+public long getPid()
+{
+return initialized ? sigar.getPid() : -1;
+}
+
public void warnIfRunningInDegradedMode()
{
if (initialized)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/47e8ef9e/src/java/org/apache/cassandra/utils/UUIDGen.java
--
diff --git a/src/java/org/apache/cassandra/utils/UUIDGen.java
b/src/java/org/apache/cassandra/utils/UUIDGen.java
index a777a50..df07e1f 100644
--- a/src/java/org/apache/cassandra/utils/UUIDGen.java
+++ b/src/java/org/apache/cassandra/utils/UUIDGen.java
@@ -26,6 +26,7 @@ import java.util.Random;
import java.util.UUID;
import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Charsets;
/**
@@ -309,10 +310,21 @@ public class UUIDGen
{
try
{
+// Identify the host.
MessageDigest messageDigest = MessageDigest.getInstance("MD5");
for(InetAddress addr : data)
messageDigest.update(addr.getAddress());
+// Identify the process on the load: we use both the PID and class
loader hash.
+long pid = SigarLibrary.instance.getPid();
+if (pid < 0)
+pid = new Random(Sys
[10/10] cassandra git commit: Merge branch 'cassandra-3.3' into trunk
Merge branch 'cassandra-3.3' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ee5aec2d Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ee5aec2d Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ee5aec2d Branch: refs/heads/trunk Commit: ee5aec2d2ed7795d64a052e191af4aec5c807dba Parents: 22f02fe bf917cb Author: Sylvain Lebresne Authored: Mon Jan 18 15:26:06 2016 +0100 Committer: Sylvain Lebresne Committed: Mon Jan 18 15:26:06 2016 +0100 -- CHANGES.txt | 1 + .../org/apache/cassandra/service/StartupChecks.java | 2 +- src/java/org/apache/cassandra/utils/SigarLibrary.java | 9 - src/java/org/apache/cassandra/utils/UUIDGen.java| 12 4 files changed, 22 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/ee5aec2d/CHANGES.txt --
[03/10] cassandra git commit: Make UUID LSB unique per-process
Make UUID LSB unique per-process
patch by slebresne; reviewed by benedict for CASSANDRA-7925
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/47e8ef9e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/47e8ef9e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/47e8ef9e
Branch: refs/heads/cassandra-3.3
Commit: 47e8ef9e9ce70c54115681f854f483a53992c988
Parents: 79b9492
Author: Sylvain Lebresne
Authored: Fri Jan 15 15:25:03 2016 +0100
Committer: Sylvain Lebresne
Committed: Mon Jan 18 15:24:50 2016 +0100
--
CHANGES.txt | 1 +
.../org/apache/cassandra/service/StartupChecks.java | 2 +-
src/java/org/apache/cassandra/utils/SigarLibrary.java | 9 -
src/java/org/apache/cassandra/utils/UUIDGen.java| 12
4 files changed, 22 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/47e8ef9e/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 2bfba80..f571c29 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.2.5
+ * Make UUID LSB unique per process (CASSANDRA-7925)
* Avoid NPE when performing sstable tasks (scrub etc.) (CASSANDRA-10980)
* Make sure client gets tombstone overwhelmed warning (CASSANDRA-9465)
* Fix error streaming section more than 2GB (CASSANDRA-10961)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/47e8ef9e/src/java/org/apache/cassandra/service/StartupChecks.java
--
diff --git a/src/java/org/apache/cassandra/service/StartupChecks.java
b/src/java/org/apache/cassandra/service/StartupChecks.java
index 19f32b6..34bc824 100644
--- a/src/java/org/apache/cassandra/service/StartupChecks.java
+++ b/src/java/org/apache/cassandra/service/StartupChecks.java
@@ -195,7 +195,7 @@ public class StartupChecks
{
public void execute()
{
-new SigarLibrary().warnIfRunningInDegradedMode();
+SigarLibrary.instance.warnIfRunningInDegradedMode();
}
};
http://git-wip-us.apache.org/repos/asf/cassandra/blob/47e8ef9e/src/java/org/apache/cassandra/utils/SigarLibrary.java
--
diff --git a/src/java/org/apache/cassandra/utils/SigarLibrary.java
b/src/java/org/apache/cassandra/utils/SigarLibrary.java
index 7cf4d71..0312204 100644
--- a/src/java/org/apache/cassandra/utils/SigarLibrary.java
+++ b/src/java/org/apache/cassandra/utils/SigarLibrary.java
@@ -25,6 +25,8 @@ public class SigarLibrary
{
private Logger logger = LoggerFactory.getLogger(SigarLibrary.class);
+public static final SigarLibrary instance = new SigarLibrary();
+
private Sigar sigar;
private FileSystemMap mounts = null;
private boolean initialized = false;
@@ -37,7 +39,7 @@ public class SigarLibrary
// TODO: Determine if file system is remote or local
// TODO: Determine if disk latency is within acceptable limits
-public SigarLibrary()
+private SigarLibrary()
{
logger.info("Initializing SIGAR library");
try
@@ -154,6 +156,11 @@ public class SigarLibrary
}
}
+public long getPid()
+{
+return initialized ? sigar.getPid() : -1;
+}
+
public void warnIfRunningInDegradedMode()
{
if (initialized)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/47e8ef9e/src/java/org/apache/cassandra/utils/UUIDGen.java
--
diff --git a/src/java/org/apache/cassandra/utils/UUIDGen.java
b/src/java/org/apache/cassandra/utils/UUIDGen.java
index a777a50..df07e1f 100644
--- a/src/java/org/apache/cassandra/utils/UUIDGen.java
+++ b/src/java/org/apache/cassandra/utils/UUIDGen.java
@@ -26,6 +26,7 @@ import java.util.Random;
import java.util.UUID;
import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Charsets;
/**
@@ -309,10 +310,21 @@ public class UUIDGen
{
try
{
+// Identify the host.
MessageDigest messageDigest = MessageDigest.getInstance("MD5");
for(InetAddress addr : data)
messageDigest.update(addr.getAddress());
+// Identify the process on the load: we use both the PID and class
loader hash.
+long pid = SigarLibrary.instance.getPid();
+if (pid < 0)
+pid = new Random(System.currentTimeMillis()).nextLong();
+FBUtilities.updateWithLong(messageDigest, pid);
+
+ClassLoader loader = UUIDGen.class.getClassLoader();
+int loaderId = loader != null ? System.identityHashCode
[09/10] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.3
Merge branch 'cassandra-3.0' into cassandra-3.3 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/bf917cbd Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/bf917cbd Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/bf917cbd Branch: refs/heads/trunk Commit: bf917cbd90bcaa5be8b76e857e58561a00a82c8d Parents: a6170ef 2445ea7 Author: Sylvain Lebresne Authored: Mon Jan 18 15:25:51 2016 +0100 Committer: Sylvain Lebresne Committed: Mon Jan 18 15:25:51 2016 +0100 -- CHANGES.txt | 1 + .../org/apache/cassandra/service/StartupChecks.java | 2 +- src/java/org/apache/cassandra/utils/SigarLibrary.java | 9 - src/java/org/apache/cassandra/utils/UUIDGen.java| 12 4 files changed, 22 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/bf917cbd/CHANGES.txt -- diff --cc CHANGES.txt index c7a5c23,41978ba..840c795 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -11,7 -7,18 +11,8 @@@ Merged from 3.0 tombstone (CASSANDRA-10743) * MV should use the maximum timestamp of the primary key (CASSANDRA-10910) * Fix potential assertion error during compaction (CASSANDRA-10944) - * Fix counting of received sstables in streaming (CASSANDRA-10949) - * Implement hints compression (CASSANDRA-9428) - * Fix potential assertion error when reading static columns (CASSANDRA-10903) - * Avoid NoSuchElementException when executing empty batch (CASSANDRA-10711) - * Avoid building PartitionUpdate in toString (CASSANDRA-10897) - * Reduce heap spent when receiving many SSTables (CASSANDRA-10797) - * Add back support for 3rd party auth providers to bulk loader (CASSANDRA-10873) - * Eliminate the dependency on jgrapht for UDT resolution (CASSANDRA-10653) - * (Hadoop) Close Clusters and Sessions in Hadoop Input/Output classes (CASSANDRA-10837) - * Fix sstableloader not working with upper case keyspace name (CASSANDRA-10806) Merged from 2.2: + * Make UUID LSB unique per process (CASSANDRA-7925) * Avoid NPE when performing sstable tasks (scrub etc.) (CASSANDRA-10980) * Make sure client gets tombstone overwhelmed warning (CASSANDRA-9465) * Fix error streaming section more than 2GB (CASSANDRA-10961) http://git-wip-us.apache.org/repos/asf/cassandra/blob/bf917cbd/src/java/org/apache/cassandra/service/StartupChecks.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/bf917cbd/src/java/org/apache/cassandra/utils/UUIDGen.java --
[07/10] cassandra git commit: Merge branch 'cassandra-2.2' into cassandra-3.0
Merge branch 'cassandra-2.2' into cassandra-3.0 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/2445ea77 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/2445ea77 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/2445ea77 Branch: refs/heads/trunk Commit: 2445ea77ec7951cf738db16be9b04fb793de73be Parents: a3360af 47e8ef9 Author: Sylvain Lebresne Authored: Mon Jan 18 15:25:38 2016 +0100 Committer: Sylvain Lebresne Committed: Mon Jan 18 15:25:38 2016 +0100 -- CHANGES.txt | 1 + .../org/apache/cassandra/service/StartupChecks.java | 2 +- src/java/org/apache/cassandra/utils/SigarLibrary.java | 9 - src/java/org/apache/cassandra/utils/UUIDGen.java| 12 4 files changed, 22 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/2445ea77/CHANGES.txt -- diff --cc CHANGES.txt index 478fe08,f571c29..41978ba --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,23 -1,5 +1,24 @@@ -2.2.5 +3.0.3 + * Remove checksum files after replaying hints (CASSANDRA-10947) + * Support passing base table metadata to custom 2i validation (CASSANDRA-10924) + * Ensure stale index entries are purged during reads (CASSANDRA-11013) + * Fix AssertionError when removing from list using UPDATE (CASSANDRA-10954) + * Fix UnsupportedOperationException when reading old sstable with range + tombstone (CASSANDRA-10743) + * MV should use the maximum timestamp of the primary key (CASSANDRA-10910) + * Fix potential assertion error during compaction (CASSANDRA-10944) + * Fix counting of received sstables in streaming (CASSANDRA-10949) + * Implement hints compression (CASSANDRA-9428) + * Fix potential assertion error when reading static columns (CASSANDRA-10903) + * Avoid NoSuchElementException when executing empty batch (CASSANDRA-10711) + * Avoid building PartitionUpdate in toString (CASSANDRA-10897) + * Reduce heap spent when receiving many SSTables (CASSANDRA-10797) + * Add back support for 3rd party auth providers to bulk loader (CASSANDRA-10873) + * Eliminate the dependency on jgrapht for UDT resolution (CASSANDRA-10653) + * (Hadoop) Close Clusters and Sessions in Hadoop Input/Output classes (CASSANDRA-10837) + * Fix sstableloader not working with upper case keyspace name (CASSANDRA-10806) +Merged from 2.2: + * Make UUID LSB unique per process (CASSANDRA-7925) * Avoid NPE when performing sstable tasks (scrub etc.) (CASSANDRA-10980) * Make sure client gets tombstone overwhelmed warning (CASSANDRA-9465) * Fix error streaming section more than 2GB (CASSANDRA-10961) http://git-wip-us.apache.org/repos/asf/cassandra/blob/2445ea77/src/java/org/apache/cassandra/service/StartupChecks.java --
[jira] [Resolved] (CASSANDRA-10973) disk_balance_decommission_test is failing on trunk
[
https://issues.apache.org/jira/browse/CASSANDRA-10973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcus Eriksson resolved CASSANDRA-10973.
-
Resolution: Duplicate
Fix Version/s: (was: 3.x)
> disk_balance_decommission_test is failing on trunk
> --
>
> Key: CASSANDRA-10973
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10973
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
>Reporter: Alan Boudreault
>Assignee: Marcus Eriksson
> Labels: dtest
>
> http://cassci.datastax.com/job/trunk_dtest/891/testReport/junit/disk_balance_test/TestDiskBalance/disk_balance_decommission_test/
> {code}
> ==
> FAIL: disk_balance_decommission_test (disk_balance_test.TestDiskBalance)
> --
> Traceback (most recent call last):
> File "/home/aboudreault/git/cstar/cassandra-dtest/disk_balance_test.py",
> line 74, in disk_balance_decommission_test
> self.assert_balanced(node)
> File "/home/aboudreault/git/cstar/cassandra-dtest/disk_balance_test.py",
> line 127, in assert_balanced
> assert_almost_equal(*sums, error=0.2, error_message=node.name)
> File "/home/aboudreault/git/cstar/cassandra-dtest/assertions.py", line 65,
> in assert_almost_equal
> assert vmin > vmax * (1.0 - error) or vmin == vmax, "values not within
> %.2f%% of the max: %s (%s)" % (error * 100, args, kwargs['error_message'])
> AssertionError: values not within 20.00% of the max: (482095, 477840, 612940)
> (node2)
> >> begin captured logging <<
> dtest: DEBUG: cluster ccm directory: /tmp/dtest-SLbi3e
> - >> end captured logging << -
> --
> Ran 1 test in 121.295s
> FAILED (failures=1)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-7715) Add a credentials cache to the PasswordAuthenticator
[ https://issues.apache.org/jira/browse/CASSANDRA-7715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Adamson updated CASSANDRA-7715: Reviewer: Mike Adamson > Add a credentials cache to the PasswordAuthenticator > > > Key: CASSANDRA-7715 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7715 > Project: Cassandra > Issue Type: New Feature > Components: CQL >Reporter: Mike Adamson >Assignee: Sam Tunnicliffe >Priority: Minor > Fix For: 3.x > > > If the PasswordAuthenticator cached credentials for a short time it would > reduce the overhead of user journeys when they need to do multiple > authentications in quick succession. > This cache should work in the same way as the cache in CassandraAuthorizer in > that if it's TTL is set to 0 the cache will be disabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7715) Add a credentials cache to the PasswordAuthenticator
[
https://issues.apache.org/jira/browse/CASSANDRA-7715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15105461#comment-15105461
]
Mike Adamson commented on CASSANDRA-7715:
-
I'm not sure about being able to modify cache settings via JMX. This seems like
an attack vector to me. In a (very hypothetical) scenario, if an attacker got
hold of user credentials and JMX credentials, they could change the cache
timeouts so that a cache containing a known password would never change. I know
this is a fragile attack because all an administrator would need to do would be
to change the cache timeouts themselves and the cache would reinitialize.
I think that the cache should be be cleared for a user if the authentication
fails. Another small issue but it gives a small advantage to the attacker if
it's left in after a failure. This only really affects the {{CredentialsCache}}
in the {{PasswordAuthenticator}}.
Could we have a method to invalidate the cache for a specific user? I'm just
thinking that the only option an administrator has if they change a users
password is to {{invalidate}} the whole cache.
In cassandra.yaml the {{credentials_update_interval_in_ms}} value (although
commented out) ought to be the same value as {{credentials_validity_in_ms}}. We
are saying that it defaults to the same value so it probably ought to have it.
It would be nice if {{MBEAN_NAME_BASE}} was (somehow) overridable by concrete
implementations. This would make this class more usable by 3rd party
implementers.
> Add a credentials cache to the PasswordAuthenticator
>
>
> Key: CASSANDRA-7715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7715
> Project: Cassandra
> Issue Type: New Feature
> Components: CQL
>Reporter: Mike Adamson
>Assignee: Sam Tunnicliffe
>Priority: Minor
> Fix For: 3.x
>
>
> If the PasswordAuthenticator cached credentials for a short time it would
> reduce the overhead of user journeys when they need to do multiple
> authentications in quick succession.
> This cache should work in the same way as the cache in CassandraAuthorizer in
> that if it's TTL is set to 0 the cache will be disabled.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9949) maxPurgeableTimestamp needs to check memtables too
[ https://issues.apache.org/jira/browse/CASSANDRA-9949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15105471#comment-15105471 ] Marcus Eriksson commented on CASSANDRA-9949: The approach looks good to me, two comments; * could you rephrase [this comment|https://github.com/stef1927/cassandra/blob/aabec6bab0d4558b7ce9bfbdec17909ff0b90bc9/src/java/org/apache/cassandra/db/AtomicBTreeColumns.java#L474-L476]? I don't get why we can't use the reconciled version. * We should probably check the memtable min timestamp in CompactionController.getFullyExpiredSSTables as well > maxPurgeableTimestamp needs to check memtables too > -- > > Key: CASSANDRA-9949 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9949 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths >Reporter: Jonathan Ellis >Assignee: Stefania > Fix For: 2.2.x, 3.0.x, 3.x > > > overlapIterator/maxPurgeableTimestamp don't include the memtables, so a > very-out-of-order write could be ignored -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8103) Secondary Indices for Static Columns
[
https://issues.apache.org/jira/browse/CASSANDRA-8103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15105478#comment-15105478
]
Sam Tunnicliffe commented on CASSANDRA-8103:
Thanks, v2 is mostly fine, just a couple of comments:
* {{filterStaleEntries}} still takes an unused {{partitionKey}} argument
* {{SecondaryIndexOnStaticColumnTest::testRegularColumn}} isn't related to
regular columns, but is concerned with static column. Could you rename it
please.
* In {{StatementRestrictions.hasNonPrimaryRegularColumnsRestriction}} &
associated getter, the {{NonPrimary}} is redundant.
* Could you remove the {{@Override}} annotations you added as per the [coding
style guide|https://wiki.apache.org/cassandra/CodeStyle]
> Secondary Indices for Static Columns
>
>
> Key: CASSANDRA-8103
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8103
> Project: Cassandra
> Issue Type: New Feature
> Components: CQL
>Reporter: Ron Cohen
>Assignee: Taiyuan Zhang
> Fix For: 3.x
>
> Attachments: 8103-v2.patch, 8103.patch, in_progress.patch,
> smoke-test.cql
>
>
> We should add secondary index support for static columns.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10937) OOM on multiple nodes on write load (v. 3.0.0), problem also present on DSE-4.8.3, but there it survives more time
[
https://issues.apache.org/jira/browse/CASSANDRA-10937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15105495#comment-15105495
]
Jack Krupansky commented on CASSANDRA-10937:
I still don't see any reason to believe that there is a bug here and that the
primary issue is that you are overloading the cluster. Sure, Cassandra should
do a better job of shedding/failing excessive incoming requests, and there is
an open Jira ticket to add just such a freature, but even with that new
feature, the net effect will be the same - it will still be up to the
application and operations to properly size the cluster and throttle
application load before it gets to Cassandra.
OOM is not typically an indication of a software bug. Sure, sometimes code has
memory leaks, but with a highly dynamic system such as Cassandra, it typically
means either a misconfigured JVM or just very heavy load. Sometimes OOM simply
means that there is a lot of background processing going on (like compactions
or hinted handoff) that is having trouble keeping up with incoming requests.
Sometimes OOM occurs because you have too large a heap which defers GC but then
GC takes too long and further incoming requests simply generate more pressure
on the heap faster than that massive GC can deal with it.
It is indeed tricky to make sure the JVM has enough heap but not too much. DSE
typically runs with a larger heap by default. You can try increasing your heap
to 10 or 12G. But if you make the heap too big, the big GC can bite you as
described above. In that case, the heap needs to be reduced. Typically you
don't need a heap smaller than 8 GB. If OOM occurs with a 8 GB heap it
typically means the load on that node is simply too heavy.
Be sure to review the recommendations in this blog post on reasonable
recommendations:
http://www.datastax.com/dev/blog/how-not-to-benchmark-cassandra
A few questions that will help us better understand what you are really trying
to do:
1. How much reading are you doing and when relative to writes?
2. Are you doing any updates or deletes? (Cause compaction, which can fall
behind your write/update load.)
3. How much data is on the cluster (rows)?
4. How many tables?
5. What RF? RF=3 would be the recommendation, but if you have a heavy read load
you may need RF=5.
6. Have you tested using cassandra-stress? That's the gold standard around here.
7. Are your clients using token-aware routing? (Otherwise a write must be
bounced from the coordinating node to the node owning the token for the
partition key.)
8. Are you using batches for your writes? If, so, do all the writes in one
batch have the same partition key? (If not, adds more network hops.)
> OOM on multiple nodes on write load (v. 3.0.0), problem also present on
> DSE-4.8.3, but there it survives more time
> --
>
> Key: CASSANDRA-10937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10937
> Project: Cassandra
> Issue Type: Bug
> Environment: Cassandra : 3.0.0
> Installed as open archive, no connection to any OS specific installer.
> Java:
> Java(TM) SE Runtime Environment (build 1.8.0_65-b17)
> OS :
> Linux version 2.6.32-431.el6.x86_64
> (mockbu...@x86-023.build.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red
> Hat 4.4.7-4) (GCC) ) #1 SMP Sun Nov 10 22:19:54 EST 2013
> We have:
> 8 guests ( Linux OS as above) on 2 (VMWare managed) physical hosts. Each
> physical host keeps 4 guests.
> Physical host parameters(shared by all 4 guests):
> Model: HP ProLiant DL380 Gen9
> Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
> 46 logical processors.
> Hyperthreading - enabled
> Each guest assigned to have:
> 1 disk 300 Gb for seq. log (NOT SSD)
> 1 disk 4T for data (NOT SSD)
> 11 CPU cores
> Disks are local, not shared.
> Memory on each host - 24 Gb total.
> 8 (or 6, tested both) Gb - cassandra heap
> (lshw and cpuinfo attached in file test2.rar)
>Reporter: Peter Kovgan
>Priority: Critical
> Attachments: gc-stat.txt, more-logs.rar, some-heap-stats.rar,
> test2.rar, test3.rar, test4.rar, test5.rar, test_2.1.rar,
> test_2.1_logs_older.rar, test_2.1_restart_attempt_log.rar
>
>
> 8 cassandra nodes.
> Load test started with 4 clients(different and not equal machines), each
> running 1000 threads.
> Each thread assigned in round-robin way to run one of 4 different inserts.
> Consistency->ONE.
> I attach the full CQL schema of tables and the query of insert.
> Replication factor - 2:
> create keyspace OBLREPOSITORY_NY with replication =
> {'class':'NetworkTopologyStrategy','NY':2};
> Initiall throughput is:
> 215.000 inserts /sec
> or
> 54Mb/sec, considering single insert size a bit larger than 256byte.
> Data:
> all fields(5-6) are short strings, e
[jira] [Commented] (CASSANDRA-10924) Pass base table's metadata to Index.validateOptions
[
https://issues.apache.org/jira/browse/CASSANDRA-10924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15105505#comment-15105505
]
Andrés de la Peña commented on CASSANDRA-10924:
---
Great news! It will be very useful for custom implementations. Thanks a lot for
your help!
> Pass base table's metadata to Index.validateOptions
> ---
>
> Key: CASSANDRA-10924
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10924
> Project: Cassandra
> Issue Type: Improvement
> Components: CQL, Local Write-Read Paths
>Reporter: Andrés de la Peña
>Assignee: Andrés de la Peña
>Priority: Minor
> Labels: 2i, index, validation
> Fix For: 3.0.3, 3.3
>
> Attachments: CASSANDRA-10924-v0.diff, CASSANDRA-10924-v1.diff
>
>
> Some custom index implementations require the base table's metadata to
> validate their creation options. For example, the options of these
> implementations can contain information about which base table's columns are
> going to be indexed and how, so the implementation needs to know the
> existence and the type of the columns to be indexed to properly validate.
> The attached patch proposes to add base table's {{CFMetaData}} to Index'
> optional static method to validate the custom index options:
> {{public static Map validateOptions(CFMetaData cfm,
> Map options);}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-11028) Streaming errors caused by corrupt tables need more logging
Jeff Ferland created CASSANDRA-11028: Summary: Streaming errors caused by corrupt tables need more logging Key: CASSANDRA-11028 URL: https://issues.apache.org/jira/browse/CASSANDRA-11028 Project: Cassandra Issue Type: Bug Reporter: Jeff Ferland Example output: ERROR [STREAM-IN-/10.0.10.218] 2016-01-17 16:01:38,431 StreamSession.java:505 - [Stream #e6ca4590-bc66-11e5-84be-571ffcecc993] Streaming error occurred java.lang.IllegalArgumentException: Unknown type 0 In some cases logging shows a message more like: ERROR [STREAM-IN-/10.0.10.12] 2016-01-05 14:44:38,690 StreamSession.java:505 - [Stream #472d28e0-b347-11e5-8b40-bb4d80df86f4] Streaming error occurred java.io.IOException: Too many retries for Header (cfId: 6b262d58-8730-36ca-8e3e-f0a40beaf92f, #0, version: ka, estimated keys: 58880, transfer size: 2159040, compressed?: true, repairedAt: 0) In the majority of cases, however, no information identifying the column family is shown, and never identifying the source file that was being streamed. Errors do no stop the streaming process, but do mark the streaming as failed at the end. This usually results in a log message pattern like: INFO [StreamReceiveTask:252] 2016-01-18 04:45:01,190 StreamResultFuture.java:180 - [Stream #e6ca4590-bc66-11e5-84be-571ffcecc993] Session with /10.0.10.219 is complete WARN [StreamReceiveTask:252] 2016-01-18 04:45:01,215 StreamResultFuture.java:207 - [Stream #e6ca4590-bc66-11e5-84be-571ffcecc993] Stream failed ERROR [main] 2016-01-18 04:45:01,217 CassandraDaemon.java:579 - Exception encountered during startup ... which is highly confusing given the error occurred hours before. Request: more detail in logging messages for stream failure indicating what column family was being used, and if possible a clarification between network issues and corrupt file issues. Actual cause of errors / solution is running nodetool scrub on the offending node. It's rather expensive scrubbing the whole space blindly versus targeting issue tables. In our particular case, out of order keys were caused by a bug in a previous version of Cassandra. WARN [CompactionExecutor:19552] 2016-01-18 16:02:10,155 OutputHandler.java:52 - 378490 out of order rows found while scrubbing SSTableReader(path='/mnt/cassandra/data/keyspace/cf-888a52f96d1d389790ee586a6100916c/keyspace-cf-ka-133-Data.db'); Those have been written (in order) to a new sstable (SSTableReader(path='/mnt/cassandra/data/keyspace/cf-888a52f96d1d389790ee586a6100916c/keyspace-cf-ka-179-Data.db')) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-10937) OOM on multiple nodes on write load (v. 3.0.0), problem also present on DSE-4.8.3, but there it survives more time
[ https://issues.apache.org/jira/browse/CASSANDRA-10937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15105495#comment-15105495 ] Jack Krupansky edited comment on CASSANDRA-10937 at 1/18/16 5:16 PM: - I still don't see any reason to believe that there is a bug here and that the primary issue is that you are overloading the cluster. Sure, Cassandra should do a better job of shedding/failing excessive incoming requests, and there is an open Jira ticket to add just such a freature, but even with that new feature, the net effect will be the same - it will still be up to the application and operations to properly size the cluster and throttle application load before it gets to Cassandra. OOM is not typically an indication of a software bug. Sure, sometimes code has memory leaks, but with a highly dynamic system such as Cassandra, it typically means either a misconfigured JVM or just very heavy load. Sometimes OOM simply means that there is a lot of background processing going on (like compactions or hinted handoff) that is having trouble keeping up with incoming requests. Sometimes OOM occurs because you have too large a heap which defers GC but then GC takes too long and further incoming requests simply generate more pressure on the heap faster than that massive GC can deal with it. It is indeed tricky to make sure the JVM has enough heap but not too much. DSE typically runs with a larger heap by default. You can try increasing your heap to 10 or 12G. But if you make the heap too big, the big GC can bite you as described above. In that case, the heap needs to be reduced. Typically you don't need a heap smaller than 8 GB. If OOM occurs with a 8 GB heap it typically means the load on that node is simply too heavy. Be sure to review the recommendations in this blog post on reasonable recommendations: http://www.datastax.com/dev/blog/how-not-to-benchmark-cassandra A few questions that will help us better understand what you are really trying to do: 1. How much reading are you doing and when relative to writes? 2. Are you doing any updates or deletes? (Cause compaction, which can fall behind your write/update load.) 3. How much data is on the cluster (rows)? 4. How many tables? 5. What RF? RF=3 would be the recommendation, but if you have a heavy read load you may need RF=5, although heavy load usually means you just need a lot more nodes so that the fraction of incoming requests going to a particular node are dramatically reduced. RF>3 is only needed if there is high load for each particular row or partition. 6. Have you tested using cassandra-stress? That's the gold standard around here. 7. Are your clients using token-aware routing? (Otherwise a write must be bounced from the coordinating node to the node owning the token for the partition key.) 8. Are you using batches for your writes? If, so, do all the writes in one batch have the same partition key? (If not, adds more network hops.) 9. What expectations did you have as to how many writes/reads a given number of nodes should be able to handle? was (Author: jkrupan): I still don't see any reason to believe that there is a bug here and that the primary issue is that you are overloading the cluster. Sure, Cassandra should do a better job of shedding/failing excessive incoming requests, and there is an open Jira ticket to add just such a freature, but even with that new feature, the net effect will be the same - it will still be up to the application and operations to properly size the cluster and throttle application load before it gets to Cassandra. OOM is not typically an indication of a software bug. Sure, sometimes code has memory leaks, but with a highly dynamic system such as Cassandra, it typically means either a misconfigured JVM or just very heavy load. Sometimes OOM simply means that there is a lot of background processing going on (like compactions or hinted handoff) that is having trouble keeping up with incoming requests. Sometimes OOM occurs because you have too large a heap which defers GC but then GC takes too long and further incoming requests simply generate more pressure on the heap faster than that massive GC can deal with it. It is indeed tricky to make sure the JVM has enough heap but not too much. DSE typically runs with a larger heap by default. You can try increasing your heap to 10 or 12G. But if you make the heap too big, the big GC can bite you as described above. In that case, the heap needs to be reduced. Typically you don't need a heap smaller than 8 GB. If OOM occurs with a 8 GB heap it typically means the load on that node is simply too heavy. Be sure to review the recommendations in this blog post on reasonable recommendations: http://www.datastax.com/dev/blog/how-not-to-benchmark-cassandra A few questions that will help us better understand what you ar
[jira] [Commented] (CASSANDRA-10937) OOM on multiple nodes on write load (v. 3.0.0), problem also present on DSE-4.8.3, but there it survives more time
[
https://issues.apache.org/jira/browse/CASSANDRA-10937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15105572#comment-15105572
]
Peter Kovgan commented on CASSANDRA-10937:
--
I will go through all the recommendations and I will provide more detailed
picture.
Trust me , I look for the most sane reason to continue use cassandra and I will
reduce load
to find the acceptable, optimal one. I do my best!!! If load reduction helps -
we will use cassandra.
But short answer is that:
I have no problem with OOM itself, assuming OOM is just a failure indicator for
the case of too intensive load.
My problem is this indication is too delayed in time. (48, 89 hours!!! -
depends on heap size)
There are clear signs that % of "IO wait" grows gradually throughout all the
test.
(sar metrics tell, that %iowait is progressively growing - time that CPU spends
in IO grows day to day...gradually, slowly)
It looks like problems accumulate and grow with time.
I'm not claiming this is not workable system.
I'm claiming it looks like a memory leak, based on unrestricted publishers and
progressively growing IO demand.
The "exit" becomes narrower and narrower and "entrance" remains the same.
For me the best solution is to find a reason of the progressive(growing) IO and
prevent this issue.
How to prevent it (by restricting publishers, by optimizing IO, etc) I do not
know.
But I would be more happy to fail in first 1-5 hours , when most of cassandra
processes have repeated multiple times and system had a chance to estimate its
load and make conclusions.
When it fails after 4 days of test, when everybody shanked hands and greeted
each other "we like it", it is another matter.
You should not convince me, because I'm a positive opportunist, I will find a
way.
But some people will be frustrated. That's why I would take it seriously.
> OOM on multiple nodes on write load (v. 3.0.0), problem also present on
> DSE-4.8.3, but there it survives more time
> --
>
> Key: CASSANDRA-10937
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10937
> Project: Cassandra
> Issue Type: Bug
> Environment: Cassandra : 3.0.0
> Installed as open archive, no connection to any OS specific installer.
> Java:
> Java(TM) SE Runtime Environment (build 1.8.0_65-b17)
> OS :
> Linux version 2.6.32-431.el6.x86_64
> (mockbu...@x86-023.build.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red
> Hat 4.4.7-4) (GCC) ) #1 SMP Sun Nov 10 22:19:54 EST 2013
> We have:
> 8 guests ( Linux OS as above) on 2 (VMWare managed) physical hosts. Each
> physical host keeps 4 guests.
> Physical host parameters(shared by all 4 guests):
> Model: HP ProLiant DL380 Gen9
> Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
> 46 logical processors.
> Hyperthreading - enabled
> Each guest assigned to have:
> 1 disk 300 Gb for seq. log (NOT SSD)
> 1 disk 4T for data (NOT SSD)
> 11 CPU cores
> Disks are local, not shared.
> Memory on each host - 24 Gb total.
> 8 (or 6, tested both) Gb - cassandra heap
> (lshw and cpuinfo attached in file test2.rar)
>Reporter: Peter Kovgan
>Priority: Critical
> Attachments: gc-stat.txt, more-logs.rar, some-heap-stats.rar,
> test2.rar, test3.rar, test4.rar, test5.rar, test_2.1.rar,
> test_2.1_logs_older.rar, test_2.1_restart_attempt_log.rar
>
>
> 8 cassandra nodes.
> Load test started with 4 clients(different and not equal machines), each
> running 1000 threads.
> Each thread assigned in round-robin way to run one of 4 different inserts.
> Consistency->ONE.
> I attach the full CQL schema of tables and the query of insert.
> Replication factor - 2:
> create keyspace OBLREPOSITORY_NY with replication =
> {'class':'NetworkTopologyStrategy','NY':2};
> Initiall throughput is:
> 215.000 inserts /sec
> or
> 54Mb/sec, considering single insert size a bit larger than 256byte.
> Data:
> all fields(5-6) are short strings, except one is BLOB of 256 bytes.
> After about a 2-3 hours of work, I was forced to increase timeout from 2000
> to 5000ms, for some requests failed for short timeout.
> Later on(after aprox. 12 hous of work) OOM happens on multiple nodes.
> (all failed nodes logs attached)
> I attach also java load client and instructions how set-up and use
> it.(test2.rar)
> Update:
> Later on test repeated with lesser load (10 mes/sec) with more relaxed
> CPU (idle 25%), with only 2 test clients, but anyway test failed.
> Update:
> DSE-4.8.3 also failed on OOM (3 nodes from 8), but here it survived 48 hours,
> not 10-12.
> Attachments:
> test2.rar -contains most of material
> more-logs.rar - contains additional nodes logs
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-10937) OOM on multiple nodes on write load (v. 3.0.0), problem also present on DSE-4.8.3, but there it survives more time
[ https://issues.apache.org/jira/browse/CASSANDRA-10937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15105572#comment-15105572 ] Peter Kovgan edited comment on CASSANDRA-10937 at 1/18/16 5:56 PM: --- I will go through all the recommendations and I will provide more detailed picture. Trust me , I look for the most sane reason to continue use cassandra and I will reduce load to find the acceptable, optimal one. I do my best!!! If load reduction helps - we will use cassandra. But short answer is that: I have no problem with OOM itself, assuming OOM is just a failure indicator for the case of too intensive load. My problem is this indication is too delayed in time. (48, 89 hours!!! - depends on heap size) There are clear signs that % of "IO wait" grows gradually throughout all the test. (sar metrics tell, that %iowait is progressively growing - time that CPU spends in IO grows day to day...gradually, slowly) It looks like problems accumulate and grow with time. I'm not claiming this is not workable system. I'm claiming it looks like a memory leak, based on unrestricted publishers and progressively growing IO demand. The "exit" becomes narrower and narrower and "entrance" remains the same. For me the best solution is to find a reason of the progressive(growing) IO and prevent this issue. How to prevent it (by restricting publishers, by optimizing IO, etc) I do not know. But I would be more happy to fail in first 1-5 hours , when most of cassandra processes have repeated multiple times and system had a chance to estimate its load and make conclusions. (even friendly warning "you need a rescue node, otherwise you will become unstable" is a solution) When it fails after 4 days of test, when everybody shanked hands and greeted each other "we like it", it is another matter. You should not convince me, because I'm a positive opportunist, I will find a way. But some people will be frustrated. That's why I would take it seriously. was (Author: tierhetze): I will go through all the recommendations and I will provide more detailed picture. Trust me , I look for the most sane reason to continue use cassandra and I will reduce load to find the acceptable, optimal one. I do my best!!! If load reduction helps - we will use cassandra. But short answer is that: I have no problem with OOM itself, assuming OOM is just a failure indicator for the case of too intensive load. My problem is this indication is too delayed in time. (48, 89 hours!!! - depends on heap size) There are clear signs that % of "IO wait" grows gradually throughout all the test. (sar metrics tell, that %iowait is progressively growing - time that CPU spends in IO grows day to day...gradually, slowly) It looks like problems accumulate and grow with time. I'm not claiming this is not workable system. I'm claiming it looks like a memory leak, based on unrestricted publishers and progressively growing IO demand. The "exit" becomes narrower and narrower and "entrance" remains the same. For me the best solution is to find a reason of the progressive(growing) IO and prevent this issue. How to prevent it (by restricting publishers, by optimizing IO, etc) I do not know. But I would be more happy to fail in first 1-5 hours , when most of cassandra processes have repeated multiple times and system had a chance to estimate its load and make conclusions. When it fails after 4 days of test, when everybody shanked hands and greeted each other "we like it", it is another matter. You should not convince me, because I'm a positive opportunist, I will find a way. But some people will be frustrated. That's why I would take it seriously. > OOM on multiple nodes on write load (v. 3.0.0), problem also present on > DSE-4.8.3, but there it survives more time > -- > > Key: CASSANDRA-10937 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10937 > Project: Cassandra > Issue Type: Bug > Environment: Cassandra : 3.0.0 > Installed as open archive, no connection to any OS specific installer. > Java: > Java(TM) SE Runtime Environment (build 1.8.0_65-b17) > OS : > Linux version 2.6.32-431.el6.x86_64 > (mockbu...@x86-023.build.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red > Hat 4.4.7-4) (GCC) ) #1 SMP Sun Nov 10 22:19:54 EST 2013 > We have: > 8 guests ( Linux OS as above) on 2 (VMWare managed) physical hosts. Each > physical host keeps 4 guests. > Physical host parameters(shared by all 4 guests): > Model: HP ProLiant DL380 Gen9 > Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz > 46 logical processors. > Hyperthreading - enabled > Each guest assigned to have: > 1 disk 300 Gb for seq. log (NOT SSD) > 1 disk 4T for data (NOT SSD) > 11 CPU cores >
[jira] [Commented] (CASSANDRA-11021) Inserting static column fails with secondary index on clustering key
[
https://issues.apache.org/jira/browse/CASSANDRA-11021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15105620#comment-15105620
]
Sam Tunnicliffe commented on CASSANDRA-11021:
-
Thanks for the patch, I've made a couple of small tweaks & pushed it to my repo
so that CI will run against it. The changes I made were:
* Changed the logic slightly in the {{insertRow}} method in
{{CassandraIndexer}} to bail early whenever the row is static, but the indexed
column is not. Your fix would works fine for this particular issue, but this
saves some unneccesary processing when a static value is updated.
* Also added something similar in {{updateRow}}, although this doesn't trigger
the same error, some additional work is performed that we can skip.
* Relocated the unit test to
{{org.apache.cassandra.cql3.validation.entites.SecondaryIndexTest}} (the
fixture it was originally in relates to the internal index of SSTable files,
rather than to secondary indexes).
Once CI has verified everything, I'll commit.
||branch||testall||dtest||
|[11021-3.0|https://github.com/beobal/cassandra/tree/11021-3.0]|[testall|http://cassci.datastax.com/view/Dev/view/beobal/job/beobal-11021-3.0-testall]|[dtest|http://cassci.datastax.com/view/Dev/view/beobal/job/beobal-11021-3.0-dtest]|
|[11021-3.3|https://github.com/beobal/cassandra/tree/11021-3.3]|[testall|http://cassci.datastax.com/view/Dev/view/beobal/job/beobal-11021-3.3-testall]|[dtest|http://cassci.datastax.com/view/Dev/view/beobal/job/beobal-11021-3.3-dtest]|
|[11021-trunk|https://github.com/beobal/cassandra/tree/11021-trunk]|[testall|http://cassci.datastax.com/view/Dev/view/beobal/job/beobal-11021-trunk-testall]|[dtest|http://cassci.datastax.com/view/Dev/view/beobal/job/beobal-11021-trunk-dtest]|
> Inserting static column fails with secondary index on clustering key
>
>
> Key: CASSANDRA-11021
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11021
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
>Reporter: Stefan Podkowinski
>Assignee: Stefan Podkowinski
>
> Creating a secondary index on a clustering key fails with an exception in
> case a static column is involved.
> {code}
> CREATE TABLE test (k int, t int, s text static, v text, PRIMARY KEY (k, t));
> CREATE INDEX ix ON test (t);
> INSERT INTO test(k, t, s, v) VALUES (0, 1, 'abc', 'def');
> {code}
> {code}
> ERROR [SharedPool-Worker-2] 2016-01-15 11:42:27,484 StorageProxy.java:1336 -
> Failed to apply mutation locally : {}
> java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.cassandra.db.AbstractClusteringPrefix.get(AbstractClusteringPrefix.java:59)
> ~[main/:na]
> at
> org.apache.cassandra.index.internal.composites.ClusteringColumnIndex.getIndexedValue(ClusteringColumnIndex.java:58)
> ~[main/:na]
> at
> org.apache.cassandra.index.internal.CassandraIndex.getIndexedValue(CassandraIndex.java:598)
> ~[main/:na]
> at
> org.apache.cassandra.index.internal.CassandraIndex.insert(CassandraIndex.java:490)
> ~[main/:na]
> at
> org.apache.cassandra.index.internal.CassandraIndex.access$100(CassandraIndex.java:53)
> ~[main/:na]
> at
> org.apache.cassandra.index.internal.CassandraIndex$1.indexPrimaryKey(CassandraIndex.java:437)
> ~[main/:na]
> at
> org.apache.cassandra.index.internal.CassandraIndex$1.insertRow(CassandraIndex.java:347)
> ~[main/:na]
> at
> org.apache.cassandra.index.SecondaryIndexManager$WriteTimeTransaction.onInserted(SecondaryIndexManager.java:795)
> ~[main/:na]
> at
> org.apache.cassandra.db.partitions.AtomicBTreePartition$RowUpdater.apply(AtomicBTreePartition.java:275)
> ~[main/:na]
> at
> org.apache.cassandra.db.partitions.AtomicBTreePartition.addAllWithSizeDelta(AtomicBTreePartition.java:154)
> ~[main/:na]
> at org.apache.cassandra.db.Memtable.put(Memtable.java:240) ~[main/:na]
> at
> org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1145)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:494)
> ~[main/:na]
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:384)
> ~[main/:na]
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:205)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageProxy$$Lambda$166/492512700.run(Unknown
> Source) ~[na:na]
> at
> org.apache.cassandra.service.StorageProxy$8.runMayThrow(StorageProxy.java:1330)
> ~[main/:na]
> at
> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2480)
> [main/:n
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11028) Streaming errors caused by corrupt tables need more logging
[ https://issues.apache.org/jira/browse/CASSANDRA-11028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15105685#comment-15105685 ] Paulo Motta commented on CASSANDRA-11028: - Thanks for the report [~autocracy]. While working on CASSANDRA-10961 I added more detailed debug logging to stream writer and reader, printing source sstable, keyspace, table and faulty partition key in case of error/corruption on receiver side, so this should be improved in upcoming releases. Some additional logging was also added on CASSANDRA-9294. > Streaming errors caused by corrupt tables need more logging > --- > > Key: CASSANDRA-11028 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11028 > Project: Cassandra > Issue Type: Bug >Reporter: Jeff Ferland > > Example output: ERROR [STREAM-IN-/10.0.10.218] 2016-01-17 16:01:38,431 > StreamSession.java:505 - [Stream #e6ca4590-bc66-11e5-84be-571ffcecc993] > Streaming error occurred > java.lang.IllegalArgumentException: Unknown type 0 > In some cases logging shows a message more like: > ERROR [STREAM-IN-/10.0.10.12] 2016-01-05 14:44:38,690 StreamSession.java:505 > - [Stream #472d28e0-b347-11e5-8b40-bb4d80df86f4] Streaming error occurred > java.io.IOException: Too many retries for Header (cfId: > 6b262d58-8730-36ca-8e3e-f0a40beaf92f, #0, version: ka, estimated keys: 58880, > transfer size: 2159040, compressed?: true, repairedAt: 0) > In the majority of cases, however, no information identifying the column > family is shown, and never identifying the source file that was being > streamed. > Errors do no stop the streaming process, but do mark the streaming as failed > at the end. This usually results in a log message pattern like: > INFO [StreamReceiveTask:252] 2016-01-18 04:45:01,190 > StreamResultFuture.java:180 - [Stream #e6ca4590-bc66-11e5-84be-571ffcecc993] > Session with /10.0.10.219 is complete > WARN [StreamReceiveTask:252] 2016-01-18 04:45:01,215 > StreamResultFuture.java:207 - [Stream #e6ca4590-bc66-11e5-84be-571ffcecc993] > Stream failed > ERROR [main] 2016-01-18 04:45:01,217 CassandraDaemon.java:579 - Exception > encountered during startup > ... which is highly confusing given the error occurred hours before. > Request: more detail in logging messages for stream failure indicating what > column family was being used, and if possible a clarification between network > issues and corrupt file issues. > Actual cause of errors / solution is running nodetool scrub on the offending > node. It's rather expensive scrubbing the whole space blindly versus > targeting issue tables. In our particular case, out of order keys were caused > by a bug in a previous version of Cassandra. > WARN [CompactionExecutor:19552] 2016-01-18 16:02:10,155 > OutputHandler.java:52 - 378490 out of order rows found while scrubbing > SSTableReader(path='/mnt/cassandra/data/keyspace/cf-888a52f96d1d389790ee586a6100916c/keyspace-cf-ka-133-Data.db'); > Those have been written (in order) to a new sstable > (SSTableReader(path='/mnt/cassandra/data/keyspace/cf-888a52f96d1d389790ee586a6100916c/keyspace-cf-ka-179-Data.db')) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (CASSANDRA-11028) Streaming errors caused by corrupt tables need more logging
[ https://issues.apache.org/jira/browse/CASSANDRA-11028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paulo Motta resolved CASSANDRA-11028. - Resolution: Duplicate > Streaming errors caused by corrupt tables need more logging > --- > > Key: CASSANDRA-11028 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11028 > Project: Cassandra > Issue Type: Bug >Reporter: Jeff Ferland > > Example output: ERROR [STREAM-IN-/10.0.10.218] 2016-01-17 16:01:38,431 > StreamSession.java:505 - [Stream #e6ca4590-bc66-11e5-84be-571ffcecc993] > Streaming error occurred > java.lang.IllegalArgumentException: Unknown type 0 > In some cases logging shows a message more like: > ERROR [STREAM-IN-/10.0.10.12] 2016-01-05 14:44:38,690 StreamSession.java:505 > - [Stream #472d28e0-b347-11e5-8b40-bb4d80df86f4] Streaming error occurred > java.io.IOException: Too many retries for Header (cfId: > 6b262d58-8730-36ca-8e3e-f0a40beaf92f, #0, version: ka, estimated keys: 58880, > transfer size: 2159040, compressed?: true, repairedAt: 0) > In the majority of cases, however, no information identifying the column > family is shown, and never identifying the source file that was being > streamed. > Errors do no stop the streaming process, but do mark the streaming as failed > at the end. This usually results in a log message pattern like: > INFO [StreamReceiveTask:252] 2016-01-18 04:45:01,190 > StreamResultFuture.java:180 - [Stream #e6ca4590-bc66-11e5-84be-571ffcecc993] > Session with /10.0.10.219 is complete > WARN [StreamReceiveTask:252] 2016-01-18 04:45:01,215 > StreamResultFuture.java:207 - [Stream #e6ca4590-bc66-11e5-84be-571ffcecc993] > Stream failed > ERROR [main] 2016-01-18 04:45:01,217 CassandraDaemon.java:579 - Exception > encountered during startup > ... which is highly confusing given the error occurred hours before. > Request: more detail in logging messages for stream failure indicating what > column family was being used, and if possible a clarification between network > issues and corrupt file issues. > Actual cause of errors / solution is running nodetool scrub on the offending > node. It's rather expensive scrubbing the whole space blindly versus > targeting issue tables. In our particular case, out of order keys were caused > by a bug in a previous version of Cassandra. > WARN [CompactionExecutor:19552] 2016-01-18 16:02:10,155 > OutputHandler.java:52 - 378490 out of order rows found while scrubbing > SSTableReader(path='/mnt/cassandra/data/keyspace/cf-888a52f96d1d389790ee586a6100916c/keyspace-cf-ka-133-Data.db'); > Those have been written (in order) to a new sstable > (SSTableReader(path='/mnt/cassandra/data/keyspace/cf-888a52f96d1d389790ee586a6100916c/keyspace-cf-ka-179-Data.db')) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-11029) cassandra failed to start up due to Too many open files
Xiaoping Li created CASSANDRA-11029: --- Summary: cassandra failed to start up due to Too many open files Key: CASSANDRA-11029 URL: https://issues.apache.org/jira/browse/CASSANDRA-11029 Project: Cassandra Issue Type: Bug Components: Lifecycle Environment: Ubuntu 14.04.2 LTS java version "1.8.0_60" cassandra 2.2.1 apache version Reporter: Xiaoping Li Priority: Critical Fix For: 2.2.x Attachments: cassandra.log.rtf I have a cassandra node running in AWS for test purpose. When I was going to test it today, just found it was down. And I have checked log file, the error is : ERROR 19:40:36 JVM state determined to be unstable. Exiting forcefully due to: java.io.FileNotFoundException: /var/lib/cassandra/data/talkatonegroup/invite_external_members-4b19b590664911e5906a018f9ca3e4f8/la-1-big-CompressionInfo.db (Too many open files) And even when I try to restart, it also shows the same error message. Any ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11029) cassandra failed to start up due to Too many open files
[ https://issues.apache.org/jira/browse/CASSANDRA-11029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15105767#comment-15105767 ] Paulo Motta commented on CASSANDRA-11029: - You may need to increase your O/S default resource limits according to the [recommended settings|https://docs.datastax.com/en/cassandra/2.1/cassandra/install/installRecommendSettings.html]. See [this document|https://docs.datastax.com/en/cassandra/2.1/cassandra/troubleshooting/trblshootTooManyFiles_r.html] for more information. Please reopen issue if that's not the case. > cassandra failed to start up due to Too many open files > --- > > Key: CASSANDRA-11029 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11029 > Project: Cassandra > Issue Type: Bug > Components: Lifecycle > Environment: Ubuntu 14.04.2 LTS > java version "1.8.0_60" > cassandra 2.2.1 apache version >Reporter: Xiaoping Li >Priority: Critical > Attachments: cassandra.log.rtf > > > I have a cassandra node running in AWS for test purpose. > When I was going to test it today, just found it was down. And I have checked > log file, the error is : > ERROR 19:40:36 JVM state determined to be unstable. Exiting forcefully due > to: > java.io.FileNotFoundException: > /var/lib/cassandra/data/talkatonegroup/invite_external_members-4b19b590664911e5906a018f9ca3e4f8/la-1-big-CompressionInfo.db > (Too many open files) > And even when I try to restart, it also shows the same error message. > Any ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (CASSANDRA-11029) cassandra failed to start up due to Too many open files
[ https://issues.apache.org/jira/browse/CASSANDRA-11029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paulo Motta resolved CASSANDRA-11029. - Resolution: Not A Problem Fix Version/s: (was: 2.2.x) > cassandra failed to start up due to Too many open files > --- > > Key: CASSANDRA-11029 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11029 > Project: Cassandra > Issue Type: Bug > Components: Lifecycle > Environment: Ubuntu 14.04.2 LTS > java version "1.8.0_60" > cassandra 2.2.1 apache version >Reporter: Xiaoping Li >Priority: Critical > Attachments: cassandra.log.rtf > > > I have a cassandra node running in AWS for test purpose. > When I was going to test it today, just found it was down. And I have checked > log file, the error is : > ERROR 19:40:36 JVM state determined to be unstable. Exiting forcefully due > to: > java.io.FileNotFoundException: > /var/lib/cassandra/data/talkatonegroup/invite_external_members-4b19b590664911e5906a018f9ca3e4f8/la-1-big-CompressionInfo.db > (Too many open files) > And even when I try to restart, it also shows the same error message. > Any ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11029) cassandra failed to start up due to Too many open files
[ https://issues.apache.org/jira/browse/CASSANDRA-11029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15105836#comment-15105836 ] Xiaoping Li commented on CASSANDRA-11029: - [~pauloricardomg]Thanks. I see the problem is because of open file limit. The reason is because I run as root, not as cassandra(since I define higher open files for cassandra). Thanks for the input > cassandra failed to start up due to Too many open files > --- > > Key: CASSANDRA-11029 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11029 > Project: Cassandra > Issue Type: Bug > Components: Lifecycle > Environment: Ubuntu 14.04.2 LTS > java version "1.8.0_60" > cassandra 2.2.1 apache version >Reporter: Xiaoping Li >Priority: Critical > Attachments: cassandra.log.rtf > > > I have a cassandra node running in AWS for test purpose. > When I was going to test it today, just found it was down. And I have checked > log file, the error is : > ERROR 19:40:36 JVM state determined to be unstable. Exiting forcefully due > to: > java.io.FileNotFoundException: > /var/lib/cassandra/data/talkatonegroup/invite_external_members-4b19b590664911e5906a018f9ca3e4f8/la-1-big-CompressionInfo.db > (Too many open files) > And even when I try to restart, it also shows the same error message. > Any ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11029) cassandra failed to start up due to Too many open files
[ https://issues.apache.org/jira/browse/CASSANDRA-11029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15105976#comment-15105976 ] Paulo Motta commented on CASSANDRA-11029: - Next time you may try asking for help on the [mailing lists|http://www.planetcassandra.org/apache-cassandra-mailing-lists/] or #cassandra on irc.freenode.net before opening an issue. Thanks! > cassandra failed to start up due to Too many open files > --- > > Key: CASSANDRA-11029 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11029 > Project: Cassandra > Issue Type: Bug > Components: Lifecycle > Environment: Ubuntu 14.04.2 LTS > java version "1.8.0_60" > cassandra 2.2.1 apache version >Reporter: Xiaoping Li >Priority: Critical > Attachments: cassandra.log.rtf > > > I have a cassandra node running in AWS for test purpose. > When I was going to test it today, just found it was down. And I have checked > log file, the error is : > ERROR 19:40:36 JVM state determined to be unstable. Exiting forcefully due > to: > java.io.FileNotFoundException: > /var/lib/cassandra/data/talkatonegroup/invite_external_members-4b19b590664911e5906a018f9ca3e4f8/la-1-big-CompressionInfo.db > (Too many open files) > And even when I try to restart, it also shows the same error message. > Any ideas? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-11030) non-ascii characters incorrectly displayed/inserted on cqlsh on Windows
Paulo Motta created CASSANDRA-11030:
---
Summary: non-ascii characters incorrectly displayed/inserted on
cqlsh on Windows
Key: CASSANDRA-11030
URL: https://issues.apache.org/jira/browse/CASSANDRA-11030
Project: Cassandra
Issue Type: Bug
Reporter: Paulo Motta
Assignee: Paulo Motta
Priority: Minor
{noformat}
C:\Users\Paulo\Repositories\cassandra [2.2-10948 +6 ~1 -0 !]> .\bin\cqlsh.bat
--encoding utf-8
Connected to test at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 2.2.4-SNAPSHOT | CQL spec 3.3.1 | Native protocol v4]
Use HELP for help.
cqlsh> INSERT INTO bla.test (bla ) VALUES ('não') ;
cqlsh> select * from bla.test;
bla
-
n?o
(1 rows)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[cassandra] Git Push Summary
Repository: cassandra Updated Tags: refs/tags/cassandra-3.2.1 [created] 9a44974b0
[cassandra] Git Push Summary
Repository: cassandra Updated Tags: refs/tags/3.2.1-tentative [deleted] 2ac95bd6c
svn commit: r11950 - in /release/cassandra: 3.1.1/ 3.2.1/ debian/dists/32x/ debian/dists/32x/main/binary-amd64/ debian/dists/32x/main/binary-i386/ debian/dists/32x/main/source/ debian/pool/main/c/cass
Author: jake Date: Tue Jan 19 01:55:39 2016 New Revision: 11950 Log: 3.2.1 and cleanup Added: release/cassandra/3.2.1/ release/cassandra/3.2.1/apache-cassandra-3.2.1-bin.tar.gz (with props) release/cassandra/3.2.1/apache-cassandra-3.2.1-bin.tar.gz.asc release/cassandra/3.2.1/apache-cassandra-3.2.1-bin.tar.gz.asc.md5 release/cassandra/3.2.1/apache-cassandra-3.2.1-bin.tar.gz.asc.sha1 release/cassandra/3.2.1/apache-cassandra-3.2.1-bin.tar.gz.md5 release/cassandra/3.2.1/apache-cassandra-3.2.1-bin.tar.gz.sha1 release/cassandra/3.2.1/apache-cassandra-3.2.1-src.tar.gz (with props) release/cassandra/3.2.1/apache-cassandra-3.2.1-src.tar.gz.asc release/cassandra/3.2.1/apache-cassandra-3.2.1-src.tar.gz.asc.md5 release/cassandra/3.2.1/apache-cassandra-3.2.1-src.tar.gz.asc.sha1 release/cassandra/3.2.1/apache-cassandra-3.2.1-src.tar.gz.md5 release/cassandra/3.2.1/apache-cassandra-3.2.1-src.tar.gz.sha1 release/cassandra/debian/pool/main/c/cassandra/cassandra-tools_3.2.1_all.deb (with props) release/cassandra/debian/pool/main/c/cassandra/cassandra_3.2.1.diff.gz (with props) release/cassandra/debian/pool/main/c/cassandra/cassandra_3.2.1.dsc release/cassandra/debian/pool/main/c/cassandra/cassandra_3.2.1.orig.tar.gz (with props) release/cassandra/debian/pool/main/c/cassandra/cassandra_3.2.1.orig.tar.gz.asc release/cassandra/debian/pool/main/c/cassandra/cassandra_3.2.1_all.deb (with props) Removed: release/cassandra/3.1.1/ release/cassandra/debian/pool/main/c/cassandra/cassandra_3.0.0.diff.gz release/cassandra/debian/pool/main/c/cassandra/cassandra_3.0.0.dsc release/cassandra/debian/pool/main/c/cassandra/cassandra_3.0.0.orig.tar.gz release/cassandra/debian/pool/main/c/cassandra/cassandra_3.0.0.orig.tar.gz.asc release/cassandra/debian/pool/main/c/cassandra/cassandra_3.0.0_all.deb release/cassandra/debian/pool/main/c/cassandra/cassandra_3.0.0~beta2.diff.gz release/cassandra/debian/pool/main/c/cassandra/cassandra_3.0.0~beta2.dsc release/cassandra/debian/pool/main/c/cassandra/cassandra_3.0.0~beta2.orig.tar.gz release/cassandra/debian/pool/main/c/cassandra/cassandra_3.0.0~beta2.orig.tar.gz.asc release/cassandra/debian/pool/main/c/cassandra/cassandra_3.0.0~beta2_all.deb release/cassandra/debian/pool/main/c/cassandra/cassandra_3.0.0~rc1.diff.gz release/cassandra/debian/pool/main/c/cassandra/cassandra_3.0.0~rc1.dsc release/cassandra/debian/pool/main/c/cassandra/cassandra_3.0.0~rc1.orig.tar.gz release/cassandra/debian/pool/main/c/cassandra/cassandra_3.0.0~rc1.orig.tar.gz.asc release/cassandra/debian/pool/main/c/cassandra/cassandra_3.0.0~rc1_all.deb release/cassandra/debian/pool/main/c/cassandra/cassandra_3.0.0~rc2.diff.gz release/cassandra/debian/pool/main/c/cassandra/cassandra_3.0.0~rc2.dsc release/cassandra/debian/pool/main/c/cassandra/cassandra_3.0.0~rc2.orig.tar.gz release/cassandra/debian/pool/main/c/cassandra/cassandra_3.0.0~rc2.orig.tar.gz.asc release/cassandra/debian/pool/main/c/cassandra/cassandra_3.0.0~rc2_all.deb Modified: release/cassandra/debian/dists/32x/InRelease release/cassandra/debian/dists/32x/Release release/cassandra/debian/dists/32x/Release.gpg release/cassandra/debian/dists/32x/main/binary-amd64/Packages release/cassandra/debian/dists/32x/main/binary-amd64/Packages.gz release/cassandra/debian/dists/32x/main/binary-i386/Packages release/cassandra/debian/dists/32x/main/binary-i386/Packages.gz release/cassandra/debian/dists/32x/main/source/Sources.gz Added: release/cassandra/3.2.1/apache-cassandra-3.2.1-bin.tar.gz == Binary file - no diff available. Propchange: release/cassandra/3.2.1/apache-cassandra-3.2.1-bin.tar.gz -- svn:mime-type = application/octet-stream Added: release/cassandra/3.2.1/apache-cassandra-3.2.1-bin.tar.gz.asc == --- release/cassandra/3.2.1/apache-cassandra-3.2.1-bin.tar.gz.asc (added) +++ release/cassandra/3.2.1/apache-cassandra-3.2.1-bin.tar.gz.asc Tue Jan 19 01:55:39 2016 @@ -0,0 +1,17 @@ +-BEGIN PGP SIGNATURE- +Version: GnuPG v1 + +iQIcBAABAgAGBQJWmFp1AAoJEHSdbuwDU7Esb5kP/1AXc7WlsQvrWRuxz3L3Wr3U +PXIlO9pK6j/Q9Cb2f4tkOY/UuHfw3UVmsn5opOoyghqbZeYxGaPDJrIe8BY1V3Wi +fIwJMtiOX9YyrA8w3pSLOvCU/5HhPie2ukLpadkB1I4QAFWeEBwca0ceeAsncvmG +XY4kgSIB6mB7BKBFDIMp8wyNY8ro2i2eL5dvPKXudmmbocNmGE3jZlQhudS9Bc5v +BNY67fK52nvyd35ZmW2L8htJE7Hewq5CFarTJ2BuCDYj6EDr17hqD/ZUWoB/6dR3 +VzZbNSA+eBa0b6QLCoiYIwzi7oku/Bop9US9hJnrtkeEzHYQD4mXuEyBs7RRmdpb +qs8GziyCmT7WND1G8PPpnViZ1mUAgt+a+J6S5Vs+dXynaqYbJBxMFyn85M5ESSmK +l4hs1qYBNJKIqQ3OnKZE1dhN18R0bu/eKlg3xzg0xOjb6/oBigefH6E3epDy6yuQ +DcB+8C4ATWUX4mgW55jxFUxdKUaDa0Li0jwGXShQ17PuZZB40HDKmjbvcjlefFZ7 +W6sP0ncXntaSjTmOZXAF9+OS
[jira] [Commented] (CASSANDRA-8180) Optimize disk seek using min/max column name meta data when the LIMIT clause is used
[
https://issues.apache.org/jira/browse/CASSANDRA-8180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15106233#comment-15106233
]
Stefania commented on CASSANDRA-8180:
-
Rebased and squashed [~blambov]:
||trunk||
|[patch|https://github.com/stef1927/cassandra/commits/8180]|
|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-8180-testall/]|
|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-8180-dtest/]|
There is one issue, {{RangeTombstone.Bound.TOP}} and {{BOTTOM}} are not
currently updating the min and max clustering values of {{StatsMetadata}} since
their clustering values are empty arrays, see
{{MetadataCollector.updateClusteringValues()}} and {{BTW.applyToMarker()}}. As
a result, [some range tombstone
tests|http://cassci.datastax.com/job/stef1927-8180-testall/1/testReport/junit/junit/]
are failing. It's not clear to me how to capture this information in the
metadata without changing its format given that the smallest / biggest values
would depend on the individual clustering types, cc [~slebresne] as well.
> Optimize disk seek using min/max column name meta data when the LIMIT clause
> is used
>
>
> Key: CASSANDRA-8180
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8180
> Project: Cassandra
> Issue Type: Improvement
> Components: Local Write-Read Paths
> Environment: Cassandra 2.0.10
>Reporter: DOAN DuyHai
>Assignee: Stefania
>Priority: Minor
> Fix For: 3.x
>
> Attachments: 8180_001.yaml, 8180_002.yaml
>
>
> I was working on an example of sensor data table (timeseries) and face a use
> case where C* does not optimize read on disk.
> {code}
> cqlsh:test> CREATE TABLE test(id int, col int, val text, PRIMARY KEY(id,col))
> WITH CLUSTERING ORDER BY (col DESC);
> cqlsh:test> INSERT INTO test(id, col , val ) VALUES ( 1, 10, '10');
> ...
> >nodetool flush test test
> ...
> cqlsh:test> INSERT INTO test(id, col , val ) VALUES ( 1, 20, '20');
> ...
> >nodetool flush test test
> ...
> cqlsh:test> INSERT INTO test(id, col , val ) VALUES ( 1, 30, '30');
> ...
> >nodetool flush test test
> {code}
> After that, I activate request tracing:
> {code}
> cqlsh:test> SELECT * FROM test WHERE id=1 LIMIT 1;
> activity |
> timestamp| source| source_elapsed
> ---+--+---+
> execute_cql3_query |
> 23:48:46,498 | 127.0.0.1 | 0
> Parsing SELECT * FROM test WHERE id=1 LIMIT 1; |
> 23:48:46,498 | 127.0.0.1 | 74
>Preparing statement |
> 23:48:46,499 | 127.0.0.1 |253
> Executing single-partition query on test |
> 23:48:46,499 | 127.0.0.1 |930
> Acquiring sstable references |
> 23:48:46,499 | 127.0.0.1 |943
>Merging memtable tombstones |
> 23:48:46,499 | 127.0.0.1 | 1032
>Key cache hit for sstable 3 |
> 23:48:46,500 | 127.0.0.1 | 1160
>Seeking to partition beginning in data file |
> 23:48:46,500 | 127.0.0.1 | 1173
>Key cache hit for sstable 2 |
> 23:48:46,500 | 127.0.0.1 | 1889
>Seeking to partition beginning in data file |
> 23:48:46,500 | 127.0.0.1 | 1901
>Key cache hit for sstable 1 |
> 23:48:46,501 | 127.0.0.1 | 2373
>Seeking to partition beginning in data file |
> 23:48:46,501 | 127.0.0.1 | 2384
> Skipped 0/3 non-slice-intersecting sstables, included 0 due to tombstones |
> 23:48:46,501 | 127.0.0.1 | 2768
> Merging data from memtables and 3 sstables |
> 23:48:46,501 | 127.0.0.1 | 2784
> Read 2 live and 0 tombstoned cells |
> 23:48:46,501 | 127.0.0.1 | 2976
> Request complete |
> 23:48:46,501 | 127.0.0.1 | 3551
> {code}
> We can clearly see that C* hits 3 SSTables on disk instead of just one,
> although it has the min/max column meta data to decide which SSTable contains
> the most recent data.
> Funny enough, if we add a clause on the clustering column to the select, this
> time C* o
[jira] [Created] (CASSANDRA-11031) Support “ALLOW FILTERING" for Partition Key
ZhaoYang created CASSANDRA-11031: Summary: Support “ALLOW FILTERING" for Partition Key Key: CASSANDRA-11031 URL: https://issues.apache.org/jira/browse/CASSANDRA-11031 Project: Cassandra Issue Type: New Feature Components: CQL Reporter: ZhaoYang Assignee: ZhaoYang Fix For: 2.1.x, 3.x Currently, Allow Filtering only works for secondary Index column or clustering columns. And it's slow, because Cassandra will read all data from SSTABLE from hard-disk to memory to filter. But we can support allow filtering on Partition Key, as far as I know, Partition Key is in memory, so we can easily filter them, and then read required data from SSTable. This will similar to "Select * from table" which scan through entire cluster. CREATE TABLE my_table ( pk1 text, pk2 text, c1 text, c2 text, v1 text, v2 text, PRIMARY KEY ((pk1,pk2),c1,c2) ) ; Select * from my_table where pk1 = "partition key part 1" allow filtering; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9472) Reintroduce off heap memtables
[
https://issues.apache.org/jira/browse/CASSANDRA-9472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15106293#comment-15106293
]
Stefania commented on CASSANDRA-9472:
-
Hi [~benedict], thank you for your review.
bq. Storing an offheap variable in BTreeRow looks error-prone to maintain, and
forwarding all calls in AbstractBTreeRow to a virtual onHeapCopy is
unnecessarily costly for objects that never are offheap. I think it makes more
sense for the AtomicBTreePartition to check f it stores on/offheap data when
answering a query, and to return a suitable iterator that knows to copy on-heap
when necessary - much as it used to, but moved one step outwards so that it can
capture DecoratedKey and staticRow.
I agree that a virtual method is unnecessarily costly. I think I went down the
route of storing the information in {{AbstractBTreeRow}} but I stopped when I
encountered the static methods. If you can pull it off (or any other more
efficient way to store this information) I am definitely OK with it.
bq. As far as segmentation faults on toString when peer == 0, this is only a
problem for the debugger, because we don't call toString() ourselves. I don't
mind adding guards, but they should really throw exceptions since any access to
these objects when they've been freed is unequivocally a bug.
This should be fine, I've verified that we can still run {{NativeCellTest}}
with the debugger and with no adverse effect.
> Reintroduce off heap memtables
> --
>
> Key: CASSANDRA-9472
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9472
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Benedict
>Assignee: Stefania
> Fix For: 3.x
>
>
> CASSANDRA-8099 removes off heap memtables. We should reintroduce them ASAP.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9472) Reintroduce off heap memtables
[ https://issues.apache.org/jira/browse/CASSANDRA-9472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefania updated CASSANDRA-9472: Assignee: Benedict (was: Stefania) > Reintroduce off heap memtables > -- > > Key: CASSANDRA-9472 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9472 > Project: Cassandra > Issue Type: Improvement >Reporter: Benedict >Assignee: Benedict > Fix For: 3.x > > > CASSANDRA-8099 removes off heap memtables. We should reintroduce them ASAP. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9472) Reintroduce off heap memtables
[ https://issues.apache.org/jira/browse/CASSANDRA-9472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefania updated CASSANDRA-9472: Reviewer: Stefania (was: Benedict) > Reintroduce off heap memtables > -- > > Key: CASSANDRA-9472 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9472 > Project: Cassandra > Issue Type: Improvement >Reporter: Benedict >Assignee: Benedict > Fix For: 3.x > > > CASSANDRA-8099 removes off heap memtables. We should reintroduce them ASAP. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9472) Reintroduce off heap memtables
[ https://issues.apache.org/jira/browse/CASSANDRA-9472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefania updated CASSANDRA-9472: Assignee: Benedict (was: Stefania) > Reintroduce off heap memtables > -- > > Key: CASSANDRA-9472 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9472 > Project: Cassandra > Issue Type: Improvement >Reporter: Benedict >Assignee: Benedict > Fix For: 3.x > > > CASSANDRA-8099 removes off heap memtables. We should reintroduce them ASAP. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (CASSANDRA-9472) Reintroduce off heap memtables
[ https://issues.apache.org/jira/browse/CASSANDRA-9472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefania reassigned CASSANDRA-9472: --- Assignee: Stefania (was: Benedict) > Reintroduce off heap memtables > -- > > Key: CASSANDRA-9472 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9472 > Project: Cassandra > Issue Type: Improvement >Reporter: Benedict >Assignee: Stefania > Fix For: 3.x > > > CASSANDRA-8099 removes off heap memtables. We should reintroduce them ASAP. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-9472) Reintroduce off heap memtables
[
https://issues.apache.org/jira/browse/CASSANDRA-9472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15106293#comment-15106293
]
Stefania edited comment on CASSANDRA-9472 at 1/19/16 6:09 AM:
--
Hi [~benedict], thank you for your review.
bq. Storing an offheap variable in BTreeRow looks error-prone to maintain, and
forwarding all calls in AbstractBTreeRow to a virtual onHeapCopy is
unnecessarily costly for objects that never are offheap. I think it makes more
sense for the AtomicBTreePartition to check f it stores on/offheap data when
answering a query, and to return a suitable iterator that knows to copy on-heap
when necessary - much as it used to, but moved one step outwards so that it can
capture DecoratedKey and staticRow.
I agree that a virtual method is unnecessarily costly. I think I went down the
route of storing the information in {{AbstractBTreeRow}} but I stopped when I
encountered the static methods. If you can pull it off I am definitely OK with
it.
bq. As far as segmentation faults on toString when peer == 0, this is only a
problem for the debugger, because we don't call toString() ourselves. I don't
mind adding guards, but they should really throw exceptions since any access to
these objects when they've been freed is unequivocally a bug.
This should be fine, I've verified that we can still run {{NativeCellTest}}
with the debugger and with no adverse effect.
was (Author: stefania):
Hi [~benedict], thank you for your review.
bq. Storing an offheap variable in BTreeRow looks error-prone to maintain, and
forwarding all calls in AbstractBTreeRow to a virtual onHeapCopy is
unnecessarily costly for objects that never are offheap. I think it makes more
sense for the AtomicBTreePartition to check f it stores on/offheap data when
answering a query, and to return a suitable iterator that knows to copy on-heap
when necessary - much as it used to, but moved one step outwards so that it can
capture DecoratedKey and staticRow.
I agree that a virtual method is unnecessarily costly. I think I went down the
route of storing the information in {{AbstractBTreeRow}} but I stopped when I
encountered the static methods. If you can pull it off (or any other more
efficient way to store this information) I am definitely OK with it.
bq. As far as segmentation faults on toString when peer == 0, this is only a
problem for the debugger, because we don't call toString() ourselves. I don't
mind adding guards, but they should really throw exceptions since any access to
these objects when they've been freed is unequivocally a bug.
This should be fine, I've verified that we can still run {{NativeCellTest}}
with the debugger and with no adverse effect.
> Reintroduce off heap memtables
> --
>
> Key: CASSANDRA-9472
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9472
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Benedict
>Assignee: Benedict
> Fix For: 3.x
>
>
> CASSANDRA-8099 removes off heap memtables. We should reintroduce them ASAP.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8103) Secondary Indices for Static Columns
[ https://issues.apache.org/jira/browse/CASSANDRA-8103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Taiyuan Zhang updated CASSANDRA-8103: - Attachment: (was: 8103-v2.patch) > Secondary Indices for Static Columns > > > Key: CASSANDRA-8103 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8103 > Project: Cassandra > Issue Type: New Feature > Components: CQL >Reporter: Ron Cohen >Assignee: Taiyuan Zhang > Fix For: 3.x > > Attachments: 8103.patch, in_progress.patch, smoke-test.cql > > > We should add secondary index support for static columns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8103) Secondary Indices for Static Columns
[ https://issues.apache.org/jira/browse/CASSANDRA-8103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Taiyuan Zhang updated CASSANDRA-8103: - Attachment: 8103-v3.patch Thanks for the comment. Updated as suggested (8103-v3.patch). > Secondary Indices for Static Columns > > > Key: CASSANDRA-8103 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8103 > Project: Cassandra > Issue Type: New Feature > Components: CQL >Reporter: Ron Cohen >Assignee: Taiyuan Zhang > Fix For: 3.x > > Attachments: 8103-v3.patch, 8103.patch, in_progress.patch, > smoke-test.cql > > > We should add secondary index support for static columns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9949) maxPurgeableTimestamp needs to check memtables too
[
https://issues.apache.org/jira/browse/CASSANDRA-9949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15106398#comment-15106398
]
Stefania commented on CASSANDRA-9949:
-
Thank you for your review [~krummas], please see [this commit|
https://github.com/stef1927/cassandra/commit/96dacb1c2f555d34753901e48fecf1f8f0190393].
I've clarified the comment best I could and added a test to show what I mean.
If it still doesn't make much sense to you, perhaps we should just change the
test and pick the reconciled timestamp. I must admit I am not 100% sure of
what's best.
I've updated {{CompactionController.getFullyExpiredSSTables()}} and added a
unit test for it.
> maxPurgeableTimestamp needs to check memtables too
> --
>
> Key: CASSANDRA-9949
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9949
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths
>Reporter: Jonathan Ellis
>Assignee: Stefania
> Fix For: 2.2.x, 3.0.x, 3.x
>
>
> overlapIterator/maxPurgeableTimestamp don't include the memtables, so a
> very-out-of-order write could be ignored
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
