[jira] [Commented] (CASSANDRA-6936) Make all byte representations of types comparable by their unsigned byte representation only
[ https://issues.apache.org/jira/browse/CASSANDRA-6936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14524461#comment-14524461 ] Jonathan Ellis commented on CASSANDRA-6936: --- [~iamaleksey] I think you had some comments on this from NGCC discussion? Make all byte representations of types comparable by their unsigned byte representation only Key: CASSANDRA-6936 URL: https://issues.apache.org/jira/browse/CASSANDRA-6936 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Benedict Labels: performance Fix For: 3.x This could be a painful change, but is necessary for implementing a trie-based index, and settling for less would be suboptimal; it also should make comparisons cheaper all-round, and since comparison operations are pretty much the majority of C*'s business, this should be easily felt (see CASSANDRA-6553 and CASSANDRA-6934 for an example of some minor changes with major performance impacts). No copying/special casing/slicing should mean fewer opportunities to introduce performance regressions as well. Since I have slated for 3.0 a lot of non-backwards-compatible sstable changes, hopefully this shouldn't be too much more of a burden. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9240) Performance issue after a restart
[ https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-9240: -- Reviewer: T Jake Luciani [~tjake] to review Performance issue after a restart - Key: CASSANDRA-9240 URL: https://issues.apache.org/jira/browse/CASSANDRA-9240 Project: Cassandra Issue Type: Bug Reporter: Alan Boudreault Assignee: Benedict Priority: Minor Fix For: 3.x Attachments: Cassandra.snapshots.zip, cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log, cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log, cassandra_trunk-clientrequest-read.log, cassandra_trunk.log, cassandra_trunk_no_restart-clientrequest-read.log, cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, runs.log, trace_query.cql I have noticed a performance issue while I was working on compaction perf tests for CASSANDRA-7409. The performance for my use case is very bad after a restart. It is mostly a read performance issue but not strictly. I have attached my use case (see run_issue.sh and issue.yaml) and all test logs for 2.1.4, 2.1.5 and trunk: * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k ops/second and ~2-2.5k of read latency. * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency. * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5. EDIT: branch cassandra-2.1 is OK. I can help to bisect and/or profile on Monday if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[1/6] cassandra git commit: Fix streaming hang when retrying
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.0 1e35fa4b5 - a11bc7865
refs/heads/cassandra-2.1 6d06f32a7 - 0276c2aab
refs/heads/trunk be6295ede - bbc796dfd
Fix streaming hang when retrying
patch by yukim; reviewed by jmckenzie for CASSANDRA-9132
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a11bc786
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a11bc786
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a11bc786
Branch: refs/heads/cassandra-2.0
Commit: a11bc78656dd4740960c0a1eb6265f910337a16b
Parents: 1e35fa4
Author: Yuki Morishita yu...@apache.org
Authored: Fri May 1 16:52:47 2015 -0500
Committer: Yuki Morishita yu...@apache.org
Committed: Fri May 1 16:52:47 2015 -0500
--
CHANGES.txt | 1 +
.../cassandra/streaming/messages/IncomingFileMessage.java | 7 +++
2 files changed, 8 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/a11bc786/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 2dabbf9..e9287ec 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -57,6 +57,7 @@
(CASSANDRA-8808)
* Fix MT mismatch between empty and GC-able data (CASSANDRA-8979)
* Fix incorrect validation when snapshotting single table (CASSANDRA-8056)
+ * Fix streaming hang when retrying (CASSANDRA-9132)
2.0.14
http://git-wip-us.apache.org/repos/asf/cassandra/blob/a11bc786/src/java/org/apache/cassandra/streaming/messages/IncomingFileMessage.java
--
diff --git
a/src/java/org/apache/cassandra/streaming/messages/IncomingFileMessage.java
b/src/java/org/apache/cassandra/streaming/messages/IncomingFileMessage.java
index a403390..5e0e087 100644
--- a/src/java/org/apache/cassandra/streaming/messages/IncomingFileMessage.java
+++ b/src/java/org/apache/cassandra/streaming/messages/IncomingFileMessage.java
@@ -46,8 +46,15 @@ public class IncomingFileMessage extends StreamMessage
{
return new IncomingFileMessage(reader.read(in), header);
}
+catch (IOException eof)
+{
+// Reading from remote failed(i.e. reached EOF before reading
expected length of data).
+// This can be caused by network/node failure thus we are not
retrying
+throw eof;
+}
catch (Throwable e)
{
+// Otherwise, we can retry
session.doRetry(header, e);
return null;
}
[2/6] cassandra git commit: Fix streaming hang when retrying
Fix streaming hang when retrying
patch by yukim; reviewed by jmckenzie for CASSANDRA-9132
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a11bc786
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a11bc786
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a11bc786
Branch: refs/heads/cassandra-2.1
Commit: a11bc78656dd4740960c0a1eb6265f910337a16b
Parents: 1e35fa4
Author: Yuki Morishita yu...@apache.org
Authored: Fri May 1 16:52:47 2015 -0500
Committer: Yuki Morishita yu...@apache.org
Committed: Fri May 1 16:52:47 2015 -0500
--
CHANGES.txt | 1 +
.../cassandra/streaming/messages/IncomingFileMessage.java | 7 +++
2 files changed, 8 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/a11bc786/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 2dabbf9..e9287ec 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -57,6 +57,7 @@
(CASSANDRA-8808)
* Fix MT mismatch between empty and GC-able data (CASSANDRA-8979)
* Fix incorrect validation when snapshotting single table (CASSANDRA-8056)
+ * Fix streaming hang when retrying (CASSANDRA-9132)
2.0.14
http://git-wip-us.apache.org/repos/asf/cassandra/blob/a11bc786/src/java/org/apache/cassandra/streaming/messages/IncomingFileMessage.java
--
diff --git
a/src/java/org/apache/cassandra/streaming/messages/IncomingFileMessage.java
b/src/java/org/apache/cassandra/streaming/messages/IncomingFileMessage.java
index a403390..5e0e087 100644
--- a/src/java/org/apache/cassandra/streaming/messages/IncomingFileMessage.java
+++ b/src/java/org/apache/cassandra/streaming/messages/IncomingFileMessage.java
@@ -46,8 +46,15 @@ public class IncomingFileMessage extends StreamMessage
{
return new IncomingFileMessage(reader.read(in), header);
}
+catch (IOException eof)
+{
+// Reading from remote failed(i.e. reached EOF before reading
expected length of data).
+// This can be caused by network/node failure thus we are not
retrying
+throw eof;
+}
catch (Throwable e)
{
+// Otherwise, we can retry
session.doRetry(header, e);
return null;
}
[jira] [Commented] (CASSANDRA-6477) Global indexes
[ https://issues.apache.org/jira/browse/CASSANDRA-6477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14524483#comment-14524483 ] Jonathan Ellis commented on CASSANDRA-6477: --- A couple questions: # What is the thinking behind using a customized batchlog vs the existing one? # Is each replica making GI updates, or just one? # What do we do to GI on repair of the base data table? Global indexes -- Key: CASSANDRA-6477 URL: https://issues.apache.org/jira/browse/CASSANDRA-6477 Project: Cassandra Issue Type: New Feature Components: API, Core Reporter: Jonathan Ellis Assignee: Carl Yeksigian Labels: cql Fix For: 3.x Local indexes are suitable for low-cardinality data, where spreading the index across the cluster is a Good Thing. However, for high-cardinality data, local indexes require querying most nodes in the cluster even if only a handful of rows is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9282) Warn on unlogged batches
[ https://issues.apache.org/jira/browse/CASSANDRA-9282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14524471#comment-14524471 ] Jonathan Ellis commented on CASSANDRA-9282: --- Maybe, but we have pretty clear precedent for warning on dangerous things server-side, e.g. tombstone thresholds, and doing it server side means we don't need to wait for 6+ code bases to update AND users to upgrade to the new one. Warn on unlogged batches Key: CASSANDRA-9282 URL: https://issues.apache.org/jira/browse/CASSANDRA-9282 Project: Cassandra Issue Type: Bug Components: API Reporter: Jonathan Ellis Assignee: T Jake Luciani Fix For: 2.1.x At least until CASSANDRA-8303 is done and we can block them entirely, we should log a warning when unlogged batches across multiple partition keys are used. This could either be done by backporting NoSpamLogger and blindly logging every time, or we could add a threshold and warn when more than 10 keys are seen. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9248) Tests for cqlsh parser
[
https://issues.apache.org/jira/browse/CASSANDRA-9248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tyler Hobbs updated CASSANDRA-9248:
---
Attachment: 9248.txt
+1, committed to trunk as {{00cd9b9f}}. Thanks!
Tests for cqlsh parser
--
Key: CASSANDRA-9248
URL: https://issues.apache.org/jira/browse/CASSANDRA-9248
Project: Cassandra
Issue Type: Test
Reporter: Jim Witschey
Assignee: Jim Witschey
Priority: Minor
Fix For: 3.0
Attachments: 9248.txt
Adds tests for the cqlsh parser in isolation. There are still more to write,
but it's a start. Branch here:
https://github.com/mambocab/cassandra/tree/cqlsh-parser-tests
And changes for review here:
https://github.com/apache/cassandra/compare/trunk...mambocab:cqlsh-parser-tests
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[ https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14524374#comment-14524374 ] Benedict commented on CASSANDRA-9240: - OK, so I've pushed a fix for this [here|https://github.com/belliottsmith/cassandra/tree/9240] This both reintroduces the pooling, and shares the mapped segments between all readers, for both pooled and regular compressed files. This could be tidied up, but since we're eliminating a lot of the functionality in these classes in the near future time is probably better spent cleaning them up as that happens. FTR, I managed to reproduce the problem with a regular heap with mmap enabled on trunk, and could eliminate the problem without reintroducing pooling, at least with those heap settings. So the cleaning of the buffers was likely the main culprit. I could strangely reproduce some reduced throughput on trunk with standard mode without pooling vs with, but not on my branch. That's a little odd, but probably a rabbit hole not worth diving into. Performance issue after a restart - Key: CASSANDRA-9240 URL: https://issues.apache.org/jira/browse/CASSANDRA-9240 Project: Cassandra Issue Type: Bug Reporter: Alan Boudreault Assignee: Benedict Priority: Minor Fix For: 3.x Attachments: Cassandra.snapshots.zip, cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log, cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log, cassandra_trunk-clientrequest-read.log, cassandra_trunk.log, cassandra_trunk_no_restart-clientrequest-read.log, cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, runs.log, trace_query.cql I have noticed a performance issue while I was working on compaction perf tests for CASSANDRA-7409. The performance for my use case is very bad after a restart. It is mostly a read performance issue but not strictly. I have attached my use case (see run_issue.sh and issue.yaml) and all test logs for 2.1.4, 2.1.5 and trunk: * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k ops/second and ~2-2.5k of read latency. * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency. * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5. EDIT: branch cassandra-2.1 is OK. I can help to bisect and/or profile on Monday if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[3/6] cassandra git commit: Fix streaming hang when retrying
Fix streaming hang when retrying
patch by yukim; reviewed by jmckenzie for CASSANDRA-9132
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a11bc786
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a11bc786
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a11bc786
Branch: refs/heads/trunk
Commit: a11bc78656dd4740960c0a1eb6265f910337a16b
Parents: 1e35fa4
Author: Yuki Morishita yu...@apache.org
Authored: Fri May 1 16:52:47 2015 -0500
Committer: Yuki Morishita yu...@apache.org
Committed: Fri May 1 16:52:47 2015 -0500
--
CHANGES.txt | 1 +
.../cassandra/streaming/messages/IncomingFileMessage.java | 7 +++
2 files changed, 8 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/a11bc786/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 2dabbf9..e9287ec 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -57,6 +57,7 @@
(CASSANDRA-8808)
* Fix MT mismatch between empty and GC-able data (CASSANDRA-8979)
* Fix incorrect validation when snapshotting single table (CASSANDRA-8056)
+ * Fix streaming hang when retrying (CASSANDRA-9132)
2.0.14
http://git-wip-us.apache.org/repos/asf/cassandra/blob/a11bc786/src/java/org/apache/cassandra/streaming/messages/IncomingFileMessage.java
--
diff --git
a/src/java/org/apache/cassandra/streaming/messages/IncomingFileMessage.java
b/src/java/org/apache/cassandra/streaming/messages/IncomingFileMessage.java
index a403390..5e0e087 100644
--- a/src/java/org/apache/cassandra/streaming/messages/IncomingFileMessage.java
+++ b/src/java/org/apache/cassandra/streaming/messages/IncomingFileMessage.java
@@ -46,8 +46,15 @@ public class IncomingFileMessage extends StreamMessage
{
return new IncomingFileMessage(reader.read(in), header);
}
+catch (IOException eof)
+{
+// Reading from remote failed(i.e. reached EOF before reading
expected length of data).
+// This can be caused by network/node failure thus we are not
retrying
+throw eof;
+}
catch (Throwable e)
{
+// Otherwise, we can retry
session.doRetry(header, e);
return null;
}
[6/6] cassandra git commit: Merge branch 'cassandra-2.1' into trunk
Merge branch 'cassandra-2.1' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/bbc796df Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/bbc796df Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/bbc796df Branch: refs/heads/trunk Commit: bbc796dfd092e01c8c42e1885819f6ba193260f9 Parents: be6295e 0276c2a Author: Yuki Morishita yu...@apache.org Authored: Fri May 1 16:56:37 2015 -0500 Committer: Yuki Morishita yu...@apache.org Committed: Fri May 1 16:56:37 2015 -0500 -- CHANGES.txt | 1 + .../cassandra/streaming/messages/IncomingFileMessage.java | 7 +++ 2 files changed, 8 insertions(+) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/bbc796df/CHANGES.txt -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/bbc796df/src/java/org/apache/cassandra/streaming/messages/IncomingFileMessage.java --
[jira] [Created] (CASSANDRA-9289) Recover from unknown table when deserializing internode messages
Tyler Hobbs created CASSANDRA-9289: -- Summary: Recover from unknown table when deserializing internode messages Key: CASSANDRA-9289 URL: https://issues.apache.org/jira/browse/CASSANDRA-9289 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Tyler Hobbs Priority: Minor Fix For: 3.x As discussed in CASSANDRA-9136, it would be nice to gracefully recover from seeing an unknown table in a message from another node. Currently, we close the connection and reconnect, which can cause other concurrent queries to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
cassandra git commit: Add tests for cqlsh's CQL parser
Repository: cassandra
Updated Branches:
refs/heads/trunk bbc796dfd - 00cd9b9ff
Add tests for cqlsh's CQL parser
Patch by Jim Witschey; reviewed by Tyler Hobbs for CASSANDRA-9248
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/00cd9b9f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/00cd9b9f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/00cd9b9f
Branch: refs/heads/trunk
Commit: 00cd9b9fff711a678c37b1f93ecd8417f22c256e
Parents: bbc796d
Author: Jim Witschey jim.witsc...@gmail.com
Authored: Fri May 1 17:45:11 2015 -0500
Committer: Tyler Hobbs tylerlho...@gmail.com
Committed: Fri May 1 17:45:56 2015 -0500
--
pylib/cqlshlib/test/test_cql_parsing.py | 674 -
pylib/cqlshlib/test/test_cqlsh_parsing.py | 26 +
2 files changed, 673 insertions(+), 27 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/00cd9b9f/pylib/cqlshlib/test/test_cql_parsing.py
--
diff --git a/pylib/cqlshlib/test/test_cql_parsing.py
b/pylib/cqlshlib/test/test_cql_parsing.py
index f88b839..717f9b6 100644
--- a/pylib/cqlshlib/test/test_cql_parsing.py
+++ b/pylib/cqlshlib/test/test_cql_parsing.py
@@ -17,78 +17,698 @@
# to configure behavior, define $CQL_TEST_HOST to the destination address
# for Thrift connections, and $CQL_TEST_PORT to the associated port.
-from .basecase import BaseTestCase, cqlsh
-from .cassconnect import get_test_keyspace, testrun_cqlsh, testcall_cqlsh
+from unittest import TestCase
+from operator import itemgetter
-class TestCqlParsing(BaseTestCase):
-def setUp(self):
-self.cqlsh_runner = testrun_cqlsh(cqlver=cqlsh.DEFAULT_CQLVER,
env={'COLUMNS': '10'})
-self.cqlsh = self.cqlsh_runner.__enter__()
+from ..cql3handling import CqlRuleSet
-def tearDown(self):
-pass
+class TestCqlParsing(TestCase):
def test_parse_string_literals(self):
-pass
+for n in ['eggs', 'Sausage 1', 'spam\nspam\n\tsausage', '']:
+self.assertSequenceEqual(tokens_with_types(CqlRuleSet.lex(n)),
+ [(n, 'quotedStringLiteral')])
+self.assertSequenceEqual(tokens_with_types(CqlRuleSet.lex('eggs')),
+ [('eggs', 'quotedStringLiteral')])
def test_parse_numbers(self):
-pass
+for n in ['6', '398', '18018']:
+self.assertSequenceEqual(tokens_with_types(CqlRuleSet.lex(n)),
+ [(n, 'wholenumber')])
def test_parse_uuid(self):
-pass
+uuids = ['4feeae80-e9cc-11e4-b571-0800200c9a66',
+ '7142303f-828f-4806-be9e-7a973da0c3f9',
+ 'dff8d435-9ca0-487c-b5d0-b0fe5c5768a8']
+for u in uuids:
+self.assertSequenceEqual(tokens_with_types(CqlRuleSet.lex(u)),
+ [(u, 'uuid')])
def test_comments_in_string_literals(self):
-pass
+comment_strings = ['sausage -- comment',
+ 'eggs and spam // comment string',
+ 'spam eggs sausage and spam /* still in string']
+for s in comment_strings:
+self.assertSequenceEqual(tokens_with_types(CqlRuleSet.lex(s)),
+ [(s, 'quotedStringLiteral')])
def test_colons_in_string_literals(self):
-pass
+comment_strings = ['Movie Title: The Movie',
+ ':a:b:c:',
+ '(=) :: (Monad m) = m a - (a - m b) - m b']
+for s in comment_strings:
+self.assertSequenceEqual(tokens_with_types(CqlRuleSet.lex(s)),
+ [(s, 'quotedStringLiteral')])
def test_partial_parsing(self):
-pass
+[parsed] = CqlRuleSet.cql_parse('INSERT INTO ks.test')
+self.assertSequenceEqual(parsed.matched, [])
+self.assertSequenceEqual(tokens_with_types(parsed.remainder),
+ [('INSERT', 'K_INSERT'),
+ ('INTO', 'K_INTO'),
+ ('ks', 'identifier'),
+ ('.', 'op'),
+ ('test', 'identifier')])
def test_parse_select(self):
-pass
+parsed = parse_cqlsh_statements('SELECT FROM ks.tab;')
+self.assertSequenceEqual(tokens_with_types(parsed),
+ [('SELECT', 'K_SELECT'),
+ ('FROM', 'K_FROM'),
+ ('ks', 'identifier'),
+ ('.', 'op'),
+ ('tab', 'identifier'),
+ (';',
[jira] [Commented] (CASSANDRA-9207) COPY FROM command does not restore all records
[
https://issues.apache.org/jira/browse/CASSANDRA-9207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14524162#comment-14524162
]
Tyler Hobbs commented on CASSANDRA-9207:
{{cqlshrc.sample}} is just an example file, it won't actually affect your cqlsh
settings. You should be editing {{~/.cassandra/cqlshrc}}, assuming you're on
Linux or OSX.
COPY FROM command does not restore all records
--
Key: CASSANDRA-9207
URL: https://issues.apache.org/jira/browse/CASSANDRA-9207
Project: Cassandra
Issue Type: Bug
Components: Core, Tools
Environment: centOs, cassandra 2.1.1
Reporter: Gaurav
Assignee: Tyler Hobbs
Labels: COPY, cqlsh
Fix For: 2.1.x
Attachments: cron_task.csv
Steps to reproduce:
1. run COPY TO command on keyspace A. (on my environment problem occurred on
only 1 table out of 19, only thing that io noticed is this table has 1500
records and other tables has 1000 records)
2. pull the csv file and place it on machine (say machine B)where other
database resides. (In my case this was another instance of amazon machine)
3. Now, run the COPY TO command on machine B. (both keyspaces, one on machine
A and one on machine B has same schema.)
Observation:
1. when COPY TO command is run for table having records 1500. Command gave
following output:
Processing 1000 records
1573 records copied.
but when i tried to verify it by running below mentioned commands, i received
only 273 records.
1. SELECT COUNT(*) FROM table-name;
2. SELECT * FROM table-name;
Note: please let me know if other information needs to be shared with you.
Also, is there any other way to take the back-up of keyspace and restoring it
on other machine.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9136) Improve error handling when table is queried before the schema has fully propagated
[
https://issues.apache.org/jira/browse/CASSANDRA-9136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14524258#comment-14524258
]
Tyler Hobbs commented on CASSANDRA-9136:
Since we haven't decided what the best solution for recovery is and we all
agree on the error message part, I've opened CASSANDRA-9289 to deal with
recovery separately. I'll have a patch and test for the error message shortly.
Improve error handling when table is queried before the schema has fully
propagated
---
Key: CASSANDRA-9136
URL: https://issues.apache.org/jira/browse/CASSANDRA-9136
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: 3 Nodes GCE, N1-Standard-2, Ubuntu 12, 1 Node on 2.1.4,
2 on 2.0.14
Reporter: Russell Alexander Spitzer
Assignee: Tyler Hobbs
Fix For: 2.1.x
This error occurs during a rolling upgrade between 2.0.14 and 2.1.4.
h3. Repo
With all the nodes on 2.0.14 make the following tables
{code}
CREATE KEYSPACE test WITH replication = {
'class': 'SimpleStrategy',
'replication_factor': '2'
};
USE test;
CREATE TABLE compact (
k int,
c int,
d int,
PRIMARY KEY ((k), c)
) WITH COMPACT STORAGE;
CREATE TABLE norm (
k int,
c int,
d int,
PRIMARY KEY ((k), c)
) ;
{code}
Then load some data into these tables. I used the python driver
{code}
from cassandra.cluster import Cluster
s = Cluster().connect()
for x in range (1000):
for y in range (1000):
s.execute_async(INSERT INTO test.compact (k,c,d) VALUES
(%d,%d,%d)%(x,y,y))
s.execute_async(INSERT INTO test.norm (k,c,d) VALUES
(%d,%d,%d)%(x,y,y))
{code}
Upgrade one node from 2.0.14 - 2.1.4
From the 2.1.4 node, create a new table.
Query that table
On the 2.0.14 nodes you get these exceptions because the schema didn't
propagate there. This exception kills the TCP connection between the nodes.
{code}
ERROR [Thread-19] 2015-04-08 18:48:45,337 CassandraDaemon.java (line 258)
Exception in thread Thread[Thread-19,5,main]
java.lang.NullPointerException
at
org.apache.cassandra.db.RangeSliceCommandSerializer.deserialize(RangeSliceCommand.java:247)
at
org.apache.cassandra.db.RangeSliceCommandSerializer.deserialize(RangeSliceCommand.java:156)
at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:149)
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:131)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:74)
{code}
Run cqlsh on the upgraded node and queries will fail until the TCP connection
is established again, easiest to repo with CL = ALL
{code}
cqlsh SELECT count(*) FROM test.norm where k = 22 ;
ReadTimeout: code=1200 [Coordinator node timed out waiting for replica nodes'
responses] message=Operation timed out - received only 1 responses.
info={'received_responses': 1, 'required_responses': 2, 'consistency': 'ALL'}
cqlsh SELECT count(*) FROM test.norm where k = 21 ;
ReadTimeout: code=1200 [Coordinator node timed out waiting for replica nodes'
responses] message=Operation timed out - received only 1 responses.
info={'received_responses': 1, 'required_responses': 2, 'consistency': 'ALL'}
{code}
So connection made:
{code}
DEBUG [Thread-227] 2015-04-09 05:09:02,718 IncomingTcpConnection.java (line
107) Set version for /10.240.14.115 to 8 (will use 7)
{code}
Connection broken by query of table before schema propagated:
{code}
ERROR [Thread-227] 2015-04-09 05:10:24,015 CassandraDaemon.java (line 258)
Exception in thread Thread[Thread-227,5,main]
java.lang.NullPointerException
at
org.apache.cassandra.db.RangeSliceCommandSerializer.deserialize(RangeSliceCommand.java:247)
at
org.apache.cassandra.db.RangeSliceCommandSerializer.deserialize(RangeSliceCommand.java:156)
at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:149)
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:131)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:74)
{code}
All query to that node will fail with timeouts now until...
Connection re-established
{code}
DEBUG [Thread-228] 2015-04-09 05:11:00,323 IncomingTcpConnection.java (line
107) Set version for /10.240.14.115 to 8 (will use 7)
{code}
Now queries work again.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[5/6] cassandra git commit: Merge branch 'cassandra-2.0' into cassandra-2.1
Merge branch 'cassandra-2.0' into cassandra-2.1 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/0276c2aa Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/0276c2aa Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/0276c2aa Branch: refs/heads/cassandra-2.1 Commit: 0276c2aab72c7b18b9015b0491013d958dafc46a Parents: 6d06f32 a11bc78 Author: Yuki Morishita yu...@apache.org Authored: Fri May 1 16:56:30 2015 -0500 Committer: Yuki Morishita yu...@apache.org Committed: Fri May 1 16:56:30 2015 -0500 -- CHANGES.txt | 1 + .../cassandra/streaming/messages/IncomingFileMessage.java | 7 +++ 2 files changed, 8 insertions(+) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/0276c2aa/CHANGES.txt -- diff --cc CHANGES.txt index a6e4e41,e9287ec..a8cd1db --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -13,93 -4,10 +13,94 @@@ Merged from 2.0 * Allow scrub to handle corrupted compressed chunks (CASSANDRA-9140) * Fix assertion error when resetlocalschema is run during repair (CASSANDRA-9249) * Disable single sstable tombstone compactions for DTCS by default (CASSANDRA-9234) - * Do more agressive ttl expiration checks to be able to - drop more sstables (CASSANDRA-8243) * IncomingTcpConnection thread is not named (CASSANDRA-9262) * Close incoming connections when MessagingService is stopped (CASSANDRA-9238) ++ * Fix streaming hang when retrying (CASSANDRA-9132) + + +2.1.5 + * Re-add deprecated cold_reads_to_omit param for backwards compat (CASSANDRA-9203) + * Make anticompaction visible in compactionstats (CASSANDRA-9098) + * Improve nodetool getendpoints documentation about the partition + key parameter (CASSANDRA-6458) + * Don't check other keyspaces for schema changes when an user-defined + type is altered (CASSANDRA-9187) + * Allow takeColumnFamilySnapshot to take a list of tables (CASSANDRA-8348) + * Limit major sstable operations to their canonical representation (CASSANDRA-8669) + * cqlsh: Add tests for INSERT and UPDATE tab completion (CASSANDRA-9125) + * cqlsh: quote column names when needed in COPY FROM inserts (CASSANDRA-9080) + * Add generate-idea-files target to build.xml (CASSANDRA-9123) + * Do not load read meter for offline operations (CASSANDRA-9082) + * cqlsh: Make CompositeType data readable (CASSANDRA-8919) + * cqlsh: Fix display of triggers (CASSANDRA-9081) + * Fix NullPointerException when deleting or setting an element by index on + a null list collection (CASSANDRA-9077) + * Buffer bloom filter serialization (CASSANDRA-9066) + * Fix anti-compaction target bloom filter size (CASSANDRA-9060) + * Make FROZEN and TUPLE unreserved keywords in CQL (CASSANDRA-9047) + * Prevent AssertionError from SizeEstimatesRecorder (CASSANDRA-9034) + * Avoid overwriting index summaries for sstables with an older format that + does not support downsampling; rebuild summaries on startup when this + is detected (CASSANDRA-8993) + * Fix potential data loss in CompressedSequentialWriter (CASSANDRA-8949) + * Make PasswordAuthenticator number of hashing rounds configurable (CASSANDRA-8085) + * Fix AssertionError when binding nested collections in DELETE (CASSANDRA-8900) + * Check for overlap with non-early sstables in LCS (CASSANDRA-8739) + * Only calculate max purgable timestamp if we have to (CASSANDRA-8914) + * (cqlsh) Greatly improve performance of COPY FROM (CASSANDRA-8225) + * IndexSummary effectiveIndexInterval is now a guideline, not a rule (CASSANDRA-8993) + * Use correct bounds for page cache eviction of compressed files (CASSANDRA-8746) + * SSTableScanner enforces its bounds (CASSANDRA-8946) + * Cleanup cell equality (CASSANDRA-8947) + * Introduce intra-cluster message coalescing (CASSANDRA-8692) + * DatabaseDescriptor throws NPE when rpc_interface is used (CASSANDRA-8839) + * Don't check if an sstable is live for offline compactions (CASSANDRA-8841) + * Don't set clientMode in SSTableLoader (CASSANDRA-8238) + * Fix SSTableRewriter with disabled early open (CASSANDRA-8535) + * Allow invalidating permissions and cache time (CASSANDRA-8722) + * Log warning when queries that will require ALLOW FILTERING in Cassandra 3.0 + are executed (CASSANDRA-8418) + * Fix cassandra-stress so it respects the CL passed in user mode (CASSANDRA-8948) + * Fix rare NPE in ColumnDefinition#hasIndexOption() (CASSANDRA-8786) + * cassandra-stress reports per-operation statistics, plus misc (CASSANDRA-8769) + * Use long for key count in cfstats (CASSANDRA-8913) + * Make SSTableRewriter.abort() more robust to failure (CASSANDRA-8832) + * Remove cold_reads_to_omit from STCS (CASSANDRA-8860) + * Make
[4/6] cassandra git commit: Merge branch 'cassandra-2.0' into cassandra-2.1
Merge branch 'cassandra-2.0' into cassandra-2.1 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/0276c2aa Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/0276c2aa Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/0276c2aa Branch: refs/heads/trunk Commit: 0276c2aab72c7b18b9015b0491013d958dafc46a Parents: 6d06f32 a11bc78 Author: Yuki Morishita yu...@apache.org Authored: Fri May 1 16:56:30 2015 -0500 Committer: Yuki Morishita yu...@apache.org Committed: Fri May 1 16:56:30 2015 -0500 -- CHANGES.txt | 1 + .../cassandra/streaming/messages/IncomingFileMessage.java | 7 +++ 2 files changed, 8 insertions(+) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/0276c2aa/CHANGES.txt -- diff --cc CHANGES.txt index a6e4e41,e9287ec..a8cd1db --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -13,93 -4,10 +13,94 @@@ Merged from 2.0 * Allow scrub to handle corrupted compressed chunks (CASSANDRA-9140) * Fix assertion error when resetlocalschema is run during repair (CASSANDRA-9249) * Disable single sstable tombstone compactions for DTCS by default (CASSANDRA-9234) - * Do more agressive ttl expiration checks to be able to - drop more sstables (CASSANDRA-8243) * IncomingTcpConnection thread is not named (CASSANDRA-9262) * Close incoming connections when MessagingService is stopped (CASSANDRA-9238) ++ * Fix streaming hang when retrying (CASSANDRA-9132) + + +2.1.5 + * Re-add deprecated cold_reads_to_omit param for backwards compat (CASSANDRA-9203) + * Make anticompaction visible in compactionstats (CASSANDRA-9098) + * Improve nodetool getendpoints documentation about the partition + key parameter (CASSANDRA-6458) + * Don't check other keyspaces for schema changes when an user-defined + type is altered (CASSANDRA-9187) + * Allow takeColumnFamilySnapshot to take a list of tables (CASSANDRA-8348) + * Limit major sstable operations to their canonical representation (CASSANDRA-8669) + * cqlsh: Add tests for INSERT and UPDATE tab completion (CASSANDRA-9125) + * cqlsh: quote column names when needed in COPY FROM inserts (CASSANDRA-9080) + * Add generate-idea-files target to build.xml (CASSANDRA-9123) + * Do not load read meter for offline operations (CASSANDRA-9082) + * cqlsh: Make CompositeType data readable (CASSANDRA-8919) + * cqlsh: Fix display of triggers (CASSANDRA-9081) + * Fix NullPointerException when deleting or setting an element by index on + a null list collection (CASSANDRA-9077) + * Buffer bloom filter serialization (CASSANDRA-9066) + * Fix anti-compaction target bloom filter size (CASSANDRA-9060) + * Make FROZEN and TUPLE unreserved keywords in CQL (CASSANDRA-9047) + * Prevent AssertionError from SizeEstimatesRecorder (CASSANDRA-9034) + * Avoid overwriting index summaries for sstables with an older format that + does not support downsampling; rebuild summaries on startup when this + is detected (CASSANDRA-8993) + * Fix potential data loss in CompressedSequentialWriter (CASSANDRA-8949) + * Make PasswordAuthenticator number of hashing rounds configurable (CASSANDRA-8085) + * Fix AssertionError when binding nested collections in DELETE (CASSANDRA-8900) + * Check for overlap with non-early sstables in LCS (CASSANDRA-8739) + * Only calculate max purgable timestamp if we have to (CASSANDRA-8914) + * (cqlsh) Greatly improve performance of COPY FROM (CASSANDRA-8225) + * IndexSummary effectiveIndexInterval is now a guideline, not a rule (CASSANDRA-8993) + * Use correct bounds for page cache eviction of compressed files (CASSANDRA-8746) + * SSTableScanner enforces its bounds (CASSANDRA-8946) + * Cleanup cell equality (CASSANDRA-8947) + * Introduce intra-cluster message coalescing (CASSANDRA-8692) + * DatabaseDescriptor throws NPE when rpc_interface is used (CASSANDRA-8839) + * Don't check if an sstable is live for offline compactions (CASSANDRA-8841) + * Don't set clientMode in SSTableLoader (CASSANDRA-8238) + * Fix SSTableRewriter with disabled early open (CASSANDRA-8535) + * Allow invalidating permissions and cache time (CASSANDRA-8722) + * Log warning when queries that will require ALLOW FILTERING in Cassandra 3.0 + are executed (CASSANDRA-8418) + * Fix cassandra-stress so it respects the CL passed in user mode (CASSANDRA-8948) + * Fix rare NPE in ColumnDefinition#hasIndexOption() (CASSANDRA-8786) + * cassandra-stress reports per-operation statistics, plus misc (CASSANDRA-8769) + * Use long for key count in cfstats (CASSANDRA-8913) + * Make SSTableRewriter.abort() more robust to failure (CASSANDRA-8832) + * Remove cold_reads_to_omit from STCS (CASSANDRA-8860) + * Make
[jira] [Commented] (CASSANDRA-7486) Compare CMS and G1 pause times
[ https://issues.apache.org/jira/browse/CASSANDRA-7486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14524278#comment-14524278 ] Matt Stump commented on CASSANDRA-7486: --- Before we talk about changing the defaults I would like to see tests run on something more representative of customer hardware. At the very least we should be doing comparisons of CMS vs G1 for different workloads on cstar. I did some initial testing and didn't see a huge benefit, but I very well could have been doing something wrong. I'm both hopeful and skeptical. Compare CMS and G1 pause times -- Key: CASSANDRA-7486 URL: https://issues.apache.org/jira/browse/CASSANDRA-7486 Project: Cassandra Issue Type: Test Components: Config Reporter: Jonathan Ellis Assignee: Shawn Kumar Fix For: 2.1.x See http://www.slideshare.net/MonicaBeckwith/garbage-first-garbage-collector-g1-7486gc-migration-to-expectations-and-advanced-tuning and https://twitter.com/rbranson/status/482113561431265281 May want to default 2.1 to G1. 2.1 is a different animal from 2.0 after moving most of memtables off heap. Suspect this will help G1 even more than CMS. (NB this is off by default but needs to be part of the test.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9279) Gossip (and mutations) lock up on startup
[
https://issues.apache.org/jira/browse/CASSANDRA-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14524479#comment-14524479
]
Jonathan Ellis commented on CASSANDRA-9279:
---
[~sebastian.este...@datastax.com] are there exceptions earlier in startup?
Gossip (and mutations) lock up on startup
-
Key: CASSANDRA-9279
URL: https://issues.apache.org/jira/browse/CASSANDRA-9279
Project: Cassandra
Issue Type: Bug
Reporter: Sebastian Estevez
Fix For: 2.0.x
Attachments: Screen Shot 2015-04-30 at 4.41.57 PM.png
Cluster running 2.0.14.352 on EC2 - c3.4xl's
2 nodes out of 8 exhibited the following behavior
When starting up the node we noticed it was gray in OpsCenter. Other
monitoring tool showed it as up.
Turned out gossip tasks were piling up and we could see the following in the
system.log:
{code}
WARN [GossipTasks:1] 2015-04-30 20:22:29,512 Gossiper.java (line 671) Gossip
stage has 4270 pending tasks; skipping status check (no nodes will be marked
down)
WARN [GossipTasks:1] 2015-04-30 20:22:30,612 Gossiper.java (line 671) Gossip
stage has 4272 pending tasks; skipping status check (no nodes will be marked
down)
WARN [GossipTasks:1] 2015-04-30 20:22:31,713 Gossiper.java (line 671) Gossip
stage has 4273 pending tasks; skipping status check (no nodes will be marked
down)
...
{code}
and tpstats shows blocked tasks--gossip and mutations:
{code}
GossipStage 1 3904 29384 0
0
{code}
the CPU's are inactive (See attachment)
and dstat output:
{code}
You did not select any stats, using -cdngy by default.
total-cpu-usage -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read writ| recv send| in out | int csw
2 0 97 0 0 0|1324k 1381k| 0 0 | 0 0 |6252 5548
0 0 100 0 0 0| 064k| 42k 1017k| 0 0 |3075 2537
0 0 99 0 0 0| 0 8192B| 39k 794k| 0 0 |6999 7039
0 0 100 0 0 0| 0 0 | 39k 759k| 0 0 |3067 2726
0 0 99 0 0 0| 0 184k| 48k 1086k| 0 0 |4829 4178
0 0 99 0 0 0| 0 8192B| 34k 802k| 0 0 |1671 1240
0 0 100 0 0 0| 0 8192B| 48k 1067k| 0 0 |1878 1193
{code}
I managed to grab a thread dump:
https://gist.githubusercontent.com/anonymous/3b7b4698c32032603493/raw/read.md
and dmesg:
https://gist.githubusercontent.com/anonymous/5982b15337c9afbd5d49/raw/f3c2e4411b9d59e90f4615d93c7c1ad25922e170/read.md
Restarting the node solved the issue (it came up normally), we don't know
what is causing it but apparently (per the thread dump) gossip threads are
blocked writing the system keyspace and the writes waiting on the commitlog.
Gossip:
{code}
GossipStage:1 daemon prio=10 tid=0x7ffa23471800 nid=0xa13fa waiting on
condition [0x7ff9cbe26000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for 0x0005d3f50960 (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:964)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282)
at
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:731)
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351)
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:336)
at org.apache.cassandra.db.RowMutation.apply(RowMutation.java:211)
at
org.apache.cassandra.cql3.statements.ModificationStatement.executeInternal(ModificationStatement.java:709)
at
org.apache.cassandra.cql3.QueryProcessor.processInternal(QueryProcessor.java:208)
at
org.apache.cassandra.db.SystemKeyspace.updatePeerInfo(SystemKeyspace.java:379)
- locked 0x0005d3f41ed8 (a java.lang.Class for
org.apache.cassandra.db.SystemKeyspace)
at
org.apache.cassandra.service.StorageService.updatePeerInfo(StorageService.java:1414)
at
org.apache.cassandra.service.StorageService.handleStateNormal(StorageService.java:1524)
at
org.apache.cassandra.service.StorageService.onChange(StorageService.java:1350)
at
org.apache.cassandra.gms.Gossiper.doOnChangeNotifications(Gossiper.java:1083)
at org.apache.cassandra.gms.Gossiper.applyNewStates(Gossiper.java:1065)
at
[jira] [Updated] (CASSANDRA-9285) LEAK DETECTED in sstwriter
[
https://issues.apache.org/jira/browse/CASSANDRA-9285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre N. updated CASSANDRA-9285:
-
Description:
reproduce bug :
{code}
public static void main(String[] args) throws Exception {
System.setProperty(cassandra.debugrefcount,true);
String ks = ks1;
String table = t1;
String schema = CREATE TABLE + ks + . + table + (a1 INT, PRIMARY
KEY (a1));;
String insert = INSERT INTO + ks + . + table + (a1) VALUES(?);;
File dir = new File(/var/tmp/ + ks + / + table);
dir.mkdirs();
CQLSSTableWriter writer =
CQLSSTableWriter.builder().forTable(schema).using(insert).inDirectory(dir).build();
writer.addRow(1);
writer.close();
writer = null;
Thread.sleep(1000);System.gc();
Thread.sleep(1000);System.gc();
}
{code}
{quote}
[2015-05-01 16:09:59,139] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - LEAK DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@79fa9da9) to class
org.apache.cassandra.io.util.SafeMemory$MemoryTidy@2053866990:Memory@[7f87f8043b20..7f87f8043b48)
was not released before the reference was garbage collected
[2015-05-01 16:09:59,143] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - Allocate trace
org.apache.cassandra.utils.concurrent.Ref$State@79fa9da9:
Thread[Thread-2,5,main]
at java.lang.Thread.getStackTrace(Thread.java:1552)
at org.apache.cassandra.utils.concurrent.Ref$Debug.init(Ref.java:200)
at org.apache.cassandra.utils.concurrent.Ref$State.init(Ref.java:133)
at org.apache.cassandra.utils.concurrent.Ref.init(Ref.java:60)
at org.apache.cassandra.io.util.SafeMemory.init(SafeMemory.java:32)
at
org.apache.cassandra.io.util.SafeMemoryWriter.init(SafeMemoryWriter.java:33)
at
org.apache.cassandra.io.sstable.IndexSummaryBuilder.init(IndexSummaryBuilder.java:111)
at
org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.init(SSTableWriter.java:576)
at
org.apache.cassandra.io.sstable.SSTableWriter.init(SSTableWriter.java:140)
at
org.apache.cassandra.io.sstable.AbstractSSTableSimpleWriter.getWriter(AbstractSSTableSimpleWriter.java:58)
at
org.apache.cassandra.io.sstable.SSTableSimpleUnsortedWriter$DiskWriter.run(SSTableSimpleUnsortedWriter.java:227)
[2015-05-01 16:09:59,144] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - LEAK DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@664382e3) to class
org.apache.cassandra.io.util.SafeMemory$MemoryTidy@899100784:Memory@[7f87f8043990..7f87f8043994)
was not released before the reference was garbage collected
[2015-05-01 16:09:59,144] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - Allocate trace
org.apache.cassandra.utils.concurrent.Ref$State@664382e3:
Thread[Thread-2,5,main]
at java.lang.Thread.getStackTrace(Thread.java:1552)
at org.apache.cassandra.utils.concurrent.Ref$Debug.init(Ref.java:200)
at org.apache.cassandra.utils.concurrent.Ref$State.init(Ref.java:133)
at org.apache.cassandra.utils.concurrent.Ref.init(Ref.java:60)
at org.apache.cassandra.io.util.SafeMemory.init(SafeMemory.java:32)
at
org.apache.cassandra.io.util.SafeMemoryWriter.init(SafeMemoryWriter.java:33)
at
org.apache.cassandra.io.sstable.IndexSummaryBuilder.init(IndexSummaryBuilder.java:110)
at
org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.init(SSTableWriter.java:576)
at
org.apache.cassandra.io.sstable.SSTableWriter.init(SSTableWriter.java:140)
at
org.apache.cassandra.io.sstable.AbstractSSTableSimpleWriter.getWriter(AbstractSSTableSimpleWriter.java:58)
at
org.apache.cassandra.io.sstable.SSTableSimpleUnsortedWriter$DiskWriter.run(SSTableSimpleUnsortedWriter.java:227)
[2015-05-01 16:09:59,144] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - LEAK DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@3cca0ac2) to class
org.apache.cassandra.io.util.SafeMemory$MemoryTidy@499043670:Memory@[7f87f8039940..7f87f8039c60)
was not released before the reference was garbage collected
[2015-05-01 16:09:59,144] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - Allocate trace
org.apache.cassandra.utils.concurrent.Ref$State@3cca0ac2:
Thread[Thread-2,5,main]
at java.lang.Thread.getStackTrace(Thread.java:1552)
at org.apache.cassandra.utils.concurrent.Ref$Debug.init(Ref.java:200)
at org.apache.cassandra.utils.concurrent.Ref$State.init(Ref.java:133)
at org.apache.cassandra.utils.concurrent.Ref.init(Ref.java:60)
at org.apache.cassandra.io.util.SafeMemory.init(SafeMemory.java:32)
at
[jira] [Comment Edited] (CASSANDRA-9284) support joins if partition key is the same in both tables
[ https://issues.apache.org/jira/browse/CASSANDRA-9284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523251#comment-14523251 ] Ryan Svihla edited comment on CASSANDRA-9284 at 5/1/15 2:32 PM: This could add some flexibility to data modeling practices and would allow one giant table to be split up into smaller tables. One would still have to be mindful of the total retrieval size in a way we do not have to consider now, but this would I believe help people new to Cassandra still keep their table models conceptually broken up into smaller easier to understand pieces. Also is has the side effect of reducing partition size into smaller logical units wrt compaction. Some data models that would probably go over the in_memory_compaction_limit previously would no longer as that data would be split up into different tables. This effectively allows another type of sharding for large logical rows. was (Author: rssvihla): This could add some flexibility to data modeling practices and would allow one giant table to be split up into smaller tables. One would still have to be mindful of the total retrieval size in a way we do not have to consider now, but this would I believe help people new to Cassandra still keep their table models conceptually broken up into smaller easier to understand pieces. support joins if partition key is the same in both tables - Key: CASSANDRA-9284 URL: https://issues.apache.org/jira/browse/CASSANDRA-9284 Project: Cassandra Issue Type: Improvement Reporter: Jon Haddad Priority: Minor Based off this conversation: https://mail-archives.apache.org/mod_mbox/cassandra-dev/201505.mbox/%3CCACUnPaAfJqU%2B86fFd4S6MS7Wv0KhpT_vavdkvDS%2Bm4Madi8_cg%40mail.gmail.com%3E It would be nice to have the flexibility of joins if we knew the query could be satisfied on a single node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9285) LEAK DETECTED in sstwriter
[
https://issues.apache.org/jira/browse/CASSANDRA-9285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523262#comment-14523262
]
T Jake Luciani commented on CASSANDRA-9285:
---
So this is only happening with the CQL writer? not on the server itself?
LEAK DETECTED in sstwriter
--
Key: CASSANDRA-9285
URL: https://issues.apache.org/jira/browse/CASSANDRA-9285
Project: Cassandra
Issue Type: Bug
Reporter: Pierre N.
Fix For: 2.1.x
reproduce bug :
{code}
public static void main(String[] args) throws Exception {
System.setProperty(cassandra.debugrefcount,true);
String ks = ks1;
String table = t1;
String schema = CREATE TABLE + ks + . + table + (a1 INT,
PRIMARY KEY (a1));;
String insert = INSERT INTO + ks + . + table + (a1) VALUES(?);;
File dir = new File(/var/tmp/ + ks + / + table);
dir.mkdirs();
CQLSSTableWriter writer =
CQLSSTableWriter.builder().forTable(schema).using(insert).inDirectory(dir).build();
writer.addRow(1);
writer.close();
writer = null;
Thread.sleep(1000);System.gc();
Thread.sleep(1000);System.gc();
}
{code}
{quote}
[2015-05-01 16:09:59,139] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - LEAK DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@79fa9da9) to class
org.apache.cassandra.io.util.SafeMemory$MemoryTidy@2053866990:Memory@[7f87f8043b20..7f87f8043b48)
was not released before the reference was garbage collected
[2015-05-01 16:09:59,143] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - Allocate trace
org.apache.cassandra.utils.concurrent.Ref$State@79fa9da9:
Thread[Thread-2,5,main]
at java.lang.Thread.getStackTrace(Thread.java:1552)
at org.apache.cassandra.utils.concurrent.Ref$Debug.init(Ref.java:200)
at org.apache.cassandra.utils.concurrent.Ref$State.init(Ref.java:133)
at org.apache.cassandra.utils.concurrent.Ref.init(Ref.java:60)
at org.apache.cassandra.io.util.SafeMemory.init(SafeMemory.java:32)
at
org.apache.cassandra.io.util.SafeMemoryWriter.init(SafeMemoryWriter.java:33)
at
org.apache.cassandra.io.sstable.IndexSummaryBuilder.init(IndexSummaryBuilder.java:111)
at
org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.init(SSTableWriter.java:576)
at
org.apache.cassandra.io.sstable.SSTableWriter.init(SSTableWriter.java:140)
at
org.apache.cassandra.io.sstable.AbstractSSTableSimpleWriter.getWriter(AbstractSSTableSimpleWriter.java:58)
at
org.apache.cassandra.io.sstable.SSTableSimpleUnsortedWriter$DiskWriter.run(SSTableSimpleUnsortedWriter.java:227)
[2015-05-01 16:09:59,144] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - LEAK DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@664382e3) to class
org.apache.cassandra.io.util.SafeMemory$MemoryTidy@899100784:Memory@[7f87f8043990..7f87f8043994)
was not released before the reference was garbage collected
[2015-05-01 16:09:59,144] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - Allocate trace
org.apache.cassandra.utils.concurrent.Ref$State@664382e3:
Thread[Thread-2,5,main]
at java.lang.Thread.getStackTrace(Thread.java:1552)
at org.apache.cassandra.utils.concurrent.Ref$Debug.init(Ref.java:200)
at org.apache.cassandra.utils.concurrent.Ref$State.init(Ref.java:133)
at org.apache.cassandra.utils.concurrent.Ref.init(Ref.java:60)
at org.apache.cassandra.io.util.SafeMemory.init(SafeMemory.java:32)
at
org.apache.cassandra.io.util.SafeMemoryWriter.init(SafeMemoryWriter.java:33)
at
org.apache.cassandra.io.sstable.IndexSummaryBuilder.init(IndexSummaryBuilder.java:110)
at
org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.init(SSTableWriter.java:576)
at
org.apache.cassandra.io.sstable.SSTableWriter.init(SSTableWriter.java:140)
at
org.apache.cassandra.io.sstable.AbstractSSTableSimpleWriter.getWriter(AbstractSSTableSimpleWriter.java:58)
at
org.apache.cassandra.io.sstable.SSTableSimpleUnsortedWriter$DiskWriter.run(SSTableSimpleUnsortedWriter.java:227)
[2015-05-01 16:09:59,144] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - LEAK DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@3cca0ac2) to class
org.apache.cassandra.io.util.SafeMemory$MemoryTidy@499043670:Memory@[7f87f8039940..7f87f8039c60)
was not released before the reference was garbage collected
[2015-05-01 16:09:59,144] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - Allocate trace
[jira] [Updated] (CASSANDRA-9285) LEAK DETECTED in sstwriter
[
https://issues.apache.org/jira/browse/CASSANDRA-9285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
T Jake Luciani updated CASSANDRA-9285:
--
Fix Version/s: 2.1.x
LEAK DETECTED in sstwriter
--
Key: CASSANDRA-9285
URL: https://issues.apache.org/jira/browse/CASSANDRA-9285
Project: Cassandra
Issue Type: Bug
Reporter: Pierre N.
Fix For: 2.1.x
reproduce bug :
{code}
public static void main(String[] args) throws Exception {
System.setProperty(cassandra.debugrefcount,true);
String ks = ks1;
String table = t1;
String schema = CREATE TABLE + ks + . + table + (a1 INT,
PRIMARY KEY (a1));;
String insert = INSERT INTO + ks + . + table + (a1) VALUES(?);;
File dir = new File(/var/tmp/ + ks + / + table);
dir.mkdirs();
CQLSSTableWriter writer =
CQLSSTableWriter.builder().forTable(schema).using(insert).inDirectory(dir).build();
writer.addRow(1);
writer.close();
writer = null;
Thread.sleep(1000);System.gc();
Thread.sleep(1000);System.gc();
}
{code}
{quote}
[2015-05-01 16:09:59,139] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - LEAK DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@79fa9da9) to class
org.apache.cassandra.io.util.SafeMemory$MemoryTidy@2053866990:Memory@[7f87f8043b20..7f87f8043b48)
was not released before the reference was garbage collected
[2015-05-01 16:09:59,143] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - Allocate trace
org.apache.cassandra.utils.concurrent.Ref$State@79fa9da9:
Thread[Thread-2,5,main]
at java.lang.Thread.getStackTrace(Thread.java:1552)
at org.apache.cassandra.utils.concurrent.Ref$Debug.init(Ref.java:200)
at org.apache.cassandra.utils.concurrent.Ref$State.init(Ref.java:133)
at org.apache.cassandra.utils.concurrent.Ref.init(Ref.java:60)
at org.apache.cassandra.io.util.SafeMemory.init(SafeMemory.java:32)
at
org.apache.cassandra.io.util.SafeMemoryWriter.init(SafeMemoryWriter.java:33)
at
org.apache.cassandra.io.sstable.IndexSummaryBuilder.init(IndexSummaryBuilder.java:111)
at
org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.init(SSTableWriter.java:576)
at
org.apache.cassandra.io.sstable.SSTableWriter.init(SSTableWriter.java:140)
at
org.apache.cassandra.io.sstable.AbstractSSTableSimpleWriter.getWriter(AbstractSSTableSimpleWriter.java:58)
at
org.apache.cassandra.io.sstable.SSTableSimpleUnsortedWriter$DiskWriter.run(SSTableSimpleUnsortedWriter.java:227)
[2015-05-01 16:09:59,144] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - LEAK DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@664382e3) to class
org.apache.cassandra.io.util.SafeMemory$MemoryTidy@899100784:Memory@[7f87f8043990..7f87f8043994)
was not released before the reference was garbage collected
[2015-05-01 16:09:59,144] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - Allocate trace
org.apache.cassandra.utils.concurrent.Ref$State@664382e3:
Thread[Thread-2,5,main]
at java.lang.Thread.getStackTrace(Thread.java:1552)
at org.apache.cassandra.utils.concurrent.Ref$Debug.init(Ref.java:200)
at org.apache.cassandra.utils.concurrent.Ref$State.init(Ref.java:133)
at org.apache.cassandra.utils.concurrent.Ref.init(Ref.java:60)
at org.apache.cassandra.io.util.SafeMemory.init(SafeMemory.java:32)
at
org.apache.cassandra.io.util.SafeMemoryWriter.init(SafeMemoryWriter.java:33)
at
org.apache.cassandra.io.sstable.IndexSummaryBuilder.init(IndexSummaryBuilder.java:110)
at
org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.init(SSTableWriter.java:576)
at
org.apache.cassandra.io.sstable.SSTableWriter.init(SSTableWriter.java:140)
at
org.apache.cassandra.io.sstable.AbstractSSTableSimpleWriter.getWriter(AbstractSSTableSimpleWriter.java:58)
at
org.apache.cassandra.io.sstable.SSTableSimpleUnsortedWriter$DiskWriter.run(SSTableSimpleUnsortedWriter.java:227)
[2015-05-01 16:09:59,144] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - LEAK DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@3cca0ac2) to class
org.apache.cassandra.io.util.SafeMemory$MemoryTidy@499043670:Memory@[7f87f8039940..7f87f8039c60)
was not released before the reference was garbage collected
[2015-05-01 16:09:59,144] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - Allocate trace
org.apache.cassandra.utils.concurrent.Ref$State@3cca0ac2:
Thread[Thread-2,5,main]
at
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[ https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523310#comment-14523310 ] Alan Boudreault commented on CASSANDRA-9240: oh right. Sorry, I had forgotten I tested that a few days ago. Performance issue after a restart - Key: CASSANDRA-9240 URL: https://issues.apache.org/jira/browse/CASSANDRA-9240 Project: Cassandra Issue Type: Bug Reporter: Alan Boudreault Assignee: T Jake Luciani Priority: Minor Fix For: 3.x Attachments: Cassandra.snapshots.zip, cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log, cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log, cassandra_trunk-clientrequest-read.log, cassandra_trunk.log, cassandra_trunk_no_restart-clientrequest-read.log, cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, runs.log, trace_query.cql I have noticed a performance issue while I was working on compaction perf tests for CASSANDRA-7409. The performance for my use case is very bad after a restart. It is mostly a read performance issue but not strictly. I have attached my use case (see run_issue.sh and issue.yaml) and all test logs for 2.1.4, 2.1.5 and trunk: * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k ops/second and ~2-2.5k of read latency. * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency. * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5. EDIT: branch cassandra-2.1 is OK. I can help to bisect and/or profile on Monday if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9276) StreamingTransferTest fails under test-compression due to bad assertion
[
https://issues.apache.org/jira/browse/CASSANDRA-9276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523327#comment-14523327
]
T Jake Luciani commented on CASSANDRA-9276:
---
You're right. this should be assert cis.getTotalCompressedBytesRead()
totalSize
StreamingTransferTest fails under test-compression due to bad assertion
---
Key: CASSANDRA-9276
URL: https://issues.apache.org/jira/browse/CASSANDRA-9276
Project: Cassandra
Issue Type: Test
Reporter: Ariel Weisberg
Assignee: Ariel Weisberg
Fix For: 3.x
https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/streaming/compress/CompressedStreamReader.java#L85
{noformat}
assert in.getBytesRead() totalSize;
{noformat}
My guess is that total size is the compressed size, not the uncompressed
size. Remove the assertion and the test passes.
Total size is calculated with
{noformat}
long size = 0;
// calculate total length of transferring chunks
for (CompressionMetadata.Chunk chunk : compressionInfo.chunks)
size += chunk.length + 4; // 4 bytes for CRC
return size;
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[2/2] cassandra git commit: Merge branch 'cassandra-2.0' into cassandra-2.1
Merge branch 'cassandra-2.0' into cassandra-2.1 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/33c59132 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/33c59132 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/33c59132 Branch: refs/heads/cassandra-2.1 Commit: 33c59132f730599859407a8460f1433e36f83fdf Parents: 5fcf8df 1e35fa4 Author: Aleksey Yeschenko alek...@apache.org Authored: Fri May 1 19:01:00 2015 +0300 Committer: Aleksey Yeschenko alek...@apache.org Committed: Fri May 1 19:01:00 2015 +0300 -- CHANGES.txt | 5 + .../org/apache/cassandra/db/index/SecondaryIndex.java | 10 -- .../apache/cassandra/db/index/SecondaryIndexManager.java | 2 +- .../org/apache/cassandra/db/ColumnFamilyStoreTest.java| 10 +++--- .../cassandra/db/index/PerRowSecondaryIndexTest.java | 8 5 files changed, 13 insertions(+), 22 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/33c59132/CHANGES.txt -- diff --cc CHANGES.txt index 55b474b,2dabbf9..b0e4fc6 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,101 -1,13 +1,103 @@@ -2.0.15: +2.1.6 + * Reduce runWithCompactionsDisabled poll interval to 1ms (CASSANDRA-9271) + * Fix PITR commitlog replay (CASSANDRA-9195) + * GCInspector logs very different times (CASSANDRA-9124) + * Fix deleting from an empty list (CASSANDRA-9198) + * Update tuple and collection types that use a user-defined type when that UDT + is modified (CASSANDRA-9148, CASSANDRA-9192) +Merged from 2.0: + * Fix index selection during rebuild with certain table layouts (CASSANDRA-9281) * Fix partition-level-delete-only workload accounting (CASSANDRA-9194) * Allow scrub to handle corrupted compressed chunks (CASSANDRA-9140) * Fix assertion error when resetlocalschema is run during repair (CASSANDRA-9249) * Disable single sstable tombstone compactions for DTCS by default (CASSANDRA-9234) - * Do more agressive ttl expiration checks to be able to - drop more sstables (CASSANDRA-8243) * IncomingTcpConnection thread is not named (CASSANDRA-9262) * Close incoming connections when MessagingService is stopped (CASSANDRA-9238) + ++ +2.1.5 + * Re-add deprecated cold_reads_to_omit param for backwards compat (CASSANDRA-9203) + * Make anticompaction visible in compactionstats (CASSANDRA-9098) + * Improve nodetool getendpoints documentation about the partition + key parameter (CASSANDRA-6458) + * Don't check other keyspaces for schema changes when an user-defined + type is altered (CASSANDRA-9187) + * Allow takeColumnFamilySnapshot to take a list of tables (CASSANDRA-8348) + * Limit major sstable operations to their canonical representation (CASSANDRA-8669) + * cqlsh: Add tests for INSERT and UPDATE tab completion (CASSANDRA-9125) + * cqlsh: quote column names when needed in COPY FROM inserts (CASSANDRA-9080) + * Add generate-idea-files target to build.xml (CASSANDRA-9123) + * Do not load read meter for offline operations (CASSANDRA-9082) + * cqlsh: Make CompositeType data readable (CASSANDRA-8919) + * cqlsh: Fix display of triggers (CASSANDRA-9081) + * Fix NullPointerException when deleting or setting an element by index on + a null list collection (CASSANDRA-9077) + * Buffer bloom filter serialization (CASSANDRA-9066) + * Fix anti-compaction target bloom filter size (CASSANDRA-9060) + * Make FROZEN and TUPLE unreserved keywords in CQL (CASSANDRA-9047) + * Prevent AssertionError from SizeEstimatesRecorder (CASSANDRA-9034) + * Avoid overwriting index summaries for sstables with an older format that + does not support downsampling; rebuild summaries on startup when this + is detected (CASSANDRA-8993) + * Fix potential data loss in CompressedSequentialWriter (CASSANDRA-8949) + * Make PasswordAuthenticator number of hashing rounds configurable (CASSANDRA-8085) + * Fix AssertionError when binding nested collections in DELETE (CASSANDRA-8900) + * Check for overlap with non-early sstables in LCS (CASSANDRA-8739) + * Only calculate max purgable timestamp if we have to (CASSANDRA-8914) + * (cqlsh) Greatly improve performance of COPY FROM (CASSANDRA-8225) + * IndexSummary effectiveIndexInterval is now a guideline, not a rule (CASSANDRA-8993) + * Use correct bounds for page cache eviction of compressed files (CASSANDRA-8746) + * SSTableScanner enforces its bounds (CASSANDRA-8946) + * Cleanup cell equality (CASSANDRA-8947) + * Introduce intra-cluster message coalescing (CASSANDRA-8692) + * DatabaseDescriptor throws NPE when rpc_interface is used (CASSANDRA-8839) + * Don't check if an sstable is live for offline compactions (CASSANDRA-8841) + * Don't
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[ https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523367#comment-14523367 ] T Jake Luciani commented on CASSANDRA-9240: --- I've gone back to CASSANDRA-8464 and I don't see this issue. Going to start bisecting. Performance issue after a restart - Key: CASSANDRA-9240 URL: https://issues.apache.org/jira/browse/CASSANDRA-9240 Project: Cassandra Issue Type: Bug Reporter: Alan Boudreault Assignee: T Jake Luciani Priority: Minor Fix For: 3.x Attachments: Cassandra.snapshots.zip, cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log, cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log, cassandra_trunk-clientrequest-read.log, cassandra_trunk.log, cassandra_trunk_no_restart-clientrequest-read.log, cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, runs.log, trace_query.cql I have noticed a performance issue while I was working on compaction perf tests for CASSANDRA-7409. The performance for my use case is very bad after a restart. It is mostly a read performance issue but not strictly. I have attached my use case (see run_issue.sh and issue.yaml) and all test logs for 2.1.4, 2.1.5 and trunk: * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k ops/second and ~2-2.5k of read latency. * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency. * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5. EDIT: branch cassandra-2.1 is OK. I can help to bisect and/or profile on Monday if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-9283) Deprecate unlogged batches
Jonathan Ellis created CASSANDRA-9283: - Summary: Deprecate unlogged batches Key: CASSANDRA-9283 URL: https://issues.apache.org/jira/browse/CASSANDRA-9283 Project: Cassandra Issue Type: Bug Reporter: Jonathan Ellis Assignee: T Jake Luciani Fix For: 3.0 Officially mark unlogged batches deprecated. Note that the main good use case for unlogged batches, of multiple updates in a single partition key, is free when done as a logged batch. So really unlogged batches mainly serve as a honeypot to trick new users into abusing them in misguided bulk loading attempts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[ https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523287#comment-14523287 ] Alan Boudreault commented on CASSANDRA-9240: Is there a way to by-pass this direct buffer slowdown? So I could restart my compaction tests. Performance issue after a restart - Key: CASSANDRA-9240 URL: https://issues.apache.org/jira/browse/CASSANDRA-9240 Project: Cassandra Issue Type: Bug Reporter: Alan Boudreault Assignee: T Jake Luciani Priority: Minor Fix For: 3.x Attachments: Cassandra.snapshots.zip, cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log, cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log, cassandra_trunk-clientrequest-read.log, cassandra_trunk.log, cassandra_trunk_no_restart-clientrequest-read.log, cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, runs.log, trace_query.cql I have noticed a performance issue while I was working on compaction perf tests for CASSANDRA-7409. The performance for my use case is very bad after a restart. It is mostly a read performance issue but not strictly. I have attached my use case (see run_issue.sh and issue.yaml) and all test logs for 2.1.4, 2.1.5 and trunk: * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k ops/second and ~2-2.5k of read latency. * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency. * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5. EDIT: branch cassandra-2.1 is OK. I can help to bisect and/or profile on Monday if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9194) Delete-only workloads crash Cassandra
[
https://issues.apache.org/jira/browse/CASSANDRA-9194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523336#comment-14523336
]
Jim Witschey commented on CASSANDRA-9194:
-
Awesome, thanks!
For posterity: The dtest for this was committed
[here|https://github.com/riptano/cassandra-dtest/pull/236] and the metrics used
were corrected [here|https://github.com/riptano/cassandra-dtest/pull/240].
Delete-only workloads crash Cassandra
-
Key: CASSANDRA-9194
URL: https://issues.apache.org/jira/browse/CASSANDRA-9194
Project: Cassandra
Issue Type: Bug
Environment: 2.0.14
Reporter: Robert Wille
Assignee: Benedict
Fix For: 2.0.x
Attachments: 9194.2.txt, 9194.txt
The size of a tombstone is not properly accounted for in the memtable. A
memtable which has only tombstones will never get flushed. It will grow until
the JVM runs out of memory. The following program easily demonstrates the
problem.
{code}
Cluster.Builder builder = Cluster.builder();
Cluster c =
builder.addContactPoints(cas121.devf3.com).build();
Session s = c.connect();
s.execute(CREATE KEYSPACE IF NOT EXISTS test WITH replication
= { 'class' : 'SimpleStrategy', 'replication_factor' : 1 });
s.execute(CREATE TABLE IF NOT EXISTS test.test(id INT PRIMARY
KEY));
PreparedStatement stmt = s.prepare(DELETE FROM test.test WHERE
id = :id);
int id = 0;
while (true)
{
s.execute(stmt.bind(id));
id++;
}{code}
This program should run forever, but eventually Cassandra runs out of heap
and craps out. You needn't wait for Cassandra to crash. If you run nodetool
cfstats test.test while it is running, you'll see Memtable cell count grow,
but Memtable data size will remain 0.
This issue was fixed once before. I received a patch for version 2.0.5 (I
believe), which contained the fix, but the fix has apparently been lost,
because it is clearly broken, and I don't see the fix in the change logs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-9285) LEAK DETECTED in sstwriter
Pierre N. created CASSANDRA-9285:
Summary: LEAK DETECTED in sstwriter
Key: CASSANDRA-9285
URL: https://issues.apache.org/jira/browse/CASSANDRA-9285
Project: Cassandra
Issue Type: Bug
Reporter: Pierre N.
I think new IndexWriter iWriter in SSTableWriter is not correctly closed in
SSTableWriter.close() (at least, iWriter.summary is not closed)
reproduce bug :
{code}
public static void main(String[] args) throws Exception {
System.setProperty(cassandra.debugrefcount,true);
String ks = ks1;
String table = t1;
String schema = CREATE TABLE + ks + . + table + (a1 INT, PRIMARY
KEY (a1));;
String insert = INSERT INTO + ks + . + table + (a1) VALUES(?);;
File dir = new File(/var/tmp/ + ks + / + table);
dir.mkdirs();
CQLSSTableWriter writer =
CQLSSTableWriter.builder().forTable(schema).using(insert).inDirectory(dir).build();
writer.addRow(1);
writer.close();
writer = null;
Thread.sleep(1000);System.gc();
Thread.sleep(1000);System.gc();
}
{code}
{quote}
[2015-05-01 16:09:59,139] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - LEAK DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@79fa9da9) to class
org.apache.cassandra.io.util.SafeMemory$MemoryTidy@2053866990:Memory@[7f87f8043b20..7f87f8043b48)
was not released before the reference was garbage collected
[2015-05-01 16:09:59,143] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - Allocate trace
org.apache.cassandra.utils.concurrent.Ref$State@79fa9da9:
Thread[Thread-2,5,main]
at java.lang.Thread.getStackTrace(Thread.java:1552)
at org.apache.cassandra.utils.concurrent.Ref$Debug.init(Ref.java:200)
at org.apache.cassandra.utils.concurrent.Ref$State.init(Ref.java:133)
at org.apache.cassandra.utils.concurrent.Ref.init(Ref.java:60)
at org.apache.cassandra.io.util.SafeMemory.init(SafeMemory.java:32)
at
org.apache.cassandra.io.util.SafeMemoryWriter.init(SafeMemoryWriter.java:33)
at
org.apache.cassandra.io.sstable.IndexSummaryBuilder.init(IndexSummaryBuilder.java:111)
at
org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.init(SSTableWriter.java:576)
at
org.apache.cassandra.io.sstable.SSTableWriter.init(SSTableWriter.java:140)
at
org.apache.cassandra.io.sstable.AbstractSSTableSimpleWriter.getWriter(AbstractSSTableSimpleWriter.java:58)
at
org.apache.cassandra.io.sstable.SSTableSimpleUnsortedWriter$DiskWriter.run(SSTableSimpleUnsortedWriter.java:227)
[2015-05-01 16:09:59,144] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - LEAK DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@664382e3) to class
org.apache.cassandra.io.util.SafeMemory$MemoryTidy@899100784:Memory@[7f87f8043990..7f87f8043994)
was not released before the reference was garbage collected
[2015-05-01 16:09:59,144] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - Allocate trace
org.apache.cassandra.utils.concurrent.Ref$State@664382e3:
Thread[Thread-2,5,main]
at java.lang.Thread.getStackTrace(Thread.java:1552)
at org.apache.cassandra.utils.concurrent.Ref$Debug.init(Ref.java:200)
at org.apache.cassandra.utils.concurrent.Ref$State.init(Ref.java:133)
at org.apache.cassandra.utils.concurrent.Ref.init(Ref.java:60)
at org.apache.cassandra.io.util.SafeMemory.init(SafeMemory.java:32)
at
org.apache.cassandra.io.util.SafeMemoryWriter.init(SafeMemoryWriter.java:33)
at
org.apache.cassandra.io.sstable.IndexSummaryBuilder.init(IndexSummaryBuilder.java:110)
at
org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.init(SSTableWriter.java:576)
at
org.apache.cassandra.io.sstable.SSTableWriter.init(SSTableWriter.java:140)
at
org.apache.cassandra.io.sstable.AbstractSSTableSimpleWriter.getWriter(AbstractSSTableSimpleWriter.java:58)
at
org.apache.cassandra.io.sstable.SSTableSimpleUnsortedWriter$DiskWriter.run(SSTableSimpleUnsortedWriter.java:227)
[2015-05-01 16:09:59,144] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - LEAK DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@3cca0ac2) to class
org.apache.cassandra.io.util.SafeMemory$MemoryTidy@499043670:Memory@[7f87f8039940..7f87f8039c60)
was not released before the reference was garbage collected
[2015-05-01 16:09:59,144] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - Allocate trace
org.apache.cassandra.utils.concurrent.Ref$State@3cca0ac2:
Thread[Thread-2,5,main]
at java.lang.Thread.getStackTrace(Thread.java:1552)
at
[jira] [Updated] (CASSANDRA-8915) Improve MergeIterator performance
[
https://issues.apache.org/jira/browse/CASSANDRA-8915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
T Jake Luciani updated CASSANDRA-8915:
--
Fix Version/s: 3.x
Improve MergeIterator performance
-
Key: CASSANDRA-8915
URL: https://issues.apache.org/jira/browse/CASSANDRA-8915
Project: Cassandra
Issue Type: Improvement
Reporter: Branimir Lambov
Assignee: Branimir Lambov
Priority: Minor
Fix For: 3.x
The implementation of {{MergeIterator}} uses a priority queue and applies a
pair of {{poll}}+{{add}} operations for every item in the resulting sequence.
This is quite inefficient as {{poll}} necessarily applies at least {{log N}}
comparisons (up to {{2log N}}), and {{add}} often requires another {{log N}},
for example in the case where the inputs largely don't overlap (where {{N}}
is the number of iterators being merged).
This can easily be replaced with a simple custom structure that can perform
replacement of the top of the queue in a single step, which will very often
complete after a couple of comparisons and in the worst case scenarios will
match the complexity of the current implementation.
This should significantly improve merge performance for iterators with
limited overlap (e.g. levelled compaction).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9284) support joins if partition key is the same in both tables
[ https://issues.apache.org/jira/browse/CASSANDRA-9284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523251#comment-14523251 ] Ryan Svihla commented on CASSANDRA-9284: This could add some flexibility to data modeling practices and would allow one giant table to be split up into smaller tables. One would still have to be mindful of the total retrieval size in a way we do not have to consider now, but this would I believe help people new to Cassandra still keep their table models conceptually broken up into smaller easier to understand pieces. support joins if partition key is the same in both tables - Key: CASSANDRA-9284 URL: https://issues.apache.org/jira/browse/CASSANDRA-9284 Project: Cassandra Issue Type: Improvement Reporter: Jon Haddad Priority: Minor Based off this conversation: https://mail-archives.apache.org/mod_mbox/cassandra-dev/201505.mbox/%3CCACUnPaAfJqU%2B86fFd4S6MS7Wv0KhpT_vavdkvDS%2Bm4Madi8_cg%40mail.gmail.com%3E It would be nice to have the flexibility of joins if we knew the query could be satisfied on a single node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
cassandra git commit: Fix index selection during rebuild with certain table layouts
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.0 f0c7a6f03 - 1e35fa4b5
Fix index selection during rebuild with certain table layouts
patch by Sam Tunnicliffe; reviewed by Jeremiah Jordan for CASSANDRA-9281
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1e35fa4b
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1e35fa4b
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1e35fa4b
Branch: refs/heads/cassandra-2.0
Commit: 1e35fa4b5a9afb4f37147a7a932a7892f2f450c1
Parents: f0c7a6f
Author: Sam Tunnicliffe s...@beobal.com
Authored: Fri May 1 10:17:39 2015 +0100
Committer: Aleksey Yeschenko alek...@apache.org
Committed: Fri May 1 18:55:08 2015 +0300
--
CHANGES.txt | 3 +++
.../apache/cassandra/db/index/SecondaryIndexManager.java | 2 +-
.../org/apache/cassandra/db/ColumnFamilyStoreTest.java| 10 +++---
.../cassandra/db/index/PerRowSecondaryIndexTest.java | 9 +
4 files changed, 12 insertions(+), 12 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/1e35fa4b/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index dfdad51..2dabbf9 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.0.15:
+ * Fix index selection during rebuild with certain table layouts
(CASSANDRA-9281)
* Fix partition-level-delete-only workload accounting (CASSANDRA-9194)
* Allow scrub to handle corrupted compressed chunks (CASSANDRA-9140)
* Fix assertion error when resetlocalschema is run during repair
(CASSANDRA-9249)
@@ -57,9 +58,11 @@
* Fix MT mismatch between empty and GC-able data (CASSANDRA-8979)
* Fix incorrect validation when snapshotting single table (CASSANDRA-8056)
+
2.0.14
* Bind JMX to localhost unless explicitly configured otherwise
(CASSANDRA-9085)
+
2.0.13:
* Add offline tool to relevel sstables (CASSANDRA-8301)
* Preserve stream ID for more protocol errors (CASSANDRA-8848)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/1e35fa4b/src/java/org/apache/cassandra/db/index/SecondaryIndexManager.java
--
diff --git a/src/java/org/apache/cassandra/db/index/SecondaryIndexManager.java
b/src/java/org/apache/cassandra/db/index/SecondaryIndexManager.java
index 1db7de6..d5e88d0 100644
--- a/src/java/org/apache/cassandra/db/index/SecondaryIndexManager.java
+++ b/src/java/org/apache/cassandra/db/index/SecondaryIndexManager.java
@@ -589,7 +589,7 @@ public class SecondaryIndexManager
{
for (ColumnDefinition column : baseCfs.metadata.allColumns())
{
-if (candidate.indexes(column.name))
+if (candidate.getColumnDefs().contains(column))
{
filtered.add(candidate.getIndexName());
break;
http://git-wip-us.apache.org/repos/asf/cassandra/blob/1e35fa4b/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
b/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
index 8f4a18f..001f838 100644
--- a/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
+++ b/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
@@ -2177,14 +2177,16 @@ public class ColumnFamilyStoreTest extends SchemaLoader
@Test
public void testRebuildSecondaryIndex() throws IOException
{
+ByteBuffer indexedColumnName = ByteBufferUtil.bytes(indexed);
RowMutation rm;
rm = new RowMutation(PerRowSecondaryIndex,
ByteBufferUtil.bytes(k1));
-rm.add(Indexed1, ByteBufferUtil.bytes(indexed),
ByteBufferUtil.bytes(foo), 1);
+rm.add(Indexed1, indexedColumnName, ByteBufferUtil.bytes(foo), 1);
rm.apply();
assertTrue(Arrays.equals(k1.getBytes(),
PerRowSecondaryIndexTest.TestIndex.LAST_INDEXED_KEY.array()));
-
Keyspace.open(PerRowSecondaryIndex).getColumnFamilyStore(Indexed1).forceBlockingFlush();
+ColumnFamilyStore cfs =
Keyspace.open(PerRowSecondaryIndex).getColumnFamilyStore(Indexed1);
+cfs.forceBlockingFlush();
PerRowSecondaryIndexTest.TestIndex.reset();
@@ -2193,7 +2195,9 @@ public class ColumnFamilyStoreTest extends SchemaLoader
PerRowSecondaryIndexTest.TestIndex.reset();
-PerRowSecondaryIndexTest.TestIndex.ACTIVE = false;
+ColumnDefinition indexedColumnDef =
cfs.metadata.getColumnDefinition(indexedColumnName);
+
cfs.indexManager.getIndexForColumn(indexedColumnName).getColumnDefs().remove(indexedColumnDef);
+
[1/2] cassandra git commit: Fix index selection during rebuild with certain table layouts
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.1 5fcf8dfad - 33c59132f
Fix index selection during rebuild with certain table layouts
patch by Sam Tunnicliffe; reviewed by Jeremiah Jordan for CASSANDRA-9281
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1e35fa4b
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1e35fa4b
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1e35fa4b
Branch: refs/heads/cassandra-2.1
Commit: 1e35fa4b5a9afb4f37147a7a932a7892f2f450c1
Parents: f0c7a6f
Author: Sam Tunnicliffe s...@beobal.com
Authored: Fri May 1 10:17:39 2015 +0100
Committer: Aleksey Yeschenko alek...@apache.org
Committed: Fri May 1 18:55:08 2015 +0300
--
CHANGES.txt | 3 +++
.../apache/cassandra/db/index/SecondaryIndexManager.java | 2 +-
.../org/apache/cassandra/db/ColumnFamilyStoreTest.java| 10 +++---
.../cassandra/db/index/PerRowSecondaryIndexTest.java | 9 +
4 files changed, 12 insertions(+), 12 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/1e35fa4b/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index dfdad51..2dabbf9 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.0.15:
+ * Fix index selection during rebuild with certain table layouts
(CASSANDRA-9281)
* Fix partition-level-delete-only workload accounting (CASSANDRA-9194)
* Allow scrub to handle corrupted compressed chunks (CASSANDRA-9140)
* Fix assertion error when resetlocalschema is run during repair
(CASSANDRA-9249)
@@ -57,9 +58,11 @@
* Fix MT mismatch between empty and GC-able data (CASSANDRA-8979)
* Fix incorrect validation when snapshotting single table (CASSANDRA-8056)
+
2.0.14
* Bind JMX to localhost unless explicitly configured otherwise
(CASSANDRA-9085)
+
2.0.13:
* Add offline tool to relevel sstables (CASSANDRA-8301)
* Preserve stream ID for more protocol errors (CASSANDRA-8848)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/1e35fa4b/src/java/org/apache/cassandra/db/index/SecondaryIndexManager.java
--
diff --git a/src/java/org/apache/cassandra/db/index/SecondaryIndexManager.java
b/src/java/org/apache/cassandra/db/index/SecondaryIndexManager.java
index 1db7de6..d5e88d0 100644
--- a/src/java/org/apache/cassandra/db/index/SecondaryIndexManager.java
+++ b/src/java/org/apache/cassandra/db/index/SecondaryIndexManager.java
@@ -589,7 +589,7 @@ public class SecondaryIndexManager
{
for (ColumnDefinition column : baseCfs.metadata.allColumns())
{
-if (candidate.indexes(column.name))
+if (candidate.getColumnDefs().contains(column))
{
filtered.add(candidate.getIndexName());
break;
http://git-wip-us.apache.org/repos/asf/cassandra/blob/1e35fa4b/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
b/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
index 8f4a18f..001f838 100644
--- a/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
+++ b/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
@@ -2177,14 +2177,16 @@ public class ColumnFamilyStoreTest extends SchemaLoader
@Test
public void testRebuildSecondaryIndex() throws IOException
{
+ByteBuffer indexedColumnName = ByteBufferUtil.bytes(indexed);
RowMutation rm;
rm = new RowMutation(PerRowSecondaryIndex,
ByteBufferUtil.bytes(k1));
-rm.add(Indexed1, ByteBufferUtil.bytes(indexed),
ByteBufferUtil.bytes(foo), 1);
+rm.add(Indexed1, indexedColumnName, ByteBufferUtil.bytes(foo), 1);
rm.apply();
assertTrue(Arrays.equals(k1.getBytes(),
PerRowSecondaryIndexTest.TestIndex.LAST_INDEXED_KEY.array()));
-
Keyspace.open(PerRowSecondaryIndex).getColumnFamilyStore(Indexed1).forceBlockingFlush();
+ColumnFamilyStore cfs =
Keyspace.open(PerRowSecondaryIndex).getColumnFamilyStore(Indexed1);
+cfs.forceBlockingFlush();
PerRowSecondaryIndexTest.TestIndex.reset();
@@ -2193,7 +2195,9 @@ public class ColumnFamilyStoreTest extends SchemaLoader
PerRowSecondaryIndexTest.TestIndex.reset();
-PerRowSecondaryIndexTest.TestIndex.ACTIVE = false;
+ColumnDefinition indexedColumnDef =
cfs.metadata.getColumnDefinition(indexedColumnName);
+
cfs.indexManager.getIndexForColumn(indexedColumnName).getColumnDefs().remove(indexedColumnDef);
+
[jira] [Created] (CASSANDRA-9284) support joins if partition key is the same in both tables
Jon Haddad created CASSANDRA-9284: - Summary: support joins if partition key is the same in both tables Key: CASSANDRA-9284 URL: https://issues.apache.org/jira/browse/CASSANDRA-9284 Project: Cassandra Issue Type: Improvement Reporter: Jon Haddad Priority: Minor Based off this conversation: https://mail-archives.apache.org/mod_mbox/cassandra-dev/201505.mbox/%3CCACUnPaAfJqU%2B86fFd4S6MS7Wv0KhpT_vavdkvDS%2Bm4Madi8_cg%40mail.gmail.com%3E It would be nice to have the flexibility of joins if we knew the query could be satisfied on a single node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9285) LEAK DETECTED in sstwriter
[
https://issues.apache.org/jira/browse/CASSANDRA-9285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre N. updated CASSANDRA-9285:
-
Description:
I think iWriter of SSTableWriter is not correctly closed in
SSTableWriter.close() (at least, iWriter.summary is not closed)
reproduce bug :
{code}
public static void main(String[] args) throws Exception {
System.setProperty(cassandra.debugrefcount,true);
String ks = ks1;
String table = t1;
String schema = CREATE TABLE + ks + . + table + (a1 INT, PRIMARY
KEY (a1));;
String insert = INSERT INTO + ks + . + table + (a1) VALUES(?);;
File dir = new File(/var/tmp/ + ks + / + table);
dir.mkdirs();
CQLSSTableWriter writer =
CQLSSTableWriter.builder().forTable(schema).using(insert).inDirectory(dir).build();
writer.addRow(1);
writer.close();
writer = null;
Thread.sleep(1000);System.gc();
Thread.sleep(1000);System.gc();
}
{code}
{quote}
[2015-05-01 16:09:59,139] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - LEAK DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@79fa9da9) to class
org.apache.cassandra.io.util.SafeMemory$MemoryTidy@2053866990:Memory@[7f87f8043b20..7f87f8043b48)
was not released before the reference was garbage collected
[2015-05-01 16:09:59,143] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - Allocate trace
org.apache.cassandra.utils.concurrent.Ref$State@79fa9da9:
Thread[Thread-2,5,main]
at java.lang.Thread.getStackTrace(Thread.java:1552)
at org.apache.cassandra.utils.concurrent.Ref$Debug.init(Ref.java:200)
at org.apache.cassandra.utils.concurrent.Ref$State.init(Ref.java:133)
at org.apache.cassandra.utils.concurrent.Ref.init(Ref.java:60)
at org.apache.cassandra.io.util.SafeMemory.init(SafeMemory.java:32)
at
org.apache.cassandra.io.util.SafeMemoryWriter.init(SafeMemoryWriter.java:33)
at
org.apache.cassandra.io.sstable.IndexSummaryBuilder.init(IndexSummaryBuilder.java:111)
at
org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.init(SSTableWriter.java:576)
at
org.apache.cassandra.io.sstable.SSTableWriter.init(SSTableWriter.java:140)
at
org.apache.cassandra.io.sstable.AbstractSSTableSimpleWriter.getWriter(AbstractSSTableSimpleWriter.java:58)
at
org.apache.cassandra.io.sstable.SSTableSimpleUnsortedWriter$DiskWriter.run(SSTableSimpleUnsortedWriter.java:227)
[2015-05-01 16:09:59,144] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - LEAK DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@664382e3) to class
org.apache.cassandra.io.util.SafeMemory$MemoryTidy@899100784:Memory@[7f87f8043990..7f87f8043994)
was not released before the reference was garbage collected
[2015-05-01 16:09:59,144] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - Allocate trace
org.apache.cassandra.utils.concurrent.Ref$State@664382e3:
Thread[Thread-2,5,main]
at java.lang.Thread.getStackTrace(Thread.java:1552)
at org.apache.cassandra.utils.concurrent.Ref$Debug.init(Ref.java:200)
at org.apache.cassandra.utils.concurrent.Ref$State.init(Ref.java:133)
at org.apache.cassandra.utils.concurrent.Ref.init(Ref.java:60)
at org.apache.cassandra.io.util.SafeMemory.init(SafeMemory.java:32)
at
org.apache.cassandra.io.util.SafeMemoryWriter.init(SafeMemoryWriter.java:33)
at
org.apache.cassandra.io.sstable.IndexSummaryBuilder.init(IndexSummaryBuilder.java:110)
at
org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.init(SSTableWriter.java:576)
at
org.apache.cassandra.io.sstable.SSTableWriter.init(SSTableWriter.java:140)
at
org.apache.cassandra.io.sstable.AbstractSSTableSimpleWriter.getWriter(AbstractSSTableSimpleWriter.java:58)
at
org.apache.cassandra.io.sstable.SSTableSimpleUnsortedWriter$DiskWriter.run(SSTableSimpleUnsortedWriter.java:227)
[2015-05-01 16:09:59,144] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - LEAK DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@3cca0ac2) to class
org.apache.cassandra.io.util.SafeMemory$MemoryTidy@499043670:Memory@[7f87f8039940..7f87f8039c60)
was not released before the reference was garbage collected
[2015-05-01 16:09:59,144] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - Allocate trace
org.apache.cassandra.utils.concurrent.Ref$State@3cca0ac2:
Thread[Thread-2,5,main]
at java.lang.Thread.getStackTrace(Thread.java:1552)
at org.apache.cassandra.utils.concurrent.Ref$Debug.init(Ref.java:200)
at org.apache.cassandra.utils.concurrent.Ref$State.init(Ref.java:133)
at
Git Push Summary
Repository: cassandra Updated Branches: refs/heads/cassandra-2.1.5 [deleted] 3c0a337eb
[jira] [Commented] (CASSANDRA-9282) Warn on unlogged batches
[ https://issues.apache.org/jira/browse/CASSANDRA-9282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523271#comment-14523271 ] T Jake Luciani commented on CASSANDRA-9282: --- Ideally these should be pushed to the clients/ or the drivers should warn on these. Warn on unlogged batches Key: CASSANDRA-9282 URL: https://issues.apache.org/jira/browse/CASSANDRA-9282 Project: Cassandra Issue Type: Bug Components: API Reporter: Jonathan Ellis Assignee: T Jake Luciani Fix For: 2.1.x At least until CASSANDRA-8303 is done and we can block them entirely, we should log a warning when unlogged batches across multiple partition keys are used. This could either be done by backporting NoSpamLogger and blindly logging every time, or we could add a threshold and warn when more than 10 keys are seen. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9285) LEAK DETECTED in sstwriter
[
https://issues.apache.org/jira/browse/CASSANDRA-9285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523289#comment-14523289
]
Pierre N. commented on CASSANDRA-9285:
--
Didn't installed 2.1.5 as a server yet.
Following may fix the issue
{code:none}
diff --git a/src/java/org/apache/cassandra/io/compress/CompressionMetadata.java
b/src/java/org/apache/cassandra/io/compress/CompressionMetadata.java
index 9ac2f89..1cd7956 100644
--- a/src/java/org/apache/cassandra/io/compress/CompressionMetadata.java
+++ b/src/java/org/apache/cassandra/io/compress/CompressionMetadata.java
@@ -409,6 +409,7 @@ public class CompressionMetadata
writeHeader(out, dataLength, chunks);
for (int i = 0 ; i count ; i++)
out.writeLong(offsets.getLong(i * 8L));
+offsets.close();
}
finally
{
diff --git a/src/java/org/apache/cassandra/io/sstable/SSTableWriter.java
b/src/java/org/apache/cassandra/io/sstable/SSTableWriter.java
index a39c134..9648636 100644
--- a/src/java/org/apache/cassandra/io/sstable/SSTableWriter.java
+++ b/src/java/org/apache/cassandra/io/sstable/SSTableWriter.java
@@ -486,6 +486,7 @@ public class SSTableWriter extends SSTable
case EARLY: case CLOSE: case NORMAL:
iwriter.close();
dataFile.close();
+iwriter.summary.close();
if (type == FinishType.CLOSE)
iwriter.bf.close();
}
{code}
LEAK DETECTED in sstwriter
--
Key: CASSANDRA-9285
URL: https://issues.apache.org/jira/browse/CASSANDRA-9285
Project: Cassandra
Issue Type: Bug
Reporter: Pierre N.
Fix For: 2.1.x
reproduce bug :
{code}
public static void main(String[] args) throws Exception {
System.setProperty(cassandra.debugrefcount,true);
String ks = ks1;
String table = t1;
String schema = CREATE TABLE + ks + . + table + (a1 INT,
PRIMARY KEY (a1));;
String insert = INSERT INTO + ks + . + table + (a1) VALUES(?);;
File dir = new File(/var/tmp/ + ks + / + table);
dir.mkdirs();
CQLSSTableWriter writer =
CQLSSTableWriter.builder().forTable(schema).using(insert).inDirectory(dir).build();
writer.addRow(1);
writer.close();
writer = null;
Thread.sleep(1000);System.gc();
Thread.sleep(1000);System.gc();
}
{code}
{quote}
[2015-05-01 16:09:59,139] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - LEAK DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@79fa9da9) to class
org.apache.cassandra.io.util.SafeMemory$MemoryTidy@2053866990:Memory@[7f87f8043b20..7f87f8043b48)
was not released before the reference was garbage collected
[2015-05-01 16:09:59,143] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - Allocate trace
org.apache.cassandra.utils.concurrent.Ref$State@79fa9da9:
Thread[Thread-2,5,main]
at java.lang.Thread.getStackTrace(Thread.java:1552)
at org.apache.cassandra.utils.concurrent.Ref$Debug.init(Ref.java:200)
at org.apache.cassandra.utils.concurrent.Ref$State.init(Ref.java:133)
at org.apache.cassandra.utils.concurrent.Ref.init(Ref.java:60)
at org.apache.cassandra.io.util.SafeMemory.init(SafeMemory.java:32)
at
org.apache.cassandra.io.util.SafeMemoryWriter.init(SafeMemoryWriter.java:33)
at
org.apache.cassandra.io.sstable.IndexSummaryBuilder.init(IndexSummaryBuilder.java:111)
at
org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.init(SSTableWriter.java:576)
at
org.apache.cassandra.io.sstable.SSTableWriter.init(SSTableWriter.java:140)
at
org.apache.cassandra.io.sstable.AbstractSSTableSimpleWriter.getWriter(AbstractSSTableSimpleWriter.java:58)
at
org.apache.cassandra.io.sstable.SSTableSimpleUnsortedWriter$DiskWriter.run(SSTableSimpleUnsortedWriter.java:227)
[2015-05-01 16:09:59,144] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - LEAK DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@664382e3) to class
org.apache.cassandra.io.util.SafeMemory$MemoryTidy@899100784:Memory@[7f87f8043990..7f87f8043994)
was not released before the reference was garbage collected
[2015-05-01 16:09:59,144] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - Allocate trace
org.apache.cassandra.utils.concurrent.Ref$State@664382e3:
Thread[Thread-2,5,main]
at java.lang.Thread.getStackTrace(Thread.java:1552)
at org.apache.cassandra.utils.concurrent.Ref$Debug.init(Ref.java:200)
at org.apache.cassandra.utils.concurrent.Ref$State.init(Ref.java:133)
at
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[ https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523288#comment-14523288 ] T Jake Luciani commented on CASSANDRA-9240: --- Use a larger heap? Performance issue after a restart - Key: CASSANDRA-9240 URL: https://issues.apache.org/jira/browse/CASSANDRA-9240 Project: Cassandra Issue Type: Bug Reporter: Alan Boudreault Assignee: T Jake Luciani Priority: Minor Fix For: 3.x Attachments: Cassandra.snapshots.zip, cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log, cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log, cassandra_trunk-clientrequest-read.log, cassandra_trunk.log, cassandra_trunk_no_restart-clientrequest-read.log, cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, runs.log, trace_query.cql I have noticed a performance issue while I was working on compaction perf tests for CASSANDRA-7409. The performance for my use case is very bad after a restart. It is mostly a read performance issue but not strictly. I have attached my use case (see run_issue.sh and issue.yaml) and all test logs for 2.1.4, 2.1.5 and trunk: * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k ops/second and ~2-2.5k of read latency. * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency. * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5. EDIT: branch cassandra-2.1 is OK. I can help to bisect and/or profile on Monday if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9279) Gossip (and mutations) lock up on startup
[
https://issues.apache.org/jira/browse/CASSANDRA-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philip Thompson updated CASSANDRA-9279:
---
Fix Version/s: 2.0.x
Gossip (and mutations) lock up on startup
-
Key: CASSANDRA-9279
URL: https://issues.apache.org/jira/browse/CASSANDRA-9279
Project: Cassandra
Issue Type: Bug
Reporter: Sebastian Estevez
Assignee: Brandon Williams
Fix For: 2.0.x
Attachments: Screen Shot 2015-04-30 at 4.41.57 PM.png
Cluster running 2.0.14.352 on EC2 - c3.4xl's
2 nodes out of 8 exhibited the following behavior
When starting up the node we noticed it was gray in OpsCenter. Other
monitoring tool showed it as up.
Turned out gossip tasks were piling up and we could see the following in the
system.log:
{code}
WARN [GossipTasks:1] 2015-04-30 20:22:29,512 Gossiper.java (line 671) Gossip
stage has 4270 pending tasks; skipping status check (no nodes will be marked
down)
WARN [GossipTasks:1] 2015-04-30 20:22:30,612 Gossiper.java (line 671) Gossip
stage has 4272 pending tasks; skipping status check (no nodes will be marked
down)
WARN [GossipTasks:1] 2015-04-30 20:22:31,713 Gossiper.java (line 671) Gossip
stage has 4273 pending tasks; skipping status check (no nodes will be marked
down)
...
{code}
and tpstats shows blocked tasks--gossip and mutations:
{code}
GossipStage 1 3904 29384 0
0
{code}
the CPU's are inactive (See attachment)
and dstat output:
{code}
You did not select any stats, using -cdngy by default.
total-cpu-usage -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read writ| recv send| in out | int csw
2 0 97 0 0 0|1324k 1381k| 0 0 | 0 0 |6252 5548
0 0 100 0 0 0| 064k| 42k 1017k| 0 0 |3075 2537
0 0 99 0 0 0| 0 8192B| 39k 794k| 0 0 |6999 7039
0 0 100 0 0 0| 0 0 | 39k 759k| 0 0 |3067 2726
0 0 99 0 0 0| 0 184k| 48k 1086k| 0 0 |4829 4178
0 0 99 0 0 0| 0 8192B| 34k 802k| 0 0 |1671 1240
0 0 100 0 0 0| 0 8192B| 48k 1067k| 0 0 |1878 1193
{code}
I managed to grab a thread dump:
https://gist.githubusercontent.com/anonymous/3b7b4698c32032603493/raw/read.md
and dmesg:
https://gist.githubusercontent.com/anonymous/5982b15337c9afbd5d49/raw/f3c2e4411b9d59e90f4615d93c7c1ad25922e170/read.md
Restarting the node solved the issue (it came up normally), we don't know
what is causing it but apparently (per the thread dump) gossip threads are
blocked writing the system keyspace and the writes waiting on the commitlog.
Gossip:
{code}
GossipStage:1 daemon prio=10 tid=0x7ffa23471800 nid=0xa13fa waiting on
condition [0x7ff9cbe26000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for 0x0005d3f50960 (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:964)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282)
at
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:731)
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351)
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:336)
at org.apache.cassandra.db.RowMutation.apply(RowMutation.java:211)
at
org.apache.cassandra.cql3.statements.ModificationStatement.executeInternal(ModificationStatement.java:709)
at
org.apache.cassandra.cql3.QueryProcessor.processInternal(QueryProcessor.java:208)
at
org.apache.cassandra.db.SystemKeyspace.updatePeerInfo(SystemKeyspace.java:379)
- locked 0x0005d3f41ed8 (a java.lang.Class for
org.apache.cassandra.db.SystemKeyspace)
at
org.apache.cassandra.service.StorageService.updatePeerInfo(StorageService.java:1414)
at
org.apache.cassandra.service.StorageService.handleStateNormal(StorageService.java:1524)
at
org.apache.cassandra.service.StorageService.onChange(StorageService.java:1350)
at
org.apache.cassandra.gms.Gossiper.doOnChangeNotifications(Gossiper.java:1083)
at org.apache.cassandra.gms.Gossiper.applyNewStates(Gossiper.java:1065)
at
org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:1023)
at
[jira] [Updated] (CASSANDRA-9279) Gossip (and mutations) lock up on startup
[
https://issues.apache.org/jira/browse/CASSANDRA-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philip Thompson updated CASSANDRA-9279:
---
Assignee: Brandon Williams
Gossip (and mutations) lock up on startup
-
Key: CASSANDRA-9279
URL: https://issues.apache.org/jira/browse/CASSANDRA-9279
Project: Cassandra
Issue Type: Bug
Reporter: Sebastian Estevez
Assignee: Brandon Williams
Fix For: 2.0.x
Attachments: Screen Shot 2015-04-30 at 4.41.57 PM.png
Cluster running 2.0.14.352 on EC2 - c3.4xl's
2 nodes out of 8 exhibited the following behavior
When starting up the node we noticed it was gray in OpsCenter. Other
monitoring tool showed it as up.
Turned out gossip tasks were piling up and we could see the following in the
system.log:
{code}
WARN [GossipTasks:1] 2015-04-30 20:22:29,512 Gossiper.java (line 671) Gossip
stage has 4270 pending tasks; skipping status check (no nodes will be marked
down)
WARN [GossipTasks:1] 2015-04-30 20:22:30,612 Gossiper.java (line 671) Gossip
stage has 4272 pending tasks; skipping status check (no nodes will be marked
down)
WARN [GossipTasks:1] 2015-04-30 20:22:31,713 Gossiper.java (line 671) Gossip
stage has 4273 pending tasks; skipping status check (no nodes will be marked
down)
...
{code}
and tpstats shows blocked tasks--gossip and mutations:
{code}
GossipStage 1 3904 29384 0
0
{code}
the CPU's are inactive (See attachment)
and dstat output:
{code}
You did not select any stats, using -cdngy by default.
total-cpu-usage -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read writ| recv send| in out | int csw
2 0 97 0 0 0|1324k 1381k| 0 0 | 0 0 |6252 5548
0 0 100 0 0 0| 064k| 42k 1017k| 0 0 |3075 2537
0 0 99 0 0 0| 0 8192B| 39k 794k| 0 0 |6999 7039
0 0 100 0 0 0| 0 0 | 39k 759k| 0 0 |3067 2726
0 0 99 0 0 0| 0 184k| 48k 1086k| 0 0 |4829 4178
0 0 99 0 0 0| 0 8192B| 34k 802k| 0 0 |1671 1240
0 0 100 0 0 0| 0 8192B| 48k 1067k| 0 0 |1878 1193
{code}
I managed to grab a thread dump:
https://gist.githubusercontent.com/anonymous/3b7b4698c32032603493/raw/read.md
and dmesg:
https://gist.githubusercontent.com/anonymous/5982b15337c9afbd5d49/raw/f3c2e4411b9d59e90f4615d93c7c1ad25922e170/read.md
Restarting the node solved the issue (it came up normally), we don't know
what is causing it but apparently (per the thread dump) gossip threads are
blocked writing the system keyspace and the writes waiting on the commitlog.
Gossip:
{code}
GossipStage:1 daemon prio=10 tid=0x7ffa23471800 nid=0xa13fa waiting on
condition [0x7ff9cbe26000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for 0x0005d3f50960 (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:964)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282)
at
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:731)
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351)
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:336)
at org.apache.cassandra.db.RowMutation.apply(RowMutation.java:211)
at
org.apache.cassandra.cql3.statements.ModificationStatement.executeInternal(ModificationStatement.java:709)
at
org.apache.cassandra.cql3.QueryProcessor.processInternal(QueryProcessor.java:208)
at
org.apache.cassandra.db.SystemKeyspace.updatePeerInfo(SystemKeyspace.java:379)
- locked 0x0005d3f41ed8 (a java.lang.Class for
org.apache.cassandra.db.SystemKeyspace)
at
org.apache.cassandra.service.StorageService.updatePeerInfo(StorageService.java:1414)
at
org.apache.cassandra.service.StorageService.handleStateNormal(StorageService.java:1524)
at
org.apache.cassandra.service.StorageService.onChange(StorageService.java:1350)
at
org.apache.cassandra.gms.Gossiper.doOnChangeNotifications(Gossiper.java:1083)
at org.apache.cassandra.gms.Gossiper.applyNewStates(Gossiper.java:1065)
at
org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:1023)
at
[jira] [Commented] (CASSANDRA-9279) Gossip (and mutations) lock up on startup
[
https://issues.apache.org/jira/browse/CASSANDRA-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523369#comment-14523369
]
Brandon Williams commented on CASSANDRA-9279:
-
All we really know here is that the gossip and mutation threads are blocked
trying to write mutations (to the system ks, in the case of gossip.) I suspect
something lower-level is wrong with the system, but this certainly isn't
related to gossip.
Gossip (and mutations) lock up on startup
-
Key: CASSANDRA-9279
URL: https://issues.apache.org/jira/browse/CASSANDRA-9279
Project: Cassandra
Issue Type: Bug
Reporter: Sebastian Estevez
Assignee: Brandon Williams
Fix For: 2.0.x
Attachments: Screen Shot 2015-04-30 at 4.41.57 PM.png
Cluster running 2.0.14.352 on EC2 - c3.4xl's
2 nodes out of 8 exhibited the following behavior
When starting up the node we noticed it was gray in OpsCenter. Other
monitoring tool showed it as up.
Turned out gossip tasks were piling up and we could see the following in the
system.log:
{code}
WARN [GossipTasks:1] 2015-04-30 20:22:29,512 Gossiper.java (line 671) Gossip
stage has 4270 pending tasks; skipping status check (no nodes will be marked
down)
WARN [GossipTasks:1] 2015-04-30 20:22:30,612 Gossiper.java (line 671) Gossip
stage has 4272 pending tasks; skipping status check (no nodes will be marked
down)
WARN [GossipTasks:1] 2015-04-30 20:22:31,713 Gossiper.java (line 671) Gossip
stage has 4273 pending tasks; skipping status check (no nodes will be marked
down)
...
{code}
and tpstats shows blocked tasks--gossip and mutations:
{code}
GossipStage 1 3904 29384 0
0
{code}
the CPU's are inactive (See attachment)
and dstat output:
{code}
You did not select any stats, using -cdngy by default.
total-cpu-usage -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read writ| recv send| in out | int csw
2 0 97 0 0 0|1324k 1381k| 0 0 | 0 0 |6252 5548
0 0 100 0 0 0| 064k| 42k 1017k| 0 0 |3075 2537
0 0 99 0 0 0| 0 8192B| 39k 794k| 0 0 |6999 7039
0 0 100 0 0 0| 0 0 | 39k 759k| 0 0 |3067 2726
0 0 99 0 0 0| 0 184k| 48k 1086k| 0 0 |4829 4178
0 0 99 0 0 0| 0 8192B| 34k 802k| 0 0 |1671 1240
0 0 100 0 0 0| 0 8192B| 48k 1067k| 0 0 |1878 1193
{code}
I managed to grab a thread dump:
https://gist.githubusercontent.com/anonymous/3b7b4698c32032603493/raw/read.md
and dmesg:
https://gist.githubusercontent.com/anonymous/5982b15337c9afbd5d49/raw/f3c2e4411b9d59e90f4615d93c7c1ad25922e170/read.md
Restarting the node solved the issue (it came up normally), we don't know
what is causing it but apparently (per the thread dump) gossip threads are
blocked writing the system keyspace and the writes waiting on the commitlog.
Gossip:
{code}
GossipStage:1 daemon prio=10 tid=0x7ffa23471800 nid=0xa13fa waiting on
condition [0x7ff9cbe26000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for 0x0005d3f50960 (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:964)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282)
at
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:731)
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351)
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:336)
at org.apache.cassandra.db.RowMutation.apply(RowMutation.java:211)
at
org.apache.cassandra.cql3.statements.ModificationStatement.executeInternal(ModificationStatement.java:709)
at
org.apache.cassandra.cql3.QueryProcessor.processInternal(QueryProcessor.java:208)
at
org.apache.cassandra.db.SystemKeyspace.updatePeerInfo(SystemKeyspace.java:379)
- locked 0x0005d3f41ed8 (a java.lang.Class for
org.apache.cassandra.db.SystemKeyspace)
at
org.apache.cassandra.service.StorageService.updatePeerInfo(StorageService.java:1414)
at
org.apache.cassandra.service.StorageService.handleStateNormal(StorageService.java:1524)
at
org.apache.cassandra.service.StorageService.onChange(StorageService.java:1350)
at
[jira] [Updated] (CASSANDRA-9279) Gossip (and mutations) lock up on startup
[
https://issues.apache.org/jira/browse/CASSANDRA-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Williams updated CASSANDRA-9279:
Assignee: (was: Brandon Williams)
Gossip (and mutations) lock up on startup
-
Key: CASSANDRA-9279
URL: https://issues.apache.org/jira/browse/CASSANDRA-9279
Project: Cassandra
Issue Type: Bug
Reporter: Sebastian Estevez
Fix For: 2.0.x
Attachments: Screen Shot 2015-04-30 at 4.41.57 PM.png
Cluster running 2.0.14.352 on EC2 - c3.4xl's
2 nodes out of 8 exhibited the following behavior
When starting up the node we noticed it was gray in OpsCenter. Other
monitoring tool showed it as up.
Turned out gossip tasks were piling up and we could see the following in the
system.log:
{code}
WARN [GossipTasks:1] 2015-04-30 20:22:29,512 Gossiper.java (line 671) Gossip
stage has 4270 pending tasks; skipping status check (no nodes will be marked
down)
WARN [GossipTasks:1] 2015-04-30 20:22:30,612 Gossiper.java (line 671) Gossip
stage has 4272 pending tasks; skipping status check (no nodes will be marked
down)
WARN [GossipTasks:1] 2015-04-30 20:22:31,713 Gossiper.java (line 671) Gossip
stage has 4273 pending tasks; skipping status check (no nodes will be marked
down)
...
{code}
and tpstats shows blocked tasks--gossip and mutations:
{code}
GossipStage 1 3904 29384 0
0
{code}
the CPU's are inactive (See attachment)
and dstat output:
{code}
You did not select any stats, using -cdngy by default.
total-cpu-usage -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read writ| recv send| in out | int csw
2 0 97 0 0 0|1324k 1381k| 0 0 | 0 0 |6252 5548
0 0 100 0 0 0| 064k| 42k 1017k| 0 0 |3075 2537
0 0 99 0 0 0| 0 8192B| 39k 794k| 0 0 |6999 7039
0 0 100 0 0 0| 0 0 | 39k 759k| 0 0 |3067 2726
0 0 99 0 0 0| 0 184k| 48k 1086k| 0 0 |4829 4178
0 0 99 0 0 0| 0 8192B| 34k 802k| 0 0 |1671 1240
0 0 100 0 0 0| 0 8192B| 48k 1067k| 0 0 |1878 1193
{code}
I managed to grab a thread dump:
https://gist.githubusercontent.com/anonymous/3b7b4698c32032603493/raw/read.md
and dmesg:
https://gist.githubusercontent.com/anonymous/5982b15337c9afbd5d49/raw/f3c2e4411b9d59e90f4615d93c7c1ad25922e170/read.md
Restarting the node solved the issue (it came up normally), we don't know
what is causing it but apparently (per the thread dump) gossip threads are
blocked writing the system keyspace and the writes waiting on the commitlog.
Gossip:
{code}
GossipStage:1 daemon prio=10 tid=0x7ffa23471800 nid=0xa13fa waiting on
condition [0x7ff9cbe26000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for 0x0005d3f50960 (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:964)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282)
at
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:731)
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351)
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:336)
at org.apache.cassandra.db.RowMutation.apply(RowMutation.java:211)
at
org.apache.cassandra.cql3.statements.ModificationStatement.executeInternal(ModificationStatement.java:709)
at
org.apache.cassandra.cql3.QueryProcessor.processInternal(QueryProcessor.java:208)
at
org.apache.cassandra.db.SystemKeyspace.updatePeerInfo(SystemKeyspace.java:379)
- locked 0x0005d3f41ed8 (a java.lang.Class for
org.apache.cassandra.db.SystemKeyspace)
at
org.apache.cassandra.service.StorageService.updatePeerInfo(StorageService.java:1414)
at
org.apache.cassandra.service.StorageService.handleStateNormal(StorageService.java:1524)
at
org.apache.cassandra.service.StorageService.onChange(StorageService.java:1350)
at
org.apache.cassandra.gms.Gossiper.doOnChangeNotifications(Gossiper.java:1083)
at org.apache.cassandra.gms.Gossiper.applyNewStates(Gossiper.java:1065)
at
org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:1023)
at
[jira] [Commented] (CASSANDRA-7066) Simplify (and unify) cleanup of compaction leftovers
[ https://issues.apache.org/jira/browse/CASSANDRA-7066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14522888#comment-14522888 ] Stefania commented on CASSANDRA-7066: - I am having doubts on whether we should eliminate temporary files entirely. There is a benefit in knowing if a file is ready for use just by looking at its file name, rather than relying on a transaction log file to find out. I am thinking of cases where the users might move files around, like bulk upload, or standalone operations. I feel a bit uneasy relying on users to also coy the transaction log files. So I would propose still keeping the tmp files, even though they are tracked by the transaction logs. Any thoughts? Simplify (and unify) cleanup of compaction leftovers Key: CASSANDRA-7066 URL: https://issues.apache.org/jira/browse/CASSANDRA-7066 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Benedict Assignee: Stefania Priority: Minor Labels: compaction Fix For: 3.x Currently we manage a list of in-progress compactions in a system table, which we use to cleanup incomplete compactions when we're done. The problem with this is that 1) it's a bit clunky (and leaves us in positions where we can unnecessarily cleanup completed files, or conversely not cleanup files that have been superceded); and 2) it's only used for a regular compaction - no other compaction types are guarded in the same way, so can result in duplication if we fail before deleting the replacements. I'd like to see each sstable store in its metadata its direct ancestors, and on startup we simply delete any sstables that occur in the union of all ancestor sets. This way as soon as we finish writing we're capable of cleaning up any leftovers, so we never get duplication. It's also much easier to reason about. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[2/2] cassandra git commit: Merge branch 'cassandra-2.1' into trunk
Merge branch 'cassandra-2.1' into trunk
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/6404e015
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/6404e015
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/6404e015
Branch: refs/heads/trunk
Commit: 6404e015ffe916fe24bacb3772b388e633af1261
Parents: c799a98 f43efaa
Author: Aleksey Yeschenko alek...@apache.org
Authored: Fri May 1 19:50:06 2015 +0300
Committer: Aleksey Yeschenko alek...@apache.org
Committed: Fri May 1 19:50:06 2015 +0300
--
.../db/commitlog/CommitLogArchiver.java | 2 +-
.../db/commitlog/CommitLogReplayer.java | 11 +-
.../db/commitlog/CommitLogStressTest.java | 3 +-
.../db/RecoveryManagerTruncateTest.java | 181 +++
4 files changed, 149 insertions(+), 48 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/6404e015/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/6404e015/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java
--
diff --cc src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java
index f6d1cc4,57f4b90..23ee9e3
--- a/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java
+++ b/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java
@@@ -271,17 -264,22 +271,17 @@@ public class CommitLogReplaye
public void recover(File file) throws IOException
{
-logger.info(Replaying {}, file.getPath());
CommitLogDescriptor desc =
CommitLogDescriptor.fromFileName(file.getName());
-final long segmentId = desc.id;
-logger.info(Replaying {} (CL version {}, messaging version {}),
-file.getPath(),
-desc.version,
-desc.getMessagingVersion());
RandomAccessReader reader = RandomAccessReader.open(new
File(file.getAbsolutePath()));
-
try
{
-assert reader.length() = Integer.MAX_VALUE;
-int offset = getStartOffset(segmentId, desc.version);
-if (offset 0)
+if (desc.version CommitLogDescriptor.VERSION_21)
{
-logger.debug(skipping replay of fully-flushed {}, file);
+if (logAndCheckIfShouldSkip(file, desc))
+return;
+if (globalPosition.segment == desc.id)
+reader.seek(globalPosition.position);
- replaySyncSection(reader, -1, desc, replayFilter);
++replaySyncSection(reader, -1, desc);
return;
}
@@@ -365,208 -411,71 +365,207 @@@
continue;
}
- if (!replaySyncSection(sectionReader, replayEnd, desc,
replayFilter))
-if (logger.isDebugEnabled())
-logger.debug(replaying mutation for {}.{}: {},
mutation.getKeyspaceName(), ByteBufferUtil.bytesToHex(mutation.key()), { +
StringUtils.join(mutation.getColumnFamilies().iterator(), , ) + });
++if (!replaySyncSection(sectionReader, replayEnd, desc))
+break;
+}
+}
+finally
+{
+FileUtils.closeQuietly(reader);
+logger.info(Finished reading {}, file);
+}
+}
-final long entryLocation = reader.getFilePointer();
-Runnable runnable = new WrappedRunnable()
-{
-public void runMayThrow() throws IOException
-{
-if
(Schema.instance.getKSMetaData(mutation.getKeyspaceName()) == null)
-return;
-if (pointInTimeExceeded(mutation))
-return;
-
-final Keyspace keyspace =
Keyspace.open(mutation.getKeyspaceName());
-
-// Rebuild the mutation, omitting column families
that
-//a) the user has requested that we ignore,
-//b) have already been flushed,
-// or c) are part of a cf that was dropped.
-// Keep in mind that the cf.name() is suspect. do
every thing based on the cfid instead.
-Mutation newMutation = null;
-for (ColumnFamily columnFamily :
replayFilter.filter(mutation))
-{
-
[jira] [Commented] (CASSANDRA-9279) Gossip (and mutations) lock up on startup
[
https://issues.apache.org/jira/browse/CASSANDRA-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523435#comment-14523435
]
Benedict commented on CASSANDRA-9279:
-
The Commit-Log-Allocator isn't running.
Gossip (and mutations) lock up on startup
-
Key: CASSANDRA-9279
URL: https://issues.apache.org/jira/browse/CASSANDRA-9279
Project: Cassandra
Issue Type: Bug
Reporter: Sebastian Estevez
Fix For: 2.0.x
Attachments: Screen Shot 2015-04-30 at 4.41.57 PM.png
Cluster running 2.0.14.352 on EC2 - c3.4xl's
2 nodes out of 8 exhibited the following behavior
When starting up the node we noticed it was gray in OpsCenter. Other
monitoring tool showed it as up.
Turned out gossip tasks were piling up and we could see the following in the
system.log:
{code}
WARN [GossipTasks:1] 2015-04-30 20:22:29,512 Gossiper.java (line 671) Gossip
stage has 4270 pending tasks; skipping status check (no nodes will be marked
down)
WARN [GossipTasks:1] 2015-04-30 20:22:30,612 Gossiper.java (line 671) Gossip
stage has 4272 pending tasks; skipping status check (no nodes will be marked
down)
WARN [GossipTasks:1] 2015-04-30 20:22:31,713 Gossiper.java (line 671) Gossip
stage has 4273 pending tasks; skipping status check (no nodes will be marked
down)
...
{code}
and tpstats shows blocked tasks--gossip and mutations:
{code}
GossipStage 1 3904 29384 0
0
{code}
the CPU's are inactive (See attachment)
and dstat output:
{code}
You did not select any stats, using -cdngy by default.
total-cpu-usage -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read writ| recv send| in out | int csw
2 0 97 0 0 0|1324k 1381k| 0 0 | 0 0 |6252 5548
0 0 100 0 0 0| 064k| 42k 1017k| 0 0 |3075 2537
0 0 99 0 0 0| 0 8192B| 39k 794k| 0 0 |6999 7039
0 0 100 0 0 0| 0 0 | 39k 759k| 0 0 |3067 2726
0 0 99 0 0 0| 0 184k| 48k 1086k| 0 0 |4829 4178
0 0 99 0 0 0| 0 8192B| 34k 802k| 0 0 |1671 1240
0 0 100 0 0 0| 0 8192B| 48k 1067k| 0 0 |1878 1193
{code}
I managed to grab a thread dump:
https://gist.githubusercontent.com/anonymous/3b7b4698c32032603493/raw/read.md
and dmesg:
https://gist.githubusercontent.com/anonymous/5982b15337c9afbd5d49/raw/f3c2e4411b9d59e90f4615d93c7c1ad25922e170/read.md
Restarting the node solved the issue (it came up normally), we don't know
what is causing it but apparently (per the thread dump) gossip threads are
blocked writing the system keyspace and the writes waiting on the commitlog.
Gossip:
{code}
GossipStage:1 daemon prio=10 tid=0x7ffa23471800 nid=0xa13fa waiting on
condition [0x7ff9cbe26000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for 0x0005d3f50960 (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:964)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282)
at
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:731)
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351)
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:336)
at org.apache.cassandra.db.RowMutation.apply(RowMutation.java:211)
at
org.apache.cassandra.cql3.statements.ModificationStatement.executeInternal(ModificationStatement.java:709)
at
org.apache.cassandra.cql3.QueryProcessor.processInternal(QueryProcessor.java:208)
at
org.apache.cassandra.db.SystemKeyspace.updatePeerInfo(SystemKeyspace.java:379)
- locked 0x0005d3f41ed8 (a java.lang.Class for
org.apache.cassandra.db.SystemKeyspace)
at
org.apache.cassandra.service.StorageService.updatePeerInfo(StorageService.java:1414)
at
org.apache.cassandra.service.StorageService.handleStateNormal(StorageService.java:1524)
at
org.apache.cassandra.service.StorageService.onChange(StorageService.java:1350)
at
org.apache.cassandra.gms.Gossiper.doOnChangeNotifications(Gossiper.java:1083)
at org.apache.cassandra.gms.Gossiper.applyNewStates(Gossiper.java:1065)
at
org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:1023)
at
[jira] [Commented] (CASSANDRA-9279) Gossip (and mutations) lock up on startup
[
https://issues.apache.org/jira/browse/CASSANDRA-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523455#comment-14523455
]
Brandon Williams commented on CASSANDRA-9279:
-
That certainly explains it.
Gossip (and mutations) lock up on startup
-
Key: CASSANDRA-9279
URL: https://issues.apache.org/jira/browse/CASSANDRA-9279
Project: Cassandra
Issue Type: Bug
Reporter: Sebastian Estevez
Fix For: 2.0.x
Attachments: Screen Shot 2015-04-30 at 4.41.57 PM.png
Cluster running 2.0.14.352 on EC2 - c3.4xl's
2 nodes out of 8 exhibited the following behavior
When starting up the node we noticed it was gray in OpsCenter. Other
monitoring tool showed it as up.
Turned out gossip tasks were piling up and we could see the following in the
system.log:
{code}
WARN [GossipTasks:1] 2015-04-30 20:22:29,512 Gossiper.java (line 671) Gossip
stage has 4270 pending tasks; skipping status check (no nodes will be marked
down)
WARN [GossipTasks:1] 2015-04-30 20:22:30,612 Gossiper.java (line 671) Gossip
stage has 4272 pending tasks; skipping status check (no nodes will be marked
down)
WARN [GossipTasks:1] 2015-04-30 20:22:31,713 Gossiper.java (line 671) Gossip
stage has 4273 pending tasks; skipping status check (no nodes will be marked
down)
...
{code}
and tpstats shows blocked tasks--gossip and mutations:
{code}
GossipStage 1 3904 29384 0
0
{code}
the CPU's are inactive (See attachment)
and dstat output:
{code}
You did not select any stats, using -cdngy by default.
total-cpu-usage -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read writ| recv send| in out | int csw
2 0 97 0 0 0|1324k 1381k| 0 0 | 0 0 |6252 5548
0 0 100 0 0 0| 064k| 42k 1017k| 0 0 |3075 2537
0 0 99 0 0 0| 0 8192B| 39k 794k| 0 0 |6999 7039
0 0 100 0 0 0| 0 0 | 39k 759k| 0 0 |3067 2726
0 0 99 0 0 0| 0 184k| 48k 1086k| 0 0 |4829 4178
0 0 99 0 0 0| 0 8192B| 34k 802k| 0 0 |1671 1240
0 0 100 0 0 0| 0 8192B| 48k 1067k| 0 0 |1878 1193
{code}
I managed to grab a thread dump:
https://gist.githubusercontent.com/anonymous/3b7b4698c32032603493/raw/read.md
and dmesg:
https://gist.githubusercontent.com/anonymous/5982b15337c9afbd5d49/raw/f3c2e4411b9d59e90f4615d93c7c1ad25922e170/read.md
Restarting the node solved the issue (it came up normally), we don't know
what is causing it but apparently (per the thread dump) gossip threads are
blocked writing the system keyspace and the writes waiting on the commitlog.
Gossip:
{code}
GossipStage:1 daemon prio=10 tid=0x7ffa23471800 nid=0xa13fa waiting on
condition [0x7ff9cbe26000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for 0x0005d3f50960 (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:964)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282)
at
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:731)
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351)
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:336)
at org.apache.cassandra.db.RowMutation.apply(RowMutation.java:211)
at
org.apache.cassandra.cql3.statements.ModificationStatement.executeInternal(ModificationStatement.java:709)
at
org.apache.cassandra.cql3.QueryProcessor.processInternal(QueryProcessor.java:208)
at
org.apache.cassandra.db.SystemKeyspace.updatePeerInfo(SystemKeyspace.java:379)
- locked 0x0005d3f41ed8 (a java.lang.Class for
org.apache.cassandra.db.SystemKeyspace)
at
org.apache.cassandra.service.StorageService.updatePeerInfo(StorageService.java:1414)
at
org.apache.cassandra.service.StorageService.handleStateNormal(StorageService.java:1524)
at
org.apache.cassandra.service.StorageService.onChange(StorageService.java:1350)
at
org.apache.cassandra.gms.Gossiper.doOnChangeNotifications(Gossiper.java:1083)
at org.apache.cassandra.gms.Gossiper.applyNewStates(Gossiper.java:1065)
at
org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:1023)
at
[jira] [Commented] (CASSANDRA-9048) Delimited File Bulk Loader
[ https://issues.apache.org/jira/browse/CASSANDRA-9048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523488#comment-14523488 ] Jonathan Ellis commented on CASSANDRA-9048: --- Whatever we end up with under the hood, I think that cqlsh and COPY are the right front end to present to users rather than a separate loader executable. Delimited File Bulk Loader -- Key: CASSANDRA-9048 URL: https://issues.apache.org/jira/browse/CASSANDRA-9048 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Brian Hess Attachments: CASSANDRA-9048.patch There is a strong need for bulk loading data from delimited files into Cassandra. Starting with delimited files means that the data is not currently in the SSTable format, and therefore cannot immediately leverage Cassandra's bulk loading tool, sstableloader, directly. A tool supporting delimited files much closer matches the format of the data more often than the SSTable format itself, and a tool that loads from delimited files is very useful. In order for this bulk loader to be more generally useful to customers, it should handle a number of options at a minimum: - support specifying the input file or to read the data from stdin (so other command-line programs can pipe into the loader) - supply the CQL schema for the input data - support all data types other than collections (collections is a stretch goal/need) - an option to specify the delimiter - an option to specify comma as the decimal delimiter (for international use casese) - an option to specify how NULL values are specified in the file (e.g., the empty string or the string NULL) - an option to specify how BOOLEAN values are specified in the file (e.g., TRUE/FALSE or 0/1) - an option to specify the Date and Time format - an option to skip some number of rows at the beginning of the file - an option to only read in some number of rows from the file - an option to indicate how many parse errors to tolerate - an option to specify a file that will contain all the lines that did not parse correctly (up to the maximum number of parse errors) - an option to specify the CQL port to connect to (with 9042 as the default). Additional options would be useful, but this set of options/features is a start. A word on COPY. COPY comes via CQLSH which requires the client to be the same version as the server (e.g., 2.0 CQLSH does not work with 2.1 Cassandra, etc). This tool should be able to connect to any version of Cassandra (within reason). For example, it should be able to handle 2.0.x and 2.1.x. Moreover, CQLSH's COPY command does not support a number of the options above. Lastly, the performance of COPY in 2.0.x is not high enough to be considered a bulk ingest tool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8085) Make PasswordAuthenticator number of hashing rounds configurable
[ https://issues.apache.org/jira/browse/CASSANDRA-8085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523401#comment-14523401 ] Wei Deng commented on CASSANDRA-8085: - Should we expose this as a cassandra.yaml setting for PasswordAuthenticator, or we would like to leave it as configurable by -D option only, as it is an advanced option? Make PasswordAuthenticator number of hashing rounds configurable Key: CASSANDRA-8085 URL: https://issues.apache.org/jira/browse/CASSANDRA-8085 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Tyler Hobbs Assignee: Sam Tunnicliffe Fix For: 2.0.15, 2.1.5 Attachments: 8085-2.0.txt, 8085-2.1.txt, 8085-3.0.txt Running 2^10 rounds of bcrypt can take a while. In environments (like PHP) where connections are not typically long-lived, authenticating can add substantial overhead. On IRC, one user saw the time to connect, authenticate, and execute a query jump from 5ms to 150ms with authentication enabled ([debug logs|http://pastebin.com/bSUufbr0]). CASSANDRA-7715 is a more complete fix for this, but in the meantime (and even after 7715), this is a good option. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
cassandra git commit: Add CASSANDRA-9195 unit tests
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.1 33c59132f - f43efaa1b
Add CASSANDRA-9195 unit tests
patch by Branimir Lambov
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f43efaa1
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f43efaa1
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f43efaa1
Branch: refs/heads/cassandra-2.1
Commit: f43efaa1bdeec1057a8097fdb5ab62b4c9c19e67
Parents: 33c5913
Author: Branimir Lambov branimir.lam...@datastax.com
Authored: Fri May 1 19:42:50 2015 +0300
Committer: Aleksey Yeschenko alek...@apache.org
Committed: Fri May 1 19:44:03 2015 +0300
--
.../db/commitlog/CommitLogArchiver.java | 2 +-
.../db/RecoveryManagerTruncateTest.java | 179 +++
2 files changed, 142 insertions(+), 39 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/f43efaa1/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java
--
diff --git a/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java
b/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java
index 602cf94..91f3179 100644
--- a/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java
+++ b/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java
@@ -56,7 +56,7 @@ public class CommitLogArchiver
final String archiveCommand;
final String restoreCommand;
final String restoreDirectories;
-public final long restorePointInTime;
+public long restorePointInTime;
public final TimeUnit precision;
public CommitLogArchiver()
http://git-wip-us.apache.org/repos/asf/cassandra/blob/f43efaa1/test/unit/org/apache/cassandra/db/RecoveryManagerTruncateTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/RecoveryManagerTruncateTest.java
b/test/unit/org/apache/cassandra/db/RecoveryManagerTruncateTest.java
index 1f7d388..817b8e9 100644
--- a/test/unit/org/apache/cassandra/db/RecoveryManagerTruncateTest.java
+++ b/test/unit/org/apache/cassandra/db/RecoveryManagerTruncateTest.java
@@ -19,15 +19,14 @@
package org.apache.cassandra.db;
import static org.apache.cassandra.Util.column;
-import static org.junit.Assert.assertNotNull;
-import static org.junit.Assert.assertNull;
+import static org.junit.Assert.*;
import java.io.IOException;
+import org.junit.Test;
import org.apache.cassandra.SchemaLoader;
import org.apache.cassandra.Util;
import org.apache.cassandra.db.commitlog.CommitLog;
-import org.junit.Test;
import org.apache.cassandra.utils.ByteBufferUtil;
/**
@@ -35,46 +34,150 @@ import org.apache.cassandra.utils.ByteBufferUtil;
*/
public class RecoveryManagerTruncateTest extends SchemaLoader
{
- @Test
- public void testTruncate() throws IOException
- {
- Keyspace keyspace = Keyspace.open(Keyspace1);
- ColumnFamilyStore cfs =
keyspace.getColumnFamilyStore(Standard1);
+@Test
+public void testTruncate() throws IOException
+{
+Keyspace keyspace = Keyspace.open(Keyspace1);
+ColumnFamilyStore cfs = keyspace.getColumnFamilyStore(Standard1);
- Mutation rm;
- ColumnFamily cf;
+Mutation rm;
+ColumnFamily cf;
- // add a single cell
+// add a single cell
cf = ArrayBackedSortedColumns.factory.create(Keyspace1, Standard1);
- cf.addColumn(column(col1, val1, 1L));
+cf.addColumn(column(col1, val1, 1L));
rm = new Mutation(Keyspace1, ByteBufferUtil.bytes(keymulti), cf);
- rm.apply();
+rm.apply();
+long time = System.currentTimeMillis();
- // Make sure data was written
- assertNotNull(getFromTable(keyspace, Standard1, keymulti,
col1));
+// Make sure data was written
+assertNotNull(getFromTable(keyspace, Standard1, keymulti, col1));
- // and now truncate it
- cfs.truncateBlocking();
+// and now truncate it
+cfs.truncateBlocking();
CommitLog.instance.resetUnsafe();
- CommitLog.instance.recover();
-
- // and validate truncation.
- assertNull(getFromTable(keyspace, Standard1, keymulti,
col1));
- }
-
- private Cell getFromTable(Keyspace keyspace, String cfName, String
keyName, String columnName)
- {
- ColumnFamily cf;
- ColumnFamilyStore cfStore =
keyspace.getColumnFamilyStore(cfName);
- if (cfStore == null)
- {
- return null;
- }
- cf =
[1/2] cassandra git commit: Add CASSANDRA-9195 unit tests
Repository: cassandra
Updated Branches:
refs/heads/trunk c799a98f0 - 6404e015f
Add CASSANDRA-9195 unit tests
patch by Branimir Lambov
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f43efaa1
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f43efaa1
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f43efaa1
Branch: refs/heads/trunk
Commit: f43efaa1bdeec1057a8097fdb5ab62b4c9c19e67
Parents: 33c5913
Author: Branimir Lambov branimir.lam...@datastax.com
Authored: Fri May 1 19:42:50 2015 +0300
Committer: Aleksey Yeschenko alek...@apache.org
Committed: Fri May 1 19:44:03 2015 +0300
--
.../db/commitlog/CommitLogArchiver.java | 2 +-
.../db/RecoveryManagerTruncateTest.java | 179 +++
2 files changed, 142 insertions(+), 39 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/f43efaa1/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java
--
diff --git a/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java
b/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java
index 602cf94..91f3179 100644
--- a/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java
+++ b/src/java/org/apache/cassandra/db/commitlog/CommitLogArchiver.java
@@ -56,7 +56,7 @@ public class CommitLogArchiver
final String archiveCommand;
final String restoreCommand;
final String restoreDirectories;
-public final long restorePointInTime;
+public long restorePointInTime;
public final TimeUnit precision;
public CommitLogArchiver()
http://git-wip-us.apache.org/repos/asf/cassandra/blob/f43efaa1/test/unit/org/apache/cassandra/db/RecoveryManagerTruncateTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/RecoveryManagerTruncateTest.java
b/test/unit/org/apache/cassandra/db/RecoveryManagerTruncateTest.java
index 1f7d388..817b8e9 100644
--- a/test/unit/org/apache/cassandra/db/RecoveryManagerTruncateTest.java
+++ b/test/unit/org/apache/cassandra/db/RecoveryManagerTruncateTest.java
@@ -19,15 +19,14 @@
package org.apache.cassandra.db;
import static org.apache.cassandra.Util.column;
-import static org.junit.Assert.assertNotNull;
-import static org.junit.Assert.assertNull;
+import static org.junit.Assert.*;
import java.io.IOException;
+import org.junit.Test;
import org.apache.cassandra.SchemaLoader;
import org.apache.cassandra.Util;
import org.apache.cassandra.db.commitlog.CommitLog;
-import org.junit.Test;
import org.apache.cassandra.utils.ByteBufferUtil;
/**
@@ -35,46 +34,150 @@ import org.apache.cassandra.utils.ByteBufferUtil;
*/
public class RecoveryManagerTruncateTest extends SchemaLoader
{
- @Test
- public void testTruncate() throws IOException
- {
- Keyspace keyspace = Keyspace.open(Keyspace1);
- ColumnFamilyStore cfs =
keyspace.getColumnFamilyStore(Standard1);
+@Test
+public void testTruncate() throws IOException
+{
+Keyspace keyspace = Keyspace.open(Keyspace1);
+ColumnFamilyStore cfs = keyspace.getColumnFamilyStore(Standard1);
- Mutation rm;
- ColumnFamily cf;
+Mutation rm;
+ColumnFamily cf;
- // add a single cell
+// add a single cell
cf = ArrayBackedSortedColumns.factory.create(Keyspace1, Standard1);
- cf.addColumn(column(col1, val1, 1L));
+cf.addColumn(column(col1, val1, 1L));
rm = new Mutation(Keyspace1, ByteBufferUtil.bytes(keymulti), cf);
- rm.apply();
+rm.apply();
+long time = System.currentTimeMillis();
- // Make sure data was written
- assertNotNull(getFromTable(keyspace, Standard1, keymulti,
col1));
+// Make sure data was written
+assertNotNull(getFromTable(keyspace, Standard1, keymulti, col1));
- // and now truncate it
- cfs.truncateBlocking();
+// and now truncate it
+cfs.truncateBlocking();
CommitLog.instance.resetUnsafe();
- CommitLog.instance.recover();
-
- // and validate truncation.
- assertNull(getFromTable(keyspace, Standard1, keymulti,
col1));
- }
-
- private Cell getFromTable(Keyspace keyspace, String cfName, String
keyName, String columnName)
- {
- ColumnFamily cf;
- ColumnFamilyStore cfStore =
keyspace.getColumnFamilyStore(cfName);
- if (cfStore == null)
- {
- return null;
- }
- cf =
[jira] [Updated] (CASSANDRA-9286) Add Keyspace/Table details to CollectionType.java error message
[
https://issues.apache.org/jira/browse/CASSANDRA-9286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sequoyha pelletier updated CASSANDRA-9286:
--
Description:
The error message for too many element in a collection does not give keyspace
or column family information. This makes it a pain point to try to determine
which table is the offending table.
Example Error message:
{noformat}
ERROR [Native-Transport-Requests:809453] 2015-04-23 22:48:21,189
CollectionType.java (line 116) Detected collection with 136234 elements, more
than the 65535 limit. Only the first 65535 elements will be returned to the
client. Please see http://cassandra.apache.org/doc/cql3/CQL.html#collections
for more details.
{noformat}
Currently, to try to pinpoint the table in question. We need to trace all
requests and then try to match up the timestamps in the CQL tracing session
with the log timestamps to try and match. If prepared statements are used, this
is a dead end due to the logged tracing information missing the query. In which
case, we have to look at other 3rd party methods for capturing the queries to
try and match up. This is extremely tedious when many tables have collections
and a high number of ops against them.
Requesting that the error contain the keyspace.table name in the error message.
was:
The error message for too many element in a collection does not give keyspace
or column family information. This makes it a pain point to try to determine
which table is the offending table.
Example Error message:
{noformat}
ERROR [Native-Transport-Requests:809453] 2015-04-23 22:48:21,189
CollectionType.java (line 116) Detected collection with 136234 elements, more
than the 65535 limit. Only the first 65535 elements will be returned to the
client. Please see http://cassandra.apache.org/doc/cql3/CQL.html#collections
for more details.
{norformat}
Currently, to try to pinpoint the table in question. We need to trace all
requests and then try to match up the timestamps in the CQL tracing session
with the log timestamps to try and match. If prepared statements are used, this
is a dead end due to the logged tracing information missing the query. In which
case, we have to look at other 3rd party methods for capturing the queries to
try and match up. This is extremely tedious when many tables have collections
and a high number of ops against them.
Requesting that the error contain the keyspace.table name in the error message.
Add Keyspace/Table details to CollectionType.java error message
---
Key: CASSANDRA-9286
URL: https://issues.apache.org/jira/browse/CASSANDRA-9286
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: sequoyha pelletier
Priority: Minor
The error message for too many element in a collection does not give keyspace
or column family information. This makes it a pain point to try to determine
which table is the offending table.
Example Error message:
{noformat}
ERROR [Native-Transport-Requests:809453] 2015-04-23 22:48:21,189
CollectionType.java (line 116) Detected collection with 136234 elements, more
than the 65535 limit. Only the first 65535 elements will be returned to the
client. Please see http://cassandra.apache.org/doc/cql3/CQL.html#collections
for more details.
{noformat}
Currently, to try to pinpoint the table in question. We need to trace all
requests and then try to match up the timestamps in the CQL tracing session
with the log timestamps to try and match. If prepared statements are used,
this is a dead end due to the logged tracing information missing the query.
In which case, we have to look at other 3rd party methods for capturing the
queries to try and match up. This is extremely tedious when many tables have
collections and a high number of ops against them.
Requesting that the error contain the keyspace.table name in the error
message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9048) Delimited File Bulk Loader
[
https://issues.apache.org/jira/browse/CASSANDRA-9048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523451#comment-14523451
]
Aleksey Yeschenko commented on CASSANDRA-9048:
--
Let me clarify my position here.
At some point in 3.X series we are going to have Spark built in, for reasons
unrelated to CSV {{COPY}}.
We are also going to have CASSANDRA-8234 utilizing it for data migration. Given
those preconditions, I aimed to use that very same framework for CSV {{COPY}} -
implemented as just another Spark input (via
https://github.com/databricks/spark-csv) and the same output logic. In the
meantime, Tyler sped up our built-in {{COPY}} by another 10x, making it ~100x
faster than 2.0 {{COPY}} is, fast enough as an interim solution.
Now, that may or may not work. After talking to Brian a few times, I see his
point, and mostly agree with it. Here's what we settled on, for now: once 3.0
is out, get someone to prototype the Spark-based loader. See how performant it
is, and whether or not we can have all the required features that way. If the
answer to either of those questions is no, we'll take Brian's loader, make it
faster/more feature-full, and start shipping it as a standard tool with
Cassandra at some point in 3.X.
Delimited File Bulk Loader
--
Key: CASSANDRA-9048
URL: https://issues.apache.org/jira/browse/CASSANDRA-9048
Project: Cassandra
Issue Type: Improvement
Components: Tools
Reporter: Brian Hess
Attachments: CASSANDRA-9048.patch
There is a strong need for bulk loading data from delimited files into
Cassandra. Starting with delimited files means that the data is not
currently in the SSTable format, and therefore cannot immediately leverage
Cassandra's bulk loading tool, sstableloader, directly.
A tool supporting delimited files much closer matches the format of the data
more often than the SSTable format itself, and a tool that loads from
delimited files is very useful.
In order for this bulk loader to be more generally useful to customers, it
should handle a number of options at a minimum:
- support specifying the input file or to read the data from stdin (so other
command-line programs can pipe into the loader)
- supply the CQL schema for the input data
- support all data types other than collections (collections is a stretch
goal/need)
- an option to specify the delimiter
- an option to specify comma as the decimal delimiter (for international use
casese)
- an option to specify how NULL values are specified in the file (e.g., the
empty string or the string NULL)
- an option to specify how BOOLEAN values are specified in the file (e.g.,
TRUE/FALSE or 0/1)
- an option to specify the Date and Time format
- an option to skip some number of rows at the beginning of the file
- an option to only read in some number of rows from the file
- an option to indicate how many parse errors to tolerate
- an option to specify a file that will contain all the lines that did not
parse correctly (up to the maximum number of parse errors)
- an option to specify the CQL port to connect to (with 9042 as the default).
Additional options would be useful, but this set of options/features is a
start.
A word on COPY. COPY comes via CQLSH which requires the client to be the
same version as the server (e.g., 2.0 CQLSH does not work with 2.1 Cassandra,
etc). This tool should be able to connect to any version of Cassandra
(within reason). For example, it should be able to handle 2.0.x and 2.1.x.
Moreover, CQLSH's COPY command does not support a number of the options
above. Lastly, the performance of COPY in 2.0.x is not high enough to be
considered a bulk ingest tool.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9285) LEAK DETECTED in sstwriter
[
https://issues.apache.org/jira/browse/CASSANDRA-9285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philip Thompson updated CASSANDRA-9285:
---
Assignee: Carl Yeksigian
LEAK DETECTED in sstwriter
--
Key: CASSANDRA-9285
URL: https://issues.apache.org/jira/browse/CASSANDRA-9285
Project: Cassandra
Issue Type: Bug
Reporter: Pierre N.
Assignee: Carl Yeksigian
Fix For: 2.1.x
reproduce bug :
{code}
public static void main(String[] args) throws Exception {
System.setProperty(cassandra.debugrefcount,true);
String ks = ks1;
String table = t1;
String schema = CREATE TABLE + ks + . + table + (a1 INT,
PRIMARY KEY (a1));;
String insert = INSERT INTO + ks + . + table + (a1) VALUES(?);;
File dir = new File(/var/tmp/ + ks + / + table);
dir.mkdirs();
CQLSSTableWriter writer =
CQLSSTableWriter.builder().forTable(schema).using(insert).inDirectory(dir).build();
writer.addRow(1);
writer.close();
writer = null;
Thread.sleep(1000);System.gc();
Thread.sleep(1000);System.gc();
}
{code}
{quote}
[2015-05-01 16:09:59,139] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - LEAK DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@79fa9da9) to class
org.apache.cassandra.io.util.SafeMemory$MemoryTidy@2053866990:Memory@[7f87f8043b20..7f87f8043b48)
was not released before the reference was garbage collected
[2015-05-01 16:09:59,143] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - Allocate trace
org.apache.cassandra.utils.concurrent.Ref$State@79fa9da9:
Thread[Thread-2,5,main]
at java.lang.Thread.getStackTrace(Thread.java:1552)
at org.apache.cassandra.utils.concurrent.Ref$Debug.init(Ref.java:200)
at org.apache.cassandra.utils.concurrent.Ref$State.init(Ref.java:133)
at org.apache.cassandra.utils.concurrent.Ref.init(Ref.java:60)
at org.apache.cassandra.io.util.SafeMemory.init(SafeMemory.java:32)
at
org.apache.cassandra.io.util.SafeMemoryWriter.init(SafeMemoryWriter.java:33)
at
org.apache.cassandra.io.sstable.IndexSummaryBuilder.init(IndexSummaryBuilder.java:111)
at
org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.init(SSTableWriter.java:576)
at
org.apache.cassandra.io.sstable.SSTableWriter.init(SSTableWriter.java:140)
at
org.apache.cassandra.io.sstable.AbstractSSTableSimpleWriter.getWriter(AbstractSSTableSimpleWriter.java:58)
at
org.apache.cassandra.io.sstable.SSTableSimpleUnsortedWriter$DiskWriter.run(SSTableSimpleUnsortedWriter.java:227)
[2015-05-01 16:09:59,144] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - LEAK DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@664382e3) to class
org.apache.cassandra.io.util.SafeMemory$MemoryTidy@899100784:Memory@[7f87f8043990..7f87f8043994)
was not released before the reference was garbage collected
[2015-05-01 16:09:59,144] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - Allocate trace
org.apache.cassandra.utils.concurrent.Ref$State@664382e3:
Thread[Thread-2,5,main]
at java.lang.Thread.getStackTrace(Thread.java:1552)
at org.apache.cassandra.utils.concurrent.Ref$Debug.init(Ref.java:200)
at org.apache.cassandra.utils.concurrent.Ref$State.init(Ref.java:133)
at org.apache.cassandra.utils.concurrent.Ref.init(Ref.java:60)
at org.apache.cassandra.io.util.SafeMemory.init(SafeMemory.java:32)
at
org.apache.cassandra.io.util.SafeMemoryWriter.init(SafeMemoryWriter.java:33)
at
org.apache.cassandra.io.sstable.IndexSummaryBuilder.init(IndexSummaryBuilder.java:110)
at
org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.init(SSTableWriter.java:576)
at
org.apache.cassandra.io.sstable.SSTableWriter.init(SSTableWriter.java:140)
at
org.apache.cassandra.io.sstable.AbstractSSTableSimpleWriter.getWriter(AbstractSSTableSimpleWriter.java:58)
at
org.apache.cassandra.io.sstable.SSTableSimpleUnsortedWriter$DiskWriter.run(SSTableSimpleUnsortedWriter.java:227)
[2015-05-01 16:09:59,144] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - LEAK DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@3cca0ac2) to class
org.apache.cassandra.io.util.SafeMemory$MemoryTidy@499043670:Memory@[7f87f8039940..7f87f8039c60)
was not released before the reference was garbage collected
[2015-05-01 16:09:59,144] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - Allocate trace
org.apache.cassandra.utils.concurrent.Ref$State@3cca0ac2:
[jira] [Commented] (CASSANDRA-6477) Global indexes
[ https://issues.apache.org/jira/browse/CASSANDRA-6477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523507#comment-14523507 ] Carl Yeksigian commented on CASSANDRA-6477: --- At NGCC, there was some discussion about how we are going to handle eventual consistency in this design. Benedict's proposal earlier in this ticket, where each data replica performs a local read-before-write and pushes out the value to the index replicas is the one which does provide eventual consistency guarantees in line with user's expectations. In order to make sure that the values are consistent in the GI, it will also need to use a batch log at quorum to ensure that the update eventually gets applied. Here is the process that GI will take on mutation: - If the mutations will be affected by global indexes (either the target or an included column), coordinator creates batch log which will wait for a quorum to ack mutations before being deleted - Coordinator pushes mutations out to data replicas - Data replica checks if mutation touches global index - Data replica takes lock on global index row in order to make sure that the generated mutations are consistent - Data replica creates index mutation, blocks until that is complete - Data replica applies data mutation, acks mutation The ordering of the index mutation before the data mutation means that if we fail at index mutation, we will generate the same values the next time this mutations is applied. We will send to exactly one replica, whose position in the ring relative to the token is equidistant as ours is. Global indexes -- Key: CASSANDRA-6477 URL: https://issues.apache.org/jira/browse/CASSANDRA-6477 Project: Cassandra Issue Type: New Feature Components: API, Core Reporter: Jonathan Ellis Assignee: Carl Yeksigian Labels: cql Fix For: 3.x Local indexes are suitable for low-cardinality data, where spreading the index across the cluster is a Good Thing. However, for high-cardinality data, local indexes require querying most nodes in the cluster even if only a handful of rows is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9285) LEAK DETECTED in sstwriter
[
https://issues.apache.org/jira/browse/CASSANDRA-9285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523406#comment-14523406
]
Benedict commented on CASSANDRA-9285:
-
[~pierz]: there's no need to worry about this, btw. Not only is this not part
of a running server process (it's an external tool), the problem is completely
benign, as it is caught and corrected.
LEAK DETECTED in sstwriter
--
Key: CASSANDRA-9285
URL: https://issues.apache.org/jira/browse/CASSANDRA-9285
Project: Cassandra
Issue Type: Bug
Reporter: Pierre N.
Assignee: Carl Yeksigian
Fix For: 2.1.x
reproduce bug :
{code}
public static void main(String[] args) throws Exception {
System.setProperty(cassandra.debugrefcount,true);
String ks = ks1;
String table = t1;
String schema = CREATE TABLE + ks + . + table + (a1 INT,
PRIMARY KEY (a1));;
String insert = INSERT INTO + ks + . + table + (a1) VALUES(?);;
File dir = new File(/var/tmp/ + ks + / + table);
dir.mkdirs();
CQLSSTableWriter writer =
CQLSSTableWriter.builder().forTable(schema).using(insert).inDirectory(dir).build();
writer.addRow(1);
writer.close();
writer = null;
Thread.sleep(1000);System.gc();
Thread.sleep(1000);System.gc();
}
{code}
{quote}
[2015-05-01 16:09:59,139] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - LEAK DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@79fa9da9) to class
org.apache.cassandra.io.util.SafeMemory$MemoryTidy@2053866990:Memory@[7f87f8043b20..7f87f8043b48)
was not released before the reference was garbage collected
[2015-05-01 16:09:59,143] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - Allocate trace
org.apache.cassandra.utils.concurrent.Ref$State@79fa9da9:
Thread[Thread-2,5,main]
at java.lang.Thread.getStackTrace(Thread.java:1552)
at org.apache.cassandra.utils.concurrent.Ref$Debug.init(Ref.java:200)
at org.apache.cassandra.utils.concurrent.Ref$State.init(Ref.java:133)
at org.apache.cassandra.utils.concurrent.Ref.init(Ref.java:60)
at org.apache.cassandra.io.util.SafeMemory.init(SafeMemory.java:32)
at
org.apache.cassandra.io.util.SafeMemoryWriter.init(SafeMemoryWriter.java:33)
at
org.apache.cassandra.io.sstable.IndexSummaryBuilder.init(IndexSummaryBuilder.java:111)
at
org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.init(SSTableWriter.java:576)
at
org.apache.cassandra.io.sstable.SSTableWriter.init(SSTableWriter.java:140)
at
org.apache.cassandra.io.sstable.AbstractSSTableSimpleWriter.getWriter(AbstractSSTableSimpleWriter.java:58)
at
org.apache.cassandra.io.sstable.SSTableSimpleUnsortedWriter$DiskWriter.run(SSTableSimpleUnsortedWriter.java:227)
[2015-05-01 16:09:59,144] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - LEAK DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@664382e3) to class
org.apache.cassandra.io.util.SafeMemory$MemoryTidy@899100784:Memory@[7f87f8043990..7f87f8043994)
was not released before the reference was garbage collected
[2015-05-01 16:09:59,144] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - Allocate trace
org.apache.cassandra.utils.concurrent.Ref$State@664382e3:
Thread[Thread-2,5,main]
at java.lang.Thread.getStackTrace(Thread.java:1552)
at org.apache.cassandra.utils.concurrent.Ref$Debug.init(Ref.java:200)
at org.apache.cassandra.utils.concurrent.Ref$State.init(Ref.java:133)
at org.apache.cassandra.utils.concurrent.Ref.init(Ref.java:60)
at org.apache.cassandra.io.util.SafeMemory.init(SafeMemory.java:32)
at
org.apache.cassandra.io.util.SafeMemoryWriter.init(SafeMemoryWriter.java:33)
at
org.apache.cassandra.io.sstable.IndexSummaryBuilder.init(IndexSummaryBuilder.java:110)
at
org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.init(SSTableWriter.java:576)
at
org.apache.cassandra.io.sstable.SSTableWriter.init(SSTableWriter.java:140)
at
org.apache.cassandra.io.sstable.AbstractSSTableSimpleWriter.getWriter(AbstractSSTableSimpleWriter.java:58)
at
org.apache.cassandra.io.sstable.SSTableSimpleUnsortedWriter$DiskWriter.run(SSTableSimpleUnsortedWriter.java:227)
[2015-05-01 16:09:59,144] [Reference-Reaper:1] ERROR
org.apache.cassandra.utils.concurrent.Ref - LEAK DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@3cca0ac2) to class
org.apache.cassandra.io.util.SafeMemory$MemoryTidy@499043670:Memory@[7f87f8039940..7f87f8039c60)
was not released before the reference was
[jira] [Created] (CASSANDRA-9286) Add Keyspace/Table details to CollectionType.java error message
sequoyha pelletier created CASSANDRA-9286:
-
Summary: Add Keyspace/Table details to CollectionType.java error
message
Key: CASSANDRA-9286
URL: https://issues.apache.org/jira/browse/CASSANDRA-9286
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: sequoyha pelletier
Priority: Minor
The error message for too many element in a collection does not give keyspace
or column family information. This makes it a pain point to try to determine
which table is the offending table.
Example Error message:
{noformat}
ERROR [Native-Transport-Requests:809453] 2015-04-23 22:48:21,189
CollectionType.java (line 116) Detected collection with 136234 elements, more
than the 65535 limit. Only the first 65535 elements will be returned to the
client. Please see http://cassandra.apache.org/doc/cql3/CQL.html#collections
for more details.
{norformat}
Currently, to try to pinpoint the table in question. We need to trace all
requests and then try to match up the timestamps in the CQL tracing session
with the log timestamps to try and match. If prepared statements are used, this
is a dead end due to the logged tracing information missing the query. In which
case, we have to look at other 3rd party methods for capturing the queries to
try and match up. This is extremely tedious when many tables have collections
and a high number of ops against them.
Requesting that the error contain the keyspace.table name in the error message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[3/3] cassandra git commit: Merge branch 'cassandra-2.1' into trunk
Merge branch 'cassandra-2.1' into trunk
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/c799a98f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/c799a98f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/c799a98f
Branch: refs/heads/trunk
Commit: c799a98f0a454010cb3a63c874c404d1d2c79048
Parents: 184f5b3 33c5913
Author: Aleksey Yeschenko alek...@apache.org
Authored: Fri May 1 19:21:04 2015 +0300
Committer: Aleksey Yeschenko alek...@apache.org
Committed: Fri May 1 19:21:04 2015 +0300
--
CHANGES.txt | 5 +++
.../cassandra/db/index/SecondaryIndex.java | 10 --
.../db/index/SecondaryIndexManager.java | 2 +-
.../cassandra/db/ColumnFamilyStoreTest.java | 37
.../db/index/PerRowSecondaryIndexTest.java | 8 -
5 files changed, 43 insertions(+), 19 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/c799a98f/CHANGES.txt
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/c799a98f/src/java/org/apache/cassandra/db/index/SecondaryIndex.java
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/c799a98f/src/java/org/apache/cassandra/db/index/SecondaryIndexManager.java
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/c799a98f/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
--
diff --cc test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
index 0c221ac,84e7d20..349edbe
--- a/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
+++ b/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
@@@ -43,12 -30,8 +43,14 @@@ import java.util.concurrent.TimeUnit
import com.google.common.base.Function;
import com.google.common.collect.Iterables;
import com.google.common.collect.Sets;
++
++import org.apache.cassandra.db.index.PerRowSecondaryIndexTest;
+import org.apache.cassandra.io.sstable.*;
+import org.apache.cassandra.io.sstable.format.SSTableReader;
+import org.apache.cassandra.io.sstable.format.SSTableWriter;
import org.apache.commons.lang3.ArrayUtils;
import org.apache.commons.lang3.StringUtils;
+import org.junit.BeforeClass;
import org.junit.Test;
import org.junit.runner.RunWith;
@@@ -105,23 -70,9 +107,24 @@@ import static org.junit.Assert.assertSa
import static org.junit.Assert.assertTrue;
@RunWith(OrderedJUnit4ClassRunner.class)
-public class ColumnFamilyStoreTest extends SchemaLoader
+public class ColumnFamilyStoreTest
{
static byte[] bytes1, bytes2;
+public static final String KEYSPACE1 = ColumnFamilyStoreTest1;
+public static final String KEYSPACE2 = ColumnFamilyStoreTest2;
+public static final String KEYSPACE3 = ColumnFamilyStoreTest3;
++public static final String KEYSPACE4 = PerRowSecondaryIndex;
+public static final String CF_STANDARD1 = Standard1;
+public static final String CF_STANDARD2 = Standard2;
+public static final String CF_STANDARD3 = Standard3;
+public static final String CF_STANDARD4 = Standard4;
+public static final String CF_STANDARD5 = Standard5;
+public static final String CF_STANDARDINT = StandardInteger1;
+public static final String CF_SUPER1 = Super1;
+public static final String CF_SUPER6 = Super6;
+public static final String CF_INDEX1 = Indexed1;
+public static final String CF_INDEX2 = Indexed2;
+public static final String CF_INDEX3 = Indexed3;
static
{
@@@ -132,36 -83,6 +135,40 @@@
random.nextBytes(bytes2);
}
+@BeforeClass
+public static void defineSchema() throws ConfigurationException
+{
+SchemaLoader.prepareServer();
+SchemaLoader.createKeyspace(KEYSPACE1,
+SimpleStrategy.class,
+KSMetaData.optsWithRF(1),
+SchemaLoader.standardCFMD(KEYSPACE1,
CF_STANDARD1),
+SchemaLoader.standardCFMD(KEYSPACE1,
CF_STANDARD2),
+SchemaLoader.standardCFMD(KEYSPACE1,
CF_STANDARD3),
+SchemaLoader.standardCFMD(KEYSPACE1,
CF_STANDARD4),
+SchemaLoader.standardCFMD(KEYSPACE1,
CF_STANDARD5),
+SchemaLoader.indexCFMD(KEYSPACE1,
CF_INDEX1, true),
+SchemaLoader.indexCFMD(KEYSPACE1,
CF_INDEX2, false),
+
[2/3] cassandra git commit: Merge branch 'cassandra-2.0' into cassandra-2.1
Merge branch 'cassandra-2.0' into cassandra-2.1 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/33c59132 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/33c59132 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/33c59132 Branch: refs/heads/trunk Commit: 33c59132f730599859407a8460f1433e36f83fdf Parents: 5fcf8df 1e35fa4 Author: Aleksey Yeschenko alek...@apache.org Authored: Fri May 1 19:01:00 2015 +0300 Committer: Aleksey Yeschenko alek...@apache.org Committed: Fri May 1 19:01:00 2015 +0300 -- CHANGES.txt | 5 + .../org/apache/cassandra/db/index/SecondaryIndex.java | 10 -- .../apache/cassandra/db/index/SecondaryIndexManager.java | 2 +- .../org/apache/cassandra/db/ColumnFamilyStoreTest.java| 10 +++--- .../cassandra/db/index/PerRowSecondaryIndexTest.java | 8 5 files changed, 13 insertions(+), 22 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/33c59132/CHANGES.txt -- diff --cc CHANGES.txt index 55b474b,2dabbf9..b0e4fc6 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,101 -1,13 +1,103 @@@ -2.0.15: +2.1.6 + * Reduce runWithCompactionsDisabled poll interval to 1ms (CASSANDRA-9271) + * Fix PITR commitlog replay (CASSANDRA-9195) + * GCInspector logs very different times (CASSANDRA-9124) + * Fix deleting from an empty list (CASSANDRA-9198) + * Update tuple and collection types that use a user-defined type when that UDT + is modified (CASSANDRA-9148, CASSANDRA-9192) +Merged from 2.0: + * Fix index selection during rebuild with certain table layouts (CASSANDRA-9281) * Fix partition-level-delete-only workload accounting (CASSANDRA-9194) * Allow scrub to handle corrupted compressed chunks (CASSANDRA-9140) * Fix assertion error when resetlocalschema is run during repair (CASSANDRA-9249) * Disable single sstable tombstone compactions for DTCS by default (CASSANDRA-9234) - * Do more agressive ttl expiration checks to be able to - drop more sstables (CASSANDRA-8243) * IncomingTcpConnection thread is not named (CASSANDRA-9262) * Close incoming connections when MessagingService is stopped (CASSANDRA-9238) + ++ +2.1.5 + * Re-add deprecated cold_reads_to_omit param for backwards compat (CASSANDRA-9203) + * Make anticompaction visible in compactionstats (CASSANDRA-9098) + * Improve nodetool getendpoints documentation about the partition + key parameter (CASSANDRA-6458) + * Don't check other keyspaces for schema changes when an user-defined + type is altered (CASSANDRA-9187) + * Allow takeColumnFamilySnapshot to take a list of tables (CASSANDRA-8348) + * Limit major sstable operations to their canonical representation (CASSANDRA-8669) + * cqlsh: Add tests for INSERT and UPDATE tab completion (CASSANDRA-9125) + * cqlsh: quote column names when needed in COPY FROM inserts (CASSANDRA-9080) + * Add generate-idea-files target to build.xml (CASSANDRA-9123) + * Do not load read meter for offline operations (CASSANDRA-9082) + * cqlsh: Make CompositeType data readable (CASSANDRA-8919) + * cqlsh: Fix display of triggers (CASSANDRA-9081) + * Fix NullPointerException when deleting or setting an element by index on + a null list collection (CASSANDRA-9077) + * Buffer bloom filter serialization (CASSANDRA-9066) + * Fix anti-compaction target bloom filter size (CASSANDRA-9060) + * Make FROZEN and TUPLE unreserved keywords in CQL (CASSANDRA-9047) + * Prevent AssertionError from SizeEstimatesRecorder (CASSANDRA-9034) + * Avoid overwriting index summaries for sstables with an older format that + does not support downsampling; rebuild summaries on startup when this + is detected (CASSANDRA-8993) + * Fix potential data loss in CompressedSequentialWriter (CASSANDRA-8949) + * Make PasswordAuthenticator number of hashing rounds configurable (CASSANDRA-8085) + * Fix AssertionError when binding nested collections in DELETE (CASSANDRA-8900) + * Check for overlap with non-early sstables in LCS (CASSANDRA-8739) + * Only calculate max purgable timestamp if we have to (CASSANDRA-8914) + * (cqlsh) Greatly improve performance of COPY FROM (CASSANDRA-8225) + * IndexSummary effectiveIndexInterval is now a guideline, not a rule (CASSANDRA-8993) + * Use correct bounds for page cache eviction of compressed files (CASSANDRA-8746) + * SSTableScanner enforces its bounds (CASSANDRA-8946) + * Cleanup cell equality (CASSANDRA-8947) + * Introduce intra-cluster message coalescing (CASSANDRA-8692) + * DatabaseDescriptor throws NPE when rpc_interface is used (CASSANDRA-8839) + * Don't check if an sstable is live for offline compactions (CASSANDRA-8841) + * Don't set
[1/3] cassandra git commit: Fix index selection during rebuild with certain table layouts
Repository: cassandra
Updated Branches:
refs/heads/trunk 184f5b3ad - c799a98f0
Fix index selection during rebuild with certain table layouts
patch by Sam Tunnicliffe; reviewed by Jeremiah Jordan for CASSANDRA-9281
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1e35fa4b
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1e35fa4b
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1e35fa4b
Branch: refs/heads/trunk
Commit: 1e35fa4b5a9afb4f37147a7a932a7892f2f450c1
Parents: f0c7a6f
Author: Sam Tunnicliffe s...@beobal.com
Authored: Fri May 1 10:17:39 2015 +0100
Committer: Aleksey Yeschenko alek...@apache.org
Committed: Fri May 1 18:55:08 2015 +0300
--
CHANGES.txt | 3 +++
.../apache/cassandra/db/index/SecondaryIndexManager.java | 2 +-
.../org/apache/cassandra/db/ColumnFamilyStoreTest.java| 10 +++---
.../cassandra/db/index/PerRowSecondaryIndexTest.java | 9 +
4 files changed, 12 insertions(+), 12 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/1e35fa4b/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index dfdad51..2dabbf9 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.0.15:
+ * Fix index selection during rebuild with certain table layouts
(CASSANDRA-9281)
* Fix partition-level-delete-only workload accounting (CASSANDRA-9194)
* Allow scrub to handle corrupted compressed chunks (CASSANDRA-9140)
* Fix assertion error when resetlocalschema is run during repair
(CASSANDRA-9249)
@@ -57,9 +58,11 @@
* Fix MT mismatch between empty and GC-able data (CASSANDRA-8979)
* Fix incorrect validation when snapshotting single table (CASSANDRA-8056)
+
2.0.14
* Bind JMX to localhost unless explicitly configured otherwise
(CASSANDRA-9085)
+
2.0.13:
* Add offline tool to relevel sstables (CASSANDRA-8301)
* Preserve stream ID for more protocol errors (CASSANDRA-8848)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/1e35fa4b/src/java/org/apache/cassandra/db/index/SecondaryIndexManager.java
--
diff --git a/src/java/org/apache/cassandra/db/index/SecondaryIndexManager.java
b/src/java/org/apache/cassandra/db/index/SecondaryIndexManager.java
index 1db7de6..d5e88d0 100644
--- a/src/java/org/apache/cassandra/db/index/SecondaryIndexManager.java
+++ b/src/java/org/apache/cassandra/db/index/SecondaryIndexManager.java
@@ -589,7 +589,7 @@ public class SecondaryIndexManager
{
for (ColumnDefinition column : baseCfs.metadata.allColumns())
{
-if (candidate.indexes(column.name))
+if (candidate.getColumnDefs().contains(column))
{
filtered.add(candidate.getIndexName());
break;
http://git-wip-us.apache.org/repos/asf/cassandra/blob/1e35fa4b/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
--
diff --git a/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
b/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
index 8f4a18f..001f838 100644
--- a/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
+++ b/test/unit/org/apache/cassandra/db/ColumnFamilyStoreTest.java
@@ -2177,14 +2177,16 @@ public class ColumnFamilyStoreTest extends SchemaLoader
@Test
public void testRebuildSecondaryIndex() throws IOException
{
+ByteBuffer indexedColumnName = ByteBufferUtil.bytes(indexed);
RowMutation rm;
rm = new RowMutation(PerRowSecondaryIndex,
ByteBufferUtil.bytes(k1));
-rm.add(Indexed1, ByteBufferUtil.bytes(indexed),
ByteBufferUtil.bytes(foo), 1);
+rm.add(Indexed1, indexedColumnName, ByteBufferUtil.bytes(foo), 1);
rm.apply();
assertTrue(Arrays.equals(k1.getBytes(),
PerRowSecondaryIndexTest.TestIndex.LAST_INDEXED_KEY.array()));
-
Keyspace.open(PerRowSecondaryIndex).getColumnFamilyStore(Indexed1).forceBlockingFlush();
+ColumnFamilyStore cfs =
Keyspace.open(PerRowSecondaryIndex).getColumnFamilyStore(Indexed1);
+cfs.forceBlockingFlush();
PerRowSecondaryIndexTest.TestIndex.reset();
@@ -2193,7 +2195,9 @@ public class ColumnFamilyStoreTest extends SchemaLoader
PerRowSecondaryIndexTest.TestIndex.reset();
-PerRowSecondaryIndexTest.TestIndex.ACTIVE = false;
+ColumnDefinition indexedColumnDef =
cfs.metadata.getColumnDefinition(indexedColumnName);
+
cfs.indexManager.getIndexForColumn(indexedColumnName).getColumnDefs().remove(indexedColumnDef);
+
[jira] [Commented] (CASSANDRA-8085) Make PasswordAuthenticator number of hashing rounds configurable
[ https://issues.apache.org/jira/browse/CASSANDRA-8085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523415#comment-14523415 ] Brandon Williams commented on CASSANDRA-8085: - Let's leave it as is and CASSANDRA-7715 can obviate the need. Make PasswordAuthenticator number of hashing rounds configurable Key: CASSANDRA-8085 URL: https://issues.apache.org/jira/browse/CASSANDRA-8085 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Tyler Hobbs Assignee: Sam Tunnicliffe Fix For: 2.0.15, 2.1.5 Attachments: 8085-2.0.txt, 8085-2.1.txt, 8085-3.0.txt Running 2^10 rounds of bcrypt can take a while. In environments (like PHP) where connections are not typically long-lived, authenticating can add substantial overhead. On IRC, one user saw the time to connect, authenticate, and execute a query jump from 5ms to 150ms with authentication enabled ([debug logs|http://pastebin.com/bSUufbr0]). CASSANDRA-7715 is a more complete fix for this, but in the meantime (and even after 7715), this is a good option. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7422) Logging for Authentication and Authorization
[ https://issues.apache.org/jira/browse/CASSANDRA-7422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523425#comment-14523425 ] Przemek commented on CASSANDRA-7422: I believe Adam have a great idea, but Robert have a point to, but we can do it in small steps, and first prepare Client IP trace as Adam shows above, then next someone will propose new idea how to prepare full auditing :) I hope that somebody will reopen this idea in near future. I'm newone on this board, but I'm intrested in very in cassandra and audit features. Logging for Authentication and Authorization Key: CASSANDRA-7422 URL: https://issues.apache.org/jira/browse/CASSANDRA-7422 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Adam Holmberg Priority: Trivial Attachments: auth_logging_remote_host.patch.20140201020 We would like to enable Cassandra to log authentication and authorization change events. This facilitates audits on access to certain data. As a side effect it would also make it easier to notice ill-behaved clients connecting repeatedly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9195) PITR commitlog replay only actually replays mutation every other time
[
https://issues.apache.org/jira/browse/CASSANDRA-9195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523437#comment-14523437
]
Aleksey Yeschenko commented on CASSANDRA-9195:
--
Thanks, Branimir. Committed to 2.1 as
{{f43efaa1bdeec1057a8097fdb5ab62b4c9c19e67}} and merged into trunk, with your
further improvements.
PITR commitlog replay only actually replays mutation every other time
-
Key: CASSANDRA-9195
URL: https://issues.apache.org/jira/browse/CASSANDRA-9195
Project: Cassandra
Issue Type: Bug
Reporter: Jon Moses
Assignee: Branimir Lambov
Fix For: 2.1.6
Attachments: 9195-2.1-v2.patch, 9195-newtest-2.1.txt,
9195-newtest-trunk.txt, 9195-v2.1.patch, loader.py
Version: Cassandra 2.1.4.374 | DSE 4.7.0
The main issue here is that the restore-cycle only replays the mutations
every other try. On the first try, it will restore the snapshot as expected
and the cassandra system load will show that it's reading the mutations, but
they do not actually get replayed, and at the end you're left with only the
snapshot data (2k records).
If you re-run the restore-cycle again, the commitlogs are replayed as
expected,
and the data expected is present in the table (4k records, with a spot check
of
record 4500, as it's in the commitlog but not the snapshot).
Then if you run the cycle again, it will fail. Then again, and it will work.
The work/
not work pattern continues. Even re-running the commitlog replay a 2nd time,
without
reloading the snapshot doesn't work
The load process is:
* Modify commitlog segment to 1mb
* Archive to directory
* create keyspace/table
* insert base data
* initial snapshot
* write more data
* capture timestamp
* write more data
* final snapshot
* copy commitlogs to 2nd location
* modify cassandra-env to replay only specified keyspace
* modify commitlog properties to restore from 2nd location, with noted
timestamp
The restore cycle is:
* truncate table
* sstableload snapshot
* flush
* output data status
* restart to replay commitlogs
* output data status
See attached .py for a mostly automated reproduction scenario. It expects
DSE (and I found it with DSE 4.7.0-1), rather than actual Cassandra, but
it's not using any DSE specific features. The script looks for the configs
in the DSE locations, but they're set at the top, and there's only 2 places
where dse is restarted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[ https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523777#comment-14523777 ] Benedict commented on CASSANDRA-9240: - CompressedPoolingSegmentedFile != CompressedSegmentedFile Doh. But it's interesting, and has demonstrated some unexpected behaviour, since this seems to suggest there could be serious (atypical) performance degradation from an insufficiently sized cache. It _seems_ from the profile that we're incurring costs on reading from the buffer more than anywhere else. Which seems to suggest it may result in the OS actually dropping the pages from its cache. At the very least some cost incurred after remapping is high, in which case calling the cleaner isn't probably a good idea. It seems like the safest thing is to introduce a specialized SegmentedFile for mmapped compressed files, that retains all of the mapped sections to pass into the CompressedRandomAccessReader on construction, so that these costs are guaranteed only ever to occur once per sstable, and not once per reader. This is possible thanks to the recent work of [~stefania_alborghetti], and I think we should exploit it here to ensure this isn't a problem even after my boo-boo is fixed. Performance issue after a restart - Key: CASSANDRA-9240 URL: https://issues.apache.org/jira/browse/CASSANDRA-9240 Project: Cassandra Issue Type: Bug Reporter: Alan Boudreault Assignee: T Jake Luciani Priority: Minor Fix For: 3.x Attachments: Cassandra.snapshots.zip, cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log, cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log, cassandra_trunk-clientrequest-read.log, cassandra_trunk.log, cassandra_trunk_no_restart-clientrequest-read.log, cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, runs.log, trace_query.cql I have noticed a performance issue while I was working on compaction perf tests for CASSANDRA-7409. The performance for my use case is very bad after a restart. It is mostly a read performance issue but not strictly. I have attached my use case (see run_issue.sh and issue.yaml) and all test logs for 2.1.4, 2.1.5 and trunk: * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k ops/second and ~2-2.5k of read latency. * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency. * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5. EDIT: branch cassandra-2.1 is OK. I can help to bisect and/or profile on Monday if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9280) Streaming connections should bind to the broadcast_address of the node
[ https://issues.apache.org/jira/browse/CASSANDRA-9280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523820#comment-14523820 ] Jonathan Ellis commented on CASSANDRA-9280: --- 2.0.x? Streaming connections should bind to the broadcast_address of the node -- Key: CASSANDRA-9280 URL: https://issues.apache.org/jira/browse/CASSANDRA-9280 Project: Cassandra Issue Type: Bug Components: Core Reporter: Richard Low Assignee: Yuki Morishita Priority: Minor Currently, if you have multiple interfaces on a server, a node receiving a stream may show the stream as coming from the wrong IP in e.g. nodetool netstats. The IP is taken as the source of the socket, which may not be the same as the node’s broadcast_address. The outgoing socket should be explicitly bound to the broadcast_address. It seems like this was fixed a long time ago in CASSANDRA-737 but has since broken. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9280) Streaming connections should bind to the broadcast_address of the node
[ https://issues.apache.org/jira/browse/CASSANDRA-9280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-9280: -- Assignee: Yuki Morishita Streaming connections should bind to the broadcast_address of the node -- Key: CASSANDRA-9280 URL: https://issues.apache.org/jira/browse/CASSANDRA-9280 Project: Cassandra Issue Type: Bug Components: Core Reporter: Richard Low Assignee: Yuki Morishita Priority: Minor Currently, if you have multiple interfaces on a server, a node receiving a stream may show the stream as coming from the wrong IP in e.g. nodetool netstats. The IP is taken as the source of the socket, which may not be the same as the node’s broadcast_address. The outgoing socket should be explicitly bound to the broadcast_address. It seems like this was fixed a long time ago in CASSANDRA-737 but has since broken. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[ https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523829#comment-14523829 ] T Jake Luciani commented on CASSANDRA-9240: --- I also set disk_access_mode: standard (so not mmap) and it was also 50% slower Performance issue after a restart - Key: CASSANDRA-9240 URL: https://issues.apache.org/jira/browse/CASSANDRA-9240 Project: Cassandra Issue Type: Bug Reporter: Alan Boudreault Assignee: Benedict Priority: Minor Fix For: 3.x Attachments: Cassandra.snapshots.zip, cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log, cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log, cassandra_trunk-clientrequest-read.log, cassandra_trunk.log, cassandra_trunk_no_restart-clientrequest-read.log, cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, runs.log, trace_query.cql I have noticed a performance issue while I was working on compaction perf tests for CASSANDRA-7409. The performance for my use case is very bad after a restart. It is mostly a read performance issue but not strictly. I have attached my use case (see run_issue.sh and issue.yaml) and all test logs for 2.1.4, 2.1.5 and trunk: * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k ops/second and ~2-2.5k of read latency. * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency. * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5. EDIT: branch cassandra-2.1 is OK. I can help to bisect and/or profile on Monday if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9240) Performance issue after a restart
[ https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] T Jake Luciani updated CASSANDRA-9240: -- Assignee: Benedict (was: T Jake Luciani) Performance issue after a restart - Key: CASSANDRA-9240 URL: https://issues.apache.org/jira/browse/CASSANDRA-9240 Project: Cassandra Issue Type: Bug Reporter: Alan Boudreault Assignee: Benedict Priority: Minor Fix For: 3.x Attachments: Cassandra.snapshots.zip, cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log, cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log, cassandra_trunk-clientrequest-read.log, cassandra_trunk.log, cassandra_trunk_no_restart-clientrequest-read.log, cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, runs.log, trace_query.cql I have noticed a performance issue while I was working on compaction perf tests for CASSANDRA-7409. The performance for my use case is very bad after a restart. It is mostly a read performance issue but not strictly. I have attached my use case (see run_issue.sh and issue.yaml) and all test logs for 2.1.4, 2.1.5 and trunk: * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k ops/second and ~2-2.5k of read latency. * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency. * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5. EDIT: branch cassandra-2.1 is OK. I can help to bisect and/or profile on Monday if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[ https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523880#comment-14523880 ] T Jake Luciani commented on CASSANDRA-9240: --- It's something about the pooling itself... Performance issue after a restart - Key: CASSANDRA-9240 URL: https://issues.apache.org/jira/browse/CASSANDRA-9240 Project: Cassandra Issue Type: Bug Reporter: Alan Boudreault Assignee: Benedict Priority: Minor Fix For: 3.x Attachments: Cassandra.snapshots.zip, cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log, cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log, cassandra_trunk-clientrequest-read.log, cassandra_trunk.log, cassandra_trunk_no_restart-clientrequest-read.log, cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, runs.log, trace_query.cql I have noticed a performance issue while I was working on compaction perf tests for CASSANDRA-7409. The performance for my use case is very bad after a restart. It is mostly a read performance issue but not strictly. I have attached my use case (see run_issue.sh and issue.yaml) and all test logs for 2.1.4, 2.1.5 and trunk: * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k ops/second and ~2-2.5k of read latency. * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency. * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5. EDIT: branch cassandra-2.1 is OK. I can help to bisect and/or profile on Monday if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-9282) Warn on unlogged batches
Jonathan Ellis created CASSANDRA-9282: - Summary: Warn on unlogged batches Key: CASSANDRA-9282 URL: https://issues.apache.org/jira/browse/CASSANDRA-9282 Project: Cassandra Issue Type: Bug Components: API Reporter: Jonathan Ellis Assignee: T Jake Luciani Fix For: 2.1.x At least until CASSANDRA-8303 is done and we can block them entirely, we should log a warning when unlogged batches across multiple partition keys are used. This could either be done by backporting NoSpamLogger and blindly logging every time, or we could add a threshold and warn when more than 10 keys are seen. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-9287) CQLSSTableWriterLongTest is failing
Ariel Weisberg created CASSANDRA-9287:
-
Summary: CQLSSTableWriterLongTest is failing
Key: CASSANDRA-9287
URL: https://issues.apache.org/jira/browse/CASSANDRA-9287
Project: Cassandra
Issue Type: Test
Reporter: Ariel Weisberg
Assignee: Ariel Weisberg
Fix For: 3.x
When I ran it I get an error in setup related the auth keyspace not existing.
It exists, but it's not in the Schema singleton. Lost half a day on this before
looking at other usages of StorageService.instance.initServer() and finding
that the other usages also have
{noformat}
SchemaLoader.cleanupAndLeaveDirs();
Keyspace.setInitialized();
{noformat}
first. So some magic was apparently missing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[2/3] cassandra git commit: Use higher timeout for prepair and snapshot in repair
Use higher timeout for prepair and snapshot in repair
patch by sankalp kohli; reviewed by yukim for CASSANDRA-9261
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9b6f55bd
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9b6f55bd
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9b6f55bd
Branch: refs/heads/trunk
Commit: 9b6f55bdec6d9b7c08d7cae267b2fefbf60d7afc
Parents: f43efaa
Author: sankalp kohli kohlisank...@gmail.com
Authored: Fri May 1 14:46:57 2015 -0500
Committer: Yuki Morishita yu...@apache.org
Committed: Fri May 1 14:48:24 2015 -0500
--
CHANGES.txt| 1 +
src/java/org/apache/cassandra/repair/SnapshotTask.java | 5 +++--
src/java/org/apache/cassandra/service/ActiveRepairService.java | 2 +-
3 files changed, 5 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/9b6f55bd/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index b0e4fc6..627cc6b 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -5,6 +5,7 @@
* Fix deleting from an empty list (CASSANDRA-9198)
* Update tuple and collection types that use a user-defined type when that UDT
is modified (CASSANDRA-9148, CASSANDRA-9192)
+ * Use higher timeout for prepair and snapshot in repair (CASSANDRA-9261)
Merged from 2.0:
* Fix index selection during rebuild with certain table layouts
(CASSANDRA-9281)
* Fix partition-level-delete-only workload accounting (CASSANDRA-9194)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/9b6f55bd/src/java/org/apache/cassandra/repair/SnapshotTask.java
--
diff --git a/src/java/org/apache/cassandra/repair/SnapshotTask.java
b/src/java/org/apache/cassandra/repair/SnapshotTask.java
index 6c3afb1..53e048d 100644
--- a/src/java/org/apache/cassandra/repair/SnapshotTask.java
+++ b/src/java/org/apache/cassandra/repair/SnapshotTask.java
@@ -19,6 +19,7 @@ package org.apache.cassandra.repair;
import java.net.InetAddress;
import java.util.concurrent.RunnableFuture;
+import java.util.concurrent.TimeUnit;
import com.google.common.util.concurrent.AbstractFuture;
@@ -43,9 +44,9 @@ public class SnapshotTask extends AbstractFutureInetAddress
implements Runnabl
public void run()
{
-MessagingService.instance().sendRRWithFailure(new
SnapshotMessage(desc).createMessage(),
+MessagingService.instance().sendRR(new
SnapshotMessage(desc).createMessage(),
endpoint,
-new SnapshotCallback(this));
+new SnapshotCallback(this), TimeUnit.HOURS.toMillis(1), true);
}
/**
http://git-wip-us.apache.org/repos/asf/cassandra/blob/9b6f55bd/src/java/org/apache/cassandra/service/ActiveRepairService.java
--
diff --git a/src/java/org/apache/cassandra/service/ActiveRepairService.java
b/src/java/org/apache/cassandra/service/ActiveRepairService.java
index f71cb6b..ac5ed99 100644
--- a/src/java/org/apache/cassandra/service/ActiveRepairService.java
+++ b/src/java/org/apache/cassandra/service/ActiveRepairService.java
@@ -282,7 +282,7 @@ public class ActiveRepairService
{
PrepareMessage message = new PrepareMessage(parentRepairSession,
cfIds, ranges);
MessageOutRepairMessage msg = message.createMessage();
-MessagingService.instance().sendRRWithFailure(msg, neighbour,
callback);
+MessagingService.instance().sendRR(msg, neighbour, callback,
TimeUnit.HOURS.toMillis(1), true);
}
try
{
[1/3] cassandra git commit: Use higher timeout for prepair and snapshot in repair
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.1 f43efaa1b - 9b6f55bde
refs/heads/trunk 6404e015f - d305ce42f
Use higher timeout for prepair and snapshot in repair
patch by sankalp kohli; reviewed by yukim for CASSANDRA-9261
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9b6f55bd
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9b6f55bd
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9b6f55bd
Branch: refs/heads/cassandra-2.1
Commit: 9b6f55bdec6d9b7c08d7cae267b2fefbf60d7afc
Parents: f43efaa
Author: sankalp kohli kohlisank...@gmail.com
Authored: Fri May 1 14:46:57 2015 -0500
Committer: Yuki Morishita yu...@apache.org
Committed: Fri May 1 14:48:24 2015 -0500
--
CHANGES.txt| 1 +
src/java/org/apache/cassandra/repair/SnapshotTask.java | 5 +++--
src/java/org/apache/cassandra/service/ActiveRepairService.java | 2 +-
3 files changed, 5 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/9b6f55bd/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index b0e4fc6..627cc6b 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -5,6 +5,7 @@
* Fix deleting from an empty list (CASSANDRA-9198)
* Update tuple and collection types that use a user-defined type when that UDT
is modified (CASSANDRA-9148, CASSANDRA-9192)
+ * Use higher timeout for prepair and snapshot in repair (CASSANDRA-9261)
Merged from 2.0:
* Fix index selection during rebuild with certain table layouts
(CASSANDRA-9281)
* Fix partition-level-delete-only workload accounting (CASSANDRA-9194)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/9b6f55bd/src/java/org/apache/cassandra/repair/SnapshotTask.java
--
diff --git a/src/java/org/apache/cassandra/repair/SnapshotTask.java
b/src/java/org/apache/cassandra/repair/SnapshotTask.java
index 6c3afb1..53e048d 100644
--- a/src/java/org/apache/cassandra/repair/SnapshotTask.java
+++ b/src/java/org/apache/cassandra/repair/SnapshotTask.java
@@ -19,6 +19,7 @@ package org.apache.cassandra.repair;
import java.net.InetAddress;
import java.util.concurrent.RunnableFuture;
+import java.util.concurrent.TimeUnit;
import com.google.common.util.concurrent.AbstractFuture;
@@ -43,9 +44,9 @@ public class SnapshotTask extends AbstractFutureInetAddress
implements Runnabl
public void run()
{
-MessagingService.instance().sendRRWithFailure(new
SnapshotMessage(desc).createMessage(),
+MessagingService.instance().sendRR(new
SnapshotMessage(desc).createMessage(),
endpoint,
-new SnapshotCallback(this));
+new SnapshotCallback(this), TimeUnit.HOURS.toMillis(1), true);
}
/**
http://git-wip-us.apache.org/repos/asf/cassandra/blob/9b6f55bd/src/java/org/apache/cassandra/service/ActiveRepairService.java
--
diff --git a/src/java/org/apache/cassandra/service/ActiveRepairService.java
b/src/java/org/apache/cassandra/service/ActiveRepairService.java
index f71cb6b..ac5ed99 100644
--- a/src/java/org/apache/cassandra/service/ActiveRepairService.java
+++ b/src/java/org/apache/cassandra/service/ActiveRepairService.java
@@ -282,7 +282,7 @@ public class ActiveRepairService
{
PrepareMessage message = new PrepareMessage(parentRepairSession,
cfIds, ranges);
MessageOutRepairMessage msg = message.createMessage();
-MessagingService.instance().sendRRWithFailure(msg, neighbour,
callback);
+MessagingService.instance().sendRR(msg, neighbour, callback,
TimeUnit.HOURS.toMillis(1), true);
}
try
{
[3/3] cassandra git commit: Merge branch 'cassandra-2.1' into trunk
Merge branch 'cassandra-2.1' into trunk
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d305ce42
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d305ce42
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d305ce42
Branch: refs/heads/trunk
Commit: d305ce42fc727c6251fc682fcc2168e410e33e99
Parents: 6404e01 9b6f55b
Author: Yuki Morishita yu...@apache.org
Authored: Fri May 1 14:49:01 2015 -0500
Committer: Yuki Morishita yu...@apache.org
Committed: Fri May 1 14:49:01 2015 -0500
--
CHANGES.txt| 1 +
src/java/org/apache/cassandra/repair/SnapshotTask.java | 5 +++--
src/java/org/apache/cassandra/service/ActiveRepairService.java | 2 +-
3 files changed, 5 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/d305ce42/CHANGES.txt
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/d305ce42/src/java/org/apache/cassandra/repair/SnapshotTask.java
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/d305ce42/src/java/org/apache/cassandra/service/ActiveRepairService.java
--
diff --cc src/java/org/apache/cassandra/service/ActiveRepairService.java
index edae0f2,ac5ed99..6f42bce
--- a/src/java/org/apache/cassandra/service/ActiveRepairService.java
+++ b/src/java/org/apache/cassandra/service/ActiveRepairService.java
@@@ -262,11 -278,11 +262,11 @@@ public class ActiveRepairServic
for (ColumnFamilyStore cfs : columnFamilyStores)
cfIds.add(cfs.metadata.cfId);
-for(InetAddress neighbour : endpoints)
+for (InetAddress neighbour : endpoints)
{
-PrepareMessage message = new PrepareMessage(parentRepairSession,
cfIds, ranges);
+PrepareMessage message = new PrepareMessage(parentRepairSession,
cfIds, options.getRanges(), options.isIncremental());
MessageOutRepairMessage msg = message.createMessage();
- MessagingService.instance().sendRRWithFailure(msg, neighbour,
callback);
+ MessagingService.instance().sendRR(msg, neighbour, callback,
TimeUnit.HOURS.toMillis(1), true);
}
try
{
[jira] [Created] (CASSANDRA-9288) LongLeveledCompactionStrategyTest is failing
Ariel Weisberg created CASSANDRA-9288: - Summary: LongLeveledCompactionStrategyTest is failing Key: CASSANDRA-9288 URL: https://issues.apache.org/jira/browse/CASSANDRA-9288 Project: Cassandra Issue Type: Test Reporter: Ariel Weisberg Assignee: Ariel Weisberg Fix For: 3.x Pretty straightforward bad cast followed by some code rot where things return null that didn't used to. I suspect this has been broken since WrappingCompactionStrategy was introduced. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[ https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523875#comment-14523875 ] Benedict commented on CASSANDRA-9240: - Well now that really doesn't make sense, since it should do nothing without mmap. It won't open/close the channel, since that's wrapped by the SegmentedFile, and it uses a heap buffer, so it cannot waste time cleaning it Performance issue after a restart - Key: CASSANDRA-9240 URL: https://issues.apache.org/jira/browse/CASSANDRA-9240 Project: Cassandra Issue Type: Bug Reporter: Alan Boudreault Assignee: Benedict Priority: Minor Fix For: 3.x Attachments: Cassandra.snapshots.zip, cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log, cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log, cassandra_trunk-clientrequest-read.log, cassandra_trunk.log, cassandra_trunk_no_restart-clientrequest-read.log, cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, runs.log, trace_query.cql I have noticed a performance issue while I was working on compaction perf tests for CASSANDRA-7409. The performance for my use case is very bad after a restart. It is mostly a read performance issue but not strictly. I have attached my use case (see run_issue.sh and issue.yaml) and all test logs for 2.1.4, 2.1.5 and trunk: * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k ops/second and ~2-2.5k of read latency. * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency. * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5. EDIT: branch cassandra-2.1 is OK. I can help to bisect and/or profile on Monday if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7486) Compare CMS and G1 pause times
[ https://issues.apache.org/jira/browse/CASSANDRA-7486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523593#comment-14523593 ] Albert P Tobey commented on CASSANDRA-7486: --- if I am reading correctly there was pretty never an old generation collection under the workload I looked at. The old gen was growing but never reached the point it needed to do an old gen GC. ^ G1 doesn't work that way. Another behavior to consider is worst case pause time when there is fragmentation. ^ G1 performs compaction. It's fairly easy to trigger and observe in gc.log with Cassandra 2.0. It takes more work with 2.1 since it seems to be easier on the GC. I'll see if I can find some time to generate graphs to make all this more convincing, but time is short because I'm spending all of my time tuning users' clusters where the #1 first issue every time is getting CMS to behave. Compare CMS and G1 pause times -- Key: CASSANDRA-7486 URL: https://issues.apache.org/jira/browse/CASSANDRA-7486 Project: Cassandra Issue Type: Test Components: Config Reporter: Jonathan Ellis Assignee: Shawn Kumar Fix For: 2.1.x See http://www.slideshare.net/MonicaBeckwith/garbage-first-garbage-collector-g1-7486gc-migration-to-expectations-and-advanced-tuning and https://twitter.com/rbranson/status/482113561431265281 May want to default 2.1 to G1. 2.1 is a different animal from 2.0 after moving most of memtables off heap. Suspect this will help G1 even more than CMS. (NB this is off by default but needs to be part of the test.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9266) Repair failed with error Already repairing SSTableReader(path='/path/to/keyspace/column_family/keyspace-column_family--Data.db'), can not continue.
[
https://issues.apache.org/jira/browse/CASSANDRA-9266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523632#comment-14523632
]
Razi Khaja commented on CASSANDRA-9266:
---
I've now attached log file for all 4 nodes.
Repair failed with error Already repairing
SSTableReader(path='/path/to/keyspace/column_family/keyspace-column_family--Data.db'),
can not continue.
---
Key: CASSANDRA-9266
URL: https://issues.apache.org/jira/browse/CASSANDRA-9266
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: Cassandra 2.1.3
Reporter: Razi Khaja
Fix For: 2.1.x
Attachments: cassandra_system.log.partial.cass320.CASSANDRA-9266.txt,
cassandra_system.log.partial.cass321.CASSANDRA-9266.txt,
cassandra_system.log.partial.cass322.CASSANDRA-9266.txt,
cassandra_system.log.partial.cass323.CASSANDRA-9266.txt
I am running incremental parallel repair using the following command:
{code}
nodetool --host my_hostname repair --incremental --in-local-dc --parallel
{code}
and get the following error:
{code}
Repair failed with error Already repairing
SSTableReader(path='/path/to/keyspace/column_family/keyspace-column_family--Data.db'),
can not continue.
{code}
I have 3 data centers, each with 4 nodes. Neither incremental or full repair
is running on any of my other nodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8915) Improve MergeIterator performance
[
https://issues.apache.org/jira/browse/CASSANDRA-8915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523753#comment-14523753
]
Benedict commented on CASSANDRA-8915:
-
Oh, one other minor quibble: would you mind applying the project codestyle to
your IDE?
Improve MergeIterator performance
-
Key: CASSANDRA-8915
URL: https://issues.apache.org/jira/browse/CASSANDRA-8915
Project: Cassandra
Issue Type: Improvement
Reporter: Branimir Lambov
Assignee: Branimir Lambov
Priority: Minor
Fix For: 3.x
The implementation of {{MergeIterator}} uses a priority queue and applies a
pair of {{poll}}+{{add}} operations for every item in the resulting sequence.
This is quite inefficient as {{poll}} necessarily applies at least {{log N}}
comparisons (up to {{2log N}}), and {{add}} often requires another {{log N}},
for example in the case where the inputs largely don't overlap (where {{N}}
is the number of iterators being merged).
This can easily be replaced with a simple custom structure that can perform
replacement of the top of the queue in a single step, which will very often
complete after a couple of comparisons and in the worst case scenarios will
match the complexity of the current implementation.
This should significantly improve merge performance for iterators with
limited overlap (e.g. levelled compaction).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9048) Delimited File Bulk Loader
[ https://issues.apache.org/jira/browse/CASSANDRA-9048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523551#comment-14523551 ] Jeremy Hanna commented on CASSANDRA-9048: - The loader tool fulfills two purposes. - One is a black box with a lot of [options|https://github.com/brianmhess/cassandra-loader?files=1#options]. These options can be added to COPY FROM over time such as rate limiting, number of threads, boolean/date styles, number of retries, whether the delimiter can be embedded in quotes, and so forth. Decent logging of the process to know what didn't succeed and why would also be helpful. It would be nice to have this built-in and the best tool for the job would be great. - The second purpose is to give people some example code to do asynchronous loading on their own. This will continue to live on and evolve separately I imagine, but it would be nice to merge in a lot of the options. Delimited File Bulk Loader -- Key: CASSANDRA-9048 URL: https://issues.apache.org/jira/browse/CASSANDRA-9048 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Brian Hess Attachments: CASSANDRA-9048.patch There is a strong need for bulk loading data from delimited files into Cassandra. Starting with delimited files means that the data is not currently in the SSTable format, and therefore cannot immediately leverage Cassandra's bulk loading tool, sstableloader, directly. A tool supporting delimited files much closer matches the format of the data more often than the SSTable format itself, and a tool that loads from delimited files is very useful. In order for this bulk loader to be more generally useful to customers, it should handle a number of options at a minimum: - support specifying the input file or to read the data from stdin (so other command-line programs can pipe into the loader) - supply the CQL schema for the input data - support all data types other than collections (collections is a stretch goal/need) - an option to specify the delimiter - an option to specify comma as the decimal delimiter (for international use casese) - an option to specify how NULL values are specified in the file (e.g., the empty string or the string NULL) - an option to specify how BOOLEAN values are specified in the file (e.g., TRUE/FALSE or 0/1) - an option to specify the Date and Time format - an option to skip some number of rows at the beginning of the file - an option to only read in some number of rows from the file - an option to indicate how many parse errors to tolerate - an option to specify a file that will contain all the lines that did not parse correctly (up to the maximum number of parse errors) - an option to specify the CQL port to connect to (with 9042 as the default). Additional options would be useful, but this set of options/features is a start. A word on COPY. COPY comes via CQLSH which requires the client to be the same version as the server (e.g., 2.0 CQLSH does not work with 2.1 Cassandra, etc). This tool should be able to connect to any version of Cassandra (within reason). For example, it should be able to handle 2.0.x and 2.1.x. Moreover, CQLSH's COPY command does not support a number of the options above. Lastly, the performance of COPY in 2.0.x is not high enough to be considered a bulk ingest tool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9286) Add Keyspace/Table details to CollectionType.java error message
[
https://issues.apache.org/jira/browse/CASSANDRA-9286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philip Thompson updated CASSANDRA-9286:
---
Assignee: Carl Yeksigian
Add Keyspace/Table details to CollectionType.java error message
---
Key: CASSANDRA-9286
URL: https://issues.apache.org/jira/browse/CASSANDRA-9286
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: sequoyha pelletier
Assignee: Carl Yeksigian
Priority: Minor
The error message for too many element in a collection does not give keyspace
or column family information. This makes it a pain point to try to determine
which table is the offending table.
Example Error message:
{noformat}
ERROR [Native-Transport-Requests:809453] 2015-04-23 22:48:21,189
CollectionType.java (line 116) Detected collection with 136234 elements, more
than the 65535 limit. Only the first 65535 elements will be returned to the
client. Please see http://cassandra.apache.org/doc/cql3/CQL.html#collections
for more details.
{noformat}
Currently, to try to pinpoint the table in question. We need to trace all
requests and then try to match up the timestamps in the CQL tracing session
with the log timestamps to try and match. If prepared statements are used,
this is a dead end due to the logged tracing information missing the query.
In which case, we have to look at other 3rd party methods for capturing the
queries to try and match up. This is extremely tedious when many tables have
collections and a high number of ops against them.
Requesting that the error contain the keyspace.table name in the error
message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-9048) Delimited File Bulk Loader
[ https://issues.apache.org/jira/browse/CASSANDRA-9048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523551#comment-14523551 ] Jeremy Hanna edited comment on CASSANDRA-9048 at 5/1/15 6:14 PM: - The loader tool fulfills two purposes. - One is a black box with a lot of [options|https://github.com/brianmhess/cassandra-loader?files=1#options]. These options can be added to COPY FROM over time such as rate limiting, number of threads, boolean/date styles, number of retries, whether the delimiter can be embedded in quotes, and so forth. Decent logging of the process to know what didn't succeed and why would also be helpful. - The second purpose is to give people some example code to do asynchronous loading on their own. This will continue to live on and evolve separately I imagine, but it would be nice to merge in a lot of the options. was (Author: jeromatron): The loader tool fulfills two purposes. - One is a black box with a lot of [options|https://github.com/brianmhess/cassandra-loader?files=1#options]. These options can be added to COPY FROM over time such as rate limiting, number of threads, boolean/date styles, number of retries, whether the delimiter can be embedded in quotes, and so forth. Decent logging of the process to know what didn't succeed and why would also be helpful. It would be nice to have this built-in and the best tool for the job would be great. - The second purpose is to give people some example code to do asynchronous loading on their own. This will continue to live on and evolve separately I imagine, but it would be nice to merge in a lot of the options. Delimited File Bulk Loader -- Key: CASSANDRA-9048 URL: https://issues.apache.org/jira/browse/CASSANDRA-9048 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Brian Hess Attachments: CASSANDRA-9048.patch There is a strong need for bulk loading data from delimited files into Cassandra. Starting with delimited files means that the data is not currently in the SSTable format, and therefore cannot immediately leverage Cassandra's bulk loading tool, sstableloader, directly. A tool supporting delimited files much closer matches the format of the data more often than the SSTable format itself, and a tool that loads from delimited files is very useful. In order for this bulk loader to be more generally useful to customers, it should handle a number of options at a minimum: - support specifying the input file or to read the data from stdin (so other command-line programs can pipe into the loader) - supply the CQL schema for the input data - support all data types other than collections (collections is a stretch goal/need) - an option to specify the delimiter - an option to specify comma as the decimal delimiter (for international use casese) - an option to specify how NULL values are specified in the file (e.g., the empty string or the string NULL) - an option to specify how BOOLEAN values are specified in the file (e.g., TRUE/FALSE or 0/1) - an option to specify the Date and Time format - an option to skip some number of rows at the beginning of the file - an option to only read in some number of rows from the file - an option to indicate how many parse errors to tolerate - an option to specify a file that will contain all the lines that did not parse correctly (up to the maximum number of parse errors) - an option to specify the CQL port to connect to (with 9042 as the default). Additional options would be useful, but this set of options/features is a start. A word on COPY. COPY comes via CQLSH which requires the client to be the same version as the server (e.g., 2.0 CQLSH does not work with 2.1 Cassandra, etc). This tool should be able to connect to any version of Cassandra (within reason). For example, it should be able to handle 2.0.x and 2.1.x. Moreover, CQLSH's COPY command does not support a number of the options above. Lastly, the performance of COPY in 2.0.x is not high enough to be considered a bulk ingest tool. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7486) Compare CMS and G1 pause times
[ https://issues.apache.org/jira/browse/CASSANDRA-7486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523630#comment-14523630 ] Ariel Weisberg commented on CASSANDRA-7486: --- bq. ^ G1 doesn't work that way. I am talking about CMS. When I looked at the 12 gigabyte heap the old gen grew to 4.1 gigabytes and I didn't see any point that it shrunk. bq. I'm spending all of my time tuning users' clusters where the #1 first issue every time is getting CMS to behave. We can make the case for G1 in different ways. If we want to do it based on real world results that is fine with me. To Benedict's point I think looking at all the operations we care about on realistic time scales is something we would have to do to really know what the differences are. I wish we had this stuff in CI so it would just be a matter of changing the flags, but we aren't there yet. Compare CMS and G1 pause times -- Key: CASSANDRA-7486 URL: https://issues.apache.org/jira/browse/CASSANDRA-7486 Project: Cassandra Issue Type: Test Components: Config Reporter: Jonathan Ellis Assignee: Shawn Kumar Fix For: 2.1.x See http://www.slideshare.net/MonicaBeckwith/garbage-first-garbage-collector-g1-7486gc-migration-to-expectations-and-advanced-tuning and https://twitter.com/rbranson/status/482113561431265281 May want to default 2.1 to G1. 2.1 is a different animal from 2.0 after moving most of memtables off heap. Suspect this will help G1 even more than CMS. (NB this is off by default but needs to be part of the test.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9266) Repair failed with error Already repairing SSTableReader(path='/path/to/keyspace/column_family/keyspace-column_family--Data.db'), can not continue.
[
https://issues.apache.org/jira/browse/CASSANDRA-9266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Razi Khaja updated CASSANDRA-9266:
--
Attachment: (was: cassandra_system.log.partial.CASSANDRA-9266.txt)
Repair failed with error Already repairing
SSTableReader(path='/path/to/keyspace/column_family/keyspace-column_family--Data.db'),
can not continue.
---
Key: CASSANDRA-9266
URL: https://issues.apache.org/jira/browse/CASSANDRA-9266
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: Cassandra 2.1.3
Reporter: Razi Khaja
Fix For: 2.1.x
Attachments: cassandra_system.log.partial.cass320.CASSANDRA-9266.txt,
cassandra_system.log.partial.cass321.CASSANDRA-9266.txt,
cassandra_system.log.partial.cass322.CASSANDRA-9266.txt,
cassandra_system.log.partial.cass323.CASSANDRA-9266.txt
I am running incremental parallel repair using the following command:
{code}
nodetool --host my_hostname repair --incremental --in-local-dc --parallel
{code}
and get the following error:
{code}
Repair failed with error Already repairing
SSTableReader(path='/path/to/keyspace/column_family/keyspace-column_family--Data.db'),
can not continue.
{code}
I have 3 data centers, each with 4 nodes. Neither incremental or full repair
is running on any of my other nodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9266) Repair failed with error Already repairing SSTableReader(path='/path/to/keyspace/column_family/keyspace-column_family--Data.db'), can not continue.
[
https://issues.apache.org/jira/browse/CASSANDRA-9266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Razi Khaja updated CASSANDRA-9266:
--
Attachment: cassandra_system.log.partial.cass323.CASSANDRA-9266.txt
cassandra_system.log.partial.cass322.CASSANDRA-9266.txt
cassandra_system.log.partial.cass321.CASSANDRA-9266.txt
cassandra_system.log.partial.cass320.CASSANDRA-9266.txt
Repair failed with error Already repairing
SSTableReader(path='/path/to/keyspace/column_family/keyspace-column_family--Data.db'),
can not continue.
---
Key: CASSANDRA-9266
URL: https://issues.apache.org/jira/browse/CASSANDRA-9266
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: Cassandra 2.1.3
Reporter: Razi Khaja
Fix For: 2.1.x
Attachments: cassandra_system.log.partial.cass320.CASSANDRA-9266.txt,
cassandra_system.log.partial.cass321.CASSANDRA-9266.txt,
cassandra_system.log.partial.cass322.CASSANDRA-9266.txt,
cassandra_system.log.partial.cass323.CASSANDRA-9266.txt
I am running incremental parallel repair using the following command:
{code}
nodetool --host my_hostname repair --incremental --in-local-dc --parallel
{code}
and get the following error:
{code}
Repair failed with error Already repairing
SSTableReader(path='/path/to/keyspace/column_family/keyspace-column_family--Data.db'),
can not continue.
{code}
I have 3 data centers, each with 4 nodes. Neither incremental or full repair
is running on any of my other nodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9151) Anti-compaction is blocking ANTI_ENTROPY stage
[ https://issues.apache.org/jira/browse/CASSANDRA-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523706#comment-14523706 ] Yuki Morishita commented on CASSANDRA-9151: --- Thanks for review. Let's open another JIRA for ParentRepairSession issue. Anti-compaction is blocking ANTI_ENTROPY stage --- Key: CASSANDRA-9151 URL: https://issues.apache.org/jira/browse/CASSANDRA-9151 Project: Cassandra Issue Type: Bug Reporter: sankalp kohli Assignee: Yuki Morishita Fix For: 3.x, 2.1.x Attachments: 0001-Do-not-block-stage-when-performing-anticompaction.patch In RepairMessageVerbHandler.doVerb case ANTICOMPACTION_REQUEST We block on ANTI_ENTROPY stage. This is not good as no other message could be processed till any anticompaction is running. This can cause snapshots for new repairs to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7486) Compare CMS and G1 pause times
[ https://issues.apache.org/jira/browse/CASSANDRA-7486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523724#comment-14523724 ] Albert P Tobey commented on CASSANDRA-7486: --- I only started messing with G1 this year, so I only know the old behavior by lore I've read and heard. I have not observed significant problems it in the ~20-40 hours I've spent tuning clusters with G1 recently. I don't recommend anyone try G1 on JDK 7 u75 or JDK 8 u40 (although it's probably OK down to u20 according to the docs I've read). I did some testing on JDK7u75 and it was stable but didn't spend much time on it since JDK8u40 gave a nice bump in performance (5-10% on a customer cluster) by just switching JDKs and nothing else. From what I've read about the reference clearing issues, there is a new-ish setting to enable parallel reference collection, -XX:+ParallelRefProcEnabled. The advice in the docs is to only turn it on if a significant amount of time is spent on RefProc collection, e.g. [Ref Proc: 5.2 ms]. I pulled that from a log I had handy and that is high enough that we might want to consider enabling the flag, but in most of my observations it hovers around 0.1ms under saturation load. Compare CMS and G1 pause times -- Key: CASSANDRA-7486 URL: https://issues.apache.org/jira/browse/CASSANDRA-7486 Project: Cassandra Issue Type: Test Components: Config Reporter: Jonathan Ellis Assignee: Shawn Kumar Fix For: 2.1.x See http://www.slideshare.net/MonicaBeckwith/garbage-first-garbage-collector-g1-7486gc-migration-to-expectations-and-advanced-tuning and https://twitter.com/rbranson/status/482113561431265281 May want to default 2.1 to G1. 2.1 is a different animal from 2.0 after moving most of memtables off heap. Suspect this will help G1 even more than CMS. (NB this is off by default but needs to be part of the test.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7486) Compare CMS and G1 pause times
[ https://issues.apache.org/jira/browse/CASSANDRA-7486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523736#comment-14523736 ] Benedict commented on CASSANDRA-7486: - bq. there is a new-ish setting to enable parallel reference collection Throughput was something like 5% of other collectors in my testing, so parallelizing this would only help so much! :) My point is, we don't really fully understand G1, and unless we undertake a research project to fully understand its pathological cases, and how they compare/contrast to its history, I'd prefer we ensured it behaved under complex loads, and not just under isolated read or write loading. Compare CMS and G1 pause times -- Key: CASSANDRA-7486 URL: https://issues.apache.org/jira/browse/CASSANDRA-7486 Project: Cassandra Issue Type: Test Components: Config Reporter: Jonathan Ellis Assignee: Shawn Kumar Fix For: 2.1.x See http://www.slideshare.net/MonicaBeckwith/garbage-first-garbage-collector-g1-7486gc-migration-to-expectations-and-advanced-tuning and https://twitter.com/rbranson/status/482113561431265281 May want to default 2.1 to G1. 2.1 is a different animal from 2.0 after moving most of memtables off heap. Suspect this will help G1 even more than CMS. (NB this is off by default but needs to be part of the test.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[ https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523690#comment-14523690 ] T Jake Luciani commented on CASSANDRA-9240: --- I bisected the issue to CASSANDRA-9112 Not sure what could cause the issue there? [~benedict] Performance issue after a restart - Key: CASSANDRA-9240 URL: https://issues.apache.org/jira/browse/CASSANDRA-9240 Project: Cassandra Issue Type: Bug Reporter: Alan Boudreault Assignee: T Jake Luciani Priority: Minor Fix For: 3.x Attachments: Cassandra.snapshots.zip, cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log, cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log, cassandra_trunk-clientrequest-read.log, cassandra_trunk.log, cassandra_trunk_no_restart-clientrequest-read.log, cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, runs.log, trace_query.cql I have noticed a performance issue while I was working on compaction perf tests for CASSANDRA-7409. The performance for my use case is very bad after a restart. It is mostly a read performance issue but not strictly. I have attached my use case (see run_issue.sh and issue.yaml) and all test logs for 2.1.4, 2.1.5 and trunk: * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k ops/second and ~2-2.5k of read latency. * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency. * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5. EDIT: branch cassandra-2.1 is OK. I can help to bisect and/or profile on Monday if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8915) Improve MergeIterator performance
[
https://issues.apache.org/jira/browse/CASSANDRA-8915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523745#comment-14523745
]
Benedict commented on CASSANDRA-8915:
-
First off, I agree that the MergeIteratorAllEqual is unfortunately complex,
though I haven't yet attempted to fully reason about its behaviour. Are we
certain there's no way to reduce its complexity? If you don't think it can be
improved complexity-wise from its current position, I'll find some time to chew
on it myself as well, so we have at least had two distinct attempts to reduce
that burden.
However from a performance point of view, I have two major quibbles:
* There is a simple optimisation of AllEqual, which I have introduced, and it
is now universally faster (or as fast)
* I don't _think_ that's the best characterisation of LCS (please feel free to
prove me wrong, though, as I may have a poor mental model of LCS)
My optimisation as stands is actually pretty rubbish; it could be done much
better. What I do is construct a new heap from the equal candidates, and then
push down the top element and bubble up the remainder. All this does really is
compress the equal items, push down the one most likely to stay at the head,
and expand the remainder from the smallest item upwards so minimizing the
number of swaps. We could instead perform a merge, but even with this imperfect
implementation the result is typically half as many comparisons as NoEqual, and
never more.
As to LCS: to my knowledge, it only ever compacts two levels together, but
within each level there is no compaction/comparison, they are simply
concatenated together. Between the levels there's no guarantee of overlap, and
the values will typically be randomly distributed since we default to hashed
partitioning. Given each level is 10 times larger than the previous, we can
also expect that the number of _runs_ of equality are very low. I've introduced
simulation of varying numbers of levels being compacted together, along with
varying numbers of L0 sstables in the mix, and with varying levels of overlap
between the levels (100%, 50%, 0%; here I mean percentage of values in a level
which definitely occur in a lower level). These all come out as either the same
or faster under both versions of AllEqual.
Of course the statement likely stands for a variety of workloads with many CQL
row overwrites, so catering for the case of many runs of overlaps is still an
important one.
bq. but please do not hesitate to disagree or bring other
concerns/questions/scenarios to the discussion.
ditto :)
Improve MergeIterator performance
-
Key: CASSANDRA-8915
URL: https://issues.apache.org/jira/browse/CASSANDRA-8915
Project: Cassandra
Issue Type: Improvement
Reporter: Branimir Lambov
Assignee: Branimir Lambov
Priority: Minor
Fix For: 3.x
The implementation of {{MergeIterator}} uses a priority queue and applies a
pair of {{poll}}+{{add}} operations for every item in the resulting sequence.
This is quite inefficient as {{poll}} necessarily applies at least {{log N}}
comparisons (up to {{2log N}}), and {{add}} often requires another {{log N}},
for example in the case where the inputs largely don't overlap (where {{N}}
is the number of iterators being merged).
This can easily be replaced with a simple custom structure that can perform
replacement of the top of the queue in a single step, which will very often
complete after a couple of comparisons and in the worst case scenarios will
match the complexity of the current implementation.
This should significantly improve merge performance for iterators with
limited overlap (e.g. levelled compaction).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7486) Compare CMS and G1 pause times
[ https://issues.apache.org/jira/browse/CASSANDRA-7486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523621#comment-14523621 ] Benedict commented on CASSANDRA-7486: - bq. ^ G1 doesn't work that way. While it has no old generation, it does promote regions and if this happens a lot you can get some weird pathological fragmentation. Now, my experience with G1 is out of date, and I haven't kept up at all with its latest behaviours, but I saw some really atrocious behaviour on very simple benchmarks a few years back. At the time, If you modified references that were randomly distributed around the heap, it required traversing a majority of the heap to collect very little, and essentially thrashed. I realise it has improved, but I do not know in what ways, and so I'm wary of making it the default without being certain it no longer has pathological cases that are a problem for us. Unless we stress the collector so that it exercises its suboptimal characteristics, I am not really super confident. I hope this is simply overly cautious, but we know of users who also had serious problems with sudden degradation despite looking good in initial testing, and it would be great for that not to be a widespread problem. Compare CMS and G1 pause times -- Key: CASSANDRA-7486 URL: https://issues.apache.org/jira/browse/CASSANDRA-7486 Project: Cassandra Issue Type: Test Components: Config Reporter: Jonathan Ellis Assignee: Shawn Kumar Fix For: 2.1.x See http://www.slideshare.net/MonicaBeckwith/garbage-first-garbage-collector-g1-7486gc-migration-to-expectations-and-advanced-tuning and https://twitter.com/rbranson/status/482113561431265281 May want to default 2.1 to G1. 2.1 is a different animal from 2.0 after moving most of memtables off heap. Suspect this will help G1 even more than CMS. (NB this is off by default but needs to be part of the test.) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8897) Remove FileCacheService, instead pooling the buffers
[ https://issues.apache.org/jira/browse/CASSANDRA-8897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14522993#comment-14522993 ] Benedict commented on CASSANDRA-8897: - [~stefania_alborghetti]: I'd just like to make it more obvious that all of my changes are _suggestions_, and that I also did not fully test the changes I made, just confirmed they were viable enough to serve as suggestions/pointers. Remove FileCacheService, instead pooling the buffers Key: CASSANDRA-8897 URL: https://issues.apache.org/jira/browse/CASSANDRA-8897 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Benedict Assignee: Stefania Fix For: 3.x After CASSANDRA-8893, a RAR will be a very lightweight object and will not need caching, so we can eliminate this cache entirely. Instead we should have a pool of buffers that are page-aligned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7066) Simplify (and unify) cleanup of compaction leftovers
[ https://issues.apache.org/jira/browse/CASSANDRA-7066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14522998#comment-14522998 ] Stefania commented on CASSANDRA-7066: - To give more context you can look at the usages of desc.type.isTemporary, for example in SSTableLoader.openSSTables, CFS.loadNewSSTables(), CFS.scrubDataDirectories() and Directories.SSTableLister. I'm not necessarily against it but before I go ahead and remove temporary desc types I want to be absolutely sure this is 100% OK. To replayce desc.type.isTemporary we have to look for any transaction logs for the CF, read the list of files from these logs and see if the file name is one of them. It's fine as long as the transaction logs can be found, else we may end up loading incomplete or corrupt files. Simplify (and unify) cleanup of compaction leftovers Key: CASSANDRA-7066 URL: https://issues.apache.org/jira/browse/CASSANDRA-7066 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Benedict Assignee: Stefania Priority: Minor Labels: compaction Fix For: 3.x Currently we manage a list of in-progress compactions in a system table, which we use to cleanup incomplete compactions when we're done. The problem with this is that 1) it's a bit clunky (and leaves us in positions where we can unnecessarily cleanup completed files, or conversely not cleanup files that have been superceded); and 2) it's only used for a regular compaction - no other compaction types are guarded in the same way, so can result in duplication if we fail before deleting the replacements. I'd like to see each sstable store in its metadata its direct ancestors, and on startup we simply delete any sstables that occur in the union of all ancestor sets. This way as soon as we finish writing we're capable of cleaning up any leftovers, so we never get duplication. It's also much easier to reason about. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[5/6] cassandra git commit: Merge branch 'cassandra-2.0' into cassandra-2.1
Merge branch 'cassandra-2.0' into cassandra-2.1
Conflicts:
CHANGES.txt
src/java/org/apache/cassandra/db/Memtable.java
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4f9c9ce5
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4f9c9ce5
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4f9c9ce5
Branch: refs/heads/cassandra-2.1
Commit: 4f9c9ce55fd9993d06bf6dc4d9193e973fb8515f
Parents: db127a1 f0c7a6f
Author: Benedict Elliott Smith bened...@apache.org
Authored: Fri May 1 11:33:40 2015 +0100
Committer: Benedict Elliott Smith bened...@apache.org
Committed: Fri May 1 11:33:40 2015 +0100
--
CHANGES.txt| 1 +
src/java/org/apache/cassandra/db/Memtable.java | 4 +++-
2 files changed, 4 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/4f9c9ce5/CHANGES.txt
--
diff --cc CHANGES.txt
index c063368,dfdad51..59b1182
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,10 -1,5 +1,11 @@@
-2.0.15:
+2.1.6
+ * Fix PITR commitlog replay (CASSANDRA-9195)
+ * GCInspector logs very different times (CASSANDRA-9124)
+ * Fix deleting from an empty list (CASSANDRA-9198)
+ * Update tuple and collection types that use a user-defined type when that
UDT
+ is modified (CASSANDRA-9148, CASSANDRA-9192)
+Merged from 2.0:
+ * Fix partition-level-delete-only workload accounting (CASSANDRA-9194)
* Allow scrub to handle corrupted compressed chunks (CASSANDRA-9140)
* Fix assertion error when resetlocalschema is run during repair
(CASSANDRA-9249)
* Disable single sstable tombstone compactions for DTCS by default
(CASSANDRA-9234)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/4f9c9ce5/src/java/org/apache/cassandra/db/Memtable.java
--
diff --cc src/java/org/apache/cassandra/db/Memtable.java
index eb04bea,897d94e..9a8596a
--- a/src/java/org/apache/cassandra/db/Memtable.java
+++ b/src/java/org/apache/cassandra/db/Memtable.java
@@@ -151,62 -182,45 +151,64 @@@ public class Memtabl
}
}
-public void updateLiveRatio() throws RuntimeException
+public boolean isLive()
{
-if (!MemoryMeter.isInitialized())
-{
-// hack for openjdk. we log a warning about this in the startup
script too.
-logger.error(MemoryMeter uninitialized (jamm not specified as
java agent); assuming liveRatio of {}.
- + Usually this means cassandra-env.sh disabled
jamm because you are using a buggy JRE;
- + upgrade to the Sun JRE instead, liveRatio);
-return;
-}
+return allocator.isLive();
+}
-if (!meteringInProgress.add(cfs))
-{
-logger.debug(Metering already pending or active for {}; skipping
liveRatio update, cfs);
-return;
-}
+public boolean isClean()
+{
+return rows.isEmpty();
+}
-meterExecutor.submit(new MeteringRunnable(cfs));
+public boolean isCleanAfter(ReplayPosition position)
+{
+return isClean() || (position != null
minReplayPosition.compareTo(position) = 0);
}
-private long resolve(DecoratedKey key, ColumnFamily cf,
SecondaryIndexManager.Updater indexer)
+/**
+ * @return true if this memtable is expired. Expiration time is
determined by CF's memtable_flush_period_in_ms.
+ */
+public boolean isExpired()
+{
+int period = cfs.metadata.getMemtableFlushPeriod();
+return period 0 (System.nanoTime() - creationNano =
TimeUnit.MILLISECONDS.toNanos(period));
+}
+
+/**
+ * Should only be called by ColumnFamilyStore.apply via Keyspace.apply,
which supplies the appropriate
+ * OpOrdering.
+ *
+ * replayPosition should only be null if this is a secondary index, in
which case it is *expected* to be null
+ */
+long put(DecoratedKey key, ColumnFamily cf, SecondaryIndexManager.Updater
indexer, OpOrder.Group opGroup)
{
-AtomicSortedColumns previous = rows.get(key);
+AtomicBTreeColumns previous = rows.get(key);
+ long initialSize = 0;
if (previous == null)
{
-AtomicSortedColumns empty =
cf.cloneMeShallow(AtomicSortedColumns.factory, false);
+AtomicBTreeColumns empty =
cf.cloneMeShallow(AtomicBTreeColumns.factory, false);
+final DecoratedKey cloneKey = allocator.clone(key, opGroup);
// We'll add the columns later. This avoids wasting works if we
get beaten in the putIfAbsent
-previous = rows.putIfAbsent(new
[1/6] cassandra git commit: Fix partition-level-delete-only workload accounting
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.0 593a7257b - f0c7a6f03
refs/heads/cassandra-2.1 db127a107 - 4f9c9ce55
refs/heads/trunk 49b089893 - 527c12a01
Fix partition-level-delete-only workload accounting
patch by benedict; reviewed by jbellis for CASSANDRA-9194
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f0c7a6f0
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f0c7a6f0
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f0c7a6f0
Branch: refs/heads/cassandra-2.0
Commit: f0c7a6f03cf55ddcfe11a58c80087ed8b0806cf4
Parents: 593a725
Author: Benedict Elliott Smith bened...@apache.org
Authored: Fri May 1 11:32:31 2015 +0100
Committer: Benedict Elliott Smith bened...@apache.org
Committed: Fri May 1 11:32:31 2015 +0100
--
CHANGES.txt| 1 +
src/java/org/apache/cassandra/db/Memtable.java | 6 +-
2 files changed, 6 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/f0c7a6f0/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 4e7a5d0..dfdad51 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.0.15:
+ * Fix partition-level-delete-only workload accounting (CASSANDRA-9194)
* Allow scrub to handle corrupted compressed chunks (CASSANDRA-9140)
* Fix assertion error when resetlocalschema is run during repair
(CASSANDRA-9249)
* Disable single sstable tombstone compactions for DTCS by default
(CASSANDRA-9234)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/f0c7a6f0/src/java/org/apache/cassandra/db/Memtable.java
--
diff --git a/src/java/org/apache/cassandra/db/Memtable.java
b/src/java/org/apache/cassandra/db/Memtable.java
index 19f38be..897d94e 100644
--- a/src/java/org/apache/cassandra/db/Memtable.java
+++ b/src/java/org/apache/cassandra/db/Memtable.java
@@ -206,17 +206,21 @@ public class Memtable
{
AtomicSortedColumns previous = rows.get(key);
+long initialSize = 0;
if (previous == null)
{
AtomicSortedColumns empty =
cf.cloneMeShallow(AtomicSortedColumns.factory, false);
// We'll add the columns later. This avoids wasting works if we
get beaten in the putIfAbsent
previous = rows.putIfAbsent(new DecoratedKey(key.token,
allocator.clone(key.key)), empty);
if (previous == null)
+{
previous = empty;
+initialSize += 1;
+}
}
final PairLong, Long pair = previous.addAllWithSizeDelta(cf,
allocator, localCopyFunction, indexer);
-currentSize.addAndGet(pair.left);
+currentSize.addAndGet(initialSize + pair.left);
currentOperations.addAndGet(cf.getColumnCount() +
(cf.isMarkedForDelete() ? 1 : 0) + cf.deletionInfo().rangeCount());
return pair.right;
}
[4/6] cassandra git commit: Merge branch 'cassandra-2.0' into cassandra-2.1
Merge branch 'cassandra-2.0' into cassandra-2.1
Conflicts:
CHANGES.txt
src/java/org/apache/cassandra/db/Memtable.java
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4f9c9ce5
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4f9c9ce5
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4f9c9ce5
Branch: refs/heads/trunk
Commit: 4f9c9ce55fd9993d06bf6dc4d9193e973fb8515f
Parents: db127a1 f0c7a6f
Author: Benedict Elliott Smith bened...@apache.org
Authored: Fri May 1 11:33:40 2015 +0100
Committer: Benedict Elliott Smith bened...@apache.org
Committed: Fri May 1 11:33:40 2015 +0100
--
CHANGES.txt| 1 +
src/java/org/apache/cassandra/db/Memtable.java | 4 +++-
2 files changed, 4 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/4f9c9ce5/CHANGES.txt
--
diff --cc CHANGES.txt
index c063368,dfdad51..59b1182
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,10 -1,5 +1,11 @@@
-2.0.15:
+2.1.6
+ * Fix PITR commitlog replay (CASSANDRA-9195)
+ * GCInspector logs very different times (CASSANDRA-9124)
+ * Fix deleting from an empty list (CASSANDRA-9198)
+ * Update tuple and collection types that use a user-defined type when that
UDT
+ is modified (CASSANDRA-9148, CASSANDRA-9192)
+Merged from 2.0:
+ * Fix partition-level-delete-only workload accounting (CASSANDRA-9194)
* Allow scrub to handle corrupted compressed chunks (CASSANDRA-9140)
* Fix assertion error when resetlocalschema is run during repair
(CASSANDRA-9249)
* Disable single sstable tombstone compactions for DTCS by default
(CASSANDRA-9234)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/4f9c9ce5/src/java/org/apache/cassandra/db/Memtable.java
--
diff --cc src/java/org/apache/cassandra/db/Memtable.java
index eb04bea,897d94e..9a8596a
--- a/src/java/org/apache/cassandra/db/Memtable.java
+++ b/src/java/org/apache/cassandra/db/Memtable.java
@@@ -151,62 -182,45 +151,64 @@@ public class Memtabl
}
}
-public void updateLiveRatio() throws RuntimeException
+public boolean isLive()
{
-if (!MemoryMeter.isInitialized())
-{
-// hack for openjdk. we log a warning about this in the startup
script too.
-logger.error(MemoryMeter uninitialized (jamm not specified as
java agent); assuming liveRatio of {}.
- + Usually this means cassandra-env.sh disabled
jamm because you are using a buggy JRE;
- + upgrade to the Sun JRE instead, liveRatio);
-return;
-}
+return allocator.isLive();
+}
-if (!meteringInProgress.add(cfs))
-{
-logger.debug(Metering already pending or active for {}; skipping
liveRatio update, cfs);
-return;
-}
+public boolean isClean()
+{
+return rows.isEmpty();
+}
-meterExecutor.submit(new MeteringRunnable(cfs));
+public boolean isCleanAfter(ReplayPosition position)
+{
+return isClean() || (position != null
minReplayPosition.compareTo(position) = 0);
}
-private long resolve(DecoratedKey key, ColumnFamily cf,
SecondaryIndexManager.Updater indexer)
+/**
+ * @return true if this memtable is expired. Expiration time is
determined by CF's memtable_flush_period_in_ms.
+ */
+public boolean isExpired()
+{
+int period = cfs.metadata.getMemtableFlushPeriod();
+return period 0 (System.nanoTime() - creationNano =
TimeUnit.MILLISECONDS.toNanos(period));
+}
+
+/**
+ * Should only be called by ColumnFamilyStore.apply via Keyspace.apply,
which supplies the appropriate
+ * OpOrdering.
+ *
+ * replayPosition should only be null if this is a secondary index, in
which case it is *expected* to be null
+ */
+long put(DecoratedKey key, ColumnFamily cf, SecondaryIndexManager.Updater
indexer, OpOrder.Group opGroup)
{
-AtomicSortedColumns previous = rows.get(key);
+AtomicBTreeColumns previous = rows.get(key);
+ long initialSize = 0;
if (previous == null)
{
-AtomicSortedColumns empty =
cf.cloneMeShallow(AtomicSortedColumns.factory, false);
+AtomicBTreeColumns empty =
cf.cloneMeShallow(AtomicBTreeColumns.factory, false);
+final DecoratedKey cloneKey = allocator.clone(key, opGroup);
// We'll add the columns later. This avoids wasting works if we
get beaten in the putIfAbsent
-previous = rows.putIfAbsent(new
[2/6] cassandra git commit: Fix partition-level-delete-only workload accounting
Fix partition-level-delete-only workload accounting
patch by benedict; reviewed by jbellis for CASSANDRA-9194
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f0c7a6f0
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f0c7a6f0
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f0c7a6f0
Branch: refs/heads/cassandra-2.1
Commit: f0c7a6f03cf55ddcfe11a58c80087ed8b0806cf4
Parents: 593a725
Author: Benedict Elliott Smith bened...@apache.org
Authored: Fri May 1 11:32:31 2015 +0100
Committer: Benedict Elliott Smith bened...@apache.org
Committed: Fri May 1 11:32:31 2015 +0100
--
CHANGES.txt| 1 +
src/java/org/apache/cassandra/db/Memtable.java | 6 +-
2 files changed, 6 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/f0c7a6f0/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 4e7a5d0..dfdad51 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.0.15:
+ * Fix partition-level-delete-only workload accounting (CASSANDRA-9194)
* Allow scrub to handle corrupted compressed chunks (CASSANDRA-9140)
* Fix assertion error when resetlocalschema is run during repair
(CASSANDRA-9249)
* Disable single sstable tombstone compactions for DTCS by default
(CASSANDRA-9234)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/f0c7a6f0/src/java/org/apache/cassandra/db/Memtable.java
--
diff --git a/src/java/org/apache/cassandra/db/Memtable.java
b/src/java/org/apache/cassandra/db/Memtable.java
index 19f38be..897d94e 100644
--- a/src/java/org/apache/cassandra/db/Memtable.java
+++ b/src/java/org/apache/cassandra/db/Memtable.java
@@ -206,17 +206,21 @@ public class Memtable
{
AtomicSortedColumns previous = rows.get(key);
+long initialSize = 0;
if (previous == null)
{
AtomicSortedColumns empty =
cf.cloneMeShallow(AtomicSortedColumns.factory, false);
// We'll add the columns later. This avoids wasting works if we
get beaten in the putIfAbsent
previous = rows.putIfAbsent(new DecoratedKey(key.token,
allocator.clone(key.key)), empty);
if (previous == null)
+{
previous = empty;
+initialSize += 1;
+}
}
final PairLong, Long pair = previous.addAllWithSizeDelta(cf,
allocator, localCopyFunction, indexer);
-currentSize.addAndGet(pair.left);
+currentSize.addAndGet(initialSize + pair.left);
currentOperations.addAndGet(cf.getColumnCount() +
(cf.isMarkedForDelete() ? 1 : 0) + cf.deletionInfo().rangeCount());
return pair.right;
}
[3/6] cassandra git commit: Fix partition-level-delete-only workload accounting
Fix partition-level-delete-only workload accounting
patch by benedict; reviewed by jbellis for CASSANDRA-9194
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f0c7a6f0
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f0c7a6f0
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f0c7a6f0
Branch: refs/heads/trunk
Commit: f0c7a6f03cf55ddcfe11a58c80087ed8b0806cf4
Parents: 593a725
Author: Benedict Elliott Smith bened...@apache.org
Authored: Fri May 1 11:32:31 2015 +0100
Committer: Benedict Elliott Smith bened...@apache.org
Committed: Fri May 1 11:32:31 2015 +0100
--
CHANGES.txt| 1 +
src/java/org/apache/cassandra/db/Memtable.java | 6 +-
2 files changed, 6 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/f0c7a6f0/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 4e7a5d0..dfdad51 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.0.15:
+ * Fix partition-level-delete-only workload accounting (CASSANDRA-9194)
* Allow scrub to handle corrupted compressed chunks (CASSANDRA-9140)
* Fix assertion error when resetlocalschema is run during repair
(CASSANDRA-9249)
* Disable single sstable tombstone compactions for DTCS by default
(CASSANDRA-9234)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/f0c7a6f0/src/java/org/apache/cassandra/db/Memtable.java
--
diff --git a/src/java/org/apache/cassandra/db/Memtable.java
b/src/java/org/apache/cassandra/db/Memtable.java
index 19f38be..897d94e 100644
--- a/src/java/org/apache/cassandra/db/Memtable.java
+++ b/src/java/org/apache/cassandra/db/Memtable.java
@@ -206,17 +206,21 @@ public class Memtable
{
AtomicSortedColumns previous = rows.get(key);
+long initialSize = 0;
if (previous == null)
{
AtomicSortedColumns empty =
cf.cloneMeShallow(AtomicSortedColumns.factory, false);
// We'll add the columns later. This avoids wasting works if we
get beaten in the putIfAbsent
previous = rows.putIfAbsent(new DecoratedKey(key.token,
allocator.clone(key.key)), empty);
if (previous == null)
+{
previous = empty;
+initialSize += 1;
+}
}
final PairLong, Long pair = previous.addAllWithSizeDelta(cf,
allocator, localCopyFunction, indexer);
-currentSize.addAndGet(pair.left);
+currentSize.addAndGet(initialSize + pair.left);
currentOperations.addAndGet(cf.getColumnCount() +
(cf.isMarkedForDelete() ? 1 : 0) + cf.deletionInfo().rangeCount());
return pair.right;
}
[6/6] cassandra git commit: Merge branch 'cassandra-2.1' into trunk
Merge branch 'cassandra-2.1' into trunk
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/527c12a0
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/527c12a0
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/527c12a0
Branch: refs/heads/trunk
Commit: 527c12a0115629ad0fa7ec95a24c55fc115739a1
Parents: 49b0898 4f9c9ce
Author: Benedict Elliott Smith bened...@apache.org
Authored: Fri May 1 11:33:52 2015 +0100
Committer: Benedict Elliott Smith bened...@apache.org
Committed: Fri May 1 11:33:52 2015 +0100
--
CHANGES.txt| 1 +
src/java/org/apache/cassandra/db/Memtable.java | 4 +++-
2 files changed, 4 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/527c12a0/CHANGES.txt
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/527c12a0/src/java/org/apache/cassandra/db/Memtable.java
--
diff --cc src/java/org/apache/cassandra/db/Memtable.java
index aa5fb1b,9a8596a..ef47aba
--- a/src/java/org/apache/cassandra/db/Memtable.java
+++ b/src/java/org/apache/cassandra/db/Memtable.java
@@@ -196,8 -197,9 +197,9 @@@ public class Memtabl
previous = empty;
// allocate the row overhead after the fact; this saves over
allocating and having to free after, but
// means we can overshoot our declared limit.
-int overhead = (int)
(cfs.partitioner.getHeapSizeOf(key.getToken()) + ROW_OVERHEAD_HEAP_SIZE);
+int overhead = (int) (key.getToken().getHeapSize() +
ROW_OVERHEAD_HEAP_SIZE);
allocator.onHeap().allocate(overhead, opGroup);
+ initialSize = 8;
}
else
{
