[jira] [Created] (CASSANDRA-6952) Cannot bind variables to USE statements
Matt Stump created CASSANDRA-6952:
-
Summary: Cannot bind variables to USE statements
Key: CASSANDRA-6952
URL: https://issues.apache.org/jira/browse/CASSANDRA-6952
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: Matt Stump
Priority: Minor
Attempting to bind a variable for a USE query results in a syntax error.
Example Invocation:
{code}
ResultSet result = session.execute("USE ?", "system");
{code}
Error:
{code}
ERROR SYNTAX_ERROR: line 1:4 no viable alternative at input '?', v=2
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-6553) Benchmark counter improvements (counters++)
[ https://issues.apache.org/jira/browse/CASSANDRA-6553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951736#comment-13951736 ] Russ Hatch commented on CASSANDRA-6553: --- I think some of the write scenarios were running into CASSANDRA-6943. > Benchmark counter improvements (counters++) > --- > > Key: CASSANDRA-6553 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6553 > Project: Cassandra > Issue Type: Test >Reporter: Ryan McGuire >Assignee: Russ Hatch > Fix For: 2.1 beta2 > > Attachments: 6553.txt, 6553.uber.quorum.bdplab.read.png, > 6553.uber.quorum.bdplab.write.png, high_cl_one.png, high_cl_quorum.png, > low_cl_one.png, low_cl_quorum.png, tracing.txt, uber_cl_one.png, > uber_cl_quorum.png > > > Benchmark the difference in performance between CASSANDRA-6504 and trunk. > * Updating totally unrelated counters (different partitions) > * Updating the same counters a lot (same cells in the same partition) > * Different cells in the same few partitions (hot counter partition) > benchmark: > https://github.com/apache/cassandra/tree/1218bcacba7edefaf56cf8440d0aea5794c89a1e > (old counters) > compared to: > https://github.com/apache/cassandra/tree/714c423360c36da2a2b365efaf9c5c4f623ed133 > (new counters) > So far, the above changes should only affect the write path. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6553) Benchmark counter improvements (counters++)
[ https://issues.apache.org/jira/browse/CASSANDRA-6553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951733#comment-13951733 ] Russ Hatch commented on CASSANDRA-6553: --- No counter cache: http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.low_contention_no_cache_CL_one_3_28.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.high_contention_no_cache_CL_one_3_28.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.uber_contention_no_cache_CL_one_3_28.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.low_contention_no_cache_CL_quorum_3_28.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.high_contention_no_cache_CL_quorum_3_28.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.user_contention_no_cache_CL_quorum_3_28.json Counter cache enabled: http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.high_contention_CL_one_3_28.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.low_contention_CL_one_3_28.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.uber_contention_CL_one_3_28.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.low_contention_CL_quorum_3_28.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.high_contention_CL_quorum_3_28.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.user_contention_CL_quorum_3_28.json > Benchmark counter improvements (counters++) > --- > > Key: CASSANDRA-6553 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6553 > Project: Cassandra > Issue Type: Test >Reporter: Ryan McGuire >Assignee: Russ Hatch > Fix For: 2.1 beta2 > > Attachments: 6553.txt, 6553.uber.quorum.bdplab.read.png, > 6553.uber.quorum.bdplab.write.png, high_cl_one.png, high_cl_quorum.png, > low_cl_one.png, low_cl_quorum.png, tracing.txt, uber_cl_one.png, > uber_cl_quorum.png > > > Benchmark the difference in performance between CASSANDRA-6504 and trunk. > * Updating totally unrelated counters (different partitions) > * Updating the same counters a lot (same cells in the same partition) > * Different cells in the same few partitions (hot counter partition) > benchmark: > https://github.com/apache/cassandra/tree/1218bcacba7edefaf56cf8440d0aea5794c89a1e > (old counters) > compared to: > https://github.com/apache/cassandra/tree/714c423360c36da2a2b365efaf9c5c4f623ed133 > (new counters) > So far, the above changes should only affect the write path. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6826) Query returns different number of results depending on fetchsize
[ https://issues.apache.org/jira/browse/CASSANDRA-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951706#comment-13951706 ] Bill Mitchell commented on CASSANDRA-6826: -- I started working on a smaller testcase, but competing time pressures at work put that effort on hold. In the meantime, I was able to work around this problem by using LIMIT instead of fetch, iterating over the partitions, and using a compound comparison in the WHERE clause to establish position for the next query. This prompted me to open JAVA-295, as I had to abandon the QueryBuilder in order to construct this WHERE clause. When Cassandra 2.0.7 comes out, I will check if the fix to CASSANDRA-6825 also fixes all the issue I found with the SELECT. > Query returns different number of results depending on fetchsize > > > Key: CASSANDRA-6826 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6826 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: quad-core Windows 7 x64, single node cluster > Cassandra 2.0.5 >Reporter: Bill Mitchell >Assignee: Sylvain Lebresne > > I issue a query across the set of partitioned wide rows for one logical row, > where s, l, and partition specify the composite primary key for the row: > SELECT ec, ea, rd FROM sr WHERE s = ? and partition IN ? and l = ? ALLOW > FILTERING; > If I set fetchSize to only 1000 when the Cluster is configured, the query > sometimes does not return all the results. In the particular case I am > chasing, it returns a total of 98586 rows. If I increase the fetchsize to > 10, all the 9 actual rows are returned. This suggests there is some > problem with fetchsize re-establishing the position on the next segment of > the result set, at least when multiple partitions are being accessed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[1/2] git commit: snaptree is no longer a dependency in 2.1 and forwards
Repository: cassandra Updated Branches: refs/heads/trunk 724951b56 -> eeef4061b snaptree is no longer a dependency in 2.1 and forwards Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/68444628 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/68444628 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/68444628 Branch: refs/heads/trunk Commit: 68444628982147790276a476d66f9087b9256bc6 Parents: a5a7261 Author: Dave Brosius Authored: Fri Mar 28 21:13:21 2014 -0400 Committer: Dave Brosius Committed: Fri Mar 28 21:13:21 2014 -0400 -- build.xml | 1 - 1 file changed, 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/68444628/build.xml -- diff --git a/build.xml b/build.xml index 6656015..fb1e456 100644 --- a/build.xml +++ b/build.xml @@ -468,7 +468,6 @@ -
[2/2] git commit: Merge branch 'cassandra-2.1' into trunk
Merge branch 'cassandra-2.1' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/eeef4061 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/eeef4061 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/eeef4061 Branch: refs/heads/trunk Commit: eeef4061b7beb181ccad0b81ad7807bef3ef6367 Parents: 724951b 6844462 Author: Dave Brosius Authored: Fri Mar 28 21:14:04 2014 -0400 Committer: Dave Brosius Committed: Fri Mar 28 21:14:04 2014 -0400 -- build.xml | 1 - 1 file changed, 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/eeef4061/build.xml --
git commit: snaptree is no longer a dependency in 2.1 and forwards
Repository: cassandra Updated Branches: refs/heads/cassandra-2.1 a5a726174 -> 684446289 snaptree is no longer a dependency in 2.1 and forwards Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/68444628 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/68444628 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/68444628 Branch: refs/heads/cassandra-2.1 Commit: 68444628982147790276a476d66f9087b9256bc6 Parents: a5a7261 Author: Dave Brosius Authored: Fri Mar 28 21:13:21 2014 -0400 Committer: Dave Brosius Committed: Fri Mar 28 21:13:21 2014 -0400 -- build.xml | 1 - 1 file changed, 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/68444628/build.xml -- diff --git a/build.xml b/build.xml index 6656015..fb1e456 100644 --- a/build.xml +++ b/build.xml @@ -468,7 +468,6 @@ -
[jira] [Resolved] (CASSANDRA-6951) update query deletes instead of update, upgraded to 2.0.6
[ https://issues.apache.org/jira/browse/CASSANDRA-6951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Yeschenko resolved CASSANDRA-6951. -- Resolution: Not A Problem > update query deletes instead of update, upgraded to 2.0.6 > - > > Key: CASSANDRA-6951 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6951 > Project: Cassandra > Issue Type: Bug > Environment: ubuntu 12.04 on ec2 >Reporter: Pardeep Singh > > I'm showing two examples below to show what happens. I'm using update, but > technically it's an insert. This worked in the previous version, which I > don't remember but it was 1.2+ > cqlsh:tagloom> update user_timeline set d=null WHERE uid='aa' and pid='aa'; > cqlsh:tagloom> select * from user_timeline where uid='aa' limit 4; > (0 rows) > cqlsh:tagloom> update user_timeline set d='a' WHERE uid='aa' and pid='aa'; > cqlsh:tagloom> select * from user_timeline where uid='aa' limit 4; > uid | pid | d > -+-+--- > aa | aa | a > (1 rows) > I run it again: > cqlsh:tagloom> update user_timeline set d=null WHERE uid='aa' and pid='aa'; > cqlsh:tagloom> select * from user_timeline where uid='aa' limit 4; > (0 rows) > Seems like it's deleting the row. Was there a change that made the above > results possible or is it a bug? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6951) update query deletes instead of update, upgraded to 2.0.6
[ https://issues.apache.org/jira/browse/CASSANDRA-6951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951597#comment-13951597 ] Aleksey Yeschenko commented on CASSANDRA-6951: -- CASSANDRA-6782 is the change (this is intentional, not a bug). > update query deletes instead of update, upgraded to 2.0.6 > - > > Key: CASSANDRA-6951 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6951 > Project: Cassandra > Issue Type: Bug > Environment: ubuntu 12.04 on ec2 >Reporter: Pardeep Singh > > I'm showing two examples below to show what happens. I'm using update, but > technically it's an insert. This worked in the previous version, which I > don't remember but it was 1.2+ > cqlsh:tagloom> update user_timeline set d=null WHERE uid='aa' and pid='aa'; > cqlsh:tagloom> select * from user_timeline where uid='aa' limit 4; > (0 rows) > cqlsh:tagloom> update user_timeline set d='a' WHERE uid='aa' and pid='aa'; > cqlsh:tagloom> select * from user_timeline where uid='aa' limit 4; > uid | pid | d > -+-+--- > aa | aa | a > (1 rows) > I run it again: > cqlsh:tagloom> update user_timeline set d=null WHERE uid='aa' and pid='aa'; > cqlsh:tagloom> select * from user_timeline where uid='aa' limit 4; > (0 rows) > Seems like it's deleting the row. Was there a change that made the above > results possible or is it a bug? -- This message was sent by Atlassian JIRA (v6.2#6252)
[3/3] git commit: Merge branch 'cassandra-2.1' into trunk
Merge branch 'cassandra-2.1' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/724951b5 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/724951b5 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/724951b5 Branch: refs/heads/trunk Commit: 724951b56b6ae17e0d5bfe61a43a3b1ab8434876 Parents: 7ef6b34 a5a7261 Author: Aleksey Yeschenko Authored: Sat Mar 29 02:18:14 2014 +0300 Committer: Aleksey Yeschenko Committed: Sat Mar 29 02:18:14 2014 +0300 -- --
[1/3] git commit: Post-CASSANDRA-6945 tweak
Repository: cassandra
Updated Branches:
refs/heads/trunk 7ef6b3477 -> 724951b56
Post-CASSANDRA-6945 tweak
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/6874aaa0
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/6874aaa0
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/6874aaa0
Branch: refs/heads/trunk
Commit: 6874aaa0db16513299d18b381166044f9735c31f
Parents: 47bbfe3
Author: Aleksey Yeschenko
Authored: Sat Mar 29 02:16:06 2014 +0300
Committer: Aleksey Yeschenko
Committed: Sat Mar 29 02:16:06 2014 +0300
--
src/java/org/apache/cassandra/db/Memtable.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/6874aaa0/src/java/org/apache/cassandra/db/Memtable.java
--
diff --git a/src/java/org/apache/cassandra/db/Memtable.java
b/src/java/org/apache/cassandra/db/Memtable.java
index b26b7cd..ea79e9c 100644
--- a/src/java/org/apache/cassandra/db/Memtable.java
+++ b/src/java/org/apache/cassandra/db/Memtable.java
@@ -131,7 +131,7 @@ public class Memtable
if (previous != null)
{
liveRatio = previous.liveRatio;
-liveRatioComputedAt.set(previous.liveRatioComputedAt.get() / 2);
+liveRatioComputedAt.set(previous.liveRatioComputedAt.get() / 4);
}
Callable> provider = new Callable>()
[2/3] git commit: Merge branch 'cassandra-2.0' into cassandra-2.1
Merge branch 'cassandra-2.0' into cassandra-2.1 Conflicts: src/java/org/apache/cassandra/db/Memtable.java Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a5a72617 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a5a72617 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a5a72617 Branch: refs/heads/trunk Commit: a5a726174ab3b574544950ba3cd75186461b797f Parents: bef9526 6874aaa Author: Aleksey Yeschenko Authored: Sat Mar 29 02:17:56 2014 +0300 Committer: Aleksey Yeschenko Committed: Sat Mar 29 02:17:56 2014 +0300 -- --
[2/2] git commit: Merge branch 'cassandra-2.0' into cassandra-2.1
Merge branch 'cassandra-2.0' into cassandra-2.1 Conflicts: src/java/org/apache/cassandra/db/Memtable.java Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a5a72617 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a5a72617 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a5a72617 Branch: refs/heads/cassandra-2.1 Commit: a5a726174ab3b574544950ba3cd75186461b797f Parents: bef9526 6874aaa Author: Aleksey Yeschenko Authored: Sat Mar 29 02:17:56 2014 +0300 Committer: Aleksey Yeschenko Committed: Sat Mar 29 02:17:56 2014 +0300 -- --
[1/2] git commit: Post-CASSANDRA-6945 tweak
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.1 bef9526d1 -> a5a726174
Post-CASSANDRA-6945 tweak
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/6874aaa0
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/6874aaa0
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/6874aaa0
Branch: refs/heads/cassandra-2.1
Commit: 6874aaa0db16513299d18b381166044f9735c31f
Parents: 47bbfe3
Author: Aleksey Yeschenko
Authored: Sat Mar 29 02:16:06 2014 +0300
Committer: Aleksey Yeschenko
Committed: Sat Mar 29 02:16:06 2014 +0300
--
src/java/org/apache/cassandra/db/Memtable.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/6874aaa0/src/java/org/apache/cassandra/db/Memtable.java
--
diff --git a/src/java/org/apache/cassandra/db/Memtable.java
b/src/java/org/apache/cassandra/db/Memtable.java
index b26b7cd..ea79e9c 100644
--- a/src/java/org/apache/cassandra/db/Memtable.java
+++ b/src/java/org/apache/cassandra/db/Memtable.java
@@ -131,7 +131,7 @@ public class Memtable
if (previous != null)
{
liveRatio = previous.liveRatio;
-liveRatioComputedAt.set(previous.liveRatioComputedAt.get() / 2);
+liveRatioComputedAt.set(previous.liveRatioComputedAt.get() / 4);
}
Callable> provider = new Callable>()
git commit: Post-CASSANDRA-6945 tweak
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.0 47bbfe35a -> 6874aaa0d
Post-CASSANDRA-6945 tweak
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/6874aaa0
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/6874aaa0
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/6874aaa0
Branch: refs/heads/cassandra-2.0
Commit: 6874aaa0db16513299d18b381166044f9735c31f
Parents: 47bbfe3
Author: Aleksey Yeschenko
Authored: Sat Mar 29 02:16:06 2014 +0300
Committer: Aleksey Yeschenko
Committed: Sat Mar 29 02:16:06 2014 +0300
--
src/java/org/apache/cassandra/db/Memtable.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/6874aaa0/src/java/org/apache/cassandra/db/Memtable.java
--
diff --git a/src/java/org/apache/cassandra/db/Memtable.java
b/src/java/org/apache/cassandra/db/Memtable.java
index b26b7cd..ea79e9c 100644
--- a/src/java/org/apache/cassandra/db/Memtable.java
+++ b/src/java/org/apache/cassandra/db/Memtable.java
@@ -131,7 +131,7 @@ public class Memtable
if (previous != null)
{
liveRatio = previous.liveRatio;
-liveRatioComputedAt.set(previous.liveRatioComputedAt.get() / 2);
+liveRatioComputedAt.set(previous.liveRatioComputedAt.get() / 4);
}
Callable> provider = new Callable>()
[1/2] git commit: Ninja-in OperationType.COUNTER_CACHE_SAVE
Repository: cassandra
Updated Branches:
refs/heads/trunk e1de2a49e -> 7ef6b3477
Ninja-in OperationType.COUNTER_CACHE_SAVE
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/bef9526d
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/bef9526d
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/bef9526d
Branch: refs/heads/trunk
Commit: bef9526d1b89ccd8bf3f42664aebf05d587a0411
Parents: 1e57bd1
Author: Aleksey Yeschenko
Authored: Sat Mar 29 01:58:34 2014 +0300
Committer: Aleksey Yeschenko
Committed: Sat Mar 29 01:58:34 2014 +0300
--
src/java/org/apache/cassandra/cache/AutoSavingCache.java | 2 ++
src/java/org/apache/cassandra/db/compaction/OperationType.java | 1 +
2 files changed, 3 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/bef9526d/src/java/org/apache/cassandra/cache/AutoSavingCache.java
--
diff --git a/src/java/org/apache/cassandra/cache/AutoSavingCache.java
b/src/java/org/apache/cassandra/cache/AutoSavingCache.java
index c0718b6..7b9ae95 100644
--- a/src/java/org/apache/cassandra/cache/AutoSavingCache.java
+++ b/src/java/org/apache/cassandra/cache/AutoSavingCache.java
@@ -170,6 +170,8 @@ public class AutoSavingCache extends
InstrumentingCachehttp://git-wip-us.apache.org/repos/asf/cassandra/blob/bef9526d/src/java/org/apache/cassandra/db/compaction/OperationType.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/OperationType.java
b/src/java/org/apache/cassandra/db/compaction/OperationType.java
index df4fd96..15d18f6 100644
--- a/src/java/org/apache/cassandra/db/compaction/OperationType.java
+++ b/src/java/org/apache/cassandra/db/compaction/OperationType.java
@@ -23,6 +23,7 @@ public enum OperationType
VALIDATION("Validation"),
KEY_CACHE_SAVE("Key cache save"),
ROW_CACHE_SAVE("Row cache save"),
+COUNTER_CACHE_SAVE("Counter cache save"),
CLEANUP("Cleanup"),
SCRUB("Scrub"),
UPGRADE_SSTABLES("Upgrade sstables"),
[2/2] git commit: Merge branch 'cassandra-2.1' into trunk
Merge branch 'cassandra-2.1' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/7ef6b347 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/7ef6b347 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/7ef6b347 Branch: refs/heads/trunk Commit: 7ef6b34774fa0a1110f97fe35d86242b74e9 Parents: e1de2a4 bef9526 Author: Aleksey Yeschenko Authored: Sat Mar 29 01:59:47 2014 +0300 Committer: Aleksey Yeschenko Committed: Sat Mar 29 01:59:47 2014 +0300 -- src/java/org/apache/cassandra/cache/AutoSavingCache.java | 2 ++ src/java/org/apache/cassandra/db/compaction/OperationType.java | 1 + 2 files changed, 3 insertions(+) --
git commit: Ninja-in OperationType.COUNTER_CACHE_SAVE
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.1 1e57bd155 -> bef9526d1
Ninja-in OperationType.COUNTER_CACHE_SAVE
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/bef9526d
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/bef9526d
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/bef9526d
Branch: refs/heads/cassandra-2.1
Commit: bef9526d1b89ccd8bf3f42664aebf05d587a0411
Parents: 1e57bd1
Author: Aleksey Yeschenko
Authored: Sat Mar 29 01:58:34 2014 +0300
Committer: Aleksey Yeschenko
Committed: Sat Mar 29 01:58:34 2014 +0300
--
src/java/org/apache/cassandra/cache/AutoSavingCache.java | 2 ++
src/java/org/apache/cassandra/db/compaction/OperationType.java | 1 +
2 files changed, 3 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/bef9526d/src/java/org/apache/cassandra/cache/AutoSavingCache.java
--
diff --git a/src/java/org/apache/cassandra/cache/AutoSavingCache.java
b/src/java/org/apache/cassandra/cache/AutoSavingCache.java
index c0718b6..7b9ae95 100644
--- a/src/java/org/apache/cassandra/cache/AutoSavingCache.java
+++ b/src/java/org/apache/cassandra/cache/AutoSavingCache.java
@@ -170,6 +170,8 @@ public class AutoSavingCache extends
InstrumentingCachehttp://git-wip-us.apache.org/repos/asf/cassandra/blob/bef9526d/src/java/org/apache/cassandra/db/compaction/OperationType.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/OperationType.java
b/src/java/org/apache/cassandra/db/compaction/OperationType.java
index df4fd96..15d18f6 100644
--- a/src/java/org/apache/cassandra/db/compaction/OperationType.java
+++ b/src/java/org/apache/cassandra/db/compaction/OperationType.java
@@ -23,6 +23,7 @@ public enum OperationType
VALIDATION("Validation"),
KEY_CACHE_SAVE("Key cache save"),
ROW_CACHE_SAVE("Row cache save"),
+COUNTER_CACHE_SAVE("Counter cache save"),
CLEANUP("Cleanup"),
SCRUB("Scrub"),
UPGRADE_SSTABLES("Upgrade sstables"),
[jira] [Commented] (CASSANDRA-6764) Using Batch commitlog_sync is slow and doesn't actually batch writes
[ https://issues.apache.org/jira/browse/CASSANDRA-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951525#comment-13951525 ] Benedict commented on CASSANDRA-6764: - bq. That was definitely the intention, and that's what it did unless the math was off. Right, agreed after re-reading, but the name testExceedSegmentSize... understandably threw me :) bq. How does this differ batch vs periodic? I thought we had agreed to only change batch behaviour here? I must admit I didn't check the code, I was going by the conversation. > Using Batch commitlog_sync is slow and doesn't actually batch writes > > > Key: CASSANDRA-6764 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6764 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: John Carrino >Assignee: John Carrino > Fix For: 2.1 beta2 > > Attachments: 6764.fix.txt, cassandra_6764_v2.patch, > cassandra_6764_v3.patch > > > The assumption behind batch commit mode is that the client does it's own > batching and wants to wait until the write is durable before returning. The > problem is that the queue that cassandra uses under the covers only allows > for a single ROW (RowMutation) per thread (concurrent_writes). This means > that commitlog_sync_batch_window_in_ms should really be called sleep_between > each_concurrent_writes_rows_in_ms. > I assume the reason this slipped by for so long is that no one uses batch > mode, probably because people say "it's slow". We need durability so this > isn't an option. > However it doesn't need to be this slow. > Also, if you write a row that is larger than the commit log size it silently > (warn) fails to put it in the commit log. This is not ideal for batch mode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6764) Using Batch commitlog_sync is slow and doesn't actually batch writes
[
https://issues.apache.org/jira/browse/CASSANDRA-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951520#comment-13951520
]
Jonathan Ellis commented on CASSANDRA-6764:
---
That was definitely the intention, and that's what it did unless the math was
off.
How does this differ batch vs periodic?
{code}
long totalSize = size + ENTRY_OVERHEAD_SIZE;
if (totalSize > MAX_MUTATION_SIZE)
{
throw new IllegalArgumentException(String.format("Mutation of %s
bytes is too large for the maxiumum size of %s",
totalSize,
MAX_MUTATION_SIZE));
}
{code}
> Using Batch commitlog_sync is slow and doesn't actually batch writes
>
>
> Key: CASSANDRA-6764
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6764
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Reporter: John Carrino
>Assignee: John Carrino
> Fix For: 2.1 beta2
>
> Attachments: 6764.fix.txt, cassandra_6764_v2.patch,
> cassandra_6764_v3.patch
>
>
> The assumption behind batch commit mode is that the client does it's own
> batching and wants to wait until the write is durable before returning. The
> problem is that the queue that cassandra uses under the covers only allows
> for a single ROW (RowMutation) per thread (concurrent_writes). This means
> that commitlog_sync_batch_window_in_ms should really be called sleep_between
> each_concurrent_writes_rows_in_ms.
> I assume the reason this slipped by for so long is that no one uses batch
> mode, probably because people say "it's slow". We need durability so this
> isn't an option.
> However it doesn't need to be this slow.
> Also, if you write a row that is larger than the commit log size it silently
> (warn) fails to put it in the commit log. This is not ideal for batch mode.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (CASSANDRA-6951) update query deletes instead of update, upgraded to 2.0.6
Pardeep Singh created CASSANDRA-6951: Summary: update query deletes instead of update, upgraded to 2.0.6 Key: CASSANDRA-6951 URL: https://issues.apache.org/jira/browse/CASSANDRA-6951 Project: Cassandra Issue Type: Bug Environment: ubuntu 12.04 on ec2 Reporter: Pardeep Singh I'm showing two examples below to show what happens. I'm using update, but technically it's an insert. This worked in the previous version, which I don't remember but it was 1.2+ cqlsh:tagloom> update user_timeline set d=null WHERE uid='aa' and pid='aa'; cqlsh:tagloom> select * from user_timeline where uid='aa' limit 4; (0 rows) cqlsh:tagloom> update user_timeline set d='a' WHERE uid='aa' and pid='aa'; cqlsh:tagloom> select * from user_timeline where uid='aa' limit 4; uid | pid | d -+-+--- aa | aa | a (1 rows) I run it again: cqlsh:tagloom> update user_timeline set d=null WHERE uid='aa' and pid='aa'; cqlsh:tagloom> select * from user_timeline where uid='aa' limit 4; (0 rows) Seems like it's deleting the row. Was there a change that made the above results possible or is it a bug? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6764) Using Batch commitlog_sync is slow and doesn't actually batch writes
[ https://issues.apache.org/jira/browse/CASSANDRA-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951510#comment-13951510 ] Benedict commented on CASSANDRA-6764: - But reading it again it looks like you are right. I think both are sensible tests, however, so I'll introduce a second test that checks the maximum size also. > Using Batch commitlog_sync is slow and doesn't actually batch writes > > > Key: CASSANDRA-6764 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6764 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: John Carrino >Assignee: John Carrino > Fix For: 2.1 beta2 > > Attachments: 6764.fix.txt, cassandra_6764_v2.patch, > cassandra_6764_v3.patch > > > The assumption behind batch commit mode is that the client does it's own > batching and wants to wait until the write is durable before returning. The > problem is that the queue that cassandra uses under the covers only allows > for a single ROW (RowMutation) per thread (concurrent_writes). This means > that commitlog_sync_batch_window_in_ms should really be called sleep_between > each_concurrent_writes_rows_in_ms. > I assume the reason this slipped by for so long is that no one uses batch > mode, probably because people say "it's slow". We need durability so this > isn't an option. > However it doesn't need to be this slow. > Also, if you write a row that is larger than the commit log size it silently > (warn) fails to put it in the commit log. This is not ideal for batch mode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6764) Using Batch commitlog_sync is slow and doesn't actually batch writes
[ https://issues.apache.org/jira/browse/CASSANDRA-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951507#comment-13951507 ] Benedict commented on CASSANDRA-6764: - I assumed from the name of the test (and the ticket) it was ensuring it could write a message that was _too large_. And in this case the behaviour differs now between batch and periodic. > Using Batch commitlog_sync is slow and doesn't actually batch writes > > > Key: CASSANDRA-6764 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6764 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: John Carrino >Assignee: John Carrino > Fix For: 2.1 beta2 > > Attachments: 6764.fix.txt, cassandra_6764_v2.patch, > cassandra_6764_v3.patch > > > The assumption behind batch commit mode is that the client does it's own > batching and wants to wait until the write is durable before returning. The > problem is that the queue that cassandra uses under the covers only allows > for a single ROW (RowMutation) per thread (concurrent_writes). This means > that commitlog_sync_batch_window_in_ms should really be called sleep_between > each_concurrent_writes_rows_in_ms. > I assume the reason this slipped by for so long is that no one uses batch > mode, probably because people say "it's slow". We need durability so this > isn't an option. > However it doesn't need to be this slow. > Also, if you write a row that is larger than the commit log size it silently > (warn) fails to put it in the commit log. This is not ideal for batch mode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6764) Using Batch commitlog_sync is slow and doesn't actually batch writes
[ https://issues.apache.org/jira/browse/CASSANDRA-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951504#comment-13951504 ] Jonathan Ellis commented on CASSANDRA-6764: --- I think what this test is trying to do is, "make sure we can write the largest acceptable mutation." (CASSANDRA-3615) > Using Batch commitlog_sync is slow and doesn't actually batch writes > > > Key: CASSANDRA-6764 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6764 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: John Carrino >Assignee: John Carrino > Fix For: 2.1 beta2 > > Attachments: 6764.fix.txt, cassandra_6764_v2.patch, > cassandra_6764_v3.patch > > > The assumption behind batch commit mode is that the client does it's own > batching and wants to wait until the write is durable before returning. The > problem is that the queue that cassandra uses under the covers only allows > for a single ROW (RowMutation) per thread (concurrent_writes). This means > that commitlog_sync_batch_window_in_ms should really be called sleep_between > each_concurrent_writes_rows_in_ms. > I assume the reason this slipped by for so long is that no one uses batch > mode, probably because people say "it's slow". We need durability so this > isn't an option. > However it doesn't need to be this slow. > Also, if you write a row that is larger than the commit log size it silently > (warn) fails to put it in the commit log. This is not ideal for batch mode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6764) Using Batch commitlog_sync is slow and doesn't actually batch writes
[ https://issues.apache.org/jira/browse/CASSANDRA-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951500#comment-13951500 ] Jonathan Ellis commented on CASSANDRA-6764: --- Is this right, that periodic and batch have different max mutation sizes? That's not how CL.add looks to me. > Using Batch commitlog_sync is slow and doesn't actually batch writes > > > Key: CASSANDRA-6764 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6764 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: John Carrino >Assignee: John Carrino > Fix For: 2.1 beta2 > > Attachments: 6764.fix.txt, cassandra_6764_v2.patch, > cassandra_6764_v3.patch > > > The assumption behind batch commit mode is that the client does it's own > batching and wants to wait until the write is durable before returning. The > problem is that the queue that cassandra uses under the covers only allows > for a single ROW (RowMutation) per thread (concurrent_writes). This means > that commitlog_sync_batch_window_in_ms should really be called sleep_between > each_concurrent_writes_rows_in_ms. > I assume the reason this slipped by for so long is that no one uses batch > mode, probably because people say "it's slow". We need durability so this > isn't an option. > However it doesn't need to be this slow. > Also, if you write a row that is larger than the commit log size it silently > (warn) fails to put it in the commit log. This is not ideal for batch mode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (CASSANDRA-6948) After Bootstrap or Replace node startup, EXPIRING_MAP_REAPER is shutdown and cannot be restarted, causing callbacks to collect indefinitely
[ https://issues.apache.org/jira/browse/CASSANDRA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951487#comment-13951487 ] Benedict edited comment on CASSANDRA-6948 at 3/28/14 10:22 PM: --- Good question. -I didn't try to figure out the bootstrap code, as it's quite unfamiliar, and isn't necessary for fixing this. It happens in checkForEndpointCollision and in prepareReplacementInfo. I can't see a good reason for it, but ruling out a good reason would require understanding that area of the codebase a lot better.- Was answering the wrong question (why it shutsdown MS). ExpiringMap executor makes no sense to ever shutdown, whatever the bootstrap code does, but happened as a matter of course when MS.shutdown() is called. was (Author: benedict): Good question. -I didn't try to figure out the bootstrap code, as it's quite unfamiliar, and isn't necessary for fixing this. It happens in checkForEndpointCollision and in prepareReplacementInfo. I can't see a good reason for it, but ruling out a good reason would require understanding that area of the codebase a lot better.- Was answering the wrong question (why it shutsdown MS). ExpiringMap executor makes no sense to ever shutdown, whatever the bootstrap code does. > After Bootstrap or Replace node startup, EXPIRING_MAP_REAPER is shutdown and > cannot be restarted, causing callbacks to collect indefinitely > --- > > Key: CASSANDRA-6948 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6948 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Keith Wright >Assignee: Benedict > Attachments: 6948.debug.txt, 6948.txt, Screen Shot 2014-03-28 at > 11.27.56 AM.png, Screen Shot 2014-03-28 at 11.29.24 AM.png, > cassandra.log.min, cassandra.yaml, logs.old.tar.gz, logs.tar.gz, > system.log.1.gz, system.log.gz > > > Since ExpiringMap.shutdown() shuts down the static executor service, it > cannot be restarted (and in fact reset() makes no attempt to do so). As such > callbacks that receive no response are never removed from the map, and > eventually either than server will run out of memory or will loop around the > integer space and start reusing messageids that have not been expired, > causing assertions to be thrown and messages to fail to be sent. It appears > that this situation only arises on bootstrap or node replacement, as > MessagingService is shutdown before being attached to the listen address. > This can cause the following errors to begin occurring in the log: > ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638 > ErrorMessage.java (line 222) Unexpected exception during request > java.lang.AssertionError: Callback already exists for id -1665979622! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984) > at > org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449) > at > org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524) > at > org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521) > at > org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505) > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358) > at > org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131) > at > org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) > at > org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > at > org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [ReplicateOnWriteStage:102766] 2014-03-28 13:32:10,638 > CassandraDaemon.java (line 196) Exception in thread > Thread[ReplicateOnWriteStage:102766,5,main] > java.lang.AssertionError: Cal
[jira] [Comment Edited] (CASSANDRA-6948) After Bootstrap or Replace node startup, EXPIRING_MAP_REAPER is shutdown and cannot be restarted, causing callbacks to collect indefinitely
[ https://issues.apache.org/jira/browse/CASSANDRA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951487#comment-13951487 ] Benedict edited comment on CASSANDRA-6948 at 3/28/14 10:21 PM: --- Good question. -I didn't try to figure out the bootstrap code, as it's quite unfamiliar, and isn't necessary for fixing this. It happens in checkForEndpointCollision and in prepareReplacementInfo. I can't see a good reason for it, but ruling out a good reason would require understanding that area of the codebase a lot better.- Was answering the wrong question (why it shutsdown MS). ExpiringMap executor makes no sense to ever shutdown, whatever the bootstrap code does. was (Author: benedict): Good question. I didn't try to figure out the bootstrap code, as it's quite unfamiliar, and isn't necessary for fixing this. It happens in checkForEndpointCollision and in prepareReplacementInfo. I can't see a good reason for it, but ruling out a good reason would require understanding that area of the codebase a lot better. > After Bootstrap or Replace node startup, EXPIRING_MAP_REAPER is shutdown and > cannot be restarted, causing callbacks to collect indefinitely > --- > > Key: CASSANDRA-6948 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6948 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Keith Wright >Assignee: Benedict > Attachments: 6948.debug.txt, 6948.txt, Screen Shot 2014-03-28 at > 11.27.56 AM.png, Screen Shot 2014-03-28 at 11.29.24 AM.png, > cassandra.log.min, cassandra.yaml, logs.old.tar.gz, logs.tar.gz, > system.log.1.gz, system.log.gz > > > Since ExpiringMap.shutdown() shuts down the static executor service, it > cannot be restarted (and in fact reset() makes no attempt to do so). As such > callbacks that receive no response are never removed from the map, and > eventually either than server will run out of memory or will loop around the > integer space and start reusing messageids that have not been expired, > causing assertions to be thrown and messages to fail to be sent. It appears > that this situation only arises on bootstrap or node replacement, as > MessagingService is shutdown before being attached to the listen address. > This can cause the following errors to begin occurring in the log: > ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638 > ErrorMessage.java (line 222) Unexpected exception during request > java.lang.AssertionError: Callback already exists for id -1665979622! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984) > at > org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449) > at > org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524) > at > org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521) > at > org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505) > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358) > at > org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131) > at > org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) > at > org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > at > org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [ReplicateOnWriteStage:102766] 2014-03-28 13:32:10,638 > CassandraDaemon.java (line 196) Exception in thread > Thread[ReplicateOnWriteStage:102766,5,main] > java.lang.AssertionError: Callback already exists for id -1665979620! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@3bdb1a75, > serializer=org.apache.cassandra.db.WriteResponse$WriteResp
[jira] [Commented] (CASSANDRA-6948) After Bootstrap or Replace node startup, EXPIRING_MAP_REAPER is shutdown and cannot be restarted, causing callbacks to collect indefinitely
[ https://issues.apache.org/jira/browse/CASSANDRA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951487#comment-13951487 ] Benedict commented on CASSANDRA-6948: - Good question. I didn't try to figure out the bootstrap code, as it's quite unfamiliar, and isn't necessary for fixing this. It happens in checkForEndpointCollision and in prepareReplacementInfo. I can't see a good reason for it, but ruling out a good reason would require understanding that area of the codebase a lot better. > After Bootstrap or Replace node startup, EXPIRING_MAP_REAPER is shutdown and > cannot be restarted, causing callbacks to collect indefinitely > --- > > Key: CASSANDRA-6948 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6948 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Keith Wright >Assignee: Benedict > Attachments: 6948.debug.txt, 6948.txt, Screen Shot 2014-03-28 at > 11.27.56 AM.png, Screen Shot 2014-03-28 at 11.29.24 AM.png, > cassandra.log.min, cassandra.yaml, logs.old.tar.gz, logs.tar.gz, > system.log.1.gz, system.log.gz > > > Since ExpiringMap.shutdown() shuts down the static executor service, it > cannot be restarted (and in fact reset() makes no attempt to do so). As such > callbacks that receive no response are never removed from the map, and > eventually either than server will run out of memory or will loop around the > integer space and start reusing messageids that have not been expired, > causing assertions to be thrown and messages to fail to be sent. It appears > that this situation only arises on bootstrap or node replacement, as > MessagingService is shutdown before being attached to the listen address. > This can cause the following errors to begin occurring in the log: > ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638 > ErrorMessage.java (line 222) Unexpected exception during request > java.lang.AssertionError: Callback already exists for id -1665979622! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984) > at > org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449) > at > org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524) > at > org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521) > at > org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505) > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358) > at > org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131) > at > org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) > at > org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > at > org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [ReplicateOnWriteStage:102766] 2014-03-28 13:32:10,638 > CassandraDaemon.java (line 196) Exception in thread > Thread[ReplicateOnWriteStage:102766,5,main] > java.lang.AssertionError: Callback already exists for id -1665979620! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@3bdb1a75, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:806) > at > org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1074) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896) > at > java.util.concurrent.ThreadPoolEx
[jira] [Commented] (CASSANDRA-6948) After Bootstrap or Replace node startup, EXPIRING_MAP_REAPER is shutdown and cannot be restarted, causing callbacks to collect indefinitely
[ https://issues.apache.org/jira/browse/CASSANDRA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951482#comment-13951482 ] Jonathan Ellis commented on CASSANDRA-6948: --- Why is bootstrap shutting down the MS ExpiringMap executor? > After Bootstrap or Replace node startup, EXPIRING_MAP_REAPER is shutdown and > cannot be restarted, causing callbacks to collect indefinitely > --- > > Key: CASSANDRA-6948 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6948 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Keith Wright >Assignee: Benedict > Attachments: 6948.debug.txt, 6948.txt, Screen Shot 2014-03-28 at > 11.27.56 AM.png, Screen Shot 2014-03-28 at 11.29.24 AM.png, > cassandra.log.min, cassandra.yaml, logs.old.tar.gz, logs.tar.gz, > system.log.1.gz, system.log.gz > > > Since ExpiringMap.shutdown() shuts down the static executor service, it > cannot be restarted (and in fact reset() makes no attempt to do so). As such > callbacks that receive no response are never removed from the map, and > eventually either than server will run out of memory or will loop around the > integer space and start reusing messageids that have not been expired, > causing assertions to be thrown and messages to fail to be sent. It appears > that this situation only arises on bootstrap or node replacement, as > MessagingService is shutdown before being attached to the listen address. > This can cause the following errors to begin occurring in the log: > ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638 > ErrorMessage.java (line 222) Unexpected exception during request > java.lang.AssertionError: Callback already exists for id -1665979622! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984) > at > org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449) > at > org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524) > at > org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521) > at > org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505) > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358) > at > org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131) > at > org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) > at > org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > at > org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [ReplicateOnWriteStage:102766] 2014-03-28 13:32:10,638 > CassandraDaemon.java (line 196) Exception in thread > Thread[ReplicateOnWriteStage:102766,5,main] > java.lang.AssertionError: Callback already exists for id -1665979620! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@3bdb1a75, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:806) > at > org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1074) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6890) Remove mmap'ed i/o from read path
[ https://issues.apache.org/jira/browse/CASSANDRA-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951479#comment-13951479 ] Jonathan Ellis commented on CASSANDRA-6890: --- bq. the option to keep compression off to improve performance seems pretty compelling You're right, more so than I thought. bq. Personally, I am -1 on removing mmapped IO as an option until we have an in-process uncompressed page cache that mitigates these problems. I think I'd be okay removing it if we can get to within 10%. > Remove mmap'ed i/o from read path > - > > Key: CASSANDRA-6890 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6890 > Project: Cassandra > Issue Type: Improvement >Reporter: Joshua McKenzie >Assignee: Joshua McKenzie >Priority: Minor > Fix For: 3.0 > > Attachments: mmap_gc.jpg, mmap_jstat.txt, mmap_perf.txt, > nommap_gc.jpg, nommap_jstat.txt > > > Since we actively unmap unreferenced SSTR's and also copy data out of those > readers on the read path, the current memory mapped i/o is a lot of > complexity for very little payoff. Clean out the mmapp'ed i/o on the read > path. -- This message was sent by Atlassian JIRA (v6.2#6252)
[3/3] git commit: Merge branch 'cassandra-2.1' into trunk
Merge branch 'cassandra-2.1' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e1de2a49 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e1de2a49 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e1de2a49 Branch: refs/heads/trunk Commit: e1de2a49e931ab8194593a76229ca71cd19cd785 Parents: 9b6aabe 1e57bd1 Author: Aleksey Yeschenko Authored: Sat Mar 29 00:44:20 2014 +0300 Committer: Aleksey Yeschenko Committed: Sat Mar 29 00:44:20 2014 +0300 -- --
[2/3] git commit: Merge branch 'cassandra-2.0' into cassandra-2.1
Merge branch 'cassandra-2.0' into cassandra-2.1 Conflicts: src/java/org/apache/cassandra/db/ColumnFamilyStore.java src/java/org/apache/cassandra/db/DataTracker.java src/java/org/apache/cassandra/db/Memtable.java Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1e57bd15 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1e57bd15 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1e57bd15 Branch: refs/heads/trunk Commit: 1e57bd1553e463c0440eab7bdedf860eecf04398 Parents: 6e6863d 47bbfe3 Author: Aleksey Yeschenko Authored: Sat Mar 29 00:43:37 2014 +0300 Committer: Aleksey Yeschenko Committed: Sat Mar 29 00:43:37 2014 +0300 -- --
[1/3] git commit: Track liveRatio per-memtable, not per-CF
Repository: cassandra
Updated Branches:
refs/heads/trunk 9b6aabef5 -> e1de2a49e
Track liveRatio per-memtable, not per-CF
patch by Aleksey Yeschenko; reviewed by Jonathan Ellis for
CASSANDRA-6945
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/47bbfe35
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/47bbfe35
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/47bbfe35
Branch: refs/heads/trunk
Commit: 47bbfe35afd750ef110311725f6b2824a91552fd
Parents: 180bb75
Author: Aleksey Yeschenko
Authored: Sat Mar 29 00:36:28 2014 +0300
Committer: Aleksey Yeschenko
Committed: Sat Mar 29 00:36:28 2014 +0300
--
CHANGES.txt | 1 +
.../apache/cassandra/db/ColumnFamilyStore.java | 21 +--
.../org/apache/cassandra/db/DataTracker.java| 7 +--
src/java/org/apache/cassandra/db/Memtable.java | 58 +++-
4 files changed, 50 insertions(+), 37 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/47bbfe35/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 399cb01..e9dca70 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -35,6 +35,7 @@
(CASSANDRA-6867)
* Add CqlRecordReader using native pagination (CASSANDRA-6311)
* Add QueryHandler interface (CASSANDRA-6659)
+ * Track liveRatio per-memtable, not per-CF (CASSANDRA-6945)
Merged from 1.2:
* Add UNLOGGED, COUNTER options to BATCH documentation (CASSANDRA-6816)
* add extra SSL cipher suites (CASSANDRA-6613)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/47bbfe35/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index b58329e..792c155 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -25,7 +25,6 @@ import java.nio.ByteBuffer;
import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicInteger;
-import java.util.concurrent.atomic.AtomicLong;
import java.util.regex.Pattern;
import javax.management.*;
@@ -105,11 +104,6 @@ public class ColumnFamilyStore implements
ColumnFamilyStoreMBean
public final Directories directories;
-/** ratio of in-memory memtable size, to serialized size */
-volatile double liveRatio = 10.0; // reasonable default until we compute
what it is based on actual data
-/** ops count last time we computed liveRatio */
-private final AtomicLong liveRatioComputedAt = new AtomicLong(32);
-
public final ColumnFamilyMetrics metric;
public volatile long sampleLatencyNanos;
@@ -899,20 +893,7 @@ public class ColumnFamilyStore implements
ColumnFamilyStoreMBean
mt.put(key, columnFamily, indexer);
maybeUpdateRowCache(key);
metric.writeLatency.addNano(System.nanoTime() - start);
-
-// recompute liveRatio, if we have doubled the number of ops since
last calculated
-while (true)
-{
-long last = liveRatioComputedAt.get();
-long operations = metric.writeLatency.latency.count();
-if (operations < 2 * last)
-break;
-if (liveRatioComputedAt.compareAndSet(last, operations))
-{
-logger.debug("computing liveRatio of {} at {} ops", this,
operations);
-mt.updateLiveRatio();
-}
-}
+mt.maybeUpdateLiveRatio();
}
/**
http://git-wip-us.apache.org/repos/asf/cassandra/blob/47bbfe35/src/java/org/apache/cassandra/db/DataTracker.java
--
diff --git a/src/java/org/apache/cassandra/db/DataTracker.java
b/src/java/org/apache/cassandra/db/DataTracker.java
index a1de8e5..a9eef98 100644
--- a/src/java/org/apache/cassandra/db/DataTracker.java
+++ b/src/java/org/apache/cassandra/db/DataTracker.java
@@ -30,7 +30,6 @@ import org.slf4j.LoggerFactory;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.db.compaction.OperationType;
import org.apache.cassandra.dht.AbstractBounds;
-import org.apache.cassandra.io.sstable.Descriptor;
import org.apache.cassandra.io.sstable.SSTableReader;
import org.apache.cassandra.io.util.FileUtils;
import org.apache.cassandra.metrics.StorageMetrics;
@@ -110,7 +109,7 @@ public class DataTracker
public Memtable switchMemtable()
{
// atomically change the current memtable
-Memtable newMemtable = new Memtable(cfstore);
+Memtable newMemtable = new Memtable(cfstore, view.get().memtable)
[2/2] git commit: Merge branch 'cassandra-2.0' into cassandra-2.1
Merge branch 'cassandra-2.0' into cassandra-2.1 Conflicts: src/java/org/apache/cassandra/db/ColumnFamilyStore.java src/java/org/apache/cassandra/db/DataTracker.java src/java/org/apache/cassandra/db/Memtable.java Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1e57bd15 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1e57bd15 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1e57bd15 Branch: refs/heads/cassandra-2.1 Commit: 1e57bd1553e463c0440eab7bdedf860eecf04398 Parents: 6e6863d 47bbfe3 Author: Aleksey Yeschenko Authored: Sat Mar 29 00:43:37 2014 +0300 Committer: Aleksey Yeschenko Committed: Sat Mar 29 00:43:37 2014 +0300 -- --
[1/2] git commit: Track liveRatio per-memtable, not per-CF
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.1 6e6863d4d -> 1e57bd155
Track liveRatio per-memtable, not per-CF
patch by Aleksey Yeschenko; reviewed by Jonathan Ellis for
CASSANDRA-6945
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/47bbfe35
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/47bbfe35
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/47bbfe35
Branch: refs/heads/cassandra-2.1
Commit: 47bbfe35afd750ef110311725f6b2824a91552fd
Parents: 180bb75
Author: Aleksey Yeschenko
Authored: Sat Mar 29 00:36:28 2014 +0300
Committer: Aleksey Yeschenko
Committed: Sat Mar 29 00:36:28 2014 +0300
--
CHANGES.txt | 1 +
.../apache/cassandra/db/ColumnFamilyStore.java | 21 +--
.../org/apache/cassandra/db/DataTracker.java| 7 +--
src/java/org/apache/cassandra/db/Memtable.java | 58 +++-
4 files changed, 50 insertions(+), 37 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/47bbfe35/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 399cb01..e9dca70 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -35,6 +35,7 @@
(CASSANDRA-6867)
* Add CqlRecordReader using native pagination (CASSANDRA-6311)
* Add QueryHandler interface (CASSANDRA-6659)
+ * Track liveRatio per-memtable, not per-CF (CASSANDRA-6945)
Merged from 1.2:
* Add UNLOGGED, COUNTER options to BATCH documentation (CASSANDRA-6816)
* add extra SSL cipher suites (CASSANDRA-6613)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/47bbfe35/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index b58329e..792c155 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -25,7 +25,6 @@ import java.nio.ByteBuffer;
import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicInteger;
-import java.util.concurrent.atomic.AtomicLong;
import java.util.regex.Pattern;
import javax.management.*;
@@ -105,11 +104,6 @@ public class ColumnFamilyStore implements
ColumnFamilyStoreMBean
public final Directories directories;
-/** ratio of in-memory memtable size, to serialized size */
-volatile double liveRatio = 10.0; // reasonable default until we compute
what it is based on actual data
-/** ops count last time we computed liveRatio */
-private final AtomicLong liveRatioComputedAt = new AtomicLong(32);
-
public final ColumnFamilyMetrics metric;
public volatile long sampleLatencyNanos;
@@ -899,20 +893,7 @@ public class ColumnFamilyStore implements
ColumnFamilyStoreMBean
mt.put(key, columnFamily, indexer);
maybeUpdateRowCache(key);
metric.writeLatency.addNano(System.nanoTime() - start);
-
-// recompute liveRatio, if we have doubled the number of ops since
last calculated
-while (true)
-{
-long last = liveRatioComputedAt.get();
-long operations = metric.writeLatency.latency.count();
-if (operations < 2 * last)
-break;
-if (liveRatioComputedAt.compareAndSet(last, operations))
-{
-logger.debug("computing liveRatio of {} at {} ops", this,
operations);
-mt.updateLiveRatio();
-}
-}
+mt.maybeUpdateLiveRatio();
}
/**
http://git-wip-us.apache.org/repos/asf/cassandra/blob/47bbfe35/src/java/org/apache/cassandra/db/DataTracker.java
--
diff --git a/src/java/org/apache/cassandra/db/DataTracker.java
b/src/java/org/apache/cassandra/db/DataTracker.java
index a1de8e5..a9eef98 100644
--- a/src/java/org/apache/cassandra/db/DataTracker.java
+++ b/src/java/org/apache/cassandra/db/DataTracker.java
@@ -30,7 +30,6 @@ import org.slf4j.LoggerFactory;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.db.compaction.OperationType;
import org.apache.cassandra.dht.AbstractBounds;
-import org.apache.cassandra.io.sstable.Descriptor;
import org.apache.cassandra.io.sstable.SSTableReader;
import org.apache.cassandra.io.util.FileUtils;
import org.apache.cassandra.metrics.StorageMetrics;
@@ -110,7 +109,7 @@ public class DataTracker
public Memtable switchMemtable()
{
// atomically change the current memtable
-Memtable newMemtable = new Memtable(cfstore);
+Memtable newMemtable = new Memtable(cfstore, view
git commit: Track liveRatio per-memtable, not per-CF
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.0 180bb7571 -> 47bbfe35a
Track liveRatio per-memtable, not per-CF
patch by Aleksey Yeschenko; reviewed by Jonathan Ellis for
CASSANDRA-6945
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/47bbfe35
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/47bbfe35
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/47bbfe35
Branch: refs/heads/cassandra-2.0
Commit: 47bbfe35afd750ef110311725f6b2824a91552fd
Parents: 180bb75
Author: Aleksey Yeschenko
Authored: Sat Mar 29 00:36:28 2014 +0300
Committer: Aleksey Yeschenko
Committed: Sat Mar 29 00:36:28 2014 +0300
--
CHANGES.txt | 1 +
.../apache/cassandra/db/ColumnFamilyStore.java | 21 +--
.../org/apache/cassandra/db/DataTracker.java| 7 +--
src/java/org/apache/cassandra/db/Memtable.java | 58 +++-
4 files changed, 50 insertions(+), 37 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/47bbfe35/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 399cb01..e9dca70 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -35,6 +35,7 @@
(CASSANDRA-6867)
* Add CqlRecordReader using native pagination (CASSANDRA-6311)
* Add QueryHandler interface (CASSANDRA-6659)
+ * Track liveRatio per-memtable, not per-CF (CASSANDRA-6945)
Merged from 1.2:
* Add UNLOGGED, COUNTER options to BATCH documentation (CASSANDRA-6816)
* add extra SSL cipher suites (CASSANDRA-6613)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/47bbfe35/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
--
diff --git a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
index b58329e..792c155 100644
--- a/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
+++ b/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
@@ -25,7 +25,6 @@ import java.nio.ByteBuffer;
import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicInteger;
-import java.util.concurrent.atomic.AtomicLong;
import java.util.regex.Pattern;
import javax.management.*;
@@ -105,11 +104,6 @@ public class ColumnFamilyStore implements
ColumnFamilyStoreMBean
public final Directories directories;
-/** ratio of in-memory memtable size, to serialized size */
-volatile double liveRatio = 10.0; // reasonable default until we compute
what it is based on actual data
-/** ops count last time we computed liveRatio */
-private final AtomicLong liveRatioComputedAt = new AtomicLong(32);
-
public final ColumnFamilyMetrics metric;
public volatile long sampleLatencyNanos;
@@ -899,20 +893,7 @@ public class ColumnFamilyStore implements
ColumnFamilyStoreMBean
mt.put(key, columnFamily, indexer);
maybeUpdateRowCache(key);
metric.writeLatency.addNano(System.nanoTime() - start);
-
-// recompute liveRatio, if we have doubled the number of ops since
last calculated
-while (true)
-{
-long last = liveRatioComputedAt.get();
-long operations = metric.writeLatency.latency.count();
-if (operations < 2 * last)
-break;
-if (liveRatioComputedAt.compareAndSet(last, operations))
-{
-logger.debug("computing liveRatio of {} at {} ops", this,
operations);
-mt.updateLiveRatio();
-}
-}
+mt.maybeUpdateLiveRatio();
}
/**
http://git-wip-us.apache.org/repos/asf/cassandra/blob/47bbfe35/src/java/org/apache/cassandra/db/DataTracker.java
--
diff --git a/src/java/org/apache/cassandra/db/DataTracker.java
b/src/java/org/apache/cassandra/db/DataTracker.java
index a1de8e5..a9eef98 100644
--- a/src/java/org/apache/cassandra/db/DataTracker.java
+++ b/src/java/org/apache/cassandra/db/DataTracker.java
@@ -30,7 +30,6 @@ import org.slf4j.LoggerFactory;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.db.compaction.OperationType;
import org.apache.cassandra.dht.AbstractBounds;
-import org.apache.cassandra.io.sstable.Descriptor;
import org.apache.cassandra.io.sstable.SSTableReader;
import org.apache.cassandra.io.util.FileUtils;
import org.apache.cassandra.metrics.StorageMetrics;
@@ -110,7 +109,7 @@ public class DataTracker
public Memtable switchMemtable()
{
// atomically change the current memtable
-Memtable newMemtable = new Memtable(cfstore);
+Memtable newMemtable = new Memtable(cfstore, view
[jira] [Updated] (CASSANDRA-6825) COUNT(*) with WHERE not finding all the matching rows
[ https://issues.apache.org/jira/browse/CASSANDRA-6825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tyler Hobbs updated CASSANDRA-6825: --- Since Version: 2.0.3 Fix Version/s: 2.1 beta2 2.0.7 > COUNT(*) with WHERE not finding all the matching rows > - > > Key: CASSANDRA-6825 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6825 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: quad core Windows7 x64, single node cluster > Cassandra 2.0.5 >Reporter: Bill Mitchell >Assignee: Tyler Hobbs > Fix For: 2.0.7, 2.1 beta2 > > Attachments: cassandra.log, selectpartitions.zip, > selectrowcounts.txt, testdb_1395372407904.zip, testdb_1395372407904.zip > > > Investigating another problem, I needed to do COUNT(*) on the several > partitions of a table immediately after a test case ran, and I discovered > that count(*) on the full table and on each of the partitions returned > different counts. > In particular case, SELECT COUNT(*) FROM sr LIMIT 100; returned the > expected count from the test 9 rows. The composite primary key splits > the logical row into six distinct partitions, and when I issue a query asking > for the total across all six partitions, the returned result is only 83999. > Drilling down, I find that SELECT * from sr WHERE s = 5 AND l = 11 AND > partition = 0; returns 30,000 rows, but a SELECT COUNT(*) with the identical > WHERE predicate reports only 14,000. > This is failing immediately after running a single small test, such that > there are only two SSTables, sr-jb-1 and sr-jb-2. Compaction never needed to > run. > In selectrowcounts.txt is a copy of the cqlsh output showing the incorrect > count(*) results. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6825) COUNT(*) with WHERE not finding all the matching rows
[
https://issues.apache.org/jira/browse/CASSANDRA-6825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951437#comment-13951437
]
Tyler Hobbs commented on CASSANDRA-6825:
It loos like CASSANDRA-6327 is the cause for this. The logic for testing
sstables for inclusion when there's a composite comparator and multiple
components in the slice filter is off. This showed up for {{count(\*)}}
because counting queries are always paged internally; the second page was
erroneously skipping an sstable. If the {{select *}} query has the same page
size (10k), it will also omit results.
> COUNT(*) with WHERE not finding all the matching rows
> -
>
> Key: CASSANDRA-6825
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6825
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: quad core Windows7 x64, single node cluster
> Cassandra 2.0.5
>Reporter: Bill Mitchell
>Assignee: Tyler Hobbs
> Attachments: cassandra.log, selectpartitions.zip,
> selectrowcounts.txt, testdb_1395372407904.zip, testdb_1395372407904.zip
>
>
> Investigating another problem, I needed to do COUNT(*) on the several
> partitions of a table immediately after a test case ran, and I discovered
> that count(*) on the full table and on each of the partitions returned
> different counts.
> In particular case, SELECT COUNT(*) FROM sr LIMIT 100; returned the
> expected count from the test 9 rows. The composite primary key splits
> the logical row into six distinct partitions, and when I issue a query asking
> for the total across all six partitions, the returned result is only 83999.
> Drilling down, I find that SELECT * from sr WHERE s = 5 AND l = 11 AND
> partition = 0; returns 30,000 rows, but a SELECT COUNT(*) with the identical
> WHERE predicate reports only 14,000.
> This is failing immediately after running a single small test, such that
> there are only two SSTables, sr-jb-1 and sr-jb-2. Compaction never needed to
> run.
> In selectrowcounts.txt is a copy of the cqlsh output showing the incorrect
> count(*) results.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-6880) counters++ lock on cells, not partitions
[ https://issues.apache.org/jira/browse/CASSANDRA-6880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951398#comment-13951398 ] Aleksey Yeschenko commented on CASSANDRA-6880: -- The old (pre-1.2) index locks were per-keyspace. I'm more inclined to go global this time, though. Whether to go 1024 * CW or higher is an open question though. FWIW I'm not committing this until the current batch of CASSANDRA-6553 and the next one (with changed params + some new tests) are run, so feel free to keep brainstorming here. > counters++ lock on cells, not partitions > > > Key: CASSANDRA-6880 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6880 > Project: Cassandra > Issue Type: Improvement >Reporter: Aleksey Yeschenko >Assignee: Aleksey Yeschenko > Fix For: 2.1 beta2 > > > I'm starting to think that we should switch to locking by cells, not by > partitions, when updating counters. > With the current 2.1 counters, if nothing changes, the new recommendation > would become "use smaller partitions, batch updates to the same partition", > and that goes against what we usually recommend: > 1. Prefer wide partitions to narrow partitions > 2. Don't batch counter updates (because you risk to exaggerate > undercounting/overcounting in case of a timeout) > Locking on cells would cause C* to have to grab more locks for batch counter > updates, but would give us generally more predictable performance > (independent of partition wideness), and won't force people to remodel their > data model if they have often concurrently-updated counters in the same few > wide partitions. > (It's a small change, code-wise) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6880) counters++ lock on cells, not partitions
[ https://issues.apache.org/jira/browse/CASSANDRA-6880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951342#comment-13951342 ] Benedict commented on CASSANDRA-6880: - If we're doing per-cell locking, the likelihood of collision across different ostensible uncontended updates increases. So, I wonder if it mightn't be sensible to have a global set of locks with a larger domain? e.g. 1024 * concurrent writers, but shared across all tables (or even all keyspaces). A shared but larger address space should reduce the likelihood of collision, whilst also bounding the amount of per-table memory we use (128 * concurrent writers could be very expensive in a batch CLE environment with a lot of tables). Probably worth capping the number of stripes as well for the same reason. > counters++ lock on cells, not partitions > > > Key: CASSANDRA-6880 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6880 > Project: Cassandra > Issue Type: Improvement >Reporter: Aleksey Yeschenko >Assignee: Aleksey Yeschenko > Fix For: 2.1 beta2 > > > I'm starting to think that we should switch to locking by cells, not by > partitions, when updating counters. > With the current 2.1 counters, if nothing changes, the new recommendation > would become "use smaller partitions, batch updates to the same partition", > and that goes against what we usually recommend: > 1. Prefer wide partitions to narrow partitions > 2. Don't batch counter updates (because you risk to exaggerate > undercounting/overcounting in case of a timeout) > Locking on cells would cause C* to have to grab more locks for batch counter > updates, but would give us generally more predictable performance > (independent of partition wideness), and won't force people to remodel their > data model if they have often concurrently-updated counters in the same few > wide partitions. > (It's a small change, code-wise) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6553) Benchmark counter improvements (counters++)
[ https://issues.apache.org/jira/browse/CASSANDRA-6553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951335#comment-13951335 ] Benedict commented on CASSANDRA-6553: - Would be good to throw in at least one new test that compares large number of cell updates across a wide range of partitions (so ostensibly uncontended). 6880 should make this slower, but would be good to quantify it > Benchmark counter improvements (counters++) > --- > > Key: CASSANDRA-6553 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6553 > Project: Cassandra > Issue Type: Test >Reporter: Ryan McGuire >Assignee: Russ Hatch > Fix For: 2.1 beta2 > > Attachments: 6553.txt, 6553.uber.quorum.bdplab.read.png, > 6553.uber.quorum.bdplab.write.png, high_cl_one.png, high_cl_quorum.png, > low_cl_one.png, low_cl_quorum.png, tracing.txt, uber_cl_one.png, > uber_cl_quorum.png > > > Benchmark the difference in performance between CASSANDRA-6504 and trunk. > * Updating totally unrelated counters (different partitions) > * Updating the same counters a lot (same cells in the same partition) > * Different cells in the same few partitions (hot counter partition) > benchmark: > https://github.com/apache/cassandra/tree/1218bcacba7edefaf56cf8440d0aea5794c89a1e > (old counters) > compared to: > https://github.com/apache/cassandra/tree/714c423360c36da2a2b365efaf9c5c4f623ed133 > (new counters) > So far, the above changes should only affect the write path. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6945) Calculate liveRatio on per-memtable basis, non per-CF
[ https://issues.apache.org/jira/browse/CASSANDRA-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951320#comment-13951320 ] Jonathan Ellis commented on CASSANDRA-6945: --- Ship it! > Calculate liveRatio on per-memtable basis, non per-CF > - > > Key: CASSANDRA-6945 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6945 > Project: Cassandra > Issue Type: Bug >Reporter: Aleksey Yeschenko >Assignee: Aleksey Yeschenko > Fix For: 2.0.7 > > > Currently we recalculate live ratio every doubling of write ops to the CF, > not to an individual memtable. The value itself is also CF-bound, not > memtable-bound. This is causing at least several issues: > 1. Depending on what stage the current memtable is, the live ratio calculated > can vary *a lot* > 2. That calculated live ratio will potentially stay that way for quite a > while - the longer C* process is on, the longer it would stay incorrect > 3. Incorrect live ratio means inefficient MeteredFlusher - flushing less or > more often than needed, picking bad candidates for flushing, etc. > 4. Incorrect live ratio means incorrect size returned to the metrics consumers > 5. Compaction strategies that rely on memtable size estimation are affected > 6. All of the above is slightly amplified by the fact that all the memtables > pending flush would also use that one incorrect value > Depending on the stage the current memtable at the moment of live ratio > recalculation is, the value calculated can be *extremely* wrong (say, a > recently created, fresh memtable - would have a much higher than average live > ratio). > The suggested fix is to bind live ratio to individual memtables, not column > families as a whole, with some optimizations to make recalculations run less > often by inheriting previous memtable's stats. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6880) counters++ lock on cells, not partitions
[ https://issues.apache.org/jira/browse/CASSANDRA-6880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951317#comment-13951317 ] Jonathan Ellis commented on CASSANDRA-6880: --- (Looks reasonable to me.) > counters++ lock on cells, not partitions > > > Key: CASSANDRA-6880 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6880 > Project: Cassandra > Issue Type: Improvement >Reporter: Aleksey Yeschenko >Assignee: Aleksey Yeschenko > Fix For: 2.1 beta2 > > > I'm starting to think that we should switch to locking by cells, not by > partitions, when updating counters. > With the current 2.1 counters, if nothing changes, the new recommendation > would become "use smaller partitions, batch updates to the same partition", > and that goes against what we usually recommend: > 1. Prefer wide partitions to narrow partitions > 2. Don't batch counter updates (because you risk to exaggerate > undercounting/overcounting in case of a timeout) > Locking on cells would cause C* to have to grab more locks for batch counter > updates, but would give us generally more predictable performance > (independent of partition wideness), and won't force people to remodel their > data model if they have often concurrently-updated counters in the same few > wide partitions. > (It's a small change, code-wise) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (CASSANDRA-6685) Nodes never bootstrap if schema is empty
[
https://issues.apache.org/jira/browse/CASSANDRA-6685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryan McGuire updated CASSANDRA-6685:
Reproduced In: 2.0.5, 1.2.15 (was: 1.2.15)
> Nodes never bootstrap if schema is empty
>
>
> Key: CASSANDRA-6685
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6685
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Richard Low
>Assignee: Brandon Williams
> Fix For: 1.2.16
>
>
> Since 1.2.15, bootstrap never completes if the schema is empty. The
> bootstrapping node endlessly prints:
> bq. {{INFO 12:37:44,863 JOINING: waiting for schema information to complete}}
> until you add something to the schema (i.e. create a keyspace).
> The problem looks to be caused by CASSANDRA-6648, where
> MigrationManager.isReadForBootstrap() was changed to:
> bq. {{return Schema.instance.getVersion() != null &&
> !Schema.emptyVersion.equals(Schema.instance.getVersion());}}
> This is wrong since
> {{Schema.emptyVersion.equals(Schema.instance.getVersion())}} is always true
> if there is no schema.
> We need some different logic for determining when the schema is propagated.
> I haven't tested, but I expect this issue appears in 2.0.5 too.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (CASSANDRA-6764) Using Batch commitlog_sync is slow and doesn't actually batch writes
[ https://issues.apache.org/jira/browse/CASSANDRA-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict updated CASSANDRA-6764: Attachment: 6764.fix.txt This patch broke our unit tests: simple fix (attached) is to simply assert that we have the correct type of exception thrown, rather than the exception being taken as test failure. > Using Batch commitlog_sync is slow and doesn't actually batch writes > > > Key: CASSANDRA-6764 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6764 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: John Carrino >Assignee: John Carrino > Fix For: 2.1 beta2 > > Attachments: 6764.fix.txt, cassandra_6764_v2.patch, > cassandra_6764_v3.patch > > > The assumption behind batch commit mode is that the client does it's own > batching and wants to wait until the write is durable before returning. The > problem is that the queue that cassandra uses under the covers only allows > for a single ROW (RowMutation) per thread (concurrent_writes). This means > that commitlog_sync_batch_window_in_ms should really be called sleep_between > each_concurrent_writes_rows_in_ms. > I assume the reason this slipped by for so long is that no one uses batch > mode, probably because people say "it's slow". We need durability so this > isn't an option. > However it doesn't need to be this slow. > Also, if you write a row that is larger than the commit log size it silently > (warn) fails to put it in the commit log. This is not ideal for batch mode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Reopened] (CASSANDRA-6764) Using Batch commitlog_sync is slow and doesn't actually batch writes
[ https://issues.apache.org/jira/browse/CASSANDRA-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict reopened CASSANDRA-6764: - > Using Batch commitlog_sync is slow and doesn't actually batch writes > > > Key: CASSANDRA-6764 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6764 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: John Carrino >Assignee: John Carrino > Fix For: 2.1 beta2 > > Attachments: cassandra_6764_v2.patch, cassandra_6764_v3.patch > > > The assumption behind batch commit mode is that the client does it's own > batching and wants to wait until the write is durable before returning. The > problem is that the queue that cassandra uses under the covers only allows > for a single ROW (RowMutation) per thread (concurrent_writes). This means > that commitlog_sync_batch_window_in_ms should really be called sleep_between > each_concurrent_writes_rows_in_ms. > I assume the reason this slipped by for so long is that no one uses batch > mode, probably because people say "it's slow". We need durability so this > isn't an option. > However it doesn't need to be this slow. > Also, if you write a row that is larger than the commit log size it silently > (warn) fails to put it in the commit log. This is not ideal for batch mode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6825) COUNT(*) with WHERE not finding all the matching rows
[ https://issues.apache.org/jira/browse/CASSANDRA-6825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951200#comment-13951200 ] Tyler Hobbs commented on CASSANDRA-6825: [~wtmitchell3] thanks! I can reproduce the issue now, so I should be able to track down what's going on. > COUNT(*) with WHERE not finding all the matching rows > - > > Key: CASSANDRA-6825 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6825 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: quad core Windows7 x64, single node cluster > Cassandra 2.0.5 >Reporter: Bill Mitchell >Assignee: Tyler Hobbs > Attachments: cassandra.log, selectpartitions.zip, > selectrowcounts.txt, testdb_1395372407904.zip, testdb_1395372407904.zip > > > Investigating another problem, I needed to do COUNT(*) on the several > partitions of a table immediately after a test case ran, and I discovered > that count(*) on the full table and on each of the partitions returned > different counts. > In particular case, SELECT COUNT(*) FROM sr LIMIT 100; returned the > expected count from the test 9 rows. The composite primary key splits > the logical row into six distinct partitions, and when I issue a query asking > for the total across all six partitions, the returned result is only 83999. > Drilling down, I find that SELECT * from sr WHERE s = 5 AND l = 11 AND > partition = 0; returns 30,000 rows, but a SELECT COUNT(*) with the identical > WHERE predicate reports only 14,000. > This is failing immediately after running a single small test, such that > there are only two SSTables, sr-jb-1 and sr-jb-2. Compaction never needed to > run. > In selectrowcounts.txt is a copy of the cqlsh output showing the incorrect > count(*) results. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (CASSANDRA-6948) After Bootstrap or Replace node startup, EXPIRING_MAP_REAPER is shutdown and cannot be restarted, causing callbacks to collect indefinitely
[ https://issues.apache.org/jira/browse/CASSANDRA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict updated CASSANDRA-6948: Attachment: 6948.txt After reading CASSANDRA-3335 and CASSANDRA-3737, I cannot find a good justification for the shutdown method: # The executor service itself blocks nothing (it is independent of the callback registration, only cleans up concurrently), and is on a daemon thread, so is completely harmless. # The shutdown parameter which ostensibly prevents callbacks from being added both does not work, and would be dangerous if it _did_ work, as any thread caught before a _restart_ would block indefinitely, and having a thread restart from that position after a clearing of the map seems problematic to me also; it seems much simpler to just let everyone be, since that's what is happening already and seems to work fine. > After Bootstrap or Replace node startup, EXPIRING_MAP_REAPER is shutdown and > cannot be restarted, causing callbacks to collect indefinitely > --- > > Key: CASSANDRA-6948 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6948 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Keith Wright >Assignee: Benedict > Attachments: 6948.debug.txt, 6948.txt, Screen Shot 2014-03-28 at > 11.27.56 AM.png, Screen Shot 2014-03-28 at 11.29.24 AM.png, > cassandra.log.min, cassandra.yaml, logs.old.tar.gz, logs.tar.gz, > system.log.1.gz, system.log.gz > > > Since ExpiringMap.shutdown() shuts down the static executor service, it > cannot be restarted (and in fact reset() makes no attempt to do so). As such > callbacks that receive no response are never removed from the map, and > eventually either than server will run out of memory or will loop around the > integer space and start reusing messageids that have not been expired, > causing assertions to be thrown and messages to fail to be sent. It appears > that this situation only arises on bootstrap or node replacement, as > MessagingService is shutdown before being attached to the listen address. > This can cause the following errors to begin occurring in the log: > ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638 > ErrorMessage.java (line 222) Unexpected exception during request > java.lang.AssertionError: Callback already exists for id -1665979622! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984) > at > org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449) > at > org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524) > at > org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521) > at > org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505) > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358) > at > org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131) > at > org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) > at > org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > at > org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [ReplicateOnWriteStage:102766] 2014-03-28 13:32:10,638 > CassandraDaemon.java (line 196) Exception in thread > Thread[ReplicateOnWriteStage:102766,5,main] > java.lang.AssertionError: Callback already exists for id -1665979620! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@3bdb1a75, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.j
[jira] [Commented] (CASSANDRA-6890) Remove mmap'ed i/o from read path
[ https://issues.apache.org/jira/browse/CASSANDRA-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951179#comment-13951179 ] Benedict commented on CASSANDRA-6890: - FTR, with compression enabled it will only use mmapped IO for reading the index file, not the compressed data. I'm not tremendously surprised that we see a drop in throughput vs mmapped files, as we compress 64K chunks, so we have to read a minimum of 64K (+ readahead), and also decompress it for every row we lookup, which in this case is only ~200 bytes. I would expect the performance difference to be smaller for larger rows. Tuning performance for standard IO is trickier for the user, and we don't currently do much to help them. Personally, I am -1 on removing mmapped IO as an option until we have an in-process uncompressed page cache that mitigates these problems. > Remove mmap'ed i/o from read path > - > > Key: CASSANDRA-6890 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6890 > Project: Cassandra > Issue Type: Improvement >Reporter: Joshua McKenzie >Assignee: Joshua McKenzie >Priority: Minor > Fix For: 3.0 > > Attachments: mmap_gc.jpg, mmap_jstat.txt, mmap_perf.txt, > nommap_gc.jpg, nommap_jstat.txt > > > Since we actively unmap unreferenced SSTR's and also copy data out of those > readers on the read path, the current memory mapped i/o is a lot of > complexity for very little payoff. Clean out the mmapp'ed i/o on the read > path. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6890) Remove mmap'ed i/o from read path
[
https://issues.apache.org/jira/browse/CASSANDRA-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951160#comment-13951160
]
Joshua McKenzie commented on CASSANDRA-6890:
Good call - it looks like cassandra-stress doesn't have compression enabled on
the keyspaces it creates on write. Enabling compression gives very similar
results between the memory mapped and non-memory-mapped path. With that said -
performance differential provided by the memory-mapped code-path looks like
it's pretty substantial. Are we sure we want to remove this?
{code:title=Performance comparison}
mmap, trunk, compression off, 2G heap default ratio:
id, ops ,op/s,adj op/s, key/s,mean, med,
.95, .99,.999, max, time
4 threadCount, 1166950 , 38737, 38737, 38737, 0.1, 0.1,
0.1, 0.1, 0.5,22.4, 30.1
8 threadCount, 1619550 , 53634, -0, 53634, 0.1, 0.1,
0.2, 0.2, 0.7,18.5, 30.2
16 threadCount, 1880100 , 61833, -0, 61833, 0.3, 0.2,
0.3, 0.5, 1.0,31.7, 30.4
24 threadCount, 1951150 , 63979, -0, 63979, 0.4, 0.3,
0.4, 0.7, 1.7,30.9, 30.5
36 threadCount, 1929450 , 63171, 63175, 63171, 0.6, 0.5,
0.7, 1.0, 3.2,20.7, 30.5
54 threadCount, 1952900 , 63845, -0, 63845, 0.8, 0.8,
1.0, 1.5, 7.8,31.0, 30.6
81 threadCount, 1890200 , 61689, 61721, 61689, 1.3, 1.3,
1.4, 2.3,17.8,33.0, 30.6
mmap, trunk, compression on, 2G heap default ratio:
id, ops ,op/s,adj op/s, key/s,mean, med,
.95, .99,.999, max, time
4 threadCount, 609850, 20237, 20237, 20237, 0.2, 0.2,
0.2, 0.3, 0.6,28.8, 30.1
8 threadCount, 915400, 30249, 30264, 30249, 0.3, 0.2,
0.4, 0.6, 1.2,26.3, 30.3
16 threadCount, 936950, 30638, 30661, 30638, 0.5, 0.3,
1.1, 6.0,17.7,94.0, 30.6
24 threadCount, 1003450 , 32452, -0, 32452, 0.7, 0.3,
1.9,11.1,26.5, 172.4, 30.9
36 threadCount, 1095500 , 34665, -0, 34665, 1.0, 0.3,
1.4,23.3,50.3, 129.1, 31.6
54 threadCount, 1083200 , 33426, -0, 33426, 1.6, 0.8,
1.8,29.3,63.5, 250.6, 32.4
81 threadCount, 1080850 , 32649, -0, 32649, 2.5, 1.6,
3.0,32.5,73.8, 784.0, 33.1
no-mmap, trunk, compression on, 2G heap default ratio:
id, ops ,op/s,adj op/s, key/s,mean, med,
.95, .99,.999, max, time
4 threadCount, 581200, 19273, 19287, 19273, 0.2, 0.2,
0.3, 0.3, 0.6,24.0, 30.2
8 threadCount, 904150, 29906, 29919, 29906, 0.3, 0.2,
0.4, 0.6, 1.1,37.0, 30.2
16 threadCount, 998650, 32770, -0, 32770, 0.5, 0.3,
1.4, 4.2, 8.7,38.4, 30.5
24 threadCount, 1035250 , 33685, -0, 33685, 0.7, 0.3,
2.9, 9.1,17.9,44.2, 30.7
36 threadCount, 1059950 , 33396, -0, 33396, 1.1, 0.3,
1.5,22.4,47.5, 139.1, 31.7
54 threadCount, 1070250 , 32483, -0, 32483, 1.7, 0.8,
1.8,28.7,60.4, 253.5, 32.9
{code}
For read-heavy workloads on moderately sized data-sets, the option to keep
compression off to improve performance seems pretty compelling from all the #'s
I'm seeing.
> Remove mmap'ed i/o from read path
> -
>
> Key: CASSANDRA-6890
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6890
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Joshua McKenzie
>Assignee: Joshua McKenzie
>Priority: Minor
> Fix For: 3.0
>
> Attachments: mmap_gc.jpg, mmap_jstat.txt, mmap_perf.txt,
> nommap_gc.jpg, nommap_jstat.txt
>
>
> Since we actively unmap unreferenced SSTR's and also copy data out of those
> readers on the read path, the current memory mapped i/o is a lot of
> complexity for very little payoff. Clean out the mmapp'ed i/o on the read
> path.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-6890) Remove mmap'ed i/o from read path
[ https://issues.apache.org/jira/browse/CASSANDRA-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13950941#comment-13950941 ] Jonathan Ellis commented on CASSANDRA-6890: --- Are you explicitly disabling compression or does something on the compressed path still use mmap? > Remove mmap'ed i/o from read path > - > > Key: CASSANDRA-6890 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6890 > Project: Cassandra > Issue Type: Improvement >Reporter: Joshua McKenzie >Assignee: Joshua McKenzie >Priority: Minor > Fix For: 3.0 > > Attachments: mmap_gc.jpg, mmap_jstat.txt, mmap_perf.txt, > nommap_gc.jpg, nommap_jstat.txt > > > Since we actively unmap unreferenced SSTR's and also copy data out of those > readers on the read path, the current memory mapped i/o is a lot of > complexity for very little payoff. Clean out the mmapp'ed i/o on the read > path. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (CASSANDRA-6948) Native & ReplicateOnWriteStage AssertionError: Callback already exists
[ https://issues.apache.org/jira/browse/CASSANDRA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict reassigned CASSANDRA-6948: --- Assignee: Benedict > Native & ReplicateOnWriteStage AssertionError: Callback already exists > -- > > Key: CASSANDRA-6948 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6948 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Keith Wright >Assignee: Benedict > Attachments: cassandra.yaml, system.log.1.gz, system.log.gz > > > We are using the datastax driver and have seen the following exceptions > appear periodically in our cluster in "groups". The client will gradually > start sending less write requests to the nodes where this occurs the most > often. This may be related to CASSANDRA-6476. Exception from logs: > ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638 > ErrorMessage.java (line 222) Unexpected exception during request > java.lang.AssertionError: Callback already exists for id -1665979622! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984) > at > org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449) > at > org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524) > at > org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521) > at > org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505) > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358) > at > org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131) > at > org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) > at > org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > at > org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [ReplicateOnWriteStage:102766] 2014-03-28 13:32:10,638 > CassandraDaemon.java (line 196) Exception in thread > Thread[ReplicateOnWriteStage:102766,5,main] > java.lang.AssertionError: Callback already exists for id -1665979620! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@3bdb1a75, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:806) > at > org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1074) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6943) UpdateFunction.abortEarly can cause BTree.update to leave its Builder in a bad state, which affects future operations
[ https://issues.apache.org/jira/browse/CASSANDRA-6943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13950765#comment-13950765 ] Benedict commented on CASSANDRA-6943: - For anyone looking for a little more information: this problem would be hit when inserting a larger number of records into an existing record than were currently present in that record (excluding empty records) causing a over-spill into a new parent node, after which there was a competing update to the underlying record, and the update that overspilled aborted before finishing the construction of its replacement btree. In this case it did not clear the state of the builder for the spilled-over parent, as it only visited all those from its original root/parent down to reset the state (it would ascend to the real parent on finishing construction), so that when the next update finished it included this old partially constructed parent, resulting in a malformed btree. The simple solution is to always go up resetting all parents on completion, as well as down > UpdateFunction.abortEarly can cause BTree.update to leave its Builder in a > bad state, which affects future operations > - > > Key: CASSANDRA-6943 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6943 > Project: Cassandra > Issue Type: Bug >Reporter: Russ Hatch >Assignee: Benedict > Fix For: 2.1 beta2 > > Attachments: 6943.2.txt, 6943.txt, node4.log, node4_jstack.log, > node5.log, node5_jstack.log, node6.log, node6_jstack.log, node7.log, > node7_jstack.log, screenshot.png, screenshot2.png, stress_jstack.log > > > Running performance scenarios I have seen this characteristic drop in > performance happen several times. A similar effect was reproduced in another > test cluster. Operations eventually come to a standstill. > !screenshot.png! > !screenshot2.png! -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6948) Native & ReplicateOnWriteStage AssertionError: Callback already exists
[ https://issues.apache.org/jira/browse/CASSANDRA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13950715#comment-13950715 ] Keith Wright commented on CASSANDRA-6948: - Another stack where its not a mutate counter: java.lang.AssertionError: Callback already exists for id -1665977341! (CallbackInfo(target=/10.106.160.84, callback=org.apache.cassandra.service.WriteResponseHandler@5a74c5ff, serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) at org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) at org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) at org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:806) at org.apache.cassandra.service.StorageProxy$2.apply(StorageProxy.java:118) at org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:682) at org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:454) at org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524) at org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521) at org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505) at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358) at org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131) at org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) > Native & ReplicateOnWriteStage AssertionError: Callback already exists > -- > > Key: CASSANDRA-6948 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6948 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Keith Wright > > We are using the datastax driver and have seen the following exceptions > appear periodically in our cluster in "groups". The client will gradually > start sending less write requests to the nodes where this occurs the most > often. This may be related to CASSANDRA-6476. Exception from logs: > ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638 > ErrorMessage.java (line 222) Unexpected exception during request > java.lang.AssertionError: Callback already exists for id -1665979622! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984) > at > org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449) > at > org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524) > at > org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521) > at > org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505) > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358) > at > org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131) > at > org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) > at > org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > at > org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [ReplicateOnWriteStage:102766] 2014-03-28 13:32:10,63
[jira] [Commented] (CASSANDRA-6948) Native & ReplicateOnWriteStage AssertionError: Callback already exists
[ https://issues.apache.org/jira/browse/CASSANDRA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13950936#comment-13950936 ] Keith Wright commented on CASSANDRA-6948: - One other thing to note is that this cluster is now our production cluster. We just migrated from a 1.2 cluster where we were having issues. As part of this migration, our client code was instantiating two instances of cluster from the datastax client to allow for writing to go to both clusters. Could that be a possible cause? We are going to restart all of our nodes as we are no longer writing to both clusters. Because this is now our production cluster, I am reluctant to apply patches. > Native & ReplicateOnWriteStage AssertionError: Callback already exists > -- > > Key: CASSANDRA-6948 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6948 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Keith Wright >Assignee: Benedict > Attachments: 6948.debug.txt, Screen Shot 2014-03-28 at 11.27.56 > AM.png, Screen Shot 2014-03-28 at 11.29.24 AM.png, cassandra.log.gz, > cassandra.yaml, logs.old.tar.gz, logs.tar.gz, system.log.1.gz, system.log.gz > > > We are using the datastax driver and have seen the following exceptions > appear periodically in our cluster in "groups". The client will gradually > start sending less write requests to the nodes where this occurs the most > often. This may be related to CASSANDRA-6476. Exception from logs: > ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638 > ErrorMessage.java (line 222) Unexpected exception during request > java.lang.AssertionError: Callback already exists for id -1665979622! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984) > at > org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449) > at > org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524) > at > org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521) > at > org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505) > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358) > at > org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131) > at > org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) > at > org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > at > org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [ReplicateOnWriteStage:102766] 2014-03-28 13:32:10,638 > CassandraDaemon.java (line 196) Exception in thread > Thread[ReplicateOnWriteStage:102766,5,main] > java.lang.AssertionError: Callback already exists for id -1665979620! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@3bdb1a75, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:806) > at > org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1074) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (CASSANDRA-6948) Native & ReplicateOnWriteStage AssertionError: Callback already exists
[ https://issues.apache.org/jira/browse/CASSANDRA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keith Wright updated CASSANDRA-6948: Attachment: system.log.1.gz system.log.gz Attached for one of the nodes showing the issue. > Native & ReplicateOnWriteStage AssertionError: Callback already exists > -- > > Key: CASSANDRA-6948 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6948 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Keith Wright > Attachments: cassandra.yaml, system.log.1.gz, system.log.gz > > > We are using the datastax driver and have seen the following exceptions > appear periodically in our cluster in "groups". The client will gradually > start sending less write requests to the nodes where this occurs the most > often. This may be related to CASSANDRA-6476. Exception from logs: > ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638 > ErrorMessage.java (line 222) Unexpected exception during request > java.lang.AssertionError: Callback already exists for id -1665979622! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984) > at > org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449) > at > org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524) > at > org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521) > at > org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505) > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358) > at > org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131) > at > org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) > at > org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > at > org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [ReplicateOnWriteStage:102766] 2014-03-28 13:32:10,638 > CassandraDaemon.java (line 196) Exception in thread > Thread[ReplicateOnWriteStage:102766,5,main] > java.lang.AssertionError: Callback already exists for id -1665979620! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@3bdb1a75, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:806) > at > org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1074) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6948) Native & ReplicateOnWriteStage AssertionError: Callback already exists
[ https://issues.apache.org/jira/browse/CASSANDRA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13950744#comment-13950744 ] Benedict commented on CASSANDRA-6948: - Could you attach a jstack of the machine throwing the exception, and a copy of your cassandra.yaml? > Native & ReplicateOnWriteStage AssertionError: Callback already exists > -- > > Key: CASSANDRA-6948 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6948 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Keith Wright > > We are using the datastax driver and have seen the following exceptions > appear periodically in our cluster in "groups". The client will gradually > start sending less write requests to the nodes where this occurs the most > often. This may be related to CASSANDRA-6476. Exception from logs: > ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638 > ErrorMessage.java (line 222) Unexpected exception during request > java.lang.AssertionError: Callback already exists for id -1665979622! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984) > at > org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449) > at > org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524) > at > org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521) > at > org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505) > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358) > at > org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131) > at > org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) > at > org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > at > org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [ReplicateOnWriteStage:102766] 2014-03-28 13:32:10,638 > CassandraDaemon.java (line 196) Exception in thread > Thread[ReplicateOnWriteStage:102766,5,main] > java.lang.AssertionError: Callback already exists for id -1665979620! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@3bdb1a75, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:806) > at > org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1074) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6948) Native & ReplicateOnWriteStage AssertionError: Callback already exists
[ https://issues.apache.org/jira/browse/CASSANDRA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13950958#comment-13950958 ] Keith Wright commented on CASSANDRA-6948: - Cassandra.log contains logging output from 010 node that just showed the issue including thread dump prior to restart. Let me know if you need me to upload it again. These were completely distinct clusters. FWIW, I have also noticed that applying DDLs on nodes doesn't appear to always get applied to all nodes. I am planning on filing a ticket for that as soon as I can reproduce in a testing environment. Workaround is to just apply the same DDL on the nodes where the change did not appear. I've often found 008 (the same node showing the assertion error to the highest degree) to be the DDL culprit. > Native & ReplicateOnWriteStage AssertionError: Callback already exists > -- > > Key: CASSANDRA-6948 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6948 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Keith Wright >Assignee: Benedict > Attachments: 6948.debug.txt, Screen Shot 2014-03-28 at 11.27.56 > AM.png, Screen Shot 2014-03-28 at 11.29.24 AM.png, cassandra.log.gz, > cassandra.yaml, logs.old.tar.gz, logs.tar.gz, system.log.1.gz, system.log.gz > > > We are using the datastax driver and have seen the following exceptions > appear periodically in our cluster in "groups". The client will gradually > start sending less write requests to the nodes where this occurs the most > often. This may be related to CASSANDRA-6476. Exception from logs: > ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638 > ErrorMessage.java (line 222) Unexpected exception during request > java.lang.AssertionError: Callback already exists for id -1665979622! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984) > at > org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449) > at > org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524) > at > org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521) > at > org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505) > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358) > at > org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131) > at > org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) > at > org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > at > org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [ReplicateOnWriteStage:102766] 2014-03-28 13:32:10,638 > CassandraDaemon.java (line 196) Exception in thread > Thread[ReplicateOnWriteStage:102766,5,main] > java.lang.AssertionError: Callback already exists for id -1665979620! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@3bdb1a75, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:806) > at > org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1074) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6
[jira] [Commented] (CASSANDRA-6948) Native & ReplicateOnWriteStage AssertionError: Callback already exists
[ https://issues.apache.org/jira/browse/CASSANDRA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951044#comment-13951044 ] Keith Wright commented on CASSANDRA-6948: - Ah so its possible its the 4B since restart. I did a bulk load of data into this new cluster from hadoop directly via CQL. Was running at 150k writes/sec across the cluster. Given RF 3 that would be 450K/sec across 11 nodes for ~2 days => ~7 billion writes. From tpstats of a node: [kwright@lxpcasdal011 ~]$ nodetool tpstats Pool NameActive Pending Completed Blocked All time blocked ReadStage 3 4 1398160265 0 0 RequestResponseStage 0 0 9085977126 0 0 MutationStage 0 012080278784 0 0 ReadRepairStage 0 0 125960131 0 0 ReplicateOnWriteStage 0 0 439434930 0 0 GossipStage 0 04119112 0 0 AntiEntropyStage 0 0 0 0 0 MigrationStage0 0449 0 0 MemoryMeter 0 0588 0 0 MemtablePostFlusher 0 0 150778 0 0 FlushWriter 0 0 127833 0 86 MiscStage 0 0 0 0 0 PendingRangeCalculator0 0 23 0 0 commitlog_archiver0 0 0 0 0 InternalResponseStage 0 0259 0 0 HintedHandoff 2 4839 0 0 Unfortunately I don't have any older logs than what I provided, sorry. I'll try to extract the stacktrace info for you and attach > Native & ReplicateOnWriteStage AssertionError: Callback already exists > -- > > Key: CASSANDRA-6948 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6948 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Keith Wright >Assignee: Benedict > Attachments: 6948.debug.txt, Screen Shot 2014-03-28 at 11.27.56 > AM.png, Screen Shot 2014-03-28 at 11.29.24 AM.png, cassandra.log.gz, > cassandra.yaml, logs.old.tar.gz, logs.tar.gz, system.log.1.gz, system.log.gz > > > We are using the datastax driver and have seen the following exceptions > appear periodically in our cluster in "groups". The client will gradually > start sending less write requests to the nodes where this occurs the most > often. This may be related to CASSANDRA-6476. Exception from logs: > ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638 > ErrorMessage.java (line 222) Unexpected exception during request > java.lang.AssertionError: Callback already exists for id -1665979622! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984) > at > org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449) > at > org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524) > at > org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521) > at > org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505) > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358) > at > org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131) > at > org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) > at > org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > at > org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > at > java.util.concurrent.ThreadPoolExecutor.runWorker
[jira] [Updated] (CASSANDRA-6948) After Bootstrap or Replace node startup, EXPIRING_MAP_REAPER is shutdown and cannot be restarted, causing callbacks to collect indefinitely
[ https://issues.apache.org/jira/browse/CASSANDRA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict updated CASSANDRA-6948: Description: Since ExpiringMap.shutdown() shuts down the static executor service, it cannot be restarted (and in fact reset() makes no attempt to do so). As such callbacks that receive no response are never removed from the map, and eventually either than server will run out of memory or will loop around the integer space and start reusing messageids that have not been expired, causing assertions to be thrown and messages to fail to be sent. It appears that this situation only arises on bootstrap or node replacement, as MessagingService is shutdown before being attached to the listen address. This can cause the following errors to begin occurring in the log: ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638 ErrorMessage.java (line 222) Unexpected exception during request java.lang.AssertionError: Callback already exists for id -1665979622! (CallbackInfo(target=/10.106.160.84, callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea, serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) at org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) at org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) at org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984) at org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449) at org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524) at org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521) at org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505) at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358) at org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131) at org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) ERROR [ReplicateOnWriteStage:102766] 2014-03-28 13:32:10,638 CassandraDaemon.java (line 196) Exception in thread Thread[ReplicateOnWriteStage:102766,5,main] java.lang.AssertionError: Callback already exists for id -1665979620! (CallbackInfo(target=/10.106.160.84, callback=org.apache.cassandra.service.WriteResponseHandler@3bdb1a75, serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) at org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) at org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) at org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:806) at org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1074) at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) was: We are using the datastax driver and have seen the following exceptions appear periodically in our cluster in "groups". The client will gradually start sending less write requests to the nodes where this occurs the most often. This may be related to CASSANDRA-6476. Exception from logs: ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638 ErrorMessage.java (line 222) Unexpected exception during request java.lang.AssertionError: Callback already exists for id -1665979622! (CallbackInfo(target=/10.106.160.84, callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea, serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) at org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) at org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) at org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984) at org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449) at org.apac
[jira] [Commented] (CASSANDRA-6948) Native & ReplicateOnWriteStage AssertionError: Callback already exists
[ https://issues.apache.org/jira/browse/CASSANDRA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951060#comment-13951060 ] Benedict commented on CASSANDRA-6948: - OK, I think I have a plausible explanation: Was this the first time the nodes in the new cluster had been started? i.e. have they ever exhibited the issue after being restarted? > Native & ReplicateOnWriteStage AssertionError: Callback already exists > -- > > Key: CASSANDRA-6948 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6948 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Keith Wright >Assignee: Benedict > Attachments: 6948.debug.txt, Screen Shot 2014-03-28 at 11.27.56 > AM.png, Screen Shot 2014-03-28 at 11.29.24 AM.png, cassandra.log.min, > cassandra.yaml, logs.old.tar.gz, logs.tar.gz, system.log.1.gz, system.log.gz > > > We are using the datastax driver and have seen the following exceptions > appear periodically in our cluster in "groups". The client will gradually > start sending less write requests to the nodes where this occurs the most > often. This may be related to CASSANDRA-6476. Exception from logs: > ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638 > ErrorMessage.java (line 222) Unexpected exception during request > java.lang.AssertionError: Callback already exists for id -1665979622! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984) > at > org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449) > at > org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524) > at > org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521) > at > org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505) > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358) > at > org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131) > at > org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) > at > org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > at > org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [ReplicateOnWriteStage:102766] 2014-03-28 13:32:10,638 > CassandraDaemon.java (line 196) Exception in thread > Thread[ReplicateOnWriteStage:102766,5,main] > java.lang.AssertionError: Callback already exists for id -1665979620! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@3bdb1a75, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:806) > at > org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1074) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (CASSANDRA-6948) After Bootstrap or Replace node startup, EXPIRING_MAP_REAPER is shutdown and cannot be restarted, causing callbacks to collect indefinitely
[ https://issues.apache.org/jira/browse/CASSANDRA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict updated CASSANDRA-6948: Summary: After Bootstrap or Replace node startup, EXPIRING_MAP_REAPER is shutdown and cannot be restarted, causing callbacks to collect indefinitely (was: Native & ReplicateOnWriteStage AssertionError: Callback already exists) > After Bootstrap or Replace node startup, EXPIRING_MAP_REAPER is shutdown and > cannot be restarted, causing callbacks to collect indefinitely > --- > > Key: CASSANDRA-6948 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6948 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Keith Wright >Assignee: Benedict > Attachments: 6948.debug.txt, Screen Shot 2014-03-28 at 11.27.56 > AM.png, Screen Shot 2014-03-28 at 11.29.24 AM.png, cassandra.log.min, > cassandra.yaml, logs.old.tar.gz, logs.tar.gz, system.log.1.gz, system.log.gz > > > We are using the datastax driver and have seen the following exceptions > appear periodically in our cluster in "groups". The client will gradually > start sending less write requests to the nodes where this occurs the most > often. This may be related to CASSANDRA-6476. Exception from logs: > ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638 > ErrorMessage.java (line 222) Unexpected exception during request > java.lang.AssertionError: Callback already exists for id -1665979622! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984) > at > org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449) > at > org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524) > at > org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521) > at > org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505) > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358) > at > org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131) > at > org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) > at > org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > at > org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [ReplicateOnWriteStage:102766] 2014-03-28 13:32:10,638 > CassandraDaemon.java (line 196) Exception in thread > Thread[ReplicateOnWriteStage:102766,5,main] > java.lang.AssertionError: Callback already exists for id -1665979620! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@3bdb1a75, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:806) > at > org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1074) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6948) Native & ReplicateOnWriteStage AssertionError: Callback already exists
[ https://issues.apache.org/jira/browse/CASSANDRA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951103#comment-13951103 ] Keith Wright commented on CASSANDRA-6948: - Great, now that all nodes have been restarted, I'll make sure to let you know if I see the issue again. In fact, I'm planning on bootstrapping some more nodes next week so I'll also validate that restart after bootstrap ensures the issue does not occur. Any chance this is somehow related to the issue where I'm not seeing DDLs apply across all nodes even though they're shown as successful? > Native & ReplicateOnWriteStage AssertionError: Callback already exists > -- > > Key: CASSANDRA-6948 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6948 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Keith Wright >Assignee: Benedict > Attachments: 6948.debug.txt, Screen Shot 2014-03-28 at 11.27.56 > AM.png, Screen Shot 2014-03-28 at 11.29.24 AM.png, cassandra.log.min, > cassandra.yaml, logs.old.tar.gz, logs.tar.gz, system.log.1.gz, system.log.gz > > > We are using the datastax driver and have seen the following exceptions > appear periodically in our cluster in "groups". The client will gradually > start sending less write requests to the nodes where this occurs the most > often. This may be related to CASSANDRA-6476. Exception from logs: > ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638 > ErrorMessage.java (line 222) Unexpected exception during request > java.lang.AssertionError: Callback already exists for id -1665979622! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984) > at > org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449) > at > org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524) > at > org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521) > at > org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505) > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358) > at > org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131) > at > org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) > at > org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > at > org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [ReplicateOnWriteStage:102766] 2014-03-28 13:32:10,638 > CassandraDaemon.java (line 196) Exception in thread > Thread[ReplicateOnWriteStage:102766,5,main] > java.lang.AssertionError: Callback already exists for id -1665979620! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@3bdb1a75, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:806) > at > org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1074) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6948) Native & ReplicateOnWriteStage AssertionError: Callback already exists
[ https://issues.apache.org/jira/browse/CASSANDRA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951087#comment-13951087 ] Benedict commented on CASSANDRA-6948: - OK. Well good news is it probably won't bite you again until you start a new cluster or bootstrap a new node into the cluster, and restarting a newly bootstrapped node immediately after bootstrap should fix it. Note that replacing a node can trigger the same issue. I will update the description with details and provide a patch. > Native & ReplicateOnWriteStage AssertionError: Callback already exists > -- > > Key: CASSANDRA-6948 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6948 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Keith Wright >Assignee: Benedict > Attachments: 6948.debug.txt, Screen Shot 2014-03-28 at 11.27.56 > AM.png, Screen Shot 2014-03-28 at 11.29.24 AM.png, cassandra.log.min, > cassandra.yaml, logs.old.tar.gz, logs.tar.gz, system.log.1.gz, system.log.gz > > > We are using the datastax driver and have seen the following exceptions > appear periodically in our cluster in "groups". The client will gradually > start sending less write requests to the nodes where this occurs the most > often. This may be related to CASSANDRA-6476. Exception from logs: > ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638 > ErrorMessage.java (line 222) Unexpected exception during request > java.lang.AssertionError: Callback already exists for id -1665979622! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984) > at > org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449) > at > org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524) > at > org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521) > at > org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505) > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358) > at > org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131) > at > org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) > at > org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > at > org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [ReplicateOnWriteStage:102766] 2014-03-28 13:32:10,638 > CassandraDaemon.java (line 196) Exception in thread > Thread[ReplicateOnWriteStage:102766,5,main] > java.lang.AssertionError: Callback already exists for id -1665979620! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@3bdb1a75, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:806) > at > org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1074) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (CASSANDRA-6950) Secondary index query fails with tc range query when ordered by DESC
Andre Campeau created CASSANDRA-6950:
Summary: Secondary index query fails with tc range query when
ordered by DESC
Key: CASSANDRA-6950
URL: https://issues.apache.org/jira/browse/CASSANDRA-6950
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: RHEL 6.3 virtual guest,
apache-cassandra-2.0.6-SNAPSHOT-src.tar.gz from build #284 (also tried with
2.0.5 with CASSANDRA- patch custom-applied with same result).
Reporter: Andre Campeau
create table test4 ( name text, lname text, tc bigint, record text, PRIMARY
KEY ((name, lname), tc)) WITH CLUSTERING ORDER BY (tc DESC) AND
compaction={'class': 'LeveledCompactionStrategy'};
create index test4_index ON test4(lname);
Populate it with some data and non-zero tc values, then try:
select * from test4 where lname='blah' and tc>0 allow filtering;
And, (0 rows) returned, even though there are rows which should be found.
When I create the table using CLUSTERING ORDER BY (tc ASC), the above query
works. Rows are correctly returned based on the range check.
Tried various combinations but with descending order on tc nothing works.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-6890) Remove mmap'ed i/o from read path
[ https://issues.apache.org/jira/browse/CASSANDRA-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13950927#comment-13950927 ] Joshua McKenzie commented on CASSANDRA-6890: Tested out upping new gen to 2G on a 4G heap. No noticeable impact on performance though new gen GC count and survivor flip-flopping declined as expected. I'll start throwing some timers into the BufferedPooling* file path vs. Mmapped* unless anyone has any better ideas. > Remove mmap'ed i/o from read path > - > > Key: CASSANDRA-6890 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6890 > Project: Cassandra > Issue Type: Improvement >Reporter: Joshua McKenzie >Assignee: Joshua McKenzie >Priority: Minor > Fix For: 3.0 > > Attachments: mmap_gc.jpg, mmap_jstat.txt, mmap_perf.txt, > nommap_gc.jpg, nommap_jstat.txt > > > Since we actively unmap unreferenced SSTR's and also copy data out of those > readers on the read path, the current memory mapped i/o is a lot of > complexity for very little payoff. Clean out the mmapp'ed i/o on the read > path. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (CASSANDRA-6948) Native & ReplicateOnWriteStage AssertionError: Callback already exists
[ https://issues.apache.org/jira/browse/CASSANDRA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keith Wright updated CASSANDRA-6948: Attachment: logs.old.tar.gz cassandra.log.gz This issue actually just started occurring on another node (010). When captured the thread dump in the cassandra log file attached prior restarting the node. Have also attached older logs from the previous node (008) > Native & ReplicateOnWriteStage AssertionError: Callback already exists > -- > > Key: CASSANDRA-6948 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6948 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Keith Wright >Assignee: Benedict > Attachments: 6948.debug.txt, Screen Shot 2014-03-28 at 11.27.56 > AM.png, Screen Shot 2014-03-28 at 11.29.24 AM.png, cassandra.log.gz, > cassandra.yaml, logs.old.tar.gz, logs.tar.gz, system.log.1.gz, system.log.gz > > > We are using the datastax driver and have seen the following exceptions > appear periodically in our cluster in "groups". The client will gradually > start sending less write requests to the nodes where this occurs the most > often. This may be related to CASSANDRA-6476. Exception from logs: > ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638 > ErrorMessage.java (line 222) Unexpected exception during request > java.lang.AssertionError: Callback already exists for id -1665979622! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984) > at > org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449) > at > org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524) > at > org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521) > at > org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505) > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358) > at > org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131) > at > org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) > at > org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > at > org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [ReplicateOnWriteStage:102766] 2014-03-28 13:32:10,638 > CassandraDaemon.java (line 196) Exception in thread > Thread[ReplicateOnWriteStage:102766,5,main] > java.lang.AssertionError: Callback already exists for id -1665979620! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@3bdb1a75, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:806) > at > org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1074) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6948) Native & ReplicateOnWriteStage AssertionError: Callback already exists
[ https://issues.apache.org/jira/browse/CASSANDRA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951109#comment-13951109 ] Benedict commented on CASSANDRA-6948: - bq. Any chance this is somehow related to the issue where I'm not seeing DDLs apply across all nodes even though they're shown as successful? It's quite possible, yes, although cannot say for certain. If you don't see any further DDL problems it's probably down to this, but otherwise please file a separate issue. > Native & ReplicateOnWriteStage AssertionError: Callback already exists > -- > > Key: CASSANDRA-6948 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6948 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Keith Wright >Assignee: Benedict > Attachments: 6948.debug.txt, Screen Shot 2014-03-28 at 11.27.56 > AM.png, Screen Shot 2014-03-28 at 11.29.24 AM.png, cassandra.log.min, > cassandra.yaml, logs.old.tar.gz, logs.tar.gz, system.log.1.gz, system.log.gz > > > We are using the datastax driver and have seen the following exceptions > appear periodically in our cluster in "groups". The client will gradually > start sending less write requests to the nodes where this occurs the most > often. This may be related to CASSANDRA-6476. Exception from logs: > ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638 > ErrorMessage.java (line 222) Unexpected exception during request > java.lang.AssertionError: Callback already exists for id -1665979622! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984) > at > org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449) > at > org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524) > at > org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521) > at > org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505) > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358) > at > org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131) > at > org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) > at > org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > at > org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [ReplicateOnWriteStage:102766] 2014-03-28 13:32:10,638 > CassandraDaemon.java (line 196) Exception in thread > Thread[ReplicateOnWriteStage:102766,5,main] > java.lang.AssertionError: Callback already exists for id -1665979620! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@3bdb1a75, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:806) > at > org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1074) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6948) Native & ReplicateOnWriteStage AssertionError: Callback already exists
[ https://issues.apache.org/jira/browse/CASSANDRA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13950782#comment-13950782 ] Benedict commented on CASSANDRA-6948: - Do you have a full log history since the previous restart at least? > Native & ReplicateOnWriteStage AssertionError: Callback already exists > -- > > Key: CASSANDRA-6948 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6948 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Keith Wright > Attachments: cassandra.yaml > > > We are using the datastax driver and have seen the following exceptions > appear periodically in our cluster in "groups". The client will gradually > start sending less write requests to the nodes where this occurs the most > often. This may be related to CASSANDRA-6476. Exception from logs: > ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638 > ErrorMessage.java (line 222) Unexpected exception during request > java.lang.AssertionError: Callback already exists for id -1665979622! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984) > at > org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449) > at > org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524) > at > org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521) > at > org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505) > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358) > at > org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131) > at > org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) > at > org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > at > org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [ReplicateOnWriteStage:102766] 2014-03-28 13:32:10,638 > CassandraDaemon.java (line 196) Exception in thread > Thread[ReplicateOnWriteStage:102766,5,main] > java.lang.AssertionError: Callback already exists for id -1665979620! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@3bdb1a75, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:806) > at > org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1074) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (CASSANDRA-6943) UpdateFunction.abortEarly can cause BTree.update to leave its Builder in a bad state, which affects future operations
[ https://issues.apache.org/jira/browse/CASSANDRA-6943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Russ Hatch updated CASSANDRA-6943: -- Tester: Russ Hatch > UpdateFunction.abortEarly can cause BTree.update to leave its Builder in a > bad state, which affects future operations > - > > Key: CASSANDRA-6943 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6943 > Project: Cassandra > Issue Type: Bug >Reporter: Russ Hatch >Assignee: Benedict > Fix For: 2.1 beta2 > > Attachments: 6943.2.txt, 6943.txt, node4.log, node4_jstack.log, > node5.log, node5_jstack.log, node6.log, node6_jstack.log, node7.log, > node7_jstack.log, screenshot.png, screenshot2.png, stress_jstack.log > > > Running performance scenarios I have seen this characteristic drop in > performance happen several times. A similar effect was reproduced in another > test cluster. Operations eventually come to a standstill. > !screenshot.png! > !screenshot2.png! -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (CASSANDRA-6880) counters++ lock on cells, not partitions
[ https://issues.apache.org/jira/browse/CASSANDRA-6880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Yeschenko updated CASSANDRA-6880: - Reviewer: Jonathan Ellis > counters++ lock on cells, not partitions > > > Key: CASSANDRA-6880 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6880 > Project: Cassandra > Issue Type: Improvement >Reporter: Aleksey Yeschenko >Assignee: Aleksey Yeschenko > Fix For: 2.1 beta2 > > > I'm starting to think that we should switch to locking by cells, not by > partitions, when updating counters. > With the current 2.1 counters, if nothing changes, the new recommendation > would become "use smaller partitions, batch updates to the same partition", > and that goes against what we usually recommend: > 1. Prefer wide partitions to narrow partitions > 2. Don't batch counter updates (because you risk to exaggerate > undercounting/overcounting in case of a timeout) > Locking on cells would cause C* to have to grab more locks for batch counter > updates, but would give us generally more predictable performance > (independent of partition wideness), and won't force people to remodel their > data model if they have often concurrently-updated counters in the same few > wide partitions. > (It's a small change, code-wise) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6880) counters++ lock on cells, not partitions
[ https://issues.apache.org/jira/browse/CASSANDRA-6880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951101#comment-13951101 ] Aleksey Yeschenko commented on CASSANDRA-6880: -- Pushed to https://github.com/iamaleksey/cassandra/tree/6880 Want Russ to run the benchmarks first before sending to review, though. > counters++ lock on cells, not partitions > > > Key: CASSANDRA-6880 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6880 > Project: Cassandra > Issue Type: Improvement >Reporter: Aleksey Yeschenko >Assignee: Aleksey Yeschenko > Fix For: 2.1 beta2 > > > I'm starting to think that we should switch to locking by cells, not by > partitions, when updating counters. > With the current 2.1 counters, if nothing changes, the new recommendation > would become "use smaller partitions, batch updates to the same partition", > and that goes against what we usually recommend: > 1. Prefer wide partitions to narrow partitions > 2. Don't batch counter updates (because you risk to exaggerate > undercounting/overcounting in case of a timeout) > Locking on cells would cause C* to have to grab more locks for batch counter > updates, but would give us generally more predictable performance > (independent of partition wideness), and won't force people to remodel their > data model if they have often concurrently-updated counters in the same few > wide partitions. > (It's a small change, code-wise) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6553) Benchmark counter improvements (counters++)
[ https://issues.apache.org/jira/browse/CASSANDRA-6553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951096#comment-13951096 ] Aleksey Yeschenko commented on CASSANDRA-6553: -- [~rhatch] I want to revisit the stress params, later, but for now, could we compare these 3 branches against each other? 1. Stock 2.0 (https://github.com/apache/cassandra/tree/cassandra-2.0) 2. Stock 2.1 + Benedict's patch (https://github.com/iamaleksey/cassandra/tree/6553) 3. 2.1 with CASSANDRA-6880 and Benedict's patch (https://github.com/iamaleksey/cassandra/tree/6880) Two runs - one w/out the counter cache, and another one with counter cache on. In the meantime I'll think about the new stress params I'd like to run the benchmarks with. Thanks! > Benchmark counter improvements (counters++) > --- > > Key: CASSANDRA-6553 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6553 > Project: Cassandra > Issue Type: Test >Reporter: Ryan McGuire >Assignee: Russ Hatch > Fix For: 2.1 beta2 > > Attachments: 6553.txt, 6553.uber.quorum.bdplab.read.png, > 6553.uber.quorum.bdplab.write.png, high_cl_one.png, high_cl_quorum.png, > low_cl_one.png, low_cl_quorum.png, tracing.txt, uber_cl_one.png, > uber_cl_quorum.png > > > Benchmark the difference in performance between CASSANDRA-6504 and trunk. > * Updating totally unrelated counters (different partitions) > * Updating the same counters a lot (same cells in the same partition) > * Different cells in the same few partitions (hot counter partition) > benchmark: > https://github.com/apache/cassandra/tree/1218bcacba7edefaf56cf8440d0aea5794c89a1e > (old counters) > compared to: > https://github.com/apache/cassandra/tree/714c423360c36da2a2b365efaf9c5c4f623ed133 > (new counters) > So far, the above changes should only affect the write path. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (CASSANDRA-6943) very contentious writes result in performance degradation
[ https://issues.apache.org/jira/browse/CASSANDRA-6943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict updated CASSANDRA-6943: Attachment: 6943.2.txt Attaching patch with fix and improved tests (both long test now triggers this, and a specific unit test) > very contentious writes result in performance degradation > - > > Key: CASSANDRA-6943 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6943 > Project: Cassandra > Issue Type: Bug >Reporter: Russ Hatch > Attachments: 6943.2.txt, 6943.txt, node4.log, node4_jstack.log, > node5.log, node5_jstack.log, node6.log, node6_jstack.log, node7.log, > node7_jstack.log, screenshot.png, screenshot2.png, stress_jstack.log > > > Running performance scenarios I have seen this characteristic drop in > performance happen several times. A similar effect was reproduced in another > test cluster. Operations eventually come to a standstill. > !screenshot.png! > !screenshot2.png! -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6948) Native & ReplicateOnWriteStage AssertionError: Callback already exists
[ https://issues.apache.org/jira/browse/CASSANDRA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951022#comment-13951022 ] Benedict commented on CASSANDRA-6948: - bq. So perhaps its related to just the total number of write operations? The exceptions can only plausibly manifest after 4B write operations, however the _cause_ of the exceptions will be occurring earlier, and probably resulting in increased heap utilisation and a steady burdening of the server. bq. Worked fine for me after I gunzip. If there still doesn't work, I can try extracting the relevant stacktraces. Weird. It's possible gunzip isn't behaving, but I consistently get a 300MB nonsense file (again, I'm assuming it's meant to be a text file and not a heap dump - but jhat didn't like it either). If you _could_ extract the relevant stack traces that would be great. Also, those earlier log files from one of these servers would be great. It would be very helpful to get a full history from startup to first exception. > Native & ReplicateOnWriteStage AssertionError: Callback already exists > -- > > Key: CASSANDRA-6948 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6948 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Keith Wright >Assignee: Benedict > Attachments: 6948.debug.txt, Screen Shot 2014-03-28 at 11.27.56 > AM.png, Screen Shot 2014-03-28 at 11.29.24 AM.png, cassandra.log.gz, > cassandra.yaml, logs.old.tar.gz, logs.tar.gz, system.log.1.gz, system.log.gz > > > We are using the datastax driver and have seen the following exceptions > appear periodically in our cluster in "groups". The client will gradually > start sending less write requests to the nodes where this occurs the most > often. This may be related to CASSANDRA-6476. Exception from logs: > ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638 > ErrorMessage.java (line 222) Unexpected exception during request > java.lang.AssertionError: Callback already exists for id -1665979622! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984) > at > org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449) > at > org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524) > at > org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521) > at > org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505) > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358) > at > org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131) > at > org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) > at > org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > at > org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [ReplicateOnWriteStage:102766] 2014-03-28 13:32:10,638 > CassandraDaemon.java (line 196) Exception in thread > Thread[ReplicateOnWriteStage:102766,5,main] > java.lang.AssertionError: Callback already exists for id -1665979620! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@3bdb1a75, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:806) > at > org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1074) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(Thread
[jira] [Commented] (CASSANDRA-6948) Native & ReplicateOnWriteStage AssertionError: Callback already exists
[ https://issues.apache.org/jira/browse/CASSANDRA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951080#comment-13951080 ] Keith Wright commented on CASSANDRA-6948: - Yes I believe it is true that this is the first time the nodes have ever been restarted. > Native & ReplicateOnWriteStage AssertionError: Callback already exists > -- > > Key: CASSANDRA-6948 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6948 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Keith Wright >Assignee: Benedict > Attachments: 6948.debug.txt, Screen Shot 2014-03-28 at 11.27.56 > AM.png, Screen Shot 2014-03-28 at 11.29.24 AM.png, cassandra.log.min, > cassandra.yaml, logs.old.tar.gz, logs.tar.gz, system.log.1.gz, system.log.gz > > > We are using the datastax driver and have seen the following exceptions > appear periodically in our cluster in "groups". The client will gradually > start sending less write requests to the nodes where this occurs the most > often. This may be related to CASSANDRA-6476. Exception from logs: > ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638 > ErrorMessage.java (line 222) Unexpected exception during request > java.lang.AssertionError: Callback already exists for id -1665979622! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984) > at > org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449) > at > org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524) > at > org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521) > at > org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505) > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358) > at > org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131) > at > org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) > at > org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > at > org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [ReplicateOnWriteStage:102766] 2014-03-28 13:32:10,638 > CassandraDaemon.java (line 196) Exception in thread > Thread[ReplicateOnWriteStage:102766,5,main] > java.lang.AssertionError: Callback already exists for id -1665979620! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@3bdb1a75, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:806) > at > org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1074) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6948) Native & ReplicateOnWriteStage AssertionError: Callback already exists
[ https://issues.apache.org/jira/browse/CASSANDRA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951012#comment-13951012 ] Keith Wright commented on CASSANDRA-6948: - Worked fine for me after I gunzip. If there still doesn't work, I can try extracting the relevant stacktraces. One other point, we bootstrapped a new node into the cluster ~5 days ago and so far that node has never shown this issue. In addition, it appears that restarting the nodes causes the issue to stop. So perhaps its related to just the total number of write operations? > Native & ReplicateOnWriteStage AssertionError: Callback already exists > -- > > Key: CASSANDRA-6948 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6948 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Keith Wright >Assignee: Benedict > Attachments: 6948.debug.txt, Screen Shot 2014-03-28 at 11.27.56 > AM.png, Screen Shot 2014-03-28 at 11.29.24 AM.png, cassandra.log.gz, > cassandra.yaml, logs.old.tar.gz, logs.tar.gz, system.log.1.gz, system.log.gz > > > We are using the datastax driver and have seen the following exceptions > appear periodically in our cluster in "groups". The client will gradually > start sending less write requests to the nodes where this occurs the most > often. This may be related to CASSANDRA-6476. Exception from logs: > ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638 > ErrorMessage.java (line 222) Unexpected exception during request > java.lang.AssertionError: Callback already exists for id -1665979622! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984) > at > org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449) > at > org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524) > at > org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521) > at > org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505) > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358) > at > org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131) > at > org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) > at > org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > at > org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [ReplicateOnWriteStage:102766] 2014-03-28 13:32:10,638 > CassandraDaemon.java (line 196) Exception in thread > Thread[ReplicateOnWriteStage:102766,5,main] > java.lang.AssertionError: Callback already exists for id -1665979620! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@3bdb1a75, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:806) > at > org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1074) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6948) Native & ReplicateOnWriteStage AssertionError: Callback already exists
[ https://issues.apache.org/jira/browse/CASSANDRA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13950897#comment-13950897 ] Benedict commented on CASSANDRA-6948: - Thanks. Regrettably those logs are all still well after the problems began occurring (they appear to all be from today!), which has caused the logs to get completely spammed. Do you have any earlier logs, or have they been trashed by log recycling? > Native & ReplicateOnWriteStage AssertionError: Callback already exists > -- > > Key: CASSANDRA-6948 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6948 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Keith Wright >Assignee: Benedict > Attachments: Screen Shot 2014-03-28 at 11.27.56 AM.png, Screen Shot > 2014-03-28 at 11.29.24 AM.png, cassandra.yaml, logs.tar.gz, system.log.1.gz, > system.log.gz > > > We are using the datastax driver and have seen the following exceptions > appear periodically in our cluster in "groups". The client will gradually > start sending less write requests to the nodes where this occurs the most > often. This may be related to CASSANDRA-6476. Exception from logs: > ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638 > ErrorMessage.java (line 222) Unexpected exception during request > java.lang.AssertionError: Callback already exists for id -1665979622! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984) > at > org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449) > at > org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524) > at > org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521) > at > org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505) > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358) > at > org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131) > at > org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) > at > org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > at > org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [ReplicateOnWriteStage:102766] 2014-03-28 13:32:10,638 > CassandraDaemon.java (line 196) Exception in thread > Thread[ReplicateOnWriteStage:102766,5,main] > java.lang.AssertionError: Callback already exists for id -1665979620! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@3bdb1a75, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:806) > at > org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1074) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (CASSANDRA-6948) Native & ReplicateOnWriteStage AssertionError: Callback already exists
[ https://issues.apache.org/jira/browse/CASSANDRA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keith Wright updated CASSANDRA-6948: Attachment: (was: cassandra.log.gz) > Native & ReplicateOnWriteStage AssertionError: Callback already exists > -- > > Key: CASSANDRA-6948 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6948 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Keith Wright >Assignee: Benedict > Attachments: 6948.debug.txt, Screen Shot 2014-03-28 at 11.27.56 > AM.png, Screen Shot 2014-03-28 at 11.29.24 AM.png, cassandra.log.min, > cassandra.yaml, logs.old.tar.gz, logs.tar.gz, system.log.1.gz, system.log.gz > > > We are using the datastax driver and have seen the following exceptions > appear periodically in our cluster in "groups". The client will gradually > start sending less write requests to the nodes where this occurs the most > often. This may be related to CASSANDRA-6476. Exception from logs: > ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638 > ErrorMessage.java (line 222) Unexpected exception during request > java.lang.AssertionError: Callback already exists for id -1665979622! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984) > at > org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449) > at > org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524) > at > org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521) > at > org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505) > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358) > at > org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131) > at > org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) > at > org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > at > org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [ReplicateOnWriteStage:102766] 2014-03-28 13:32:10,638 > CassandraDaemon.java (line 196) Exception in thread > Thread[ReplicateOnWriteStage:102766,5,main] > java.lang.AssertionError: Callback already exists for id -1665979620! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@3bdb1a75, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:806) > at > org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1074) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (CASSANDRA-6948) Native & ReplicateOnWriteStage AssertionError: Callback already exists
[ https://issues.apache.org/jira/browse/CASSANDRA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keith Wright updated CASSANDRA-6948: Attachment: cassandra.log.min This is a non-gzip version of cassandra.log from node where issue was occurring and we did a thread dump. > Native & ReplicateOnWriteStage AssertionError: Callback already exists > -- > > Key: CASSANDRA-6948 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6948 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Keith Wright >Assignee: Benedict > Attachments: 6948.debug.txt, Screen Shot 2014-03-28 at 11.27.56 > AM.png, Screen Shot 2014-03-28 at 11.29.24 AM.png, cassandra.log.min, > cassandra.yaml, logs.old.tar.gz, logs.tar.gz, system.log.1.gz, system.log.gz > > > We are using the datastax driver and have seen the following exceptions > appear periodically in our cluster in "groups". The client will gradually > start sending less write requests to the nodes where this occurs the most > often. This may be related to CASSANDRA-6476. Exception from logs: > ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638 > ErrorMessage.java (line 222) Unexpected exception during request > java.lang.AssertionError: Callback already exists for id -1665979622! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984) > at > org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449) > at > org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524) > at > org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521) > at > org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505) > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358) > at > org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131) > at > org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) > at > org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > at > org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [ReplicateOnWriteStage:102766] 2014-03-28 13:32:10,638 > CassandraDaemon.java (line 196) Exception in thread > Thread[ReplicateOnWriteStage:102766,5,main] > java.lang.AssertionError: Callback already exists for id -1665979620! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@3bdb1a75, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:806) > at > org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1074) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (CASSANDRA-6948) Native & ReplicateOnWriteStage AssertionError: Callback already exists
[ https://issues.apache.org/jira/browse/CASSANDRA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keith Wright updated CASSANDRA-6948: Attachment: cassandra.log.gz OK here it is again gzip compressed. > Native & ReplicateOnWriteStage AssertionError: Callback already exists > -- > > Key: CASSANDRA-6948 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6948 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Keith Wright >Assignee: Benedict > Attachments: 6948.debug.txt, Screen Shot 2014-03-28 at 11.27.56 > AM.png, Screen Shot 2014-03-28 at 11.29.24 AM.png, cassandra.log.gz, > cassandra.yaml, logs.old.tar.gz, logs.tar.gz, system.log.1.gz, system.log.gz > > > We are using the datastax driver and have seen the following exceptions > appear periodically in our cluster in "groups". The client will gradually > start sending less write requests to the nodes where this occurs the most > often. This may be related to CASSANDRA-6476. Exception from logs: > ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638 > ErrorMessage.java (line 222) Unexpected exception during request > java.lang.AssertionError: Callback already exists for id -1665979622! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984) > at > org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449) > at > org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524) > at > org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521) > at > org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505) > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358) > at > org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131) > at > org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) > at > org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > at > org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [ReplicateOnWriteStage:102766] 2014-03-28 13:32:10,638 > CassandraDaemon.java (line 196) Exception in thread > Thread[ReplicateOnWriteStage:102766,5,main] > java.lang.AssertionError: Callback already exists for id -1665979620! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@3bdb1a75, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:806) > at > org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1074) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (CASSANDRA-6948) Native & ReplicateOnWriteStage AssertionError: Callback already exists
[ https://issues.apache.org/jira/browse/CASSANDRA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keith Wright updated CASSANDRA-6948: Attachment: (was: cassandra.log.gz) > Native & ReplicateOnWriteStage AssertionError: Callback already exists > -- > > Key: CASSANDRA-6948 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6948 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Keith Wright >Assignee: Benedict > Attachments: 6948.debug.txt, Screen Shot 2014-03-28 at 11.27.56 > AM.png, Screen Shot 2014-03-28 at 11.29.24 AM.png, cassandra.log.gz, > cassandra.yaml, logs.old.tar.gz, logs.tar.gz, system.log.1.gz, system.log.gz > > > We are using the datastax driver and have seen the following exceptions > appear periodically in our cluster in "groups". The client will gradually > start sending less write requests to the nodes where this occurs the most > often. This may be related to CASSANDRA-6476. Exception from logs: > ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638 > ErrorMessage.java (line 222) Unexpected exception during request > java.lang.AssertionError: Callback already exists for id -1665979622! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984) > at > org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449) > at > org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524) > at > org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521) > at > org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505) > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358) > at > org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131) > at > org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) > at > org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > at > org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [ReplicateOnWriteStage:102766] 2014-03-28 13:32:10,638 > CassandraDaemon.java (line 196) Exception in thread > Thread[ReplicateOnWriteStage:102766,5,main] > java.lang.AssertionError: Callback already exists for id -1665979620! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@3bdb1a75, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:806) > at > org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1074) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (CASSANDRA-6892) Cassandra 2.0.x validates Thrift columns incorrectly and causes InvalidRequestException
[
https://issues.apache.org/jira/browse/CASSANDRA-6892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tyler Hobbs updated CASSANDRA-6892:
---
Attachment: 6892-2.0.txt
Attached patch (and
[branch|https://github.com/thobbs/cassandra/tree/CASSANDRA-6892]) ensures that
key aliases and column aliases aren't used improperly for validation.
Primarily this affects inserts through Thrift. If data was inserted pre-2.0
with a key alias for the column name, sstable2json/json2sstable and scrub could
also be affected. (I say pre-2.0 because post-2.0, that data would have
(erroneously) failed validation on insert.)
I didn't look simple (or necessarily correct) to remove the aliases from
CFMetaData.column_metadata, so I added methods to get the ColumnDefinition or
validator for columns _outside_ of cql3. I'm not perfectly happy with this; if
anybody can think of a better way to enforce the correct behavior, I would
appreciate it.
> Cassandra 2.0.x validates Thrift columns incorrectly and causes
> InvalidRequestException
> ---
>
> Key: CASSANDRA-6892
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6892
> Project: Cassandra
> Issue Type: Bug
> Components: API
>Reporter: Christian Spriegel
>Assignee: Tyler Hobbs
>Priority: Minor
> Fix For: 2.0.7
>
> Attachments: 6892-2.0.txt, CASSANDRA-6892_V1.patch
>
>
> I just upgrade my local dev machine to Cassandra 2.0, which causes one of my
> automated tests to fail now. With the latest 1.2.x it was working fine.
> The Exception I get on my client (using Hector) is:
> {code}
> me.prettyprint.hector.api.exceptions.HInvalidRequestException:
> InvalidRequestException(why:(Expected 8 or 0 byte long (21))
> [MDS_0][MasterdataIndex][key2] failed validation)
> at
> me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:52)
> at
> me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:265)
> at
> me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:113)
> at
> me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)
> at
> me.prettyprint.cassandra.service.template.AbstractColumnFamilyTemplate.executeBatch(AbstractColumnFamilyTemplate.java:115)
> at
> me.prettyprint.cassandra.service.template.AbstractColumnFamilyTemplate.executeIfNotBatched(AbstractColumnFamilyTemplate.java:163)
> at
> me.prettyprint.cassandra.service.template.ColumnFamilyTemplate.update(ColumnFamilyTemplate.java:69)
> at
> com.mycompany.spring3utils.dataaccess.cassandra.AbstractCassandraDAO.doUpdate(AbstractCassandraDAO.java:482)
>
> Caused by: InvalidRequestException(why:(Expected 8 or 0 byte long (21))
> [MDS_0][MasterdataIndex][key2] failed validation)
> at
> org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:20833)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
> at
> org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:964)
> at
> org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:950)
> at
> me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:246)
> at
> me.prettyprint.cassandra.model.MutatorImpl$3.execute(MutatorImpl.java:1)
> at
> me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:104)
> at
> me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:258)
> ... 46 more
> {code}
> The schema of my column family is:
> {code}
> create column family MasterdataIndex with
> compression_options = {sstable_compression:SnappyCompressor,
> chunk_length_kb:64} and
> comparator = UTF8Type and
> key_validation_class = 'CompositeType(UTF8Type,LongType)' and
> default_validation_class = BytesType;
> {code}
> From the error message it looks like Cassandra is trying to validate the
> value with the key-validator! (My value in this case it 21 bytes long)
> I studied the Cassandra 2.0 code and found something wrong. It seems in
> CFMetaData.addDefaultKeyAliases it passes the KeyValidator into
> ColumnDefinition.partitionKeyDef. Inside ColumnDefinition the validator is
> expected to be the value validator!
> In CFMetaData:
> {code}
> private List
> addDefaultKeyAliases(List pkCols)
> {
> for (int i = 0; i < pkCols.size(); i++)
> {
> if (pkCols.get(i) == null)
> {
> Integer idx = null;
> AbstractType type = keyValidator;
> if (keyValidator instanceof CompositeType)
> {
[jira] [Commented] (CASSANDRA-6948) Native & ReplicateOnWriteStage AssertionError: Callback already exists
[ https://issues.apache.org/jira/browse/CASSANDRA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13950870#comment-13950870 ] Benedict commented on CASSANDRA-6948: - Thanks. Could you give some indication of the amount if time this box had been up for before exhibiting this, and the number of requests it would have processed (just ballpark order of magnitude)? > Native & ReplicateOnWriteStage AssertionError: Callback already exists > -- > > Key: CASSANDRA-6948 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6948 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Keith Wright >Assignee: Benedict > Attachments: cassandra.yaml, system.log.1.gz, system.log.gz > > > We are using the datastax driver and have seen the following exceptions > appear periodically in our cluster in "groups". The client will gradually > start sending less write requests to the nodes where this occurs the most > often. This may be related to CASSANDRA-6476. Exception from logs: > ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638 > ErrorMessage.java (line 222) Unexpected exception during request > java.lang.AssertionError: Callback already exists for id -1665979622! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984) > at > org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449) > at > org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524) > at > org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521) > at > org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505) > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358) > at > org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131) > at > org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) > at > org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > at > org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [ReplicateOnWriteStage:102766] 2014-03-28 13:32:10,638 > CassandraDaemon.java (line 196) Exception in thread > Thread[ReplicateOnWriteStage:102766,5,main] > java.lang.AssertionError: Callback already exists for id -1665979620! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@3bdb1a75, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:806) > at > org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1074) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6948) Native & ReplicateOnWriteStage AssertionError: Callback already exists
[ https://issues.apache.org/jira/browse/CASSANDRA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951000#comment-13951000 ] Benedict commented on CASSANDRA-6948: - The file is still a 300MB binary file, not a textual log file. Could you confirm the source file you're uploading is what you think it is? > Native & ReplicateOnWriteStage AssertionError: Callback already exists > -- > > Key: CASSANDRA-6948 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6948 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Keith Wright >Assignee: Benedict > Attachments: 6948.debug.txt, Screen Shot 2014-03-28 at 11.27.56 > AM.png, Screen Shot 2014-03-28 at 11.29.24 AM.png, cassandra.log.gz, > cassandra.yaml, logs.old.tar.gz, logs.tar.gz, system.log.1.gz, system.log.gz > > > We are using the datastax driver and have seen the following exceptions > appear periodically in our cluster in "groups". The client will gradually > start sending less write requests to the nodes where this occurs the most > often. This may be related to CASSANDRA-6476. Exception from logs: > ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638 > ErrorMessage.java (line 222) Unexpected exception during request > java.lang.AssertionError: Callback already exists for id -1665979622! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984) > at > org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449) > at > org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524) > at > org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521) > at > org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505) > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358) > at > org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131) > at > org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) > at > org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > at > org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [ReplicateOnWriteStage:102766] 2014-03-28 13:32:10,638 > CassandraDaemon.java (line 196) Exception in thread > Thread[ReplicateOnWriteStage:102766,5,main] > java.lang.AssertionError: Callback already exists for id -1665979620! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@3bdb1a75, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:806) > at > org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1074) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6948) Native & ReplicateOnWriteStage AssertionError: Callback already exists
[ https://issues.apache.org/jira/browse/CASSANDRA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13950983#comment-13950983 ] Benedict commented on CASSANDRA-6948: - bq. Cassandra.log contains logging output It seems to be a binary file. Could you confirm the contents, and repackage and reupload? The old log files, btw, also still don't go back to node restart: the first log file has them with lower frequency, but still exhibits these exceptions. > Native & ReplicateOnWriteStage AssertionError: Callback already exists > -- > > Key: CASSANDRA-6948 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6948 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Keith Wright >Assignee: Benedict > Attachments: 6948.debug.txt, Screen Shot 2014-03-28 at 11.27.56 > AM.png, Screen Shot 2014-03-28 at 11.29.24 AM.png, cassandra.log.gz, > cassandra.yaml, logs.old.tar.gz, logs.tar.gz, system.log.1.gz, system.log.gz > > > We are using the datastax driver and have seen the following exceptions > appear periodically in our cluster in "groups". The client will gradually > start sending less write requests to the nodes where this occurs the most > often. This may be related to CASSANDRA-6476. Exception from logs: > ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638 > ErrorMessage.java (line 222) Unexpected exception during request > java.lang.AssertionError: Callback already exists for id -1665979622! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984) > at > org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449) > at > org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524) > at > org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521) > at > org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505) > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358) > at > org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131) > at > org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) > at > org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > at > org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [ReplicateOnWriteStage:102766] 2014-03-28 13:32:10,638 > CassandraDaemon.java (line 196) Exception in thread > Thread[ReplicateOnWriteStage:102766,5,main] > java.lang.AssertionError: Callback already exists for id -1665979620! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@3bdb1a75, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:806) > at > org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1074) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (CASSANDRA-6948) Native & ReplicateOnWriteStage AssertionError: Callback already exists
[ https://issues.apache.org/jira/browse/CASSANDRA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict updated CASSANDRA-6948: Attachment: 6948.debug.txt Attaching a patch that should increase the amount of debugging information available to us, and also ensures the ExpiringMap shutdown code works (on the off-chance this is the cause) > Native & ReplicateOnWriteStage AssertionError: Callback already exists > -- > > Key: CASSANDRA-6948 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6948 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Keith Wright >Assignee: Benedict > Attachments: 6948.debug.txt, Screen Shot 2014-03-28 at 11.27.56 > AM.png, Screen Shot 2014-03-28 at 11.29.24 AM.png, cassandra.log.gz, > cassandra.yaml, logs.old.tar.gz, logs.tar.gz, system.log.1.gz, system.log.gz > > > We are using the datastax driver and have seen the following exceptions > appear periodically in our cluster in "groups". The client will gradually > start sending less write requests to the nodes where this occurs the most > often. This may be related to CASSANDRA-6476. Exception from logs: > ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638 > ErrorMessage.java (line 222) Unexpected exception during request > java.lang.AssertionError: Callback already exists for id -1665979622! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984) > at > org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449) > at > org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524) > at > org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521) > at > org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505) > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358) > at > org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131) > at > org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) > at > org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > at > org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [ReplicateOnWriteStage:102766] 2014-03-28 13:32:10,638 > CassandraDaemon.java (line 196) Exception in thread > Thread[ReplicateOnWriteStage:102766,5,main] > java.lang.AssertionError: Callback already exists for id -1665979620! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@3bdb1a75, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:806) > at > org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1074) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6948) Native & ReplicateOnWriteStage AssertionError: Callback already exists
[ https://issues.apache.org/jira/browse/CASSANDRA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13950954#comment-13950954 ] Benedict commented on CASSANDRA-6948: - What does the cassandra.log contain? Seems to be a binary file... bq. As part of this migration, our client code was instantiating two instances of cluster from the datastax client to allow for writing to go to both clusters This shouldn't have been an issue. Were the clusters aware of each other (i.e. multiple datacenters in same cluster) or were they completely distinct clusters? > Native & ReplicateOnWriteStage AssertionError: Callback already exists > -- > > Key: CASSANDRA-6948 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6948 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Keith Wright >Assignee: Benedict > Attachments: 6948.debug.txt, Screen Shot 2014-03-28 at 11.27.56 > AM.png, Screen Shot 2014-03-28 at 11.29.24 AM.png, cassandra.log.gz, > cassandra.yaml, logs.old.tar.gz, logs.tar.gz, system.log.1.gz, system.log.gz > > > We are using the datastax driver and have seen the following exceptions > appear periodically in our cluster in "groups". The client will gradually > start sending less write requests to the nodes where this occurs the most > often. This may be related to CASSANDRA-6476. Exception from logs: > ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638 > ErrorMessage.java (line 222) Unexpected exception during request > java.lang.AssertionError: Callback already exists for id -1665979622! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984) > at > org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449) > at > org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524) > at > org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521) > at > org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505) > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358) > at > org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131) > at > org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) > at > org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > at > org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [ReplicateOnWriteStage:102766] 2014-03-28 13:32:10,638 > CassandraDaemon.java (line 196) Exception in thread > Thread[ReplicateOnWriteStage:102766,5,main] > java.lang.AssertionError: Callback already exists for id -1665979620! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@3bdb1a75, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:806) > at > org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1074) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (CASSANDRA-6948) Native & ReplicateOnWriteStage AssertionError: Callback already exists
[ https://issues.apache.org/jira/browse/CASSANDRA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keith Wright updated CASSANDRA-6948: Attachment: Screen Shot 2014-03-28 at 11.29.24 AM.png Screen Shot 2014-03-28 at 11.27.56 AM.png Its been up and running for ~2 weeks. It started showing the issue again ~3 hours ago. I have attached a graph showing read requests for the last week as compared to another server. You can see that they were lower for this server until ~3 days ago when they recovered on their own (we did not restart the node but we did restart clients that day which likely caused the recovery). I have also attached read/write request volume. Note we're using RF 3 for all keyspaces. > Native & ReplicateOnWriteStage AssertionError: Callback already exists > -- > > Key: CASSANDRA-6948 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6948 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Keith Wright >Assignee: Benedict > Attachments: Screen Shot 2014-03-28 at 11.27.56 AM.png, Screen Shot > 2014-03-28 at 11.29.24 AM.png, cassandra.yaml, logs.tar.gz, system.log.1.gz, > system.log.gz > > > We are using the datastax driver and have seen the following exceptions > appear periodically in our cluster in "groups". The client will gradually > start sending less write requests to the nodes where this occurs the most > often. This may be related to CASSANDRA-6476. Exception from logs: > ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638 > ErrorMessage.java (line 222) Unexpected exception during request > java.lang.AssertionError: Callback already exists for id -1665979622! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984) > at > org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449) > at > org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524) > at > org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521) > at > org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505) > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358) > at > org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131) > at > org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) > at > org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > at > org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [ReplicateOnWriteStage:102766] 2014-03-28 13:32:10,638 > CassandraDaemon.java (line 196) Exception in thread > Thread[ReplicateOnWriteStage:102766,5,main] > java.lang.AssertionError: Callback already exists for id -1665979620! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@3bdb1a75, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:806) > at > org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1074) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6946) Catching RequestExecutionException instead of Throwable
[ https://issues.apache.org/jira/browse/CASSANDRA-6946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13950889#comment-13950889 ] Jonathan Ellis commented on CASSANDRA-6946: --- This is saying, "no matter what caused revoke to fail, log a human-readable error instead of a raw stack trace" > Catching RequestExecutionException instead of Throwable > --- > > Key: CASSANDRA-6946 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6946 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Ding Yuan >Priority: Trivial > Attachments: trunk-overthrow.txt > > > In CassandraAuthorizer.java, there are a few cases where a Throwable is used > to catch RequestExecutionException. Propose to catch the exact exception. > Attaching a patch against trunk for review. Thanks. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (CASSANDRA-6948) Native & ReplicateOnWriteStage AssertionError: Callback already exists
[ https://issues.apache.org/jira/browse/CASSANDRA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keith Wright updated CASSANDRA-6948: Attachment: cassandra.yaml Cassandra.yaml attached. We restarted the nodes where this was occurring so I don't have a good JStack at this time however I will attach it when it re-occurs. > Native & ReplicateOnWriteStage AssertionError: Callback already exists > -- > > Key: CASSANDRA-6948 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6948 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Keith Wright > Attachments: cassandra.yaml > > > We are using the datastax driver and have seen the following exceptions > appear periodically in our cluster in "groups". The client will gradually > start sending less write requests to the nodes where this occurs the most > often. This may be related to CASSANDRA-6476. Exception from logs: > ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638 > ErrorMessage.java (line 222) Unexpected exception during request > java.lang.AssertionError: Callback already exists for id -1665979622! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984) > at > org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449) > at > org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524) > at > org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521) > at > org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505) > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358) > at > org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131) > at > org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) > at > org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > at > org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [ReplicateOnWriteStage:102766] 2014-03-28 13:32:10,638 > CassandraDaemon.java (line 196) Exception in thread > Thread[ReplicateOnWriteStage:102766,5,main] > java.lang.AssertionError: Callback already exists for id -1665979620! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@3bdb1a75, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:806) > at > org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1074) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (CASSANDRA-6948) Native & ReplicateOnWriteStage AssertionError: Callback already exists
[ https://issues.apache.org/jira/browse/CASSANDRA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13950892#comment-13950892 ] Keith Wright edited comment on CASSANDRA-6948 at 3/28/14 3:29 PM: -- Its been up and running for ~2 weeks. It started showing the issue again ~3 hours ago. I have attached a graph showing write requests for the last week as compared to another server. You can see that they were lower for this server until ~3 days ago when they recovered on their own (we did not restart the node but we did restart clients that day which likely caused the recovery). I have also attached read/write request volume. Note we're using RF 3 for all keyspaces. was (Author: keithwrightbos): Its been up and running for ~2 weeks. It started showing the issue again ~3 hours ago. I have attached a graph showing read requests for the last week as compared to another server. You can see that they were lower for this server until ~3 days ago when they recovered on their own (we did not restart the node but we did restart clients that day which likely caused the recovery). I have also attached read/write request volume. Note we're using RF 3 for all keyspaces. > Native & ReplicateOnWriteStage AssertionError: Callback already exists > -- > > Key: CASSANDRA-6948 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6948 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Keith Wright >Assignee: Benedict > Attachments: Screen Shot 2014-03-28 at 11.27.56 AM.png, Screen Shot > 2014-03-28 at 11.29.24 AM.png, cassandra.yaml, logs.tar.gz, system.log.1.gz, > system.log.gz > > > We are using the datastax driver and have seen the following exceptions > appear periodically in our cluster in "groups". The client will gradually > start sending less write requests to the nodes where this occurs the most > often. This may be related to CASSANDRA-6476. Exception from logs: > ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638 > ErrorMessage.java (line 222) Unexpected exception during request > java.lang.AssertionError: Callback already exists for id -1665979622! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984) > at > org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449) > at > org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524) > at > org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521) > at > org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505) > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358) > at > org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131) > at > org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) > at > org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > at > org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [ReplicateOnWriteStage:102766] 2014-03-28 13:32:10,638 > CassandraDaemon.java (line 196) Exception in thread > Thread[ReplicateOnWriteStage:102766,5,main] > java.lang.AssertionError: Callback already exists for id -1665979620! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@3bdb1a75, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:806) > at > org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1074) > at
[jira] [Commented] (CASSANDRA-6947) SocketException being swallowed
[
https://issues.apache.org/jira/browse/CASSANDRA-6947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13950779#comment-13950779
]
Ding Yuan commented on CASSANDRA-6947:
--
I see. Thanks for the feedbacks!
> SocketException being swallowed
> ---
>
> Key: CASSANDRA-6947
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6947
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Ding Yuan
> Attachments: trunk-socketExceptions.txt
>
>
> There are a few cases where a SocketException is swallowed. For example: in
> the following code snippet in TCustomServerSocket.java, a SocketException is
> ignored even though the method declares that TTransportException can be
> thrown.
> {noformat}
> public void listen() throws TTransportException
> {
> // Make sure not to block on accept
> if (serverSocket != null)
> {
> try
> {
> serverSocket.setSoTimeout(100);
> }
> catch (SocketException sx)
> {
> logger.error("Could not set socket timeout.", sx);
> }
> }
> }
> {noformat}
> Propose to thrown TTransportException on SocketExceptions. Attaching a patch
> against trunk for review. Thanks!
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (CASSANDRA-6949) Performance regression in tombstone heavy workloads
Jeremiah Jordan created CASSANDRA-6949:
--
Summary: Performance regression in tombstone heavy workloads
Key: CASSANDRA-6949
URL: https://issues.apache.org/jira/browse/CASSANDRA-6949
Project: Cassandra
Issue Type: Bug
Reporter: Jeremiah Jordan
Assignee: Sam Tunnicliffe
CASSANDRA-5614 causes a huge performance regression in tombstone heavy
workloads. The isDeleted checks here cause a huge CPU overhead:
https://github.com/apache/cassandra/blob/cassandra-2.0/src/java/org/apache/cassandra/db/AtomicSortedColumns.java#L189-L196
An insert workload which does perfectly fine on 1.2, pegs CPU use at 100% on
2.0, with all of the mutation threads sitting in that loop. For example:
{noformat}
"MutationStage:20" daemon prio=10 tid=0x7fb1c4c72800 nid=0x2249 runnable
[0x7fb1b033]
java.lang.Thread.State: RUNNABLE
at org.apache.cassandra.db.marshal.BytesType.bytesCompare(BytesType.java:45)
at org.apache.cassandra.db.marshal.UTF8Type.compare(UTF8Type.java:34)
at org.apache.cassandra.db.marshal.UTF8Type.compare(UTF8Type.java:26)
at
org.apache.cassandra.db.marshal.AbstractType.compareCollectionMembers(AbstractType.java:267)
at
org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:85)
at
org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:35)
at
org.apache.cassandra.db.RangeTombstoneList.searchInternal(RangeTombstoneList.java:253)
at
org.apache.cassandra.db.RangeTombstoneList.isDeleted(RangeTombstoneList.java:210)
at org.apache.cassandra.db.DeletionInfo.isDeleted(DeletionInfo.java:136)
at org.apache.cassandra.db.DeletionInfo.isDeleted(DeletionInfo.java:123)
at
org.apache.cassandra.db.AtomicSortedColumns.addAllWithSizeDelta(AtomicSortedColumns.java:193)
at org.apache.cassandra.db.Memtable.resolve(Memtable.java:194)
at org.apache.cassandra.db.Memtable.put(Memtable.java:158)
at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:890)
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:368)
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:333)
at org.apache.cassandra.db.RowMutation.apply(RowMutation.java:201)
at
org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:56)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:60)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (CASSANDRA-6948) Native & ReplicateOnWriteStage AssertionError: Callback already exists
[ https://issues.apache.org/jira/browse/CASSANDRA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keith Wright updated CASSANDRA-6948: Attachment: logs.tar.gz All recent logs attached > Native & ReplicateOnWriteStage AssertionError: Callback already exists > -- > > Key: CASSANDRA-6948 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6948 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Keith Wright >Assignee: Benedict > Attachments: cassandra.yaml, logs.tar.gz, system.log.1.gz, > system.log.gz > > > We are using the datastax driver and have seen the following exceptions > appear periodically in our cluster in "groups". The client will gradually > start sending less write requests to the nodes where this occurs the most > often. This may be related to CASSANDRA-6476. Exception from logs: > ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638 > ErrorMessage.java (line 222) Unexpected exception during request > java.lang.AssertionError: Callback already exists for id -1665979622! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984) > at > org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449) > at > org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524) > at > org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521) > at > org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505) > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358) > at > org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131) > at > org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) > at > org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > at > org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [ReplicateOnWriteStage:102766] 2014-03-28 13:32:10,638 > CassandraDaemon.java (line 196) Exception in thread > Thread[ReplicateOnWriteStage:102766,5,main] > java.lang.AssertionError: Callback already exists for id -1665979620! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@3bdb1a75, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:806) > at > org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1074) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6948) Native & ReplicateOnWriteStage AssertionError: Callback already exists
[ https://issues.apache.org/jira/browse/CASSANDRA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13950807#comment-13950807 ] Benedict commented on CASSANDRA-6948: - These don't seem to go back to node startup? They both start with the exception, I would like to see log info from before this happened, as this is almost certainly a delayed manifestation of an earlier (potentially *much* earlier) problem > Native & ReplicateOnWriteStage AssertionError: Callback already exists > -- > > Key: CASSANDRA-6948 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6948 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Keith Wright > Attachments: cassandra.yaml, system.log.1.gz, system.log.gz > > > We are using the datastax driver and have seen the following exceptions > appear periodically in our cluster in "groups". The client will gradually > start sending less write requests to the nodes where this occurs the most > often. This may be related to CASSANDRA-6476. Exception from logs: > ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638 > ErrorMessage.java (line 222) Unexpected exception during request > java.lang.AssertionError: Callback already exists for id -1665979622! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984) > at > org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449) > at > org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524) > at > org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521) > at > org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505) > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358) > at > org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131) > at > org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) > at > org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > at > org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [ReplicateOnWriteStage:102766] 2014-03-28 13:32:10,638 > CassandraDaemon.java (line 196) Exception in thread > Thread[ReplicateOnWriteStage:102766,5,main] > java.lang.AssertionError: Callback already exists for id -1665979620! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@3bdb1a75, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:806) > at > org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1074) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6948) Native & ReplicateOnWriteStage AssertionError: Callback already exists
[ https://issues.apache.org/jira/browse/CASSANDRA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13950854#comment-13950854 ] Benedict commented on CASSANDRA-6948: - Also, could you confirm you haven't modified any of the RPC timeouts at any point using JMX? > Native & ReplicateOnWriteStage AssertionError: Callback already exists > -- > > Key: CASSANDRA-6948 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6948 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Keith Wright >Assignee: Benedict > Attachments: cassandra.yaml, system.log.1.gz, system.log.gz > > > We are using the datastax driver and have seen the following exceptions > appear periodically in our cluster in "groups". The client will gradually > start sending less write requests to the nodes where this occurs the most > often. This may be related to CASSANDRA-6476. Exception from logs: > ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638 > ErrorMessage.java (line 222) Unexpected exception during request > java.lang.AssertionError: Callback already exists for id -1665979622! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984) > at > org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449) > at > org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524) > at > org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521) > at > org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505) > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358) > at > org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131) > at > org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) > at > org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > at > org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [ReplicateOnWriteStage:102766] 2014-03-28 13:32:10,638 > CassandraDaemon.java (line 196) Exception in thread > Thread[ReplicateOnWriteStage:102766,5,main] > java.lang.AssertionError: Callback already exists for id -1665979620! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@3bdb1a75, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:806) > at > org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1074) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6948) Native & ReplicateOnWriteStage AssertionError: Callback already exists
[ https://issues.apache.org/jira/browse/CASSANDRA-6948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13950859#comment-13950859 ] Keith Wright commented on CASSANDRA-6948: - Confirmed we are running OOTB with no JMX changes whatsoever > Native & ReplicateOnWriteStage AssertionError: Callback already exists > -- > > Key: CASSANDRA-6948 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6948 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Keith Wright >Assignee: Benedict > Attachments: cassandra.yaml, system.log.1.gz, system.log.gz > > > We are using the datastax driver and have seen the following exceptions > appear periodically in our cluster in "groups". The client will gradually > start sending less write requests to the nodes where this occurs the most > often. This may be related to CASSANDRA-6476. Exception from logs: > ERROR [Native-Transport-Requests:7636] 2014-03-28 13:32:10,638 > ErrorMessage.java (line 222) Unexpected exception during request > java.lang.AssertionError: Callback already exists for id -1665979622! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@5d36d8ea, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.mutateCounter(StorageProxy.java:984) > at > org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:449) > at > org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:524) > at > org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:521) > at > org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:505) > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) > at > org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:358) > at > org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:131) > at > org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) > at > org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > at > org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > ERROR [ReplicateOnWriteStage:102766] 2014-03-28 13:32:10,638 > CassandraDaemon.java (line 196) Exception in thread > Thread[ReplicateOnWriteStage:102766,5,main] > java.lang.AssertionError: Callback already exists for id -1665979620! > (CallbackInfo(target=/10.106.160.84, > callback=org.apache.cassandra.service.WriteResponseHandler@3bdb1a75, > serializer=org.apache.cassandra.db.WriteResponse$WriteResponseSerializer@6ed37f0b)) > at > org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:549) > at > org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:601) > at > org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:806) > at > org.apache.cassandra.service.StorageProxy$8$1.runMayThrow(StorageProxy.java:1074) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1896) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (CASSANDRA-6947) SocketException being swallowed
[
https://issues.apache.org/jira/browse/CASSANDRA-6947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Ellis resolved CASSANDRA-6947.
---
Resolution: Not A Problem
these are places where we have decided that the exception should not be fatal
> SocketException being swallowed
> ---
>
> Key: CASSANDRA-6947
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6947
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Ding Yuan
> Attachments: trunk-socketExceptions.txt
>
>
> There are a few cases where a SocketException is swallowed. For example: in
> the following code snippet in TCustomServerSocket.java, a SocketException is
> ignored even though the method declares that TTransportException can be
> thrown.
> {noformat}
> public void listen() throws TTransportException
> {
> // Make sure not to block on accept
> if (serverSocket != null)
> {
> try
> {
> serverSocket.setSoTimeout(100);
> }
> catch (SocketException sx)
> {
> logger.error("Could not set socket timeout.", sx);
> }
> }
> }
> {noformat}
> Propose to thrown TTransportException on SocketExceptions. Attaching a patch
> against trunk for review. Thanks!
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-6666) Avoid accumulating tombstones after partial hint replay
[ https://issues.apache.org/jira/browse/CASSANDRA-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13950812#comment-13950812 ] Anne Sullivan commented on CASSANDRA-: -- Files uploaded to scp server - thanks. > Avoid accumulating tombstones after partial hint replay > --- > > Key: CASSANDRA- > URL: https://issues.apache.org/jira/browse/CASSANDRA- > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Jonathan Ellis >Assignee: Jonathan Ellis >Priority: Minor > Labels: hintedhandoff > Fix For: 1.2.16, 2.0.6 > > Attachments: .txt, cassandra_system.log.debug.gz > > -- This message was sent by Atlassian JIRA (v6.2#6252)
