[jira] [Commented] (CASSANDRA-3014) AnitEntropy/MerkleTree Error

2013-02-12 Thread JIRA 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-3014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13576489#comment-13576489
 ] 

David Röhr commented on CASSANDRA-3014:
---

Another repro in 1.2.1.
6 nodes cluster (vnodes, murmur3, rf:3)
very low activity
running a nodetool repair -pr loop on the cluster nodes

nodetool hangs, and same big stacktrace in logs. 

root 11025  0.0  0.0 106100  1436 pts/0S+   Feb11   0:00  \_ 
/bin/sh /usr/bin/nodetool -h HOST -p 7199 -pr repair KEYSPACE COLUMN_FAMILY

 AnitEntropy/MerkleTree Error
 

 Key: CASSANDRA-3014
 URL: https://issues.apache.org/jira/browse/CASSANDRA-3014
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 0.8.0
Reporter: Hefeng Yuan
Priority: Minor

 Hi,
 We are seeing some AntiEntropy error in our production servers, since it's 
 hard to repro in other env, pasting the stack track here for some clue. 
 We're using a cluster of 2 data centers, 9 nodes, 6 for online traffic, 3 for 
 Brisk BI. Our RF is Cassandra:5, Brisk:1.
 Our Cassandra server version is 0.8.0. Data is written using Hector 0.7.0-20 
 and cassandra 0.7.2 library.
 Partitioner is random.
 Our nodetool repair is scheduled once per week.
 The exception stack is appended in the end.
 Any help is appreciated.
 Thanks,
 Hefeng
 ERROR [AntiEntropyStage:2] 2011-08-08 04:24:39,556 
 AbstractCassandraDaemon.java (line 113) Fatal exception in thread 
 Thread[AntiEntropyStage:2,5,main]
 java.lang.AssertionError
   at org.apache.cassandra.utils.MerkleTree.inc(MerkleTree.java:154)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:262)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at

[jira] [Commented] (CASSANDRA-3014) AnitEntropy/MerkleTree Error

2013-01-17 Thread Sylvain Pognant (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-3014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13556245#comment-13556245
 ] 

Sylvain Pognant commented on CASSANDRA-3014:


Reproduced on 1.2.0
6 nodes cluster (vnodes, murmur3, rf:2)
no activity
running nodetool repair on all nodes

some of the nodetool repair were waiting forever - looking into the logs - 
same big stack trace and assertion

 AnitEntropy/MerkleTree Error
 

 Key: CASSANDRA-3014
 URL: https://issues.apache.org/jira/browse/CASSANDRA-3014
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 0.8.0
Reporter: Hefeng Yuan
Priority: Minor

 Hi,
 We are seeing some AntiEntropy error in our production servers, since it's 
 hard to repro in other env, pasting the stack track here for some clue. 
 We're using a cluster of 2 data centers, 9 nodes, 6 for online traffic, 3 for 
 Brisk BI. Our RF is Cassandra:5, Brisk:1.
 Our Cassandra server version is 0.8.0. Data is written using Hector 0.7.0-20 
 and cassandra 0.7.2 library.
 Partitioner is random.
 Our nodetool repair is scheduled once per week.
 The exception stack is appended in the end.
 Any help is appreciated.
 Thanks,
 Hefeng
 ERROR [AntiEntropyStage:2] 2011-08-08 04:24:39,556 
 AbstractCassandraDaemon.java (line 113) Fatal exception in thread 
 Thread[AntiEntropyStage:2,5,main]
 java.lang.AssertionError
   at org.apache.cassandra.utils.MerkleTree.inc(MerkleTree.java:154)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:262)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at

[jira] [Commented] (CASSANDRA-3014) AnitEntropy/MerkleTree Error

2011-08-11 Thread Sylvain Lebresne (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-3014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13083002#comment-13083002
 ] 

Sylvain Lebresne commented on CASSANDRA-3014:
-

Yeah, this has a real chance to have been fixed since 0.8.0. This seems to be a 
problem of the merkle tree being deeper that the max depth limit. So it is 
likely a problem during the creation of the tree that create a deeper tree than 
allowed (possibly by just 1). However, since 0.8.0, we have both fixed a 
related bug in the creation function (CASSANDRA-2758) and changed the creation 
function used for RandomPartitioner (CASSANDRA-2841).

 AnitEntropy/MerkleTree Error
 

 Key: CASSANDRA-3014
 URL: https://issues.apache.org/jira/browse/CASSANDRA-3014
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 0.8.0
Reporter: Hefeng Yuan
Priority: Minor

 Hi,
 We are seeing some AntiEntropy error in our production servers, since it's 
 hard to repro in other env, pasting the stack track here for some clue. 
 We're using a cluster of 2 data centers, 9 nodes, 6 for online traffic, 3 for 
 Brisk BI. Our RF is Cassandra:5, Brisk:1.
 Our Cassandra server version is 0.8.0. Data is written using Hector 0.7.0-20 
 and cassandra 0.7.2 library.
 Partitioner is random.
 Our nodetool repair is scheduled once per week.
 The exception stack is appended in the end.
 Any help is appreciated.
 Thanks,
 Hefeng
 ERROR [AntiEntropyStage:2] 2011-08-08 04:24:39,556 
 AbstractCassandraDaemon.java (line 113) Fatal exception in thread 
 Thread[AntiEntropyStage:2,5,main]
 java.lang.AssertionError
   at org.apache.cassandra.utils.MerkleTree.inc(MerkleTree.java:154)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:262)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at

[jira] [Commented] (CASSANDRA-3014) AnitEntropy/MerkleTree Error

2011-08-10 Thread Jonathan Ellis (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-3014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13082474#comment-13082474
 ] 

Jonathan Ellis commented on CASSANDRA-3014:
---

We've fixed many bugs in repair since 0.8.0.  Please upgrade to 0.8.3 and 
re-test.

 AnitEntropy/MerkleTree Error
 

 Key: CASSANDRA-3014
 URL: https://issues.apache.org/jira/browse/CASSANDRA-3014
 Project: Cassandra
  Issue Type: Bug
  Components: Core
Affects Versions: 0.8.0
Reporter: Hefeng Yuan
Priority: Minor

 Hi,
 We are seeing some AntiEntropy error in our production servers, since it's 
 hard to repro in other env, pasting the stack track here for some clue. 
 We're using a cluster of 2 data centers, 9 nodes, 6 for online traffic, 3 for 
 Brisk BI. Our RF is Cassandra:5, Brisk:1.
 Our Cassandra server version is 0.8.0. Data is written using Hector 0.7.0-20 
 and cassandra 0.7.2 library.
 Partitioner is random.
 Our nodetool repair is scheduled once per week.
 The exception stack is appended in the end.
 Any help is appreciated.
 Thanks,
 Hefeng
 ERROR [AntiEntropyStage:2] 2011-08-08 04:24:39,556 
 AbstractCassandraDaemon.java (line 113) Fatal exception in thread 
 Thread[AntiEntropyStage:2,5,main]
 java.lang.AssertionError
   at org.apache.cassandra.utils.MerkleTree.inc(MerkleTree.java:154)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:262)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:284)
   at 
 org.apache.cassandra.utils.MerkleTree.differenceHelper(MerkleTree.java:273)
   at