[ https://issues.apache.org/jira/browse/CASSANDRA-8316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14246153#comment-14246153 ]
Alan Boudreault edited comment on CASSANDRA-8316 at 12/14/14 9:59 PM: ---------------------------------------------------------------------- [~krummas], unfortunately, I'm still getting the initial issue AND the issue introduced on multiple nodes with the v2 patch + CASSANDRA-8458. 8 nodes cluster and cassandra-stress with n=1000000 {code} aboudreault@kovarro:~/dev/cstar/8316$ ccm node3 nodetool -- repair -par -inc [2014-12-14 16:30:48,037] Starting repair command #1, repairing 3 ranges for keyspace r1 (parallelism=PARALLEL, full=false) [2014-12-14 16:30:48,040] Repair failed with error Already repairing SSTableReader(path='/home/aboudreault/.ccm/local/node3/data/r1/Standard1-d11383e083d411e4869ab56034537865/r1-Standard1-ka-38-Data.db'), can not continue. {code} In addition, I noticed that it took 20 minutes restart my cluster this time. Not sure if it's related to this issue but I've attached a yourkit snapshot. Let me know if I can anything else. was (Author: aboudreault): [~krummas], unfortunately, I'm still getting the same issue introduced on multiple nodes with the v2 patch + CASSANDRA-8458. 8 nodes cluster and cassandra-stress with n=1000000 {code} aboudreault@kovarro:~/dev/cstar/8316$ ccm node3 nodetool -- repair -par -inc [2014-12-14 16:30:48,037] Starting repair command #1, repairing 3 ranges for keyspace r1 (parallelism=PARALLEL, full=false) [2014-12-14 16:30:48,040] Repair failed with error Already repairing SSTableReader(path='/home/aboudreault/.ccm/local/node3/data/r1/Standard1-d11383e083d411e4869ab56034537865/r1-Standard1-ka-38-Data.db'), can not continue. {code} In addition, I noticed that it took 20 minutes restart my cluster this time. Not sure if it's related to this issue but I've attached a yourkit snapshot. Let me know if I can anything else. > "Did not get positive replies from all endpoints" error on incremental repair > ------------------------------------------------------------------------------ > > Key: CASSANDRA-8316 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8316 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: cassandra 2.1.2 > Reporter: Loic Lambiel > Assignee: Marcus Eriksson > Fix For: 2.1.3 > > Attachments: 0001-patch.patch, 8316-v2.patch, > CassandraDaemon-2014-11-25-2.snapshot.tar.gz, > CassandraDaemon-2014-12-14.snapshot.tar.gz, test.sh > > > Hi, > I've got an issue with incremental repairs on our production 15 nodes 2.1.2 > (new cluster, not yet loaded, RF=3) > After having successfully performed an incremental repair (-par -inc) on 3 > nodes, I started receiving "Repair failed with error Did not get positive > replies from all endpoints." from nodetool on all remaining nodes : > [2014-11-14 09:12:36,488] Starting repair command #3, repairing 108 ranges > for keyspace xxxx (seq=false, full=false) > [2014-11-14 09:12:47,919] Repair failed with error Did not get positive > replies from all endpoints. > All the nodes are up and running and the local system log shows that the > repair commands got started and that's it. > I've also noticed that soon after the repair, several nodes started having > more cpu load indefinitely without any particular reason (no tasks / queries, > nothing in the logs). I then restarted C* on these nodes and retried the > repair on several nodes, which were successful until facing the issue again. > I tried to repro on our 3 nodes preproduction cluster without success > It looks like I'm not the only one having this issue: > http://www.mail-archive.com/user%40cassandra.apache.org/msg39145.html > Any idea? > Thanks > Loic -- This message was sent by Atlassian JIRA (v6.3.4#6332)