Hi All,
First, let me apologize for the long message. Repair service is driving me
insane. :)
We are running Cassandra version *2.1.11.908* in 3 data centers. While
running nodetool repair we are receiving the following error:
[2015-11-20 00:25:24,803] Repair session
2ed2f930-8f1d-11e5-a498-4b9679ec178d for range
(4375311677306593380,4384065392379704564] failed with error
org.apache.cassandra.exceptions.RepairException:
[repair #2ed2f930-8f1d-11e5-a498-4b9679ec178d on keyspace/table,
(4375311677306593380,4384065392379704564]] Sync failed between /10.10.8.104
and /10.30.30.6
Unfortunately, not much information online. Any ideas how to fix this issue
or determine root cause?
Thanks!
Detailed log below:
ERROR [AntiEntropySessions:8] 2015-11-20 00:25:24,801
RepairSession.java:303 - [repair #2ed2f930-8f1d-11e5-a498-4b9679ec178d]
session completed with the following
error org.apache.cassandra.exceptions.RepairException: [repair
#2ed2f930-8f1d-11e5-a498-4b9679ec178d on keyspace/table,
(4375311677306593380,4384065392379704564]] Sync failed between /10.10.8.104
and /10.30.30.6 at
org.apache.cassandra.repair.RepairSession.syncComplete(RepairSession.java:223)
~[cassandra-all-2.1.11.908.jar:2.1.11.908] at
org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:411)
~[cassandra-all-2.1.11.908.jar:2.1.11.908]
at
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:134)
~[cassandra-all-2.1.11.908.jar:2.1.11.908]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
~[cassandra-all-2.1.11.908.jar:2.1.11.908]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source) [na:1.8.0_60]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source) [na:1.8.0_60]
at java.lang.Thread.run(Unknown Source) [na:1.8.0_60]
ERROR [AntiEntropySessions:8] 2015-11-20 00:25:24,802
CassandraDaemon.java:227 - Exception in thread
Thread[AntiEntropySessions:8,5,RMI Runtime]
java.lang.RuntimeException:
org.apache.cassandra.exceptions.RepairException: [repair
#2ed2f930-8f1d-11e5-a498-4b9679ec178d on keyspace/table,
(4375311677306593380,4384065392379704564]] Sync failed between /10.10.8.104
and /10.30.30.6
at com.google.common.base.Throwables.propagate(Throwables.java:160)
~[guava-16.0.1.jar:na]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
~[cassandra-all-2.1.11.908.jar:2.1.11.908]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
Source) ~[na:1.8.0_60]
at java.util.concurrent.FutureTask.run(Unknown Source)
~[na:1.8.0_60]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source) [na:1.8.0_60]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source) [na:1.8.0_60]
at java.lang.Thread.run(Unknown Source) [na:1.8.0_60]
Caused by: org.apache.cassandra.exceptions.RepairException: [repair
#2ed2f930-8f1d-11e5-a498-4b9679ec178d on keyspace/table,
(4375311677306593380,4384065392379704564]] Sync failed between /10.10.8.104
and /10.30.30.6
at
org.apache.cassandra.repair.RepairSession.syncComplete(RepairSession.java:223)
~[cassandra-all-2.1.11.908.jar:2.1.11.908]
at
org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:411)
~[cassandra-all-2.1.11.908.jar:2.1.11.908]
at
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:134)
~[cassandra-all-2.1.11.908.jar:2.1.11.908]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
~[cassandra-all-2.1.11.908.jar:2.1.11.908]
... 3 common frames omitted
ERROR [Thread-3128] 2015-11-20 00:25:24,802 StorageService.java:3008 -
Repair session 2ed2f930-8f1d-11e5-a498-4b9679ec178d for range
(4375311677306593380,4384065392379704564] failed with error
org.apache.cassandra.exceptions.RepairException: [repair
#2ed2f930-8f1d-11e5-a498-4b9679ec178d on keyspace/table,
(4375311677306593380,4384065392379704564]] Sync failed between /10.10.8.104
and /10.30.30.6
java.util.concurrent.ExecutionException: java.lang.RuntimeException:
org.apache.cassandra.exceptions.RepairException: [repair
#2ed2f930-8f1d-11e5-a498-4b9679ec178d on keyspace/table,
(4375311677306593380,4384065392379704564]] Sync failed between /10.10.8.104
and /10.30.30.6
at java.util.concurrent.FutureTask.report(Unknown Source)
[na:1.8.0_60]
at java.util.concurrent.FutureTask.get(Unknown Source) [na:1.8.0_60]
at
org.apache.cassandra.service.StorageService$4.runMayThrow(StorageService.java:2999)
~[cassandra-all-2.1.11.908.jar:2.1.11.908]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
[cassandra-all-2.1.11.908.jar:2.1.11.908]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
Source) [na:1.8.0_60]
at java.util.concurrent.FutureTask.run(Unknown Source) [na:1.8.0_60]
at java.lang.Thread.run(Unknown Source) [na:1.8.0_60]
Caused by: java.lang.RuntimeException:
org.apache.cassandra.exceptions.RepairException: [repair
#2ed2f930-8f1d-11e5-a498-4b9679ec178d on keyspace/table,
(4375311677306593380,4384065392379704564]] Sync failed between /10.10.8.104
and /10.30.30.6
at com.google.common.base.Throwables.propagate(Throwables.java:160)
~[guava-16.0.1.jar:na]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
[cassandra-all-2.1.11.908.jar:2.1.11.908]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
Source) [na:1.8.0_60]
at java.util.concurrent.FutureTask.run(Unknown Source) [na:1.8.0_60]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source) ~[na:1.8.0_60]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source) ~[na:1.8.0_60]
... 1 common frames omitted
Caused by: org.apache.cassandra.exceptions.RepairException: [repair
#2ed2f930-8f1d-11e5-a498-4b9679ec178d on keyspace/table,
(4375311677306593380,4384065392379704564]] Sync failed between /10.10.8.104
and /10.30.30.6
at
org.apache.cassandra.repair.RepairSession.syncComplete(RepairSession.java:223)
~[cassandra-all-2.1.11.908.jar:2.1.11.908]
at
org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:411)
~[cassandra-all-2.1.11.908.jar:2.1.11.908]
at
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:134)
~[cassandra-all-2.1.11.908.jar:2.1.11.908]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
~[cassandra-all-2.1.11.908.jar:2.1.11.908]
... 3 common frames omitted
Thanks,
Rafael
