Jaydeepkumar Chovatia created CASSANDRA-14804:
-------------------------------------------------
Summary: Running repair on multiple nodes in parallel could halt
entire repair
Key: CASSANDRA-14804
URL: https://issues.apache.org/jira/browse/CASSANDRA-14804
Project: Cassandra
Issue Type: Bug
Components: Repair
Reporter: Jaydeepkumar Chovatia
Fix For: 3.0.18
Possible deadlock if we run repair on multiple nodes at the same time. We have
come across a situation in production in which if we repair multiple nodes at
the same time then repair hangs forever. Here are the details:
Time t1
{{node-1}} has issued repair command to {{node-2}} but due to some reason
{{node-2}} didn't receive request hence {{node-1}} is awaiting at
[prepareForRepair
|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/service/ActiveRepairService.java#L333]
for 1 hour *with lock*
Time t2
{{node-2}} sent prepare repair request to {{node-1}}, some exception occurred
on {{node-1}} and it is trying to cleanup parent session
[here|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/repair/RepairMessageVerbHandler.java#L172]
but {{node-1}} cannot get lock as 1 hour of time has not yet elapsed (above
one)
snippet of jstack on {{node-1}}
{quote}"Thread-888" #262588 daemon prio=5 os_prio=0 waiting on condition
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for (a java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
at
org.apache.cassandra.service.ActiveRepairService.prepareForRepair(ActiveRepairService.java:332)
- locked <> (a org.apache.cassandra.service.ActiveRepairService)
at
org.apache.cassandra.repair.RepairRunnable.runMayThrow(RepairRunnable.java:214)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
at
org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$9/864248990.run(Unknown
Source)
at java.lang.Thread.run(Thread.java:748)
"AntiEntropyStage:1" #1789 daemon prio=5 os_prio=0 waiting for monitor entry []
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.cassandra.service.ActiveRepairService.removeParentRepairSession(ActiveRepairService.java:421)
- waiting to lock <> (a org.apache.cassandra.service.ActiveRepairService)
at
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:172)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
at
org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$9/864248990.run(Unknown
Source)
at java.lang.Thread.run(Thread.java:748){quote}
Time t3:
{{node-2}}(and possibly other nodes {{node-3}}…) sent [prepare request
|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/service/ActiveRepairService.java#L333]
to {{node-1}}, but {{node-1}}’s AntiEntropyStage thread is busy awaiting for
lock at {{ActiveRepairService.removeParentRepairSession}}, hence {{node-2}},
{{node-3}} (and possibly other nodes) will also go in 1 hour wait *with lock*.
This rolling effect continues and stalls repair in entire ring.
If we totally stop triggering repair then system would recover slowly but here
are the two major problems with this:
1. Externally there is no way to decide whether to trigger new repair or wait
for system to recover
2. In this case system recovers eventually but it takes probably {{n}} hours
where n = #of repair requests fired, only way to come out of this situation is
either to do a rolling restart of entire ring or wait for {{n}} hours before
triggering new repair request
Please let me know if my above analysis makes sense or not.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
