Jaydeepkumar Chovatia created CASSANDRA-14674: -------------------------------------------------
Summary: Repair Validation message request could get stuck forever at sender side Key: CASSANDRA-14674 URL: https://issues.apache.org/jira/browse/CASSANDRA-14674 Project: Cassandra Issue Type: Bug Components: Repair Reporter: Jaydeepkumar Chovatia Assignee: Jaydeepkumar Chovatia Validation request message as part of repair are currently sent as [sendOneWay|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/repair/ValidationTask.java#L56] and then it waits at [Futures.getUnchecked|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/repair/RepairJob.java#L160]]. If sender doesn’t hear back from receiver for whatever reason then thread is blocked forever. I’ve reproduced following stack trace at sender side by deliberately ignoring [VALIDATION_REQUST|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/repair/RepairMessageVerbHandler.java#L114] at receiver side. {quote} "Repair#1:1" #301 daemon prio=5 os_prio=0 tid=0x00007f5a62060800 nid=0x13198 waiting on condition [0x00007f5a5cc6c000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) parking to wait for <0x00000005c6ba9630> (a com.google.common.util.concurrent.AbstractFuture$Sync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:285) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:137) at com.google.common.util.concurrent.Futures.getUnchecked(Futures.java:1509) at org.apache.cassandra.repair.RepairJob.run(RepairJob.java:160) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79) at org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$3/1858015030.run(Unknown Source) at java.lang.Thread.run(Thread.java:745) {quote} AFAIK we should be using {{sendRR}} for this instead of {{sendOneWay}}. Please let me know if my understanding is correct or not. I am working on a fix to make it {{sendRR}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org