Artur Malchanau created CASSANDRA-21819:
-------------------------------------------
Summary: AutoRepair repair_session_timeout does not abort the
timed-out repair, leaking a hung incremental repair session that blocks all
later incremental repairs of its ranges
Key: CASSANDRA-21819
URL: https://issues.apache.org/jira/browse/CASSANDRA-21819
Project: Apache Cassandra
Issue Type: Bug
Components: Consistency/Repair
Reporter: Artur Malchanau
Attachments: AutoRepairSessionTimeoutTest-repro.patch
h3. Summary
When an auto-repair task exceeds {{{}repair_session_timeout{}}}, AutoRepair
"cancels" it with {{Future.cancel(true)}} and retries the same ranges. That
cancel is a no-op: {{RepairCoordinator.run()}} only wires an asynchronous
future chain and returns right away, so the executor future is already done and
{{cancel()}} returns {{{}false{}}}. The repair keeps running, or keeps hanging,
in the background:
* the incremental repair (IR) session stays {{REPAIRING}} on every
participant, and its sstables stay {{{}pendingRepair{}}};
* every retry, and every later incremental repair of the same ranges, fails in
the prepare phase with {{{}...encountered intersecting sstables belonging to
another incremental repair session (<old id>){}}};
* the {{RepairSession}} stays registered on the coordinator.
Nothing ever cleans this up:
* LocalSessions never auto-fails the session, because the participants keep
answering each other's status requests with {{{}REPAIRING{}}}, and since
CASSANDRA-20312 each such answer refreshes {{{}lastUpdate{}}};
* the coordinator has no timeout of its own on merkle-tree responses.
In our production cluster the session stayed hung for 7 days, until an operator
ran {{{}nodetool repair_admin cancel{}}}. Incremental repair of the affected
ranges was blocked for that whole time.
h3. Root cause
[AutoRepair.java (5.0.9), lines
344-398|https://github.com/apache/cassandra/blob/cassandra-5.0.9/src/java/org/apache/cassandra/repair/autorepair/AutoRepair.java#L344-L398]:
{code:java}
f = repairRunnableExecutors.get(repairType).submit(task); // task =
RepairCoordinator
listener.await(config.getRepairSessionTimeout(repairType)); // times out ->
success = false
...
boolean cancellationStatus = f.cancel(true); // always false:
run() already returned
// sleep repair_retry_backoff, then submit a NEW RepairCoordinator for the same
ranges
{code}
[RepairCoordinator.run()/runMayThrow()|https://github.com/apache/cassandra/blob/cassandra-5.0.9/src/java/org/apache/cassandra/repair/RepairCoordinator.java#L273-L309]
starts prepare -> repair -> finalize as a future chain and returns. Nothing in
AutoRepair ever fails the parent repair, its {{{}RepairSession}}s, or the IR
{{CoordinatorSession{}}}. As a result, no {{FailSession}} is sent and the
participants keep the sstables in {{{}pendingRepair{}}}.
The same logic is present on the current {{{}cassandra-5.0{}}},
{{cassandra-6.0}} and {{trunk}} branches. Trunk only adds an {{isShutDown}}
branch, which uses the same {{{}f.cancel(true){}}}.
h3. How the session got stuck in production (trigger; not the bug itself)
5.0.9, three DCs, incremental auto-repair, {{repair.retries}} not configured
(default: disabled).
* 10:16:48 UTC: the coordinator (node C) starts IR session
{{8681eb40-cd38-11f1-b81f-4153cf3313e6}} with 9 participants. Prepare succeeds,
and all 54 VALIDATION_REQs (6 tables x 9 replicas) are acknowledged.
* 10:17:00: the LARGE_MESSAGES connections from two remote-DC replicas to node
C fail to connect ({{{}connection timed out after 2000 ms{}}}). While a
connection is failing to connect, {{OutboundConnection}} rejects any message
that doesn't fit the per-connection send queue, so three 8 MiB merkle trees are
dropped:
{noformat}
WARN [AntiEntropyStage:1] 2026-10-21T10:17:00,569 NoSpamLogger.java:107 -
/R1:7000->/C:7000-LARGE_MESSAGES-[no-channel] overloaded; dropping 8.019MiB
message (queue: 4.488MiB local, 500.008KiB endpoint, 500.008KiB global)
WARN [AntiEntropyStage:1] 2026-10-21T10:17:00,298 NoSpamLogger.java:107 -
/R2:7000->/C:7000-LARGE_MESSAGES-[no-channel] overloaded; dropping 8.315MiB
message (queue: 12.507MiB local, 8.507MiB endpoint, 8.507MiB global)
{noformat}
The connections came back about 3s later. With retries disabled, the drop is
never retried and the coordinator is never told; this matches CASSANDRA-20560,
which was closed as expected without retries. The
{{Connection.LargeMessageDroppedTasksDueToOverload}} metrics confirm 2+1 drops
at exactly that time.
* 13:16:48: AutoRepair's 3h timeout fires; the cancel is a no-op, and the
retry immediately collides:
{noformat}
WARN [AutoRepair-Repair-incremental:1] 2026-10-21T13:16:48,372
AutoRepair.java:370 - Repair failed for range
-2779633433731331207--2763034864604948441 for cell tables [...] with
cancellationStatus: false retrying after 30 seconds...
ERROR [AntiEntropyStage:1] 2026-10-21T13:17:59,740 AutoRepair.java:581 - Repair
failure for repair INCREMENTAL: [...] Repair command #89 failed with error
Incremental repair session cf929050-cd51-11f1-b81f-4153cf3313e6 has failed
{noformat}
* For the next 7 days, every incremental auto-repair touching those ranges
failed with:
{noformat}
WARN [Repair-46a1cca0-...:2] 2026-10-28T08:55:08,385
PendingAntiCompaction.java:210 - Prepare phase for incremental repair session
46a1cca0-d2ad-11f1-b81f-4153cf3313e6 has failed because it encountered
intersecting sstables belonging to another incremental repair session
(8681eb40-cd38-11f1-b81f-4153cf3313e6). ...
{noformat}
** {{repair_admin list}} on all 9 participants showed {{8681eb40...}} as
{{{}REPAIRING{}}}, with "last activity" always below one hour; the status-check
keep-alive kept it alive.
** {{Table.BytesPendingRepair}} on the coordinator stayed at about 364 MB the
whole time.
** A second session in the same cluster
({{{}349f6f90-cd39-11f1-9a8e-0962703fd491{}}}, a different coordinator, a
dropped merkle tree in the same minute) had been hung in the same way.
h3. Steps to reproduce
The attached in-JVM dtest {{AutoRepairSessionTimeoutTest}} reproduces this
deterministically in about 2 minutes:
# 3 nodes, RF=3, one table with 1000 rows, flushed.
# Incremental auto-repair runs on node1 only, with {{{}repair_session_timeout:
60s{}}}, {{repair_max_retries: 1}} and {{{}repair_retry_backoff: 1s{}}}. These
two are set in the incremental override, because the global values are ignored
on 5.0.9 (CASSANDRA-21748).
# The LocalSessions timers are shortened (status check 5s, auto-fail 15s,
cleanup 5s) to show that they don't recover the session either.
# A message filter drops exactly one {{VALIDATION_RSP}} (node2 -> node1) for
the test table. {{repair.retries}} is left at its default (disabled).
{code:java}
git checkout cassandra-5.0.9
git apply AutoRepairSessionTimeoutTest-repro.patch
export JAVA_HOME=<JDK 17>
ant jar
ant test-jvm-dtest-some
-Dtest.name=org.apache.cassandra.distributed.test.repair.AutoRepairSessionTimeoutTest
{code}
Observed on 5.0.9 (all intermediate checks pass; the final check, which asserts
the expected behaviour, fails):
{noformat}
[P1] T0: dropped VALIDATION_RSP 2->1 for incremental repair session
d76f47ca-d2f9-11f1-89c1-17ed1be92f21
[P2] session d76f47ca-... still REPAIRING on all nodes at T0+35s (> 2x
the 15s LocalSessions auto-fail timeout)
[P3] node1: WARN ... AutoRepair.java:370 - Repair failed for range
-9223372036854775808--3074457345618258603 for distributed_test_keyspace tables
[tbl] with cancellationStatus: false retrying after 1 seconds...
[P4] WARN ... PendingAntiCompaction.java:210 - Prepare phase for incremental
repair session fbcc35b0-... has failed because it encountered intersecting
sstables belonging to another incremental repair session
(d76f47ca-d2f9-11f1-89c1-17ed1be92f21) ...
[P5] node1: ERROR ... AutoRepair.java:394 - Repair failed for range ... after 2
retries, total assignments: 2,processed assignments: 1, cancellationStatus:
false
[FINAL] node1 ActiveRepairService sessionCount=1
junit.framework.AssertionFailedError: AutoRepair gave up after
repair_session_timeout, but incremental repair session
d76f47ca-d2f9-11f1-89c1-17ed1be92f21 is still REPAIRING on node1; its
pendingRepair sstables block every later incremental repair of these ranges
expected:<[FAILED]> but was:<[REPAIRING]>
{noformat}
Notes on the test:
* It uses the in-JVM message path, without the NETWORK/GOSSIP features.
* The matcher finds the table by scanning the raw message bytes for the
RepairJobDesc keyspace/table strings, because a {{ValidationResponse}} can't be
deserialized in the test classloader.
* It has only been run on 5.0.9 so far.
h3. Expected behaviour
When AutoRepair gives up on a repair after {{{}repair_session_timeout{}}}, that
repair should actually be aborted:
* the IR session is failed on all participants ({{{}FailSession{}}}), so its
{{pendingRepair}} sstables are released back to unrepaired;
* the coordinator's {{{}RepairSession{}}}/{{{}ParentRepairSession{}}} are
removed;
* the retry starts from a clean state.
h3. Suggested fix
# On timeout, abort the specific repair instead of calling
{{{}f.cancel(true){}}}. Use the parent session id
({{{}RepairCoordinator.state.id{}}}) to:
** force-shutdown that parent's \{{RepairSession}}s, so the coordinator chain
completes exceptionally;
** fail the IR {{{}CoordinatorSession{}}}, which sends {{FailSession}} to the
participants;
** remove the {{{}ParentRepairSession{}}}.
This may need a small {{{}RepairCoordinator{}}}/{{{}ActiveRepairService{}}} API
for aborting a single repair, like {{terminateSessions()}} but scoped to one
repair.
# Log the real outcome of the abort; the current {{cancellationStatus}} is
always {{false}} and misleading.
# Optionally: don't start a retry for the same ranges while the previous IR
session is still non-terminal.
# Optionally: when incremental auto-repair is enabled while {{repair.retries}}
(merkle_tree_response) is disabled, warn at startup or document it prominently.
A single dropped merkle tree otherwise hangs the repair indefinitely
(CASSANDRA-20560).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
