[
https://issues.apache.org/jira/browse/CASSANDRA-20186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18119946#comment-18119946
]
George commented on CASSANDRA-20186:
------------------------------------
I investigated this on trunk at f43a7b9265827cc1a2765ee8bebd53ffe6f261ae and
would like to align on the scheduling contract before implementing a production
change.
*Current behavior*
* {{AutoRepair.tooSoonToRunRepair()}} applies {{min_repair_interval}} from the
local node's last finish time. Incomplete/unstarted rounds and force repair
have special handling.
* {{AutoRepairUtils.myTurnToRunRepair()}} checks ongoing repairs, priority and
the oldest eligible finish time, including replica conflicts where configured.
There is no time-based spacing between ordinary starts on different nodes.
* Consequently, the local minimum interval can be respected while a small
cluster's starts remain concentrated. Increasing {{repair_check_interval}} or
reducing concurrency can slow scheduling, but does not define a spreading
policy.
Source references: [local interval
check|https://github.com/apache/cassandra/blob/f43a7b9265827cc1a2765ee8bebd53ffe6f261ae/src/java/org/apache/cassandra/repair/autorepair/AutoRepair.java#L431]
and [turn
selection|https://github.com/apache/cassandra/blob/f43a7b9265827cc1a2765ee8bebd53ffe6f261ae/src/java/org/apache/cassandra/repair/autorepair/AutoRepairUtils.java#L915].
*Local characterization*
With JDK 17, the build and 63 relevant unit tests passed (48 existing tests
plus 15 new interval checks). Four new JVM distributed tests also passed:
* Three real nodes, NETWORK/GOSSIP enabled, RF=3, completed histories older
than 24h, {{min_repair_interval=24h}}, concurrency 1, and parallel replica
repair disabled.
* For FULL, INCREMENTAL and PREVIEW_REPAIRED separately, manually driving the
scheduler at controlled offsets of 0, 5 and 10 minutes allowed all three node
rounds to complete. Immediately repeating a round on the same node was blocked
by its local interval.
* A separate history-based admission test confirmed that capacity and replica
conflicts block the next node while its peer is running; marking that peer
finished allows the next node immediately.
The distributed tests use real repair execution and history storage, with a
controlled AutoRepair clock and shortened task waits. These are scheduling
characterizations, not a wall-clock benchmark or a reproduction of the original
cluster's precise duration. I have not run the full suite, fault-injection,
cross-DC, mixed-version or production-load tests.
*Possible direction*
An independently configurable pacing policy for ordinary node-round starts
could preserve the existing finish-to-next-start minimum interval and reuse the
current eligibility/concurrency rules. A starting point for discussion is a
target spacing of {{I / N}}, where I is the configured interval and N is the
participating population in the same scheduling domain. For 12 participants and
24h, that would target one ordinary start about every 2h; concurrency remains a
separate limit.
This would require bounded extra delay/catch-up behavior, explicit membership
and configuration assumptions, and observability of deliberate pacing delays.
Newly seeded history timestamps must not be mistaken for actual repair starts.
Force, interrupted-round recovery, first repair and priority need explicit
semantics; in particular we should preserve the behavior fixed by
CASSANDRA-21115 and CASSANDRA-21552.
Reusing the current read-history/decide/write-start path would provide
best-effort spreading. Strict global spacing would require atomic admission or
persistent permits and a larger design. The pacing would belong in ordinary
turn admission, without long sleeps or changes to the repair protocol.
Would maintainers prefer:
# Best-effort smoothing within the existing scheduler, or a strict cluster-wide
schedule?
# A separate opt-in policy preserving {{min_repair_interval}} semantics, or a
different configuration model?
# Which first-run/priority exemptions and maximum extra delay/catch-up
guarantees should we target?
No production patch or community CI run yet; these are proposed semantics for
discussion.
> AutoRepair should splay repairs based on min_repair_interval
> ------------------------------------------------------------
>
> Key: CASSANDRA-20186
> URL: https://issues.apache.org/jira/browse/CASSANDRA-20186
> Project: Apache Cassandra
> Issue Type: Bug
> Reporter: Andy Tolbert
> Priority: Normal
>
> For smaller node count clusters AutoRepair currently schedules repairs in
> quick succession.
> Ideally we should try to evenly spread, to do this we could try to splay
> repairs out over min_repair_interval.
> Observationally, i've seen that with a 1 day {{{}min_repair_interval{}}}, all
> repairs might get executed within ~3hours as they are effectively throttled
> by a combination of {{parallel_repair_count}} , {{repair_check_interval}} and
> how long it takes to complete replicas for a replica.
> {noformat}
> select host_id, repair_start_ts from system_distributed.auto_repair_history
> where repair_type='FULL';
> host_id | repair_start_ts
> -------------------------------------+--------------------------------
> 6d194555-f6eb-41d0-c000-000000000001 | 2025-01-06 03:59:53.399000+0000
> 6d194555-f6eb-41d0-c000-000000000002 | 2025-01-06 05:36:49.299000+0000
> 6d194555-f6eb-41d0-c000-000000000003 | 2025-01-06 05:46:48.505000+0000
> 6d194555-f6eb-41d0-c000-000000000004 | 2025-01-06 04:04:27.126000+0000
> 6d194555-f6eb-41d0-c000-000000000005 | 2025-01-06 03:53:32.573000+0000
> 6d194555-f6eb-41d0-c000-000000000006 | 2025-01-06 06:31:49.930000+0000
> 6d194555-f6eb-41d0-c000-000000000007 | 2025-01-06 05:41:46.562000+0000
> 6d194555-f6eb-41d0-c000-000000000008 | 2025-01-06 04:45:57.136000+0000
> 6d194555-f6eb-41d0-c000-000000000009 | 2025-01-06 06:32:59.428000+0000
> 6d194555-f6eb-41d0-c000-00000000000a | 2025-01-06 04:50:14.020000+0000
> 6d194555-f6eb-41d0-c000-00000000000b | 2025-01-06 06:33:30.846000+0000
> 6d194555-f6eb-41d0-c000-00000000000c | 2025-01-06
> 04:40:05.365000+0000{noformat}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
