[
https://issues.apache.org/jira/browse/HDDS-14950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Andika updated HDDS-14950:
-------------------------------
Description:
We recently found Datanode lifecycle race condition like HDDS-14834 which
causes node inconsistency between SCM NodeStateManager and NetworkTopology.
HDDS-14834 reduces the window to make race condition highly improbable, but it
might theoretically happen.
!image-2026-04-01-10-55-37-370.png|width=738,height=219!
Practical risk is near-zero but not zero:
• The window is a handful of JVM bytecode instructions (nanoseconds of CPU
time).
• For the race to manifest, the HealthyReadOnlyNodeHandler must fully
complete (including closePipeline() I/O for every pipeline on that node) within
that window.
• The JVM can preempt a thread at any point for an arbitrary duration (GC
pause, OS scheduling), so theoretically the DeadNodeHandler thread could be
paused after the state check long enough for the entire
pipeline to play out.
We need to revisit the SCM datanode management concurrency model to prevent
similar race conditions. We also need to revisit the whole SCM event framework.
Possible approach
# A lock that makes the state-check + topology-remove atomic.
# A double-check: re-read the state after nt.remove() and call nt.add() back
if the node is no longer DEAD.
# A version/epoch on the topology mutation so stale removes are rejected.
However locking implementation might be involved
There are several reasons the state-check guard approach was preferred over
merging both handlers onto a single shared executor:
1. Event framework architecture: Ozone's EventQueue assigns each event type
its own SingleThreadExecutor. Forcing two different event types (DEAD_NODE and
HEALTHY_READONLY_NODE) onto a shared executor would
require changes to the event framework infrastructure itself — a much more
invasive change.
2. Throughput and isolation: DeadNodeHandler.onMessage() does heavy work —
closing containers, destroying pipelines, removing replicas. Serializing it
with HealthyReadOnlyNodeHandler means a slow dead-node
cleanup would block healthy-readonly transitions from being processed,
degrading cluster responsiveness during node churn.
3. Doesn't eliminate stale events: Even with a shared executor, the
fundamental problem remains — the DEAD_NODE event is stale by the time it's
dequeued. The node may have already transitioned back to
HEALTHY_READONLY via a heartbeat. A shared executor only guarantees
ordering within the queue, but it can't prevent a DEAD_NODE event (enqueued
earlier) from executing after the node has legitimately
resurrected. You'd still need the state-check guard.
4. Other topology-touching handlers exist: It's not just these two.
HealthyNodeHandler, StaleNodeHandler, and others also interact with
NetworkTopology. A shared executor for only two of them wouldn't fully
protect against races with the rest, yet pulling all of them onto one
executor would create a massive serialization bottleneck.
5. Minimal, targeted fix: The state-check in DeadNodeHandler + unconditional
add in HealthyReadOnlyNodeHandler is a two-line-level change that's correct
regardless of executor topology. It follows the general
principle that event handlers in an async system should always validate
current state before acting on a potentially stale even
Here's why locking was not the preferred approach:
1. Coarse-grained lock = bottleneck: If you put a lock around the
NetworkTopology operations in both handlers, you're serializing all topology
mutations. This has the same throughput problem as the shared
executor — DeadNodeHandler's heavy work (container close, pipeline
destroy) would hold the lock and block HealthyReadOnlyNodeHandler from making
progress.
2. Fine-grained lock = complexity and deadlock risk: A per-node lock would
avoid the global bottleneck, but introduces significant complexity. You'd need
to manage a lock map keyed by datanode ID, handle lock
lifecycle (creation, cleanup, eviction), and carefully reason about lock
ordering across all the handlers that touch topology (DeadNodeHandler,
HealthyReadOnlyNodeHandler, HealthyNodeHandler,
StaleNodeHandler, etc.) to avoid deadlocks.
3. Still doesn't address the root cause: The fundamental issue is that
DeadNodeHandler acts on a stale event — the node is no longer dead by the time
the handler runs. Locking around the topology mutation
would prevent interleaving, but if DeadNodeHandler acquires the lock
first, it still removes a node that has already resurrected. The lock ensures
atomicity of the topology operation, but the operation
itself is wrong because the premise (node is dead) is no longer true.
You'd still need the state check.
4. Locking is the wrong abstraction here: The problem isn't concurrent
mutation of a shared data structure (which is what locks solve). The problem is
acting on outdated information. The correct fix is to
re-validate the precondition (is the node still dead?) before acting.
That's an idempotency/guard pattern, not a mutual exclusion pattern.
5. `NetworkTopologyImpl` already has internal synchronization: The topology's
add() and remove() methods are themselves thread-safe. The race isn't about
corrupting the topology's internal state — it's about
the logical outcome being wrong (a live node absent from the topology).
Wrapping already-synchronized calls in another lock doesn't fix a logic error.
In summary: locking solves data structure corruption from concurrent access,
but the bug here is a semantic race — acting on a stale event. The state-check
guard directly addresses the root cause, is simpler,
has zero performance overhead, and doesn't require managing lock
infrastructure across the handler ecosystem.
was:
We recently found Datanode lifecycle race condition like HDDS-14834 which
causes node inconsistency between SCM NodeStateManager and NetworkTopology.
HDDS-14834 reduces the window to make race condition highly improbable, but it
might theoretically happen.
!image-2026-04-01-10-55-37-370.png|width=738,height=219!
Practical risk is near-zero but not zero:
• The window is a handful of JVM bytecode instructions (nanoseconds of CPU
time).
• For the race to manifest, the HealthyReadOnlyNodeHandler must fully
complete (including closePipeline() I/O for every pipeline on that node) within
that window.
• The JVM can preempt a thread at any point for an arbitrary duration (GC
pause, OS scheduling), so theoretically the DeadNodeHandler thread could be
paused after the state check long enough for the entire
pipeline to play out.
We need to revisit the SCM datanode management concurrency model to prevent
similar race conditions. We also need to revisit the whole SCM event framework.
Possible approach
# A lock that makes the state-check + topology-remove atomic.
# A double-check: re-read the state after nt.remove() and call nt.add() back
if the node is no longer DEAD.
# A version/epoch on the topology mutation so stale removes are rejected.
> Revisit SCM NodeManager concurrency
> -----------------------------------
>
> Key: HDDS-14950
> URL: https://issues.apache.org/jira/browse/HDDS-14950
> Project: Apache Ozone
> Issue Type: Improvement
> Reporter: Ivan Andika
> Assignee: Ivan Andika
> Priority: Major
> Attachments: image-2026-04-01-10-55-37-370.png
>
>
> We recently found Datanode lifecycle race condition like HDDS-14834 which
> causes node inconsistency between SCM NodeStateManager and NetworkTopology.
> HDDS-14834 reduces the window to make race condition highly improbable, but
> it might theoretically happen.
> !image-2026-04-01-10-55-37-370.png|width=738,height=219!
> Practical risk is near-zero but not zero:
> • The window is a handful of JVM bytecode instructions (nanoseconds of CPU
> time).
> • For the race to manifest, the HealthyReadOnlyNodeHandler must fully
> complete (including closePipeline() I/O for every pipeline on that node)
> within that window.
> • The JVM can preempt a thread at any point for an arbitrary duration (GC
> pause, OS scheduling), so theoretically the DeadNodeHandler thread could be
> paused after the state check long enough for the entire
> pipeline to play out.
> We need to revisit the SCM datanode management concurrency model to prevent
> similar race conditions. We also need to revisit the whole SCM event
> framework.
> Possible approach
> # A lock that makes the state-check + topology-remove atomic.
> # A double-check: re-read the state after nt.remove() and call nt.add() back
> if the node is no longer DEAD.
> # A version/epoch on the topology mutation so stale removes are rejected.
>
> However locking implementation might be involved
> There are several reasons the state-check guard approach was preferred over
> merging both handlers onto a single shared executor:
> 1. Event framework architecture: Ozone's EventQueue assigns each event type
> its own SingleThreadExecutor. Forcing two different event types (DEAD_NODE
> and HEALTHY_READONLY_NODE) onto a shared executor would
> require changes to the event framework infrastructure itself — a much
> more invasive change.
> 2. Throughput and isolation: DeadNodeHandler.onMessage() does heavy work —
> closing containers, destroying pipelines, removing replicas. Serializing it
> with HealthyReadOnlyNodeHandler means a slow dead-node
> cleanup would block healthy-readonly transitions from being processed,
> degrading cluster responsiveness during node churn.
> 3. Doesn't eliminate stale events: Even with a shared executor, the
> fundamental problem remains — the DEAD_NODE event is stale by the time it's
> dequeued. The node may have already transitioned back to
> HEALTHY_READONLY via a heartbeat. A shared executor only guarantees
> ordering within the queue, but it can't prevent a DEAD_NODE event (enqueued
> earlier) from executing after the node has legitimately
> resurrected. You'd still need the state-check guard.
> 4. Other topology-touching handlers exist: It's not just these two.
> HealthyNodeHandler, StaleNodeHandler, and others also interact with
> NetworkTopology. A shared executor for only two of them wouldn't fully
> protect against races with the rest, yet pulling all of them onto one
> executor would create a massive serialization bottleneck.
> 5. Minimal, targeted fix: The state-check in DeadNodeHandler +
> unconditional add in HealthyReadOnlyNodeHandler is a two-line-level change
> that's correct regardless of executor topology. It follows the general
> principle that event handlers in an async system should always validate
> current state before acting on a potentially stale even
>
> Here's why locking was not the preferred approach:
> 1. Coarse-grained lock = bottleneck: If you put a lock around the
> NetworkTopology operations in both handlers, you're serializing all topology
> mutations. This has the same throughput problem as the shared
> executor — DeadNodeHandler's heavy work (container close, pipeline
> destroy) would hold the lock and block HealthyReadOnlyNodeHandler from making
> progress.
> 2. Fine-grained lock = complexity and deadlock risk: A per-node lock would
> avoid the global bottleneck, but introduces significant complexity. You'd
> need to manage a lock map keyed by datanode ID, handle lock
> lifecycle (creation, cleanup, eviction), and carefully reason about lock
> ordering across all the handlers that touch topology (DeadNodeHandler,
> HealthyReadOnlyNodeHandler, HealthyNodeHandler,
> StaleNodeHandler, etc.) to avoid deadlocks.
> 3. Still doesn't address the root cause: The fundamental issue is that
> DeadNodeHandler acts on a stale event — the node is no longer dead by the
> time the handler runs. Locking around the topology mutation
> would prevent interleaving, but if DeadNodeHandler acquires the lock
> first, it still removes a node that has already resurrected. The lock ensures
> atomicity of the topology operation, but the operation
> itself is wrong because the premise (node is dead) is no longer true.
> You'd still need the state check.
> 4. Locking is the wrong abstraction here: The problem isn't concurrent
> mutation of a shared data structure (which is what locks solve). The problem
> is acting on outdated information. The correct fix is to
> re-validate the precondition (is the node still dead?) before acting.
> That's an idempotency/guard pattern, not a mutual exclusion pattern.
> 5. `NetworkTopologyImpl` already has internal synchronization: The
> topology's add() and remove() methods are themselves thread-safe. The race
> isn't about corrupting the topology's internal state — it's about
> the logical outcome being wrong (a live node absent from the topology).
> Wrapping already-synchronized calls in another lock doesn't fix a logic error.
> In summary: locking solves data structure corruption from concurrent
> access, but the bug here is a semantic race — acting on a stale event. The
> state-check guard directly addresses the root cause, is simpler,
> has zero performance overhead, and doesn't require managing lock
> infrastructure across the handler ecosystem.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]