[ 
https://issues.apache.org/jira/browse/HDDS-14950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ivan Andika updated HDDS-14950:
-------------------------------
    Description: 
We recently found Datanode lifecycle race condition like HDDS-14834 which 
causes node inconsistency between SCM NodeStateManager and NetworkTopology. 
HDDS-14834 reduces the window to make race condition highly improbable, but it 
might theoretically happen. 

!image-2026-04-01-10-55-37-370.png|width=738,height=219!

Practical risk is near-zero but not zero:
  • The window is a handful of JVM bytecode instructions (nanoseconds of CPU 
time).
  • For the race to manifest, the HealthyReadOnlyNodeHandler must fully 
complete (including closePipeline() I/O for every pipeline on that node) within 
that window.
  • The JVM can preempt a thread at any point for an arbitrary duration (GC 
pause, OS scheduling), so theoretically the DeadNodeHandler thread could be 
paused after the state check long enough for the entire
    pipeline to play out.

We need to revisit the SCM datanode management concurrency model to prevent 
similar race conditions. We also need to revisit the whole SCM event framework.

Possible approach
 # A lock that makes the state-check + topology-remove atomic.
 # A double-check: re-read the state after nt.remove() and call nt.add() back 
if the node is no longer DEAD.
 # A version/epoch on the topology mutation so stale removes are rejected.

 

However locking implementation might be involved

There are several reasons the state-check guard approach was preferred over 
merging both handlers onto a single shared executor:
  1. Event framework architecture: Ozone's EventQueue assigns each event type 
its own SingleThreadExecutor. Forcing two different event types (DEAD_NODE and 
HEALTHY_READONLY_NODE) onto a shared executor would
     require changes to the event framework infrastructure itself — a much more 
invasive change.
  2. Throughput and isolation: DeadNodeHandler.onMessage() does heavy work — 
closing containers, destroying pipelines, removing replicas. Serializing it 
with HealthyReadOnlyNodeHandler means a slow dead-node
     cleanup would block healthy-readonly transitions from being processed, 
degrading cluster responsiveness during node churn.
  3. Doesn't eliminate stale events: Even with a shared executor, the 
fundamental problem remains — the DEAD_NODE event is stale by the time it's 
dequeued. The node may have already transitioned back to
     HEALTHY_READONLY via a heartbeat. A shared executor only guarantees 
ordering within the queue, but it can't prevent a DEAD_NODE event (enqueued 
earlier) from executing after the node has legitimately
     resurrected. You'd still need the state-check guard.
  4. Other topology-touching handlers exist: It's not just these two. 
HealthyNodeHandler, StaleNodeHandler, and others also interact with 
NetworkTopology. A shared executor for only two of them wouldn't fully
     protect against races with the rest, yet pulling all of them onto one 
executor would create a massive serialization bottleneck.
  5. Minimal, targeted fix: The state-check in DeadNodeHandler + unconditional 
add in HealthyReadOnlyNodeHandler is a two-line-level change that's correct 
regardless of executor topology. It follows the general
     principle that event handlers in an async system should always validate 
current state before acting on a potentially stale even

 

Here's why locking was not the preferred approach:
  1. Coarse-grained lock = bottleneck: If you put a lock around the 
NetworkTopology operations in both handlers, you're serializing all topology 
mutations. This has the same throughput problem as the shared
     executor — DeadNodeHandler's heavy work (container close, pipeline 
destroy) would hold the lock and block HealthyReadOnlyNodeHandler from making 
progress.
  2. Fine-grained lock = complexity and deadlock risk: A per-node lock would 
avoid the global bottleneck, but introduces significant complexity. You'd need 
to manage a lock map keyed by datanode ID, handle lock
     lifecycle (creation, cleanup, eviction), and carefully reason about lock 
ordering across all the handlers that touch topology (DeadNodeHandler, 
HealthyReadOnlyNodeHandler, HealthyNodeHandler,
     StaleNodeHandler, etc.) to avoid deadlocks.
  3. Still doesn't address the root cause: The fundamental issue is that 
DeadNodeHandler acts on a stale event — the node is no longer dead by the time 
the handler runs. Locking around the topology mutation
     would prevent interleaving, but if DeadNodeHandler acquires the lock 
first, it still removes a node that has already resurrected. The lock ensures 
atomicity of the topology operation, but the operation
     itself is wrong because the premise (node is dead) is no longer true. 
You'd still need the state check.
  4. Locking is the wrong abstraction here: The problem isn't concurrent 
mutation of a shared data structure (which is what locks solve). The problem is 
acting on outdated information. The correct fix is to
     re-validate the precondition (is the node still dead?) before acting. 
That's an idempotency/guard pattern, not a mutual exclusion pattern.
  5. `NetworkTopologyImpl` already has internal synchronization: The topology's 
add() and remove() methods are themselves thread-safe. The race isn't about 
corrupting the topology's internal state — it's about
     the logical outcome being wrong (a live node absent from the topology). 
Wrapping already-synchronized calls in another lock doesn't fix a logic error.

  In summary: locking solves data structure corruption from concurrent access, 
but the bug here is a semantic race — acting on a stale event. The state-check 
guard directly addresses the root cause, is simpler,
  has zero performance overhead, and doesn't require managing lock 
infrastructure across the handler ecosystem.

  was:
We recently found Datanode lifecycle race condition like HDDS-14834 which 
causes node inconsistency between SCM NodeStateManager and NetworkTopology. 
HDDS-14834 reduces the window to make race condition highly improbable, but it 
might theoretically happen. 

!image-2026-04-01-10-55-37-370.png|width=738,height=219!

Practical risk is near-zero but not zero:
  • The window is a handful of JVM bytecode instructions (nanoseconds of CPU 
time).
  • For the race to manifest, the HealthyReadOnlyNodeHandler must fully 
complete (including closePipeline() I/O for every pipeline on that node) within 
that window.
  • The JVM can preempt a thread at any point for an arbitrary duration (GC 
pause, OS scheduling), so theoretically the DeadNodeHandler thread could be 
paused after the state check long enough for the entire
    pipeline to play out.

We need to revisit the SCM datanode management concurrency model to prevent 
similar race conditions. We also need to revisit the whole SCM event framework.

Possible approach
 # A lock that makes the state-check + topology-remove atomic.
 # A double-check: re-read the state after nt.remove() and call nt.add() back 
if the node is no longer DEAD.
 # A version/epoch on the topology mutation so stale removes are rejected.


> Revisit SCM NodeManager concurrency
> -----------------------------------
>
>                 Key: HDDS-14950
>                 URL: https://issues.apache.org/jira/browse/HDDS-14950
>             Project: Apache Ozone
>          Issue Type: Improvement
>            Reporter: Ivan Andika
>            Assignee: Ivan Andika
>            Priority: Major
>         Attachments: image-2026-04-01-10-55-37-370.png
>
>
> We recently found Datanode lifecycle race condition like HDDS-14834 which 
> causes node inconsistency between SCM NodeStateManager and NetworkTopology. 
> HDDS-14834 reduces the window to make race condition highly improbable, but 
> it might theoretically happen. 
> !image-2026-04-01-10-55-37-370.png|width=738,height=219!
> Practical risk is near-zero but not zero:
>   • The window is a handful of JVM bytecode instructions (nanoseconds of CPU 
> time).
>   • For the race to manifest, the HealthyReadOnlyNodeHandler must fully 
> complete (including closePipeline() I/O for every pipeline on that node) 
> within that window.
>   • The JVM can preempt a thread at any point for an arbitrary duration (GC 
> pause, OS scheduling), so theoretically the DeadNodeHandler thread could be 
> paused after the state check long enough for the entire
>     pipeline to play out.
> We need to revisit the SCM datanode management concurrency model to prevent 
> similar race conditions. We also need to revisit the whole SCM event 
> framework.
> Possible approach
>  # A lock that makes the state-check + topology-remove atomic.
>  # A double-check: re-read the state after nt.remove() and call nt.add() back 
> if the node is no longer DEAD.
>  # A version/epoch on the topology mutation so stale removes are rejected.
>  
> However locking implementation might be involved
> There are several reasons the state-check guard approach was preferred over 
> merging both handlers onto a single shared executor:
>   1. Event framework architecture: Ozone's EventQueue assigns each event type 
> its own SingleThreadExecutor. Forcing two different event types (DEAD_NODE 
> and HEALTHY_READONLY_NODE) onto a shared executor would
>      require changes to the event framework infrastructure itself — a much 
> more invasive change.
>   2. Throughput and isolation: DeadNodeHandler.onMessage() does heavy work — 
> closing containers, destroying pipelines, removing replicas. Serializing it 
> with HealthyReadOnlyNodeHandler means a slow dead-node
>      cleanup would block healthy-readonly transitions from being processed, 
> degrading cluster responsiveness during node churn.
>   3. Doesn't eliminate stale events: Even with a shared executor, the 
> fundamental problem remains — the DEAD_NODE event is stale by the time it's 
> dequeued. The node may have already transitioned back to
>      HEALTHY_READONLY via a heartbeat. A shared executor only guarantees 
> ordering within the queue, but it can't prevent a DEAD_NODE event (enqueued 
> earlier) from executing after the node has legitimately
>      resurrected. You'd still need the state-check guard.
>   4. Other topology-touching handlers exist: It's not just these two. 
> HealthyNodeHandler, StaleNodeHandler, and others also interact with 
> NetworkTopology. A shared executor for only two of them wouldn't fully
>      protect against races with the rest, yet pulling all of them onto one 
> executor would create a massive serialization bottleneck.
>   5. Minimal, targeted fix: The state-check in DeadNodeHandler + 
> unconditional add in HealthyReadOnlyNodeHandler is a two-line-level change 
> that's correct regardless of executor topology. It follows the general
>      principle that event handlers in an async system should always validate 
> current state before acting on a potentially stale even
>  
> Here's why locking was not the preferred approach:
>   1. Coarse-grained lock = bottleneck: If you put a lock around the 
> NetworkTopology operations in both handlers, you're serializing all topology 
> mutations. This has the same throughput problem as the shared
>      executor — DeadNodeHandler's heavy work (container close, pipeline 
> destroy) would hold the lock and block HealthyReadOnlyNodeHandler from making 
> progress.
>   2. Fine-grained lock = complexity and deadlock risk: A per-node lock would 
> avoid the global bottleneck, but introduces significant complexity. You'd 
> need to manage a lock map keyed by datanode ID, handle lock
>      lifecycle (creation, cleanup, eviction), and carefully reason about lock 
> ordering across all the handlers that touch topology (DeadNodeHandler, 
> HealthyReadOnlyNodeHandler, HealthyNodeHandler,
>      StaleNodeHandler, etc.) to avoid deadlocks.
>   3. Still doesn't address the root cause: The fundamental issue is that 
> DeadNodeHandler acts on a stale event — the node is no longer dead by the 
> time the handler runs. Locking around the topology mutation
>      would prevent interleaving, but if DeadNodeHandler acquires the lock 
> first, it still removes a node that has already resurrected. The lock ensures 
> atomicity of the topology operation, but the operation
>      itself is wrong because the premise (node is dead) is no longer true. 
> You'd still need the state check.
>   4. Locking is the wrong abstraction here: The problem isn't concurrent 
> mutation of a shared data structure (which is what locks solve). The problem 
> is acting on outdated information. The correct fix is to
>      re-validate the precondition (is the node still dead?) before acting. 
> That's an idempotency/guard pattern, not a mutual exclusion pattern.
>   5. `NetworkTopologyImpl` already has internal synchronization: The 
> topology's add() and remove() methods are themselves thread-safe. The race 
> isn't about corrupting the topology's internal state — it's about
>      the logical outcome being wrong (a live node absent from the topology). 
> Wrapping already-synchronized calls in another lock doesn't fix a logic error.
>   In summary: locking solves data structure corruption from concurrent 
> access, but the bug here is a semantic race — acting on a stale event. The 
> state-check guard directly addresses the root cause, is simpler,
>   has zero performance overhead, and doesn't require managing lock 
> infrastructure across the handler ecosystem.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to