commit 458d58cb1d953bae10b37534fe38671b03be0b7c (HEAD -> develop,
origin/develop, ticket-3399)
Author: dat.phan <[email protected]>
Date: Thu Jun 4 07:36:06 2026 +0000
ntf: fix unhandled return code when logging notification [#3399]
When failover happens, new active ntfd must process all queued
notifications. During this failover window, the CLM membership
state on new active ntfd can transiently appear as non-member
to the LOG agent, causing the log client to be marked stale and
return SA_AIS_ERR_UNAVAILABLE when ntfd tries processing
notification. The return code is not handled and gets abort().
Fix by handling SA_AIS_ERR_UNAVAILABLE, requeue the notifcation and
try again.
commit 5ddab129203be7c36e810453650a389cd92e4ea2
Author: dat.phan <[email protected]>
Date: Thu Jun 4 07:27:11 2026 +0000
ntf: add test to verify no deadlock on agent during failover [#3399]
commit 683a39c246a18e6684ed6104813f7a819e14f406
Author: dat.phan <[email protected]>
Date: Thu Jun 4 06:57:34 2026 +0000
ntf: fix deadlock on ntf agent during failover [#3399]
recoverClient() holds cb_lock and calls ncsmds_api() which tries to
acquire gl_mds_library_mutex. Meanwhile, the MDS thread holds
gl_mds_library_mutex and its callback ntfa_mds_rcv() tries to acquire
cb_lock. This creates a circular lock dependency that deadlocks the
NTF agent.
Fix by releasing cb_lock before MDS calls in recoverClient(). A
'recovering' flag is set under cb_lock to prevent concurrent recovery
of the same handle (returns SA_AIS_ERR_TRY_AGAIN). When recovering
associated reader_hdl and subscriber_hdl, reacquire cb_lock during
traverse handle-list to prevent use-after-free.
Addationally, saNtfDispatch() will now properly return the error code
from recoverClient() instead of silently continuing on non-UNAVAILABLE
failures.
Note: cb_lock must never be held when calling MDS APIs (ncsmds_api,
ntfa_mds_msg_sync_send) since MDS callbacks acquire cb_lock under
gl_mds_library_mutex, creating an inherent lock-ordering constraint.
---
**[tickets:#3399] ntf: deadlock on ntf agent during failover**
**Status:** assigned
**Milestone:** 5.27.01
**Created:** Mon Jun 01, 2026 09:14 AM UTC by PhanTranQuocDat
**Last Updated:** Mon Jun 01, 2026 09:14 AM UTC
**Owner:** PhanTranQuocDat
**Attachments:**
-
[ntf\_deadlock.bt](https://sourceforge.net/p/opensaf/tickets/3399/attachment/ntf_deadlock.bt)
(22.7 kB; application/octet-stream)
When a NTF agent program invokes NTF APIs concurrently with failover, deadlock
can happen.
The root cause is MDS thread invokes NTF agent callback during failover, which
requires to obtain both MDS lock and NTF agent lock. Concurrently, an API
calling thread obtains the NTF agent lock first and require obtaining MDS lock
to send message. The encounter rate of this issue increases with the number of
threads invoking the NTF API at the time the failover occurs.
---
Sent from sourceforge.net because [email protected] is
subscribed to https://sourceforge.net/p/opensaf/tickets/
To unsubscribe from further messages, a project admin can change settings at
https://sourceforge.net/p/opensaf/admin/tickets/options. Or, if this is a
mailing list, you can unsubscribe from the mailing list._______________________________________________
Opensaf-tickets mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/opensaf-tickets
