- **status**: assigned --> review
---
** [tickets:#2949] amf: standby amfd keep crash and node reboot cyclically**
**Status:** review
**Milestone:** 5.18.12
**Created:** Tue Oct 30, 2018 06:20 AM UTC by Thuan
**Last Updated:** Tue Oct 30, 2018 06:20 AM UTC
**Owner:** Thuan
Standby AMFD crash because:
~~~
Oct 26 19:30:53 sc-2 osafamfd[3555]: WA avd_node_get FAILED for
'safAmfNode=PL-13,safAmfCluster=myAmfCluster'
Oct 26 19:30:53 sc-2 osafamfd[3555]: src/amf/amfd/ckpt_dec.cc:2200:
dec_cs_node_config: Assertion 'status == NCSCC_RC_SUCCESS' failed.
~~~
In scenario that user delete node (scale-in). Then Standby SC just boot up.
But Active AMFD miss node CCB delete apply callback because AMFD got IMM bad
handle right before.
So, Active AMFD still not update node db and checkpoint to standby SC still
have the node which not exist anymore in IMM database. It make Standby AMFD
stuck in crash and reboot loop.
~~~
<143>1 2018-10-26T19:29:14.68+02:00 SC-1 osafamfd 12883 osafamfd [meta
sequenceId="500306"] 12883:amf/amfd/node.cc:532 >>
node_ccb_completed_delete_hdlr: 'safAmfNode=PL-13,safAmfCluster=myAmfCluster'
<143>1 2018-10-26T19:29:14.680268+02:00 SC-1 osafamfd 12883 osafamfd [meta
sequenceId="500313"] 12883:amf/amfd/node.cc:664 <<
node_ccb_completed_delete_hdlr
<143>1 2018-10-26T19:29:21.747664+02:00 SC-1 osafamfd 12883 osafamfd [meta
sequenceId="528336"] 12883:amf/amfd/imm.cc:240 >> exec: Update
'safSi=NoRed3,safApp=OpenSAF' saAmfUnassignedAlarmStatus
<143>1 2018-10-26T19:29:21.747668+02:00 SC-1 osafamfd 12883 osafamfd [meta
sequenceId="528337"] 12883:amf/amfd/imm.cc:750 >> object_name_to_class_type:
safSi=NoRed3,safApp=OpenSAF
<143>1 2018-10-26T19:29:21.747678+02:00 SC-1 osafamfd 12883 osafamfd [meta
sequenceId="528338"] 12883:amf/amfd/imm.cc:798 << object_name_to_class_type: 24
<143>1 2018-10-26T19:29:21.747683+02:00 SC-1 osafamfd 12883 osafamfd [meta
sequenceId="528339"] 12883:imm/agent/imma_oi_api.cc:2556 >>
rt_object_update_common
<143>1 2018-10-26T19:29:28.621939+02:00 SC-1 osafamfd 12883 osafamfd [meta
sequenceId="528340"] 12885:amf/amfd/timer.cc:152 >> avd_tmr_exp
<143>1 2018-10-26T19:29:28.621967+02:00 SC-1 osafamfd 12883 osafamfd [meta
sequenceId="528341"] 12885:amf/amfd/timer.cc:175 << avd_tmr_exp
<143>1 2018-10-26T19:29:31.757918+02:00 SC-1 osafamfd 12883 osafamfd [meta
sequenceId="528342"] 12883:imm/agent/imma_oi_api.cc:2849 <<
rt_object_update_common
<143>1 2018-10-26T19:29:31.757932+02:00 SC-1 osafamfd 12883 osafamfd [meta
sequenceId="528343"] 12883:amf/amfd/imm.cc:263 TR TIMEOUT
<143>1 2018-10-26T19:29:31.75794+02:00 SC-1 osafamfd 12883 osafamfd [meta
sequenceId="528344"] 12883:amf/amfd/imm.cc:275 << exec
<143>1 2018-10-26T19:29:31.758198+02:00 SC-1 osafamfd 12883 osafamfd [meta
sequenceId="528361"] 12883:amf/amfd/imm.cc:240 >> exec: Update
'safSi=NoRed3,safApp=OpenSAF' saAmfUnassignedAlarmStatus
<143>1 2018-10-26T19:29:31.758205+02:00 SC-1 osafamfd 12883 osafamfd [meta
sequenceId="528362"] 12883:amf/amfd/imm.cc:750 >> object_name_to_class_type:
safSi=NoRed3,safApp=OpenSAF
<143>1 2018-10-26T19:29:31.758214+02:00 SC-1 osafamfd 12883 osafamfd [meta
sequenceId="528363"] 12883:amf/amfd/imm.cc:798 << object_name_to_class_type: 24
<143>1 2018-10-26T19:29:31.758221+02:00 SC-1 osafamfd 12883 osafamfd [meta
sequenceId="528364"] 12883:imm/agent/imma_oi_api.cc:2556 >>
rt_object_update_common
<143>1 2018-10-26T19:29:31.758898+02:00 SC-1 osafamfd 12883 osafamfd [meta
sequenceId="528365"] 12883:imm/agent/imma_oi_api.cc:2849 <<
rt_object_update_common
<143>1 2018-10-26T19:29:31.758918+02:00 SC-1 osafamfd 12883 osafamfd [meta
sequenceId="528366"] 12883:amf/amfd/imm.cc:266 TR BADHANDLE
<143>1 2018-10-26T19:29:31.758924+02:00 SC-1 osafamfd 12883 osafamfd [meta
sequenceId="528367"] 12883:amf/amfd/imm.cc:2194 >> avd_imm_reinit_bg
<141>1 2018-10-26T19:29:31.758944+02:00 SC-1 osafamfd 12883 osafamfd [meta
sequenceId="528368"] 12883:amf/amfd/imm.cc:2207 NO Re-initializing with IMM
<143>1 2018-10-26T19:29:31.758997+02:00 SC-1 osafamfd 12883 osafamfd [meta
sequenceId="528369"] 6394:amf/amfd/imm.cc:2090 >> avd_imm_reinit_bg_thread
<143>1 2018-10-26T19:29:31.759021+02:00 SC-1 osafamfd 12883 osafamfd [meta
sequenceId="528370"] 6394:imm/agent/imma_oi_api.cc:701 >> saImmOiFinalize
<143>1 2018-10-26T19:29:31.7591+02:00 SC-1 osafamfd 12883 osafamfd [meta
sequenceId="528371"] 12883:amf/amfd/imm.cc:2227 << avd_imm_reinit_bg
<143>1 2018-10-26T19:29:31.759112+02:00 SC-1 osafamfd 12883 osafamfd [meta
sequenceId="528372"] 12883:amf/amfd/imm.cc:275 << exec
<143>1 2018-10-26T19:29:31.759118+02:00 SC-1 osafamfd 12883 osafamfd [meta
sequenceId="528373"] 12883:amf/amfd/imm.cc:426 << execute: 2
<143>1 2018-10-26T19:29:31.759211+02:00 SC-1 osafamfd 12883 osafamfd [meta
sequenceId="528374"] 6394:imm/agent/imma_proc.cc:150 >>
imma_callback_ipc_destroy
<143>1 2018-10-26T19:29:31.759235+02:00 SC-1 osafamfd 12883 osafamfd [meta
sequenceId="528375"] 6394:imm/agent/imma_proc.cc:0 << imma_callback_ipc_destroy
<143>1 2018-10-26T19:29:31.759243+02:00 SC-1 osafamfd 12883 osafamfd [meta
sequenceId="528376"] 6394:imm/agent/imma_proc.cc:208 TR Deleting client node
<143>1 2018-10-26T19:29:31.759248+02:00 SC-1 osafamfd 12883 osafamfd [meta
sequenceId="528377"] 6394:imm/agent/imma_db.cc:281 >> imma_oi_ccb_record_delete
<143>1 2018-10-26T19:29:31.759252+02:00 SC-1 osafamfd 12883 osafamfd [meta
sequenceId="528378"] 6394:imm/agent/imma_db.cc:293 T3 Removing
imma_oi_ccb_record ccb:0x6af handle:1700027a0f client:0x5631d5f1ae80 in
CRITICAL state
<143>1 2018-10-26T19:29:31.759257+02:00 SC-1 osafamfd 12883 osafamfd [meta
sequenceId="528379"] 6394:imm/agent/imma_db.cc:319 << imma_oi_ccb_record_delete
~~~
---
Sent from sourceforge.net because opensaf-tickets@lists.sourceforge.net is
subscribed to https://sourceforge.net/p/opensaf/tickets/
To unsubscribe from further messages, a project admin can change settings at
https://sourceforge.net/p/opensaf/admin/tickets/options. Or, if this is a
mailing list, you can unsubscribe from the mailing list.
_______________________________________________
Opensaf-tickets mailing list
Opensaf-tickets@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/opensaf-tickets
