- **status**: review --> fixed
- **Comment**:
commit b3c88ac2b51edd2cb54b45876c856f7cc80b40ca (HEAD -> develop,
origin/develop, ticket-3156)
Author: thuan.tran <[email protected]>
Date: Mon Feb 17 14:38:49 2020 +0700
clmd: retry once to send message to clmna [#3156]
- If a node reboot up, clmna (on that node) svc_up is not yet come
but clmd get message join request then send message to clmna failed.
It leads to amfnd (on that node) timeout init clm agent and delay
to send node up msg. This may cause amfd order reboot that node if
osafAmfDelayNodeFailoverNodeWaitTimeout is set smaller than total
time amfnd (on that node) retry until init clm agent successfully.
- One retry to send messsage to clmna help avoid this scenario.
---
** [tickets:#3156] clmd: fail to send message to clmna**
**Status:** fixed
**Milestone:** 5.20.05
**Created:** Tue Feb 18, 2020 06:20 AM UTC by Thuan Tran
**Last Updated:** Tue Feb 18, 2020 06:40 AM UTC
**Owner:** Thuan Tran
CLMD fail to send message to reply request join from clmna
because CLMD still not get svc_up for clmna.
It leads to AMFND stuck in retry loop to init clm agent
and don't send node up after 5s (> node up delay timeout set 5s)
Then AMFD send order to reboot it.
~~~
<143>1 2020-02-14T10:45:00.677961+01:00 SC-2 osafclmd 256 osafclmd [meta
sequenceId="11034"] 256:clm/clmd/clms_evt.cc:540 >> proc_node_up_msg: Node up
mesg for nodename length 4 PL-4
<139>1 2020-02-14T10:45:00.678134+01:00 SC-2 osafclmd 256 mds.log [meta
sequenceId="270"] MDS_SND_RCV:No Route Found from svc_id = CLMS(34), to svc_id
= CLMNA(36) on Adest = <0x0002040f, 2438112658>
<139>1 2020-02-14T10:45:00.678161+01:00 SC-2 osafclmd 256 mds.log [meta
sequenceId="271"] MDS_SND_RCV: Normal send Message sent Failed from svc_id =
CLMS(34), to svc_id = CLMNA(36)
<139>1 2020-02-14T10:45:00.678186+01:00 SC-2 osafclmd 256 mds.log [meta
sequenceId="272"] MDS_SND_RCV: Adest=<0x0002040f,2438112658>
<141>1 2020-02-14T10:45:00.678566+01:00 SC-2 osafclmd 256 mds.log [meta
sequenceId="273"] MDTM: svc up event for svc_id = CLMNA(36), subscri. by svc_id
= CLMS(34) pwe_id=1 Adest = <rem_node[0x2040f]:dest_tipc_id_ref[2438112658]>
<141>1 2020-02-14T10:45:00.679018+01:00 SC-2 osafclmd 256 osafclmd [meta
sequenceId="11066"] 256:clm/clmd/clms_evt.cc:643 NO proc_node_up_msg: send
failed. dest:2040f9152a592
<143>1 2020-02-14T10:45:00.679025+01:00 SC-2 osafclmd 256 osafclmd [meta
sequenceId="11067"] 256:clm/clmd/clms_evt.cc:712 << proc_node_up_msg
<143>1 2020-02-14T10:45:02.091059+01:00 SC-2 osafclmd 256 osafclmd [meta
sequenceId="12717"] 256:clm/clmd/clms_evt.cc:1534 >> proc_initialize_msg: dest
2040ffc127366
<143>1 2020-02-14T10:45:02.091069+01:00 SC-2 osafclmd 256 osafclmd [meta
sequenceId="12718"] 256:clm/clmd/clms_util.cc:118 >> clms_node_get_by_id
<143>1 2020-02-14T10:45:02.091077+01:00 SC-2 osafclmd 256 osafclmd [meta
sequenceId="12719"] 256:clm/clmd/clms_util.cc:131 << clms_node_get_by_id
<143>1 2020-02-14T10:45:02.091084+01:00 SC-2 osafclmd 256 osafclmd [meta
sequenceId="12720"] 256:clm/clmd/clms_evt.cc:1537 TR Node id = 132111
<143>1 2020-02-14T10:45:02.091093+01:00 SC-2 osafclmd 256 osafclmd [meta
sequenceId="12721"] 256:clm/clmd/clms_evt.cc:0 << proc_initialize_msg
2020-02-14 10:45:07.187 SC-2 osafamfd[267]: NO Node failover timeout
2020-02-14 10:45:07.187 SC-2 osafamfd[267]: WA Failed node 'PL-4' has
reappeared after network separation
2020-02-14 10:45:07.187 SC-2 osafamfd[267]: WA Sending node reboot order
<143>1 2020-02-14T10:45:12.09107+01:00 PL-4 osafamfnd 203 osafamfnd [meta
sequenceId="148"] 203:clm/agent/clma_mds.cc:1245 TR clma_mds_msg_sync_send
FAILED
<143>1 2020-02-14T10:45:12.091083+01:00 PL-4 osafamfnd 203 osafamfnd [meta
sequenceId="149"] 203:clm/agent/clma_mds.cc:1247 << clma_mds_msg_sync_send
<143>1 2020-02-14T10:45:12.091092+01:00 PL-4 osafamfnd 203 osafamfnd [meta
sequenceId="150"] 203:clm/agent/clma_api.cc:661 TR clma_mds_msg_sync_send
FAILED: 5
<143>1 2020-02-14T10:45:12.0911+01:00 PL-4 osafamfnd 203 osafamfnd [meta
sequenceId="151"] 203:clm/agent/clma_api.cc:699 T2 CLMA INIT FAILED
2020-02-14 10:45:12.092 PL-4 osafamfnd[203]: WA saClmInitialize_4 returned 5
~~~
Later after a timeout, clmna send another request join and clmd able to reply
to clmna.
Then AMFND retry init clm agent successfully and send node up to AMFD but AMFD
already order this node reboot.
~~~
2020-02-14 10:45:30.687 PL-4 osafclmna[180]: NO
safNode=PL-4,safCluster=myClmCluster Joined cluster, nodeid=2040f
2020-02-14 10:45:32.502 PL-4 osafamfnd[203]: NO Sending node up due to NCSMDS_UP
~~~
---
Sent from sourceforge.net because [email protected] is
subscribed to https://sourceforge.net/p/opensaf/tickets/
To unsubscribe from further messages, a project admin can change settings at
https://sourceforge.net/p/opensaf/admin/tickets/options. Or, if this is a
mailing list, you can unsubscribe from the mailing list._______________________________________________
Opensaf-tickets mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/opensaf-tickets
