HI, what is the configured TIPC link tolerance time? Depending on the tolerance time, the other node will get service down.
/Neel. // On Tuesday 04 March 2014 08:53 PM, Tony Hart wrote: > We’re seeing a problem where there is a loss of connectivity between a > payload (cmm02B) and the controller (the connectivity returns but is away > just long enough to trigger a TIPC timeout) in this case the payload is > dropped from the cluster but the payload doesn’t restart. The payload is > flagged as not being in the cluster and its presence state is UNINSTANTIATED. > Its still running the osaf processes though. > > Is this something that’s been fixed in the current release (we’re running > 4.3.1) > > $ immlist safNode=cmm02b,safCluster=myClmCluster > Name Type Value(s) > ======================================================================== > safNode SA_STRING_T safNode=cmm02b > saClmNodeLockCallbackTimeout SA_TIME_T 50000000000 > (0xba43b7400, Thu Jan 1 00:00:50 1970) > saClmNodeIsMember SA_UINT32_T 0 (0x0) > saClmNodeInitialViewNumber SA_UINT64_T 28 (0x1c) > saClmNodeID SA_UINT32_T 73743 > (0x1200f) > saClmNodeEE SA_NAME_T <Empty> > saClmNodeDisableReboot SA_UINT32_T 0 (0x0) > saClmNodeCurrAddressFamily SA_UINT32_T <Empty> > saClmNodeCurrAddress SA_STRING_T <Empty> > saClmNodeBootTimeStamp SA_TIME_T > 1393879646000000000 (0x13580e27277a2c00, Mon Mar 3 20:47:26 2014) > saClmNodeAdminState SA_UINT32_T 1 (0x1) > saClmNodeAddressFamily SA_UINT32_T <Empty> > saClmNodeAddress SA_STRING_T <Empty> > SaImmAttrImplementerName SA_STRING_T safClmService > SaImmAttrClassName SA_STRING_T SaClmNode > SaImmAttrAdminOwnerName SA_STRING_T IMMLOADER > > > $ immlist $(amf-find node | grep CMM02B) > Name Type Value(s) > ======================================================================== > safAmfNode SA_STRING_T > safAmfNode=CMM02B > saAmfNodeSuFailoverMax SA_UINT32_T 2 (0x2) > saAmfNodeSuFailOverProb SA_TIME_T 1200000000000 > (0x1176592e000, Thu Jan 1 00:20:00 1970) > saAmfNodeOperState SA_UINT32_T 2 (0x2) > saAmfNodeFailfastOnTerminationFailure SA_UINT32_T 0 (0x0) > saAmfNodeFailfastOnInstantiationFailure SA_UINT32_T 0 (0x0) > saAmfNodeClmNode SA_NAME_T > safNode=cmm02b,safCluster=myClmCluster (38) > saAmfNodeCapacity SA_STRING_T <Empty> > saAmfNodeAutoRepair SA_UINT32_T 1 (0x1) > saAmfNodeAdminState SA_UINT32_T 1 (0x1) > SaImmAttrImplementerName SA_STRING_T safAmfService > SaImmAttrClassName SA_STRING_T SaAmfNode > SaImmAttrAdminOwnerName SA_STRING_T IMMLOADER > > > cmm02b$ ps aux | grep osaf > root 1417 0.0 0.0 225880 2028 ? Ssl Mar03 0:08 > /usr/lib64/opensaf/osafamfnd osafamfnd > root 1429 0.0 0.0 157100 1416 ? Ssl Mar03 0:00 > /usr/lib64/opensaf/osafsmfnd osafsmfnd > opensaf 1438 0.0 0.1 174256 5764 ? Ssl Mar03 0:00 > /usr/lib64/opensaf/osafmsgnd osafmsgnd > opensaf 1454 0.0 0.0 155732 1448 ? Ssl Mar03 0:00 > /usr/lib64/opensaf/osaflcknd osaflcknd > opensaf 1463 0.0 0.0 158148 2296 ? Ssl Mar03 0:00 > /usr/lib64/opensaf/osafckptnd osafckptnd > opensaf 1472 0.0 0.0 155020 1392 ? Ssl Mar03 0:02 > /usr/lib64/opensaf/osafamfwd osafamfwd > opensaf 4704 0.0 0.3 182240 11992 ? Ssl 14:20 0:01 > /usr/lib64/opensaf/osafimmnd osafimmnd > > > > SCM1 (1.1.15) > ------------------- > 2014-03-04T14:20:18.808187+00:00 scm1 osafamfd[1771]: NO Node 'PLD0211' left > the cluster > 2014-03-04T14:20:18.851318+00:00 scm1 kernel: TIPC: Established link > <1.1.15:eth2-1.1.27:bond0> on network plane A > 2014-03-04T14:20:18.852749+00:00 scm1 osafsmfd[1965]: ER saClmClusterNodeGet > failed, rc=SA_AIS_ERR_NOT_EXIST (12) > 2014-03-04T14:20:18.858472+00:00 scm1 kernel: TIPC: Established link > <1.1.15:eth2-1.1.23:bond0> on network plane A > 2014-03-04T14:20:18.871084+00:00 scm1 osafsmfd[1965]: ER saClmClusterNodeGet > failed, rc=SA_AIS_ERR_NOT_EXIST (12) > 2014-03-04T14:20:18.956307+00:00 scm1 kernel: TIPC: Resetting link > <1.1.15:eth2-1.1.32:eth2>, peer not responding > 2014-03-04T14:20:18.956330+00:00 scm1 kernel: TIPC: Lost link > <1.1.15:eth2-1.1.32:eth2> on network plane A > 2014-03-04T14:20:18.956335+00:00 scm1 kernel: TIPC: Lost contact with <1.1.32> > 2014-03-04T14:20:18.956340+00:00 scm1 kernel: TIPC: Established link > <1.1.15:eth2-1.1.32:eth2> on network plane A > 2014-03-04T14:20:18.958227+00:00 scm1 osafimmnd[1667]: NO Global discard node > received for nodeId:1200f pid:1347 > 2014-03-04T14:20:18.958270+00:00 scm1 osafimmnd[1667]: NO Implementer > disconnected 51 <0, 1200f(down)> (MsgQueueService73743) > 2014-03-04T14:20:18.965240+00:00 scm1 osafimmnd[1667]: NO Implementer > connected: 71 (MsgQueueService73743) <92377, 10f0f> > 2014-03-04T14:20:18.968251+00:00 scm1 osafimmnd[1667]: NO Implementer locally > disconnected. Marking it as doomed 71 <92377, 10f0f> (MsgQueueService73743) > 2014-03-04T14:20:18.971785+00:00 scm1 osafimmnd[1667]: NO Global discard node > received for nodeId:1170f pid:0 > 2014-03-04T14:20:18.973013+00:00 scm1 osafimmnd[1667]: NO Global discard node > received for nodeId:1200f pid:0 > 2014-03-04T14:20:18.976586+00:00 scm1 osafimmnd[1667]: NO Implementer > disconnected 71 <92377, 10f0f> (MsgQueueService73743) > 2014-03-04T14:20:19.025760+00:00 scm1 osafimmd[1657]: NO Node 11e0f request > sync sync-pid:23769 epoch:0 > 2014-03-04T14:20:19.076427+00:00 scm1 osafamfd[1771]: NO Node 'PLD0214' left > the cluster > 2014-03-04T14:20:19.215220+00:00 scm1 osafimmd[1657]: NO Node 11c0f request > sync sync-pid:23629 epoch:0 > 2014-03-04T14:20:19.296817+00:00 scm1 osafamfd[1771]: WA avd_msg_sanity_chk: > invalid node ID (11e0f) > 2014-03-04T14:20:19.300899+00:00 scm1 osafamfd[1771]: WA avd_msg_sanity_chk: > invalid node ID (11e0f) > 2014-03-04T14:20:19.305377+00:00 scm1 osafamfd[1771]: NO Node 'CMM02B' left > the cluster > 2014-03-04T14:20:19.357458+00:00 scm1 osafimmd[1657]: NO Node 1200f request > sync sync-pid:4704 epoch:0 > > > cmm02B (1.1.32) > ----------------------- > 2014-03-04T14:20:18.495174+00:00 cmm02b kernel: TIPC: Resetting link > <1.1.32:eth2-1.1.10:bond0>, peer not responding > 2014-03-04T14:20:18.495203+00:00 cmm02b kernel: TIPC: Lost link > <1.1.32:eth2-1.1.10:bond0> on network plane A > 2014-03-04T14:20:18.495209+00:00 cmm02b kernel: TIPC: Lost contact with > <1.1.10> > 2014-03-04T14:20:18.501981+00:00 cmm02b kernel: TIPC: Resetting link > <1.1.32:eth2-1.1.15:eth2>, peer not responding > 2014-03-04T14:20:18.502012+00:00 cmm02b kernel: TIPC: Lost link > <1.1.32:eth2-1.1.15:eth2> on network plane A > 2014-03-04T14:20:18.502016+00:00 cmm02b kernel: TIPC: Lost contact with > <1.1.15> > 2014-03-04T14:20:18.502020+00:00 cmm02b kernel: TIPC: Resetting link > <1.1.32:eth2-1.1.11:bond0>, peer not responding > 2014-03-04T14:20:18.502023+00:00 cmm02b kernel: TIPC: Lost link > <1.1.32:eth2-1.1.11:bond0> on network plane A > 2014-03-04T14:20:18.502026+00:00 cmm02b kernel: TIPC: Lost contact with > <1.1.11> > 2014-03-04T14:20:18.502110+00:00 cmm02b kernel: TIPC: Resetting link > <1.1.32:eth2-1.1.1:bond0>, peer not responding > 2014-03-04T14:20:18.502115+00:00 cmm02b kernel: TIPC: Lost link > <1.1.32:eth2-1.1.1:bond0> on network plane A > 2014-03-04T14:20:18.502118+00:00 cmm02b kernel: TIPC: Lost contact with > <1.1.1> > 2014-03-04T14:20:18.549154+00:00 cmm02b kernel: TIPC: Resetting link > <1.1.32:eth2-1.1.14:bond0>, peer not responding > 2014-03-04T14:20:18.549180+00:00 cmm02b kernel: TIPC: Lost link > <1.1.32:eth2-1.1.14:bond0> on network plane A > 2014-03-04T14:20:18.549184+00:00 cmm02b kernel: TIPC: Lost contact with > <1.1.14> > 2014-03-04T14:20:18.671107+00:00 cmm02b kernel: TIPC: Established link > <1.1.32:eth2-1.1.14:bond0> on network plane A > 2014-03-04T14:20:18.743482+00:00 cmm02b kernel: TIPC: Established link > <1.1.32:eth2-1.1.11:bond0> on network plane A > 2014-03-04T14:20:18.866277+00:00 cmm02b kernel: TIPC: Established link > <1.1.32:eth2-1.1.10:bond0> on network plane A > 2014-03-04T14:20:18.869280+00:00 cmm02b kernel: TIPC: Established link > <1.1.32:eth2-1.1.1:bond0> on network plane A > 2014-03-04T14:20:18.954740+00:00 cmm02b kernel: TIPC: Established link > <1.1.32:eth2-1.1.15:eth2> on network plane A > 2014-03-04T14:20:18.959226+00:00 cmm02b osafimmnd[1347]: WA MESSAGE:38632 OUT > OF ORDER my highest processed:38600, exiting > 2014-03-04T14:20:18.967269+00:00 cmm02b osafamfnd[1417]: NO > 'safComp=IMMND,safSu=CMM02B,safSg=NoRed,safApp=OpenSAF' faulted due to > 'avaDown' : Recovery is 'componentRestart' > 2014-03-04T14:20:19.052569+00:00 cmm02b osafimmnd[4704]: Started > 2014-03-04T14:20:19.157393+00:00 cmm02b osafimmnd[4704]: NO SERVER STATE: > IMM_SERVER_ANONYMOUS --> IMM_SERVER_CLUSTER_WAITING > 2014-03-04T14:20:19.257835+00:00 cmm02b osafimmnd[4704]: NO SERVER STATE: > IMM_SERVER_CLUSTER_WAITING --> IMM_SERVER_LOADING_PENDING > 2014-03-04T14:20:19.358134+00:00 cmm02b osafimmnd[4704]: NO SERVER STATE: > IMM_SERVER_LOADING_PENDING --> IMM_SERVER_SYNC_PENDING > 2014-03-04T14:20:19.358452+00:00 cmm02b osafimmnd[4704]: NO NODE STATE-> > IMM_NODE_ISOLATED > 2014-03-04T14:20:19.955686+00:00 cmm02b osafimmnd[4704]: NO NODE STATE-> > IMM_NODE_W_AVAILABLE > 2014-03-04T14:20:20.022473+00:00 cmm02b osafimmnd[4704]: NO SERVER STATE: > IMM_SERVER_SYNC_PENDING --> IMM_SERVER_SYNC_CLIENT > 2014-03-04T14:20:26.925158+00:00 cmm02b kernel: TIPC: Resetting link > <1.1.32:eth2-1.1.31:eth2>, peer not responding > 2014-03-04T14:20:26.925184+00:00 cmm02b kernel: TIPC: Lost link > <1.1.32:eth2-1.1.31:eth2> on network plane A > 2014-03-04T14:20:26.925191+00:00 cmm02b kernel: TIPC: Lost contact with > <1.1.31> > 2014-03-04T14:20:27.893115+00:00 cmm02b kernel: TIPC: Resetting link > <1.1.32:eth2-1.1.27:bond0>, peer not responding > 2014-03-04T14:20:27.893148+00:00 cmm02b kernel: TIPC: Lost link > <1.1.32:eth2-1.1.27:bond0> on network plane A > 2014-03-04T14:20:27.893154+00:00 cmm02b kernel: TIPC: Lost contact with > <1.1.27> > 2014-03-04T14:20:32.026411+00:00 cmm02b osafimmnd[4704]: NO NODE STATE-> > IMM_NODE_FULLY_AVAILABLE 2144 > 2014-03-04T14:20:32.026463+00:00 cmm02b osafimmnd[4704]: NO > RepositoryInitModeT is SA_IMM_INIT_FROM_FILE > 2014-03-04T14:20:32.026493+00:00 cmm02b osafimmnd[4704]: NO Epoch set to 22 > in ImmModel > 2014-03-04T14:20:32.031737+00:00 cmm02b osafimmnd[4704]: NO Implementer > connected: 72 (MsgQueueService73743) <67, 1200f> > 2014-03-04T14:20:32.035966+00:00 cmm02b osafimmnd[4704]: NO Implementer > connected: 73 (MsgQueueService73231) <0, 11e0f> > 2014-03-04T14:20:32.041233+00:00 cmm02b osafimmnd[4704]: NO SERVER STATE: > IMM_SERVER_SYNC_CLIENT --> IMM SERVER READY > 2014-03-04T14:20:32.042213+00:00 cmm02b osafimmnd[4704]: NO Implementer > connected: 74 (MsgQueueService72719) <0, 11c0f> > 2014-03-04T14:20:32.047252+00:00 cmm02b osafimmnd[4704]: NO Implementer > connected: 75 (MsgQueueService71439) <0, 1170f> > 2014-03-04T14:20:46.911220+00:00 cmm02b osafimmnd[4704]: NO Implementer > connected: 76 (MsgQueueService73487) <0, 10f0f> > 2014-03-04T14:20:46.920751+00:00 cmm02b osafimmnd[4704]: NO Implementer > disconnected 76 <0, 10f0f> (MsgQueueService73487) > 2014-03-04T14:20:48.012244+00:00 cmm02b osafimmnd[4704]: NO Implementer > connected: 77 (MsgQueueService72463) <0, 10f0f> > 2014-03-04T14:20:48.014779+00:00 cmm02b osafimmnd[4704]: NO Implementer > disconnected 77 <0, 10f0f> (MsgQueueService72463) > 2014-03-04T14:21:13.653100+00:00 cmm02b kernel: TIPC: Established link > <1.1.32:eth2-1.1.31:eth2> on network plane A > 2014-03-04T14:21:14.052200+00:00 cmm02b osafimmnd[4704]: NO NODE STATE-> > IMM_NODE_R_AVAILABLE > 2014-03-04T14:21:20.913433+00:00 cmm02b osafimmnd[4704]: NO NODE STATE-> > IMM_NODE_FULLY_AVAILABLE 14277 > 2014-03-04T14:21:20.913963+00:00 cmm02b osafimmnd[4704]: NO Epoch set to 23 > in ImmModel > 2014-03-04T14:21:21.419157+00:00 cmm02b osafimmnd[4704]: NO Implementer > connected: 78 (MsgQueueService73487) <0, 11f0f> > 2014-03-04T14:21:40.874192+00:00 cmm02b kernel: TIPC: Established link > <1.1.32:eth2-1.1.27:bond0> on network plane A > 2014-03-04T14:21:42.179625+00:00 cmm02b osafimmnd[4704]: NO NODE STATE-> > IMM_NODE_R_AVAILABLE > 2014-03-04T14:21:46.871328+00:00 cmm02b osafimmnd[4704]: NO NODE STATE-> > IMM_NODE_FULLY_AVAILABLE 14277 > 2014-03-04T14:21:46.871755+00:00 cmm02b osafimmnd[4704]: NO Epoch set to 24 > in ImmModel > 2014-03-04T14:21:47.649858+00:00 cmm02b osafimmnd[4704]: NO Implementer > connected: 79 (MsgQueueService72463) <0, 11b0f> > > ------------------------------------------------------------------------------ > Subversion Kills Productivity. Get off Subversion & Make the Move to Perforce. > With Perforce, you get hassle-free workflows. Merge that actually works. > Faster operations. Version large binaries. Built-in WAN optimization and the > freedom to use Git, Perforce or both. Make the move to Perforce. > http://pubads.g.doubleclick.net/gampad/clk?id=122218951&iu=/4140/ostg.clktrk > _______________________________________________ > Opensaf-users mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/opensaf-users ------------------------------------------------------------------------------ Subversion Kills Productivity. Get off Subversion & Make the Move to Perforce. With Perforce, you get hassle-free workflows. Merge that actually works. Faster operations. Version large binaries. Built-in WAN optimization and the freedom to use Git, Perforce or both. Make the move to Perforce. http://pubads.g.doubleclick.net/gampad/clk?id=122218951&iu=/4140/ostg.clktrk _______________________________________________ Opensaf-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/opensaf-users
