Re: [devel] [PATCH 0 of 2] Review Request for fm: failover using OPENSAF_MANAGE_TIPC flag and failover after down of critical services [#721]
Ok, now I see what you mean. In the case of failover without OS reboot, the FM process is terminated and will not perform the waiting. / Anders Widell 2014-02-04 12:56, Mathivanan Naickan Palanivelu skrev: > - anders.wid...@ericsson.com wrote: > >> I am not sure if I fully understand the use case. You say that this is >> >> for failover without OS reboot, but at the same time you say that FM >> will stay alive until it is killed by the OS reboot, and the peer FM >> will not take over until it receives a service down of FM. Could you >> elaborate a bit on this? >> > Support for failover without OS reboot would mean support for > failover by means of /etc/init.d/opensafd 'stop' without rmmod tipc > (i.e. The OS is up, non-OpenSAF applications are on and using TIPC). > Now, a mechanism that supports this scenario should also handle the > case when AMFND 'crashes' and node reboot got trigerred. > > In both the cases, the peer FM will failover only after downs of > all - AMFD, AMFND, IMMD, IMMND, FM are received. > > - In the 'stop' case, all processes are terminated by AMFND first. > Subsequently > AMFND and AMFD exit last. > - In the 'amfnd crash' case, all middleware processes terminate themselves > upon detecting > AMFND crash. But local FM will not exit upon detecting AMFND crash because > there is no guarantee that application processes have exited at that point of > time and therefore the local FM will 'wait' until the OS terminates it. > > It is for this 'waiting' period, that the proposal to run a system_fencer > script > on systems with delayed(by design or fault) reboots, has been made in a > separate patch > for TLC approval. > > > Cheers, > Mathi. > >> thanks >> / Anders Widell >> >> 2014-01-29 19:00, mathi.naic...@oracle.com skrev: >>> Summary: failover using OPENSAF_MANAGE_TIPC flag and failover after >> down of critical services [#721] >>> Review request for Trac Ticket(s): #721 >>> Peer Reviewer(s): ramesh.bet...@oracle.com, >> anders.wid...@ericsson.com >>> Pull request to: <> >>> Affected branch(es): opensaf-4.4.x, default >>> Development branch: <> >>> >>> >>> Impacted area Impact y/n >>> >>>Docsn >>>Build systemn >>>RPM/packaging n >>>Configuration files n >>>Startup scripts n >>>SAF servicesn >>>OpenSAF servicesy >>>Core libraries n >>>Samples n >>>Tests n >>>Other n >>> >>> >>> Comments (indicate scope for each "y" above): >>> - >>> Two cases are supported by this patch >>> >>> 1) Failover in 4.4.x, and default will be the same as previous >> releases. >>> i.e. as usual FM trigerres failover upon receiving the NODE_DOWN >> event. >>> 2) The main goal of the patch is w.r.t failover without OS reboot >> through >>> /etc/init.d/opensafd stop is more controlled now in the scenario >> "when amfnd itself crashes". >>> FM trigers failover after DOWNs of AMF, IMM and FM. >>> >>> More info of these cases: >>> 1) The flag OPENSAF_MANAGE_TIPC=yes is used to control when failover >> is trigerred. >>> This way, the default failover behaviour will now be the same as the >> previous releases. >>> There is no change involved. >>> >>> 2) In the usecase of failover (involving /etc/init.d/opensafd stop) >> without OS reboot cycle, >>> the flag nid.conf is set to OPENSAF_MANAGE_TIPC=no. This usecase is >> currently >>> intact, however some considerations need to be made ito handle the >> scenrio when amfnd itself crashes. >>> For this FM shall subscribe to service downs of AMFD, AMFND, IMMD, >> IMMND AND FM and "FM >>> by way of installing amfnd_down_callback shall exit only when the OS >> reboot terminates FM". >>> The peer FM will the failover once downs of all these services are >> received. >>> changeset c1875b7073b5fa2a1b9ae8755a15f6e8a6bf1aaf >>> Author: Mathivanan N.P. >>> Date: Wed, 29 Jan 2014 23:08:05 +0530 >>> >>> fm: failover using OPENSAF_MANAGE_TIPC flag and subscribe to AMF, >> IMM downs >>> [#721] To support failover without OS reboot, FM subscribed to >> AMFND down >>> events. But this may not be sufficient in scenarios when AMFND >> itself >>> crashes or exits. In this scenario, the exit/kill of OpenSAF >> processes need >>> not be in order and immediate. This can create a scenario where >> some OpenSAF >>> process may still be running, but FM has already started failover >>> processing. To avoid this, FM subscribes to the down events of >> critical >>> opensaf services - AMFD, AMFND, IMMD, IMMND In a false/quick >> failover >>> scenario, these are the services that can lead to problems like >>> implementerset not cleared or dangling AMF state assignments. >> Currently, all >>> OpenSAF services exit upon receiving the AMFND down events,
Re: [devel] [PATCH 0 of 2] Review Request for fm: failover using OPENSAF_MANAGE_TIPC flag and failover after down of critical services [#721]
- anders.wid...@ericsson.com wrote: > I am not sure if I fully understand the use case. You say that this is > > for failover without OS reboot, but at the same time you say that FM > will stay alive until it is killed by the OS reboot, and the peer FM > will not take over until it receives a service down of FM. Could you > elaborate a bit on this? > Support for failover without OS reboot would mean support for failover by means of /etc/init.d/opensafd 'stop' without rmmod tipc (i.e. The OS is up, non-OpenSAF applications are on and using TIPC). Now, a mechanism that supports this scenario should also handle the case when AMFND 'crashes' and node reboot got trigerred. In both the cases, the peer FM will failover only after downs of all - AMFD, AMFND, IMMD, IMMND, FM are received. - In the 'stop' case, all processes are terminated by AMFND first. Subsequently AMFND and AMFD exit last. - In the 'amfnd crash' case, all middleware processes terminate themselves upon detecting AMFND crash. But local FM will not exit upon detecting AMFND crash because there is no guarantee that application processes have exited at that point of time and therefore the local FM will 'wait' until the OS terminates it. It is for this 'waiting' period, that the proposal to run a system_fencer script on systems with delayed(by design or fault) reboots, has been made in a separate patch for TLC approval. Cheers, Mathi. > thanks > / Anders Widell > > 2014-01-29 19:00, mathi.naic...@oracle.com skrev: > > Summary: failover using OPENSAF_MANAGE_TIPC flag and failover after > down of critical services [#721] > > Review request for Trac Ticket(s): #721 > > Peer Reviewer(s): ramesh.bet...@oracle.com, > anders.wid...@ericsson.com > > Pull request to: <> > > Affected branch(es): opensaf-4.4.x, default > > Development branch: <> > > > > > > Impacted area Impact y/n > > > > Docsn > > Build systemn > > RPM/packaging n > > Configuration files n > > Startup scripts n > > SAF servicesn > > OpenSAF servicesy > > Core libraries n > > Samples n > > Tests n > > Other n > > > > > > Comments (indicate scope for each "y" above): > > - > > Two cases are supported by this patch > > > > 1) Failover in 4.4.x, and default will be the same as previous > releases. > > i.e. as usual FM trigerres failover upon receiving the NODE_DOWN > event. > > 2) The main goal of the patch is w.r.t failover without OS reboot > through > > /etc/init.d/opensafd stop is more controlled now in the scenario > "when amfnd itself crashes". > > FM trigers failover after DOWNs of AMF, IMM and FM. > > > > More info of these cases: > > 1) The flag OPENSAF_MANAGE_TIPC=yes is used to control when failover > is trigerred. > > This way, the default failover behaviour will now be the same as the > previous releases. > > There is no change involved. > > > > 2) In the usecase of failover (involving /etc/init.d/opensafd stop) > without OS reboot cycle, > > the flag nid.conf is set to OPENSAF_MANAGE_TIPC=no. This usecase is > currently > > intact, however some considerations need to be made ito handle the > scenrio when amfnd itself crashes. > > > > For this FM shall subscribe to service downs of AMFD, AMFND, IMMD, > IMMND AND FM and "FM > > by way of installing amfnd_down_callback shall exit only when the OS > reboot terminates FM". > > The peer FM will the failover once downs of all these services are > received. > > > > changeset c1875b7073b5fa2a1b9ae8755a15f6e8a6bf1aaf > > Author: Mathivanan N.P. > > Date: Wed, 29 Jan 2014 23:08:05 +0530 > > > > fm: failover using OPENSAF_MANAGE_TIPC flag and subscribe to AMF, > IMM downs > > [#721] To support failover without OS reboot, FM subscribed to > AMFND down > > events. But this may not be sufficient in scenarios when AMFND > itself > > crashes or exits. In this scenario, the exit/kill of OpenSAF > processes need > > not be in order and immediate. This can create a scenario where > some OpenSAF > > process may still be running, but FM has already started failover > > processing. To avoid this, FM subscribes to the down events of > critical > > opensaf services - AMFD, AMFND, IMMD, IMMND In a false/quick > failover > > scenario, these are the services that can lead to problems like > > implementerset not cleared or dangling AMF state assignments. > Currently, all > > OpenSAF services exit upon receiving the AMFND down events, but > even this > > does not guarantee immediate and ordered delivery of down events. > The next > > patch in this series of 2 patches makes FM to install > > ava_amfnd_down_callback such that FM waits forever till the OS > kills FM. > > > > changeset 77232a084d58b68c8b470f7b90f
Re: [devel] [PATCH 0 of 2] Review Request for fm: failover using OPENSAF_MANAGE_TIPC flag and failover after down of critical services [#721]
I am not sure if I fully understand the use case. You say that this is for failover without OS reboot, but at the same time you say that FM will stay alive until it is killed by the OS reboot, and the peer FM will not take over until it receives a service down of FM. Could you elaborate a bit on this? thanks / Anders Widell 2014-01-29 19:00, mathi.naic...@oracle.com skrev: > Summary: failover using OPENSAF_MANAGE_TIPC flag and failover after down of > critical services [#721] > Review request for Trac Ticket(s): #721 > Peer Reviewer(s): ramesh.bet...@oracle.com, anders.wid...@ericsson.com > Pull request to: <> > Affected branch(es): opensaf-4.4.x, default > Development branch: <> > > > Impacted area Impact y/n > > Docsn > Build systemn > RPM/packaging n > Configuration files n > Startup scripts n > SAF servicesn > OpenSAF servicesy > Core libraries n > Samples n > Tests n > Other n > > > Comments (indicate scope for each "y" above): > - > Two cases are supported by this patch > > 1) Failover in 4.4.x, and default will be the same as previous releases. > i.e. as usual FM trigerres failover upon receiving the NODE_DOWN event. > 2) The main goal of the patch is w.r.t failover without OS reboot through > /etc/init.d/opensafd stop is more controlled now in the scenario "when amfnd > itself crashes". > FM trigers failover after DOWNs of AMF, IMM and FM. > > More info of these cases: > 1) The flag OPENSAF_MANAGE_TIPC=yes is used to control when failover is > trigerred. > This way, the default failover behaviour will now be the same as the previous > releases. > There is no change involved. > > 2) In the usecase of failover (involving /etc/init.d/opensafd stop) without > OS reboot cycle, > the flag nid.conf is set to OPENSAF_MANAGE_TIPC=no. This usecase is currently > intact, however some considerations need to be made ito handle the scenrio > when amfnd itself crashes. > > For this FM shall subscribe to service downs of AMFD, AMFND, IMMD, IMMND AND > FM and "FM > by way of installing amfnd_down_callback shall exit only when the OS reboot > terminates FM". > The peer FM will the failover once downs of all these services are received. > > changeset c1875b7073b5fa2a1b9ae8755a15f6e8a6bf1aaf > Author: Mathivanan N.P. > Date: Wed, 29 Jan 2014 23:08:05 +0530 > > fm: failover using OPENSAF_MANAGE_TIPC flag and subscribe to AMF, IMM > downs > [#721] To support failover without OS reboot, FM subscribed to AMFND > down > events. But this may not be sufficient in scenarios when AMFND itself > crashes or exits. In this scenario, the exit/kill of OpenSAF processes > need > not be in order and immediate. This can create a scenario where some > OpenSAF > process may still be running, but FM has already started failover > processing. To avoid this, FM subscribes to the down events of critical > opensaf services - AMFD, AMFND, IMMD, IMMND In a false/quick failover > scenario, these are the services that can lead to problems like > implementerset not cleared or dangling AMF state assignments. > Currently, all > OpenSAF services exit upon receiving the AMFND down events, but even > this > does not guarantee immediate and ordered delivery of down events. The > next > patch in this series of 2 patches makes FM to install > ava_amfnd_down_callback such that FM waits forever till the OS kills FM. > > changeset 77232a084d58b68c8b470f7b90fa6a2df541183d > Author: Mathivanan N.P. > Date: Wed, 29 Jan 2014 23:13:27 +0530 > > fm: install ava_install_amf_down_cb and wait till OS reboot terminates > FM > [#721] FM installs amfnd down callback and upon receiving the amfnd down > event, shall wait till the OS reboot(trigerred by amfnd crash or exit) > terminates FM. The STANDBY FM will now start the failover after > recieving > the downs of all AMFD, AMFND, IMMD, IMMND and FM > > > Complete diffstat: > -- > osaf/services/infrastructure/fm/fms/fm.h |2 + > osaf/services/infrastructure/fm/fms/fm_amf.c | 18 +++ > osaf/services/infrastructure/fm/fms/fm_cb.h |8 + > osaf/services/infrastructure/fm/fms/fm_evt.h |2 + > osaf/services/infrastructure/fm/fms/fm_main.c| 86 > -- > osaf/services/infrastructure/fm/fms/fm_mds.c | 155 > +-- > osaf/services/infrastructure/fm/fms/fm_mds.h |1 + > osaf/services/infrastructure/nid/config/nid.conf |2 +- > 8 files changed, 222 insertions(+), 52 deletions(-) >
Re: [devel] [PATCH 0 of 2] Review Request for fm: failover using OPENSAF_MANAGE_TIPC flag and failover after down of critical services [#721]
Or, why not simply: for (;;) pause();
As I side note, static code analysis tools can complain about infinite
loops (I wrote one myself, in in that case it wasn't even infinite - it
was terminated by a signal, which the tool could not infer). One
solution would be to have e.g. a library function osaf_sleep_forever()
to make it visible that it is intended to be infinite. Or, in this
particular case, maybe it shouldn't sleep forever. It could sleep for,
say, an hour, and if the process is still alive after that it can log a
message about this and then call opensaf_reboot().
/ Anders Widell
2014-02-04 07:31, Ramesh Betham skrev:
> Ack. With the following Minor comments.
>
> 1. Let IMMND UP event too go through mbx processing.
> 2. Realistically this may not happen, let amfnd_down_callback() have
> while(1) loop to wait on OS reboot.
> while (1) {
> m_NCS_TASK_SLEEP(0xfff0);
> }
>
> Thanks,
> Ramesh.
>
> On 1/29/2014 11:30 PM, mathi.naic...@oracle.com wrote:
>> Summary: failover using OPENSAF_MANAGE_TIPC flag and failover after
>> down of critical services [#721]
>> Review request for Trac Ticket(s): #721
>> Peer Reviewer(s): ramesh.bet...@oracle.com, anders.wid...@ericsson.com
>> Pull request to: <>
>> Affected branch(es): opensaf-4.4.x, default
>> Development branch: <>
>>
>>
>> Impacted area Impact y/n
>>
>> Docsn
>> Build systemn
>> RPM/packaging n
>> Configuration files n
>> Startup scripts n
>> SAF servicesn
>> OpenSAF servicesy
>> Core libraries n
>> Samples n
>> Tests n
>> Other n
>>
>>
>> Comments (indicate scope for each "y" above):
>> -
>> Two cases are supported by this patch
>>
>> 1) Failover in 4.4.x, and default will be the same as previous releases.
>> i.e. as usual FM trigerres failover upon receiving the NODE_DOWN event.
>> 2) The main goal of the patch is w.r.t failover without OS reboot
>> through
>> /etc/init.d/opensafd stop is more controlled now in the scenario
>> "when amfnd itself crashes".
>> FM trigers failover after DOWNs of AMF, IMM and FM.
>>
>> More info of these cases:
>> 1) The flag OPENSAF_MANAGE_TIPC=yes is used to control when failover
>> is trigerred.
>> This way, the default failover behaviour will now be the same as the
>> previous releases.
>> There is no change involved.
>>
>> 2) In the usecase of failover (involving /etc/init.d/opensafd stop)
>> without OS reboot cycle,
>> the flag nid.conf is set to OPENSAF_MANAGE_TIPC=no. This usecase is
>> currently
>> intact, however some considerations need to be made ito handle the
>> scenrio when amfnd itself crashes.
>>
>> For this FM shall subscribe to service downs of AMFD, AMFND, IMMD,
>> IMMND AND FM and "FM
>> by way of installing amfnd_down_callback shall exit only when the OS
>> reboot terminates FM".
>> The peer FM will the failover once downs of all these services are
>> received.
>>
>> changeset c1875b7073b5fa2a1b9ae8755a15f6e8a6bf1aaf
>> Author:Mathivanan N.P.
>> Date:Wed, 29 Jan 2014 23:08:05 +0530
>>
>> fm: failover using OPENSAF_MANAGE_TIPC flag and subscribe to AMF,
>> IMM downs
>> [#721] To support failover without OS reboot, FM subscribed to
>> AMFND down
>> events. But this may not be sufficient in scenarios when AMFND
>> itself
>> crashes or exits. In this scenario, the exit/kill of OpenSAF
>> processes need
>> not be in order and immediate. This can create a scenario where
>> some OpenSAF
>> process may still be running, but FM has already started failover
>> processing. To avoid this, FM subscribes to the down events of
>> critical
>> opensaf services - AMFD, AMFND, IMMD, IMMND In a false/quick
>> failover
>> scenario, these are the services that can lead to problems like
>> implementerset not cleared or dangling AMF state assignments.
>> Currently, all
>> OpenSAF services exit upon receiving the AMFND down events, but
>> even this
>> does not guarantee immediate and ordered delivery of down events.
>> The next
>> patch in this series of 2 patches makes FM to install
>> ava_amfnd_down_callback such that FM waits forever till the OS
>> kills FM.
>>
>> changeset 77232a084d58b68c8b470f7b90fa6a2df541183d
>> Author:Mathivanan N.P.
>> Date:Wed, 29 Jan 2014 23:13:27 +0530
>>
>> fm: install ava_install_amf_down_cb and wait till OS reboot
>> terminates FM
>> [#721] FM installs amfnd down callback and upon receiving the
>> amfnd down
>> event, shall wait till the OS reboot(trigerred by amfnd crash or
>> exit)
>> terminates FM. The STANDBY FM will now start the failover after
>> recieving
>> the downs of all AMFD, AMFND, IMMD, IMMND and FM
>>
>>
>> Complete diffstat:
>> --
>> osaf/services/infrastru
Re: [devel] [PATCH 0 of 2] Review Request for fm: failover using OPENSAF_MANAGE_TIPC flag and failover after down of critical services [#721]
Ack. With the following Minor comments.
1. Let IMMND UP event too go through mbx processing.
2. Realistically this may not happen, let amfnd_down_callback() have
while(1) loop to wait on OS reboot.
while (1) {
m_NCS_TASK_SLEEP(0xfff0);
}
Thanks,
Ramesh.
On 1/29/2014 11:30 PM, mathi.naic...@oracle.com wrote:
> Summary: failover using OPENSAF_MANAGE_TIPC flag and failover after down of
> critical services [#721]
> Review request for Trac Ticket(s): #721
> Peer Reviewer(s): ramesh.bet...@oracle.com, anders.wid...@ericsson.com
> Pull request to: <>
> Affected branch(es): opensaf-4.4.x, default
> Development branch: <>
>
>
> Impacted area Impact y/n
>
> Docsn
> Build systemn
> RPM/packaging n
> Configuration files n
> Startup scripts n
> SAF servicesn
> OpenSAF servicesy
> Core libraries n
> Samples n
> Tests n
> Other n
>
>
> Comments (indicate scope for each "y" above):
> -
> Two cases are supported by this patch
>
> 1) Failover in 4.4.x, and default will be the same as previous releases.
> i.e. as usual FM trigerres failover upon receiving the NODE_DOWN event.
> 2) The main goal of the patch is w.r.t failover without OS reboot through
> /etc/init.d/opensafd stop is more controlled now in the scenario "when amfnd
> itself crashes".
> FM trigers failover after DOWNs of AMF, IMM and FM.
>
> More info of these cases:
> 1) The flag OPENSAF_MANAGE_TIPC=yes is used to control when failover is
> trigerred.
> This way, the default failover behaviour will now be the same as the previous
> releases.
> There is no change involved.
>
> 2) In the usecase of failover (involving /etc/init.d/opensafd stop) without
> OS reboot cycle,
> the flag nid.conf is set to OPENSAF_MANAGE_TIPC=no. This usecase is currently
> intact, however some considerations need to be made ito handle the scenrio
> when amfnd itself crashes.
>
> For this FM shall subscribe to service downs of AMFD, AMFND, IMMD, IMMND AND
> FM and "FM
> by way of installing amfnd_down_callback shall exit only when the OS reboot
> terminates FM".
> The peer FM will the failover once downs of all these services are received.
>
> changeset c1875b7073b5fa2a1b9ae8755a15f6e8a6bf1aaf
> Author: Mathivanan N.P.
> Date: Wed, 29 Jan 2014 23:08:05 +0530
>
> fm: failover using OPENSAF_MANAGE_TIPC flag and subscribe to AMF, IMM
> downs
> [#721] To support failover without OS reboot, FM subscribed to AMFND
> down
> events. But this may not be sufficient in scenarios when AMFND itself
> crashes or exits. In this scenario, the exit/kill of OpenSAF processes
> need
> not be in order and immediate. This can create a scenario where some
> OpenSAF
> process may still be running, but FM has already started failover
> processing. To avoid this, FM subscribes to the down events of critical
> opensaf services - AMFD, AMFND, IMMD, IMMND In a false/quick failover
> scenario, these are the services that can lead to problems like
> implementerset not cleared or dangling AMF state assignments.
> Currently, all
> OpenSAF services exit upon receiving the AMFND down events, but even
> this
> does not guarantee immediate and ordered delivery of down events. The
> next
> patch in this series of 2 patches makes FM to install
> ava_amfnd_down_callback such that FM waits forever till the OS kills FM.
>
> changeset 77232a084d58b68c8b470f7b90fa6a2df541183d
> Author: Mathivanan N.P.
> Date: Wed, 29 Jan 2014 23:13:27 +0530
>
> fm: install ava_install_amf_down_cb and wait till OS reboot terminates
> FM
> [#721] FM installs amfnd down callback and upon receiving the amfnd down
> event, shall wait till the OS reboot(trigerred by amfnd crash or exit)
> terminates FM. The STANDBY FM will now start the failover after
> recieving
> the downs of all AMFD, AMFND, IMMD, IMMND and FM
>
>
> Complete diffstat:
> --
> osaf/services/infrastructure/fm/fms/fm.h |2 +
> osaf/services/infrastructure/fm/fms/fm_amf.c | 18 +++
> osaf/services/infrastructure/fm/fms/fm_cb.h |8 +
> osaf/services/infrastructure/fm/fms/fm_evt.h |2 +
> osaf/services/infrastructure/fm/fms/fm_main.c| 86
> --
> osaf/services/infrastructure/fm/fms/fm_mds.c | 155
> +--
> osaf/services/infrastructure/fm/fms/fm_mds.h |1 +
> osaf/services/infrastructure/nid/config/nid.conf |2 +-
> 8 files changed, 222 insertions(+), 52 deletions(-)
>
>
> Testing Commands:
> -
> 1) Start a opens
