Summary: clm: harden processing node down events and agents during failover
[#1120]
Review request for Trac Ticket(s): #1120
Peer Reviewer(s): <<LIST THE TECH REVIEWER(S) / MAINTAINER(S) HERE>>
Pull request to: <<LIST THE PERSON WITH PUSH ACCESS HERE>>
Affected branch(es): 4.3 and above
Development branch: <<IF ANY GIVE THE REPO URL>>
--------------------------------
Impacted area Impact y/n
--------------------------------
Docs n
Build system n
RPM/packaging n
Configuration files n
Startup scripts n
SAF services y
OpenSAF services n
Core libraries n
Samples n
Tests n
Other n
Comments (indicate scope for each "y" above):
---------------------------------------------
This patch series touches 3 flows:
1) handling of stale node downs during controller failover
2) handling of agent downs first before node downs during failover
3) fixes a bug that sends trackcallback to agents on a node that just got
rebooted and
came back again with the same address.
Thanks to HansN for pursing this topic and suggesting the theory and testing of
the patches.
changeset 4243080696763d51a77048570b8275bbb6b8fff5
Author: Mathivanan N.P.<mathi.naic...@oracle.com>
Date: Wed, 24 Sep 2014 15:05:16 -0400
clm: avoid stale node down processing and unexpected track callback
[#1120]
There is a possiblity that the checkpointing message for a NODE_DOWN
reaches
the STANDBY first, i.e. before the MDS delivers the NODE_DOWN event to
the
standby. This can result in stale node_down record getting stored in the
node_down list which is a designated list for processing of node downs
that
occur during role change from standby to active. The patch introduces a
variable that checks whether the checkpoint event for node_down has
arrived
first, followed by a check during role change to ignore such stale
events.
changeset 0859c122fc22c43e92703547d952abeb854b7edc
Author: Mathivanan N.P.<mathi.naic...@oracle.com>
Date: Wed, 24 Sep 2014 15:11:14 -0400
clm: during failover, process agent down before node downs [#1120] It is
quite possible that the agent downs are for the agents that were
running on
the same node that went down. So, process agent downs first, before
processing node downs.
changeset 9b4c11f1ade1d4633abccb3bedf0a6b6111b3e01
Author: Mathivanan N.P.<mathi.naic...@oracle.com>
Date: Wed, 24 Sep 2014 15:16:02 -0400
clm: do not send track for the node that left the cluster because of
reboot
[#1120] It is possible that when a payload that goes down during
controller
failover, can reboot and come back fast. As a part of failover
processing,
it is possible that the agent downs for these nodes reach the new ACTIVE
before node down events. In this scenario, when sending a track callback
with COMPLETED step, we need to check whether the track is for an agent
that
is on a node for which we have received node downevent. And if so, don't
send track for any clm agent(subscriber) on that node.
Complete diffstat:
------------------
osaf/services/saf/clmsv/clms/clms_cb.h | 6 ++++++
osaf/services/saf/clmsv/clms/clms_evt.c | 51
+++++++++++++++++++++++++++++++++++++++++++++++++--
osaf/services/saf/clmsv/clms/clms_imm.c | 25 +++++++++++++++++++++----
3 files changed, 76 insertions(+), 6 deletions(-)
Testing Commands:
-----------------
1) Randomly reboot payload nodes
2) Trigger controller failover.
Testing, Expected Results:
--------------------------
There should not be any stale node downs for AMFND or other CLM agents running
on the node that just got rebooted and came back up and running.
Conditions of Submission:
-------------------------
Ack from HansN and Hans or Ramesh.
Arch Built Started Linux distro
-------------------------------------------
mips n n
mips64 n n
x86 n n
x86_64 y y
powerpc n n
powerpc64 n n
Reviewer Checklist:
-------------------
[Submitters: make sure that your review doesn't trigger any checkmarks!]
Your checkin has not passed review because (see checked entries):
___ Your RR template is generally incomplete; it has too many blank entries
that need proper data filled in.
___ You have failed to nominate the proper persons for review and push.
___ Your patches do not have proper short+long header
___ You have grammar/spelling in your header that is unacceptable.
___ You have exceeded a sensible line length in your headers/comments/text.
___ You have failed to put in a proper Trac Ticket # into your commits.
___ You have incorrectly put/left internal data in your comments/files
(i.e. internal bug tracking tool IDs, product names etc)
___ You have not given any evidence of testing beyond basic build tests.
Demonstrate some level of runtime or other sanity testing.
___ You have ^M present in some of your files. These have to be removed.
___ You have needlessly changed whitespace or added whitespace crimes
like trailing spaces, or spaces before tabs.
___ You have mixed real technical changes with whitespace and other
cosmetic code cleanup changes. These have to be separate commits.
___ You need to refactor your submission into logical chunks; there is
too much content into a single commit.
___ You have extraneous garbage in your review (merge commits etc)
___ You have giant attachments which should never have been sent;
Instead you should place your content in a public tree to be pulled.
___ You have too many commits attached to an e-mail; resend as threaded
commits, or place in a public tree for a pull.
___ You have resent this content multiple times without a clear indication
of what has changed between each re-send.
___ You have failed to adequately and individually address all of the
comments and change requests that were proposed in the initial review.
___ You have a misconfigured ~/.hgrc file (i.e. username, email etc)
___ Your computer have a badly configured date and time; confusing the
the threaded patch review.
___ Your changes affect IPC mechanism, and you don't present any results
for in-service upgradability test.
___ Your changes affect user manual and documentation, your patch series
do not contain the patch that updates the Doxygen manual.
------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
Opensaf-devel mailing list
Opensaf-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/opensaf-devel
