On 8/13/2014 3:30 PM, Hans Feldt wrote:
> Why is the finalizesync message so big?
>
> I think that is a concern if we have a mw that broadcasts big blobs without
> any upper limit.
I to have the same opinion.
>
> As a fix now MDS could instead of failing the broadcast send when the message
> if greater than 2^16, revert to the old multi-unicast.
Ok I will do that .
-AVM
>
> Thanks,
> Hans
>
>> -----Original Message-----
>> From: Neelakanta Reddy [mailto:reddy.neelaka...@oracle.com]
>> Sent: den 13 augusti 2014 11:34
>> To: mahesh.va...@oracle.com; Hans Feldt; Anders Widell; Anders Björnerstedt
>> Cc: opensaf-devel@lists.sourceforge.net
>> Subject: Re: [devel] [PATCH 0 of 3] Review Request for mds: use TIPC
>> multicast for MDS broadcast [#851]
>>
>> Hi Mahesh,
>>
>>>> - A single multicast message can accommodate max of MDS_DIRECT_BUF_MAXSIZE
>>>> (2^16).
>>
>> The above limit can be removed, because IMM can send the event with grater
>> than (2^16) size.
>>
>>
>> After running multiple regression test and do a failover the following is
>> observed:
>>
>> The testcase is Failover, the problem happened when Faiover node is joining.
>> At the end of syncing IMMND co-ordinator sends the finalizesync message,
>> which will be sent to IMMD and IMMD broadcast to all
>> IMMNDs.
>> The finalizesync meaasage may have the event size more than 64000. Since,
>> MDS drops packets with the size greater than 64000
>> because of
>> this there is mismatch in FEVS count and out of order is observed.This will
>> lead to cluster re-start.
>>
>> syslog at co-ordinator node:
>>
>> Aug 13 10:24:30 SLES-SLOT-2 osafimmnd[17711]: NO NODE STATE->
>> IMM_NODE_R_AVAILABLE
>> Aug 13 10:24:30 SLES-SLOT-2 osafimmd[6462]: NO Successfully announced
>> sync. New ruling epoch:19
>> Aug 13 10:24:30 SLES-SLOT-2 osafimmloadd: NO Sync starting
>> Aug 13 10:24:53 SLES-SLOT-2 osafimmloadd: IN Synced 421 objects in total
>> Aug 13 10:24:53 SLES-SLOT-2 osafimmnd[17711]: NO NODE STATE->
>> IMM_NODE_FULLY_AVAILABLE 16063
>> Aug 13 10:24:53 SLES-SLOT-2 osafimmloadd: NO Sync ending normally
>> Aug 13 10:24:53 SLES-SLOT-2 osafimmd[6462]: NO MDTM: Not possible to
>> send size:94807 TIPC multicast to svc_id: 25
>> Aug 13 10:24:53 SLES-SLOT-2 osafimmd[6462]: NO MDTM: Not possible to
>> send size:94807 TIPC multicast to svc_id: 25
>> Aug 13 10:24:53 SLES-SLOT-2 osafimmd[6462]: NO MDTM: Not possible to
>> send size:94807 TIPC multicast to svc_id: 25
>> Aug 13 10:24:53 SLES-SLOT-2 osafimmd[6462]: NO MDTM: Not possible to
>> send size:95147 TIPC multicast to svc_id: 25
>> Aug 13 10:26:44 SLES-SLOT-2 syslog-ng[1190]: Log statistics;
>> dropped='pipe(/dev/xconsole)=1825', dropped='pipe(/dev/tty10)=0',
>> processed='center(queued)=54989', processed='center(received)=22910',
>> processed='destination(messages)=22908',
>> processed='destination(mailinfo)=2',
>> processed='destination(mailwarn)=0',
>> processed='destination(localmessages)=21235',
>> processed='destination(newserr)=0', processed='destination(mailerr)=0',
>> processed='destination(netmgm)=0', processed='destination(warn)=3868',
>> processed='destination(console)=3487', processed='destination(null)=0',
>> processed='destination(mail)=2', processed='destination(xconsole)=3487',
>> processed='destination(firewall)=0', processed='destination(acpid)=0',
>> processed='destination(newscrit)=0',
>> processed='destination(newsnotice)=0', processed='source(src)=22910'
>> Aug 13 10:32:29 SLES-SLOT-2 osafimmnd[17711]: WA MESSAGE:78692 OUT OF
>> ORDER my highest processed:78690, exiting
>> Aug 13 10:32:29 SLES-SLOT-2 osafimmpbed: WA PBE lost contact with parent
>> IMMND - Exiting
>> Aug 13 10:32:29 SLES-SLOT-2 osafamfnd[6533]: NO
>> 'safSu=SC-2,safSg=NoRed,safApp=OpenSAF' component restart probation
>> timer started (timeout: 60000000000 ns)
>> Aug 13 10:32:29 SLES-SLOT-2 osafamfnd[6533]: NO Restarting a component
>> of 'safSu=SC-2,safSg=NoRed,safApp=OpenSAF' (comp restart count: 1)
>> Aug 13 10:32:29 SLES-SLOT-2 osafamfnd[6533]: NO
>> 'safComp=IMMND,safSu=SC-2,safSg=NoRed,safApp=OpenSAF' faulted due to
>> 'avaDown' : Recovery is 'componentRestart'
>> Aug 13 10:32:29 SLES-SLOT-2 osafimmd[6462]: WA IMMND coordinator at
>> 2020f apparently crashed => electing new coord
>> Aug 13 10:32:29 SLES-SLOT-2 osafimmd[6462]: ER Failed to find candidate
>> for new IMMND coordinator
>> Aug 13 10:32:29 SLES-SLOT-2 osafimmd[6462]: ER Active IMMD has to
>> restart the IMMSv. All IMMNDs will restart
>> Aug 13 10:32:29 SLES-SLOT-2 osafimmnd[6099]: Started
>> Aug 13 10:32:29 SLES-SLOT-2 osafimmnd[6099]: NO Persistent Back-End
>> capability configured, Pbe file:imm.db (suffix may get added)
>> Aug 13 10:32:29 SLES-SLOT-2 osafimmnd[6099]: NO SERVER STATE:
>> IMM_SERVER_ANONYMOUS --> IMM_SERVER_CLUSTER_WAITING
>> Aug 13 10:32:30 SLES-SLOT-2 osafimmnd[6099]: NO IMMND received reset
>> order from IMMD, but has just restarted - ignoring
>> Aug 13 10:32:30 SLES-SLOT-2 osafimmd[6462]: ER IMM RELOAD => ensure
>> cluster restart by IMMD exit at both SCs, exiting
>> Aug 13 10:32:30 SLES-SLOT-2 osafamfnd[6533]: NO
>> 'safComp=IMMD,safSu=SC-2,safSg=2N,safApp=OpenSAF' faulted due to
>> 'avaDown' : Recovery is 'nodeFailfast'
>> Aug 13 10:32:30 SLES-SLOT-2 osafamfnd[6533]: ER
>> safComp=IMMD,safSu=SC-2,safSg=2N,safApp=OpenSAF Faulted due to:avaDown
>> Recovery is:nodeFailfast
>> Aug 13 10:32:30 SLES-SLOT-2 osafamfnd[6533]: Rebooting OpenSAF NodeId =
>> 131599 EE Name = , Reason: Component faulted: recovery is node failfast,
>> OwnNodeId = 131599, SupervisionTime = 60
>>
>>
>> /Neel.
>>
>>
>> On Thursday 07 August 2014 02:05 PM, mahesh.va...@oracle.com wrote:
>>> Summary: mds: use TIPC multicast for MDS broadcast [#851]
>>> Review request for Trac Ticket(s): #851
>>> Peer Reviewer(s): Hans/Anders.widell/Anders.bj
>>> Pull request to: <<LIST THE PERSON WITH PUSH ACCESS HERE>>
>>> Affected branch(es): default
>>> Development branch: default
>>>
>>> --------------------------------
>>> Impacted area Impact y/n
>>> --------------------------------
>>> Docs n
>>> Build system n
>>> RPM/packaging n
>>> Configuration files n
>>> Startup scripts n
>>> SAF services n
>>> OpenSAF services n
>>> Core libraries y
>>> Samples n
>>> Tests n
>>> Other n
>>>
>>>
>>> Comments (indicate scope for each "y" above):
>>> ---------------------------------------------
>>>
>>> changeset c6de7935bc53f5b96ee0cf604e29d2784c6e4fca
>>> Author: A V Mahesh <mahesh.va...@oracle.com>
>>> Date: Thu, 07 Aug 2014 12:13:04 +0530
>>>
>>> mds: use TIPC multicast for MDS broadcast [#851] Brief note on Multicast
>>> Enhancement Ticket:
>>>
>>> -----------------------------------------------------------------------------
>>> Currently Opensaf Broadcast message was implemented as multiple
>>> unicasts (that means Broadcast message was reaching N nodes in T-1 to
>>> T-N
>>> time based on the number of nodes) ,after the ticket #851 changes , the
>>> Opensaf Broadcast message will be utilizing TIPC Multicast , that means
>>> message may reach T-1 time irrelevant of number of nodes in the cluster.
>>>
>>> This Enhancement of TIPC multicast make sure or receives broadcast
>>> message
>>> at same instant of Time . So this enhancement provides an improvement of
>>> node cluster or node cluster should take same bring up time ,and it also
>>> eliminate timing issue that were facing because multiple unicasts.
>>>
>>> But it also improves load time and sync time because of reduced
>>> unnecessary
>>> load to the sending process.
>>>
>>> Code changes :
>>>
>>> -----------------------------------------------------------------------------
>>>
>>> - The Code changes are only effects to MDS TIPC transport , NO Changes
>>> in
>>> MDS TCP transport .This change are with in-service upgrade support.
>>>
>>> - Now the MDS TIPC transport sendto() for SENDTYPE BCAST & RBCAST
>>> addrtype
>>> is TIPC_ADDR_MCAST.
>>>
>>> - A single multicast message can accommodate max of
>>> MDS_DIRECT_BUF_MAXSIZE
>>> (2^16).
>>>
>>> - MDS_SVC_INFO structure has a new variable , which maintains count of
>>> previous Opensaf version subscribers count for this service.
>>>
>>> a subscribe/unsubscribe of previous Opensaf version service for this
>>> service.
>>>
>>> - If the count is ZERO means all are nodes in the cluster has new
>>> version
>>> Opensaf
>>>
>>> - If the count is grater than ZERO means nodes in the cluster has both
>>> old and new version Opensaf.
>>>
>>> - If the count is ZERO the SENDTYPE BCAST & RBCAST messages will be sent
>>> as TIPC Multicast and this is at service level.
>>>
>>> - If the count is grater than ZERO SENDTYPE BCAST & RBCAST messages will
>>> be sent as previous multiple unicasts and this is at service level.
>>>
>>> Opensaf Services Code Tuning :
>>>
>>> -----------------------------------------------------------------------------
>>> While adopting new Multicast messaging I came across following
>>> Opensaf integration issue in very complex test cases ,in n normal
>>> conditions/use case these issue may not occur.
>>>
>>> But I have created ticket for all of the so these tickets required to be
>>> fixed to Multicast to work with out any issues even in complex
>>> conditions.
>>>
>>> 1. https://sourceforge.net/p/opensaf/tickets/952/ 2.
>>> https://sourceforge.net/p/opensaf/tickets/953/ 3.
>>> https://sourceforge.net/p/opensaf/tickets/954/ 4.
>>> https://sourceforge.net/p/opensaf/tickets/955/ 5.
>>> https://sourceforge.net/p/opensaf/tickets/946/
>>>
>>> changeset 5f86111a818ccff18f6a9cd93d12ac7b15b3a7d3
>>> Author: A V Mahesh <mahesh.va...@oracle.com>
>>> Date: Thu, 07 Aug 2014 12:14:11 +0530
>>>
>>> imm: tune imm macros according to mds mcast size [#851]
>>>
>>> - No functional changes.
>>>
>>> - The IMM IMMSV_DEFAULT_MAX_SYNC_BATCH_SIZE is limited to 90%
>>> MDS_DIRECT_BUF_MAXSIZE (2^16) to accommodate IMM header data.
>>>
>>> changeset e446a201760b8e1e2674fa022b622af0aa5ce34f
>>> Author: A V Mahesh <mahesh.va...@oracle.com>
>>> Date: Thu, 07 Aug 2014 13:56:07 +0530
>>>
>>> mds: support multicast for n_way configuration [#851]
>>>
>>> - The existing Opensaf Broadcast message was implemented such that the
>>> Bcast
>>> messes will be send to only the active subscribers as multiple unicast,
>>> assuming that the subscription list may have n_way_active & n_way.
>>>
>>> - But currently we don't have any n_way configuration in the current
>>> Opensaf
>>> middle ware , and so far we didn't come across such n_way configuration
>>> application.
>>>
>>> - In case of Multicast Messaging TIPC to send a copy of the message to
>>> every
>>> port in the sender's cluster irrelevant of role of the subscriber (
>>> assuming application will filter it based on active/standby)
>>>
>>> - So to match existing Opensaf Broadcast message , i am providing this
>>> Option patch as well.
>>>
>>> - If community thinks this is required we will push this optional patch
>>> ,
>>> otherwise we will attach this to ticket for future use when we will
>>> suppor
>>> n_way .
>>>
>>> Code changes :
>>>
>>> Before sending the Multicast Messaging , query the subscribers for any
>>> standby exist (n_way subscribers list) , if (if standby exist) send old
>>> type Broadcast message as multiple unicast only to actives , else in the
>>> case of n_way_active configuration sends Multicast Message.
>>>
>>>
>>> Complete diffstat:
>>> ------------------
>>> osaf/libs/common/immsv/include/immsv_api.h | 6 ++--
>>> osaf/libs/core/mds/include/mds_core.h | 5 ++++
>>> osaf/libs/core/mds/mds_c_api.c | 19 ++++++++++++--
>>> osaf/libs/core/mds/mds_c_db.c | 29
>>> +++++++++++++++++++++++
>>> osaf/libs/core/mds/mds_c_sndrcv.c | 126
>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---------------------------------
>>> osaf/libs/core/mds/mds_dt_tipc.c | 69
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++----
>>> osaf/libs/core/mds/mds_main.c | 2 +-
>>> osaf/services/saf/immsv/immloadd/imm_loader.cc | 8 +++++-
>>> 8 files changed, 208 insertions(+), 56 deletions(-)
>>>
>>>
>>> Testing Commands:
>>> -----------------
>>> s Patch has improved ~50% of performance
>>>
>>> I have considered IMM concurrently syncing multiple Payload for
>>> measuring Benchmarks of the Multicast Messaging ( this is one of the case
>>> where
>>> Multicast/Broadcast being used with high frequency ).
>>>
>>>
>>> With Multicast changes ,all restarted Node now joining cluster with in
>>> ~42 seconds , which used to take ~74 seconds when One million IMM objects
>>> are involved in sync process.
>>>
>>>
>>> Following are the benchmarking figure:
>>>
>>>
>>> OpenSAF 4.5 release ( Default staging) WITH #851 Multicast patch:-
>>> -----------------------------------------------------------------------
>>> Jul 15 15:40:59 SLES-SLOT2 opensafd: Starting OpenSAF Services
>>> Jul 15 15:41:43 SLES-SLOT2 opensafd: OpenSAF(4.5.M0 - ) services
>>> successfully started <=== 42 Seconds
>>>
>>> Jul 15 15:40:58 SLES-SLOT3 opensafd: Starting OpenSAF Services
>>> Jul 15 15:41:42 SLES-SLOT3 opensafd: OpenSAF(4.5.M0 - ) services
>>> successfully started <=== 44 Seconds
>>>
>>> Jul 15 15:41:30 SLES-SLOT4 opensafd: Starting OpenSAF Services
>>> Jul 15 15:42:12 SLES-SLOT4 opensafd: OpenSAF(4.5.M0 - ) services
>>> successfully started <=== 42 Seconds
>>> -----------------------------------------------------------------------
>>>
>>>
>>> OpenSAF 4.5 release ( Default staging) WITH-OUT #851 Multicast patch:-
>>> ----------------------------------------------------------------------
>>> Jul 15 15:21:21 SLES-SLOT2 opensafd: Starting OpenSAF Services
>>> Jul 15 15:22:36 SLES-SLOT2 opensafd: OpenSAF(4.5.M0 - ) services
>>> successfully started <=== 74 Seconds
>>>
>>> Jul 15 15:21:20 SLES-SLOT3 opensafd: Starting OpenSAF Services
>>> Jul 15 15:22:34 SLES-SLOT3 opensafd: OpenSAF(4.5.M0 - ) services
>>> successfully started <=== 74 Seconds
>>>
>>> Jul 15 15:21:51 SLES-SLOT4 opensafd: Starting OpenSAF Services
>>> Jul 15 15:23:05 SLES-SLOT4 opensafd: OpenSAF(4.5.M0 - ) services
>>> successfully started <=== 74 Seconds
>>> ------------------------------------------------------------------------
>>>
>>>
>>> Note : need to make sure IMMND(s) of each restated node SERVER STATE
>>> should be in `IMM_SERVER_SYNC_CLIENT` at same point of time.
>>>
>>>
>>> You can achieve this programmatically by hacking/commenting code
>>> `cb->mLostNodes--;` line : 580 of `immnd_proc.c` file.
>>>
>>>
>>>
>>> Testing, Expected Results:
>>> --------------------------
>>> <<PASTE COMMAND OUTPUTS / TEST RESULTS>>
>>>
>>>
>>> Conditions of Submission:
>>> -------------------------
>>> Ack from Reviewer
>>>
>>> Arch Built Started Linux distro
>>> -------------------------------------------
>>> mips n n
>>> mips64 n n
>>> x86 n n
>>> x86_64 y y
>>> powerpc n n
>>> powerpc64 n n
>>>
>>>
>>> Reviewer Checklist:
>>> -------------------
>>> [Submitters: make sure that your review doesn't trigger any checkmarks!]
>>>
>>>
>>> Your checkin has not passed review because (see checked entries):
>>>
>>> ___ Your RR template is generally incomplete; it has too many blank entries
>>> that need proper data filled in.
>>>
>>> ___ You have failed to nominate the proper persons for review and push.
>>>
>>> ___ Your patches do not have proper short+long header
>>>
>>> ___ You have grammar/spelling in your header that is unacceptable.
>>>
>>> ___ You have exceeded a sensible line length in your headers/comments/text.
>>>
>>> ___ You have failed to put in a proper Trac Ticket # into your commits.
>>>
>>> ___ You have incorrectly put/left internal data in your comments/files
>>> (i.e. internal bug tracking tool IDs, product names etc)
>>>
>>> ___ You have not given any evidence of testing beyond basic build tests.
>>> Demonstrate some level of runtime or other sanity testing.
>>>
>>> ___ You have ^M present in some of your files. These have to be removed.
>>>
>>> ___ You have needlessly changed whitespace or added whitespace crimes
>>> like trailing spaces, or spaces before tabs.
>>>
>>> ___ You have mixed real technical changes with whitespace and other
>>> cosmetic code cleanup changes. These have to be separate commits.
>>>
>>> ___ You need to refactor your submission into logical chunks; there is
>>> too much content into a single commit.
>>>
>>> ___ You have extraneous garbage in your review (merge commits etc)
>>>
>>> ___ You have giant attachments which should never have been sent;
>>> Instead you should place your content in a public tree to be pulled.
>>>
>>> ___ You have too many commits attached to an e-mail; resend as threaded
>>> commits, or place in a public tree for a pull.
>>>
>>> ___ You have resent this content multiple times without a clear indication
>>> of what has changed between each re-send.
>>>
>>> ___ You have failed to adequately and individually address all of the
>>> comments and change requests that were proposed in the initial review.
>>>
>>> ___ You have a misconfigured ~/.hgrc file (i.e. username, email etc)
>>>
>>> ___ Your computer have a badly configured date and time; confusing the
>>> the threaded patch review.
>>>
>>> ___ Your changes affect IPC mechanism, and you don't present any results
>>> for in-service upgradability test.
>>>
>>> ___ Your changes affect user manual and documentation, your patch series
>>> do not contain the patch that updates the Doxygen manual.
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Infragistics Professional
>>> Build stunning WinForms apps today!
>>> Reboot your WinForms applications with our WinForms controls.
>>> Build a bridge from your legacy apps to the future.
>>> http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk
>>> _______________________________________________
>>> Opensaf-devel mailing list
>>> Opensaf-devel@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/opensaf-devel
------------------------------------------------------------------------------
_______________________________________________
Opensaf-devel mailing list
Opensaf-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/opensaf-devel
