Hi, This is a series of 4 patches. It is all or nothing. I sent out v2 for patch 1 after Surya’s comments. So is this an ack for the complete series? Thanks, Hans
From: A V Mahesh [mailto:mahesh.va...@oracle.com] Sent: den 16 maj 2014 03:51 To: Hans Feldt; Hans Feldt Cc: opensaf-devel@lists.sourceforge.net; Suryanarayana Garlapati Subject: Re: [devel] [PATCH 3 of 4] mds: use TIPC segmentation/reassembly [#654] Importance: High Hi Hans , ACK from me for your published patch , please go-ahead and push the patch as it is. I discussion/syncdup with Surya, following are the summary points : By default MDS does FULL encode/decode between different nodes (inter-node communication), also the behavior is same when the MDS messaging happens between 32-bit and 64-bit processes. your published Patch: #654 is not impacting any of these default functionality . Even though MDS is providing a compile time flag called "mds_arch", when this flag is set explicitly to the same value on different nodes, then only MDS messaging across nodes will happen with FLAT encode/decode routines, but most of the OpenSAF services (mds application) are still doing FULL encode/decode even when the MDS is triggered FLAT encode/decode callbacks ( Middle ware applications are not using advantage of "mds_arch" ). Till now none of the OpenSAF user are using the 'mds_arch' field and all are going with the MDS default settings. So they are NO impacts of deprecating "mds_arch" and considering that for versions. -AVM On 5/7/2014 12:20 PM, SuryaNarayana Garlapati wrote: OK, Here goes the solution. There is a 8 bit field in each message that is being exchanged between the services (which contains the message priority, mds prot_ver(mds protocol and mds version)) when message send is attempted. 6 bits are allocated for the mds prot_ver, present value for this field is 0xA8. This field can be used for versioning(and intended for this type of changes in MDS). This version can be dynamically learnt for a destination, when the first message is received from that destination and is stored in the Subscription result table. MDS prot_ver value will be changed in this release. So we can identify the old and new ones. If the destination is an old one, we will send with the old logic and if its a new one we will send with the bigger message. There is other solution, but this includes quite code changes. For example, In each node bind a new type say X and in each process subscribe for this X. When we get up this means, it indicates a new node and messaging is with bigger size else the normal one. Only a single extra bind is done for a node. There is one more solution, but it breaks the inservice upgradabilty, where in which the vdest id range is decreased. Regards Surya On Wednesday 07 May 2014 08:44 AM, A V Mahesh wrote: Surya, Thank for reiterating arch_word of the MDS feature ,we all in sync. On 5/6/2014 3:55 PM, SuryaNarayana Garlapati wrote: MDS version unless we get alternate bits/variables used for MDS version. [Surya] Thats the reason i am asking for some time. [AVM] If we get some alternate bits/variables used for MDS version this will the best Option and we can retain arch_word MDS feature. On 5/6/2014 3:55 PM, SuryaNarayana Garlapati wrote: 6) This current patch Honors the default Opensaf configuration ` mds_arch=0` behavior ( point 1 & 2 ) and it removed hidden `same arc message optimization` feature related code and used those 3bits/variable for the purpose of MDS version. [Surya] Current patch has still problems. I am repeating the same, but for your convenience, they are listed below. 1. Its taking out MDS optimization feature for the performance enhancement of the messaging. [AVM] we all in sync on this , now we are at the point whether we should expose this or not to OpenSAF users? 2. 64 bit and 32 bit combination will not work on the local node as it is doing flat encode which is wrong. [AVM] The current patch do ENC_TYPE_FULL for mixed 32/64 bit clients on the same Node. -AVM On 5/6/2014 3:55 PM, SuryaNarayana Garlapati wrote: Before going ahead, Following is the explanation for the arch_word of the MDS. Arch word(4bits) is combination of architecture and bit size of the machine. 3 bits are allocated for architecture and 1 bit is allocated for bit size. architecture of value 0 means unspecified. Message encoding is done as follows: 1. If the source and destination of the message is same process, then copy callback is called. 2. If source and destination arch_word are same, then encode flat is called. But there are some exceptions here, Say there are two nodes, A(Intel) and B(Powerpc) communicating with each other and are 64 bit each and opensaf code was compiled without giving any mds_arch input. In this case mds_arch is set to default 0. So as per the rule, encode flat callback should be called. But in this case if the flat encode is called, then the communication between A and B will not work as one is little endian and other is big endian. For handling this type of situations where the mds_arch is unspecified, mds checks for the mds_arch and if it is set to 0, MDS will call the encode full such that the communication will work between the nodes as the message is fully encoded. Encode flat will also be called if the source and destination are on the same node and arch_word is same. 3. If source and destination arch_word are different, then encode full is called. In this case the source and destination can be on the same or different node. On Tuesday 06 May 2014 09:05 AM, A V Mahesh wrote: Hi All, I re-verified the behavior of OpenSAF MDS_ENC_TYPE_FULL/MDS_ENC_TYPE_FLAT messaging and its optimization , following is my observation and option: 1) In the default OpenSAF configuration (` mds_arch=0` ) , messaging across the node is ENC_TYPE_FULL and messaging with-in the node is ENC_TYPE_FLAT . [Surya] See the explanation in point 2 and point 3. 2) Even though Opensaf is configured with default `mds_arch=0`, if their is difference of 32--bit application communicating to 64-bit middle-ware ( mixed 32/64 bit clients on the same Node), messaging with-in the node is also ENC_TYPE_FULL. [Surya] See the explanation in point 3. 3) when OpenSAF is configured with `mds_arch=1... to 7` messaging across & with-in the node is ENC_TYPE_FLAT , hear we have advantage of message optimization , only when explicitly enabling this configuration . [Surya] See the exception explanation in point 2. MDS was always designed to have this optimization a long time. It was just the build system was not using it. 4) Unfortunate the same arc message optimization feature got broken in few services such as RDE & PLM doesn't support ENC_TYPE_FLAT , we need fix these issue along with in-service upgrade to make it work properly . [Surya] Optimization feature is triggered by MDS. RDE & PLM are missing the implementation for this callbacks. 5) One more important note is that the `same arc message optimization` feature was not exposed to OpenSAF user till now. [Surya] It was always present in the confiure. Its just that, it was not used any time and default option was choosed for the mds_arch. 6) This current patch Honors the default Opensaf configuration ` mds_arch=0` behavior ( point 1 & 2 ) and it removed hidden `same arc message optimization` feature related code and used those 3bits/variable for the purpose of MDS version. [Surya] Current patch has still problems. I am repeating the same, but for your convenience, they are listed below. 1. Its taking out MDS optimization feature for the performance enhancement of the messaging. 2. 64 bit and 32 bit combination will not work on the local node as it is doing flat encode which is wrong. So I agree with Hans and let us not regressed for Un exposed feature , [Surya] Feature is present, the only thing is it was not used. and we will use the ARCHTYPE 3 bits for MDS version unless we get alternate bits/variables used for MDS version. [Surya] Thats the reason i am asking for some time. I also NOT much in favor of little bit expensive `new Node install & subscribe its Mds version` (My previously published approach) So following are current options : - Use ARCHTYPE 3 bits for MDS version and discard un exposed feature ( `mds_arch` config option) - Find out alter native bits for MDS version fix all the ENC_TYPE_FLAT issue with in-service upgrade and expose `same arc message optimization` feature to OpenSAF users. - Use `new Node install & subscribe` option for Mds version (My previously published approach) and fix all the ENC_TYPE_FLAT issue with in-service upgrade and expose `same arc message optimization` feature to OpenSAF users. -AVM On 5/1/2014 2:45 PM, SuryaNarayana Garlapati wrote: On Thursday 01 May 2014 02:30 PM, Hans Feldt wrote: Let's be clear that MDS a pure internal opensaf service since quite a long time. There are no "applications" to consider at all. This means the only thing to care about is in-service performance and making sure characteristics are improved, not regressed. [Surya] I am saying in the context of internal messaging between the services which use MDS. Yes, thats what even i am trying to. I am not sure we have to maintain some legacy requirement of mixed 32/64 bit clients on the same system. I don't see why we should. [Surya] This is not limited to even 32/64 bit. Its how the encoding is handled for the destinations. Thanks, Hans -----Original Message----- From: SuryaNarayana Garlapati [mailto:suryanarayana.garlap...@oracle.com] Sent: den 1 maj 2014 10:54 To: A V Mahesh Cc: opensaf-devel@lists.sourceforge.net<mailto:opensaf-devel@lists.sourceforge.net> Subject: Re: [devel] [PATCH 3 of 4] mds: use TIPC segmentation/reassembly [#654] On Thursday 01 May 2014 01:04 PM, A V Mahesh wrote: On 5/1/2014 10:16 AM, SuryaNarayana Garlapati wrote: The archtype is not a legacy one, it was introduced to have the optimization for the message encoding by the applications. Passing compile flags `mds_arch=1` `mds_arch=2` (mds_arch, [The arch-type input to MDS - valid values 1-7]) , is not a standard Open-Source ( enterprise things for legacy product ) option for compilations. [Surya] Wrong, if you see any standard linux configure options for any linux application code distributions, they do provide the options for supporting the options to provide such powerpc, intel, 32/64bit etc. So similar is the configure options for opensaf as well. For example you can see the configure options for the net-snmp as well. Open source user not need to Understand/get-Educate about this internal options , [Surya] This is not a internal option, configure option which is exposed to user as well. we are ok loose advantage of flat encode optimization , from now we want to send full_encode to different node. [Surya] This is not only done for optimization, this internally effects the message performance between the opensaf services which do huge messaging. I am on way of thinking the solution for this. Lets wait. We are also raising enhancement ticket for code clean. -AVM On 5/1/2014 10:16 AM, SuryaNarayana Garlapati wrote: On Thursday 01 May 2014 09:56 AM, A V Mahesh wrote: Hi Surya, On 5/1/2014 7:28 AM, Suryanarayana Garlapati wrote: Nack. MDS ARCHWORD is used for deciding the factor whether the message which is being sent to the destination should be encoded full/flat/copied. This bits were introduced to do a optimization in the way the callbacks encode full/flat are called. For example, if there are two processes(can be present in the same node or different node) and there architecture and bit size is same, so in this case, the encode flat can be called (or simply a flat copy of the message can be done in the UB). If these bits were not used, earlier the messages was being full encoded when the messages were sent to another node. At the same time, the arch field is given in the configure such that this can be used for different architectures such as the powerpc, etc. [AVM] “ARCHTYPE” is getting only used while cross-compiling, and to distinguish architecture flavors: x86, PowerPC, m68k ,ect . In the code "arch_word" is used while encoding whether to do MDS_ENC_TYPE_FLAT/MDS_ENC_TYPE_FULL. Keeping “ARCHTYPE” doesn't have any significance in current Open-sourced Opnesaf code , [Surya] This is provided to accommodate different architectures and is a build option which optimizes the applications way of message encoding(full/flat) and corresponding decode(full/flat). So that means, you want to remove this support and introduce the over head again. One more thing, with the new changes if you have two nodes one with powerpc and intel, both will not be able to communicate each other as both will result in flat encode/flat decode(as the value is one with the latest changes). why because In the current MDS messaging we do full encoded when the messages were sent to another process/node, irrelevant of “ARCHTYPE” [Surya] Wrong, the decision is done always on the archword only. I already verified cross 32/64 middle-ware & application combination , all the fowls are working fine . [Surya] As you said this might work, but you are loosing the optimization and would always result in full encode/full decode of application messages which is an overhead. I am thinking of a solution for this and will get back to you. [AVM] we already have an alternative option of new node do install & subscribe its Node Mds version, to handle in-service Upgrade , but we used legacy “ARCHTYPE” 3 bits for Mds version which is optimized solution then install & subscribe. [Surya] Thats what we had discussed before as well. The archtype is not a legacy one, it was introduced to have the optimization for the message encoding by the applications. -AVM Regards Surya ----- Original Message ----- From: osafde...@gmail.com<mailto:osafde...@gmail.com> To: mahesh.va...@oracle.com<mailto:mahesh.va...@oracle.com> Cc: opensaf-devel@lists.sourceforge.net<mailto:opensaf-devel@lists.sourceforge.net> Sent: Tuesday, April 29, 2014 10:58:53 AM GMT +05:30 Chennai, Kolkata, Mumbai, New Delhi Subject: [devel] [PATCH 3 of 4] mds: use TIPC segmentation/reassembly [#654] configure.ac | 21 --------------------- osaf/libs/core/mds/Makefile.am | 1 - osaf/libs/core/mds/include/mds_dt2c.h | 17 ++++++++--------- osaf/libs/core/mds/mds_c_sndrcv.c | 14 +++++--------- osaf/libs/core/mds/mds_dt_tipc.c | 11 ++++++++++- osaf/libs/core/mds/mds_log.c | 4 ++-- 6 files changed, 25 insertions(+), 43 deletions(-) TIPC supports sending message with a maximum length of 66000 bytes. MDS supports sending even bigger messages, upto ~46Mb. MDS has used an internal fragmentation of just 1400 bytes (MDTM_NORMAL_MSG_FRAG_SIZE) despite that the receive buffer is 8000 bytes (MDS_DIRECT_BUF_MAXSIZE) This patch changes MDS to use TIPC segmentation/reassembly. This gives benefits such as secure transmission (TIPC does retransmission). TIPC send congestion is reduced since TIPC counts messages not bytes. Less processing done in user space in the sending opensaf or client process. Since the receive buffer of older MDS cores is limited to 8000 bytes we have be sure not to send larger messages than that to older cores. Otherwise we have problems with in-service upgradeability. Each MDS TIPC port is bound to a functional address - a port name. The port name consists of 32 bit type and 32 bit instance. The 4 msb of the instance word contains the so called "archword" field. The msb indicates if this MDS core is 32 or 64 bit. The remaining 3 bits is now used as a version field. Version 0 is used by older versions of MDS, version 1 is now used to indicate the capability for longer fragmentation. This information is part of the published port name thus a subscriber will get them using the TIPC discovery service. These bits are ignored by old MDS cores, so it can receive messages from a new MDS core. When sending a message this bit is used to understand if the peer uses TIPC segmentation/reassembly (or not) and adapt the limit accordingly. The possibility to set mds_arch using configure has been removed. The change is in-service compliant and can be upgraded into a system. diff --git a/configure.ac b/configure.ac --- a/configure.ac +++ b/configure.ac @@ -470,27 +470,6 @@ if test "$enable_java" = no; then fi ############################################# -# MDS ARCH TYPE: MDS advises message encode/decode -# optimization to its clients if the sender and receiver -# are of "compatible-type". For this optimization to work, -# the sender's platform and receiver's platform should be -# similar (for e.g. both PowerPC) and use an libncs_core built -# with the same value for "mds_arch". An "mds_arch" value of 0 -# is the default value and stands for a platform which is NOT -# compatible with any platform. -############################################# - -AC_ARG_VAR([mds_arch], [The arch-type input to MDS - valid values 1-7]) - -if test "$mds_arch" = ""; then - mdsarchtype=0 -else - mdsarchtype=$mds_arch -fi - -AC_SUBST([MDS_WORD_ARCH_FLAGS], [" -DMDS_ARCH_TYPE=$mdsarchtype"]) - -############################################# # Checks for libraries. ############################################# PKG_CHECK_MODULES([XML2], [libxml-2.0]) diff --git a/osaf/libs/core/mds/Makefile.am b/osaf/libs/core/mds/Makefile.am --- a/osaf/libs/core/mds/Makefile.am +++ b/osaf/libs/core/mds/Makefile.am @@ -23,7 +23,6 @@ SUBDIRS = include noinst_LTLIBRARIES = libmds.la libmds_la_CPPFLAGS = \ - @MDS_WORD_ARCH_FLAGS@ \ $(AM_CPPFLAGS) libmds_la_LDFLAGS = -static diff --git a/osaf/libs/core/mds/include/mds_dt2c.h b/osaf/libs/core/mds/include/mds_dt2c.h --- a/osaf/libs/core/mds/include/mds_dt2c.h +++ b/osaf/libs/core/mds/include/mds_dt2c.h @@ -33,17 +33,16 @@ typedef uint8_t MDS_SVC_ARCHWORD_TYPE; /*MDS app-svc arch and word_size combination */ -/* MDS_WORD_SIZE_TYPE and MDS_ARCH_TYPE are compile-time macros */ - -#ifndef MDS_ARCH_TYPE -#define MDS_ARCH_TYPE 0 /* Stands for unspecified architecture type */ -#elif (MDS_ARCH_TYPE > 7) -#error MDS_ARCH_TYPE should be in the range 0 to 7. -#endif - #define MDS_WORD_SIZE_TYPE ((sizeof(long)/4) - 1) /* 0 for 32-bit, 1 for 64-bit */ -#define MDS_SELF_ARCHWORD ((MDS_SVC_ARCHWORD_TYPE) ((MDS_WORD_SIZE_TYPE<<3) | MDS_ARCH_TYPE)) +/* + * 4 bit ARCHWORD usage: + * Bit 3 is wordsize + * Bit 2:0 is a version field indicating capabilities. + * Version 0 uses 1400 bytes fragmentation. + * Version 1 uses TIPC max msg (66000 bytes) fragmentation. + */ +#define MDS_SELF_ARCHWORD ((MDS_SVC_ARCHWORD_TYPE) ((MDS_WORD_SIZE_TYPE<<3) | 1)) typedef enum { MDS_VIEW_NORMAL, diff --git a/osaf/libs/core/mds/mds_c_sndrcv.c b/osaf/libs/core/mds/mds_c_sndrcv.c --- a/osaf/libs/core/mds/mds_c_sndrcv.c +++ b/osaf/libs/core/mds/mds_c_sndrcv.c @@ -1483,10 +1483,10 @@ static uint32_t mcm_msg_encode_full_or_f msg_send.dest_pwe_id = m_MDS_GET_PWE_ID_FROM_SVC_HDL(svc_cb->svc_hdl); msg_send.dest_vdest_id = dest_vdest_id; msg_send.src_svc_sub_part_ver = svc_cb->svc_sub_part_ver; + msg_send.msg_arch_word = to_msg->rem_svc_arch_word; if (msg_send.msg.encoding == MDS_ENC_TYPE_FULL) { if (NULL == bcast_ptr) { - msg_send.msg_fmt_ver = cbinfo.info.enc.o_msg_fmt_ver; /* archword will be filled in next label */ - msg_send.msg_arch_word = to_msg->rem_svc_arch_word; + msg_send.msg_fmt_ver = cbinfo.info.enc.o_msg_fmt_ver; } } else { if (NULL == bcast_ptr) { @@ -6502,14 +6502,10 @@ static uint32_t mcm_query_for_node_dest_ if (m_MDS_GET_ADEST == adest) { *to = DESTINATION_SAME_PROCESS; } else if (MDS_SELF_ARCHWORD == arch_word) { - if ((0 == (MDS_SELF_ARCHWORD & 0x7) && (0 == (arch_word & 0x7)))) { - if (m_MDS_GET_NODE_ID_FROM_ADEST(m_MDS_GET_ADEST) == m_MDS_GET_NODE_ID_FROM_ADEST(adest)) { - *to = DESTINATION_ON_NODE; /* This hash define may give a wrong impression, but actually it means to do flat_enc */ - } else { - *to = DESTINATION_OFF_NODE; - } + if (m_MDS_GET_NODE_ID_FROM_ADEST(m_MDS_GET_ADEST) == m_MDS_GET_NODE_ID_FROM_ADEST(adest)) { + *to = DESTINATION_ON_NODE; /* This hash define may give a wrong impression, but actually it means to do flat_enc */ } else { - *to = DESTINATION_ON_NODE; + *to = DESTINATION_OFF_NODE; } } else { *to = DESTINATION_OFF_NODE; diff --git a/osaf/libs/core/mds/mds_dt_tipc.c b/osaf/libs/core/mds/mds_dt_tipc.c --- a/osaf/libs/core/mds/mds_dt_tipc.c +++ b/osaf/libs/core/mds/mds_dt_tipc.c @@ -2035,7 +2035,16 @@ uint32_t mds_mdtm_send_tipc(MDTM_SEND_RE m_MDS_LOG_INFO("MDTM: User Sending Data lenght=%d Fr_svc=%d to_svc=%d\n", len, req->src_svc_id, req->dest_svc_id); - int frag_size = MDTM_NORMAL_MSG_FRAG_SIZE; + // determine fragment limit using a bit in destination archword + int frag_size; + int version = req->msg_arch_word & 0x7; + if (version > 0) { + // normal mode, use TIPC fragmentation + frag_size = TIPC_MAX_USER_MSG_SIZE; + } else { + // old mode, completely skip TIPC fragmentation + frag_size = MDTM_NORMAL_MSG_FRAG_SIZE; + } if (len > frag_size) { /* Packet needs to be fragmented and send */ diff --git a/osaf/libs/core/mds/mds_log.c b/osaf/libs/core/mds/mds_log.c --- a/osaf/libs/core/mds/mds_log.c +++ b/osaf/libs/core/mds/mds_log.c @@ -64,8 +64,8 @@ uint32_t mds_log_init(char *log_file_nam if ((fh = fopen(lf, "a+")) != NULL) { fclose(fh); - log_mds_notify("BEGIN MDS LOGGING| PID=%d|ARCH=%d|64bit=%ld\n", - process_id, MDS_ARCH_TYPE, (long)MDS_WORD_SIZE_TYPE); + log_mds_notify("BEGIN MDS LOGGING| PID=%d|ARCHW=%x|64bit=%ld\n", + process_id, MDS_SELF_ARCHWORD, (long)MDS_WORD_SIZE_TYPE); } return NCSCC_RC_SUCCESS; ------------------------------------------------------------------------------ "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE Instantly run your Selenium tests across 300+ browser/OS combos. Get unparalleled scalability from the best Selenium testing platform available. Simple to use. Nothing to install. Get started now for free." http://p.sf.net/sfu/SauceLabs _______________________________________________ Opensaf-devel mailing list Opensaf-devel@lists.sourceforge.net<mailto:Opensaf-devel@lists.sourceforge.net> https://lists.sourceforge.net/lists/listinfo/opensaf-devel ------------------------------------------------------------------------------ "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE Instantly run your Selenium tests across 300+ browser/OS combos. Get unparalleled scalability from the best Selenium testing platform available. Simple to use. Nothing to install. Get started now for free." http://p.sf.net/sfu/SauceLabs _______________________________________________ Opensaf-devel mailing list Opensaf-devel@lists.sourceforge.net<mailto:Opensaf-devel@lists.sourceforge.net> https://lists.sourceforge.net/lists/listinfo/opensaf-devel ------------------------------------------------------------------------------ "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE Instantly run your Selenium tests across 300+ browser/OS combos. Get unparalleled scalability from the best Selenium testing platform available Simple to use. Nothing to install. Get started now for free." http://p.sf.net/sfu/SauceLabs _______________________________________________ Opensaf-devel mailing list Opensaf-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/opensaf-devel
