Hi Alex,
I can fetch your code. Will try it and reply if any concern…
* 430dbfdd4 - (HEAD -> ticket-70) amf: fix crash of amfd [#70] (5 days ago)
<Alex Jones>
* bf998b543 - amfd: don't start contained if it is in locked-in state [#70] (3
weeks ago) <Alex Jones>
* 8b8f2fe30 - amf: fix restart of container app and check container model [#70]
(6 weeks ago) <Alex Jones>
* 9c9f7e04c - amf: add support for container/contained [#70] (6 weeks ago)
<Alex Jones>
* cf9d75653 - amf: add support for container/contained [#70] (6 weeks ago)
<Alex Jones>
* 199d81e0e - amf: add support for container/contained [#70] (6 weeks ago)
<Alex Jones>
* d53a063d3 - amfnd: add support for container/contained [#70] (6 weeks ago)
<Alex Jones>
* 308b8d733 - amfd: add support for container/contained [#70] (6 weeks ago)
<Alex Jones>
Best Regards,
Thuan
From: Jones, Alex <ajo...@rbbn.com>
Sent: Wednesday, September 26, 2018 12:10 AM
To: Tran Thuan <thuan.t...@dektech.com.au>; nagen...@hasolutions.in; 'Gary Lee'
<gary....@dektech.com.au>; hans.nordeb...@ericsson.com;
ravisekhar.ko...@oracle.com
Cc: opensaf-devel@lists.sourceforge.net
Subject: Re: [devel] [PATCH 1/1] amf: add support for container/contained [#70]
Hi Thuan,
I pushed the latest ticket-70 to my review repo.
For these concerns you have try cleaning out your code, and pull the latest
from my review repo for ticket-70. Contained should not get instantiated until
you unlock-in the contained su. I'm guessing you are missing some patches...
Let me know if you still see these issues.
Alex
On 09/24/2018 12:24 AM, Tran Thuan wrote:
_____
NOTICE: This email was received from an EXTERNAL sender
_____
Hi Alex,
Can you send out new version of review?
Then I can fetch latest version from your review repo.
Some more concerns without using your additional patch plm-70 yet.
1. After unlock-in, unlock container, contained got instantiated but admin
state is still locked-in
safSu=SU1,safSg=Contained_2N,safApp=Contained_2N
saAmfSUAdminState=LOCKED-INSTANTIATION(3)
saAmfSUOperState=ENABLED(1)
saAmfSUPresenceState=INSTANTIATED(3)
saAmfSUReadinessState=OUT-OF-SERVICE(1)
--
safSu=SU1,safSg=Container,safApp=Container
saAmfSUAdminState=UNLOCKED(1)
saAmfSUOperState=ENABLED(1)
saAmfSUPresenceState=INSTANTIATED(3)
saAmfSUReadinessState=IN-SERVICE(2)
2. Also, I don't see how to make contained get assignment?
root@SC-1:~# amf-adm unlock safSu=SU1,safSg=Contained_2N,safApp=Contained_2N
error - saImmOmAdminOperationInvoke_2 admin-op RETURNED:
SA_AIS_ERR_BAD_OPERATION (20)
error-string: State transition invalid, state 3, op 1
root@SC-1:~# amf-adm unlock-in safSu=SU1,safSg=Contained_2N,safApp=Contained_2N
error - saImmOmAdminOperationInvoke_2 admin-op RETURNED:
SA_AIS_ERR_BAD_OPERATION (20)
error-string: Can't instantiate
'safSu=SU1,safSg=Contained_2N,safApp=Contained_2N', whose presence state is '3'
3. When I kill -9 `pidof amf_container_demo`, it seems recovery fail?
2018-09-24 10:59:59.273 SC-1 osafamfnd[478]: NO
'safSu=SU1,safSg=Contained_2N,safApp=Contained_2N' component restart probation
timer started (timeout: 4000000000 ns)
2018-09-24 10:59:59.273 SC-1 osafamfnd[478]: NO Restarting a component of
'safSu=SU1,safSg=Contained_2N,safApp=Contained_2N' (comp restart count: 1)
2018-09-24 10:59:59.273 SC-1 osafamfnd[478]: NO
'safComp=Contained_1,safSu=SU1,safSg=Contained_2N,safApp=Contained_2N' faulted
due to 'avaDown' : Recovery is 'componentRestart'
2018-09-24 10:59:59.274 SC-1 osafamfnd[478]: ER ncsmds_api for 0 FAILED,
dest=2010f00000280
2018-09-24 10:59:59.274 SC-1 osafamfnd[478]: NO Component CLC fsm exited with
error for
comp:safComp=Contained_1,safSu=SU1,safSg=Contained_2N,safApp=Contained_2N
2018-09-24 11:00:03.295 SC-1 osafamfnd[478]: NO
'safSu=SU1,safSg=Contained_2N,safApp=Contained_2N' Component or SU restart
probation timer expired
safSu=SU1,safSg=Contained_2N,safApp=Contained_2N
saAmfSUAdminState=LOCKED-INSTANTIATION(3)
saAmfSUOperState=ENABLED(1)
saAmfSUPresenceState=INSTANTIATED(3)
saAmfSUReadinessState=OUT-OF-SERVICE(1)
--
safSu=SU1,safSg=Container,safApp=Container
saAmfSUAdminState=UNLOCKED(1)
saAmfSUOperState=ENABLED(1)
saAmfSUPresenceState=INSTANTIATED(3)
saAmfSUReadinessState=IN-SERVICE(2)
safSi=Contained_2N_1,safApp=Contained_2N
saAmfSIAdminState=UNLOCKED(1)
saAmfSIAssignmentState=UNASSIGNED(1)
safSi=Container,safApp=Container
saAmfSIAdminState=UNLOCKED(1)
saAmfSIAssignmentState=PARTIALLY_ASSIGNED(3)
Best Regards,
Thuan
From: Jones, Alex <mailto:ajo...@rbbn.com> <ajo...@rbbn.com>
Sent: Saturday, September 22, 2018 1:01 AM
To: Tran Thuan <mailto:thuan.t...@dektech.com.au> <thuan.t...@dektech.com.au>;
nagen...@hasolutions.in <mailto:nagen...@hasolutions.in> ; 'Gary Lee'
<mailto:gary....@dektech.com.au> <gary....@dektech.com.au>;
hans.nordeb...@ericsson.com <mailto:hans.nordeb...@ericsson.com> ;
ravisekhar.ko...@oracle.com <mailto:ravisekhar.ko...@oracle.com>
Cc: opensaf-devel@lists.sourceforge.net
<mailto:opensaf-devel@lists.sourceforge.net>
Subject: Re: [devel] [PATCH 1/1] amf: add support for container/contained [#70]
Hi Tran,
Here is a patch you can overlay to fix this crash.
Alex
On 09/21/2018 06:31 AM, Tran Thuan wrote:
_____
NOTICE: This email was received from an EXTERNAL sender
_____
Hi Alex,
I think you need update samples/amf/container/README.
Also when I try following steps, AMFD crash.
root@SC-1:/opt/amf_demo# immcfg -f AppConfig-container.xml
root@SC-1:/opt/amf_demo# immcfg -f AppConfig-contained-2N.xml
root@SC-1:/opt/amf_demo# amf-adm unlock-in
safSu=SU1,safSg=Container,safApp=Container
root@SC-1:/opt/amf_demo# amf-adm unlock
safSu=SU1,safSg=Container,safApp=Container
root@SC-1:/opt/amf_demo# amf-adm lock
safSu=SU1,safSg=Container,safApp=Container
2018-09-21 16:58:38.338 SC-1 osafamfnd[512]: NO Assigning
'safSi=Container,safApp=Container' ACTIVE to
'safSu=SU1,safSg=Container,safApp=Container'
2018-09-21 16:58:38.339 SC-1 amf_container_demo[739]: csi set callback for
comp: safComp=Container,safSu=SU1,safSg=Container,safApp=Container
2018-09-21 16:58:38.339 SC-1 amf_container_demo[739]: CSI Set - add
'safCsi=Container1,safSi=Container,safApp=Container' HAState Active
2018-09-21 16:58:38.339 SC-1 amf_container_demo[739]: name: Contained1,
value: AAAA
2018-09-21 16:58:38.339 SC-1 amf_container_demo[739]: name: Contained1,
value: BBBB
2018-09-21 16:58:38.340 SC-1 osafamfnd[512]: NO Assigned
'safSi=Container,safApp=Container' ACTIVE to
'safSu=SU1,safSg=Container,safApp=Container'
2018-09-21 16:58:38.413 SC-1 osafamfnd[512]: NO
'safSu=SU1,safSg=Contained_2N,safApp=Contained_2N' Presence State
UNINSTANTIATED => INSTANTIATING
2018-09-21 16:58:38.414 SC-1 amf_container_demo[739]: =====Contained
Instantiate Callback====>
2018-09-21 16:58:38.414 SC-1 amf_container_demo[739]:
comp:safComp=Contained_1,safSu=SU1,safSg=Contained_2N,safApp=Contained_2N
2018-09-21 16:58:38.414 SC-1 amf_container_demo[739]: responding with
TRY_AGAIN
2018-09-21 16:58:38.417 SC-1 amf_container_demo[739]:
<===========================================
2018-09-21 16:58:38.418 SC-1 amf_container_demo[739]: =====Contained Clean
Up Callback====>
2018-09-21 16:58:38.418 SC-1 amf_container_demo[739]:
comp:safComp=Contained_1,safSu=SU1,safSg=Contained_2N,safApp=Contained_2N
2018-09-21 16:58:38.418 SC-1 amf_container_demo[739]:
<===========================================
2018-09-21 16:58:38.422 SC-1 amf_container_demo[739]: =====Contained
Instantiate Callback====>
2018-09-21 16:58:38.422 SC-1 amf_container_demo[739]:
comp:safComp=Contained_1,safSu=SU1,safSg=Contained_2N,safApp=Contained_2N
2018-09-21 16:58:38.422 SC-1 amf_container_demo[739]:
<===========================================
2018-09-21 16:58:38.424 SC-1 osafamfnd[512]: NO
'safSu=SU1,safSg=Contained_2N,safApp=Contained_2N' Presence State
INSTANTIATING => INSTANTIATED
2018-09-21 17:19:07.776 SC-1 osafamfnd[512]: ER AMFD has unexpectedly
crashed. Rebooting node
2018-09-21 17:19:07.776 SC-1 osafamfnd[512]: Rebooting OpenSAF NodeId =
131343 EE Name = , Reason: AMFD has unexpectedly crashed. Rebooting node,
OwnNodeId = 131343, SupervisionTime = 60
2018-09-21 17:19:07.778 SC-1 osafimmnd[438]: NO Implementer locally
disconnected. Marking it as doomed 5 <23, 2010f> (safAmfService)
2018-09-21 17:19:07.799 SC-1 opensaf_reboot: Rebooting local node;
timeout=60
Best Regards,
Thuan
-----Original Message-----
From: nagen...@hasolutions.in <mailto:nagen...@hasolutions.in>
<mailto:nagen...@hasolutions.in> <nagen...@hasolutions.in>
Sent: Friday, September 7, 2018 12:57 PM
To: Alex Jones <mailto:ajo...@rbbn.com> <ajo...@rbbn.com>; Gary Lee
<mailto:gary....@dektech.com.au> <gary....@dektech.com.au>;
hans.nordeb...@ericsson.com <mailto:hans.nordeb...@ericsson.com> ;
ravisekhar.ko...@oracle.com <mailto:ravisekhar.ko...@oracle.com>
Cc: opensaf-devel@lists.sourceforge.net
<mailto:opensaf-devel@lists.sourceforge.net>
Subject: Re: [devel] [PATCH 1/1] amf: add support for container/contained
[#70]
Hi Alex,
Thanks for the patch.
From my side Ack. I wish that I could have tested the following area (I
assume you would have covered it):
- Headless enabled test cases
- CSI Dep, SI Dep testing(in 2N red model)
- Combinations of Admin operations on Container and contained (in all 5 red
models for contained) with fault scenarios.
- Escalations of contained components.
Thanks,
Nagendra, 91-9866424860
High Availability Solutions Pvt. Ltd. (www.hasolutions.in
<https://protect-us.mimecast.com/s/EO45CQWOyWhZZ5Wcx7jbh?domain=hasolutions.in>
)
- OpenSAF Support and Services
--------- Original Message --------- Subject: Re: [PATCH 1/1] amf: add
support for container/contained [#70]
From: "Alex Jones" <mailto:ajo...@rbbn.com> <ajo...@rbbn.com>
Date: 9/6/18 8:52 pm
To: nagen...@hasolutions.in <mailto:nagen...@hasolutions.in> , "Gary Lee"
<mailto:gary....@dektech.com.au> <gary....@dektech.com.au>,
hans.nordeb...@ericsson.com <mailto:hans.nordeb...@ericsson.com> ,
ravisekhar.ko...@oracle.com <mailto:ravisekhar.ko...@oracle.com>
Cc: opensaf-devel@lists.sourceforge.net
<mailto:opensaf-devel@lists.sourceforge.net>
Hi Nagu,
Here's a patch that fixes your issue in test #1.
For the other code review issues, is it OK if I just add them when I
push the final patch. Or do you want to review them now?
Alex
On 08/30/2018 01:44 AM, nagen...@hasolutions.in
<mailto:nagen...@hasolutions.in> wrote:
NOTICE: This email was received from an EXTERNAL sender
Hi Alex,
Thanks for your response.
For Test #2, I had configured all SUs on the single node SC-1. So, 2
container SUs and 2 contained SUs are on the same node. In such cases, we
can have the implementation as having only one SU of that node(higher rank
SUs may be) to be the container for all the contained SUs of that node.
Thanks,
Nagendra, 91-9866424860
High Availability Solutions Pvt. Ltd. (www.hasolutions.in
<https://protect-us.mimecast.com/s/EO45CQWOyWhZZ5Wcx7jbh?domain=hasolutions.in>
)
- OpenSAF Support and Services
--------- Original Message --------- Subject: Re: [PATCH 1/1] amf: add
support for container/contained [#70]
From: "Alex Jones" <mailto:ajo...@rbbn.com> <ajo...@rbbn.com>
Date: 8/29/18 9:29 pm
To: nagen...@hasolutions.in <mailto:nagen...@hasolutions.in> , "Gary Lee"
<mailto:gary....@dektech.com.au> <gary....@dektech.com.au>,
hans.nordeb...@ericsson.com <mailto:hans.nordeb...@ericsson.com> ,
ravisekhar.ko...@oracle.com
Cc: opensaf-devel@lists.sourceforge.net
<mailto:opensaf-devel@lists.sourceforge.net>
Hi Nagu,
I have a fix for your issue test #1. I will send out a patch along with
changes for code review #1 and #2.
For issue test #2, I think this needs to be handled in the
configuration. In this case because there is no explicit node set for the
contained SUs, su.cc:map_su_to_node will assign a node in the node group.
The code is assigning it to SC-2 in this case, because another SU has been
assigned to SC-1, even though there is no container on SC-2. I'm not sure
how we can get around this without explicitly setting the contained host
node in the configuration. Since the container csi has not yet been
assigned, we can't map it to a container, and so we can't figure out which
container we should be on the same node as. Am I right here?
Alex
On 08/28/2018 09:56 AM, nagen...@hasolutions.in
<mailto:nagen...@hasolutions.in> wrote:
NOTICE: This email was received from an EXTERNAL sender
Hi Alex,
Code review:
1. Header for few functions are missing.
2. Clc.cc: Need to add '0' in place avnd_comp_clc_inst_try_again_hdler in
other fsm states.
Testing:
1. Uploaded AppConfig-container.xml and AppConfig-contained-2N.xml
Performed:
amf-adm unlock-in safSu=SU1,safSg=Container,safApp=Container
amf-adm unlock safSu=SU1,safSg=Container,safApp=Container
Even I don't perform the following, the contained components are
instantiated.
amf-adm unlock-in safSu=SU1,safSg=Contained_2N,safApp=Contained_2N
amf-adm unlock safSu=SU1,safSg=Contained_2N,safApp=Contained_2N
Aug 28 19:15:11 nags-VirtualBox osafamfnd[28278]: NO
'safSu=SU1,safSg=Contained_2N,safApp=Contained_2N' Presence State
UNINSTANTIATED => INSTANTIATING
immlist safSu=SU1,safSg=Contained_2N,safApp=Contained_2N will show
saAmfSUPresenceState 3(instantiated) and saAmfSUAdminState 3(locked-in)
Now further admin operation on
safSu=SU1,safSg=Contained_2N,safApp=Contained_2N will fail:
root@nags-VirtualBox:/home/nags/views/ajones-review/samples/amf/container#
amf-adm unlock-in safSu=SU1,safSg=Contained_2N,safApp=Contained_2N
error - saImmOmAdminOperationInvoke_2 admin-op RETURNED:
SA_AIS_ERR_BAD_OPERATION (20)
error-string: Can't instantiate
'safSu=SU1,safSg=Contained_2N,safApp=Contained_2N', whose presence state is
'3'
2.This is related to Specs 6.2.2 Assignment of the Container CSI: "If there
are multiple container components on a node which have the active HA state
for a particular container CSI, and one or more service units on the same
node whose contained components are configured with the same container CSI,
it is implementation- defined how the Availability Management Framework
selects container components to handle the life cycle of the contained
components of these service units.
However, all contained components of a service unit must have the same
associated container component."
Uploaded AppConfig-container.xml and AppConfig-contained-2N.xml with once
difference that all SUs of container and contained are configured on SC-1.
Perform the following operations, but
safSu=SU2,safSg=Contained_2N,safApp=Contained_2N will not get assignments.
amf-adm unlock-in safSu=SU1,safSg=Contained_2N,safApp=Contained_2N
amf-adm unlock safSu=SU1,safSg=Contained_2N,safApp=Contained_2N
amf-adm unlock-in safSu=SU2,safSg=Contained_2N,safApp=Contained_2N
amf-adm unlock safSu=SU2,safSg=Contained_2N,safApp=Contained_2N
amf-adm unlock-in safSu=SU1,safSg=Container,safApp=Container
amf-adm unlock safSu=SU1,safSg=Container,safApp=Container
amf-adm unlock-in safSu=SU2,safSg=Container,safApp=Container
amf-adm unlock safSu=SU2,safSg=Container,safApp=Container
root@nags-VirtualBox:/home/nags/views/ajones-review/samples/amf/container#
amf-state siass
safSISU=safSu=SC-1\,safSg=NoRed\,safApp=OpenSAF,safSi=NoRed1,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1)
safSISU=safSu=SC-1\,safSg=2N\,safApp=OpenSAF,safSi=SC-2N,safApp=OpenSAF
saAmfSISUHAState=ACTIVE(1)
saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1)
safSISU=safSu=SU1\,safSg=Contained_2N\,safApp=Contained_2N,safSi=Contained_2
N_1,safApp=Contained_2N
saAmfSISUHAState=ACTIVE(1)
saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1)
safSISU=safSu=SU1\,safSg=Container\,safApp=Container,safSi=Container,safApp=
Container
saAmfSISUHAState=ACTIVE(1)
saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1)
safSISU=safSu=SU2\,safSg=Container\,safApp=Container,safSi=Container,safApp=
Container
saAmfSISUHAState=ACTIVE(1)
saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1)
I will do further testing.
The documentation need to be done if you haven't tested :
- Headless enabled
- CSI Dep, SI Dep testimg
- Etc.
Thanks,
Nagendra, 91-9866424860
High Availability Solutions Pvt. Ltd. (www.hasolutions.in
<https://protect-us.mimecast.com/s/EO45CQWOyWhZZ5Wcx7jbh?domain=hasolutions.in>
)
- OpenSAF Support and Services
--------- Original Message --------- Subject: Re: [PATCH 1/1] amf: add
support for container/contained [#70]
From: "Alex Jones" <mailto:ajo...@rbbn.com> <ajo...@rbbn.com>
Date: 8/15/18 11:10 pm
To: "Gary Lee" <mailto:gary....@dektech.com.au> <gary....@dektech.com.au>,
hans.nordeb...@ericsson.com <mailto:hans.nordeb...@ericsson.com> ,
ravisekhar.ko...@oracle.com <mailto:ravisekhar.ko...@oracle.com> ,
nagen...@hasolutions.in <mailto:nagen...@hasolutions.in>
Cc: opensaf-devel@lists.sourceforge.net
<mailto:opensaf-devel@lists.sourceforge.net>
G'day Gary,
I see you were adding the XML file dynamically with "immcfg -f". I
hadn't tried that. I hadn't tried killing the sample app, either.
Here is a patch that should fix both issues. Apply it on top of the
latest big one I sent.
Alex
On 08/13/2018 10:37 PM, Gary Lee wrote:
NOTICE: This email was received from an EXTERNAL sender
Hi Alex
I modified AppConfig-container.xml and changed saAmfSgtRedundancyModel from
4 (NwayAct) to 1 (2N).
The xml still loads and I could unlock, resulting in:
root@SC-1:/var/log# immlist safVersion=1,safSgType=Container
Name Type Value(s)
========================================================================
safVersion SA_STRING_T
safVersion=1
saAmfSgtValidSuTypes SA_NAME_T
safVersion=1,safSuType=Container (32)
saAmfSgtRedundancyModel SA_UINT32_T 1 (0x1)
safSISU=safSu=SU2\,safSg=Container\,safApp=Container,safSi=Container,safApp=
Container
saAmfSISUHAState=STANDBY(2)
saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1)
safSISU=safSu=SU1\,safSg=Container\,safApp=Container,safSi=Container,safApp=
Container
saAmfSISUHAState=ACTIVE(1)
saAmfSISUHAReadinessState=READY_FOR_ASSIGNMENT(1)
Also, have you tried killing the amf_container_demo binary?
Thanks
Gary
On 14/08/18 05:00, Alex Jones wrote:
Hi Gary,
I just resubmitted a new patch which breaks out the different
components, and addresses the other comments here. But, #2 (rejecting all
but NWay-active for container) should already be in there. Is there a
specific test you ran that didn't work?
Alex
On 08/13/2018 02:43 AM, Gary Lee wrote:
NOTICE: This email was received from an EXTERNAL sender
Hi Alex
Some initial comments:
0. Is it possible to split up the patch into amfd / amfnd / common /
samples. Just makes it easier to reply inline.
1. Please compile the container demo by default, and make
amf_container_script world executable.
Eg.
diff --git a/samples/amf/Makefile.am b/samples/amf/Makefile.am index
447dedd..7ebf9c3 100644
--- a/samples/amf/Makefile.am
+++ b/samples/amf/Makefile.am
@@ -19,5 +19,5 @@ include $(top_srcdir)/Makefile.common
MAINTAINERCLEANFILES = Makefile.in
-SUBDIRS = sa_aware non_sa_aware wrapper proxy api_demo +SUBDIRS =
sa_aware non_sa_aware wrapper proxy api_demo container
diff --git a/samples/amf/container/amf_container_script
b/samples/amf/container/amf_container_script
old mode 100644
new mode 100755
diff --git a/samples/configure.ac b/samples/configure.ac index
7cf803e..9765d54 100644
--- a/samples/configure.ac
+++ b/samples/configure.ac
@@ -67,6 +67,7 @@ AC_CONFIG_FILES([ \
amf/wrapper/Makefile \
amf/proxy/Makefile \
amf/api_demo/Makefile \
+ amf/container/Makefile \
cpsv/Makefile \
cpsv/ckpt_demo/Makefile \
cpsv/ckpt_track_demo/Makefile \
2. We should probably reject CCBs that set saAmfSgtRedundancyModel to
anything other than NWayActive, for Containers.
3. Do we need to bump the msg format version to
AVSV_AVD_AVND_MSG_FMT_VER_8? An old amfnd will assert if it gets an
AVSV_D2N_CONTAINED_SU_MSG_INFO msg.
Thanks
Gary
----------------------------------------------------------------------------
--
Check out the vibrant tech community on one of the world's most engaging
tech sites, Slashdot.org! http://sdm.link/slashdot
<https://protect-us.mimecast.com/s/qqUZCR6Lz6CZZALcN2_lT?domain=sdm.link>
_______________________________________________
Opensaf-devel mailing list
Opensaf-devel@lists.sourceforge.net
<mailto:Opensaf-devel@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/opensaf-devel
<https://protect-us.mimecast.com/s/rLi0CVONEOUQQDAtyfptK?domain=lists.sourceforge.net>
_______________________________________________
Opensaf-devel mailing list
Opensaf-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/opensaf-devel
