Re: [hwloc-devel] hwloc embedding vs libltdl

[Brice Goglin]

Le 08/05/2013 13:09, Jeff Squyres (jsquyres) a écrit :
> 2.  My thoughts were preventing hwloc from going thru some of the pain that 
> OMPI went thru w embedding. Libibverbs has the same problem. If you have 
> middleware that uses plugins that, in turn, uses plugins, it's a bit 
> complicated to support fully static builds  properly (OMPI and hwloc do, but 
> libibverbs doesn't easily, but still, it may require a rebuild of hwloc with 
> enable-static). Also, the problem with opening DSOs in private namespaces 
> still exists; there's no good solution for that (when both the middleware and 
> hwloc use plugins).  

But this issue is only in the embedders (OMPI), not in the embeddee
(hwloc), right?

I can get plugins to work in tests/embedded by adding 2 lines to its
configure.ac (see the attached patch, which also removes your error
message and creates a shared lib containing libhwloc_embedded).

In short, I don't really see what risk we would be taking on the hwloc
side if we keep embedding+plugins possible (and still don't enable
plugins by default).

> 3. Open MPI also get flack for embedding lib ltdl and libevent. Although 
> libltdl now has the built in options for using an external libltdl (which is 
> what the distros use), why go down this road if we don't need to embed 
> libltdl?

OK

Brice

diff --git a/config/hwloc.m4 b/config/hwloc.m4
index 059b72e..818fcc6 100644
--- a/config/hwloc.m4
+++ b/config/hwloc.m4
@@ -1040,12 +1040,6 @@ EOF])
 AS_IF([test "x$enable_plugins" = "xyes" -a "x$hwloc_windows" = "xyes"],
   [AC_MSG_WARN([Plugins not supported on non-native Windows build, plugins support cannot be enabled.])
AC_MSG_ERROR([Cannot continue])])
-# plugins are not supported in embedded mode (indeed, all the LTDL
-# setup stuff is up in hwloc's private configure.ac -- not down
-# here in hwloc.m4)
-AS_IF([test "x$enable_plugins" = "xyes" -a "$hwloc_mode" = "embedded"],
-  [AC_MSG_WARN([Embedded mode not supported with plugins])
-   AC_MSG_ERROR([Cannot continue])])

 AC_ARG_WITH([hwloc-plugins-path],
 		AC_HELP_STRING([--with-hwloc-plugins-path=dir:...],
diff --git a/tests/embedded/Makefile.am b/tests/embedded/Makefile.am
index 9f658f0..eae3c8c 100644
--- a/tests/embedded/Makefile.am
+++ b/tests/embedded/Makefile.am
@@ -10,12 +10,13 @@ AM_CPPFLAGS = $(HWLOC_EMBEDDED_CPPFLAGS)
 TESTS = main

 noinst_PROGRAMS = main
-noinst_LTLIBRARIES = libdo_test.la
+lib_LTLIBRARIES = libdo_test.la

 libdo_test_la_SOURCES = do_test.c
+libdo_test_la_LIBADD = $(HWLOC_EMBEDDED_LDADD) $(HWLOC_EMBEDDED_LIBS)

 main_SOURCES = main.c
-main_LDADD = libdo_test.la $(HWLOC_EMBEDDED_LDADD) $(HWLOC_EMBEDDED_LIBS)
+main_LDADD = libdo_test.la
 main_DEPENDENCIES = libdo_test.la

 EXTRA_DIST = run-embedded-tests.sh
diff --git a/tests/embedded/configure.ac b/tests/embedded/configure.ac
index d697e58..8e57207 100644
--- a/tests/embedded/configure.ac
+++ b/tests/embedded/configure.ac
@@ -19,10 +19,12 @@ cat <

Re: [OMPI devel] [OMPI svn] svn:open-mpi r28456 - trunk

[Thomas Naughton]

Hi Ralph,

On Tue, 7 May 2013, Ralph Castain wrote:


2) Avoid adding "ignored" frameworks to the autogenerated "frameworks.h"
   header file.


This simply applies the same ignored() function that is used elsewhere in
the autogen.pl script for omitting "ignored" MCA directories from the
processing.  This just picks up the ".ompi_ignore" (and/or ".ompi_unignore) 
files.   The intent being that if you ignore a component (subdir) that will

be removed from the list, but you could also remove an entire framework if
you put the ignore file in the top-level of the framework.



That is new - I would suggest not doing that as it behaves differently than
you might expect. The .ompi_ignore in a component prevents that component
from building at all, so it won't ever be opened etc. However, the
framework *must* build the base code no matter what - and that means the
framework will be opened, selected, and closed at the minimum.


I would prefer we keep ompi_ignore cleanly defined. You can ignore all 
components by simply putting --enable-mca-no-build= on your 
configure line.


The intent being that if for whatever reason you ignore a framework in the
"${project}/mca/" space, you will not have it automatically show up in the
project's "frameworks.h" file.

On Tue, 7 May 2013, Ralph Castain wrote:


We use the frameworks.h file to "discover" the frameworks in
ompi_info.  Even if no components are built for that framework,
there still are MCA params relating to the base of that framework.
Sounds silly, I know - but there may be reasons to access those
params - e.g., to set verbosity to verify that no components are
being selected.

I think we need those frameworks to be listed...



Ok, I didn't realize the 'ompi_info' aspect.  Good to know.
However, I think honouring the ignore behavior is good in this case
b/c if you drop an ignore file in a framework, you won't get any
other autogen touches (i.e., no Makefile's are autogenerated).  So
it seems that having no Makefiles but including the framework in the
"framework.h" would break regardless.  Again, this is more of a
safety guard.


Actually, I disagree. As stated above, the framework will *always* build the
base code and be opened, selected, and closed - so you at least need
access to the verbosity parameter so you can verify those operations.
Keeping it in ompi_info is of value.



I guess I misunderstood the scope of use for the ".ompi_ignore" file.  I
thought that it could be placed at the top of the framework and it would
ignore the entire directory.

I just did a quick test with the earlier version of autogen.pl (r28241) and
it does indeed generate the Makefiles for that directory.  So it does seem
reasonable that if autogen.pl processes the directory for Makefile stuff*,
that it should process it for the "frameworks.h" entry.

I'll revert that part of the changeset to previous functionality.

Sorry, my bad,
--tjn

 _
  Thomas Naughton  naught...@ornl.gov
  Research Associate   (865) 576-4184

Re: [hwloc-devel] hwloc embedding vs libltdl

[Jeff Squyres (jsquyres)]

I don't think libltdl has a .pc. :(

Sent from my phone. No type good.

On May 8, 2013, at 2:26 AM, "Brice Goglin" 
> wrote:

Le 08/05/2013 02:47, Jeff Squyres (jsquyres) a écrit :

How's this patch?

The only question I have is: how do we figure out what libraries to put in the 
.pc file in the --disable-shared --enable-static case?

There should be a ltdl.pc for this. But I don't see any. Is there a standard 
way to extract this line from ltdl.la ?
dependency_libs=' -ldl'

How about we do not support plugins when --enable-static is given?

Brice







On May 7, 2013, at 8:24 PM, Samuel Thibault 
 wrote:



Jeff Squyres (jsquyres), le Wed 08 May 2013 02:21:02 +0200, a écrit :


On May 7, 2013, at 6:25 PM, Brice Goglin 
 wrote:



I don't have anything against this. What was the reason for not using
the default/system libltdl in OMPI? libtool was buggy when you started
using it?



I neglected to answer this.

Yes, plus libltdl grew new functionality that we needed (global/local symbol 
visibility).

We might be getting to the point soon where we can rely on the installed 
libltdl to be new enough everywhere, but we haven't had that conversation.


We could already check that the installed version is new enough for our
needs.

Samuel
___
hwloc-devel mailing list
hwloc-de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-devel







___
hwloc-devel mailing list
hwloc-de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-devel

___
hwloc-devel mailing list
hwloc-de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-devel

Re: [hwloc-devel] hwloc embedding vs libltdl

[Jeff Squyres (jsquyres)]

1.  Ok. 
2.  My thoughts were preventing hwloc from going thru some of the pain that 
OMPI went thru w embedding. Libibverbs has the same problem. If you have 
middleware that uses plugins that, in turn, uses plugins, it's a bit 
complicated to support fully static builds  properly (OMPI and hwloc do, but 
libibverbs doesn't easily, but still, it may require a rebuild of hwloc with 
enable-static). Also, the problem with opening DSOs in private namespaces still 
exists; there's no good solution for that (when both the middleware and hwloc 
use plugins).  

>From hwlocs perspective, the middleware author can fix the static build 
>option, but I figured: why even go here?

3. Open MPI also get flack for embedding lib ltdl and libevent. Although 
libltdl now has the built in options for using an external libltdl (which is 
what the distros use), why go down this road if we don't need to embed libltdl?

Sent from my phone. No type good. 

On May 8, 2013, at 2:53 AM, "Brice Goglin"  wrote:

> Actually, are we going too far here?
> 
> 1) The original problem was that embedding couldn't build (it couldn't
> even autoreconf) because of the embedded ltdl. Not because of plugins
> being enabled. That's fixed with my patch and with yours.
> tests/embedded/ runs fine now.
> 
> 2) Now, we're disallowing plugins entirely in the embedded case too.
> That's kind of orthogonal to (1). I don't think we currently have a
> single line of code to support this. If people want plugins and
> embedded, thay will need to setup ltdl in their own configure. I don't
> see any reason to prevent this. We may have users wanting this one day,
> so I think we should remove the current error message.
> 
> 3) And we're disallowing included ltdl too. I am actually not against
> this one. We don't use advanced ltdl features, and I don't plan to
> change the plugin management code. So the installed ltdl should be fine.
> But now that (1) is fixed, there's no direct reason to do (3) immediately.
> 
> Brice
> 
> ___
> hwloc-devel mailing list
> hwloc-de...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-devel

Re: [hwloc-devel] hwloc embedding vs libltdl

[Brice Goglin]

Actually, are we going too far here?

1) The original problem was that embedding couldn't build (it couldn't
even autoreconf) because of the embedded ltdl. Not because of plugins
being enabled. That's fixed with my patch and with yours.
tests/embedded/ runs fine now.

2) Now, we're disallowing plugins entirely in the embedded case too.
That's kind of orthogonal to (1). I don't think we currently have a
single line of code to support this. If people want plugins and
embedded, thay will need to setup ltdl in their own configure. I don't
see any reason to prevent this. We may have users wanting this one day,
so I think we should remove the current error message.

3) And we're disallowing included ltdl too. I am actually not against
this one. We don't use advanced ltdl features, and I don't plan to
change the plugin management code. So the installed ltdl should be fine.
But now that (1) is fixed, there's no direct reason to do (3) immediately.

Brice

Re: [hwloc-devel] [hwloc-svn] svn:hwloc r5606 - trunk

[Brice Goglin]

We actually used C++ during make check (we test the C++ build of
doc/hwloc-hello.c)
(got a build failure report from https://ci.inria.fr/hwloc/)

Brice



Le 08/05/2013 02:27, svn-commit-mai...@open-mpi.org a écrit :
> Author: jsquyres (Jeff Squyres)
> Date: 2013-05-07 20:27:25 EDT (Tue, 07 May 2013)
> New Revision: 5606
> URL: https://svn.open-mpi.org/trac/hwloc/changeset/5606
>
> Log:
> Revert r5604 -- it's redundant with LT_LANG([C]).
>
> Text files modified: 
>trunk/configure.ac | 4 
>1 files changed, 0 insertions(+), 4 deletions(-)
>
> Modified: trunk/configure.ac
> ==
> --- trunk/configure.acTue May  7 20:18:05 2013(r5605)
> +++ trunk/configure.ac2013-05-07 20:27:25 EDT (Tue, 07 May 2013)  
> (r5606)
> @@ -166,10 +166,6 @@
>  # Compiler support -- we don't need that stuff.
>  AM_ENABLE_SHARED
>  AM_DISABLE_STATIC
> -# Tell libtool that we don't need Fortran or C++ support.
> -FC=no
> -F77=no
> -CXX=no
>  AM_PROG_LIBTOOL([dlopen win32-dll])
>  LT_LANG([C])
>  LT_CONFIG_LTDL_DIR([src/libltdl])
> ___
> hwloc-svn mailing list
> hwloc-...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-svn

Re: [hwloc-devel] hwloc embedding vs libltdl

[Brice Goglin]

Le 08/05/2013 02:47, Jeff Squyres (jsquyres) a écrit :
> How's this patch?
>
> The only question I have is: how do we figure out what libraries to put in 
> the .pc file in the --disable-shared --enable-static case?

There should be a ltdl.pc for this. But I don't see any. Is there a
standard way to extract this line from ltdl.la ?
dependency_libs=' -ldl'

How about we do not support plugins when --enable-static is given?

Brice




>
>
> On May 7, 2013, at 8:24 PM, Samuel Thibault  wrote:
>
>> Jeff Squyres (jsquyres), le Wed 08 May 2013 02:21:02 +0200, a écrit :
>>> On May 7, 2013, at 6:25 PM, Brice Goglin  wrote:
>>>
 I don't have anything against this. What was the reason for not using
 the default/system libltdl in OMPI? libtool was buggy when you started
 using it?
>>>
>>> I neglected to answer this.
>>>
>>> Yes, plus libltdl grew new functionality that we needed (global/local 
>>> symbol visibility).
>>>
>>> We might be getting to the point soon where we can rely on the installed 
>>> libltdl to be new enough everywhere, but we haven't had that conversation.
>> We could already check that the installed version is new enough for our
>> needs.
>>
>> Samuel
>> ___
>> hwloc-devel mailing list
>> hwloc-de...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-devel
>
>
>
> ___
> hwloc-devel mailing list
> hwloc-de...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-devel