Re: [hwloc-devel] backends and plugins
Brice Goglin, le Wed 22 Aug 2012 07:52:07 +0200, a écrit : > Le 21/08/2012 21:18, Samuel Thibault a écrit : > > Brice Goglin, le Tue 21 Aug 2012 18:49:48 +0200, a écrit : > >> 1) We load plugins and list existing components once per topology. We > >> should do it only once per process. But that requires some locking in > >> case multiple topologies are loaded simultaneously, which means we need > >> thread-safety. Do we want pthread_mutex() for this? > > I'd say so. We can test in configure.ac whether -lpthread is really > > needed for that (it is not on most systems, which optimizes things away > > in non-libpthread cases). > > So pthread_mutex() is always available, at least with -lpthread, on all > platforms we support? IIRC yes. For windows that needs a recent enough version of mingw, but that should be fine. Samuel
Re: [hwloc-devel] backends and plugins
Le 21/08/2012 21:18, Samuel Thibault a écrit : > Brice Goglin, le Tue 21 Aug 2012 18:49:48 +0200, a écrit : >> 1) We load plugins and list existing components once per topology. We >> should do it only once per process. But that requires some locking in >> case multiple topologies are loaded simultaneously, which means we need >> thread-safety. Do we want pthread_mutex() for this? > I'd say so. We can test in configure.ac whether -lpthread is really > needed for that (it is not on most systems, which optimizes things away > in non-libpthread cases). So pthread_mutex() is always available, at least with -lpthread, on all platforms we support? I am asking this because "shared" component registration (right after shared loading of plugins) needs mutex too. If there's no mutex, I can disable plugin support. But nothing would works if I disable component registration too. Brice
Re: [hwloc-devel] backends and plugins
Brice Goglin, le Tue 21 Aug 2012 18:49:48 +0200, a écrit : > 1) We load plugins and list existing components once per topology. We > should do it only once per process. But that requires some locking in > case multiple topologies are loaded simultaneously, which means we need > thread-safety. Do we want pthread_mutex() for this? I'd say so. We can test in configure.ac whether -lpthread is really needed for that (it is not on most systems, which optimizes things away in non-libpthread cases). > 2) Some internal functions are now exported to plugins. Do we want a > special namespace? "hwloc__" instead of "hwloc_"? Is there anyway to > make them visibile to plugins but not to applications? Not that I know of. > 3) I currently use the system libltdl. People usually ask libtool to > copy it's libtldl implementation inside the project source at autogen to > make sure it's compatible/working. So we won't have libltdl in SVN, > we'll have it in tarballs, and we'll build it (with a sub-configure) if > plugins are enabled. Is anybody against this? It looks fine to me. Samuel
Re: [hwloc-devel] backends and plugins
I implemented most of this in the component branch. The changes in the core look heavy but it's not too bad in the end. Now, each old backend registers a "component" that can be instantiated into a "backend" structure depending on the topology configuration. By default, you get your OS backend and the PCI backend. But you could also get a XML backend instead for instance. Each backend may contain a discover() callback (doing the main discovery, e.g. initial discovery through the OS) and a notify_new_object() callback to react when somebody else discovers something (currently only used by Linux to add OS devices when the PCI backend adds PCI devices). The list of components to create is defined by configure, depending on the OS, available libraries, and on plugin configuration. There are two types of plugins. "core" (for the above "components/backends") and "xml" (for libxml2 and nolibxml implementations behind the XML backend). At configure time, we can ask to build some components as plugins. Only "core-libpci" and "xml-libxml" for now (others do not depend on any external lib). Those components will be installed in $(libdir)/hwloc and loaded in topology_init(). Plugin are entirely disabled by default (even the core support is disabled) so it shouldn't break much. Testing with and without plugins didn't show any problem so far. Things to improve: 1) We load plugins and list existing components once per topology. We should do it only once per process. But that requires some locking in case multiple topologies are loaded simultaneously, which means we need thread-safety. Do we want pthread_mutex() for this? 2) Some internal functions are now exported to plugins. Do we want a special namespace? "hwloc__" instead of "hwloc_"? Is there anyway to make them visibile to plugins but not to applications? 3) I currently use the system libltdl. People usually ask libtool to copy it's libtldl implementation inside the project source at autogen to make sure it's compatible/working. So we won't have libltdl in SVN, we'll have it in tarballs, and we'll build it (with a sub-configure) if plugins are enabled. Is anybody against this? Brice Le 07/08/2012 11:02, Brice Goglin a écrit : > Antoine Rougier finished his internship recently so here's a summary of > what he did in the "backends" branch. For the record, the goal of his > work was to explore how to change our backends into proper plugins so > that we can easily avoid hard dependencies between the hwloc core and > external libraries such as CUDA, libpci, ... > > > He defined three types of backends: > * "Base" is a normal OS backend such as Linux/Solaris/... This guy > manages PU, Cores, Sockets, ... Only one of these can be used at the > same time. > * "Additional" is what is added to "Base" backends. This is libpci, and > things added inside PCI devices (CUDA, ...). Multiple of these may be > used at the same time. They are likely invoked sequentially after the > Base backend discovery. > * "Global" is a special case for XML, Synthetic and Custom, which > replaces both Base and Additional. It's not clear that this type will be > needed in the end, "Base" with a special flag to disable "Additional" > backends might be enough. > > During init(), all existing backends are placed on a list of "available" > backends. And a list of "enabled" backends is initialized to empty. > Between init() and load(), calling set_xml(), set_synthetic(), > set_fsroot(), ... will append some backends to the "enabled" list > (actually two lists, one for Base, one for Additional). > During load(), we check whether some Base backends are "enabled". > Otherwise we enable the default one of the current OS. Then we actually > discover things using the (unique) enabled Base backend followed by all > enabled Additional backends. > > > > Aside from the main "discover" callback, backends may also define some > callbacks to be invoked when new object are created. The main example is > Linux creating "OS devices" when a new "PCI device" is added by the PCI > backend. CUDA could use that too to fill GPU PCI devices. This is not > strictly needed since adding these devices could still be done later, > once the PCI backend is done. We'll see. > > I'll work on putting all this in a branch soon. > > > > All this is about making interaction between backends nicer. Once this > is done, we'll be able to make actual plugins out of all this. One > problem that will come is that some backends are almost directly used > from outside the core. For instance exporting a topology to XML is a > public API call going down to XML plugin. lstopo and hwloc-ps using > Linux-specific tid_get_cpubind() calls causes similar problems. > > Instead of allowing random API calls into plugin internals, we could > keep these backends internal, i.e. not making them plugins. At least for > OS backends, it makes sense. "synthetic" and "custom" also have no > reason to be pluginified either, they depend on nothing. > > XML
Re: [hwloc-devel] backends and plugins
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 07/08/12 19:02, Brice Goglin wrote: > Aside from the main "discover" callback, backends may also define > some callbacks to be invoked when new object are created. The main > example is Linux creating "OS devices" when a new "PCI device" is > added by the PCI backend. That could also be useful to some folks for non-PCI devices, say if a CPU gets hotplugged in/out (or more likely added/removed from a cpuset/cgroup you're in). - -- Christopher SamuelSenior Systems Administrator VLSCI - Victorian Life Sciences Computation Initiative Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545 http://www.vlsci.org.au/ http://twitter.com/vlsci -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAlAoe3QACgkQO2KABBYQAh8DagCeKwDn0lPdX1D7GlLD0ksuIX/t jvEAn2l7+FQhnYvdPoN1CUd6Y6oyHSTv =mBxD -END PGP SIGNATURE-
Re: [hwloc-devel] backends and plugins
Brice Goglin, le Tue 07 Aug 2012 13:29:31 +0200, a écrit : > Le 07/08/2012 13:06, Jeff Squyres a écrit : > >> Aside from the main "discover" callback, backends may also define some > >> callbacks to be invoked when new object are created. The main example is > >> Linux creating "OS devices" when a new "PCI device" is added by the PCI > >> backend. CUDA could use that too to fill GPU PCI devices. This is not > >> strictly needed since adding these devices could still be done later, > >> once the PCI backend is done. We'll see. > > This is a nifty idea. Is the idea that callback can be registered to be > > fired when a specific PCI vendor / device ID are found? > > I am not sure yet. Linux would use the callback for some classes of > devices. CUDA for some vendor ID. We could do the general case (all > object types) and have the callback check the device attribute, but it > could be overkill. I'd say it's actually simpler and shouldn't be too expensive. Samuel
Re: [hwloc-devel] backends and plugins
Le 07/08/2012 13:06, Jeff Squyres a écrit : >> Aside from the main "discover" callback, backends may also define some >> callbacks to be invoked when new object are created. The main example is >> Linux creating "OS devices" when a new "PCI device" is added by the PCI >> backend. CUDA could use that too to fill GPU PCI devices. This is not >> strictly needed since adding these devices could still be done later, >> once the PCI backend is done. We'll see. > This is a nifty idea. Is the idea that callback can be registered to be > fired when a specific PCI vendor / device ID are found? I am not sure yet. Linux would use the callback for some classes of devices. CUDA for some vendor ID. We could do the general case (all object types) and have the callback check the device attribute, but it could be overkill. >> Instead of allowing random API calls into plugin internals, we could >> keep these backends internal, i.e. not making them plugins. At least for >> OS backends, it makes sense. "synthetic" and "custom" also have no >> reason to be pluginified either, they depend on nothing. > It might be nice to view all plugins as the same -- regardless of whether > they are internal (i.e., part of libhwloc) or external (i.e., a standalone > DSO). That way, the majority of the core code doesn't have to know/care > whether plugins are internal or external. > > It would also allow slurping external plugins to be internal, which will be > fairly important for embedded mode. A specific case which has come up for > this multiple times is when higher-level MPI bindings packages (e.g., Python) > dlopen libmpi into a private namespace. When OMPI then tries to dlopen its > own DSO/external plugins, they can't find the symbols in libmpi that they > depend on (because libmpi is in a private namespace). Hence, OMPI has to be > built in a slurp-all-plugins-to-be-internal mode to support such > configurations. > > As such, we'll need hwloc to also support this > slurp-all-plugins-to-be-internal kind of mode, too. > > I can help with the build mojo for this, if desired. I don't enough about all this so we'll your help for sure :) We'll see once backends are cleaned up. Brice
Re: [hwloc-devel] backends and plugins
On Aug 7, 2012, at 5:02 AM, Brice Goglin wrote: > Antoine Rougier finished his internship recently so here's a summary of > what he did in the "backends" branch. For the record, the goal of his > work was to explore how to change our backends into proper plugins so > that we can easily avoid hard dependencies between the hwloc core and > external libraries such as CUDA, libpci, ... In general, this sounds most excellent. Some specific comments, below. > Aside from the main "discover" callback, backends may also define some > callbacks to be invoked when new object are created. The main example is > Linux creating "OS devices" when a new "PCI device" is added by the PCI > backend. CUDA could use that too to fill GPU PCI devices. This is not > strictly needed since adding these devices could still be done later, > once the PCI backend is done. We'll see. This is a nifty idea. Is the idea that callback can be registered to be fired when a specific PCI vendor / device ID are found? > All this is about making interaction between backends nicer. Once this > is done, we'll be able to make actual plugins out of all this. One > problem that will come is that some backends are almost directly used > from outside the core. For instance exporting a topology to XML is a > public API call going down to XML plugin. lstopo and hwloc-ps using > Linux-specific tid_get_cpubind() calls causes similar problems. Another possibility is deprecating the old functions and making new functions, like hwloc_export_topo_to_file(..., HWLOC_FILE_FORMAT_XML), or something like that. I.e., call a dispatch routine before the actual back-end routine -- this would preserve the abstraction/plugin barrier. (you mention another viable possibility below, too -- I mention this one only for completeness) I don't know if that's attractive or not, but I offer the possibility. :-) > Instead of allowing random API calls into plugin internals, we could > keep these backends internal, i.e. not making them plugins. At least for > OS backends, it makes sense. "synthetic" and "custom" also have no > reason to be pluginified either, they depend on nothing. It might be nice to view all plugins as the same -- regardless of whether they are internal (i.e., part of libhwloc) or external (i.e., a standalone DSO). That way, the majority of the core code doesn't have to know/care whether plugins are internal or external. It would also allow slurping external plugins to be internal, which will be fairly important for embedded mode. A specific case which has come up for this multiple times is when higher-level MPI bindings packages (e.g., Python) dlopen libmpi into a private namespace. When OMPI then tries to dlopen its own DSO/external plugins, they can't find the symbols in libmpi that they depend on (because libmpi is in a private namespace). Hence, OMPI has to be built in a slurp-all-plugins-to-be-internal mode to support such configurations. As such, we'll need hwloc to also support this slurp-all-plugins-to-be-internal kind of mode, too. I can help with the build mojo for this, if desired. > XML would like to be a plugin so that we stop depending on libxml2, but > we have an intermediate level to ease this. The main XML functions do > not depend on libxml2, they can remain internal and call either libxml2 > or our minimalistic no-libxml2 support underneath. So we can keep the > common code and the minimalistic support internal, and only move the > libxml2-specific callbacks to a plugin. > > Summary of what plugins we could have in the end: > * for main backends: > + internal: synthetic, xml-core, custom, linux, solaris, ... > + plugins: pci, cuda, display, ... > * for "low-level" xml backends: > + internal: minimalistic xml support > + plugin: libxml2 > * and maybe lstopo backends one day > + internal: console, txt, fig, windows > + plugin: cairo > > Brice > > ___ > hwloc-devel mailing list > hwloc-de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-devel -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
[hwloc-devel] backends and plugins
Antoine Rougier finished his internship recently so here's a summary of what he did in the "backends" branch. For the record, the goal of his work was to explore how to change our backends into proper plugins so that we can easily avoid hard dependencies between the hwloc core and external libraries such as CUDA, libpci, ... He defined three types of backends: * "Base" is a normal OS backend such as Linux/Solaris/... This guy manages PU, Cores, Sockets, ... Only one of these can be used at the same time. * "Additional" is what is added to "Base" backends. This is libpci, and things added inside PCI devices (CUDA, ...). Multiple of these may be used at the same time. They are likely invoked sequentially after the Base backend discovery. * "Global" is a special case for XML, Synthetic and Custom, which replaces both Base and Additional. It's not clear that this type will be needed in the end, "Base" with a special flag to disable "Additional" backends might be enough. During init(), all existing backends are placed on a list of "available" backends. And a list of "enabled" backends is initialized to empty. Between init() and load(), calling set_xml(), set_synthetic(), set_fsroot(), ... will append some backends to the "enabled" list (actually two lists, one for Base, one for Additional). During load(), we check whether some Base backends are "enabled". Otherwise we enable the default one of the current OS. Then we actually discover things using the (unique) enabled Base backend followed by all enabled Additional backends. Aside from the main "discover" callback, backends may also define some callbacks to be invoked when new object are created. The main example is Linux creating "OS devices" when a new "PCI device" is added by the PCI backend. CUDA could use that too to fill GPU PCI devices. This is not strictly needed since adding these devices could still be done later, once the PCI backend is done. We'll see. I'll work on putting all this in a branch soon. All this is about making interaction between backends nicer. Once this is done, we'll be able to make actual plugins out of all this. One problem that will come is that some backends are almost directly used from outside the core. For instance exporting a topology to XML is a public API call going down to XML plugin. lstopo and hwloc-ps using Linux-specific tid_get_cpubind() calls causes similar problems. Instead of allowing random API calls into plugin internals, we could keep these backends internal, i.e. not making them plugins. At least for OS backends, it makes sense. "synthetic" and "custom" also have no reason to be pluginified either, they depend on nothing. XML would like to be a plugin so that we stop depending on libxml2, but we have an intermediate level to ease this. The main XML functions do not depend on libxml2, they can remain internal and call either libxml2 or our minimalistic no-libxml2 support underneath. So we can keep the common code and the minimalistic support internal, and only move the libxml2-specific callbacks to a plugin. Summary of what plugins we could have in the end: * for main backends: + internal: synthetic, xml-core, custom, linux, solaris, ... + plugins: pci, cuda, display, ... * for "low-level" xml backends: + internal: minimalistic xml support + plugin: libxml2 * and maybe lstopo backends one day + internal: console, txt, fig, windows + plugin: cairo Brice
