Re: Bug#657949: Cannot install libhdf5-mpi-dev and libnetcdf-dev
Le mercredi 01 février 2012 à 19:43 -0600, Steve M. Robbins a écrit : Unfortunately they were still not available for that at the time of my last poking. Diverging from upstream is not a good idea, so we still have to live in a non perfect world... I think we can no longer live in the status quo (see all the blockers of #631019), so something has to give. Even if it is painful, perhaps Debian could pioneer something and pass patches back to upstream? I asked upstream for their opinions on this subject. Here is the answer: There are no current plans to change the parallel HDF5 library name to another name. However, this has been a source of confusion over the years, so I entered a bug report for it. (I don't know of any ramifications to renaming the parallel library, but I think the best solution would be for us to provide it with a different name.) (the bug report is not public). As Julien suggested a few days ago on IRC, we could provide a patch for this but I don't think it is a technical issue but more organisation/politic issue for their users. We could try to make Debian move to the renaming of all MPI HDF5 libraries but this will be diverging a lot from upstream while confusing our users (but I don't know if it would confuse less or more than the current solution). Sylvestre -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/1329409321.4035.121.ca...@pomegues.inria.fr
Re: Bug#657949: Cannot install libhdf5-mpi-dev and libnetcdf-dev
Hi Francesco, Do you recommend that we build the next NetCDF from 4.1.1 or should we use the 4.1.3 from experimental as the base? Regards Alastair On 2012-02-07 13:17, Francesco P. Lovergine wrote: On Tue, Feb 07, 2012 at 09:28:00AM +, Alastair McKinstry wrote: On 2012-02-02 01:43, Steve M. Robbins wrote: Hi, I'd like to contribute towards a solution for this. I'm forwarding to debian-devel to get some others' ideas. Naively, I don't understand why netcdf can't offer multiple variants, just as hdf5 does. Or, at least, one package libnetcdf-mpi-dev that links with the default MPI implementation. I am not involved in the netcdf. You should report a bug on this package. I'm prepared to do so, but I'd first like to get agreement that netcdf is where the problem lies. Netcdf maintainers, please chime in! I think we can no longer live in the status quo (see all the blockers of #631019), so something has to give. Even if it is painful, perhaps Debian could pioneer something and pass patches back to upstream? Thoughts? -Steve As of now, I have several packages (eg ADIOS, CDO) that used to build against netcdf and libhdf5-mpi-dev that don't. Without fixes to netCDF (I appreciate what Francesco says about netcdf upstream not giving the libraries proper names), there needs to be a regression: either the packages build with netcdf but no MPI, or MPI but no netcdf. The problem is the following: with latest update to hdf5, the chain of dependencies changed, so that now libnetcdf6 depends on the pure serial version of hdf5, while the previous one depended on serial or parallel: Version: 1:4.1.1-6+b1 Depends: libc6 (= 2.7), libcurl3-gnutls (= 7.16.2), libgcc1 (= 1:4.1.1), libgfortran3 (= 4.3), libhdf5-7 (= 1.8.7), libquadmath0 (= 4.6), libstdc++6 (= 4.4.0) Version: 1:4.1.1-6 Depends: libc6 (= 2.7), libcurl3-gnutls (= 7.16.2-1), libgcc1 (= 1:4.1.1), libgfortran3 (= 4.3), libhdf5-serial-1.8.4 | libhdf5-1.8.4, libquadmath0 (= 4.6), libstdc++6 (= 4.4.0) So at least at packaging level, that should be fixed to follow the previous criteria. That said, indeed NetCDF provides nc_create_par and nc_open_par in both serial and parallel versions, but needs to be built with --enable-parallel to take advantage of parallel I/O in HDF5, else it works in pure serial mode. -- Alastair McKinstry , alast...@sceal.ie , mckins...@debian.org http://blog.sceal.ie Anyone who believes exponential growth can go on forever in a finite world is either a madman or an economist - Kenneth Boulter, Economist. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4f328767.1060...@debian.org
Re: Bug#657949: Cannot install libhdf5-mpi-dev and libnetcdf-dev
On Wed, Feb 08, 2012 at 02:32:07PM +, Alastair McKinstry wrote: Hi Francesco, Do you recommend that we build the next NetCDF from 4.1.1 or should we use the 4.1.3 from experimental as the base? Regards Alastair AFAIK Sylvestre is going to reset the dependencies chain in hdf5 to avoid that kind of problem. About 4.1.3 in experimental, it still needs a bit of work, and I'm going to split in separate packages current netcdf 4.1.1 before, in order to have a decent organization of all solibs to have a smooth migration to 4.1.3. You have free access to the git repository, so a branch can be prepared for having a parallel flavor too. -- Francesco P. Lovergine -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120208152707.gb3...@gandalf.is-a-geek.org
Re: Bug#657949: Cannot install libhdf5-mpi-dev and libnetcdf-dev
On 2012-02-02 01:43, Steve M. Robbins wrote:
Hi,
I'd like to contribute towards a solution for this. I'm forwarding to
debian-devel to get some others' ideas.
Naively, I don't understand why netcdf can't offer multiple variants,
just as hdf5 does. Or, at least, one package libnetcdf-mpi-dev that
links with the default MPI implementation.
I am not involved in the netcdf. You should report a bug on this
package.
I'm prepared to do so, but I'd first like to get agreement that
netcdf is where the problem lies. Netcdf maintainers, please
chime in!
I think we can no longer live in the status quo (see all the blockers
of #631019), so something has to give. Even if it is painful, perhaps
Debian could pioneer something and pass patches back to upstream?
Thoughts?
-Steve
As of now, I have several packages (eg ADIOS, CDO) that used to build
against netcdf and libhdf5-mpi-dev
that don't. Without fixes to netCDF (I appreciate what Francesco says
about netcdf upstream
not giving the libraries proper names), there needs to be a regression:
either the packages
build with netcdf but no MPI, or MPI but no netcdf.
I could split the package, and provide two versions, eg. adios-mpi and
adios-serial, but this to me is going backwards.
In an increasingly parallel world, we need binaries that will run in
parallel when its available. eg. detect
an MPI environmnent, and if so, use the parallel version of libraries.
Do others think this is the way to go, or that way lies madness?
That is, can we work out the details of what would be needed for
automatic parallelism,
what we can do and what upstream changes might be needed?
e.g. we might add some shim code at the start of programs that do:
if (mpi_detected || ($ENV{NETCDF_SERIAL_WANTED})
dlopen(netcdf_mpi_version)
else
dlopen(netcdf_serial_version)
(Some netCDF programs, even running under MPI, might run in serial mode
in order to use
features such as compression that don't work in parallel netcdf).
We need to come up with:
(1) A bigger picture of where we want Debian to go (may involve upstream
changes)
(2) A plan for that we can do for the next release.
Regards
Alastair
--
Alastair McKinstry , alast...@sceal.ie , mckins...@debian.org
http://blog.sceal.ie
Anyone who believes exponential growth can go on forever in a finite world
is either a madman or an economist - Kenneth Boulter, Economist.
--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/4f30eea0.7030...@debian.org
Re: Bug#657949: Cannot install libhdf5-mpi-dev and libnetcdf-dev
On Tue, Feb 07, 2012 at 09:28:00AM +, Alastair McKinstry wrote: On 2012-02-02 01:43, Steve M. Robbins wrote: Hi, I'd like to contribute towards a solution for this. I'm forwarding to debian-devel to get some others' ideas. Naively, I don't understand why netcdf can't offer multiple variants, just as hdf5 does. Or, at least, one package libnetcdf-mpi-dev that links with the default MPI implementation. I am not involved in the netcdf. You should report a bug on this package. I'm prepared to do so, but I'd first like to get agreement that netcdf is where the problem lies. Netcdf maintainers, please chime in! I think we can no longer live in the status quo (see all the blockers of #631019), so something has to give. Even if it is painful, perhaps Debian could pioneer something and pass patches back to upstream? Thoughts? -Steve As of now, I have several packages (eg ADIOS, CDO) that used to build against netcdf and libhdf5-mpi-dev that don't. Without fixes to netCDF (I appreciate what Francesco says about netcdf upstream not giving the libraries proper names), there needs to be a regression: either the packages build with netcdf but no MPI, or MPI but no netcdf. The problem is the following: with latest update to hdf5, the chain of dependencies changed, so that now libnetcdf6 depends on the pure serial version of hdf5, while the previous one depended on serial or parallel: Version: 1:4.1.1-6+b1 Depends: libc6 (= 2.7), libcurl3-gnutls (= 7.16.2), libgcc1 (= 1:4.1.1), libgfortran3 (= 4.3), libhdf5-7 (= 1.8.7), libquadmath0 (= 4.6), libstdc++6 (= 4.4.0) Version: 1:4.1.1-6 Depends: libc6 (= 2.7), libcurl3-gnutls (= 7.16.2-1), libgcc1 (= 1:4.1.1), libgfortran3 (= 4.3), libhdf5-serial-1.8.4 | libhdf5-1.8.4, libquadmath0 (= 4.6), libstdc++6 (= 4.4.0) So at least at packaging level, that should be fixed to follow the previous criteria. That said, indeed NetCDF provides nc_create_par and nc_open_par in both serial and parallel versions, but needs to be built with --enable-parallel to take advantage of parallel I/O in HDF5, else it works in pure serial mode. -- Francesco P. Lovergine -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120207131730.gb7...@gandalf.is-a-geek.org
Re: Bug#657949: Cannot install libhdf5-mpi-dev and libnetcdf-dev
On Wed, Feb 01, 2012 at 07:43:31PM -0600, Steve M. Robbins wrote: The solution is having upstream adopting a sane naming scheme for mpi-enabled flavor libraries instead of using always the same names for all. Francesco, please clarify: are you speaking of the hdf5 upstream or the netcdf upstream? (Both?) I mean first of all hdf5 upstream. Note that anyway both them use different APIs for serial and parallel programming models. So having the same library names for completely different things IMHO is defective by design and confusing. As a principle we could install only mpi-enabled libraries (the serial model and API could be anyway used) but that would imply that people should coexist with such kind of stuff installed always, if used or not. Also some serial-only supports could be missed and anomalies appearing here and there: both them are quite complicated beasts. I would avoid to take such kind of decision without a deep analysis. What problem are you trying to solve with that: co-installable -dev packages or just coinstallable lib packages? Unfortunately they were still not available for that at the time of my last poking. Diverging from upstream is not a good idea, so we still have to live in a non perfect world... I think we can no longer live in the status quo (see all the blockers of #631019), so something has to give. Even if it is painful, perhaps Debian could pioneer something and pass patches back to upstream? Thoughts? I'm afraid it is quite difficult having such kind of proposal accepted by upstreams. It implies changes for both them in library use, that they could be not ready to introduce. In 2009 I asked about that in hdf-forum without a positive answer. -- Francesco P. Lovergine -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120207134136.gc7...@gandalf.is-a-geek.org
Re: Bug#657949: Cannot install libhdf5-mpi-dev and libnetcdf-dev
Hi, I'd like to contribute towards a solution for this. I'm forwarding to debian-devel to get some others' ideas. On Wed, Feb 01, 2012 at 09:57:39AM +0100, Sylvestre Ledru wrote: Le mardi 31 janvier 2012 à 21:56 -0600, Steve M. Robbins a écrit : Naively, I don't understand why netcdf can't offer multiple variants, just as hdf5 does. Or, at least, one package libnetcdf-mpi-dev that links with the default MPI implementation. I am not involved in the netcdf. You should report a bug on this package. I'm prepared to do so, but I'd first like to get agreement that netcdf is where the problem lies. Netcdf maintainers, please chime in! On Wed, Feb 01, 2012 at 05:44:49PM +0100, Francesco P. Lovergine wrote: On Tue, Jan 31, 2012 at 04:41:06PM +0100, Sylvestre Ledru wrote: Even if I am not happy about this change, it is expected. libnetcdf-dev depends on libnetcdf7 which depends on libhdf5-7. libhdf5-openmpi-7 conflicts with libhdf5-7. Before I had the silly idea to become a hdf5 maintainer, I reported this bug myself #591346. For now, I haven't find the right solution to tackle this issue ... Suggestions are welcome. The solution is having upstream adopting a sane naming scheme for mpi-enabled flavor libraries instead of using always the same names for all. Francesco, please clarify: are you speaking of the hdf5 upstream or the netcdf upstream? (Both?) What problem are you trying to solve with that: co-installable -dev packages or just coinstallable lib packages? Unfortunately they were still not available for that at the time of my last poking. Diverging from upstream is not a good idea, so we still have to live in a non perfect world... I think we can no longer live in the status quo (see all the blockers of #631019), so something has to give. Even if it is painful, perhaps Debian could pioneer something and pass patches back to upstream? Thoughts? -Steve signature.asc Description: Digital signature
