[sage-devel] Re: C/C++ Library Versioning in Sage
Hi Martin, Michael, Tim, and [sage-devel], Michael did propose two solutions for certain problems: 1. move all library versioning patches to the Debian packaging directory 2. provide makefile targets to obtain unversioned libraries Thinking about it, a third solution comes to my mind, let me explain how I got there. In the past, all packages (say e.g. Pari, Flint) were taken from their source destination, and then directly spkg were made out them. If libraries were provided, they were put under $SAGE_ROOT/local/lib, and it was tried hard to make sure, that the system linker/loader was taking these, and not any other version of them somewhere lying around. The debianization of Sage means (among other things) that certain packages that already existed as debian packages are taken out of Sage (Pari), as well as some other packages, which were made into debain packages in that process. The libraries provided by these debianized packages are no longer under $SAGE_ROOT/local/lib, but the debian system has to know itself where to find them (not really, see below). On non-debian (or better, non-GNU/Linux) systems, and even on debian systems also, it might not only be favorable, but a sheer necessity to be able to install not the debian version of a package, but the spkg version. For example, the Pari spkg does patch (directly in the sources) the exit/abort behaviour, and one might want to use this patched version of Pari, even on a Debian system, where the debian upstream Pari does (and shall) not have this exist/abort patch. This directly leads to a third solution to the problems Michael sees: 3. For those packages, that are debianized, the input for creating a corresponding spkg is no longer the raw source, but the debianized version, i.e. a debian package. In the spkg-install script, one can (and does) patch the sources and the build process anyway, by these means one ensures that the libraries created are put under $SAGE_ROOT/ local/lib and nowhere else. And any soname versioning is simply patched out, if it causes trouble. And Sage works with either the original debian or with the spkg version of a package, the latter having priority if it is installed. To follow this way seems necessary to me, if one wants to ensure that the Sage debian branch and Sage mainline branch do not diverge too much. Perhaps one has to create fake or virtual debian packages from time to time, but that's not too much additional effort. Certainly not compared with the benefit of keeping mainline and debianization together, but even in absolute time. As for those packages that had been debianized in the process, it is pretty easy to provide the spkg using the new debian package as input: just replace (in spkg-install) the new build structure (makefiles and such) by the old ones, which are still in the repository, and are known to work. And the introduced soname versioning will just disappear, too :-) The Windows native 64-Bit port in mind, it seems to be necessary anyway, to use the flexibility of doubling a package's build structure here and there. Admittedly, there is one drawback to solution 3), which needs further discussion: In devel/sage/setup.py, the environment variable SAGE_DEBIAN currently is used as a master switch for all those packages, to add each and every of the debian_include_dirs at once. Things might work well nevertheless. But if not, it's time anyway to clean up this grown over the years file, there is more than one trac ticket addressing this (e.g. #2732). At last, one word concerning the use of libtool. For Sage purposes, this seems to me to be like shooting with cannons on sparrows. In my eyes, it does make sense to (newly) introduce the use of libtool, automake, autoconf for one of the debianized packages, if and only if that package is intended to be spreaded to the world not only as a debian package, or as a part of Sage, but also as a standalone (!) package under Solaris, BSD, and other non-GNU/Linux systems. Which is well worth some effort. But that clearly is orthogonal to Sage, and the needs of Sage. Cheers, gsw --~--~-~--~~~---~--~~ To post to this group, send email to sage-devel@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sage-devel URLs: http://www.sagemath.org -~--~~~~--~~--~--~---
[sage-devel] Re: C/C++ Library Versioning in Sage
On Sep 28, 5:29 am, Georg S. Weber [EMAIL PROTECTED] wrote: At last, one word concerning the use of libtool. For Sage purposes, this seems to me to be like shooting with cannons on sparrows. In my eyes, it does make sense to (newly) introduce the use of libtool, automake, autoconf for one of the debianized packages, if and only if that package is intended to be spreaded to the world not only as a debian package, or as a part of Sage, but also as a standalone (!) package under Solaris, BSD, and other non-GNU/Linux systems. Library versioning is really supposed to be maintained upstream, since they're the people who know which interfaces have been broken and thus when version bumps are required. It also avoids every distribution maintaining its own patch for library versioning for a particular piece of software. Once the library versioning gets merged upstream, however, Sage then must either patch the upstream build system (which seems annoying to maintain) or use the versioned library. Because Sage is ported to Solaris, BSD, etc., this means the versioning must support all these non-GNU/Linux platforms as well. I probably should have thought about this more before submitting my versioning patches upstream. I think it may be helpful to be concrete about which packages are involved. The packages not already using automake that Francois Bissey and I have added shared library versioning to are the following: symmetrica (here the upstream makefile is like 3 lines long) singular (has a rather complex build system using autoconf but not automake) flint (makefile build system) zn_poly (uses a python script to write a makefile) ntl (uses a perl script to write a makefile) polybori (uses scons; versioning was added by the upstream developers) tachyon (uses a makefile; here there was previously only a static library) Almost all of these projects have an active userbase outside of Sage. I should probably also comment on the current state of the SAGE_DEBIAN setup. Because my Sage packages are now in Debian (rather than in my Sage Debian apt repository), I'm no longer building Debian packages from the Sage sources (I need to use clean upstream sources for packages distributed in Debian, and it's easier to be sure that I've got clean upstream code if I download from the upstream website). So, the dist/ debian directories in spkgs are currently unused. Those directories should perhaps be cleaned up, since I certainly don't have to time to maintain them for building new Debian packages after every Sage release in a Sage Debian apt repository (which is the only use they might have at this point). I do have an easy mechanism for using the Sage version of a particular spkg, rather than the Debian version for the Sage build (I have a list of spkgs that are built when I build the Sage package, included sage_scripts, examples, extcode, etc., and could add pari to that list if necessary). But I try to avoid this whenever possible; ideally, these bugfixes should be sent to the upstream developer and fixed for all Debian users, rather than just in Debian's Sage package. -Tim Abbott --~--~-~--~~~---~--~~ To post to this group, send email to sage-devel@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sage-devel URLs: http://www.sagemath.org -~--~~~~--~~--~--~---
[sage-devel] Re: C/C++ Library Versioning in Sage
On Sep 27, 8:50 am, Tim Abbott [EMAIL PROTECTED] wrote: On Sep 26, 3:39 pm, mabshoff [EMAIL PROTECTED] wrote: Hi Tim, For example: Take the FLINT.spkg - ever since we used the patches Tim submitted the spkg was broken twice by either not adjusting the link name for libfint.so or after installing an updated version Sage would I think this is really the victim of Sage not using the install target from the upstream FLINT package. Sage should not need to be maintaining code to copy FLINT's libraries into place and install the right symbolic links. I believe if we were to automakeify FLINT, the install targets would do the right thing with installing symlinks appropriately on all platforms supported by libtool. I still don't see the benefit of versioned libraries for Sage. And automakifying FLINT will only make the build process slower and more complicated. I fixed a gmp linking issue in FLINT on Cygwin trivially in 3 seconds while a similar fix with link order for LinBox when using Givaro was way more involved. Well, in theory: yes. But autoconf does not work on Windows and I guess this is referring to native Windows, as opposed to Cygwin? Or does autoconf not work on Cygwin either? It does not work on native Windows, it mostly works in Cygwin. It can be made to use the native MS compilers via some POSIX shell, but that is prone to break, i.e. spaces in paths and so on Sage must already have some 30 packages that use autotools; is the worry that we're making a native Windows port even harder by adding 10 more? Because the autotools are a 20 year old mess of incompatible versions and generally have the elegance of a car with square tires :). We don't need to push another ten software packages toward autotools and forgo the use of a saner cross-platform build system. Many of the spkgs which do not have a autohell base build system are small packages and making them autohell based will increase their build time significantly since the configure process can run for quite a while. This is even worst on Cygwin where shell scripts run dog slow. various spkgs do neither use autoconf or SCons. And I do not want to move more code to use autoconf since IMHO it plainly sucks. There are alternatives that work on Windows for sure, but that is likely even more work since few people use SCons for example. One can always have two build systems, i.e one for Windows and one for the rest. In the future, I think most people will want to get Sage via their binary Linux distribution, rather than having to compile it from source themselves. Nope, once the Cygwin version is out that will be the most downloaded version. Back in the day when we had a Cygwin build about 2/3rds of the downloads were Cygwin and I see no reason that will not change in the future. I wholly expect this number of Windows downloads to be even more dominant once we have a native version. Compiling Sage from source is a huge barrier to entry. I disagree, building Sage is trivial compared to the complexity of similar packages. It might be that some users will not use Sage when it has to be build from sources, but we have binaries, too. But no binary Linux distribution can distribute the Sage dependencies unless they do proper library versioning. Sure, but that is not the problem I need to solve :) The patches for adding sonames that Francois and I wrote were indeed not designed with Solaris and BSD's non-GNU toolchain in mind. The right way to solve this problem is to use libtool, since that the standard way to do portable versioning for libraries. The easiest way to use libtool is to use automake, and I'd be willing to generate patches to make the Sage dependencies to which I've added versioning use automake (with the exception of Singular, which probably needs a more incremental solution). I'm not convinced that Scons is an option for this; its documentation does not suggest to me that it has a portable mechanism for specifying the soname of a library. I have not checked into this, but PolyBoRi seems to do fine. I am still not convinced versioned libraries are a necessity for Sage itself. Just because the Debian packaging rules demand it does not make a case. It does make sense for Debian, but not for Sage IMHO. So let the Debian packagers deal with versioned libraries, I just don't see any reason to support it in Sage especially if they break ports. But the above problems do also cause trouble when I upgrade an spkg when doing merges, i.e. if I updated ntl I would hate to rebuild all of Sage. As is the soname of FLINT is changed *every* time the minor version is bumped. Well, if FLINT were using automake, it would be easy for them to only change the soname when it is necessary (i.e. when an interface is removed). The soname bumps are a nuisance for me as well, since packages have to pass through the Debian NEW queue every time their soname changes Yeah, this is
[sage-devel] Re: C/C++ Library Versioning in Sage
Hi guys, The present discussion was precipitated by my submission of the zn_poly 0.9 spkg, which included some modifications to the makefile suggested by Tim, specifically the library versioning stuff. I still don't totally understand what's going on, but the ensuing discussion has been very illuminating (thanks everyone!) I think from my point of view the crux of the matter is this exchange: On Sep 28, 2008, at 1:25 PM, mabshoff wrote: On Sep 27, 8:50 am, Tim Abbott [EMAIL PROTECTED] wrote: But no binary Linux distribution can distribute the Sage dependencies unless they do proper library versioning. Sure, but that is not the problem I need to solve :) So, in other words, it's useful for the purposes of Debian packaging for me to do proper library versioning, at least as far as indicating when interfaces change. On the other hand, Michael doesn't think versioning is important for Sage, and he points out that the method suggested by Tim is GNU-specific, whereas I don't want zn_poly to be GNU-specific (as far as I can help it). But at the end of the day, as long as I provide the versioning information, both Sage and Debian can trivially patch the makefile to make it work how they want. I think I know enough about the Sage build system to be able to handle this myself for the spkg (with occasional guidance from mabshoff). So my question becomes: how do I provide versioning information in a non-GNU-specific way? I guess if I can do this then I can make everyone happy. david --~--~-~--~~~---~--~~ To post to this group, send email to sage-devel@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sage-devel URLs: http://www.sagemath.org -~--~~~~--~~--~--~---
[sage-devel] Re: C/C++ Library Versioning in Sage
On Sep 28, 10:16 am, Tim Abbott [EMAIL PROTECTED] wrote: On Sep 28, 5:29 am, Georg S. Weber [EMAIL PROTECTED] wrote: Hi, At last, one word concerning the use of libtool. For Sage purposes, this seems to me to be like shooting with cannons on sparrows. In my eyes, it does make sense to (newly) introduce the use of libtool, automake, autoconf for one of the debianized packages, if and only if that package is intended to be spreaded to the world not only as a debian package, or as a part of Sage, but also as a standalone (!) package under Solaris, BSD, and other non-GNU/Linux systems. Sage is self contained for a reason since any other setup while using components from the system is sure to cause endless trouble, i.e. just imagine debugging a Sage setup where ten components are provided by the system. IMHO there will only be one official and supported version of Sage and that will be the official, vanilla tarball. Whatever distribution package will be close, but not identical. I also doubt that any distribution will keep up with Sage releases and we do not have the manpower or interest to support stable releases of Sage. William has always claimed that once development calms down this will be possible, but he said that a year ago and since then development has sped up and not slowed down. I think that any Sage in Debian stable will be useless after six months while the version in Debian testing might have a chance to stay somewhat close to current. Library versioning is really supposed to be maintained upstream, since they're the people who know which interfaces have been broken and thus when version bumps are required. It also avoids every distribution maintaining its own patch for library versioning for a particular piece of software. Once the library versioning gets merged upstream, however, Sage then must either patch the upstream build system (which seems annoying to maintain) or use the versioned library. We don't have to patch the build system, we have to do some post- processing of the installed libraries which is mostly trivial. If upstream goes with versioned libraries I will not fight it, but complain if it breaks due to GNUisms. If upstream does not fix the problem I will patch it out of their build system :) Because Sage is ported to Solaris, BSD, etc., this means the versioning must support all these non-GNU/Linux platforms as well. I probably should have thought about this more before submitting my versioning patches upstream. Well, I did not complain vocal enough, so some patches were written that will likely end up being discarded. Wasting code and effort is always a bad thing. I think it may be helpful to be concrete about which packages are involved. The packages not already using automake that Francois Bissey and I have added shared library versioning to are the following: symmetrica (here the upstream makefile is like 3 lines long) singular (has a rather complex build system using autoconf but not automake) flint (makefile build system) zn_poly (uses a python script to write a makefile) ntl (uses a perl script to write a makefile) polybori (uses scons; versioning was added by the upstream developers) tachyon (uses a makefile; here there was previously only a static library) The build system for ntl certainly sucks, but it is a cross platforms build system that actually has supported MSVC for probably a decade. That makes it pretty unique for any mathematical software library out there. I seriously doubt you can convince Victor Shoup to use autotools for NTL. Almost all of these projects have an active userbase outside of Sage. I should probably also comment on the current state of the SAGE_DEBIAN setup. Because my Sage packages are now in Debian (rather than in my Sage Debian apt repository), I'm no longer building Debian packages from the Sage sources (I need to use clean upstream sources for packages distributed in Debian, and it's easier to be sure that I've got clean upstream code if I download from the upstream website). So, the dist/ debian directories in spkgs are currently unused. Those directories should perhaps be cleaned up, since I certainly don't have to time to maintain them for building new Debian packages after every Sage release in a Sage Debian apt repository (which is the only use they might have at this point). I do have an easy mechanism for using the Sage version of a particular spkg, rather than the Debian version for the Sage build (I have a list of spkgs that are built when I build the Sage package, included sage_scripts, examples, extcode, etc., and could add pari to that list if necessary). But I try to avoid this whenever possible; ideally, these bugfixes should be sent to the upstream developer and fixed for all Debian users, rather than just in Debian's Sage package. Our patches go into upstream, some times not as fast as they should. The main issue I have with using
[sage-devel] Re: C/C++ Library Versioning in Sage
On Mon, 29 Sep 2008, mabshoff wrote: On Sep 27, 8:50 am, Tim Abbott [EMAIL PROTECTED] wrote: On Sep 26, 3:39 pm, mabshoff [EMAIL PROTECTED] wrote: Hi Tim, For example: Take the FLINT.spkg - ever since we used the patches Tim submitted the spkg was broken twice by either not adjusting the link name for libfint.so or after installing an updated version Sage would I think this is really the victim of Sage not using the install target from the upstream FLINT package. Sage should not need to be maintaining code to copy FLINT's libraries into place and install the right symbolic links. I believe if we were to automakeify FLINT, the install targets would do the right thing with installing symlinks appropriately on all platforms supported by libtool. I still don't see the benefit of versioned libraries for Sage. And automakifying FLINT will only make the build process slower and more complicated. I fixed a gmp linking issue in FLINT on Cygwin trivially in 3 seconds while a similar fix with link order for LinBox when using Givaro was way more involved. big snip Yeah, this is certainly bad, but I also pretty much doubt for most of the upstream authors pay attention to sonames and preseving upward compatibility. There was a recent discussion on the mpfr mailing list where some people complained that backward compatibility was broken since the mpfr 2.3.1 test suite could no longer link against the mpfr 2.3.2 library and vice versa and as it turned out that some auxiliary function had been removed that was never intended to be used. So where you draw the line? My personal conclusion (and I know I am repeating myself) is that in Sage's situation we just should not care and not deal with versioned libraries and be done with it. Whatever upstream packagers do is orthogonal to that, but any fix that will introduce GNUisms will be removed from the spkg or the updated spkg will not be merged. +1 Because sage is self contained it has no need for library versioning. Distro are different, we may let several versions of a library co-exist for that you need versioning. But it's certainly not sage's job to maintain it - in fact if it wasn't a pain you probably want to strip it. Cheers, Francois --~--~-~--~~~---~--~~ To post to this group, send email to sage-devel@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sage-devel URLs: http://www.sagemath.org -~--~~~~--~~--~--~---
[sage-devel] Re: C/C++ Library Versioning in Sage
On Sep 28, 10:46 am, David Harvey [EMAIL PROTECTED] wrote: Hi guys, Hi, The present discussion was precipitated by my submission of the zn_poly 0.9 spkg, which included some modifications to the makefile suggested by Tim, specifically the library versioning stuff. I still don't totally understand what's going on, but the ensuing discussion has been very illuminating (thanks everyone!) I think from my point of view the crux of the matter is this exchange: On Sep 28, 2008, at 1:25 PM, mabshoff wrote: On Sep 27, 8:50 am, Tim Abbott [EMAIL PROTECTED] wrote: But no binary Linux distribution can distribute the Sage dependencies unless they do proper library versioning. Sure, but that is not the problem I need to solve :) So, in other words, it's useful for the purposes of Debian packaging for me to do proper library versioning, at least as far as indicating when interfaces change. Debian and many other distributions require sonames. So it is more than useful, it is mandated in those cases :) On the other hand, Michael doesn't think versioning is important for Sage, and he points out that the method suggested by Tim is GNU-specific, whereas I don't want zn_poly to be GNU-specific (as far as I can help it). But at the end of the day, as long as I provide the versioning information, both Sage and Debian can trivially patch the makefile to make it work how they want. I think I know enough about the Sage build system to be able to handle this myself for the spkg (with occasional guidance from mabshoff). So my question becomes: how do I provide versioning information in a non-GNU-specific way? I guess if I can do this then I can make everyone happy. Well, Tim and many other people believe that the only solution here is autotools, but I certainly disagree. You picked a python script to write a makefile, so it can be easily enough worked around by providing a library build target that honors Debian's policy, i.e. sonames and an install target that creates the proper links. Then just leave the plain old version that we is there already so that Sage for example can use that. david Cheers, Michael --~--~-~--~~~---~--~~ To post to this group, send email to sage-devel@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sage-devel URLs: http://www.sagemath.org -~--~~~~--~~--~--~---
[sage-devel] Re: C/C++ Library Versioning in Sage
On Sep 26, 3:39 pm, mabshoff [EMAIL PROTECTED] wrote: For example: Take the FLINT.spkg - ever since we used the patches Tim submitted the spkg was broken twice by either not adjusting the link name for libfint.so or after installing an updated version Sage would I think this is really the victim of Sage not using the install target from the upstream FLINT package. Sage should not need to be maintaining code to copy FLINT's libraries into place and install the right symbolic links. I believe if we were to automakeify FLINT, the install targets would do the right thing with installing symlinks appropriately on all platforms supported by libtool. Well, in theory: yes. But autoconf does not work on Windows and I guess this is referring to native Windows, as opposed to Cygwin? Or does autoconf not work on Cygwin either? Sage must already have some 30 packages that use autotools; is the worry that we're making a native Windows port even harder by adding 10 more? various spkgs do neither use autoconf or SCons. And I do not want to move more code to use autoconf since IMHO it plainly sucks. There are alternatives that work on Windows for sure, but that is likely even more work since few people use SCons for example. One can always have two build systems, i.e one for Windows and one for the rest. In the future, I think most people will want to get Sage via their binary Linux distribution, rather than having to compile it from source themselves. Compiling Sage from source is a huge barrier to entry. But no binary Linux distribution can distribute the Sage dependencies unless they do proper library versioning. The patches for adding sonames that Francois and I wrote were indeed not designed with Solaris and BSD's non-GNU toolchain in mind. The right way to solve this problem is to use libtool, since that the standard way to do portable versioning for libraries. The easiest way to use libtool is to use automake, and I'd be willing to generate patches to make the Sage dependencies to which I've added versioning use automake (with the exception of Singular, which probably needs a more incremental solution). I'm not convinced that Scons is an option for this; its documentation does not suggest to me that it has a portable mechanism for specifying the soname of a library. But the above problems do also cause trouble when I upgrade an spkg when doing merges, i.e. if I updated ntl I would hate to rebuild all of Sage. As is the soname of FLINT is changed *every* time the minor version is bumped. Well, if FLINT were using automake, it would be easy for them to only change the soname when it is necessary (i.e. when an interface is removed). The soname bumps are a nuisance for me as well, since packages have to pass through the Debian NEW queue every time their soname changes -Tim Abbott --~--~-~--~~~---~--~~ To post to this group, send email to sage-devel@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sage-devel URLs: http://www.sagemath.org -~--~~~~--~~--~--~---
[sage-devel] Re: C/C++ Library Versioning in Sage
On Saturday 27 September 2008, Tim Abbott wrote: The patches for adding sonames that Francois and I wrote were indeed not designed with Solaris and BSD's non-GNU toolchain in mind. The right way to solve this problem is to use libtool, since that the standard way to do portable versioning for libraries. The easiest way to use libtool is to use automake, and I'd be willing to generate patches to make the Sage dependencies to which I've added versioning use automake (with the exception of Singular, which probably needs a more incremental solution). I'd be willing to help with that. My automake skills are probably way behind your's but there ought to be some easier candidates. Cheers, Martin -- name: Martin Albrecht _pgp: http://pgp.mit.edu:11371/pks/lookup?op=getsearch=0x8EF0DC99 _www: http://www.informatik.uni-bremen.de/~malb _jab: [EMAIL PROTECTED] --~--~-~--~~~---~--~~ To post to this group, send email to sage-devel@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sage-devel URLs: http://www.sagemath.org -~--~~~~--~~--~--~---
[sage-devel] Re: C/C++ Library Versioning in Sage
On Sep 26, 3:23 am, Martin Albrecht [EMAIL PROTECTED] wrote: Hi Michael and [sage-devel], at http://trac.sagemath.org/sage_trac/ticket/4169#comment:7 mabshoff makes the following comment: The version patches by Tim break on every BSD and Solaris where we do not use the GNU ld per default. We now work around this by linking gld to ld, but this is *not* a long term solution. Sage in general does not benefit from versioned libraries, indeed they case a bunch of problems when linking extensions, due to the links the archives get larger and on top of that it makes the memory address space on Cygwin even more scare, so I intend to remove every one of those version patches in the future. Those patches should be moved into the Debian packaging directory or alternatively you should provide a makefile target for not versioned libraries. As this seems to lay out a general strategy this seems too big to not discuss it here. Here are some -- clueless -- questions: - Sage in general does not benefit from versioned libraries Is adding a proper soname / library versioning to a library what you call versioned libraries? Isn't that somehow standardised and good practice? Sure, on Linux and some Unix platforms. But doing it right, i.e. for non-GNU linkers is complicated. I do not oppose using soname friends, but if we introduce them they ought to work on more than Linux :) - indeed they case a bunch of problems when linking extensions Why? For example: Take the FLINT.spkg - ever since we used the patches Tim submitted the spkg was broken twice by either not adjusting the link name for libfint.so or after installing an updated version Sage would no longer start since we linked against libflint-1.0.13.so. We do need to force a rebuild of the extensions that link against FLINT anyway, so this is manageable, but just imagine what happens if we changed ntl. - due to the links the archives get larger Do you mean the symbolic links? I don't see how they contribute to the size significantly. Not the link, but not all tar utilities are smart enough to deal with links properly. And saving a hundred or so MB install size seems to matter. - and on top of that it makes the memory address space on Cygwin even more scare Can you elaborate on that? Cygwin is stupid and often causes libraries to have collisions in address space. So when you compile a bunch of DLL extensions for Python (even Python's own) you need to rebase all the DLLs that can be potentially loaded from an application. In Sage's case we are talking about 190 extension in addition to numerous other DLLs. For example: as is if I build Python 2.5 on my Cygwin install Python fails to start since the socket extension has a address conflict with another DLL. So one needs do drop out of all Cygwin shells and use the ash shell and a script rebaseall (which in turn calls rebase.exe) to rebase all libraries. To be 100% save you need to drop out of the build process and rebase all DLLs everytime we build a native Python extension. On top of the above the total address space on Cygwin is limited to 1.3GB. So having mutliple DLLs due to symlinks and sonames we constrict *and* fragment the available address space even more. - so I intend to remove every one of those version patches in the future *or* offer the option to build versioned or non-versioned libraries. That should make either camp happy. To me this seems to be an issue that is better discussed first. Quite likely, you have very good arguments for the statements above (and you know way more about this than I'll ever know), but I would like to hear them first. Well, all of the above IMHO clearly adds up to the conclusion I arrived at :). I am not saying versioned libraries are bad in general, but for the Sage project the disadvantages way outweigh the benefit. So while versioned libraries are essential for the Debian project they are a hassle for the Sage project. Cheers, Martin Cheers, Michael -- name: Martin Albrecht _pgp:http://pgp.mit.edu:11371/pks/lookup?op=getsearch=0x8EF0DC99 _www:http://www.informatik.uni-bremen.de/~malb _jab: [EMAIL PROTECTED] --~--~-~--~~~---~--~~ To post to this group, send email to sage-devel@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sage-devel URLs: http://www.sagemath.org -~--~~~~--~~--~--~---
[sage-devel] Re: C/C++ Library Versioning in Sage
- Sage in general does not benefit from versioned libraries Is adding a proper soname / library versioning to a library what you call versioned libraries? Isn't that somehow standardised and good practice? Sure, on Linux and some Unix platforms. But doing it right, i.e. for non-GNU linkers is complicated. I do not oppose using soname friends, but if we introduce them they ought to work on more than Linux :) Agreed. So the main objection is improper use of soname and friends by not using autoconf or whatever the currently popular wanna-be autoconf-killer is? - indeed they case a bunch of problems when linking extensions Why? For example: Take the FLINT.spkg - ever since we used the patches Tim submitted the spkg was broken twice by either not adjusting the link name for libfint.so or after installing an updated version Sage would no longer start since we linked against libflint-1.0.13.so. We do need to force a rebuild of the extensions that link against FLINT anyway, so this is manageable, but just imagine what happens if we changed ntl. Version numbers are supposed to change whenever binary incompatibility is introduced so changing the name actually helps. Also, when upgrading Sage we're rebuilding the library anyway. The not adjusted linkname can be fixed by using proper autoconf and friends. - due to the links the archives get larger Do you mean the symbolic links? I don't see how they contribute to the size significantly. Not the link, but not all tar utilities are smart enough to deal with links properly. And saving a hundred or so MB install size seems to matter. Which 'tar' tool doesn't understand symbolic links? That should be a crime. - and on top of that it makes the memory address space on Cygwin even more scare Can you elaborate on that? Cygwin is stupid and often causes libraries to have collisions in address space. So when you compile a bunch of DLL extensions for Python (even Python's own) you need to rebase all the DLLs that can be potentially loaded from an application. In Sage's case we are talking about 190 extension in addition to numerous other DLLs. For example: as is if I build Python 2.5 on my Cygwin install Python fails to start since the socket extension has a address conflict with another DLL. So one needs do drop out of all Cygwin shells and use the ash shell and a script rebaseall (which in turn calls rebase.exe) to rebase all libraries. To be 100% save you need to drop out of the build process and rebase all DLLs everytime we build a native Python extension. On top of the above the total address space on Cygwin is limited to 1.3GB. So having mutliple DLLs due to symlinks and sonames we constrict *and* fragment the available address space even more. We still only link *one* version -- say foo.so pointing to foo.so.1.2 -- so I don't see how we populate the limited 1.3GB address space with redundant libraries. To me this seems to be an issue that is better discussed first. Quite likely, you have very good arguments for the statements above (and you know way more about this than I'll ever know), but I would like to hear them first. Well, all of the above IMHO clearly adds up to the conclusion I arrived at :). I am not saying versioned libraries are bad in general, but for the Sage project the disadvantages way outweigh the benefit. So while versioned libraries are essential for the Debian project they are a hassle for the Sage project. So far it seems to me that all of your remarks above can be addressed by proper versioning (via autoconf and friends). Cheers, Martin -- name: Martin Albrecht _pgp: http://pgp.mit.edu:11371/pks/lookup?op=getsearch=0x8EF0DC99 _www: http://www.informatik.uni-bremen.de/~malb _jab: [EMAIL PROTECTED] --~--~-~--~~~---~--~~ To post to this group, send email to sage-devel@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sage-devel URLs: http://www.sagemath.org -~--~~~~--~~--~--~---
[sage-devel] Re: C/C++ Library Versioning in Sage
On Sep 26, 12:28 pm, Martin Albrecht [EMAIL PROTECTED] wrote: - Sage in general does not benefit from versioned libraries Is adding a proper soname / library versioning to a library what you call versioned libraries? Isn't that somehow standardised and good practice? Sure, on Linux and some Unix platforms. But doing it right, i.e. for non-GNU linkers is complicated. I do not oppose using soname friends, but if we introduce them they ought to work on more than Linux :) Agreed. So the main objection is improper use of soname and friends by not using autoconf or whatever the currently popular wanna-be autoconf-killer is? Well, in theory: yes. But autoconf does not work on Windows and various spkgs do neither use autoconf or SCons. And I do not want to move more code to use autoconf since IMHO it plainly sucks. There are alternatives that work on Windows for sure, but that is likely even more work since few people use SCons for example. One can always have two build systems, i.e one for Windows and one for the rest. - indeed they case a bunch of problems when linking extensions Why? For example: Take the FLINT.spkg - ever since we used the patches Tim submitted the spkg was broken twice by either not adjusting the link name for libfint.so or after installing an updated version Sage would no longer start since we linked against libflint-1.0.13.so. We do need to force a rebuild of the extensions that link against FLINT anyway, so this is manageable, but just imagine what happens if we changed ntl. Version numbers are supposed to change whenever binary incompatibility is introduced so changing the name actually helps. Also, when upgrading Sage we're rebuilding the library anyway. The not adjusted linkname can be fixed by using proper autoconf and friends. But the above problems do also cause trouble when I upgrade an spkg when doing merges, i.e. if I updated ntl I would hate to rebuild all of Sage. As is the soname of FLINT is changed *every* time the minor version is bumped. - due to the links the archives get larger Do you mean the symbolic links? I don't see how they contribute to the size significantly. Not the link, but not all tar utilities are smart enough to deal with links properly. And saving a hundred or so MB install size seems to matter. Which 'tar' tool doesn't understand symbolic links? That should be a crime. I agree, but there is a shitload of unpackers out there that are plain stupid. It does not have to be traditional tar. - and on top of that it makes the memory address space on Cygwin even more scare Can you elaborate on that? Cygwin is stupid and often causes libraries to have collisions in address space. So when you compile a bunch of DLL extensions for Python (even Python's own) you need to rebase all the DLLs that can be potentially loaded from an application. In Sage's case we are talking about 190 extension in addition to numerous other DLLs. For example: as is if I build Python 2.5 on my Cygwin install Python fails to start since the socket extension has a address conflict with another DLL. So one needs do drop out of all Cygwin shells and use the ash shell and a script rebaseall (which in turn calls rebase.exe) to rebase all libraries. To be 100% save you need to drop out of the build process and rebase all DLLs everytime we build a native Python extension. On top of the above the total address space on Cygwin is limited to 1.3GB. So having mutliple DLLs due to symlinks and sonames we constrict *and* fragment the available address space even more. We still only link *one* version -- say foo.so pointing to foo.so.1.2 -- so I don't see how we populate the limited 1.3GB address space with redundant libraries. *Every* DLL on that list is rebase, it does not matter if they are loaded or not. And that causes address space fragmentation. To me this seems to be an issue that is better discussed first. Quite likely, you have very good arguments for the statements above (and you know way more about this than I'll ever know), but I would like to hear them first. Well, all of the above IMHO clearly adds up to the conclusion I arrived at :). I am not saying versioned libraries are bad in general, but for the Sage project the disadvantages way outweigh the benefit. So while versioned libraries are essential for the Debian project they are a hassle for the Sage project. So far it seems to me that all of your remarks above can be addressed by proper versioning (via autoconf and friends). I strongly disagree. gmp for example provides no versioning info at all on Windows when using either MinGW or Cygwin and you can be sure that TG is not someone who prefers not to do the right thing. It is plainly just not worth it in Sage to put up with all the added complexity if we don't even use the functionality. I don't mind if the packages we use provide
