Re: [sage-devel] Re: Add 'gcc' libraries to Sage binaries ( 0.5% bloat)
Kasper Peeters wrote: I'd propose that we include in any binary distribution gcc's C, C++ and Fortran shared libraries. I personally think that this is a _very_ bad idea. As others have emphasised, most systems out there have a proper package management tool, which can moreover take care of dependencies. By doing all that yourself, you also burden yourself with the task of keeping the Sage-packaged external programs and libraries up to date. I'm not saying there are not ways of doing this. But a small, though not insignificant number of people seem to be getting bitten by a failure to have the required libraries. In fact, after a fortran compiler was removed from Sage on Linux (there is still one for OS X), there were instances of those without the Fortran library, so the addition of the Fortran library was made a 'blocker' by William. http://trac.sagemath.org/sage_trac/ticket/8049 So the Fortran library is shipped. I just looked at an install of Mathematica on Solaris and notice they ship 'libzip.so' and 'libsqlite.so' as well as the java runtime environment libmawt.so. In fact, there are tons of libraries which could be found on some systems, but Wolfram Research obvious feel the desire to include them. While on this topic, can anyone point me to a good read on why Sage includes every known piece of software under the sun in its distribution? I would personally prefer to get rid of that _all_ and instead use the energy to support deb/rpm/ pkg/... maintainers. Cheers, Kasper The reason is most of the packages in Sage have small modifications. To suggest to users that they download X, but modifications A, B and C, then download Y, but make modifications D, E and F would put off too many people. Have a look in $SAGE_ROOT/spkg/standard, then look for the packages that end in .p0.spkg, .p1.spkg, .p2.spkg. All of them have had some changes. Often that are modifications of the source code. ATLAS has modifications to take care of a bug in a memset() in some older Solaris releases. libz has changes to allow a 64-bit build on OS X. Those are two I can think of, but there are tons of them. To be honest, as Sage consists of about 100 items, it would be hard to know where to start with some bug reports when you don't know precisely what versions of the libraries people have, and whether those libraries have been updated to a version Sage is not tested with. Also, some of the libraries might change to GPL 3. If those libraries did not form part of the core operating system, Sage should not link to them. As much as I can see why people do not like this, I can see a lot of logic in William's approach. Dave -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
Re: [sage-devel] Re: Add 'gcc' libraries to Sage binaries ( 0.5% bloat)
As much as I can see why people do not like this, I can see a lot of logic in William's approach. To add to what David said, some of this is historical. The 100 packages sage builds now are unwieldy, but it was not always so. Earlier distributions contained much less, and shipping them all seemed like a reasonable trade for build anywhere. It's worked surprisingly well, and we are attached to it now. A thought occurs -- perhaps we could have sage .spkg packages that in fact are just dummies that really use the system libraries. That might provide a transition path to mixed .spkg/system libraries. Nick -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
Re: [sage-devel] Re: Add 'gcc' libraries to Sage binaries ( 0.5% bloat)
On Tue, Feb 23, 2010 at 1:37 AM, Dr. David Kirkby david.kir...@onetel.net wrote: Kasper Peeters wrote: I'd propose that we include in any binary distribution gcc's C, C++ and Fortran shared libraries. I personally think that this is a _very_ bad idea. As others have emphasised, most systems out there have a proper package management tool, which can moreover take care of dependencies. By doing all that yourself, you also burden yourself with the task of keeping the Sage-packaged external programs and libraries up to date. I'm not saying there are not ways of doing this. But a small, though not insignificant number of people seem to be getting bitten by a failure to have the required libraries. In fact, after a fortran compiler was removed from Sage on Linux (there is still one for OS X), there were instances of those without the Fortran library, so the addition of the Fortran library was made a 'blocker' by William. http://trac.sagemath.org/sage_trac/ticket/8049 So the Fortran library is shipped. I just looked at an install of Mathematica on Solaris and notice they ship 'libzip.so' and 'libsqlite.so' as well as the java runtime environment libmawt.so. In fact, there are tons of libraries which could be found on some systems, but Wolfram Research obvious feel the desire to include them. While on this topic, can anyone point me to a good read on why Sage includes every known piece of software under the sun in its distribution? I would personally prefer to get rid of that _all_ and instead use the energy to support deb/rpm/ pkg/... maintainers. Cheers, Kasper The reason is most of the packages in Sage have small modifications. To suggest to users that they download X, but modifications A, B and C, then download Y, but make modifications D, E and F would put off too many people. Have a look in $SAGE_ROOT/spkg/standard, then look for the packages that end in .p0.spkg, .p1.spkg, .p2.spkg. All of them have had some changes. Often that are modifications of the source code. ATLAS has modifications to take care of a bug in a memset() in some older Solaris releases. libz has changes to allow a 64-bit build on OS X. Those are two I can think of, but there are tons of them. To be honest, as Sage consists of about 100 items, it would be hard to know where to start with some bug reports when you don't know precisely what versions of the libraries people have, and whether those libraries have been updated to a version Sage is not tested with. Also, some of the libraries might change to GPL 3. If those libraries did not form part of the core operating system, Sage should not link to them. This GPL3 comment is not really an issue. As much as I can see why people do not like this, I can see a lot of logic in William's approach. I'm going to add a few comments supporting this. The goal of Sage is to compete with the likes of all of Magma, Mathematica and Matlab, and as such it is a necessarily a complicated piece of software, being more complicated than any one of those Ma's. Most people don't appreciate how complicated Sage is. At this point, I don't think I even do. It would be better for end users if we built standard rpm/deb/etc. packages that integrate well with the rest of each Linux, OS X, Solaris, Windows, etc., operating system, and of course regularly tested that the full test suite passes on each system, and when packages on those systems get old or too new, deal with those problems. That would be wonderful. Unfortunately, we have to be realistic, given the resources that we have available. How many major software projects with a similar level of complexity to Sage actually do this? I can think of exactly one: Open Office. That project is at least as complicated as Sage (maybe more). Who creates all of their packages? Do the Open Office developers? Or the projects such as MacPorts, Debian, Ubuntu, Mandriva, RedHat, etc.? Are there any Open Office devs reading this? Doesn't the OpenOffice project have dozens of fulltime developers, employed by IBM and Sun? Sage still has 0 fulltime devs. -- William -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
Re: [sage-devel] Re: Add 'gcc' libraries to Sage binaries ( 0.5% bloat)
Nick Alexander wrote: As much as I can see why people do not like this, I can see a lot of logic in William's approach. To add to what David said, some of this is historical. The 100 packages sage builds now are unwieldy, but it was not always so. Earlier distributions contained much less, and shipping them all seemed like a reasonable trade for build anywhere. It's worked surprisingly well, and we are attached to it now. A thought occurs -- perhaps we could have sage .spkg packages that in fact are just dummies that really use the system libraries. That might provide a transition path to mixed .spkg/system libraries. Nick One obvious advantage of shipping all the code is that you know more precisely what the person is using. I believe the thread below indicates one of issues. (Read the full thread if you wish, but here is a summary). 1) Alex Ghitza builds an Arch Linux binary using gcc 4.4.2 and puts it on the Sage web site. It works for Alex 2) Akbar Hidaya downloads that binary and finds it does not work for him. 3) The binary stops working for Alex Ghitza once he updates his gcc from 4.4.2 to 4.4.3. 4) Alex correctly guess that the user has updated his gcc. http://groups.google.com/group/sage-support/browse_thread/thread/aba48495d9c09e03 I think this is an rare example, but it shows how just updating gcc can cause a failure. I'm sure having a version of gcc too old will also cause problems, which is why I suggested including the gcc libraries. Now consider what happens when the end user can have loads of different libraries at different versions. Debugging problems would become much more problematic. Simply insisting that the user upgrade their libraries will not be practical if the user does not have root access. Many system admins will be unwilling to update system libraries for users. They simply can't take the risk the updated libraries will not cause problems for other uses. In an ideal world, updated libraries would be backwards compatible and introduce no new bugs. In practice, neither are always true. I'm also unaware of any way of determining library versions in general. Dave -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
Re: [sage-devel] Re: Add 'gcc' libraries to Sage binaries ( 0.5% bloat)
On Feb 23, 2010, at 12:48 AM, Kasper Peeters wrote: While on this topic, can anyone point me to a good read on why Sage includes every known piece of software under the sun in its distribution? I would personally prefer to get rid of that _all_ and instead use the energy to support deb/rpm/ pkg/... maintainers. This issue has come up several times before, I don't have a specific thread to point to, but I would recommend searching the mailing list archives. There are projects underway to split Sage into separate packages, but it's a massive effort to do so and no one's successfully gotten a fully working Sage that way (though we're closer than we've ever been). I'm unconvinced it would be less effort on our part (maintaing and up-streaming all of deb/rpm/ports/cygwin/??? rather than just our own extremely simple setup) as well. - Robert -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
Re: [sage-devel] Re: Add 'gcc' libraries to Sage binaries ( 0.5% bloat)
On Feb 23, 2010, at 1:44 PM, Kasper Peeters wrote: It would be better for end users if we built standard rpm/deb/etc. packages that integrate well with the rest of each Linux, OS X, Solaris, Windows, etc., operating system, and of course regularly tested that the full test suite passes on each system, and when packages on those systems get old or too new, deal with those problems. That would be wonderful. Unfortunately, we have to be realistic, given the resources that we have available. The problem is that, by relying on your own versions of many tools and libraries, you essentially prevent other people from helping you with packaging for those rpm/deb/etc. systems. Because those other people are now burdened with the task of separating those libraries and patching things on the Sage side. Which is harder for them, because they do not know what to watch out for. As a result, you are burdened with more work, because now _you_ have to take care of making Sage installation/compilation easy, instead of having distribution packagers take care of that. We would have to take care of making Sage installation/compilation easy anyways, as many of the packages we rely on (often the most fragile ones) are rather specialized math programs that aren't always in all the standard distributions, and it would fall on us to do all the work to get them into all the packaging systems. And we would probably have to roll our own for Solaris, OS X, or any other platform that doesn't yet have a large packaging distribution infrastructure. It would also make Sage releases much more dependent on outside people and projects--right now if one finds a bug in an upstream package one can get it into the next sage release rather than wait for upstream to accept and push it. Also if there were one package repository/system that all supported platforms used, things would be much simpler. I have some personal experience with Cython. I'm very greatful for whoever makes .dep and .rpm packages of Cython, but I do have to admit that they're not always up to date and I'm glad I can get the new Cython into Sage before waiting for every packaging system we would rely on to finally upgrade to the latest. The consequence is clearly visible: there are no up-to-date packages of Sage for any of the Linux distributions. I'm playing devils advocate here a bit, but there are up-to-date binaries of Sage for many Linux distributions, so it would seem that's (for the moment) a more effective way to go. Note that if they accepted monolithic packages this would be much easier. Which means that for many users, installing Sage is actually _harder_ than installing most other software. That is sad but true. But don't get me wrong: I am definitely not saying that it is easy to make Sage deal with distribution-supplied libraries, and I fully understand the historical reasons for shipping Sage with such a large number of spkg's. What I am saying is that, in the long run, it is much better to steer clear of that. And therefore, that it would be unwise to add more of your own versions of libraries, _especially_ when it concerns libraries which every distribution already ships with (like gcc-lib). I can see that Macports is having problems with the 'we ship everything ourselves' logic, and I don't see how Sage is going to avoid that. Despite my defense of the way things are, I am very excited about the recent push for the Gentoo port, and I'm curious to see how it will be maintaining it over, say, the next year. We will see if, for example, library versioning issues are a major headache or non-issue. Perhaps eventually this will even become the default way of building/ distributing Sage for Linux. I'm not sure how we would move away from the monolithic source and binary model for other platforms though, especially given our mission of being a viable alternative to the commercial offerings. - Robert -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
Re: [sage-devel] Re: Add 'gcc' libraries to Sage binaries ( 0.5% bloat)
On Feb 23, 2010, at 9:53 PM, Robert Bradshaw wrote: On Feb 23, 2010, at 1:44 PM, Kasper Peeters wrote: It would be better for end users if we built standard rpm/deb/etc. packages that integrate well with the rest of each Linux, OS X, Solaris, Windows, etc., operating system, and of course regularly tested that the full test suite passes on each system, and when packages on those systems get old or too new, deal with those problems. That would be wonderful. Unfortunately, we have to be realistic, given the resources that we have available. The problem is that, by relying on your own versions of many tools and libraries, you essentially prevent other people from helping you with packaging for those rpm/deb/etc. systems. Because those other people are now burdened with the task of separating those libraries and patching things on the Sage side. Which is harder for them, because they do not know what to watch out for. As a result, you are burdened with more work, because now _you_ have to take care of making Sage installation/compilation easy, instead of having distribution packagers take care of that. We would have to take care of making Sage installation/compilation easy anyways, as many of the packages we rely on (often the most fragile ones) are rather specialized math programs that aren't always in all the standard distributions, and it would fall on us to do all the work to get them into all the packaging systems. And we would probably have to roll our own for Solaris, OS X, or any other platform that doesn't yet have a large packaging distribution infrastructure. It would also make Sage releases much more dependent on outside people and projects--right now if one finds a bug in an upstream package one can get it into the next sage release rather than wait for upstream to accept and push it. Also if there were one package repository/system that all supported platforms used, things would be much simpler. I have some personal experience with Cython. I'm very greatful for whoever makes .dep and .rpm packages of Cython, but I do have to admit that they're not always up to date and I'm glad I can get the new Cython into Sage before waiting for every packaging system we would rely on to finally upgrade to the latest. The consequence is clearly visible: there are no up-to-date packages of Sage for any of the Linux distributions. I'm playing devils advocate here a bit, but there are up-to-date binaries of Sage for many Linux distributions, so it would seem that's (for the moment) a more effective way to go. Note that if they accepted monolithic packages this would be much easier. Which means that for many users, installing Sage is actually _harder_ than installing most other software. That is sad but true. But don't get me wrong: I am definitely not saying that it is easy to make Sage deal with distribution-supplied libraries, and I fully understand the historical reasons for shipping Sage with such a large number of spkg's. What I am saying is that, in the long run, it is much better to steer clear of that. And therefore, that it would be unwise to add more of your own versions of libraries, _especially_ when it concerns libraries which every distribution already ships with (like gcc-lib). I can see that Macports is having problems with the 'we ship everything ourselves' logic, and I don't see how Sage is going to avoid that. Despite my defense of the way things are, I am very excited about the recent push for the Gentoo port, and I'm curious to see how it will be maintaining it over, say, the next year. We will see if, for example, library versioning issues are a major headache or non- issue. Perhaps eventually this will even become the default way of building/distributing Sage for Linux. I'm not sure how we would move away from the monolithic source and binary model for other platforms though, especially given our mission of being a viable alternative to the commercial offerings. [pressed send too soon] Another thing I really like about the monolithic approach is that it makes it really easy to install Sage as non-root, as many times as you want, and to move and delete them. Of course some users could care less about this. - Robert -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
Re: [sage-devel] Re: Add 'gcc' libraries to Sage binaries ( 0.5% bloat)
Bill Hart wrote: On 22 Feb, 11:27, Dr. David Kirkby david.kir...@onetel.net wrote: This came up on the thread mercurial on t2 but I thought I'd start a new thread on it. I'd propose that we include in any binary distribution gcc's C, C++ and Fortran shared libraries. They would be placed in $SAGE_LOCAL/lib. Then we can ensure that people will run Sage with what libraries Sage was built with, rather than what versions they may or may not have lying around. Comments? Dave Are we sure this would work? Won't those libraries depend on what kernel is installed, etc, etc? I'm completely ignorant on this, so may be talk out my proverbial. Bill. I would not have thought so, though I don't claim to be an expert on this. I just checked the sage-4.3.2-linux-64bit-ubuntu_9.10-x86_64-Linux binary, and find the fortran library libgfortran.so is already there, as otherwise one would need to have gfortran installed to run Sage. Since that is not causing any particular kernel issues, I doubt the C or C++ libraries would either. (As always, there can always be unwelcome surprises.) The file COPYING.LIB in the gcc source distribution specifically allows copying the libraries for the purposes of running code on other systems, without including the source code for your proprietary program. I don't think the gcc developers would bother if the version of the library depended on the kernel version in use on the other system. Of course, we already have to produce different binaries for different Linux distributions. So I'd suggest just copying the gcc libraries from whatever system one built Sage on. The libraries I put in the Solaris binary were copied from a Sun running the very first Solaris 10 release (03/2005). The kernel on both t2 and my other Solaris box are both late 2009. So even with 4.5 years of kernel updates, the gcc libraries still work. (I know Linux tends to have poorer backward compatibility than Solaris, but I doubt that would be an issue even on Linux). Dave -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
Re: [sage-devel] Re: Add 'gcc' libraries to Sage binaries ( 0.5% bloat)
Volker Braun wrote: If you want to go that route you probably want to include glibc (contains standard math library) as well. While a viable possibility, there are two obvious arguments against it: On Feb 22, 11:27 am, Dr. David Kirkby david.kir...@onetel.net wrote: * Always have the libraries. Instead, check on first startup that all required packages are installed and print an error message instead ('Please yum install libgfortran'). I assume one would need root to do that, but perhaps not. In any case, we already include the fortran library. The question is whether to extend that to C and C++ too. * Always have the exact same versions Sage was built with. Downside: if your distribution comes out with a bugfix you don't automatically profit from it. True, though the Sage developers tend to build the binaries with late gcc versions. In which case bug fixes in the libraries are more likely to be included in the Sage binary than they are on the users system unless that user has kept more up to date with gcc than the Sage developers. Problems are more likely to occur if the user has an older buggy gcc, but the Sage binary was built and tested using a newer gcc. Also, if your distribution keeps breaking the C++ ABI then maybe its time to switch. Well, it was annoying that simply updating gcc to 4.4.3 from a slightly earlier release caused problems. But including the libraries should avoid that. At least with the libraries included, the user gets to use Sage with libraries where Sage has been tested. I know Mathematica on Solaris ships with libraries that are included on Solaris. I assume Wolfram Research feel happier knowing that Mathematica runs with libraries which it was tested on, rather than what the end user may have. Dave -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
Re: [sage-devel] Re: Add 'gcc' libraries to Sage binaries ( 0.5% bloat)
Bill Hart wrote: Actually, this is really silly. The must be a multitude of packages for Solaris which are distributed with binaries and which need these libraries. How do they deal with this issue? Is there a package installation system like apt-get or yum or rpm on Solaris? They must surely deal with this issue somehow? Bill. Yes - it called 'pkgadd' Blastwave is a site where you pick a package you want (e.g. python) and it downloads that along with any dependences. Here are those for phthon http://www.blastwave.org/jir/pkgcontents.ftd?software=pythonstyle=briefstate=5arch=i386 One of which is GNU Compiler Collection Version 4.3.3 Core RunTime which is basically the libraries. Sunfreeware (run by a consultant for Wolfram Reserarch) http://www.sunfreeware.com/ has loads of packages in binary format, though the site has stagnated. Let's pick the first package (a2ps-4.14), a2ps-4.14 Having libgcc_s.so.1 in /usr/local/lib is required. This can be done by installing libgcc-3.4.6 or gcc-3.4.6 or higher. So it basically boils down to one of two choices. 1) Install gcc. 2) Install just the gcc libraries. When I created the binary which runs on 't2nb' (which is a zone on Solaris) http://t2nb.math.washington.edu:8000 I included the libraries. There is no gcc on that Solaris zone, so Sage would not run without them. In contrast, if you have gcc installed, and the libraries can be found via LD_LIBRARY_PATH or some other means, then those libraries are not needed. I personally think it is a bit safer to include them, but I can see counter arguments. The most convincing one is from Robert Bradsure who said: --- Are there issues if the gcc and core C libraries don't match? For example, If a user uses cython in the notebook, will it pick up the users g++ and Sage's libstdc++? What about someone who starts with a binary then starts developing with it or installs optional packages? -- The truth is, I don't know the answer to that one. It's clear the current system does cause some problems (not just on Solaris), but I must admit Robert's argument may be valid. Dave -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org