[sage-devel] Re: Add 'gcc' libraries to Sage binaries ( 0.5% bloat)
Are there issues if the gcc and core C libraries don't match? For example, If a user uses cython in the notebook, will it pick up the users g++ and Sage's libstdc++? What about someone who starts with a binary then starts developing with it or installs optional packages? On OS X, the MacPorts folks have just been bitten by exactly these kind of problems. They ship everything themselves, including gcc and the libraries. Unfortunately, Apple does not always use the same compiler flags as MacPorts, and there are thus incompatible libraries floating around on those systems (which means: if you use MacPorts, and it links to a standard Apple library which is not present in MacPorts, your program will crash). Packing your own versions of system libraries is _never_ a good idea, and bound to lead to all sorts of problems in the long run. Cheers, Kasper -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
Re: [sage-devel] Re: Add 'gcc' libraries to Sage binaries ( 0.5% bloat)
Kasper Peeters wrote: I'd propose that we include in any binary distribution gcc's C, C++ and Fortran shared libraries. I personally think that this is a _very_ bad idea. As others have emphasised, most systems out there have a proper package management tool, which can moreover take care of dependencies. By doing all that yourself, you also burden yourself with the task of keeping the Sage-packaged external programs and libraries up to date. I'm not saying there are not ways of doing this. But a small, though not insignificant number of people seem to be getting bitten by a failure to have the required libraries. In fact, after a fortran compiler was removed from Sage on Linux (there is still one for OS X), there were instances of those without the Fortran library, so the addition of the Fortran library was made a 'blocker' by William. http://trac.sagemath.org/sage_trac/ticket/8049 So the Fortran library is shipped. I just looked at an install of Mathematica on Solaris and notice they ship 'libzip.so' and 'libsqlite.so' as well as the java runtime environment libmawt.so. In fact, there are tons of libraries which could be found on some systems, but Wolfram Research obvious feel the desire to include them. While on this topic, can anyone point me to a good read on why Sage includes every known piece of software under the sun in its distribution? I would personally prefer to get rid of that _all_ and instead use the energy to support deb/rpm/ pkg/... maintainers. Cheers, Kasper The reason is most of the packages in Sage have small modifications. To suggest to users that they download X, but modifications A, B and C, then download Y, but make modifications D, E and F would put off too many people. Have a look in $SAGE_ROOT/spkg/standard, then look for the packages that end in .p0.spkg, .p1.spkg, .p2.spkg. All of them have had some changes. Often that are modifications of the source code. ATLAS has modifications to take care of a bug in a memset() in some older Solaris releases. libz has changes to allow a 64-bit build on OS X. Those are two I can think of, but there are tons of them. To be honest, as Sage consists of about 100 items, it would be hard to know where to start with some bug reports when you don't know precisely what versions of the libraries people have, and whether those libraries have been updated to a version Sage is not tested with. Also, some of the libraries might change to GPL 3. If those libraries did not form part of the core operating system, Sage should not link to them. As much as I can see why people do not like this, I can see a lot of logic in William's approach. Dave -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
Re: [sage-devel] Re: Add 'gcc' libraries to Sage binaries ( 0.5% bloat)
As much as I can see why people do not like this, I can see a lot of logic in William's approach. To add to what David said, some of this is historical. The 100 packages sage builds now are unwieldy, but it was not always so. Earlier distributions contained much less, and shipping them all seemed like a reasonable trade for build anywhere. It's worked surprisingly well, and we are attached to it now. A thought occurs -- perhaps we could have sage .spkg packages that in fact are just dummies that really use the system libraries. That might provide a transition path to mixed .spkg/system libraries. Nick -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
Re: [sage-devel] Re: Add 'gcc' libraries to Sage binaries ( 0.5% bloat)
On Tue, Feb 23, 2010 at 1:37 AM, Dr. David Kirkby david.kir...@onetel.net wrote: Kasper Peeters wrote: I'd propose that we include in any binary distribution gcc's C, C++ and Fortran shared libraries. I personally think that this is a _very_ bad idea. As others have emphasised, most systems out there have a proper package management tool, which can moreover take care of dependencies. By doing all that yourself, you also burden yourself with the task of keeping the Sage-packaged external programs and libraries up to date. I'm not saying there are not ways of doing this. But a small, though not insignificant number of people seem to be getting bitten by a failure to have the required libraries. In fact, after a fortran compiler was removed from Sage on Linux (there is still one for OS X), there were instances of those without the Fortran library, so the addition of the Fortran library was made a 'blocker' by William. http://trac.sagemath.org/sage_trac/ticket/8049 So the Fortran library is shipped. I just looked at an install of Mathematica on Solaris and notice they ship 'libzip.so' and 'libsqlite.so' as well as the java runtime environment libmawt.so. In fact, there are tons of libraries which could be found on some systems, but Wolfram Research obvious feel the desire to include them. While on this topic, can anyone point me to a good read on why Sage includes every known piece of software under the sun in its distribution? I would personally prefer to get rid of that _all_ and instead use the energy to support deb/rpm/ pkg/... maintainers. Cheers, Kasper The reason is most of the packages in Sage have small modifications. To suggest to users that they download X, but modifications A, B and C, then download Y, but make modifications D, E and F would put off too many people. Have a look in $SAGE_ROOT/spkg/standard, then look for the packages that end in .p0.spkg, .p1.spkg, .p2.spkg. All of them have had some changes. Often that are modifications of the source code. ATLAS has modifications to take care of a bug in a memset() in some older Solaris releases. libz has changes to allow a 64-bit build on OS X. Those are two I can think of, but there are tons of them. To be honest, as Sage consists of about 100 items, it would be hard to know where to start with some bug reports when you don't know precisely what versions of the libraries people have, and whether those libraries have been updated to a version Sage is not tested with. Also, some of the libraries might change to GPL 3. If those libraries did not form part of the core operating system, Sage should not link to them. This GPL3 comment is not really an issue. As much as I can see why people do not like this, I can see a lot of logic in William's approach. I'm going to add a few comments supporting this. The goal of Sage is to compete with the likes of all of Magma, Mathematica and Matlab, and as such it is a necessarily a complicated piece of software, being more complicated than any one of those Ma's. Most people don't appreciate how complicated Sage is. At this point, I don't think I even do. It would be better for end users if we built standard rpm/deb/etc. packages that integrate well with the rest of each Linux, OS X, Solaris, Windows, etc., operating system, and of course regularly tested that the full test suite passes on each system, and when packages on those systems get old or too new, deal with those problems. That would be wonderful. Unfortunately, we have to be realistic, given the resources that we have available. How many major software projects with a similar level of complexity to Sage actually do this? I can think of exactly one: Open Office. That project is at least as complicated as Sage (maybe more). Who creates all of their packages? Do the Open Office developers? Or the projects such as MacPorts, Debian, Ubuntu, Mandriva, RedHat, etc.? Are there any Open Office devs reading this? Doesn't the OpenOffice project have dozens of fulltime developers, employed by IBM and Sun? Sage still has 0 fulltime devs. -- William -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
[sage-devel] Re: Add 'gcc' libraries to Sage binaries ( 0.5% bloat)
Just from a purely pragmatic perspective, I would love if for binary builds the libraries (especially libstdc++) used during building would be *available* (not necessarily linked to) by default. Several times I ran into a situation where I quickly wanted to test something with a new version of sage on a (redhat or fedora) machine for which no precisely matching binary was available. libstdc++ seems to be a particularly volatile library (i.e., gets upgraded with pretty much every OS release), and hence binaries that were one version off would not work. After tracking down the appropriate version of libstdc++ and putting that in $SAGE_ROOT/local/lib, everything worked fine. In other situations I gave up on finding the appropriate library and submitted to a 3 hour compile from source or abandoned the project. I would have much preferred to have done something like ln -s $SAGE_ROOT/buildtime/lib/* $SAGE_ROOT/local/lib and got on with the job. I do not control which version of the OS these machines are running and do not have administrative permissions on them. -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
Re: [sage-devel] Re: Add 'gcc' libraries to Sage binaries ( 0.5% bloat)
Nick Alexander wrote: As much as I can see why people do not like this, I can see a lot of logic in William's approach. To add to what David said, some of this is historical. The 100 packages sage builds now are unwieldy, but it was not always so. Earlier distributions contained much less, and shipping them all seemed like a reasonable trade for build anywhere. It's worked surprisingly well, and we are attached to it now. A thought occurs -- perhaps we could have sage .spkg packages that in fact are just dummies that really use the system libraries. That might provide a transition path to mixed .spkg/system libraries. Nick One obvious advantage of shipping all the code is that you know more precisely what the person is using. I believe the thread below indicates one of issues. (Read the full thread if you wish, but here is a summary). 1) Alex Ghitza builds an Arch Linux binary using gcc 4.4.2 and puts it on the Sage web site. It works for Alex 2) Akbar Hidaya downloads that binary and finds it does not work for him. 3) The binary stops working for Alex Ghitza once he updates his gcc from 4.4.2 to 4.4.3. 4) Alex correctly guess that the user has updated his gcc. http://groups.google.com/group/sage-support/browse_thread/thread/aba48495d9c09e03 I think this is an rare example, but it shows how just updating gcc can cause a failure. I'm sure having a version of gcc too old will also cause problems, which is why I suggested including the gcc libraries. Now consider what happens when the end user can have loads of different libraries at different versions. Debugging problems would become much more problematic. Simply insisting that the user upgrade their libraries will not be practical if the user does not have root access. Many system admins will be unwilling to update system libraries for users. They simply can't take the risk the updated libraries will not cause problems for other uses. In an ideal world, updated libraries would be backwards compatible and introduce no new bugs. In practice, neither are always true. I'm also unaware of any way of determining library versions in general. Dave -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
[sage-devel] Re: Add 'gcc' libraries to Sage binaries ( 0.5% bloat)
There is a fundamental difference between math and system libraries. Specialist software receives much less testing, especially on exotic architectures. You can easily be the first one who builds X on Y and run into some obscure bug. So it is valuable to collect mathematical programs and make sure that they build on a variety of systems and collect the patches to do so. System libraries, on the other hand, receive much more attention. Just building a whole distribution with a given libstdc++, say, is a great testcase. Since you mention zlib, Fedora applies 2 patches to that library. Does the zlib spkg also include those? Compiler and more general system libraries are even worse. A lot of work goes into core libraries, and it would be a total waste to duplicate that effort. The default should be to not duplicate system libraries. Only in those (few) cases where the OS does not have a usable packaging system / ships a broken library / etc, the sage binary distribution (for that OS) should include the library as a workaround. -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
Re: [sage-devel] Re: Add 'gcc' libraries to Sage binaries ( 0.5% bloat)
On Feb 23, 2010, at 12:48 AM, Kasper Peeters wrote: While on this topic, can anyone point me to a good read on why Sage includes every known piece of software under the sun in its distribution? I would personally prefer to get rid of that _all_ and instead use the energy to support deb/rpm/ pkg/... maintainers. This issue has come up several times before, I don't have a specific thread to point to, but I would recommend searching the mailing list archives. There are projects underway to split Sage into separate packages, but it's a massive effort to do so and no one's successfully gotten a fully working Sage that way (though we're closer than we've ever been). I'm unconvinced it would be less effort on our part (maintaing and up-streaming all of deb/rpm/ports/cygwin/??? rather than just our own extremely simple setup) as well. - Robert -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
[sage-devel] Re: Add 'gcc' libraries to Sage binaries ( 0.5% bloat)
It would be better for end users if we built standard rpm/deb/etc. packages that integrate well with the rest of each Linux, OS X, Solaris, Windows, etc., operating system, and of course regularly tested that the full test suite passes on each system, and when packages on those systems get old or too new, deal with those problems. That would be wonderful. Unfortunately, we have to be realistic, given the resources that we have available. The problem is that, by relying on your own versions of many tools and libraries, you essentially prevent other people from helping you with packaging for those rpm/deb/etc. systems. Because those other people are now burdened with the task of separating those libraries and patching things on the Sage side. Which is harder for them, because they do not know what to watch out for. As a result, you are burdened with more work, because now _you_ have to take care of making Sage installation/compilation easy, instead of having distribution packagers take care of that. The consequence is clearly visible: there are no up-to-date packages of Sage for any of the Linux distributions. Which means that for many users, installing Sage is actually _harder_ than installing most other software. But don't get me wrong: I am definitely not saying that it is easy to make Sage deal with distribution-supplied libraries, and I fully understand the historical reasons for shipping Sage with such a large number of spkg's. What I am saying is that, in the long run, it is much better to steer clear of that. And therefore, that it would be unwise to add more of your own versions of libraries, _especially_ when it concerns libraries which every distribution already ships with (like gcc-lib). I can see that Macports is having problems with the 'we ship everything ourselves' logic, and I don't see how Sage is going to avoid that. Cheers, Kasper -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
Re: [sage-devel] Re: Add 'gcc' libraries to Sage binaries ( 0.5% bloat)
On Feb 23, 2010, at 1:44 PM, Kasper Peeters wrote: It would be better for end users if we built standard rpm/deb/etc. packages that integrate well with the rest of each Linux, OS X, Solaris, Windows, etc., operating system, and of course regularly tested that the full test suite passes on each system, and when packages on those systems get old or too new, deal with those problems. That would be wonderful. Unfortunately, we have to be realistic, given the resources that we have available. The problem is that, by relying on your own versions of many tools and libraries, you essentially prevent other people from helping you with packaging for those rpm/deb/etc. systems. Because those other people are now burdened with the task of separating those libraries and patching things on the Sage side. Which is harder for them, because they do not know what to watch out for. As a result, you are burdened with more work, because now _you_ have to take care of making Sage installation/compilation easy, instead of having distribution packagers take care of that. We would have to take care of making Sage installation/compilation easy anyways, as many of the packages we rely on (often the most fragile ones) are rather specialized math programs that aren't always in all the standard distributions, and it would fall on us to do all the work to get them into all the packaging systems. And we would probably have to roll our own for Solaris, OS X, or any other platform that doesn't yet have a large packaging distribution infrastructure. It would also make Sage releases much more dependent on outside people and projects--right now if one finds a bug in an upstream package one can get it into the next sage release rather than wait for upstream to accept and push it. Also if there were one package repository/system that all supported platforms used, things would be much simpler. I have some personal experience with Cython. I'm very greatful for whoever makes .dep and .rpm packages of Cython, but I do have to admit that they're not always up to date and I'm glad I can get the new Cython into Sage before waiting for every packaging system we would rely on to finally upgrade to the latest. The consequence is clearly visible: there are no up-to-date packages of Sage for any of the Linux distributions. I'm playing devils advocate here a bit, but there are up-to-date binaries of Sage for many Linux distributions, so it would seem that's (for the moment) a more effective way to go. Note that if they accepted monolithic packages this would be much easier. Which means that for many users, installing Sage is actually _harder_ than installing most other software. That is sad but true. But don't get me wrong: I am definitely not saying that it is easy to make Sage deal with distribution-supplied libraries, and I fully understand the historical reasons for shipping Sage with such a large number of spkg's. What I am saying is that, in the long run, it is much better to steer clear of that. And therefore, that it would be unwise to add more of your own versions of libraries, _especially_ when it concerns libraries which every distribution already ships with (like gcc-lib). I can see that Macports is having problems with the 'we ship everything ourselves' logic, and I don't see how Sage is going to avoid that. Despite my defense of the way things are, I am very excited about the recent push for the Gentoo port, and I'm curious to see how it will be maintaining it over, say, the next year. We will see if, for example, library versioning issues are a major headache or non-issue. Perhaps eventually this will even become the default way of building/ distributing Sage for Linux. I'm not sure how we would move away from the monolithic source and binary model for other platforms though, especially given our mission of being a viable alternative to the commercial offerings. - Robert -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
Re: [sage-devel] Re: Add 'gcc' libraries to Sage binaries ( 0.5% bloat)
On Feb 23, 2010, at 9:53 PM, Robert Bradshaw wrote: On Feb 23, 2010, at 1:44 PM, Kasper Peeters wrote: It would be better for end users if we built standard rpm/deb/etc. packages that integrate well with the rest of each Linux, OS X, Solaris, Windows, etc., operating system, and of course regularly tested that the full test suite passes on each system, and when packages on those systems get old or too new, deal with those problems. That would be wonderful. Unfortunately, we have to be realistic, given the resources that we have available. The problem is that, by relying on your own versions of many tools and libraries, you essentially prevent other people from helping you with packaging for those rpm/deb/etc. systems. Because those other people are now burdened with the task of separating those libraries and patching things on the Sage side. Which is harder for them, because they do not know what to watch out for. As a result, you are burdened with more work, because now _you_ have to take care of making Sage installation/compilation easy, instead of having distribution packagers take care of that. We would have to take care of making Sage installation/compilation easy anyways, as many of the packages we rely on (often the most fragile ones) are rather specialized math programs that aren't always in all the standard distributions, and it would fall on us to do all the work to get them into all the packaging systems. And we would probably have to roll our own for Solaris, OS X, or any other platform that doesn't yet have a large packaging distribution infrastructure. It would also make Sage releases much more dependent on outside people and projects--right now if one finds a bug in an upstream package one can get it into the next sage release rather than wait for upstream to accept and push it. Also if there were one package repository/system that all supported platforms used, things would be much simpler. I have some personal experience with Cython. I'm very greatful for whoever makes .dep and .rpm packages of Cython, but I do have to admit that they're not always up to date and I'm glad I can get the new Cython into Sage before waiting for every packaging system we would rely on to finally upgrade to the latest. The consequence is clearly visible: there are no up-to-date packages of Sage for any of the Linux distributions. I'm playing devils advocate here a bit, but there are up-to-date binaries of Sage for many Linux distributions, so it would seem that's (for the moment) a more effective way to go. Note that if they accepted monolithic packages this would be much easier. Which means that for many users, installing Sage is actually _harder_ than installing most other software. That is sad but true. But don't get me wrong: I am definitely not saying that it is easy to make Sage deal with distribution-supplied libraries, and I fully understand the historical reasons for shipping Sage with such a large number of spkg's. What I am saying is that, in the long run, it is much better to steer clear of that. And therefore, that it would be unwise to add more of your own versions of libraries, _especially_ when it concerns libraries which every distribution already ships with (like gcc-lib). I can see that Macports is having problems with the 'we ship everything ourselves' logic, and I don't see how Sage is going to avoid that. Despite my defense of the way things are, I am very excited about the recent push for the Gentoo port, and I'm curious to see how it will be maintaining it over, say, the next year. We will see if, for example, library versioning issues are a major headache or non- issue. Perhaps eventually this will even become the default way of building/distributing Sage for Linux. I'm not sure how we would move away from the monolithic source and binary model for other platforms though, especially given our mission of being a viable alternative to the commercial offerings. [pressed send too soon] Another thing I really like about the monolithic approach is that it makes it really easy to install Sage as non-root, as many times as you want, and to move and delete them. Of course some users could care less about this. - Robert -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
[sage-devel] Re: Add 'gcc' libraries to Sage binaries ( 0.5% bloat)
Are we sure this would work? Won't those libraries depend on what kernel is installed, etc, etc? I'm completely ignorant on this, so may be talk out my proverbial. Bill. On 22 Feb, 11:27, Dr. David Kirkby david.kir...@onetel.net wrote: This came up on the thread mercurial on t2 but I thought I'd start a new thread on it. I'd propose that we include in any binary distribution gcc's C, C++ and Fortran shared libraries. They would be placed in $SAGE_LOCAL/lib. Then we can ensure that people will run Sage with what libraries Sage was built with, rather than what versions they may or may not have lying around. The amount of bloat this would add to the binary would be very small. For Solaris, the compressed sizes of the files are: -rwxr-xr-x 1 drkirkby staff 1.5M Feb 22 10:10 libstdc++.so.6.0.10.gz -rwxr-xr-x 1 drkirkby staff 717K Feb 22 10:10 libgfortran.so.3.0.0.gz -rw-r--r-- 1 drkirkby staff 80K Feb 22 10:10 libgcc_s.so.1.gz So adding all 3 adds 2.3 MB of extra code to the binary. But given the binary is 500 MB (not untypical), that is less than 0.5% of bloat. By doing this, we ensure that people * Always have the libraries. * Always have the exact same versions Sage was built with. I believe the Fortran library might already be included for Linux (I have not checked), but I'd suggest all 3 were added to binaries. The C library is the one people most likely will have, but given it is by far the smallest, we might as well include it to be 100% sure. Comments? Dave -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
[sage-devel] Re: Add 'gcc' libraries to Sage binaries ( 0.5% bloat)
Its a good thing that we already make available binaries for those people with less Linux experience Whatever we can do to make Sage work out of the box is good (i.e. I know its 99% there but it will be even better if we can avoid asking people to ensure certain things are installed and are certain versions - even when people choose to build) So, in a nutshell, its a +1 from me FWIW (Of course this is subject to any vetos from anyone saying why this wont work under certain circumstances) On Feb 22, 10:52 pm, Bill Hart goodwillh...@googlemail.com wrote: Are we sure this would work? Won't those libraries depend on what kernel is installed, etc, etc? I'm completely ignorant on this, so may be talk out my proverbial. Bill. On 22 Feb, 11:27, Dr. David Kirkby david.kir...@onetel.net wrote: This came up on the thread mercurial on t2 but I thought I'd start a new thread on it. I'd propose that we include in any binary distribution gcc's C, C++ and Fortran shared libraries. They would be placed in $SAGE_LOCAL/lib. Then we can ensure that people will run Sage with what libraries Sage was built with, rather than what versions they may or may not have lying around. The amount of bloat this would add to the binary would be very small. For Solaris, the compressed sizes of the files are: -rwxr-xr-x 1 drkirkby staff 1.5M Feb 22 10:10 libstdc++.so.6.0.10.gz -rwxr-xr-x 1 drkirkby staff 717K Feb 22 10:10 libgfortran.so.3.0.0.gz -rw-r--r-- 1 drkirkby staff 80K Feb 22 10:10 libgcc_s.so.1.gz So adding all 3 adds 2.3 MB of extra code to the binary. But given the binary is 500 MB (not untypical), that is less than 0.5% of bloat. By doing this, we ensure that people * Always have the libraries. * Always have the exact same versions Sage was built with. I believe the Fortran library might already be included for Linux (I have not checked), but I'd suggest all 3 were added to binaries. The C library is the one people most likely will have, but given it is by far the smallest, we might as well include it to be 100% sure. Comments? Dave -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
[sage-devel] Re: Add 'gcc' libraries to Sage binaries ( 0.5% bloat)
If you want to go that route you probably want to include glibc
(contains standard math library) as well. While a viable possibility,
there are two obvious arguments against it:
On Feb 22, 11:27 am, Dr. David Kirkby david.kir...@onetel.net
wrote:
* Always have the libraries.
Instead, check on first startup that all required packages are
installed and print an error message instead ('Please yum install
libgfortran').
* Always have the exact same versions Sage was built with.
Downside: if your distribution comes out with a bugfix you don't
automatically profit from it. Also, if your distribution keeps
breaking the C++ ABI then maybe its time to switch.
--
To post to this group, send an email to sage-devel@googlegroups.com
To unsubscribe from this group, send an email to
sage-devel+unsubscr...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/sage-devel
URL: http://www.sagemath.org
Re: [sage-devel] Re: Add 'gcc' libraries to Sage binaries ( 0.5% bloat)
Bill Hart wrote: On 22 Feb, 11:27, Dr. David Kirkby david.kir...@onetel.net wrote: This came up on the thread mercurial on t2 but I thought I'd start a new thread on it. I'd propose that we include in any binary distribution gcc's C, C++ and Fortran shared libraries. They would be placed in $SAGE_LOCAL/lib. Then we can ensure that people will run Sage with what libraries Sage was built with, rather than what versions they may or may not have lying around. Comments? Dave Are we sure this would work? Won't those libraries depend on what kernel is installed, etc, etc? I'm completely ignorant on this, so may be talk out my proverbial. Bill. I would not have thought so, though I don't claim to be an expert on this. I just checked the sage-4.3.2-linux-64bit-ubuntu_9.10-x86_64-Linux binary, and find the fortran library libgfortran.so is already there, as otherwise one would need to have gfortran installed to run Sage. Since that is not causing any particular kernel issues, I doubt the C or C++ libraries would either. (As always, there can always be unwelcome surprises.) The file COPYING.LIB in the gcc source distribution specifically allows copying the libraries for the purposes of running code on other systems, without including the source code for your proprietary program. I don't think the gcc developers would bother if the version of the library depended on the kernel version in use on the other system. Of course, we already have to produce different binaries for different Linux distributions. So I'd suggest just copying the gcc libraries from whatever system one built Sage on. The libraries I put in the Solaris binary were copied from a Sun running the very first Solaris 10 release (03/2005). The kernel on both t2 and my other Solaris box are both late 2009. So even with 4.5 years of kernel updates, the gcc libraries still work. (I know Linux tends to have poorer backward compatibility than Solaris, but I doubt that would be an issue even on Linux). Dave -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
Re: [sage-devel] Re: Add 'gcc' libraries to Sage binaries ( 0.5% bloat)
Volker Braun wrote:
If you want to go that route you probably want to include glibc
(contains standard math library) as well.
While a viable possibility,
there are two obvious arguments against it:
On Feb 22, 11:27 am, Dr. David Kirkby david.kir...@onetel.net
wrote:
* Always have the libraries.
Instead, check on first startup that all required packages are
installed and print an error message instead ('Please yum install
libgfortran').
I assume one would need root to do that, but perhaps not.
In any case, we already include the fortran library. The question is whether to
extend that to C and C++ too.
* Always have the exact same versions Sage was built with.
Downside: if your distribution comes out with a bugfix you don't
automatically profit from it.
True, though the Sage developers tend to build the binaries with late gcc
versions. In which case bug fixes in the libraries are more likely to be
included in the Sage binary than they are on the users system unless that user
has kept more up to date with gcc than the Sage developers.
Problems are more likely to occur if the user has an older buggy gcc, but the
Sage binary was built and tested using a newer gcc.
Also, if your distribution keeps
breaking the C++ ABI then maybe its time to switch.
Well, it was annoying that simply updating gcc to 4.4.3 from a slightly earlier
release caused problems. But including the libraries should avoid that.
At least with the libraries included, the user gets to use Sage with libraries
where Sage has been tested.
I know Mathematica on Solaris ships with libraries that are included on Solaris.
I assume Wolfram Research feel happier knowing that Mathematica runs with
libraries which it was tested on, rather than what the end user may have.
Dave
--
To post to this group, send an email to sage-devel@googlegroups.com
To unsubscribe from this group, send an email to
sage-devel+unsubscr...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/sage-devel
URL: http://www.sagemath.org
[sage-devel] Re: Add 'gcc' libraries to Sage binaries ( 0.5% bloat)
On Feb 22, 1:49 pm, Willem Jan Palenstijn w...@usecode.org wrote: This text describes RedHat's policy on libgcc_s and libstdc++:http://kbase.redhat.com/faq/docs/DOC-8313 It seems to suggest that if we want to include libstdc++ or libgcc_s, we should include both. Well it literally says: Both libgcc and libstdc++ are core libraries, applications should never bundle their own versions of these libraries. Volker -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
[sage-devel] Re: Add 'gcc' libraries to Sage binaries ( 0.5% bloat)
On 22 Feb., 12:27, Dr. David Kirkby david.kir...@onetel.net wrote: This came up on the thread mercurial on t2 but I thought I'd start a new thread on it. I'd propose that we include in any binary distribution gcc's C, C++ and Fortran shared libraries. They would be placed in $SAGE_LOCAL/lib. Then we can ensure that people will run Sage with what libraries Sage was built with, rather than what versions they may or may not have lying around. The amount of bloat this would add to the binary would be very small. For Solaris, the compressed sizes of the files are: -rwxr-xr-x 1 drkirkby staff 1.5M Feb 22 10:10 libstdc++.so.6.0.10.gz -rwxr-xr-x 1 drkirkby staff 717K Feb 22 10:10 libgfortran.so.3.0.0.gz -rw-r--r-- 1 drkirkby staff 80K Feb 22 10:10 libgcc_s.so.1.gz So adding all 3 adds 2.3 MB of extra code to the binary. But given the binary is 500 MB (not untypical), that is less than 0.5% of bloat. By doing this, we ensure that people * Always have the libraries. * Always have the exact same versions Sage was built with. I believe the Fortran library might already be included for Linux (I have not checked), but I'd suggest all 3 were added to binaries. The C library is the one people most likely will have, but given it is by far the smallest, we might as well include it to be 100% sure. Comments? Dave Hi Dave, this only is reasonable, if for building Sage we do not use the (or a) system's default compiler. So on GNU/Linux, or Mac OS X, this is almost a non-issue --- it's clear what this default compiler is, and Sage uses it. It's far less clear on Solaris, or e.g. on Windows. If some future Sage is built with some Microsoft Visual C++ compiler, we will have to tell the casual user exactly which additional msvc libraries (for msvc 2008, or 2010, or ...) need to be installed on a virgin Windows in order for Sage to be able to run. (These additional libraries are free for download, but not free to distribute by others than Microsoft.) But then, these libraries will be installed in some system location. Back to the Solaris case --- as long as we do not have some GCC spkg, eventually optional, that installs the full GCC, I'm opposed to install its compiler specific libraries under $SAGE_ROOT. Instead, Sage should rely on them to be found in the standard system locations --- or in the ones the user explicitly communicates to Sage via environment variables. Of course it would make sense to provide for Solaris some minimal GCC library install specifically adapted for the need of potential Sage users. (Just like in the case of Microsoft not urging users to install the full msvc, but providing some library-only package.) But IMHO, this is outside the scope of our current model of fully relocatable yet open (Sage) installs, so that would need to be offered somewhat in parallel. (Otherwise, Sage would need to be yet far more paranoid about taking in which dynamic libraries and how than today, see also Robert's remark.) Cheers, Georg -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
[sage-devel] Re: Add 'gcc' libraries to Sage binaries ( 0.5% bloat)
-1 from me to including these libraries. Q1. Are there any other well-known packages which do this? If not, it is not a standard thing to do, probably for good reason. Cython uses the C compiler (if I understand correctly). I think this kills the idea dead. Q2. Would building Sage with the Sun CC compiler remove the need to have standard GNU libraries accessible somewhere? If so, how hard would it be to have Sage build with the Sun CC? Bill. On 22 Feb, 20:44, Georg S. Weber georgswe...@googlemail.com wrote: On 22 Feb., 12:27, Dr. David Kirkby david.kir...@onetel.net wrote: This came up on the thread mercurial on t2 but I thought I'd start a new thread on it. I'd propose that we include in any binary distribution gcc's C, C++ and Fortran shared libraries. They would be placed in $SAGE_LOCAL/lib. Then we can ensure that people will run Sage with what libraries Sage was built with, rather than what versions they may or may not have lying around. The amount of bloat this would add to the binary would be very small. For Solaris, the compressed sizes of the files are: -rwxr-xr-x 1 drkirkby staff 1.5M Feb 22 10:10 libstdc++.so.6.0.10.gz -rwxr-xr-x 1 drkirkby staff 717K Feb 22 10:10 libgfortran.so.3.0.0.gz -rw-r--r-- 1 drkirkby staff 80K Feb 22 10:10 libgcc_s.so.1.gz So adding all 3 adds 2.3 MB of extra code to the binary. But given the binary is 500 MB (not untypical), that is less than 0.5% of bloat. By doing this, we ensure that people * Always have the libraries. * Always have the exact same versions Sage was built with. I believe the Fortran library might already be included for Linux (I have not checked), but I'd suggest all 3 were added to binaries. The C library is the one people most likely will have, but given it is by far the smallest, we might as well include it to be 100% sure. Comments? Dave Hi Dave, this only is reasonable, if for building Sage we do not use the (or a) system's default compiler. So on GNU/Linux, or Mac OS X, this is almost a non-issue --- it's clear what this default compiler is, and Sage uses it. It's far less clear on Solaris, or e.g. on Windows. If some future Sage is built with some Microsoft Visual C++ compiler, we will have to tell the casual user exactly which additional msvc libraries (for msvc 2008, or 2010, or ...) need to be installed on a virgin Windows in order for Sage to be able to run. (These additional libraries are free for download, but not free to distribute by others than Microsoft.) But then, these libraries will be installed in some system location. Back to the Solaris case --- as long as we do not have some GCC spkg, eventually optional, that installs the full GCC, I'm opposed to install its compiler specific libraries under $SAGE_ROOT. Instead, Sage should rely on them to be found in the standard system locations --- or in the ones the user explicitly communicates to Sage via environment variables. Of course it would make sense to provide for Solaris some minimal GCC library install specifically adapted for the need of potential Sage users. (Just like in the case of Microsoft not urging users to install the full msvc, but providing some library-only package.) But IMHO, this is outside the scope of our current model of fully relocatable yet open (Sage) installs, so that would need to be offered somewhat in parallel. (Otherwise, Sage would need to be yet far more paranoid about taking in which dynamic libraries and how than today, see also Robert's remark.) Cheers, Georg -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
[sage-devel] Re: Add 'gcc' libraries to Sage binaries ( 0.5% bloat)
Actually, this is really silly. The must be a multitude of packages for Solaris which are distributed with binaries and which need these libraries. How do they deal with this issue? Is there a package installation system like apt-get or yum or rpm on Solaris? They must surely deal with this issue somehow? Bill. On 22 Feb, 21:01, Bill Hart goodwillh...@googlemail.com wrote: -1 from me to including these libraries. Q1. Are there any other well-known packages which do this? If not, it is not a standard thing to do, probably for good reason. Cython uses the C compiler (if I understand correctly). I think this kills the idea dead. Q2. Would building Sage with the Sun CC compiler remove the need to have standard GNU libraries accessible somewhere? If so, how hard would it be to have Sage build with the Sun CC? Bill. On 22 Feb, 20:44, Georg S. Weber georgswe...@googlemail.com wrote: On 22 Feb., 12:27, Dr. David Kirkby david.kir...@onetel.net wrote: This came up on the thread mercurial on t2 but I thought I'd start a new thread on it. I'd propose that we include in any binary distribution gcc's C, C++ and Fortran shared libraries. They would be placed in $SAGE_LOCAL/lib. Then we can ensure that people will run Sage with what libraries Sage was built with, rather than what versions they may or may not have lying around. The amount of bloat this would add to the binary would be very small. For Solaris, the compressed sizes of the files are: -rwxr-xr-x 1 drkirkby staff 1.5M Feb 22 10:10 libstdc++.so.6.0.10.gz -rwxr-xr-x 1 drkirkby staff 717K Feb 22 10:10 libgfortran.so.3.0.0.gz -rw-r--r-- 1 drkirkby staff 80K Feb 22 10:10 libgcc_s.so.1.gz So adding all 3 adds 2.3 MB of extra code to the binary. But given the binary is 500 MB (not untypical), that is less than 0.5% of bloat. By doing this, we ensure that people * Always have the libraries. * Always have the exact same versions Sage was built with. I believe the Fortran library might already be included for Linux (I have not checked), but I'd suggest all 3 were added to binaries. The C library is the one people most likely will have, but given it is by far the smallest, we might as well include it to be 100% sure. Comments? Dave Hi Dave, this only is reasonable, if for building Sage we do not use the (or a) system's default compiler. So on GNU/Linux, or Mac OS X, this is almost a non-issue --- it's clear what this default compiler is, and Sage uses it. It's far less clear on Solaris, or e.g. on Windows. If some future Sage is built with some Microsoft Visual C++ compiler, we will have to tell the casual user exactly which additional msvc libraries (for msvc 2008, or 2010, or ...) need to be installed on a virgin Windows in order for Sage to be able to run. (These additional libraries are free for download, but not free to distribute by others than Microsoft.) But then, these libraries will be installed in some system location. Back to the Solaris case --- as long as we do not have some GCC spkg, eventually optional, that installs the full GCC, I'm opposed to install its compiler specific libraries under $SAGE_ROOT. Instead, Sage should rely on them to be found in the standard system locations --- or in the ones the user explicitly communicates to Sage via environment variables. Of course it would make sense to provide for Solaris some minimal GCC library install specifically adapted for the need of potential Sage users. (Just like in the case of Microsoft not urging users to install the full msvc, but providing some library-only package.) But IMHO, this is outside the scope of our current model of fully relocatable yet open (Sage) installs, so that would need to be offered somewhat in parallel. (Otherwise, Sage would need to be yet far more paranoid about taking in which dynamic libraries and how than today, see also Robert's remark.) Cheers, Georg -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
Re: [sage-devel] Re: Add 'gcc' libraries to Sage binaries ( 0.5% bloat)
Bill Hart wrote: Actually, this is really silly. The must be a multitude of packages for Solaris which are distributed with binaries and which need these libraries. How do they deal with this issue? Is there a package installation system like apt-get or yum or rpm on Solaris? They must surely deal with this issue somehow? Bill. Yes - it called 'pkgadd' Blastwave is a site where you pick a package you want (e.g. python) and it downloads that along with any dependences. Here are those for phthon http://www.blastwave.org/jir/pkgcontents.ftd?software=pythonstyle=briefstate=5arch=i386 One of which is GNU Compiler Collection Version 4.3.3 Core RunTime which is basically the libraries. Sunfreeware (run by a consultant for Wolfram Reserarch) http://www.sunfreeware.com/ has loads of packages in binary format, though the site has stagnated. Let's pick the first package (a2ps-4.14), a2ps-4.14 Having libgcc_s.so.1 in /usr/local/lib is required. This can be done by installing libgcc-3.4.6 or gcc-3.4.6 or higher. So it basically boils down to one of two choices. 1) Install gcc. 2) Install just the gcc libraries. When I created the binary which runs on 't2nb' (which is a zone on Solaris) http://t2nb.math.washington.edu:8000 I included the libraries. There is no gcc on that Solaris zone, so Sage would not run without them. In contrast, if you have gcc installed, and the libraries can be found via LD_LIBRARY_PATH or some other means, then those libraries are not needed. I personally think it is a bit safer to include them, but I can see counter arguments. The most convincing one is from Robert Bradsure who said: --- Are there issues if the gcc and core C libraries don't match? For example, If a user uses cython in the notebook, will it pick up the users g++ and Sage's libstdc++? What about someone who starts with a binary then starts developing with it or installs optional packages? -- The truth is, I don't know the answer to that one. It's clear the current system does cause some problems (not just on Solaris), but I must admit Robert's argument may be valid. Dave -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
