Re: [sage-devel] Re: GSoC project: make the Sage build system more distribution friendly
hi there. On Fri, Apr 12, 2013 at 05:13:14PM -0600, David Roe wrote: But speaking as someone who contributes a lot to the Sage library itself, I'm strongly in support of Robert Bradshaw's proposal that we combine the different Sage repositories (root, devel, script and ext). It's much easier to work on issues that involve code in these different repositories once you only have to make a commit to a single unified repo. i'm not a sage developer, but i get your point (apart from root - which doesn't contain a single line of code). missing useful git support for submodules is the force appearently. I'm not sure whether the spkgs you refer to correspond to the four repositories I mentioned above, but if so then the merging is intended to be permanent. in turn, i cannot associate any of the four repositories above to the (former) spkg sage.spkg. additionally http://hg.sagemath.org/sage-root is dead, but iirc it contains the toplevel Makefile and README. i think, we are talking about the same merge... To the git transition team: Have you considered using git submodules for the different Sage components? It would certainly solve this issue. Would it also have other advantages? Is there a reason not to do it? Exactly what do you mean by different Sage components? I do know that there was some discussion of git submodules, but it was discarded for technical reasons that I don't recall. components are all non-packaging/distribution related parts. like sage-scripts, sage-notebook, extcode, sage-tex the python modules and the c library. these were (before the git-transition) more or less cleanly split from each other. guess why (who did this?). also, all packages had a unified interface (the spkg-install script), and were treated alike by the top-level-stuff (okay, they should have been -- i don't know for sure). regards felix PS: yes i could think of a roadmap do deal with most of this. but its more work and less distribution-friendly. lets burn all bridges first. -- You received this message because you are subscribed to the Google Groups sage-devel group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [sage-devel] Re: GSoC project: make the Sage build system more distribution friendly
Hi there. On Thu, Apr 11, 2013 at 11:48:17PM +0200, Tobias Hansen wrote: so theres the inevitable question to ask: would it be an option to eventually split c_lib and the python modules to different packages? If they are each in a selfcontained folder it's not too much hassle to repack them from the one tarball. (Ok, still some hassle.) However, I don't see why they should not be in one source package. Because linking to c_lib has to be done differently when it is not installed on the system when the package is built? An important implication of having stuff together in a source package is usually that they have to be updated together. That is the case for the parts of Sage. By the way, does c_lib have a stable ABI so that it is reasonable to have it as a public shared library? Would that be useful? i don't see the c_lib/python sage lib as critical as the gentoo people do. but: if some distribution -- for whatever reason -- requires/fancies seperate packages, seperate packages would make sage more distribution friendly. (no actual problem here). what frightens me is that the spkgs for the core parts have vanished in https://github.com/sagemath/sage.git (the git-transition). if this is serious (and not a temporary kludge), we really would have to repack tarballs. this (imo) is not just 'some hassle': try to cherry-pick a patch from current sage-python that makes the release candidate work with, say singular-(sage-singular-version+0.0.1). this scenario is realistic, as it doesnt seem possible to always ship exact dependencies within debian/gentoo/whatever. now try to cherry-pick that patch after repacking and loosing history. im not saying it's not possible. but it would be seriously distribution-unfriendly. It would still be nice if the top level script could be used by distributions. [...] i'm not convinced. once all parts (including python-sage and c_lib) are in a distributable/configurable shape, any distribution will be able to pick them up easily. especially there will be no need for distributions to use sage's built in top-level script. whatever a top level script does, it will never fit the needs of all distributions at once. just think about building a multiarch ready package out of c_lib, while it is only accessible within a tarball containing the sources of five other packages, through a patchwork of a sage-toplevel script and an spkg-install script calling scons install through a static makefile (or setup.py or whatever). Sounds reasonable. Would you take care that the Sage distribution also build python-sage, c_lib, etc using the new configurable process? that would be the primary objective. treat python-sage, c_lib, sage-scripts, extcode, whatnoteverelse as *packages* just like all the other packages. now understand the sage-toplevel repository as a sage distribution (distributing spkgs or successors). sage will be distribution-friendly as soon as all packages have sane build systems. regards felix -- You received this message because you are subscribed to the Google Groups sage-devel group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [sage-devel] Re: GSoC project: make the Sage build system more distribution friendly
Am 12.04.2013 10:25, schrieb Felix Salfelder: Hi there. On Thu, Apr 11, 2013 at 11:48:17PM +0200, Tobias Hansen wrote: so theres the inevitable question to ask: would it be an option to eventually split c_lib and the python modules to different packages? If they are each in a selfcontained folder it's not too much hassle to repack them from the one tarball. (Ok, still some hassle.) However, I don't see why they should not be in one source package. Because linking to c_lib has to be done differently when it is not installed on the system when the package is built? An important implication of having stuff together in a source package is usually that they have to be updated together. That is the case for the parts of Sage. By the way, does c_lib have a stable ABI so that it is reasonable to have it as a public shared library? Would that be useful? i don't see the c_lib/python sage lib as critical as the gentoo people do. but: if some distribution -- for whatever reason -- requires/fancies seperate packages, seperate packages would make sage more distribution friendly. (no actual problem here). what frightens me is that the spkgs for the core parts have vanished in https://github.com/sagemath/sage.git (the git-transition). if this is serious (and not a temporary kludge), we really would have to repack tarballs. Only if we want multiple Sage Debian source packages. this (imo) is not just 'some hassle': try to cherry-pick a patch from current sage-python that makes the release candidate work with, say singular-(sage-singular-version+0.0.1). this scenario is realistic, as it doesnt seem possible to always ship exact dependencies within debian/gentoo/whatever. now try to cherry-pick that patch after repacking and loosing history. im not saying it's not possible. but it would be seriously distribution-unfriendly. To the git transition team: Have you considered using git submodules for the different Sage components? It would certainly solve this issue. Would it also have other advantages? Is there a reason not to do it? Cheers, Tobias -- You received this message because you are subscribed to the Google Groups sage-devel group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [sage-devel] Re: GSoC project: make the Sage build system more distribution friendly
On 12/04/13 20:59, Tobias Hansen wrote: Am 12.04.2013 10:25, schrieb Felix Salfelder: Hi there. On Thu, Apr 11, 2013 at 11:48:17PM +0200, Tobias Hansen wrote: so theres the inevitable question to ask: would it be an option to eventually split c_lib and the python modules to different packages? If they are each in a selfcontained folder it's not too much hassle to repack them from the one tarball. (Ok, still some hassle.) However, I don't see why they should not be in one source package. Because linking to c_lib has to be done differently when it is not installed on the system when the package is built? An important implication of having stuff together in a source package is usually that they have to be updated together. That is the case for the parts of Sage. By the way, does c_lib have a stable ABI so that it is reasonable to have it as a public shared library? Would that be useful? i don't see the c_lib/python sage lib as critical as the gentoo people do. but: if some distribution -- for whatever reason -- requires/fancies seperate packages, seperate packages would make sage more distribution friendly. (no actual problem here). what frightens me is that the spkgs for the core parts have vanished in https://github.com/sagemath/sage.git (the git-transition). if this is serious (and not a temporary kludge), we really would have to repack tarballs. Only if we want multiple Sage Debian source packages. this (imo) is not just 'some hassle': try to cherry-pick a patch from current sage-python that makes the release candidate work with, say singular-(sage-singular-version+0.0.1). this scenario is realistic, as it doesnt seem possible to always ship exact dependencies within debian/gentoo/whatever. now try to cherry-pick that patch after repacking and loosing history. im not saying it's not possible. but it would be seriously distribution-unfriendly. To the git transition team: Have you considered using git submodules for the different Sage components? It would certainly solve this issue. Would it also have other advantages? Is there a reason not to do it? I'd like to note that on gentoo we only use things like sage-singular-xxx as an exception - singular is a particularly good example, since it was the last one we had years ago. Compared to debian we have a little bit more freedom on the versions a user can install. My own experience over the years is that singular and pari are possibly the most problematic one followed by polybori (but it is improving dramatically in that regard in the latest release), networkx and lately ppl. The biggest reason of doing sage-foo for everything is to use mpir instead of gmp. sage-on-gentoo is using gmp as switching to mpir distro wide would require agreement from the toolchain people. There are way of giving people the choice but that involves changing every package currently using gmp Francois -- You received this message because you are subscribed to the Google Groups sage-devel group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [sage-devel] Re: GSoC project: make the Sage build system more distribution friendly
On 04/11/2013 11:48 PM, Tobias Hansen wrote: If they are each in a selfcontained folder it's not too much hassle to repack them from the one tarball. (Ok, still some hassle.) However, I don't see why they should not be in one source package. Because linking to c_lib has to be done differently when it is not installed on the system when the package is built? An important implication of having stuff together in a source package is usually that they have to be updated together. That is the case for the parts of Sage. I agree. I see no reason at all for having separate packages for c_lib and the sage library for example. Perhaps Gentoo has its reasons, but normally I expect this to remain one package. -- You received this message because you are subscribed to the Google Groups sage-devel group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [sage-devel] Re: GSoC project: make the Sage build system more distribution friendly
On Fri, Apr 12, 2013 at 09:23:45PM +1200, Francois Bissey wrote: I'd like to note that on gentoo we only use things like sage-singular-xxx as an exception - singular is a particularly good example, since it was the last one we had years ago. Compared to debian we have a little bit more freedom on the versions a user can install. My own experience over the years is that singular and pari are possibly the most problematic one followed by polybori (but it is improving dramatically in that regard in the latest release), networkx and lately ppl. but i guess, you would switch to unprefixed versions, if they worked as well. The biggest reason of doing sage-foo for everything is to use mpir instead of gmp. sage-on-gentoo is using gmp as switching to mpir distro wide would require agreement from the toolchain people. There are way of giving people the choice but that involves changing every package currently using gmp in debian one would probably postfix libraries linked against mpir instead of gmp with -mpir, like for example libmpfr-mpir, libmpfi-mpir or libpari-mpir. this is not difficult, does not require a distro-wide switch and is only indirectly related to sage. providing mpir versions is something we/debian should do after sage is packaged and working (and there is an indication to do so). regards felix -- You received this message because you are subscribed to the Google Groups sage-devel group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [sage-devel] Re: GSoC project: make the Sage build system more distribution friendly
On 13/04/2013, at 4:24, Felix Salfelder fe...@salfelder.org wrote: On Fri, Apr 12, 2013 at 09:23:45PM +1200, Francois Bissey wrote: I'd like to note that on gentoo we only use things like sage-singular-xxx as an exception - singular is a particularly good example, since it was the last one we had years ago. Compared to debian we have a little bit more freedom on the versions a user can install. My own experience over the years is that singular and pari are possibly the most problematic one followed by polybori (but it is improving dramatically in that regard in the latest release), networkx and lately ppl. but i guess, you would switch to unprefixed versions, if they worked as well. Yes and we currently do not use prefixed version singular was the last one several year ago. Francois -- You received this message because you are subscribed to the Google Groups sage-devel group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [sage-devel] Re: GSoC project: make the Sage build system more distribution friendly
First a disclaimer: I was at Sage Days 47 (working on the transition to git), but I'm not a git expert and I don't know the technical justification for some of the choices made (I think the best people to speak up in that direction would be Andrew Ohana, Keshav Kini, Robert Bradshaw, Timo Kluck or Julian Ruth). I also use OS X so I personally have no interest in working on packaging efforts, though I'm grateful to those of you who do, both for the increased exposure of Sage in the Linux world and for the progress toward the goal of being able to checkout different versions of Sage with different spkgs installed and have them build and work. But speaking as someone who contributes a lot to the Sage library itself, I'm strongly in support of Robert Bradshaw's proposal that we combine the different Sage repositories (root, devel, script and ext). It's much easier to work on issues that involve code in these different repositories once you only have to make a commit to a single unified repo. On Fri, Apr 12, 2013 at 2:59 AM, Tobias Hansen than...@debian.org wrote: Am 12.04.2013 10:25, schrieb Felix Salfelder: Hi there. On Thu, Apr 11, 2013 at 11:48:17PM +0200, Tobias Hansen wrote: so theres the inevitable question to ask: would it be an option to eventually split c_lib and the python modules to different packages? If they are each in a selfcontained folder it's not too much hassle to repack them from the one tarball. (Ok, still some hassle.) However, I don't see why they should not be in one source package. Because linking to c_lib has to be done differently when it is not installed on the system when the package is built? An important implication of having stuff together in a source package is usually that they have to be updated together. That is the case for the parts of Sage. By the way, does c_lib have a stable ABI so that it is reasonable to have it as a public shared library? Would that be useful? i don't see the c_lib/python sage lib as critical as the gentoo people do. but: if some distribution -- for whatever reason -- requires/fancies seperate packages, seperate packages would make sage more distribution friendly. (no actual problem here). what frightens me is that the spkgs for the core parts have vanished in https://github.com/sagemath/sage.git (the git-transition). if this is serious (and not a temporary kludge), we really would have to repack tarballs. I'm not sure whether the spkgs you refer to correspond to the four repositories I mentioned above, but if so then the merging is intended to be permanent. this (imo) is not just 'some hassle': try to cherry-pick a patch from current sage-python that makes the release candidate work with, say singular-(sage-singular-version+0.0.1). this scenario is realistic, as it doesnt seem possible to always ship exact dependencies within debian/gentoo/whatever. now try to cherry-pick that patch after repacking and loosing history. im not saying it's not possible. but it would be seriously distribution-unfriendly. To the git transition team: Have you considered using git submodules for the different Sage components? It would certainly solve this issue. Would it also have other advantages? Is there a reason not to do it? Exactly what do you mean by different Sage components? I do know that there was some discussion of git submodules, but it was discarded for technical reasons that I don't recall. David -- You received this message because you are subscribed to the Google Groups sage-devel group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [sage-devel] Re: GSoC project: make the Sage build system more distribution friendly
On 04/11/2013 01:53 AM, R. Andrew Ohana wrote: For those so inclined to browse the current status of the git transition, the repository is at https://github.com/sagemath/sage. Yes, it is completely obvious that this Debian packaging effort should start from the GIT repository, not from the current way Sage is distributed. Concerning a top-level configure script: having such a script is something I'm already planning anyway. So, yes, we should absolutely have a top-level configure script. Almost all of the stuff currently in spkg/base/prereq* and spkg/install should move there, and then these files should be removed. -- You received this message because you are subscribed to the Google Groups sage-devel group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [sage-devel] Re: GSoC project: make the Sage build system more distribution friendly
Hi there. On Thu, Apr 11, 2013 at 01:04:23AM +0200, Tobias Hansen wrote: Remember the git transition. spkg's will go away. There will be just one tarball containing everything in the future. The sources of other projects will be better separated, so it will be easy to get a Sage tarball without sources of other projects. Taking this into account, it probably makes sense to have only one Sage Debian source package. i have looked at the git repo at https://github.com/sagemath/sage. as it seems, this repo contains the contents of (formerly?) hg.sagemath.org/sage-main (and some more stuff) within src, while the other spkgs have been replaced by a set of patches and rules (couldnt find references to the original tarballs?), sorted into build/pkgs. in particular the contents of sage.spkg have been relocated and merged with other stuff (ext, doc). i don't know why or how serious this is. so current plans are to merge everything together (instead of for example splitting up sage.spkg into c_lib and python-sage)? Switching to autotools is something we can't decide alone. What do Sage developers say? Do you mean with Sage (the library) all Sage components including notebook etc? sage the library is the contents of sage.spkg. i.e. a shared library (in c_lib) and a python module (in sage). (or *was* the contents of that spkg). It would still be nice if the top level script could be used by distributions. There are still several things to build and other things to do, if external dependencies are not built, and we should not implement all this in debian/rules. One could start with the option to build all or none of the dependencies and then maybe go further to allow more combinations. But I'm also not entirely sure if combining system and bundled dependencies is needed. Maybe an alternative build script for building with system libraries would be a better idea. Are there other opinions? the top-level stuff in a way *is* a distribution. using it to build debian packages makes things worse -- finally the purpose of debian/rules will be mostly dissecting the components. look at singular for a smaller example [1]. (i'm not saying, a top level configure script isnt useful for local/manual installation). regards felix [1] http://git.debian.org/?p=debian-science/packages/singular.git -- You received this message because you are subscribed to the Google Groups sage-devel group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [sage-devel] Re: GSoC project: make the Sage build system more distribution friendly
Also, a top-level configure script is the natural point to select alternatives to 3rd party code. For example, one should be able to ./configure --with-mpir=/usr --with-gap=/usr/local to use an external mpir and gap install... On Thursday, April 11, 2013 8:00:30 AM UTC+1, Jeroen Demeyer wrote: On 04/11/2013 01:53 AM, R. Andrew Ohana wrote: For those so inclined to browse the current status of the git transition, the repository is at https://github.com/sagemath/sage. Yes, it is completely obvious that this Debian packaging effort should start from the GIT repository, not from the current way Sage is distributed. Concerning a top-level configure script: having such a script is something I'm already planning anyway. So, yes, we should absolutely have a top-level configure script. Almost all of the stuff currently in spkg/base/prereq* and spkg/install should move there, and then these files should be removed. -- You received this message because you are subscribed to the Google Groups sage-devel group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [sage-devel] Re: GSoC project: make the Sage build system more distribution friendly
On Thursday, April 11, 2013 4:29:33 AM UTC-4, Volker Braun wrote: Also, a top-level configure script is the natural point to select alternatives to 3rd party code. For example, one should be able to ./configure --with-mpir=/usr --with-gap=/usr/local to use an external mpir and gap install... Would the top-level configure script also include optional/experimental packages? Best, Travis -- You received this message because you are subscribed to the Google Groups sage-devel group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [sage-devel] Re: GSoC project: make the Sage build system more distribution friendly
On 04/11/2013 01:36 PM, Travis Scrimshaw wrote: On Thursday, April 11, 2013 4:29:33 AM UTC-4, Volker Braun wrote: Also, a top-level configure script is the natural point to select alternatives to 3rd party code. For example, one should be able to ./configure --with-mpir=/usr --with-gap=/usr/local to use an external mpir and gap install... Would the top-level configure script also include optional/experimental packages? Maybe yes, but let's not consider that for the moment. -- You received this message because you are subscribed to the Google Groups sage-devel group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [sage-devel] Re: GSoC project: make the Sage build system more distribution friendly
Hi there. On Thu, Apr 11, 2013 at 11:41:17AM +1200, François Bissey wrote: From the point of view of a linux distribution, my opinion is that the package management system should be extracted. If it comes from your distro the packages and upgrades are handled by the distro mechanism, except for stuff that (can) live in the final user home directory. i do not really understand this. are you saying that the linux distribution should take over the role of the sage package mechanism? if so, would that mean that sage components should work independently of the sage packaging method? Sage itself has currently several components that are shipped separately and at least one that should be split: sage_root: which has all the elements of the basic build system and traditionally the scripts to start sage. sage_scripts: a collection of various scripts and command to run and tests sage. extcode: various bits and pieces accumulated over time. I understand it will disappear in the git migration being integrated elsewhere. sage: the python extension itself plus the c library. The c library is the element we think should be split (and we do in sage-on-gentoo). are you implying that its a good idea to split components and ship them as seperate packages? the sage-to-git transition apparently does the reverse. how does this affect the gentoo-packaging? Whether to keep this structure in Debian or after the git transition is not for me to answer. But I strongly believe the c library needs to be available separately from the python library. in debian, one source package can create multiple binary packages. this for example makes sense, when seperate (but related) lib*, *-dev, *-doc, *-dbg packages are convenient. packing unrelated stuff from a single source repo do different binary packages usually leads to overhead within the rules (which will probably not even work for the next release). so theres the inevitable question to ask: would it be an option to eventually split c_lib and the python modules to different packages? The c library is built with scons which has its detractors (that includes me) but is seriously too small to justify autotooling in my opinion. all i (need to?) know about scons is, that there is no visible concept of configure. particularly there is no way to pass --with-this-and-that=/my/favourite/path switches in a practical way. so if the c_lib is small, that would make transition to something else even fast. It would still be nice if the top level script could be used by distributions. There are still several things to build and other things to do, if external dependencies are not built, and we should not implement all this in debian/rules. One could start with the option to build all or none of the dependencies and then maybe go further to allow more combinations. But I'm also not entirely sure if combining system and bundled dependencies is needed. Maybe an alternative build script for building with system libraries would be a better idea. Are there other opinions? The questions has arisen several time in the past 5 years but in spite of some suggestions on how to achieve this, no one has done the work. You are welcome to have a go at it. If you start it you may get a surprising number of helpers. I think most of the inertia is in starting it. i'm not convinced. once all parts (including python-sage and c_lib) are in a distributable/configurable shape, any distribution will be able to pick them up easily. especially there will be no need for distributions to use sage's built in top-level script. whatever a top level script does, it will never fit the needs of all distributions at once. just think about building a multiarch ready package out of c_lib, while it is only accessible within a tarball containing the sources of five other packages, through a patchwork of a sage-toplevel script and an spkg-install script calling scons install through a static makefile (or setup.py or whatever). have fun felix -- You received this message because you are subscribed to the Google Groups sage-devel group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [sage-devel] Re: GSoC project: make the Sage build system more distribution friendly
On Thu, 11 Apr 2013 22:27:48 Felix Salfelder wrote: Hi there. On Thu, Apr 11, 2013 at 11:41:17AM +1200, François Bissey wrote: From the point of view of a linux distribution, my opinion is that the package management system should be extracted. If it comes from your distro the packages and upgrades are handled by the distro mechanism, except for stuff that (can) live in the final user home directory. i do not really understand this. are you saying that the linux distribution should take over the role of the sage package mechanism? if so, would that mean that sage components should work independently of the sage packaging method? Yes. Do you as a distro packager want one of your applications take over the management of some of your system components? Never mind that it would probably need privilege do to so. Sage package mechanism include upgrade capability. Do you want a sage debian package just upgrading itself by compiling bits all over the place independently of dpkg? Never mind that would probably mess upgrade and fixes from dpkg. Sage itself has currently several components that are shipped separately and at least one that should be split: sage_root: which has all the elements of the basic build system and traditionally the scripts to start sage. sage_scripts: a collection of various scripts and command to run and tests sage. extcode: various bits and pieces accumulated over time. I understand it will disappear in the git migration being integrated elsewhere. sage: the python extension itself plus the c library. The c library is the element we think should be split (and we do in sage-on-gentoo). are you implying that its a good idea to split components and ship them as seperate packages? the sage-to-git transition apparently does the reverse. how does this affect the gentoo-packaging? We'll change the way we package things. The current packaging in Gentoo reflects the packaging upstream to a great degree apart from the separation of c_lib. This separation is motivated by the fact that c_lib and the python library have two different build systems and it is hard to combine them in a single package from the Gentoo point of view. Another point for me is that c_lib is a rather slow moving thing not updated very often. The other packages are just plain files to install. Separating c_lib in its own package with its own version numbering may enable me to provide several version of sage on the system at the same time. That last bit is very much more of a wish than something concrete but this separation would make things easier. There are a lot of other obstacles. Whether to keep this structure in Debian or after the git transition is not for me to answer. But I strongly believe the c library needs to be available separately from the python library. in debian, one source package can create multiple binary packages. this for example makes sense, when seperate (but related) lib*, *-dev, *-doc, *-dbg packages are convenient. packing unrelated stuff from a single source repo do different binary packages usually leads to overhead within the rules (which will probably not even work for the next release). No arguing there. Of course I am not doing a binary distro :) so theres the inevitable question to ask: would it be an option to eventually split c_lib and the python modules to different packages? The python modules need c_lib but the reverse is untrue. It is one of these things where either can be done depending on your agenda. The c library is built with scons which has its detractors (that includes me) but is seriously too small to justify autotooling in my opinion. all i (need to?) know about scons is, that there is no visible concept of configure. particularly there is no way to pass --with-this-and-that=/my/favourite/path switches in a practical way. so if the c_lib is small, that would make transition to something else even fast. It is small. I often thought of just building it with a simple Makefile rather than scons. It would still be nice if the top level script could be used by distributions. There are still several things to build and other things to do, if external dependencies are not built, and we should not implement all this in debian/rules. One could start with the option to build all or none of the dependencies and then maybe go further to allow more combinations. But I'm also not entirely sure if combining system and bundled dependencies is needed. Maybe an alternative build script for building with system libraries would be a better idea. Are there other opinions? The questions has arisen several time in the past 5 years but in spite of some suggestions on how to achieve this, no one has done the work. You are welcome to have a go at it. If you start it you may get a surprising number of helpers. I think most of the inertia is in starting it. i'm
Re: [sage-devel] Re: GSoC project: make the Sage build system more distribution friendly
Let's drop debian-science from CC. For those interested, the thread on sage-devel is at https://groups.google.com/forum/?fromgroups=#!topic/sage-devel/1HGbf4EZGb0 Am 11.04.2013 22:27, schrieb Felix Salfelder: in debian, one source package can create multiple binary packages. this for example makes sense, when seperate (but related) lib*, *-dev, *-doc, *-dbg packages are convenient. packing unrelated stuff from a single source repo do different binary packages usually leads to overhead within the rules (which will probably not even work for the next release). so theres the inevitable question to ask: would it be an option to eventually split c_lib and the python modules to different packages? If they are each in a selfcontained folder it's not too much hassle to repack them from the one tarball. (Ok, still some hassle.) However, I don't see why they should not be in one source package. Because linking to c_lib has to be done differently when it is not installed on the system when the package is built? An important implication of having stuff together in a source package is usually that they have to be updated together. That is the case for the parts of Sage. By the way, does c_lib have a stable ABI so that it is reasonable to have it as a public shared library? Would that be useful? It would still be nice if the top level script could be used by distributions. [...] i'm not convinced. once all parts (including python-sage and c_lib) are in a distributable/configurable shape, any distribution will be able to pick them up easily. especially there will be no need for distributions to use sage's built in top-level script. whatever a top level script does, it will never fit the needs of all distributions at once. just think about building a multiarch ready package out of c_lib, while it is only accessible within a tarball containing the sources of five other packages, through a patchwork of a sage-toplevel script and an spkg-install script calling scons install through a static makefile (or setup.py or whatever). Sounds reasonable. Would you take care that the Sage distribution also build python-sage, c_lib, etc using the new configurable process? Cheers, Tobias -- You received this message because you are subscribed to the Google Groups sage-devel group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [sage-devel] Re: GSoC project: make the Sage build system more distribution friendly
Hi, I'll give some answers from my point of view as one of the lead of sage-on- gentoo https://github.com/cschwan/sage-on-gentoo. Some of the problems discussed we encountered and have an opinion on. Whether some of the things we do or have done are applicable to Debian is best answered by Debian people themselves. On Thu, 11 Apr 2013 01:04:23 Tobias Hansen wrote: Am 10.04.2013 17:43, schrieb Felix Salfelder: several parts of fixing/changing/improving the sage (the system) build system is unrelated to distributing sage the library. so changes to the build system in Sage is somewhat vague. i think a build system for the sage library should at least check availibility (especially optional dependencies). more checks for versions/header functionality/existing symbols are implemented within autotools. the universal way would be maintaining these checks (which is unrealistic). catching common caveats should be possible. I mostly thought about the build system of the Sage system until now and didn't think about the build process of the library. Since availability of dependencies is ensured by the Sage system, the build process of the Sage library doesn't check for dependencies, right? Your idea is to add dependency checks to the library build, and that may be a good idea, but one could also consider to let the top level build scripts keep the job of handling the dependencies and do the checks there. I'm not saying that this would be better, just that this was my idea until now. Indeed sage is built from a top makefile and dependencies between the various spkg and sage itself are defined in spkg/standard/deps. This is used for building standard sage. For distribution purpose it is useful to separate these in two class, build time dependencies and pure run-time dependencies. For example scipy is not required at build time but will be required at run time. Sage also has some optional dependencies, some which are built time are usually checked in module_list.py (at the end). The sage package management system is invoked to check whether the package is installed. Run time optional dependencies are either checked through the sage package management system or presence of key files. From the point of view of a linux distribution, my opinion is that the package management system should be extracted. If it comes from your distro the packages and upgrades are handled by the distro mechanism, except for stuff that (can) live in the final user home directory. Every distribution has its own way to organize patches. I think Sage already has a good test coverage and a Debian package should of course run all the tests during build. iirc the tests are contained within sage.spkg (where the library is), so that should be possible. other tests may come within other spkgs. Remember the git transition. spkg's will go away. There will be just one tarball containing everything in the future. The sources of other projects will be better separated, so it will be easy to get a Sage tarball without sources of other projects. Taking this into account, it probably makes sense to have only one Sage Debian source package. Yes and no, see below. It seems to me that you understood that Sage should build all spkg's in any case and just install them somewhere else. I think that means the step you call fix Sage the operating system is not needed right? Now that I have clarified this, any new thoughts? for debian or other distributions, i dont see the need for any changes to sage (the system). making sagelib distributable doesn't help skipping spkgs. so there are still two subprojects: - improve sage (the library) distributability - enhance setup.py (better switch to autotools) - get libcsage.deb and python-sage.deb packages - run the unit tests from debian/rules - (also pack other stuff) - pave the way for local spkgs (e.g. installed to ~/.sage/local) - [..] Switching to autotools is something we can't decide alone. What do Sage developers say? Do you mean with Sage (the library) all Sage components including notebook etc? Sage itself has currently several components that are shipped separately and at least one that should be split: sage_root: which has all the elements of the basic build system and traditionally the scripts to start sage. sage_scripts: a collection of various scripts and command to run and tests sage. extcode: various bits and pieces accumulated over time. I understand it will disappear in the git migration being integrated elsewhere. sage: the python extension itself plus the c library. The c library is the element we think should be split (and we do in sage-on-gentoo). Whether to keep this structure in Debian or after the git transition is not for me to answer. But I strongly believe the c library needs to be available separately from the python library. The c library is built with scons which
Re: [sage-devel] Re: GSoC project: make the Sage build system more distribution friendly
On Wed, Apr 10, 2013 at 4:04 PM, Tobias Hansen than...@debian.org wrote: Am 10.04.2013 17:43, schrieb Felix Salfelder: several parts of fixing/changing/improving the sage (the system) build system is unrelated to distributing sage the library. so changes to the build system in Sage is somewhat vague. i think a build system for the sage library should at least check availibility (especially optional dependencies). more checks for versions/header functionality/existing symbols are implemented within autotools. the universal way would be maintaining these checks (which is unrealistic). catching common caveats should be possible. I mostly thought about the build system of the Sage system until now and didn't think about the build process of the library. Since availability of dependencies is ensured by the Sage system, the build process of the Sage library doesn't check for dependencies, right? Your idea is to add dependency checks to the library build, and that may be a good idea, but one could also consider to let the top level build scripts keep the job of handling the dependencies and do the checks there. I'm not saying that this would be better, just that this was my idea until now. Every distribution has its own way to organize patches. I think Sage already has a good test coverage and a Debian package should of course run all the tests during build. iirc the tests are contained within sage.spkg (where the library is), so that should be possible. other tests may come within other spkgs. Remember the git transition. spkg's will go away. There will be just one tarball containing everything in the future. The sources of other projects will be better separated, so it will be easy to get a Sage tarball without sources of other projects. Taking this into account, it probably makes sense to have only one Sage Debian source package. spkgs won't fully go away -- with the initial transition we are simply emulating the current spkg model. That said, there has been some work in transitioning to the ebuild format to use portage as a package manager -- but from what I understand, this is still a ways off. One thing to note with the git transition is that I made an effort to (mostly) separate the build system from the sage library (by this I mean the core components of sage: the python library, csage, documentation, scripts, ...). So hopefully this will help in determining what is needed in packaging sage. For those so inclined to browse the current status of the git transition, the repository is at https://github.com/sagemath/sage. It seems to me that you understood that Sage should build all spkg's in any case and just install them somewhere else. I think that means the step you call fix Sage the operating system is not needed right? Now that I have clarified this, any new thoughts? for debian or other distributions, i dont see the need for any changes to sage (the system). making sagelib distributable doesn't help skipping spkgs. so there are still two subprojects: - improve sage (the library) distributability - enhance setup.py (better switch to autotools) - get libcsage.deb and python-sage.deb packages - run the unit tests from debian/rules - (also pack other stuff) - pave the way for local spkgs (e.g. installed to ~/.sage/local) - [..] Switching to autotools is something we can't decide alone. What do Sage developers say? Do you mean with Sage (the library) all Sage components including notebook etc? - fix sage the system - implement/incorporate top level configure script - figure out flags, pass down to spkg-compilation - build sage the upstream way skipping some spkgs - [..] in fact, the second is not needed, neither for me, nor for the debian package. the project description implies something different, which is confusing. It would still be nice if the top level script could be used by distributions. There are still several things to build and other things to do, if external dependencies are not built, and we should not implement all this in debian/rules. One could start with the option to build all or none of the dependencies and then maybe go further to allow more combinations. But I'm also not entirely sure if combining system and bundled dependencies is needed. Maybe an alternative build script for building with system libraries would be a better idea. Are there other opinions? Cheers, Tobias -- You received this message because you are subscribed to the Google Groups sage-devel group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel?hl=en. For more options, visit https://groups.google.com/groups/opt_out. -- Andrew -- You received this message
Re: [sage-devel] Re: GSoC project: make the Sage build system more distribution friendly
On Sun, Mar 17, 2013 at 9:28 AM, Julien Puydt julien.pu...@laposte.net wrote: Le 17/03/2013 15:22, leif a écrit : than...@debian.org wrote: Am Samstag, 16. März 2013 20:57:13 UTC+1 schrieb leif: But if we switch to git, improve Sage's package management (as a first step, split vanilla upstream sources off the spkgs :P ), ... Is splitting the vanilla upstream sources off planned? That would be very helpful for distributions. Another helpful thing would be a clear distinction between fixes/adjustment to the library and Sage glue, because we need all the Sage glue, but want to decide ourselves which other patches to apply. We discussed this many times, and hopefully due to switching to git this will finally happen, because the src/ (upstream) dirs in spkgs aren't tracked, should always be vanilla (modulo fixing weird file permissions in a few cases), and hence do not change the often many times we bump an spkg's patch level (making just small changes to the Sage part which is under our revision control, including patches to upstream). Here is what is used when managing a debian package with git, to store upstream sources in a branch and get a unpatched (pristine) source tarball when needed : http://joeyh.name/code/pristine-tar/ when a new upstream gets out, a script makes it easy to update the upstream source by just pointing to the new tarball: http://honk.sigxcpu.org/projects/git-buildpackage/manual-html/gbp.import.html As has been mentioned, we're hoping to move to something like that at the workshop later this month as we've greatly outgrown the home-grown solution that currently exists. As is, any one-character change to an spkg's spkg-install script, say, means you have to again download the whole compressed archive (often 12 MB+) just to check the changes or reinstall the package. Although we have an upgrade path (untarred sage source dist accessible via http or rsync) in addition to the source distribution tarball, the former still just contains the compressed, self-contained spkgs. (Even worse, sometimes unmodified spkgs get repackaged from release to release such that not just their file modifications times but also their md5sums differ, for no real reason.) I.e., you currently cannot simply pull changes to (Sage's part of) spkgs. A branch for upstream, a branch for packaging, and scripts which make it easy to work with both ; see for example: Another thing are patch headers. Sometimes I can't find out what a patch in a spkg does, for example I found patches that just move a few lines of code somewhere else. I can't apply a patch when I don't know what it does. In Debian we have this specification for patch headers to document what they do: http://dep.debian.net/deps/dep3/ Well, every spkg contains an SPKG.txt file with a changelog and -- hopefully -- a Patches section (if appropriate), which in a perfect world precisely documents what's done why (including what patches are applied for what reason, whether they come from or went upstream etc.). Each patch should be its own documentation -- even if it's just to say Look in SPKG.txt, silly!. It's extremely silly. SPKG.txt and manually creating/applying/checking in patches is basically a poor man's revision control, with all the data on an external website, and some critical pieces not in revision control at all (e.g. what packages and versions are in a specific release of Sage). (Each entry of the changelog should also contain a Sage trac ticket number -- unfortunately sometimes more or less just this, which means further reading.) There's usually also a Special Update/Build Instructions section, which for example states patch foo can and should be removed once we upgrade to upstream version x.y, or check whether patch foo is still required. Unfortunately the distinction between generic fixes to upstream and Sage-specific modifications isn't always that clear; a single patch may even include both. In addition, there's the Mercurial history (but the repo is again only in the spkg itself), although few people provide detailed commit messages. This is perhaps suboptimal, but similar to the patches to the Sage library (see below), which hardly ever contain details, but just the corresponding trac ticket numbers in their commit messages. I'd be happy if Sage in whole was more modular / less monolithic; I'm not very optimistic regarding the Sage /library/ though. What do you mean with the Sage /library/? The Sage library is contained in a sage-x.y.z spkg, has its own repo (one can pull from for official/final releases, hg.sagemath.org), and contains the heart of Sage, i.e., genuine Sage code, as opposed to upstream components. In a Sage installation, it lives in $SAGE_ROOT/devel/sage/ (meant to get [cloned and] modified by pretty ordinary users as well; every Sage user is [also] a Sage developer). H... there is no .hg in $SAGE_ROOT/devel/sage/, but
Re: [sage-devel] Re: GSoC project: make the Sage build system more distribution friendly
On 17/03/13 21:38, Julien Puydt wrote: Le 17/03/2013 00:36, than...@debian.org a écrit : Is splitting the vanilla upstream sources off planned? That would be very helpful for distributions. Another helpful thing would be a clear distinction between fixes/adjustment to the library and Sage glue, because we need all the Sage glue, but want to decide ourselves which other patches to apply. Let me add that there is also no clear distinction between tests which check the code does correct calculations and tests which check things are installed in some place and not in another ; see for example: http://trac.sagemath.org/sage_trac/ticket/12627 having such tests spread here and there makes fractioning sage in distribution packages more painful than should be. Fortunately the instances of tests for that are not as many as you make it sound. There are indeed the issues of trac 12627. Actually in the case of Singular there is the added issue that sage install a singular file and calls it when upstream only ships Singular-3-1-5. In this case that make the sage call a bit more version independent. But there are doctests that indeed specify a path or part of a path that is sage specific, local/share/pari in sage/interfaces/gp.py and local/share/maxima in doc/en/constructions/plotting.rst are the only examples I can see. Other test are for precise version number when other versions works just as well (R, pari version is also tested but you should follow sage closely there). Francois -- You received this message because you are subscribed to the Google Groups sage-devel group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [sage-devel] Re: GSoC project: make the Sage build system more distribution friendly
On 2013-03-17 09:38, Julien Puydt wrote: The impression I have is that by default, a patch in an spkg has not. It really should be. It doesn't mean that upstream will *accept* the patch of course. Or sometimes, Sage contains patches to spkgs which are just meant for the Sage - package interface which would not make sense in the upstream tree. -- You received this message because you are subscribed to the Google Groups sage-devel group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [sage-devel] Re: GSoC project: make the Sage build system more distribution friendly
Le 16/03/2013 18:36, leif a écrit : Julien Puydt wrote: Le 16/03/2013 16:57, Tobias Hansen a écrit : == Desirable skills: == familiarity with shell scripts, makefiles and C libraries You forgot python. Autotools, C++, Common Lisp / ECL, Cython, Perl, ... ? ;-) Do they know about Sage-on-Gentoo, Jan's Ubuntu PPA, ... ? Ubuntu's sagemath ppa is a monolithic package, hence has nothing at all to do with Tobias' proposition. The proposition isn't to fork sage to get it into a single distribution, but to modify it with upstream so any distribution can easily package it correctly. Snark on #debian-science and #sagemath -- You received this message because you are subscribed to the Google Groups sage-devel group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
