[sage-devel] Re: spkg refactoring and development model
On 6/22/07, William Stein [EMAIL PROTECTED] wrote: Could you say something about the fact that currently there is no way to uninstall a SAGE package, since we don't track what files are actually installed. That said, we definitely *could* implement something simple that stores a list of all files installed in the install certificate in the spkg/installed directory. I'm thinking of something like computing a list of all files before then after the install, hence getting a list of what was installed (plus a list of files overwritten). Anyway, I'm just randomly throwing this out for comment. Maybe people who do gentoo/rpm/deb's can make a more intelligent comment. For SAGE uninstall isn't a big deal, since essentially nobody ever wants to uninstall a package. It's mostly painless to remove a package if: -- you know where all its files are. Most of them are in SAGE_ROOT/local/include/ and SAGE_ROOT/local/lib/ . It's pretty to figure out what goes where from the spkg-install script. -- you know that SAGE won't be using the functionality provided by this package itself. It would be really cool if by removing maxima for example, the uninstall script also figured out that that SAGE_ROOT/devel/sage-main/sage/calculus/*.py would also have to be removed. Really cool, but probably a bit overkill -- you remove the file with its name located in SAGE_ROOT/spkg/installed/ BTW, I have attached a small patch that makes installing an SPKG from a hard drive possible. On 6/19/07, Brian Granger [EMAIL PROTECTED] wrote: Isn't newest_version already on the users system if they already have sage installed - it is usd to build sage in the first place. It's already there, along with all standard packages that are needed to build SAGE. I'm not sure why it's still being downloaded. This is a great summary of all spkg things! This should definitely be put into the SAGE docs somewhere. There is a section on creating an SPKG (http://modular.math.washington.edu/sage/doc/html/prog/node24.html), I can update it to reflect the new changes. didier --~--~-~--~~~---~--~~ To post to this group, send email to sage-devel@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sage-devel URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/ -~--~~~~--~~--~--~--- diff -r 1b20b47c02d7 sage-spkg --- a/sage-spkg Sat Jun 02 22:31:55 2007 -0700 +++ b/sage-spkg Wed Jun 27 14:16:26 2007 -0400 @@ -83,6 +83,12 @@ PKG_NAME=`basename $PKG_NAME` PKG_NAME=`basename $PKG_NAME` PKG_SRC=$1 PKG_BASE=`echo $PKG_NAME | sed -e s/-.*//` + +# This is a local package +if [ -f $PKG_SRC ]; then +echo Installing $1 from hard drive +PKG_SRC=$1 +fi if [ ! -f $PKG_SRC ]; then if [ -f $SAGE_PACKAGES/standard/$PKG_NAME.spkg ]; then
[sage-devel] Re: spkg refactoring and development model
On 6/27/07, didier deshommes [EMAIL PROTECTED] wrote: It's mostly painless to remove a package if: -- you know where all its files are. Most of them are in SAGE_ROOT/local/include/ and SAGE_ROOT/local/lib/ . It's pretty to figure out what goes where from the spkg-install script. -- you know that SAGE won't be using the functionality provided by this package itself. It would be really cool if by removing maxima for example, the uninstall script also figured out that that SAGE_ROOT/devel/sage-main/sage/calculus/*.py would also have to be removed. Really cool, but probably a bit overkill -- you remove the file with its name located in SAGE_ROOT/spkg/installed/ Pain! Then starting SAGE and type x^2 would break. That would be nuts. I'm only referring to uninstalling optional packages. BTW, I have attached a small patch that makes installing an SPKG from a hard drive possible. Thanks! On 6/19/07, Brian Granger [EMAIL PROTECTED] wrote: Isn't newest_version already on the users system if they already have sage installed - it is usd to build sage in the first place. It's already there, along with all standard packages that are needed to build SAGE. I'm not sure why it's still being downloaded. It's downloaded just because I'm worried that if I ever improve it or fix a bug in it, then it needs to be downloaded to the new version will be used. This hasn't happened in a long time, but it used to happen regularly. This is a great summary of all spkg things! This should definitely be put into the SAGE docs somewhere. There is a section on creating an SPKG (http://modular.math.washington.edu/sage/doc/html/prog/node24.html), I can update it to reflect the new changes. didier diff -r 1b20b47c02d7 sage-spkg --- a/sage-spkg Sat Jun 02 22:31:55 2007 -0700 +++ b/sage-spkg Wed Jun 27 14:16:26 2007 -0400 @@ -83,6 +83,12 @@ PKG_NAME=`basename $PKG_NAME` PKG_NAME=`basename $PKG_NAME` PKG_SRC=$1 PKG_BASE=`echo $PKG_NAME | sed -e s/-.*//` + +# This is a local package +if [ -f $PKG_SRC ]; then +echo Installing $1 from hard drive +PKG_SRC=$1 +fi if [ ! -f $PKG_SRC ]; then if [ -f $SAGE_PACKAGES/standard/$PKG_NAME.spkg ]; then -- William Stein Associate Professor of Mathematics University of Washington http://www.williamstein.org --~--~-~--~~~---~--~~ To post to this group, send email to sage-devel@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sage-devel URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/ -~--~~~~--~~--~--~---
[sage-devel] Re: spkg refactoring and development model
On Jun 27, 2007, at 12:05 PM, didier deshommes wrote: On 6/22/07, William Stein [EMAIL PROTECTED] wrote: Could you say something about the fact that currently there is no way to uninstall a SAGE package, since we don't track what files are actually installed. That said, we definitely *could* implement something simple that stores a list of all files installed in the install certificate in the spkg/installed directory. I'm thinking of something like computing a list of all files before then after the install, hence getting a list of what was installed (plus a list of files overwritten). Anyway, I'm just randomly throwing this out for comment. Maybe people who do gentoo/rpm/deb's can make a more intelligent comment. For SAGE uninstall isn't a big deal, since essentially nobody ever wants to uninstall a package. It's mostly painless to remove a package if: -- you know where all its files are. Most of them are in SAGE_ROOT/local/include/ and SAGE_ROOT/local/lib/ . It's pretty to figure out what goes where from the spkg-install script. -- you know that SAGE won't be using the functionality provided by this package itself. It would be really cool if by removing maxima for example, the uninstall script also figured out that that SAGE_ROOT/devel/sage-main/sage/calculus/*.py would also have to be removed. Really cool, but probably a bit overkill -- you remove the file with its name located in SAGE_ROOT/spkg/ installed/ I think this works *only* if each package provides unique copies of things like libraries . For example, if a package uses libfac or libsing, and you remove the singular package, you pull the rug out from under the former. One reason package management is so difficult is this problem: you have to do reference counting on each file that SAGE uses. Computer history is littered with the corpses of packaging systems. :-} Justin -- Justin C. Walker, Curmudgeon-At-Large, Director Institute for the Enhancement of the Director's Income The path of least resistance: it's not just for electricity any more. --~--~-~--~~~---~--~~ To post to this group, send email to sage-devel@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sage-devel URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/ -~--~~~~--~~--~--~---
[sage-devel] Re: spkg refactoring and development model
On 6/18/07, Nick Alexander [EMAIL PROTECTED] wrote: didier deshommes [EMAIL PROTECTED] writes: On Jun 1, 10:59 am, William Stein [EMAIL PROTECTED] wrote: package_dir spkg-install-- (required) shell script run to install the package spkg-rebuild -- (optional) download latest version of package from web page and recreate the src directory spkg-check-- (optional) runs the packages test suite after spkg-install succeeds README.txt -- (optional) what used to be SAGE.txt /patches-- (optional) any files needed by spkg-install for patching /src-- (warning if not there) exact copy of what is available at some website. /.hg-- (optional) mercurial repository that has spkg-install, spkg-rebuild, spkg-check, README.txt and all contents of patches. This is unbelievably cool. I just had to hard reinstall IPython because I hacked it too hard and couldn't hg revert to the previous revision. A little tutorial on how one hacks a SAGE package would really help developers such as Dave Saunders, who is working on linbox_wrap, too. I'm not sure how to write this since it seems to be so easy to me. I'll start from the begining, ie I won't worry about dumbing it down too much and stating obvious things. Q: what is a SAGE package? A: A SAGE package (spkg) is a script that builds an install a piece of software for your SAGE installation. Thus an spkg usually has 2 things in it: - The source code of the software - The script that will build and install the software into SAGE. Q: What does one look like? A: Look in SAGE_ROOT/spkg/standard and you'll find a bunch of files with .spkg extensions. For example box-1.0.spkg might be a package living there. An spkg is just a tar ball. If you decompress box-1.0.spkg, you'll see something that looks like this: box-1.0/ spkg-install -- (required) shell script run to install the package spkg-check -- (optional) runs the packages test suite after spkg-install succeeds SAGE.txt -- Any modification done to the src/ directory are recorded here /patches-- (optional) any files needed by spkg-install for patching /src-- the software that has to be installed. /.hg-- A mercurial repository that has spkg-install, spkg-rebuild, spkg-check, SAGE.txt and all contents patches. Q: What's new about this format? What's all the fuss about? A: It was decided that: -- Having the installation scripts under version control would be a good idea for people that have to make and modify spkg's frequently. -- Separating source code and installation scripts would would keep us from the temptation of forking other projects. I also think this will force us --~--~-~--~~~---~--~~ To post to this group, send email to sage-devel@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sage-devel URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/ -~--~~~~--~~--~--~---
[sage-devel] Re: spkg refactoring and development model
[Apologies, I hit Send too soon] I'm not sure how to write this since it seems to be so easy to me. I'll start from the beginning, ie I won't worry about dumbing it down too much and stating obvious things. Q: what is a SAGE package? A: A SAGE package (spkg) is a script that builds an install a piece of software for your SAGE installation. Thus an spkg usually has 2 things in it: - The source code of the software - The script that will build and install the software into SAGE. Q: What does one look like? A: Look in SAGE_ROOT/spkg/standard and you'll find a bunch of files with .spkg extensions. For example box-1.0.spkg might be a package living there. An spkg is just a tar ball. If you decompress box-1.0.spkg, you'll see something that looks like this: box-1.0/ spkg-install -- (required) shell script run to install the package spkg-check -- (optional) runs the packages test suite after spkg-install succeeds SAGE.txt -- Any modification done to the src/ directory are recorded here /patches-- (optional) any files needed by spkg-install for patching /src-- the software that has to be installed. /.hg-- A mercurial repository that has spkg-install, spkg-rebuild, spkg-check, SAGE.txt and all contents patches. Q: What's new about this format? What's all the fuss about? A: It was decided that: -- Having the installation scripts under version control would be a good idea for people that have to make and modify spkg's frequently. -- Separating source code and our patches would would keep us from the temptation of forking other projects. I also think this will force us to send patches upstream when we're tired of managing too much of them. Q: OK, so how do I create one? A: Suppose you have a piece of software called pa-1.0.tar.gz written in C/C++ that you've downloaded from sourceforge. Here's how I would do it: -- make a fresh directory. I'll call mine new/ -- save pa-1.0.tar.gz in it -- untar pa-1.0.tar.gz -- rename pa-1.0/ to src/ -- create a spkg-install script. This is a shell script that will run ./configure, make, make install. You want to install the library in SAGE_ROOT/local/lib/, so might want to set --prefix=SAGE_ROOT/local/lib/ for the configure script. Header files go into SAGE_ROOT/local/include. Any additional libraries that you want to compile pa-1.0 against are in SAGE_ROOT/local/lib/ , BTW. -- if you have to modify pa-1.0 's source itself in any way, don't do it directly. Instead, put the file you've modified in the patches/ directory, and in your spkg-install script, run something like cp patches/file.c src/file.c before configuring and building. -- put spkg-install and patches/ under version control: hg init ; hg add spkg-install patches; hg ci -m new spkg created for pa! -- rename new/ to pa-1.0/ -- run: sage -spkg pa-1.0/ and it will create for you pa-1.0.spkg. You're done! Special thanks to Brian, William and Joel for the discussion and feedback. didier --~--~-~--~~~---~--~~ To post to this group, send email to sage-devel@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sage-devel URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/ -~--~~~~--~~--~--~---
[sage-devel] Re: spkg refactoring and development model
This is a great summary of all spkg things! This should definitely be put into the SAGE docs somewhere. Brian On 6/22/07, didier deshommes [EMAIL PROTECTED] wrote: [Apologies, I hit Send too soon] I'm not sure how to write this since it seems to be so easy to me. I'll start from the beginning, ie I won't worry about dumbing it down too much and stating obvious things. Q: what is a SAGE package? A: A SAGE package (spkg) is a script that builds an install a piece of software for your SAGE installation. Thus an spkg usually has 2 things in it: - The source code of the software - The script that will build and install the software into SAGE. Q: What does one look like? A: Look in SAGE_ROOT/spkg/standard and you'll find a bunch of files with .spkg extensions. For example box-1.0.spkg might be a package living there. An spkg is just a tar ball. If you decompress box-1.0.spkg, you'll see something that looks like this: box-1.0/ spkg-install -- (required) shell script run to install the package spkg-check -- (optional) runs the packages test suite after spkg-install succeeds SAGE.txt -- Any modification done to the src/ directory are recorded here /patches-- (optional) any files needed by spkg-install for patching /src-- the software that has to be installed. /.hg-- A mercurial repository that has spkg-install, spkg-rebuild, spkg-check, SAGE.txt and all contents patches. Q: What's new about this format? What's all the fuss about? A: It was decided that: -- Having the installation scripts under version control would be a good idea for people that have to make and modify spkg's frequently. -- Separating source code and our patches would would keep us from the temptation of forking other projects. I also think this will force us to send patches upstream when we're tired of managing too much of them. Q: OK, so how do I create one? A: Suppose you have a piece of software called pa-1.0.tar.gz written in C/C++ that you've downloaded from sourceforge. Here's how I would do it: -- make a fresh directory. I'll call mine new/ -- save pa-1.0.tar.gz in it -- untar pa-1.0.tar.gz -- rename pa-1.0/ to src/ -- create a spkg-install script. This is a shell script that will run ./configure, make, make install. You want to install the library in SAGE_ROOT/local/lib/, so might want to set --prefix=SAGE_ROOT/local/lib/ for the configure script. Header files go into SAGE_ROOT/local/include. Any additional libraries that you want to compile pa-1.0 against are in SAGE_ROOT/local/lib/ , BTW. -- if you have to modify pa-1.0 's source itself in any way, don't do it directly. Instead, put the file you've modified in the patches/ directory, and in your spkg-install script, run something like cp patches/file.c src/file.c before configuring and building. -- put spkg-install and patches/ under version control: hg init ; hg add spkg-install patches; hg ci -m new spkg created for pa! -- rename new/ to pa-1.0/ -- run: sage -spkg pa-1.0/ and it will create for you pa-1.0.spkg. You're done! Special thanks to Brian, William and Joel for the discussion and feedback. didier --~--~-~--~~~---~--~~ To post to this group, send email to sage-devel@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sage-devel URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/ -~--~~~~--~~--~--~---
[sage-devel] Re: spkg refactoring and development model
On 6/18/07, Brian Granger [EMAIL PROTECTED] wrote: I am talking about the both the directory structure that is needed for the script/makefiles to function properly: spkg/ standard/ base/ As well as the actual scripts and makefiles (deps, install, newest-version, etc) that have to be in particular locations in that tree to work properly. For instance, deps as to be in standard, install in spkg newest-version in standard, and a bunch of stuff has to be in base. I agree with this: the scripts in spkg/, spkg/base/ and spkg/standard/ should be under version control. SPKG.txt SPKG-README.txt I vote for SPKG.txt too. didier --~--~-~--~~~---~--~~ To post to this group, send email to sage-devel@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sage-devel URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/ -~--~~~~--~~--~--~---
[sage-devel] Re: spkg refactoring and development model
On 6/18/07, Brian Granger [EMAIL PROTECTED] wrote: Then SAGE wouldn't know that the package had been updated. Those version numbers are used to determine whether it is necessary to install a package. I didn't know SAGE was actually using them. Is this when sage -update is run? My understanding is that the decision to upgrade something was based on the timestamps if the files compared with the stub in the installed directory. Am I mistaken. The sage-update script gives an idea of how sage is upgraded from a SAGE_SERVER: - download the new install script - download the new list file. The list file contains all spkgs with their version numbers. It is from it that sage knows which packages have been updated and why specifying version numbers for now is necessary. - download the new deps file - download the newest_version script. newest_version determines the latest version of an spkg you have installed locally. - check the contents of your list file against your `ls spkg/standard/`. Anything that is in list and not in the output of ls is downloaded. didier --~--~-~--~~~---~--~~ To post to this group, send email to sage-devel@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sage-devel URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/ -~--~~~~--~~--~--~---
[sage-devel] Re: spkg refactoring and development model
On 6/18/07, Brian Granger [EMAIL PROTECTED] wrote: I didn't know SAGE was actually using them. Is this when sage -update is run? My understanding is that the decision to upgrade something was based on the timestamps if the files compared with the stub in the installed directory. Am I mistaken. Yes. It is based on the version number. Timestamps aren't used for upgrading. OK, that makes sense. Is there a standard approach for these version numbers? At times I have seen a projects version number upgraded, but the .pn version number remain the same. For instance, the python spkg is named: python-2.5.1.p3.spkg But I don't think the .p3 means the third version of the spkg for python-2.5.1. The standard is package_name[dash]official-version[dash]sage_patch_number Thus python-2.5.1.p3.spkg is the fourth (0-based) version of the Python 2.5.1 package. If the package number jumped from 2.5.0.p3 to 2.5.1.p3 that would be a mistake. OK, that really clarifies things. I thought that might be the case, but the mistakes were throwing me off. As long as everyone understands this convention and updates the versions when they need to be this sounds fine. There is one case though that is not clear. If multiple people (even 2) have made changes to an spkg and both of them have updated the sage_patch_number, there is i) possibility for confusion as both packages will have the same version number, but contain different changes and ii) someone has to merge the changes and make sure the sage_patch_number is globally unique. Again, an informal social protocol will probably suffice in this case as well, but it is something that developers need to be aware of. This seems a bit confusing to me. Especially because some spkgs don't have a .pn version, even though they were updated. Sometimes a package is updated in a trivial way, e.g., something is added to the README or a directory is reorganized, but there is absolutely no reason to force very SAGE user out there to upgrade/build that package. In such cases, I usually do not update the version number. The thing that I keep running into though is that because of these minor changes (which are in fact important to keep) I can't trust the sage_patch_number to solely determine if a package has changed. I still have to do manual diff or look at md5sums to determine that. Granted, having hg repo for each spkg will completely solve this problem though. I don't mind having a version that denotes the spkg version, I just want to make sure that those version number are consistent and standardized and have a purpose that is not redundant. Could we use md5 sums for this? In many cases, I do this anyway, because the .pn version number is not a reliable way of determining if an spkg has changed. The md5 sums are unfortunately not in order, so that doesn't work. md5 only tells you whether two packages are different, not which is newer. Definitely. One issue is that it isn't a readme, actually. It's a description of how the package was made. True, then maybe SPKG.txt makes more sense? Or maybe BUILD-NOTES.txt? SPKG-BUILD-NOTES.txt? SPKG.txt is I think clearly the best choice given our constraints. Sounds good. Brian William --~--~-~--~~~---~--~~ To post to this group, send email to sage-devel@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sage-devel URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/ -~--~~~~--~~--~--~---
[sage-devel] Re: spkg refactoring and development model
Thanks, this does clarify what goes on in an update. The sage-update script gives an idea of how sage is upgraded from a SAGE_SERVER: - download the new install script - download the new list file. The list file contains all spkgs with their version numbers. It is from it that sage knows which packages have been updated and why specifying version numbers for now is necessary. - download the new deps file - download the newest_version script. newest_version determines the latest version of an spkg you have installed locally. Isn't newest_version already on the users system if they already have sage installed - it is usd to build sage in the first place. - check the contents of your list file against your `ls spkg/standard/`. Anything that is in list and not in the output of ls is downloaded. Brian didier --~--~-~--~~~---~--~~ To post to this group, send email to sage-devel@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sage-devel URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/ -~--~~~~--~~--~--~---
[sage-devel] Re: spkg refactoring and development model
I've updated all spkg's so that they now have an hg repo with the directory structure based on the one listed above. Comments and suggestions are welcomed. You can see them here: http://sage.math.washington.edu/home/dfdeshom/custom/spkg/ Fantastic! The naming convention for spkgs that includes the .pn post-fix where n labels the spkg version is probably not needed now. My understanding is that label was loosely used as a sort of version control. I think simply having the hg repo for each solves that problem. Also, there doesn't seem to be any real consistent approach for updating those numbers. My though would be to get rid of them. The other thing that I am not sure how to handle (part of this is my lack of experience with hg) is the names of the directories themselves, for example: freetype-2.1.10.spkg When freetype gets upgraded to 2.1.11, do we simply rename the directory to freetype-2.1.11? Will hg be happy with that? Sorry, I just don't know how hg treats directory renames. One note: I would prefer that READMET.txt be kept as SAGE.txt (or renamed to something else) because then it would be clear that modifications were made to the spkg. I tend to ignore README.txt's anyway because they don't tell you why you should read them. I agree that README.txt is a little confusion, but because the spkgs are being used outside of SAGE, I would name the file something that emphasizes that it is really the readme for the spkg: SPKG.txt SPKG-README.txt or something like that. One of the things I have been working on (on my private copies) is to decouple the spkg build system from the details of sage itself. This has mostly meant changing the name sage to spkg in the context of the build system. On 6/1/07, Brian Granger [EMAIL PROTECTED] wrote: 2. The other scripts/makefiles involved in the spkg build process - like newest_version, install, save-env, etc. sage-env is under version control, but the others aren't. 3. The overall directory structure that contains the scrips/makefiles. The reason this is important is that most of the scripts make strong assumptions about where they are run from. Also, as I have begun refactoring, some of the scripts have been moved around (for instance newest_version has been moved to base/spkg-newest-version). Managing this without the associated directory structure is a pain. Are you talking about the scripts in SAGE_ROOT/local/bin ? Because they are under version control (this is where sage-env lives too). I am talking about the both the directory structure that is needed for the script/makefiles to function properly: spkg/ standard/ base/ As well as the actual scripts and makefiles (deps, install, newest-version, etc) that have to be in particular locations in that tree to work properly. For instance, deps as to be in standard, install in spkg newest-version in standard, and a bunch of stuff has to be in base. Once the overall structure of these new spkgs have settled down, I will start to convert my private spkgs (I have things like (qt and pyqt4, and a bunch of other things) into this form. Then contributing them will be really simple. Brian didier --~--~-~--~~~---~--~~ To post to this group, send email to sage-devel@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sage-devel URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/ -~--~~~~--~~--~--~---
[sage-devel] Re: spkg refactoring and development model
On 6/18/07, Brian Granger [EMAIL PROTECTED] wrote: I've updated all spkg's so that they now have an hg repo with the directory structure based on the one listed above. Comments and suggestions are welcomed. You can see them here: http://sage.math.washington.edu/home/dfdeshom/custom/spkg/ Fantastic! The naming convention for spkgs that includes the .pn post-fix where n labels the spkg version is probably not needed now. My understanding is that label was loosely used as a sort of version control. I think simply having the hg repo for each solves that problem. Also, there doesn't seem to be any real consistent approach for updating those numbers. My though would be to get rid of them. Then SAGE wouldn't know that the package had been updated. Those version numbers are used to determine whether it is necessary to install a package. The other thing that I am not sure how to handle (part of this is my lack of experience with hg) is the names of the directories themselves, for example: freetype-2.1.10.spkg When freetype gets upgraded to 2.1.11, do we simply rename the directory to freetype-2.1.11? Will hg be happy with that? Sorry, I just don't know how hg treats directory renames. With hg it's just that there is a directory .hg instead freetype-2.1.10. Hg has no idea what the directory is called that it is in, and it doesn't care. We will need an easy SAGE tool to merge two versions of a package, e.g., to pull from a latest published version of the package, to take to freetype-* packages, say that I and didier made, and -- with almost no work -- merge them into a single new package. This can easily be automated, by calling hg pull, then using sage -pkg. One note: I would prefer that READMET.txt be kept as SAGE.txt (or renamed to something else) because then it would be clear that modifications were made to the spkg. I tend to ignore README.txt's anyway because they don't tell you why you should read them. I agree that README.txt is a little confusion, but because the spkgs are being used outside of SAGE, I would name the file something that emphasizes that it is really the readme for the spkg: SPKG.txt SPKG-README.txt or something like that. One issue is that it isn't a readme, actually. It's a description of how the package was made. One of the things I have been working on (on my private copies) is to decouple the spkg build system from the details of sage itself. This has mostly meant changing the name sage to spkg in the context of the build system. On 6/1/07, Brian Granger [EMAIL PROTECTED] wrote: 2. The other scripts/makefiles involved in the spkg build process - like newest_version, install, save-env, etc. sage-env is under version control, but the others aren't. 3. The overall directory structure that contains the scrips/makefiles. The reason this is important is that most of the scripts make strong assumptions about where they are run from. Also, as I have begun refactoring, some of the scripts have been moved around (for instance newest_version has been moved to base/spkg-newest-version). Managing this without the associated directory structure is a pain. Are you talking about the scripts in SAGE_ROOT/local/bin ? Because they are under version control (this is where sage-env lives too). I am talking about the both the directory structure that is needed for the script/makefiles to function properly: spkg/ standard/ base/ As well as the actual scripts and makefiles (deps, install, newest-version, etc) that have to be in particular locations in that tree to work properly. For instance, deps as to be in standard, install in spkg newest-version in standard, and a bunch of stuff has to be in base. Once the overall structure of these new spkgs have settled down, I will start to convert my private spkgs (I have things like (qt and pyqt4, and a bunch of other things) into this form. Then contributing them will be really simple. William --~--~-~--~~~---~--~~ To post to this group, send email to sage-devel@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sage-devel URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/ -~--~~~~--~~--~--~---
[sage-devel] Re: spkg refactoring and development model
Fantastic! The naming convention for spkgs that includes the .pn post-fix where n labels the spkg version is probably not needed now. My understanding is that label was loosely used as a sort of version control. I think simply having the hg repo for each solves that problem. Also, there doesn't seem to be any real consistent approach for updating those numbers. My though would be to get rid of them. Then SAGE wouldn't know that the package had been updated. Those version numbers are used to determine whether it is necessary to install a package. I didn't know SAGE was actually using them. Is this when sage -update is run? My understanding is that the decision to upgrade something was based on the timestamps if the files compared with the stub in the installed directory. Am I mistaken. Is there a standard approach for these version numbers? At times I have seen a projects version number upgraded, but the .pn version number remain the same. For instance, the python spkg is named: python-2.5.1.p3.spkg But I don't think the .p3 means the third version of the spkg for python-2.5.1. This seems a bit confusing to me. Especially because some spkgs don't have a .pn version, even though they were updated. I don't mind having a version that denotes the spkg version, I just want to make sure that those version number are consistent and standardized and have a purpose that is not redundant. Could we use md5 sums for this? In many cases, I do this anyway, because the .pn version number is not a reliable way of determining if an spkg has changed. The other thing that I am not sure how to handle (part of this is my lack of experience with hg) is the names of the directories themselves, for example: freetype-2.1.10.spkg When freetype gets upgraded to 2.1.11, do we simply rename the directory to freetype-2.1.11? Will hg be happy with that? Sorry, I just don't know how hg treats directory renames. With hg it's just that there is a directory .hg instead freetype-2.1.10. Hg has no idea what the directory is called that it is in, and it doesn't care. We will need an easy SAGE tool to merge two versions of a package, e.g., to pull from a latest published version of the package, to take to freetype-* packages, say that I and didier made, and -- with almost no work -- merge them into a single new package. This can easily be automated, by calling hg pull, then using sage -pkg. One note: I would prefer that READMET.txt be kept as SAGE.txt (or renamed to something else) because then it would be clear that modifications were made to the spkg. I tend to ignore README.txt's anyway because they don't tell you why you should read them. I agree that README.txt is a little confusion, but because the spkgs are being used outside of SAGE, I would name the file something that emphasizes that it is really the readme for the spkg: SPKG.txt SPKG-README.txt or something like that. One issue is that it isn't a readme, actually. It's a description of how the package was made. True, then maybe SPKG.txt makes more sense? Or maybe BUILD-NOTES.txt? SPKG-BUILD-NOTES.txt? One of the things I have been working on (on my private copies) is to decouple the spkg build system from the details of sage itself. This has mostly meant changing the name sage to spkg in the context of the build system. On 6/1/07, Brian Granger [EMAIL PROTECTED] wrote: 2. The other scripts/makefiles involved in the spkg build process - like newest_version, install, save-env, etc. sage-env is under version control, but the others aren't. 3. The overall directory structure that contains the scrips/makefiles. The reason this is important is that most of the scripts make strong assumptions about where they are run from. Also, as I have begun refactoring, some of the scripts have been moved around (for instance newest_version has been moved to base/spkg-newest-version). Managing this without the associated directory structure is a pain. Are you talking about the scripts in SAGE_ROOT/local/bin ? Because they are under version control (this is where sage-env lives too). I am talking about the both the directory structure that is needed for the script/makefiles to function properly: spkg/ standard/ base/ As well as the actual scripts and makefiles (deps, install, newest-version, etc) that have to be in particular locations in that tree to work properly. For instance, deps as to be in standard, install in spkg newest-version in standard, and a bunch of stuff has to be in base. Once the overall structure of these new spkgs have settled down, I will start to convert my private spkgs (I have things like (qt and pyqt4, and a bunch of other things) into this form. Then contributing them will be really simple. William
[sage-devel] Re: spkg refactoring and development model
On 6/18/07, Brian Granger [EMAIL PROTECTED] wrote: I didn't know SAGE was actually using them. Is this when sage -update is run? My understanding is that the decision to upgrade something was based on the timestamps if the files compared with the stub in the installed directory. Am I mistaken. Yes. It is based on the version number. Timestamps aren't used for upgrading. Is there a standard approach for these version numbers? At times I have seen a projects version number upgraded, but the .pn version number remain the same. For instance, the python spkg is named: python-2.5.1.p3.spkg But I don't think the .p3 means the third version of the spkg for python-2.5.1. The standard is package_name[dash]official-version[dash]sage_patch_number Thus python-2.5.1.p3.spkg is the fourth (0-based) version of the Python 2.5.1 package. If the package number jumped from 2.5.0.p3 to 2.5.1.p3 that would be a mistake. This seems a bit confusing to me. Especially because some spkgs don't have a .pn version, even though they were updated. Sometimes a package is updated in a trivial way, e.g., something is added to the README or a directory is reorganized, but there is absolutely no reason to force very SAGE user out there to upgrade/build that package. In such cases, I usually do not update the version number. I don't mind having a version that denotes the spkg version, I just want to make sure that those version number are consistent and standardized and have a purpose that is not redundant. Could we use md5 sums for this? In many cases, I do this anyway, because the .pn version number is not a reliable way of determining if an spkg has changed. The md5 sums are unfortunately not in order, so that doesn't work. md5 only tells you whether two packages are different, not which is newer. One issue is that it isn't a readme, actually. It's a description of how the package was made. True, then maybe SPKG.txt makes more sense? Or maybe BUILD-NOTES.txt? SPKG-BUILD-NOTES.txt? SPKG.txt is I think clearly the best choice given our constraints. William --~--~-~--~~~---~--~~ To post to this group, send email to sage-devel@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sage-devel URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/ -~--~~~~--~~--~--~---
[sage-devel] Re: spkg refactoring and development model
There's something I've been thinking about for awhile about the spkg packages. It seems relevant to the discussion at hand, but overall I'm not sure that there is any immediate merit to my comments. I'm a (happy) gentoo user and one of the principle reasons I'm with gentoo is the package management -- their portage system. Of course, it's a build-from-source model and it's really remarkably similar to how the spkg scripts do things. Both spkg and gentoo are also similar to the bsd ports method of doing things. The analog of the spkg spkg-install install script is the ebuild in gentoo. Here's an ebuild for PARI from the online CVS: http://sources.gentoo.org/viewcvs.py/gentoo-x86/sci-mathematics/pari/pari-2.1.5-r4.ebuild?rev=1.5view=markup Some things to note: - in the src_unpack, gentoo applies some patches which are kept on gentoo servers - it's essentially a bash script, but I believe there are some frills added somehow The main relation of this to the current discussion is that gentoo solves the patch/mainline problem by distributing patch files which are applied on the user's computer before the build of the package in question. Sometimes there are a large number of patches applied (I don't have a specific example, but I'm pretty sure I've seen as many as 10-15 patches applied before a build.) This way it is very clear to any concerned individual exactly what is coming from upstream (the original package is downloaded as part of the install process), vs. what is coming from the gentoo packagers. In gentoo, when a new package comes out, the upgrade process can be as simple as updating the filename of the ebuild file to have the new version number. If you are applying patches, things are a bit more complicated because you have to check if the patches are still needed or applicable. However, I'm sure there are many times when the patches just carry forward with-out a hitch. A good goal would be to keep those patches headed upstream so that new versions could eliminate all the patches of the old -- but that's easier said than done. Now, here's a wilder idea which I don't think is totally undoable. It seems to me that it might be possible to detatch the portage system scripts from a gentoo system and rewrite all the spkg-install scripts as ebuilds. This has two benefits: 1) You don't have to work on spkg (and reinventing the from-source package management wheel) 2) Once those ebuilds are done, you have sage packaged for gentoo. There should be absolutely nothing left to do. Instead of installing under the $SAGE_HOME directory, you install things to the root of the tree (portage has the ability to install things to a subdirectory -- of course, the norm is to install stuff to the root of the tree and I've never had a reason to do otherwise.) In portage parlance, you could say that you have a SAGE overlay which has a bunch of mathematical packages. Actually, after a bit of searching in the gentoo forums, I came accross this: http://www.pkgcore.org/ It appears to be an attempt at non-gentoo-ifying portage, but the website is sort of useless for actually learning about the project. And, for the snake lovers, portage is written in python :) ! -- Joel On Thursday 31 May 2007 22:42, Brian Granger wrote: This looks very interesting, I hadn't seen this before. I need to learn more about hg for sure. Would this approach require having the source to the upstream project in an hg repo? On 5/31/07, Fernando Perez [EMAIL PROTECTED] wrote: On 5/31/07, Brian Granger [EMAIL PROTECTED] wrote: source. This doesn't answer the question of how to maintain those patches, that don't get sent back upstream. This definitely needs to be addressed. If I understand correctly, that's what mercurial queues are for: http://hgbook.red-bean.com/hgbookch12.html Cheers, f --~--~-~--~~~---~--~~ To post to this group, send email to sage-devel@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sage-devel URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/ -~--~~~~--~~--~--~---
[sage-devel] Re: spkg refactoring and development model
On 6/1/07, Joel B. Mohler [EMAIL PROTECTED] wrote: The main relation of this to the current discussion is that gentoo solves the patch/mainline problem by distributing patch files which are applied on the user's computer before the build of the package in question. Sometimes there are a large number of patches applied (I don't have a specific example, but I'm pretty sure I've seen as many as 10-15 patches applied before a build.) This way it is very clear to any concerned individual exactly what is coming from upstream (the original package is downloaded as part of the install process), vs. what is coming from the gentoo packagers. I'm sure I'm missing something here. Could you elaborate more on how gentoo does it to solve the patching problem? If it's just a bunch of bash scripts, couldn't we write them ourselves? It seems to me that this is basically what spkg-install is doing now. didier --~--~-~--~~~---~--~~ To post to this group, send email to sage-devel@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sage-devel URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/ -~--~~~~--~~--~--~---
[sage-devel] Re: spkg refactoring and development model
On 5/31/07, Brian Granger [EMAIL PROTECTED] wrote: For each spkg, there would be a corresponding directory containing the spkg-install script and the sage subdirectory (but _not_ the source code for the package itself). I think it would be a good idea to create a format for the spkgs that is a little more standardized. Here is an example: ipython-0.8 spkg-install # existing spkg-install script ./spkg-patches # rename the sage directory to this for clarity spkg-fetch.py # a python script that could be called to get the source for ipython Would spkg-fetch.py download package A from A's homepage or from sage.math? I think the spkg-fetch.py script is an important part of this. It would allow us to not include the source code for each package in the repo, but it would make it very easy to write a script that begins with only the scripts in the repo and builds a full source distribution of everything. I think this penalty could be paid once - the source packages could be downloaded into the hg tree - but not commited to the repo. Once that is done, the spkgs could be built in place. My thought is that the only people who would have to do this are those who are maintaining/creating spkgs. Most people will simply use the created spkgs that include the source. So if I download sage-version.tar, it would have the spkgs with their sources. If I ran sage -upgrade, it would do an hg checkout on the spkg/standard directory, update any spkg that needs to be and download (if necessary) the sources to build and compile them. Did I get that right? didier --~--~-~--~~~---~--~~ To post to this group, send email to sage-devel@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sage-devel URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/ -~--~~~~--~~--~--~---
[sage-devel] Re: spkg refactoring and development model
On Friday 01 June 2007 10:26, didier deshommes wrote: On 6/1/07, Joel B. Mohler [EMAIL PROTECTED] wrote: The main relation of this to the current discussion is that gentoo solves the patch/mainline problem by distributing patch files which are applied on the user's computer before the build of the package in question. Sometimes there are a large number of patches applied (I don't have a specific example, but I'm pretty sure I've seen as many as 10-15 patches applied before a build.) This way it is very clear to any concerned individual exactly what is coming from upstream (the original package is downloaded as part of the install process), vs. what is coming from the gentoo packagers. I'm sure I'm missing something here. Could you elaborate more on how gentoo does it to solve the patching problem? If it's just a bunch of bash scripts, couldn't we write them ourselves? It seems to me that this is basically what spkg-install is doing now. Portage downloads the upstream package from upstream's website. It also (if there are patches) downloads the patches from gentoo's repositories. The ebuild script which defines the package then patches the unzipped upstream package. The important part is that in the end, I have the original upstream tarball in my downloaded files directory. I also have the gentoo patches in my downloaded files directory. The ebuild file itself which defines the package is essentially a bash script, but the portage scripts that call into this bash script know how to build the default configure; make; make install sequence. Therefore, in theory, a simple package doesn't have to define much at all in the ebuild. Of course, if you have patches to apply, the ebuild is the code that applies them. And, yes, you are correct, it's not that much different than what the spkg-install is doing. That's why I point it out. It seems to me, though, that Brian is jumping off from spkg to make a more general package manager. I'm just wondering if the more general package manager isn't already made -- i.e. portage. -- Joel --~--~-~--~~~---~--~~ To post to this group, send email to sage-devel@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sage-devel URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/ -~--~~~~--~~--~--~---
[sage-devel] Re: spkg refactoring and development model
For each spkg, there would be a corresponding directory containing the spkg-install script and the sage subdirectory (but _not_ the source code for the package itself). I think it would be a good idea to create a format for the spkgs that is a little more standardized. Here is an example: ipython-0.8 spkg-install # existing spkg-install script ./spkg-patches # rename the sage directory to this for clarity spkg-fetch.py # a python script that could be called to get the source for ipython Would spkg-fetch.py download package A from A's homepage or from sage.math? I think the best approach would be to have spkg-fetch.py download package A from A's homepage. Patches would be kept in the spkg repo and applied by the spkg-install script upon building. In many ways this is very similar to gentoo = original source + patches - build - install. I think the spkg-fetch.py script is an important part of this. It would allow us to not include the source code for each package in the repo, but it would make it very easy to write a script that begins with only the scripts in the repo and builds a full source distribution of everything. I think this penalty could be paid once - the source packages could be downloaded into the hg tree - but not commited to the repo. Once that is done, the spkgs could be built in place. My thought is that the only people who would have to do this are those who are maintaining/creating spkgs. Most people will simply use the created spkgs that include the source. So if I download sage-version.tar, it would have the spkgs with their sources. If I ran sage -upgrade, it would do an hg checkout on the spkg/standard directory, update any spkg that needs to be and download (if necessary) the sources to build and compile them. Did I get that right? Almost. The first part is right - the full sage tarball would have all the spkg stuff and the package sources - just like it does today. But, sage -upgrade could work two different ways: 1. If you just want to get the latest stable version, it would download the newest spkgs (that contain the full source) and install them. This assumes that someone (the maintainer or the spkg) has used the spkg hg repo and the spkg-fetch.py script to create an updated spkg. But the important point is that most users won't see anything different from the current approach - they always deal with the full spkg with the package source. 2. For people who are doing lots of spkg development, we could make something like sage -devel that does what you describe - it would use the hg repo, download the source, create a new spkg and install it. The first parts of this process (creating the spkg) would be exactly what developers do to create a new spkg. Does that make sense? didier --~--~-~--~~~---~--~~ To post to this group, send email to sage-devel@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sage-devel URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/ -~--~~~~--~~--~--~---
[sage-devel] Re: spkg refactoring and development model
On 6/1/07, Brian Granger [EMAIL PROTECTED] wrote: * HG queues help with the issues of maintaining patches against a changing upstream source, but it is still a pain and nontrivial. Would the mq extension help? It's an hg extension for managing queues: http://www.selenic.com/mercurial/wiki/index.cgi/MqExtension didier --~--~-~--~~~---~--~~ To post to this group, send email to sage-devel@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sage-devel URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/ -~--~~~~--~~--~--~---
[sage-devel] Re: spkg refactoring and development model
On 6/1/07, Brian Granger [EMAIL PROTECTED] wrote: After looking some at hg queues, I have a few more comments about how to handle patching packages sources. * HG queues help with the issues of maintaining patches against a changing upstream source, but it is still a pain and nontrivial. The point of spkg's, at least as I seem them, is that they should be ridiculously simple, and that anybody can make one with about 5 minutes of directions. This is -- in my opinion -- in sharp contrast to the situation with .deb, .rpm's, cygwin packages, etc. A person with no packaging experience could easily spends literally days of hard work to really understand how to create rpm's correctly. Moreover, spkg's are supposed to be decentralized, in that anybody can just start creating them and sharing them with other people without any need to check them into any sort of central repository. I know for a fact that they accomplish both the above goals very well. I do not find spkg's themselves to be a pain to maintain. At least, the pain is *not* maintaining the spkg's, but actually solving the real problems of getting code to build and work correctly in the context of SAGE. This is almost never related to spkg's. Regarding multiple people working on spkg's, the main problem is that one person will look at an spkg-install, not understand it, delete it and create a new one, not realizing that the things they didn't understand are very subtle, involve only certain platforms, and often took hours to figure out. There are some real problems that Brian has pointed out which need to be solved: (1) It would be preferable if the package naming structure were standardized, though it should only be a warning if one violates this. E.g., sage -pkg package-dir will print non-fatal warnings if there is no src subdirectory, etc. I like this proposal, which is a slight extension of the one Brian suggested: package_dir spkg-install-- (required) shell script run to install the package spkg-rebuild -- (optional) download latest version of package from web page and recreate the src directory spkg-check-- (optional) runs the packages test suite after spkg-install succeeds README.txt -- (optional) what used to be SAGE.txt /patches-- (optional) any files needed by spkg-install for patching /src-- (warning if not there) exact copy of what is available at some website. /.hg-- (optional) mercurial repository that has spkg-install, spkg-rebuild, spkg-check, README.txt and all contents of patches. (2) I'm not enthusiastic about having a version of spkg's that requires an internet connection to use them. I would prefer simply improving the spkg-rebuild scripts. (3) Versioning issues. If one types sage -i foo then the newest version of the foo package should be installed. Currently one has to type the version number. Fixing this is just adding something to some shell script. (4) Dependencies among optional packages. Currently this is usually dealt with by simply putting sage -i foo-2.3, say in the top of spkg-install for dependent packages. This is OK, but it would probably be better if the version number were only a lower bound or didn't have to be specified. (5) Regarding all the source/revision control discussions, I think we should just put one (optional) .hg repository in each spkg. Simple. When somebody does a bunch of work to improve an spkg, they can make available an hg bundle that anybody else can apply to their spkg. * Because of this, I think mercurial queues only make sense when you really need it - when there are multiple patches that are difficult to diff/patch by hand. But in these cases, it probably does make lots of sense, and these sources should probably be in the hg repo. Thankfully, the big packages (qt/enthought) are not heavily patched. Mercurial queues are really confusing. I spent 2 hours once trying to figure them out and couldn't. As is, they thus fail the 5 minutes test for a new user test mentioned above. Of course, if it were worth it, we could probably figure out how to make them usable to people easily. * However, for the many spkgs that don't have patches applied, it seems crazy to put the package source into the hg repo. Yep. We should just put the patches and spkg-installs that we make in an hg repo. * There is still the issue that maintaining patches is a pain. It seems like there are two broad categories of paches: 1) patches that simply fix a bug in the original source 2) patches that actually provide new functionality for sage or allow the package to be installed in the sage context. In the first case, I think we really
[sage-devel] Re: spkg refactoring and development model
The point of spkg's, at least as I seem them, is that they should be ridiculously simple, and that anybody can make one with about 5 minutes of directions. This was the critical factor for me when I looked at the different options. This is -- in my opinion -- in sharp contrast to the situation with .deb, .rpm's, cygwin packages, etc. A person with no packaging experience could easily spends literally days of hard work to really understand how to create rpm's correctly. Moreover, spkg's are supposed to be decentralized, in that anybody can just start creating them and sharing them with other people without any need to check them into any sort of central repository. I know for a fact that they accomplish both the above goals very well. They are _mostly_ decentralized, but it would be nice if they were more so. A few things prevent this. One problem is when multiple people create slightly different versions of the same spkg. Having the spkg scripts hosted (even decentrally) on hg repos would dramatically improve this situation. The other thing we want to think about, is that for people to use a new spkg, they have to update two files: deps and the install script. It would be really nice if we could come up with a way that a user could simply drop the spkg in a directory and type make. I will try to think about how we could enable that and still preserve the dependency info. I do not find spkg's themselves to be a pain to maintain. At least, the pain is *not* maintaining the spkg's, but actually solving the real problems of getting code to build and work correctly in the context of SAGE. This is almost never related to spkg's. For the most part, I agree, but there are a few things I find to be a pain: 1. I have a number of spkgs that I maintain that other people also make changes to. Tracking these things without a repo is difficult and very error prone. 2. I have no source control for the spkgs I maintain. This means I have no history tracking, etc. Also, it means that I don't have any automated backups. I can setup my own hg repo, but I might as well work with everyone else to not duplicate efforts. 3. The other aspect of maintenance is the actual scripts that run the build process (outside the spkgs). As I have started refactoring those things, I have to manage everything by hand. For instance for others to benefit from my changes, I would have post a tarball of everything somewhere. That is no fun. Regarding multiple people working on spkg's, the main problem is that one person will look at an spkg-install, not understand it, delete it and create a new one, not realizing that the things they didn't understand are very subtle, involve only certain platforms, and often took hours to figure out. Yep. There are some real problems that Brian has pointed out which need to be solved: (1) It would be preferable if the package naming structure were standardized, though it should only be a warning if one violates this. E.g., sage -pkg package-dir will print non-fatal warnings if there is no src subdirectory, etc. I like this proposal, which is a slight extension of the one Brian suggested: package_dir spkg-install-- (required) shell script run to install the package spkg-rebuild -- (optional) download latest version of package from web page and recreate the src directory spkg-check-- (optional) runs the packages test suite after spkg-install succeeds README.txt -- (optional) what used to be SAGE.txt /patches-- (optional) any files needed by spkg-install for patching /src-- (warning if not there) exact copy of what is available at some website. /.hg-- (optional) mercurial repository that has spkg-install, spkg-rebuild, spkg-check, README.txt and all contents of patches. I like this a lot - especially the spkg-check. (2) I'm not enthusiastic about having a version of spkg's that requires an internet connection to use them. I would prefer simply improving the spkg-rebuild scripts. I also agree with this fully. One of the main benefits that I have found, is that I can give someone a tarball that i) has everything and ii) just works. (3) Versioning issues. If one types sage -i foo then the newest version of the foo package should be installed. Currently one has to type the version number. Fixing this is just adding something to some shell script. It would be nice to have code that actually parses the version number in an intelligent manner. Unfortunately, I really don't want to write that code in a bash script - python would be much nicer. On the other hand , I like not having python involved in the build process. (4) Dependencies among
[sage-devel] Re: spkg refactoring and development model
One other things I forgot to mention that would be nice to have. The way we are using spkg is to distribute a set of python tools that overlaps with, but that is not the same as what sage distributes. The set of tools are are distributing is more focused on numerical computing. Thus, we include things like numpy, scipy and related things, but we don't include the more mathematically focused packages. This is mainly because our users simply need a different subset of tools. Because of this, we maintain our own deps makefile that declares the dependencies among our spkgs. It would be very nice for people to be able to build a custom distribution that includes the spkgs they want/need and not have to create their own deps file. If this were possible, it would also make it very easy for people to create new spkgs and share them without everyone having ot update their install and deps files. Any ideas on how this could be done? One thought would be to have each spkg contain a file maybe called spkg-deps, that lists its dependencies. This might require more sophisticated versioning capabilities (but we probably want that anyway). The only thing I don't like about this approach is that all the spkgs would have to be unpacked for the dependency analysis to happen. This does give me an idea though. What about removing the version number off the end of the spkg name: foo-1.0.1.spkg - foo.spkg and instead putting the version number in the same place as the dependencies for the package. This could either be in a file in the spkg or in a separate file. The nice this about this approach is that all the dependency analysis could be done with this files alone. This would allow us to get rid of the newest-version script as well as the deps and install script. Instead we would need a script that looked at the version/deps files, figured out the dependency tree and then built things in the right order. The downside is that I wouldn't want to write the dependency analysis in a shell script - python again would be much nicer. I will think more about these things. To see if there is a better way. Brian --~--~-~--~~~---~--~~ To post to this group, send email to sage-devel@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sage-devel URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/ -~--~~~~--~~--~--~---
[sage-devel] Re: spkg refactoring and development model
On 5/31/07, Brian Granger [EMAIL PROTECTED] wrote: source. This doesn't answer the question of how to maintain those patches, that don't get sent back upstream. This definitely needs to be addressed. If I understand correctly, that's what mercurial queues are for: http://hgbook.red-bean.com/hgbookch12.html Cheers, f --~--~-~--~~~---~--~~ To post to this group, send email to sage-devel@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sage-devel URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/ -~--~~~~--~~--~--~---
[sage-devel] Re: spkg refactoring and development model
This looks very interesting, I hadn't seen this before. I need to learn more about hg for sure. Would this approach require having the source to the upstream project in an hg repo? On 5/31/07, Fernando Perez [EMAIL PROTECTED] wrote: On 5/31/07, Brian Granger [EMAIL PROTECTED] wrote: source. This doesn't answer the question of how to maintain those patches, that don't get sent back upstream. This definitely needs to be addressed. If I understand correctly, that's what mercurial queues are for: http://hgbook.red-bean.com/hgbookch12.html Cheers, f --~--~-~--~~~---~--~~ To post to this group, send email to sage-devel@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sage-devel URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/ -~--~~~~--~~--~--~---
