Re: [sage-devel] Re: RFC: New Build/Packaging System
Also, I just merged Volker's pull request in to <#> master. You shouldn't need to use his hashdist branch. A On Sat, Jun 21, 2014 at 7:47 PM, Aron Ahmadia wrote: > Wow, this is really impressive work! > > A > > > On Sat, Jun 21, 2014 at 7:42 PM, Volker Braun > wrote: > >> PS: I just remembered that this is using file globs, so you need hashdist >> from this branch: https://github.com/vbraun/hashdist/tree/files_glob. >> Though it should be merged upstream soon. >> >> >> >> On Saturday, June 21, 2014 11:22:26 PM UTC+1, Volker Braun wrote: >>> >>> I've made a proof-of-concept conversion of the Sage packaging system to >>> hashdist: >>> >>> https://github.com/vbraun/sagestack >>> >>> Obviously its not working, but you can build third-party packages and >>> maybe play around with hashdist. >>> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "sage-devel" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/sage-devel/52VPaADq0pU/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> sage-devel+unsubscr...@googlegroups.com. >> To post to this group, send email to sage-devel@googlegroups.com. >> Visit this group at http://groups.google.com/group/sage-devel. >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: RFC: New Build/Packaging System
Wow, this is really impressive work! A On Sat, Jun 21, 2014 at 7:42 PM, Volker Braun wrote: > PS: I just remembered that this is using file globs, so you need hashdist > from this branch: https://github.com/vbraun/hashdist/tree/files_glob. > Though it should be merged upstream soon. > > > > On Saturday, June 21, 2014 11:22:26 PM UTC+1, Volker Braun wrote: >> >> I've made a proof-of-concept conversion of the Sage packaging system to >> hashdist: >> >> https://github.com/vbraun/sagestack >> >> Obviously its not working, but you can build third-party packages and >> maybe play around with hashdist. >> > -- > You received this message because you are subscribed to a topic in the > Google Groups "sage-devel" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/sage-devel/52VPaADq0pU/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > sage-devel+unsubscr...@googlegroups.com. > To post to this group, send email to sage-devel@googlegroups.com. > Visit this group at http://groups.google.com/group/sage-devel. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
[sage-devel] Re: RFC: New Build/Packaging System
PS: I just remembered that this is using file globs, so you need hashdist from this branch: https://github.com/vbraun/hashdist/tree/files_glob. Though it should be merged upstream soon. On Saturday, June 21, 2014 11:22:26 PM UTC+1, Volker Braun wrote: > > I've made a proof-of-concept conversion of the Sage packaging system to > hashdist: > > https://github.com/vbraun/sagestack > > Obviously its not working, but you can build third-party packages and > maybe play around with hashdist. > -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
[sage-devel] Re: RFC: New Build/Packaging System
I've made a proof-of-concept conversion of the Sage packaging system to hashdist: https://github.com/vbraun/sagestack Obviously its not working, but you can build third-party packages and maybe play around with hashdist. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: RFC: New Build/Packaging System
On Tue, Jun 17, 2014 at 4:33 PM, François Bissey wrote: > I was available 55mn in but just the two of us wouldn't have done much without > someone from hashdist. > I would have liked to be present because I several interests in this. > Next time tomorrow is not possible for me as I will be in a meeting. > But generally that kind of time means morning in my time zone. > Of course if you put it on Friday it will be Saturday morning for me and that > will suck big time. > Chris Kees gave a 45 minute talk on hashdist today at Sage Days, which I video'd. I uploaded the video to youtube here: http://youtu.be/no4MyXn4Uik It's still processing right now. > Francois > > On Tue, 17 Jun 2014 16:25:08 Volker Braun wrote: >> Since nobody showed up, maybe you can suggest when? Do you have time during >> the week? >> >> cc to hashdist list in case somebody hasn't heard of this thread yet ;-) >> >> On Tuesday, June 17, 2014 5:25:23 PM UTC+1, Volker Braun wrote: >> > Good idea, whats a good time? >> > >> > I'll be online today at 23:00 BST UTC = 22:00 UTC = 15:00PDT. >> > >> > https://plus.google.com/events/cff6ldm40v1skgq5eeslmhg4j5g >> > >> > On Tuesday, June 17, 2014 4:30:18 PM UTC+1, dagss wrote: >> >> If you are interested in cooperating perhaps the quickest way to move >> >> forward is a Google Hangout session; I think a lot of this is really >> >> about >> >> what software architecture would fit the package manager of Sage, and >> >> that >> >> all the smaller issues discussed so far in the thread is more on the >> >> background noise level (as they are things easily changed about >> >> Hashdist), >> >> and it may be difficult to kick the discussion to the level we need in an >> >> email thread. > > -- > You received this message because you are subscribed to the Google Groups > "sage-devel" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to sage-devel+unsubscr...@googlegroups.com. > To post to this group, send email to sage-devel@googlegroups.com. > Visit this group at http://groups.google.com/group/sage-devel. > For more options, visit https://groups.google.com/d/optout. -- William Stein Professor of Mathematics University of Washington http://wstein.org -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: RFC: New Build/Packaging System
I was available 55mn in but just the two of us wouldn't have done much without someone from hashdist. I would have liked to be present because I several interests in this. Next time tomorrow is not possible for me as I will be in a meeting. But generally that kind of time means morning in my time zone. Of course if you put it on Friday it will be Saturday morning for me and that will suck big time. Francois On Tue, 17 Jun 2014 16:25:08 Volker Braun wrote: > Since nobody showed up, maybe you can suggest when? Do you have time during > the week? > > cc to hashdist list in case somebody hasn't heard of this thread yet ;-) > > On Tuesday, June 17, 2014 5:25:23 PM UTC+1, Volker Braun wrote: > > Good idea, whats a good time? > > > > I'll be online today at 23:00 BST UTC = 22:00 UTC = 15:00PDT. > > > > https://plus.google.com/events/cff6ldm40v1skgq5eeslmhg4j5g > > > > On Tuesday, June 17, 2014 4:30:18 PM UTC+1, dagss wrote: > >> If you are interested in cooperating perhaps the quickest way to move > >> forward is a Google Hangout session; I think a lot of this is really > >> about > >> what software architecture would fit the package manager of Sage, and > >> that > >> all the smaller issues discussed so far in the thread is more on the > >> background noise level (as they are things easily changed about > >> Hashdist), > >> and it may be difficult to kick the discussion to the level we need in an > >> email thread. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
[sage-devel] Re: RFC: New Build/Packaging System
Since nobody showed up, maybe you can suggest when? Do you have time during the week? cc to hashdist list in case somebody hasn't heard of this thread yet ;-) On Tuesday, June 17, 2014 5:25:23 PM UTC+1, Volker Braun wrote: > > Good idea, whats a good time? > > I'll be online today at 23:00 BST UTC = 22:00 UTC = 15:00PDT. > > https://plus.google.com/events/cff6ldm40v1skgq5eeslmhg4j5g > > > On Tuesday, June 17, 2014 4:30:18 PM UTC+1, dagss wrote: >> >> If you are interested in cooperating perhaps the quickest way to move >> forward is a Google Hangout session; I think a lot of this is really about >> what software architecture would fit the package manager of Sage, and that >> all the smaller issues discussed so far in the thread is more on the >> background noise level (as they are things easily changed about Hashdist), >> and it may be difficult to kick the discussion to the level we need in an >> email thread. >> > > > -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
[sage-devel] Re: RFC: New Build/Packaging System
Good idea, whats a good time? I'll be online today at 23:00 BST UTC = 22:00 UTC = 15:00PDT. https://plus.google.com/events/cff6ldm40v1skgq5eeslmhg4j5g On Tuesday, June 17, 2014 4:30:18 PM UTC+1, dagss wrote: > > If you are interested in cooperating perhaps the quickest way to move > forward is a Google Hangout session; I think a lot of this is really about > what software architecture would fit the package manager of Sage, and that > all the smaller issues discussed so far in the thread is more on the > background noise level (as they are things easily changed about Hashdist), > and it may be difficult to kick the discussion to the level we need in an > email thread. > -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: RFC: New Build/Packaging System
Just a general +1 to Dag and Ondrej's comments. I should take the blame (and responsibility) for places where documentation or testing is weak. hashdist itself has quite good coverage in its core, and we've been running nightly builds of the Proteus stack on a number of operating systems with it for the last 6 months (self-issued certificate, sorry): https://proteus.usace.army.mil/buildbot/waterfall A -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
[sage-devel] Re: RFC: New Build/Packaging System
On Tuesday, June 17, 2014 4:33:24 PM UTC+2, Volker Braun wrote: > I've spent some time looking at hashdist which is probably the closest to > what we need, but I don't think its the way to go for us right now. First, > Sage depends on the LD_LIBRARY_PATH hack on too many places. Before that is > fixed its hard to do anything with a real package management system. At the > same time, hashdist isn't ready. Its version 0.2, which is incompatible to > 0.1, there is no documentation for writing builder plugins / API. And much > of the source is, let's say, lightly documented with doctests being almost > completely absent. And no parallel build support, ... But in the future we > should definitely reconsider that, when these issues have been addressed on > both sides. > There's quite a lot of nosetests and everything was definitely developed by unit tests...I'd argue that doctests vs. nosetests depends a little bit on taste and a lot of what kind of code (and how complicated test fixtures) you need. I won't contest that code documentation is light. That's mainly my fault, and mainly because Hashdist was (perhaps still is) an experiment -- it doesn't make so much sense to document things thoroughly if they may change at any moment, and the number of coders touching the code are only a couple. This must definitely change. > > What I don't like in hashdist is that the packages don't have a clear > backing as Python classes in the code, which is related to not being able > to customize the heck out of it. There is the school of thought that says > you shouldn't be allowed to do anything that doesn't lead to provably > correct hash->build maps, and everything must be stateless functional. > Thats nice if you want a package manager for end users only, but that is > IMHO not good > enough for development purposes. I'd rather have to do an occasional "make > clean" than an extra 10k file ops every time I test a change. > Fundamentally, what does a package define? Its how to get the source code, > the version of the source code, and how to turn the source into a binary. > Right now, hashdist packages only have a hook into the third part. Why > can't we override how hashing works, or where to get the source from? Yes > that might allow you to break the promise of provably correct builds, but > we are all adults here. As long as you build stuff in my home directory I > ought to be able to break anything I want. > One of the reasons Hashdist came about was actually because "we're all adults" and to remove the strictness of Nix. Perhaps it's still too strict, but at least we said the exact same thing about Nix at one point :-) A lot of what you say is fully correct, and is simply an effect of Hashdist still being in development and still being developed. Many of the things you asked for are almost there (e.g., package builds already happen breadth-first and turning that into parallel is a mostly trivial matter). Others may simply be that we didn't think about them, and I think you'll find we're very open to somebody coming in and change how things are done. We are definitely not purists. You can definitely build a stack relying solely on LD_LIBRARY_PATH. And so on. We very much have the same goal here. What you have to ask yourself is: Will starting from scratch get you where you want faster than building on and improving Hashdist. Either way, there's work to be done. If you are interested in cooperating perhaps the quickest way to move forward is a Google Hangout session; I think a lot of this is really about what software architecture would fit the package manager of Sage, and that all the smaller issues discussed so far in the thread is more on the background noise level (as they are things easily changed about Hashdist), and it may be difficult to kick the discussion to the level we need in an email thread. Dag Sverre -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: RFC: New Build/Packaging System
Hi Volker, Thanks for considering hashdist. Few comments: On Tue, Jun 17, 2014 at 8:33 AM, Volker Braun wrote: > I've spent some time looking at hashdist which is probably the closest to > what we need, but I don't think its the way to go for us right now. First, > Sage depends on the LD_LIBRARY_PATH hack on too many places. Before that is > fixed its hard to do anything with a real package management system. We currently use RPATH, and plan to move to $ORIGIN, so that the build is relocatable. https://github.com/hashdist/hashdist/issues/13 But $ORIGIN is not implemented yet. This is another show stopper, as Sage needs to be relocatable. > At the > same time, hashdist isn't ready. Its version 0.2, which is incompatible to > 0.1, Yes, it is important to get the design right, and sometimes it is not backwards compatible. That will obviously improve. > there is no documentation for writing builder plugins / API. I created an issue for it: https://github.com/hashdist/hashdist/issues/247 But can you please clarify what you mean by "builder plugin"? > And much > of the source is, let's say, lightly documented with doctests being almost > completely absent. https://github.com/hashdist/hashdist/issues/248 > And no parallel build support, https://github.com/hashdist/hashdist/issues/246 > ... But in the future we > should definitely reconsider that, when these issues have been addressed on > both sides. Thank you for giving us feedback. I have marked all these issues with "sage", so that you can conveniently view them at one place: https://github.com/hashdist/hashdist/issues?labels=sage Is there anything else missing? > > What I don't like in hashdist is that the packages don't have a clear > backing as Python classes in the code, which is related to not being able to > customize the heck out of it. There is the school of thought that says you > shouldn't be allowed to do anything that doesn't lead to provably correct > hash->build maps, and everything must be stateless functional. Thats nice if > you want a package manager for end users only, but that is IMHO not good > enough for development purposes. I'd rather have to do an occasional "make > clean" than an extra 10k file ops every time I test a change. Fundamentally, > what does a package define? Its how to get the source code, the version of > the source code, and how to turn the source into a binary. Right now, > hashdist packages only have a hook into the third part. Why can't we > override how hashing works, or where to get the source from? Yes that might > allow you to break the promise of provably correct builds, but we are all > adults here. As long as you build stuff in my home directory I ought to be > able to break anything I want. So what you are missing is that you want to modify one line in openssl, and you know that it won't change anything in the other 50+ packages that depend on it (e.g. Python, numpy, scipy, ...), so you want to be able to simply build openssl, but keep the hash, so that it does not trigger a rebuild of the rest of the stack? Or also something else? > > Not only is package == builder Python object a clear API to override all > aspects of the package build, you then can't help yourself but load them all > into an IPython session. I found that to be a really nice mechanism to debug > and explore the whole packaging: > > > $ ./build/manager/sage-pkg shell > Python 2.7.5 (default, Feb 19 2014, 13:47:28) > Type "copyright", "credits" or "license" for more information. > > IPython 0.13.2 -- An enhanced Interactive Python. > ? -> Introduction and overview of IPython's features. > %quickref -> Quick reference. > help -> Python's own help system. > object? -> Details about 'object', use 'object??' for extra details. > Loaded packages: atlas, autotools, boehm_gc, boost_cropped, bzip2, cddlib, > cephes, cliquer, configure, conway_polynomials, cryptominisat, cvxopt, > cython, database_cremona_ellcurve, database_gap, database_stein_watkins, > database_stein_watkins_mini, database_symbolic_data, dateutil, docutils, > dot2tex, ecl, eclib, ecm, elliptic_curves, fflas_ffpack, flint, flintqs, > freetype, gap, gap_packages, gcc, gd, gdmodule, genus2reduction, gf2x, gfan, > git, git_trac, givaro, glpk, gmp, graphs, gsl, iconv, iml, ipython, jinja2, > jmol, lcalc, libfplll, libgap, libm4ri, libm4rie, libogg, libpng, libtheora, > linbox, lrcalc, matplotlib, maxima, mcqd, mercurial, mpc, mpfi, mpfr, mpir, > mpmath, ncurses, networkx, ntl, numpy, openssl, palp, pari, pari_galdata, > pari_seadata_small, patch, pexpect, pillow, pkgconf, pkgconfig, polybori, > polytopes_db, ppl, pycrypto, pygments, pynac, pyparsing, python, r, > ratpoints, readline, rpy2, rubiks, sage_c_library, sage_mode, sage_noarch, > sage_python_library, sagenb, sagetex, scipy, scons, setuptools, singular, > six, sphinx, sqlalchemy, sqlite, symmetrica, sympow, sympy, tachyon, > termcap, topcom, tornado, zlib, zn_poly > > In [1]: ppl.config > Out
[sage-devel] Re: RFC: New Build/Packaging System
I've spent some time looking at hashdist which is probably the closest to what we need, but I don't think its the way to go for us right now. First, Sage depends on the LD_LIBRARY_PATH hack on too many places. Before that is fixed its hard to do anything with a real package management system. At the same time, hashdist isn't ready. Its version 0.2, which is incompatible to 0.1, there is no documentation for writing builder plugins / API. And much of the source is, let's say, lightly documented with doctests being almost completely absent. And no parallel build support, ... But in the future we should definitely reconsider that, when these issues have been addressed on both sides. What I don't like in hashdist is that the packages don't have a clear backing as Python classes in the code, which is related to not being able to customize the heck out of it. There is the school of thought that says you shouldn't be allowed to do anything that doesn't lead to provably correct hash->build maps, and everything must be stateless functional. Thats nice if you want a package manager for end users only, but that is IMHO not good enough for development purposes. I'd rather have to do an occasional "make clean" than an extra 10k file ops every time I test a change. Fundamentally, what does a package define? Its how to get the source code, the version of the source code, and how to turn the source into a binary. Right now, hashdist packages only have a hook into the third part. Why can't we override how hashing works, or where to get the source from? Yes that might allow you to break the promise of provably correct builds, but we are all adults here. As long as you build stuff in my home directory I ought to be able to break anything I want. Not only is package == builder Python object a clear API to override all aspects of the package build, you then can't help yourself but load them all into an IPython session. I found that to be a really nice mechanism to debug and explore the whole packaging: $ ./build/manager/sage-pkg shell Python 2.7.5 (default, Feb 19 2014, 13:47:28) Type "copyright", "credits" or "license" for more information. IPython 0.13.2 -- An enhanced Interactive Python. ? -> Introduction and overview of IPython's features. %quickref -> Quick reference. help -> Python's own help system. object? -> Details about 'object', use 'object??' for extra details. Loaded packages: atlas, autotools, boehm_gc, boost_cropped, bzip2, cddlib, cephes, cliquer, configure, conway_polynomials, cryptominisat, cvxopt, cython, database_cremona_ellcurve, database_gap, database_stein_watkins, database_stein_watkins_mini, database_symbolic_data, dateutil, docutils, dot2tex, ecl, eclib, ecm, elliptic_curves, fflas_ffpack, flint, flintqs, freetype, gap, gap_packages, gcc, gd, gdmodule, genus2reduction, gf2x, gfan, git, git_trac, givaro, glpk, gmp, graphs, gsl, iconv, iml, ipython, jinja2, jmol, lcalc, libfplll, libgap, libm4ri, libm4rie, libogg, libpng, libtheora, linbox, lrcalc, matplotlib, maxima, mcqd, mercurial, mpc, mpfi, mpfr, mpir, mpmath, ncurses, networkx, ntl, numpy, openssl, palp, pari, pari_galdata, pari_seadata_small, patch, pexpect, pillow, pkgconf, pkgconfig, polybori, polytopes_db, ppl, pycrypto, pygments, pynac, pyparsing, python, r, ratpoints, readline, rpy2, rubiks, sage_c_library, sage_mode, sage_noarch, sage_python_library, sagenb, sagetex, scipy, scons, setuptools, singular, six, sphinx, sqlalchemy, sqlite, symmetrica, sympow, sympy, tachyon, termcap, topcom, tornado, zlib, zn_poly In [1]: ppl.config Out[1]: Configuration: - config.builder.check_script = spkg-check - config.builder.install_script = spkg-install - config.builder.type = SpkgInstallScript - config.category = standard - config.depends.hard = ['mpir', 'glpk'] - config.name = ppl - config.source.tarball.name = ppl-1.1.tar.bz2 - config.source.tarball.sha1 = f76fbc2d374170771fed030b79a5ffac08d907bf - config.source.version = 1.1 In [2]: type(ppl) Out[2]: sage_pkg.package.spkg_install.SpkgInstallScript In [3]: type(ppl).mro() Out[3]: [sage_pkg.package.spkg_install.SpkgInstallScript, sage_pkg.package.sage_environment_mixin.SageEnvironmentMixin, sage_pkg.package.sage_mirror_mixin.SageMirrorMixin, sage_pkg.package.base.PackageBase, object] In [4]: ppl.version_stamp Out[4]: 'c2b217a0d2c918132a0af6189d5f9a74bce4c41f' In [5]: ppl.get_all_dependencies() Out[5]: (mpir, glpk) In [6]: ppl.get_all_dependencies()[0] is mpir Out[6]: True In [7]: ppl.download() http://www.sagemath.org/packages/upstream/ppl/ppl-1.1.tar.bz2 [] In [8]: ppl.unpack() In [10]: ppl.prepare() In [11]: ppl.install() ppl Host system: Linux localhost.localdomain 3.14.4-200.fc20.x86_64 #1 SMP Tue May 13 13:51:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
Re: [sage-devel] Re: RFC: New Build/Packaging System
So how about naming them compile/hard/soft/test, is that clearer? I don't think we need separate (only) run-time dependencies... -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] Re: RFC: New Build/Packaging System
On Sun, 15 Jun 2014 15:29:58 Volker Braun wrote: > On Sunday, June 15, 2014 10:59:57 PM UTC+1, leif wrote: > > > * Git-aware: use SHA1 hashes instead of timestamps for dependency > > > calculations > > > > ? Hashs of what exactly? Modification / installation time of course > > matters... > > No, modification times precisely does not matter. Git does not record > timestamps, and switching branches therefore causes needless recompilation > even if you switch back to your the branch! > > > Not sure what "hard" dependencies are supposed to mean -- above you talk > > about hard/soft *runtime* dependencies, while at least some of the > > "hard" dependencies listed above are definitely *build* time > > dependencies (Python, freetype, libpng...). > > Runtime dependencies are of course required to build. But not the other way > round, you don't need to recompile everything because you updated "patch". > No. One example: scipy is a runtime dependency of sage but not a build time. You can build sage without scipy being installed. Same with maxima. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
[sage-devel] Re: RFC: New Build/Packaging System
On Sunday, June 15, 2014 10:59:57 PM UTC+1, leif wrote: > > > * Git-aware: use SHA1 hashes instead of timestamps for dependency > > calculations > > ? Hashs of what exactly? Modification / installation time of course > matters... > No, modification times precisely does not matter. Git does not record timestamps, and switching branches therefore causes needless recompilation even if you switch back to your the branch! > Not sure what "hard" dependencies are supposed to mean -- above you talk > about hard/soft *runtime* dependencies, while at least some of the > "hard" dependencies listed above are definitely *build* time > dependencies (Python, freetype, libpng...). > Runtime dependencies are of course required to build. But not the other way round, you don't need to recompile everything because you updated "patch". What about versions in dependencies (i.e., prerequisites)? > The version is whatever version is in Sage (Patch level) version of the Sage package / YAML file? > There is no need for patch levels. The dependency computation uses the SHA1 of the git tree object that holds the package dir as the true version. The tarball version is just for the UI. > I'd also add a couple of things from SPKG.txt: > Yes, but that can wait for a followup. > shell IPython shell :-) Another (useless?) dependency? Not a dependency, just an optional command line interface if IPython is installed in the OS Python. Hmmm, 'pkg-upgrade-v1'? Debug, ignore. Do we / can we keep package info files for various versions? There is always "git checkout" -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
[sage-devel] Re: RFC: New Build/Packaging System
Volker Braun wrote:
This is a RFC for new packaging system for "sage-the-distribution". I've
already talked about this with a few of you at the last sage days, but
finally it managed to do something about it. The goal is to be:
* Git-aware: use SHA1 hashes instead of timestamps for dependency
calculations
? Hashs of what exactly? Modification / installation time of course
matters...
* Unified machine-readable package configuration using YAML
* Dependency handling also for optional packages
* Distinction between different types of dependencies: build time,
runtime hard/soft, testing.
* Modular, allowing for easy experimentation with per-package backends
* written in pure Python without any dependencies
* doctested for Python 2.6, 2.7, 3.3, and 3.4.
So we're finally making (some) Python a prerequisite -- fine, provided
all (or at least most) packages get built with the system's Python,
i.e., do not (artificially) depend on Sage's Python.
Having all package configuration data accessible in machine-readable
form is IMHO the key to managing complexity. There are various
possibilities for the file format, but in the end I think YAML is the
best choice. It integrates very well with Python, is easy to read/write
for humans, and if things go wrong the parser can pin-point the error
very well. Downside is that it is not in the Python standard library,
but then I don't think we should that let us dictate the file format.
We'll just include the Python-only part of PyYAML as a fallback if the
OS Python does not have it.
The entire package configuration will be in a file
SAGE_ROOT/build/pkgs//package.yaml, for example
name:
matplotlib
category:
standard
source:
version:
1.3.1
tarball:
name: matplotlib-{source.version}.tar.gz
sha1: f340378c43c4c3f6219ef9fd84af4ebbe18f8feb
builder:
type:SpkgInstallScript
install_script: spkg-install
depends:
build:
- pkgconf
- setuptools
hard:
- python
- numpy
- freetype
- libpng
- gdmodule
- dateutil
- pyparsing
Not sure what "hard" dependencies are supposed to mean -- above you talk
about hard/soft *runtime* dependencies, while at least some of the
"hard" dependencies listed above are definitely *build* time
dependencies (Python, freetype, libpng...).
What about versions in dependencies (i.e., prerequisites)?
(Patch level) version of the Sage package / YAML file?
I'd also add a couple of things from SPKG.txt:
- perhaps at least short (one-line) description
- copyright / license information
- URL of upstream tarball if appropriate
- upstream contact and website
- author(s)
- perhaps (Sage) package maintainers
- list of patches?
I've been working on an implementation, which you can find at
http://trac.sagemath.org/ticket/16483. It is not feature-complete, but
can already build the standard packages in dependency order.
The package manager lives in SAGE_ROOT/build/manager. Eventually, all
build-related sage command line switches should just call it:
$ ./build/manager/sage-pkg help
usage: sage-pkg [-h] [--config CONFIG] [--log LOG]
{info,shell,list,pkg-upgrade-v1,help,download,unpack,prepare,configure,compile,check,install,build,get}
...
The Sage Package Manager
positional arguments:
{info,shell,list,pkg-upgrade-v1,help,download,unpack,prepare,configure,compile,check,install,build,get}
infoPrint information about package
shell IPython shell
:-) Another (useless?) dependency?
listList all packages
pkg-upgrade-v1 Upgrade package descriptions
helpGet help
downloadBuild package up to the "download" step
unpack Build package up to the "unpack" step
prepare Build package up to the "prepare" step
configure Build package up to the "configure" step
compile Build package up to the "compile" step
check Build package up to the "check" step
install Build package and install
build Build everything
get Download tarball/spkg/file
optional arguments:
-h, --helpshow this help message and exit
--config CONFIG Builder configuration file
--log LOG One of [DEBUG, INFO, ERROR, WARNING, CRITICAL]
Hmmm, 'pkg-upgrade-v1'?
Not immediately clear what the difference between 'get' and 'download'
is, nor what 'build' here is supposed to mean.
$ ./build/manager/sage-pkg info matplotlib
Configuration:
- config.builder.install_script = spkg-install
- config.builder.type = SpkgInstallScript
- config.category = standard
- config.depends.build = 'pkgconf', 'setuptools']
- config.depends.hard = ['python', 'numpy', '
