Hi Volker, Thanks for considering hashdist. Few comments:
On Tue, Jun 17, 2014 at 8:33 AM, Volker Braun <vbraun.n...@gmail.com> wrote: > I've spent some time looking at hashdist which is probably the closest to > what we need, but I don't think its the way to go for us right now. First, > Sage depends on the LD_LIBRARY_PATH hack on too many places. Before that is > fixed its hard to do anything with a real package management system. We currently use RPATH, and plan to move to $ORIGIN, so that the build is relocatable. https://github.com/hashdist/hashdist/issues/13 But $ORIGIN is not implemented yet. This is another show stopper, as Sage needs to be relocatable. > At the > same time, hashdist isn't ready. Its version 0.2, which is incompatible to > 0.1, Yes, it is important to get the design right, and sometimes it is not backwards compatible. That will obviously improve. > there is no documentation for writing builder plugins / API. I created an issue for it: https://github.com/hashdist/hashdist/issues/247 But can you please clarify what you mean by "builder plugin"? > And much > of the source is, let's say, lightly documented with doctests being almost > completely absent. https://github.com/hashdist/hashdist/issues/248 > And no parallel build support, https://github.com/hashdist/hashdist/issues/246 > ... But in the future we > should definitely reconsider that, when these issues have been addressed on > both sides. Thank you for giving us feedback. I have marked all these issues with "sage", so that you can conveniently view them at one place: https://github.com/hashdist/hashdist/issues?labels=sage Is there anything else missing? > > What I don't like in hashdist is that the packages don't have a clear > backing as Python classes in the code, which is related to not being able to > customize the heck out of it. There is the school of thought that says you > shouldn't be allowed to do anything that doesn't lead to provably correct > hash->build maps, and everything must be stateless functional. Thats nice if > you want a package manager for end users only, but that is IMHO not good > enough for development purposes. I'd rather have to do an occasional "make > clean" than an extra 10k file ops every time I test a change. Fundamentally, > what does a package define? Its how to get the source code, the version of > the source code, and how to turn the source into a binary. Right now, > hashdist packages only have a hook into the third part. Why can't we > override how hashing works, or where to get the source from? Yes that might > allow you to break the promise of provably correct builds, but we are all > adults here. As long as you build stuff in my home directory I ought to be > able to break anything I want. So what you are missing is that you want to modify one line in openssl, and you know that it won't change anything in the other 50+ packages that depend on it (e.g. Python, numpy, scipy, ...), so you want to be able to simply build openssl, but keep the hash, so that it does not trigger a rebuild of the rest of the stack? Or also something else? > > Not only is package == builder Python object a clear API to override all > aspects of the package build, you then can't help yourself but load them all > into an IPython session. I found that to be a really nice mechanism to debug > and explore the whole packaging: > > > $ ./build/manager/sage-pkg shell > Python 2.7.5 (default, Feb 19 2014, 13:47:28) > Type "copyright", "credits" or "license" for more information. > > IPython 0.13.2 -- An enhanced Interactive Python. > ? -> Introduction and overview of IPython's features. > %quickref -> Quick reference. > help -> Python's own help system. > object? -> Details about 'object', use 'object??' for extra details. > Loaded packages: atlas, autotools, boehm_gc, boost_cropped, bzip2, cddlib, > cephes, cliquer, configure, conway_polynomials, cryptominisat, cvxopt, > cython, database_cremona_ellcurve, database_gap, database_stein_watkins, > database_stein_watkins_mini, database_symbolic_data, dateutil, docutils, > dot2tex, ecl, eclib, ecm, elliptic_curves, fflas_ffpack, flint, flintqs, > freetype, gap, gap_packages, gcc, gd, gdmodule, genus2reduction, gf2x, gfan, > git, git_trac, givaro, glpk, gmp, graphs, gsl, iconv, iml, ipython, jinja2, > jmol, lcalc, libfplll, libgap, libm4ri, libm4rie, libogg, libpng, libtheora, > linbox, lrcalc, matplotlib, maxima, mcqd, mercurial, mpc, mpfi, mpfr, mpir, > mpmath, ncurses, networkx, ntl, numpy, openssl, palp, pari, pari_galdata, > pari_seadata_small, patch, pexpect, pillow, pkgconf, pkgconfig, polybori, > polytopes_db, ppl, pycrypto, pygments, pynac, pyparsing, python, r, > ratpoints, readline, rpy2, rubiks, sage_c_library, sage_mode, sage_noarch, > sage_python_library, sagenb, sagetex, scipy, scons, setuptools, singular, > six, sphinx, sqlalchemy, sqlite, symmetrica, sympow, sympy, tachyon, > termcap, topcom, tornado, zlib, zn_poly > > In [1]: ppl.config > Out[1]: > Configuration: > - config.builder.check_script = spkg-check > - config.builder.install_script = spkg-install > - config.builder.type = SpkgInstallScript > - config.category = standard > - config.depends.hard = ['mpir', 'glpk'] > - config.name = ppl > - config.source.tarball.name = ppl-1.1.tar.bz2 > - config.source.tarball.sha1 = f76fbc2d374170771fed030b79a5ffac08d907bf > - config.source.version = 1.1 > > In [2]: type(ppl) > Out[2]: sage_pkg.package.spkg_install.SpkgInstallScript > > In [3]: type(ppl).mro() > Out[3]: > [sage_pkg.package.spkg_install.SpkgInstallScript, > sage_pkg.package.sage_environment_mixin.SageEnvironmentMixin, > sage_pkg.package.sage_mirror_mixin.SageMirrorMixin, > sage_pkg.package.base.PackageBase, > object] > > In [4]: ppl.version_stamp > Out[4]: 'c2b217a0d2c918132a0af6189d5f9a74bce4c41f' > > In [5]: ppl.get_all_dependencies() > Out[5]: (mpir, glpk) > > In [6]: ppl.get_all_dependencies()[0] is mpir > Out[6]: True > > In [7]: ppl.download() > http://www.sagemath.org/packages/upstream/ppl/ppl-1.1.tar.bz2 > [............................................................] > > In [8]: ppl.unpack() > > In [10]: ppl.prepare() > > In [11]: ppl.install() > ==================================================== > ppl > **************************************************** > Host system: > Linux localhost.localdomain 3.14.4-200.fc20.x86_64 #1 SMP Tue May 13 > 13:51:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux > **************************************************** > C compiler: gcc > C compiler version: > ... > make[1]: Leaving directory > `/home/vbraun/Code/sage/local/var/tmp/sage-build/ppl/ppl-1.1/m4' > Successfully installed ppl > Finished installing ppl.spkg > > In [12]: That's a nice feature. In general I agree with you, it's always better to write something that exactly solves the problem that you have at hand, than using a more general package. That's what Sage did, and that's what all the HPC distributions did etc. It fixes the particular problem that the team was facing, but usually it's not very applicable for other people. With hashdist, we are trying to not only fix this for us, but also make it so that it can be useful by other people. Essentially, there is a lot of common things, that all of us need, and then there are always project specific features, so we want to make hashdist work in a way, so that you can take it, and build something specific on top of it. So we appreciate all the feedback, that you gave us. If you remember anything else for which I didn't create an issue, please let us know. Ondrej -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
