Re: [sage-devel] RFC: New Build/Packaging System
Thanks Volker, Aron, and Ondrej. I think I understand most of Volker's questions about HashDist a lot better now. One thing I wanted to clarify is that HashDist uses the abstract idea of functional package management from Nix, but HashDist is not built on the Nix package manager or a rewrite/refactorization of it. In particular, the build recipes for packages are highly customizable and flexible. We erred on the side of pragmatism in several areas of the design due to our experience building complex stacks on HPC platforms. Another point I wanted to underscore is that some of the functionality like parallel builds across packages and integrated, efficient development workflows have been part of the design since the early stages of HashDist development. There's just a limit to how fast we can implement features because we all have primary research objectives in other areas. -Chris On Mon, Jun 16, 2014 at 2:19 PM, Ondřej Čertík wrote: > On Mon, Jun 16, 2014 at 1:59 PM, Volker Braun > wrote: > > On Monday, June 16, 2014 8:40:34 PM UTC+1, Ondřej Čertík wrote: > >> > >> Volker, my understanding is that this would be useful for developing a > >> package, to be able to quickly > >> run a build, without committing. But for the end user, you always want > >> to build from some commit. > > > > > > Yes, I agree. But our primary aim here is reduce development turnaround > > time. > > Definitely. I just wanted to clarify this. I've wanted this feature > for a long time too, when developing. > > Ondrej > > -- > You received this message because you are subscribed to the Google Groups > "sage-devel" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to sage-devel+unsubscr...@googlegroups.com. > To post to this group, send email to sage-devel@googlegroups.com. > Visit this group at http://groups.google.com/group/sage-devel. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] RFC: New Build/Packaging System
On Mon, Jun 16, 2014 at 1:59 PM, Volker Braun wrote: > On Monday, June 16, 2014 8:40:34 PM UTC+1, Ondřej Čertík wrote: >> >> Volker, my understanding is that this would be useful for developing a >> package, to be able to quickly >> run a build, without committing. But for the end user, you always want >> to build from some commit. > > > Yes, I agree. But our primary aim here is reduce development turnaround > time. Definitely. I just wanted to clarify this. I've wanted this feature for a long time too, when developing. Ondrej -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] RFC: New Build/Packaging System
On Monday, June 16, 2014 8:40:34 PM UTC+1, Ondřej Čertík wrote: > > Volker, my understanding is that this would be useful for developing a > package, to be able to quickly > run a build, without committing. But for the end user, you always want > to build from some commit. Yes, I agree. But our primary aim here is reduce development turnaround time. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] RFC: New Build/Packaging System
On Mon, Jun 16, 2014 at 1:09 PM, Aron Ahmadia wrote: > > On Mon, Jun 16, 2014 at 3:07 PM, Volker Braun wrote: >> >> In particular its not possible to build from the already-existing git >> repo? I don't want to have to specify the version of sage-the-library, I >> just want to build it out of the source tree. The version is the git tree >> sha1 if the tree is clean, and random / file checksum if not clean. At the >> same time there we have at least one package that depends on Sage being able >> to start, so we need to integrate sage in our package management. > > > I'm working on a feature similar to what you're asking for this week. I'm > programmatically using the functionality in `git stash` to create a commit > of the current working tree before launching the build. It's not > implemented on anything besides napkins yet, though :) The actual command is "git stash create", which will not modify any of your source files, i.e. if you have uncommitted changes, but returns a hash of a "stash" commit, that can then be used in hashdist. Volker, my understanding is that this would be useful for developing a package, to be able to quickly run a build, without committing. But for the end user, you always want to build from some commit. I.e. if I am developing my own library, I would first put it into hashdist (with url to github), and then when developing a new version, I change the url to my local checkout, and I want be able to quickly check that it installs and works, so then I would use this feature. Ondrej -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] RFC: New Build/Packaging System
On Mon, Jun 16, 2014 at 3:07 PM, Volker Braun wrote: > In particular its not possible to build from the already-existing git > repo? I don't want to have to specify the version of sage-the-library, I > just want to build it out of the source tree. The version is the git tree > sha1 if the tree is clean, and random / file checksum if not clean. At the > same time there we have at least one package that depends on Sage being > able to start, so we need to integrate sage in our package management. I'm working on a feature similar to what you're asking for this week. I'm programmatically using the functionality in `git stash` to create a commit of the current working tree before launching the build. It's not implemented on anything besides napkins yet, though :) A -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] RFC: New Build/Packaging System
On Monday, June 16, 2014 7:32:06 PM UTC+1, Aron Ahmadia wrote: > > At this point, you'd need to depend on a commit that contained your git > tree. The full commit would be unpacked in the build, and you could do a > sub-install from there. > In particular its not possible to build from the already-existing git repo? I don't want to have to specify the version of sage-the-library, I just want to build it out of the source tree. The version is the git tree sha1 if the tree is clean, and random / file checksum if not clean. At the same time there we have at least one package that depends on Sage being able to start, so we need to integrate sage in our package management. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] RFC: New Build/Packaging System
Hi Volker, We have implemented mirror support that does what you want: https://github.com/hashdist/hashdist/blob/master/hashdist/core/source_cache.py#L740 Mirrors are specified in the hashdist config file, not profiles, but I don't think it would be hard to add support for mirrors at the profile-level. I don't see a reason to specify mirrors at the package level, which is where we tend to put the canonical source. We don't support sub-commit level Git objects, currently. At this point, you'd need to depend on a commit that contained your git tree. The full commit would be unpacked in the build, and you could do a sub-install from there. I (and the rest of the developers) would probably need to think a bit about the advantages/risks of allowing sub-commit objects to be referenced in package installs. On my first impression, it seems a little low-level for HashDist's layer of responsibility. Creating such a subtree commit object is not hard for an advanced Git developer, however... A -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] RFC: New Build/Packaging System
On Monday, June 16, 2014 7:27:54 PM UTC+1, Ondřej Čertík wrote: > > Ah, from a git repository? That's easy, you just use the git hash, see > e.g.: > sources: > - url: /nh/nest/u/ondrej/repos/ginac > key: git:edfa67d26bac695b5ef9911f3cda3ff50232e35a > I don't want to use the SHA1 of the entire repo, I only want to use a sub-directory of the repository. In other words, it would be nice to have a finer-grained control on recompilation than the entire Sage repository. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] RFC: New Build/Packaging System
On Mon, Jun 16, 2014 at 12:20 PM, Volker Braun wrote: > On Monday, June 16, 2014 7:04:51 PM UTC+1, Ondřej Čertík wrote: >> >> Yes. You just modify the url field to your own mirror. > > > No. I know that I want to download xyz.tar.gz, and I have a list of sage > mirrors ranked by speed, and I want to download from the fastest one. I don't think we support this yet. Essentially you would have some path postfix, but then you want to change the mirror at one place, and all the packages would be downloaded from this mirror? > >> > What if I want to install >> > files/directories directly from the repository? >> >> >> Could you clarify this question a bit? I am not sure what exactly you >> are asking. > > > I want to install part of Sage (say, under src/c_lib) as a "package" of the > build process. How do I do that without creating a tarball? I have a unique > hash, namely the git tree blob SHA1 for src/c_lib. Ah, from a git repository? That's easy, you just use the git hash, see e.g.: https://github.com/hashdist/hashstack/blob/master/pkgs/ginac.yaml sources: - url: git://www.ginac.de/ginac.git key: git:edfa67d26bac695b5ef9911f3cda3ff50232e35a And you can use local path as well, e.g.: sources: - url: /nh/nest/u/ondrej/repos/ginac key: git:edfa67d26bac695b5ef9911f3cda3ff50232e35a Ondrej -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] RFC: New Build/Packaging System
On Monday, June 16, 2014 7:04:51 PM UTC+1, Ondřej Čertík wrote: > > Yes. You just modify the url field to your own mirror. No. I know that I want to download xyz.tar.gz, and I have a list of sage mirrors ranked by speed, and I want to download from the fastest one. > What if I want to install > > files/directories directly from the repository? > Could you clarify this question a bit? I am not sure what exactly you > are asking. > I want to install part of Sage (say, under src/c_lib) as a "package" of the build process. How do I do that without creating a tarball? I have a unique hash, namely the git tree blob SHA1 for src/c_lib. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] RFC: New Build/Packaging System
Hi Volker,
I was also pointed to this thread. Aron answered pretty much
everything, so just a few comments:
On Mon, Jun 16, 2014 at 11:49 AM, Volker Braun wrote:
> For the record, Sage build a lot slower to build if you build packages one
> after the other.
>
> So hashdist packages can sort of change how they are built by inheriting
> ("extends:"). But can we override / extend:
> * source download (to use the Sage mirror network)
Yes. You just modify the url field to your own mirror. Assuming you
didn't modify the tarball,
you don't need to modify the hash. If you modified the tarball, then
you need to update the hash.
> * the hash computation for the source?
There is hash computation to verify the tarball, e.g.:
sources:
- url: http://archive.ipython.org/release/2.0.0/ipython-2.0.0.tar.gz
key: tar.gz:j2b4bd4nn7iorbemnd6a5mxzwbkptn2c
But the hash that comes into the build is computed from a json input
script, that contains
this source "key", then the script that builds the package, then any
dependencies etc.
> What if I want to install
> files/directories directly from the repository?
Could you clarify this question a bit? I am not sure what exactly you
are asking.
> Is there a way to skip the Nix versioned directories stuff and instead just
> install directly into the destination tree? Because thats what we do now,
So we create a profile from these versioned directories, which you can
view as a destination tree.
They contain hardlinks into versioned directories. You use the
destination tree just like a Sage tree.
> and changing the build system and 50 package build scripts at the same time
> is not really feasible.
We plan to eventually port packages that Sage uses. We've already
ported quite a few relevant packages,
e.g.:
https://github.com/hashdist/hashstack/blob/master/pkgs/linbox.yaml
https://github.com/hashdist/hashstack/blob/master/pkgs/ntl.yaml
https://github.com/hashdist/hashstack/blob/master/pkgs/flint.yaml
and more will come in the future. Essentially anyone can just send a PR:
https://github.com/hashdist/hashstack/pulls
Ondrej
P.S. We now also have a more fancy web page: https://hashdist.github.io/
--
You received this message because you are subscribed to the Google Groups
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at http://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] RFC: New Build/Packaging System
Hi Volker,
Inline-repies this time:
On Mon, Jun 16, 2014 at 1:49 PM, Volker Braun wrote:
> For the record, Sage build a lot slower to build if you build packages one
> after the other.
>
That's good to know. We're doing our best to get hashdist in shape for
generating relocatable binary builds. This should have a much bigger
impact on build times than the level of parallelism offered in compiles,
since users could specify what packages they want and then just have them
downloaded for their given operating system (assuming somebody has taken
the time to generate and cache their build dependencies somewhere public).
> So hashdist packages can sort of change how they are built by inheriting
> ("extends:"). But can we override / extend:
> * source download (to use the Sage mirror network)
>
Yes. There's a concept of a source cache where all git repositories and
tarballs are stored. You can have as many mirrors as you like. I don't
have a public one that I can show you right now. It follows the same
directory structure as the local directory source cache (which lives by
default in ~/.hashdist/src) and can be simply rsycned up to create.
> * the hash computation for the source? What if I want to install
> files/directories directly from the repository?
>
I don't understand this question.
Is there a way to skip the Nix versioned directories stuff and instead just
> install directly into the destination tree? Because thats what we do now,
> and changing the build system and 50 package build scripts at the same time
> is not really feasible.
>
The 'hit develop' command will copy built trees over into the target
profile directory. I suppose we could modify the develop command so that
it allows users to skip the build cache, but it is very convenient to have
the individual builds in their own directories for a number of reasons, and
we haven't had a request for this particular feature yet.
A
--
You received this message because you are subscribed to the Google Groups
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at http://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] RFC: New Build/Packaging System
For the record, Sage build a lot slower to build if you build packages one
after the other.
So hashdist packages can sort of change how they are built by inheriting
("extends:"). But can we override / extend:
* source download (to use the Sage mirror network)
* the hash computation for the source? What if I want to install
files/directories directly from the repository?
Is there a way to skip the Nix versioned directories stuff and instead just
install directly into the destination tree? Because thats what we do now,
and changing the build system and 50 package build scripts at the same time
is not really feasible.
--
You received this message because you are subscribed to the Google Groups
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at http://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] RFC: New Build/Packaging System
On Monday, June 16, 2014 8:44:43 AM UTC-4, Volker Braun wrote: > > Some questions from playing around with hashdist a bit: > > * How do I build packages in parallel? > * I can't build Python on Fedora 20? What are the actual hashdist > dependencies? > > On Monday, June 16, 2014 5:53:36 AM UTC+1, Chris Kees wrote: >> >> * Modular, allowing for easy experimentation with per-package backends >>> >> not sure I understand this point >> > > By which I mean: You don't have to drink the cool-aid all the way if you > don't want to.. Packages can define alternative builders, (a Python class > named by the builder.type key) that handles the package. For example there > is a: > > -- > name: > sage_c_library > > category: > standard > > source: > - src/c_lib > > builder: > type: ShellCommands > workdir:src/c_lib > commands: > - source ../bin/sage-env && scons -Q install > > depends: > compile: > - scons > hard: > - mpir > - ntl > - pari > - python > -- > Hi Volker, Chris pointed me at this thread. Thanks for taking a look at hashdist. In response to your questions: * How do I build packages in parallel? This isn't currently enabled, but it would not be hard to make an option. Right now we build up a list of packages that we would like to install, then execute parallel installation on each package. Since the dependency graphs tend to be fairly narrow for many of our longest environment builds, I suspect it's actually faster to do it this way (but have not measured). * I can't build Python on Fedora 20? What are the actual hashdist dependencies? What profile are you using and what error are you getting? The example Python profile we use assumes a few things may already be on the host system to simplify builds (and also verifies that a few of Python's optional components are installed). This is admittedly not great practice, and in the future, we'll explicitly enumerate the Python package dependencies we assume may be on the host. One thing I *don't* like doing is building my own openssl in hashdist, since crypto software should really be handled by the operating system, not an overlay package manager. I am assuming you don't have sqlite or bzip2 installed on your core Fedora 20 system. Try this install profile: python: host: false build_with: | bzip2, sqlite Regarding modular packages, hashdist defines the concept of a profile (think of it like a pip requirements.txt) as well as a package (which describes how to build a given package). Profiles can define parameters, which then modify how a package is built. Profiles can also specify alternative package providers, such as a BLAS from the system or built by the hoser. Happy to answer more questions on this mailing list if I'm not derailing your RFC. From your initial proposal, it does look like you're interested in many of the same features. -A -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] RFC: New Build/Packaging System
On Monday, June 16, 2014 1:44:43 PM UTC+1, Volker Braun wrote: > > * How do I build packages in parallel? > Just to clarify, the question is how can I build different packages at the same time. Not: run make -j 10 for each package. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] RFC: New Build/Packaging System
Some questions from playing around with hashdist a bit: * How do I build packages in parallel? * I can't build Python on Fedora 20? What are the actual hashdist dependencies? On Monday, June 16, 2014 5:53:36 AM UTC+1, Chris Kees wrote: > > * Modular, allowing for easy experimentation with per-package backends >> > not sure I understand this point > By which I mean: You don't have to drink the cool-aid all the way if you don't want to.. Packages can define alternative builders, (a Python class named by the builder.type key) that handles the package. For example there is a: -- name: sage_c_library category: standard source: - src/c_lib builder: type: ShellCommands workdir:src/c_lib commands: - source ../bin/sage-env && scons -Q install depends: compile: - scons hard: - mpir - ntl - pari - python -- -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] RFC: New Build/Packaging System
On Monday, June 16, 2014 5:53:36 AM UTC+1, Chris Kees wrote: > > Nix package manager > I'm aware of that. Going the Nix way means a lot of modifications to how packages are built, much of it at odds with how normal distributions package software. I wasn't the only one who thought that would be a bad idea. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] RFC: New Build/Packaging System
HI Volker,
If you haven't already looked at it, you might be interested in the
hashdist project: https://github.com/hashdist/hashdist. I've included a
few more comments below. I will be at sage days this week.
On Sun, Jun 15, 2014 at 3:59 PM, Volker Braun wrote:
> This is a RFC for new packaging system for "sage-the-distribution". I've
> already talked about this with a few of you at the last sage days, but
> finally it managed to do something about it. The goal is to be:
>
> * Git-aware: use SHA1 hashes instead of timestamps for dependency
> calculations
>
hashdist can use git commit ID's and pull source from github. It also uses
SHA1 hashes for other types of source archives. hashdist also hashes and
caches the builds following the approach described by E. Dolstra (Nix
package manager).
* Unified machine-readable package configuration using YAML
>
package and stack configuration files are in yaml
> * Dependency handling also for optional packages
> * Distinction between different types of dependencies: build time, runtime
> hard/soft, testing.
>
here's and example of the matplotlib package spec:
https://github.com/hashdist/hashstack/blob/master/pkgs/matplotlib/matplotlib.yaml
> * Modular, allowing for easy experimentation with per-package backends
>
not sure I understand this point
> * written in pure Python without any dependencies
>
written in pure Python with all depedencies bundled.
> * doctested for Python 2.6, 2.7, 3.3, and 3.4.
>
>
I don't think there are any major issues with Python 3, but most of the
stacks we are building are Python 2.7.
> Having all package configuration data accessible in machine-readable form
> is IMHO the key to managing complexity. There are various possibilities for
> the file format, but in the end I think YAML is the best choice. It
> integrates very well with Python, is easy to read/write for humans, and if
> things go wrong the parser can pin-point the error very well. Downside is
> that it is not in the Python standard library, but then I don't think we
> should that let us dictate the file format. We'll just include the
> Python-only part of PyYAML as a fallback if the OS Python does not have it.
>
> The entire package configuration will be in a file
> SAGE_ROOT/build/pkgs//package.yaml, for example
>
> name:
> matplotlib
>
> category:
> standard
>
> source:
> version:
> 1.3.1
> tarball:
> name: matplotlib-{source.version}.tar.gz
> sha1: f340378c43c4c3f6219ef9fd84af4ebbe18f8feb
>
> builder:
> type:SpkgInstallScript
> install_script: spkg-install
>
> depends:
> build:
> - pkgconf
> - setuptools
> hard:
> - python
> - numpy
> - freetype
> - libpng
> - gdmodule
> - dateutil
> - pyparsing
>
>
> I've been working on an implementation, which you can find at
> http://trac.sagemath.org/ticket/16483. It is not feature-complete, but
> can already build the standard packages in dependency order.
>
>
> The package manager lives in SAGE_ROOT/build/manager. Eventually, all
> build-related sage command line switches should just call it:
>
> $ ./build/manager/sage-pkg help
> usage: sage-pkg [-h] [--config CONFIG] [--log LOG]
>
>
> {info,shell,list,pkg-upgrade-v1,help,download,unpack,prepare,configure,compile,check,install,build,get}
> ...
>
> The Sage Package Manager
>
> positional arguments:
>
> {info,shell,list,pkg-upgrade-v1,help,download,unpack,prepare,configure,compile,check,install,build,get}
> infoPrint information about package
> shell IPython shell
> listList all packages
> pkg-upgrade-v1 Upgrade package descriptions
> helpGet help
> downloadBuild package up to the "download" step
> unpack Build package up to the "unpack" step
> prepare Build package up to the "prepare" step
> configure Build package up to the "configure" step
> compile Build package up to the "compile" step
> check Build package up to the "check" step
> install Build package and install
> build Build everything
> get Download tarball/spkg/file
>
> optional arguments:
> -h, --helpshow this help message and exit
> --config CONFIG Builder configuration file
> --log LOG One of [DEBUG, INFO, ERROR, WARNING, CRITICAL]
>
> $ ./build/manager/sage-pkg info matplotlib
> Configuration:
> - config.builder.install_script = spkg-install
> - config.builder.type = SpkgInstallScript
> - config.category = standard
> - config.depends.build = 'pkgconf', 'setuptools']
> - config.depends.hard = ['python', 'numpy', 'freetype', 'libpng',
> 'gdmodule', 'dateutil', 'pyparsing']
> - config.name
Re: [sage-devel] RFC: New Build/Packaging System
On Sunday, June 15, 2014 10:42:55 PM UTC+1, François wrote: > So I guess you don't record files installed. Is it at least feasible? > Its definitely something that I want. But in the interest of keeping manageable milestones I'm not going to change the spgk-install scripts for now. Recording the files installed can be added later. This is a RFC, not a RFFR (request for feature requests) -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] RFC: New Build/Packaging System
On Sun, 15 Jun 2014 14:40:13 Volker Braun wrote: > On Sunday, June 15, 2014 10:14:59 PM UTC+1, Snark wrote: > > (1) what is the relation between your work and Felix Salfelder's > > None. > No surprise if a bit sad. > > which still hasn't been fully included as far as I know? > > Because it doesn't work. There is always trac if you want to know the > details. > > (2) does it allow uninstalling packages? > > > Not yet, but it provides at least a framework for separating the package > build steps. > So I guess you don't record files installed. Is it at least feasible? > > (3) does it allow using system libs/binaries when they're available and > > suitable? > > No, but it'll provide a configuration mechanism that can be used for that. Excellent. Francois -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] RFC: New Build/Packaging System
On Sunday, June 15, 2014 10:14:59 PM UTC+1, Snark wrote: > > (1) what is the relation between your work and Felix Salfelder's None. > which still hasn't been fully included as far as I know? > Because it doesn't work. There is always trac if you want to know the details. (2) does it allow uninstalling packages? Not yet, but it provides at least a framework for separating the package build steps. > (3) does it allow using system libs/binaries when they're available and > suitable? > No, but it'll provide a configuration mechanism that can be used for that. -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
Re: [sage-devel] RFC: New Build/Packaging System
Hi, Le 15/06/2014 22:59, Volker Braun a écrit : This is a RFC for new packaging system for "sage-the-distribution". I've already talked about this with a few of you at the last sage days, but finally it managed to do something about it. The goal is to be: * Git-aware: use SHA1 hashes instead of timestamps for dependency calculations * Unified machine-readable package configuration using YAML * Dependency handling also for optional packages * Distinction between different types of dependencies: build time, runtime hard/soft, testing. * Modular, allowing for easy experimentation with per-package backends * written in pure Python without any dependencies * doctested for Python 2.6, 2.7, 3.3, and 3.4. I don't have much to comment on yet (I'll need to dig further), but I already have a few questions: (1) what is the relation between your work and Felix Salfelder's from last summer, which still hasn't been fully included as far as I know? (2) does it allow uninstalling packages? (3) does it allow using system libs/binaries when they're available and suitable? Snark on #sagemath -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
[sage-devel] RFC: New Build/Packaging System
This is a RFC for new packaging system for "sage-the-distribution". I've
already talked about this with a few of you at the last sage days, but
finally it managed to do something about it. The goal is to be:
* Git-aware: use SHA1 hashes instead of timestamps for dependency
calculations
* Unified machine-readable package configuration using YAML
* Dependency handling also for optional packages
* Distinction between different types of dependencies: build time, runtime
hard/soft, testing.
* Modular, allowing for easy experimentation with per-package backends
* written in pure Python without any dependencies
* doctested for Python 2.6, 2.7, 3.3, and 3.4.
Having all package configuration data accessible in machine-readable form
is IMHO the key to managing complexity. There are various possibilities for
the file format, but in the end I think YAML is the best choice. It
integrates very well with Python, is easy to read/write for humans, and if
things go wrong the parser can pin-point the error very well. Downside is
that it is not in the Python standard library, but then I don't think we
should that let us dictate the file format. We'll just include the
Python-only part of PyYAML as a fallback if the OS Python does not have it.
The entire package configuration will be in a file
SAGE_ROOT/build/pkgs//package.yaml, for example
name:
matplotlib
category:
standard
source:
version:
1.3.1
tarball:
name: matplotlib-{source.version}.tar.gz
sha1: f340378c43c4c3f6219ef9fd84af4ebbe18f8feb
builder:
type:SpkgInstallScript
install_script: spkg-install
depends:
build:
- pkgconf
- setuptools
hard:
- python
- numpy
- freetype
- libpng
- gdmodule
- dateutil
- pyparsing
I've been working on an implementation, which you can find at
http://trac.sagemath.org/ticket/16483. It is not feature-complete, but can
already build the standard packages in dependency order.
The package manager lives in SAGE_ROOT/build/manager. Eventually, all
build-related sage command line switches should just call it:
$ ./build/manager/sage-pkg help
usage: sage-pkg [-h] [--config CONFIG] [--log LOG]
{info,shell,list,pkg-upgrade-v1,help,download,unpack,prepare,configure,compile,check,install,build,get}
...
The Sage Package Manager
positional arguments:
{info,shell,list,pkg-upgrade-v1,help,download,unpack,prepare,configure,compile,check,install,build,get}
infoPrint information about package
shell IPython shell
listList all packages
pkg-upgrade-v1 Upgrade package descriptions
helpGet help
downloadBuild package up to the "download" step
unpack Build package up to the "unpack" step
prepare Build package up to the "prepare" step
configure Build package up to the "configure" step
compile Build package up to the "compile" step
check Build package up to the "check" step
install Build package and install
build Build everything
get Download tarball/spkg/file
optional arguments:
-h, --helpshow this help message and exit
--config CONFIG Builder configuration file
--log LOG One of [DEBUG, INFO, ERROR, WARNING, CRITICAL]
$ ./build/manager/sage-pkg info matplotlib
Configuration:
- config.builder.install_script = spkg-install
- config.builder.type = SpkgInstallScript
- config.category = standard
- config.depends.build = 'pkgconf', 'setuptools']
- config.depends.hard = ['python', 'numpy', 'freetype', 'libpng',
'gdmodule', 'dateutil', 'pyparsing']
- config.name = matplotlib
- config.source.tarball.name = matplotlib-1.3.1.tar.gz
- config.source.tarball.sha1 = f340378c43c4c3f6219ef9fd84af4ebbe18f8feb
- config.source.version = 1.3.1
Status: up to date
Installed when: 2 hours ago
--
You received this message because you are subscribed to the Google Groups
"sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sage-devel+unsubscr...@googlegroups.com.
To post to this group, send email to sage-devel@googlegroups.com.
Visit this group at http://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.
