Re: Build determinism, dependency granularity, and dependency scope
On Wed, Nov 25, 2020 at 6:15 PM Leo Famulari wrote: > > No. The way that dependencies are handled in Go-world does concord with > Guix on a conceptual level — it's definitely possible to have hundreds > of versions of each Go library — but it's impractical with the current > Guix tooling. > Java-based Guix packages also suffer from this problem (actually, I'm far more familiar with dependency management in the JVM landscape than for Go, but the use of granularly versioned and scoped, distributed dependency models by both languages appears to be similar on the surface). For example, the `java-log4j-core` Guix package (at version 2.4.1 in the Guix tree) has a dependency on `java-fasterxml-jackson-core` (at version 2.9.4), but the corresponding Log4j release asserts a dependency version of 2.6.2 in its `pom.xml` [1]. > A good stopgap option is to use vendored dependencies (heresy, I know), > assuming they are free software and the upstream sources include them. > This works fine and is better than not having Go software at all. > In the case of the Go application I was trying to package, it does not include vendored dependencies. And I don't have any relationship or check-in privileges with the project - it is simply something I wanted to use in an environment with other Guix-sourced packages. Well, I guess it would be straightforward to fork the GitHub source, run `go mod vendor` [2] and check in the vendor directory with a specific tag such as "vx.y.z-guix-vendored". Whether the project maintainers would accept such a pull request, or if it would be considered bad form to refer to a forked repository in a Guix package definition instead of the official repo if not, I don't know. In the long run, Guix's Go support needs a complete overhaul. > Indeed. [1] https://github.com/apache/logging-log4j2/blob/rel/2.4.1/pom.xml#L177 [2] https://golang.org/ref/mod#vendoring
Re: Build determinism, dependency granularity, and dependency scope
On Wed, Nov 25, 2020 at 10:52 PM raingloom wrote: > > Multiple versions leads to more maintenance burden and a bigger store. > That is true, but IMO with Guix's focus on build determinism and repeatability, it's a price that must be paid. > But I admit I didn't really investigate which exact module versions > Yggdrasil is compatible with. > As an end-user, if I'm installing the Yggdrasil package with a specific version, say 0.3.15, that corresponds to a specific Git commit tag, I want that package to maintain fidelity with the dependency versions locked down by the upstream project maintainers. As it stands, the Guix dependency declaration mechanism is lossy, flattening out dependencies to a single global scope which all packages in a profile must assume. Unfortunately, for me, that really limits its usefulness.
Build determinism, dependency granularity, and dependency scope
I have been trying to package an open source application written in Go for
Guix, and along the way as I've come to understand the mechanics better,
I've realized a few things which are a bit disconcerting. I'll refer to the
package for Yggdrasil, as it was recommended to me as a good blueprint to
follow for the project I'm trying to package.
If you take a look at the package definition for Yggdrasil 0.3.15, here are
some of the Golang dependencies:
(propagated-inputs
;; ...
("go-golang-org-x-net" ,go-golang-org-x-net)
("go-golang-org-x-text" ,go-golang-org-x-text)
;; ... )
If you look at the project's `go.mod` file [1], you have:
golang.org/x/net v0.0.0-20200301022130-244492dfa37a
golang.org/x/text v0.3.3-0.20191230102452-929e72ca90de
But if you look at the commits for the packages defined in the Guix tree,
they do not correspond. And the `go-golang-org-x-text` package in the Guix
tree (version "0.3.2") does not even meet the minimum version specified in
`go.mod`.
Also, it occurs to me that someone could decide to bump the version for one
of these packages up in the global Guix tree at any time to satisfy the
version requirements of some other package which require a newer version,
but because at the single package level there is only a reference to the
package name but not the version, all dependencies in the tree will be
carried along for the ride (!).
Now, there's nothing preventing someone from defining versioned packages in
the Guix tree, such as a `go-golang-org-x-text-929e72ca90de`, and referring
to those in dependent packages, but in practice that doesn't seem to be
done and most packages appear to have only one version, except for some
things like major language/platform versions (e.g. openjdk).
Am I missing something here?
It seems like what is needed would something like a package-scoped
"dependency constructor", allowing you to declare required versions
per-package:
(propagated-inputs
;; ...
("go-golang-org-x-net" (go-module "golang.org/x/net" "244492dfa37a"))
("go-golang-org-x-text" (go-module "golang.org/x/text"
"929e72ca90de"))
;; ... )
[1] https://github.com/yggdrasil-network/yggdrasil-go/blob/v0.3.15/go.mod
Re: Name resolution failures during build
The project I'm attempting to package is a standalone application, with a Makefile build system entrypoint - it is not a library, so I don't think `go-build-system` is of any help here (though presumably the Makefile uses golang build machinery under the covers). Do any examples of standalone applications written in Go that are already packaged for Guix jump to mind? On Fri, Nov 20, 2020 at 4:34 PM Christopher Baines wrote: > > Stephen Scheck writes: > > > Sure - I understand the reasoning - but this is an open-source project > with > > its own build system that I'm trying to package, and I believe its build > > system depends on network access (it's Go-based, and my understanding is > > that the Go dependency model relies on access to numerous external Git > > repos). Is such a project unsupportable by Guix? > > Not at all, there's a number of examples of software written in Go > packaged for Guix, it's just that you need to build it in a way that > doesn't depend on network access. > > Keeping this as a hard requirement is one of the reasons why having Guix > packages is beneficial. If some packages used the network during builds, > you'd end up in the bad situation where packages would be far more > likely to fail to build, now or in the future, and it would be harder to > reason about build reproducibility (due to the interaction with the > network). > Message-ID: <87blfrk912@cbaines.net> >
Re: Name resolution failures during build
Sure - I understand the reasoning - but this is an open-source project with its own build system that I'm trying to package, and I believe its build system depends on network access (it's Go-based, and my understanding is that the Go dependency model relies on access to numerous external Git repos). Is such a project unsupportable by Guix? On Fri, Nov 20, 2020 at 3:47 PM Christopher Baines wrote: > > Stephen Scheck writes: > > > I'm getting the following error when I try to install a package I'm > working > > on: > > > > go: github.com/BurntSushi/toml@v0.3.1: Get " > > https://proxy.golang.org/github.com/%21burnt%21sushi/toml/@v/v0.3.1.mod > ": > > dial tcp: lookup proxy.golang.org on [::1]:53: read udp > > [::1]:43973->[::1]:53: read: connection refused > > > > It uses `gnu-build-system`. Is there some package that needs to be added > to > > the build inputs for ordinary name resolution to work? > > To ensure that builds don't depend on the network, the build processes > don't have network access. > > This could mean you're missing something in the build environment (an > input), or that there's an input being ignored (and the network is being > used to try and fetch it). > > Does that make sense? > > Chris >
Name resolution failures during build
I'm getting the following error when I try to install a package I'm working on: go: github.com/BurntSushi/toml@v0.3.1: Get " https://proxy.golang.org/github.com/%21burnt%21sushi/toml/@v/v0.3.1.mod": dial tcp: lookup proxy.golang.org on [::1]:53: read udp [::1]:43973->[::1]:53: read: connection refused It uses `gnu-build-system`. Is there some package that needs to be added to the build inputs for ordinary name resolution to work?
Unaltered Git source
Hello, Is there a way to retrieve the unaltered source of a Git project in a `(package ...)` definition? I'm using `%build-inputs` to retrieve the source, but this is a directory in /gnu/store, and all of the file permissions are changed to read-only (0400), which unfortunately causes problems with the project's build.
trivial-build-system and which
This package definition always fails with #f returned by `(which "bash")`
... am I missing something?
(build-system trivial-build-system)
(arguments
`(#:builder
(begin
(use-modules (guix build utils))
(invoke "make" (string-append "SHELL=" (which "bash")) "..."
(native-inputs
`(("bash" ,bash)
("make" ,gnu-make)
Re: Package workspace relocation
I just need to add a prefix to the directory in which the Git source is cloned, because its build system requires that. I don't care which TMP directory Guix daemon uses for its build root. I just need for `/my-project` to become `/foo/bar/my-project`. It sounds like what you suggested will do the trick, if I understand it correctly. Thanks. On Mon, Nov 16, 2020 at 3:56 PM Leo Famulari wrote: > On Mon, Nov 16, 2020 at 03:15:46PM -0500, Stephen Scheck wrote: > > Is there a way to relocate the package build workspace for a Git clone > > source, e.g.: > > > > (package > > (name "my-package") > > (version "0.0.1") > > (source (origin > > (method git-fetch) > > (uri (git-reference > > (url "https://github.com/my-package";))) > > (file-name (git-file-name name version)) > > > > This results in a path like `/my-package-0.0.1` but > > because of idiosyncrasies in the project's build system, I need for it to > > be relocated to something like > `/foo/bar/my-package-0.0.1`. > > > > How can I do this? > > I'm not sure that I fully understand what you are asking. > > However, the location of the build directory is controlled by the TMPDIR > variable, which is set in the environment of the guix-daemon. You can > set it to any location that you want. > > An alternative per-package approach is to add a build phase before the > 'unpack' phase and do something like `mkdir -p foo/bar && chdir > foo/bar`. >
Package workspace relocation
Is there a way to relocate the package build workspace for a Git clone source, e.g.: (package (name "my-package") (version "0.0.1") (source (origin (method git-fetch) (uri (git-reference (url "https://github.com/my-package";))) (file-name (git-file-name name version)) This results in a path like `/my-package-0.0.1` but because of idiosyncrasies in the project's build system, I need for it to be relocated to something like `/foo/bar/my-package-0.0.1`. How can I do this? Thanks.
Guix GitLab Runner on Kubernetes
Hello, I've created some resources for deploying a custom GitLab CI/CD Runner integrated with a Guix daemon with persistent store on Kubernetes clusters. You can use such a Runner to provide Guix services to your project CI/CD jobs which need it running on hosted gitlab.com. Details are here: https://gitlab.com/singularsyntax/guix-k8s https://gitlab.com/singularsyntax/guix-gitlab-runner-k8s
Re: Gitlab CI
Ludovic Courtès writes: > Hello! I’m late to the party but I’m interested in this discussion… :-) > GitLab CI can run jobs directly in a Docker image: > https://docs.gitlab.com/ce/ci/docker/using_docker_images.html > So it would be interesting to have any easy way for people to create a Docker > image for this purpose. I’ve updated my Alpine-based Docker image, and I think it should work now for many common use cases, but it would be great if it saw some more testing. Details here: https://hub.docker.com/r/singularsyntax/guix-bootstrap Cheers -ss
Re: Gitlab CI
Jérémy Korwin-Zmijowski writes: > Hey Guixters ! > > I am experimenting with Gitlab CI for a Guile project I am working on. > It is managed with Hall and I would like to benefit the Guix features. > > Does anyone has done that before ? > Is there a way to get a Guix docker image instead of the Debian one ? Hello Jérémy, I have been trying to accomplish a similar goal - using Guix in GitLab CI for a Guile project. To that end, I’ve built some Docker images you might find interesting: Guix in Alpine Linux https://hub.docker.com/r/singularsyntax/guix-bootstrap/tags https://gitlab.com/singularsyntax-docker-hub/guix-bootstrap (the Dockerfile and pipeline for deployment to Docker Hub) My goal is to enable typical Docker workflows with Guix, for example something like this Dockerfile: FROM gnu/guix:1.1.0 RUN guix package —install Then use the resulting Docker image in the build, test, etc. stages of a GitLab CI/CD pipeline. Unfortunately, Guix does not have a natural fit with Docker containerization since `guix-daemon` must be running for any Guix commands to work. I haven’t been able to get any Dockerfile workflows like above working, since `docker build` seems to skip execution of the regular image entry point command which starts `guix-daemon`. I think to make it work, you’d need to provide some kind of shell wrapper around the `guix` command which first starts `guix-daemon`, waits for it to be ready, then runs Guix commands. But you can use the Alpine image for Docker-in-Docker workflows where you first start a container with `docker run`, then issue successive `docker exec` commands to perform work inside the container, then extract the results from the container for use in other stages of a CI/CD pipeline. I also attempted to build a “pure” Guix container not requiring installation on top of another Linux distribution by using `guix system docker-image <…>`. I even had a GitLab pipeline making daily builds which you might find interesting (but I recently disabled it as I’ll explain below): https://hub.docker.com/r/singularsyntax/guix/tags https://gitlab.com/singularsyntax-docker-hub/guix (GitLab CI/CD pipeline for deployment to Docker Hub) It works fine for obtaining a sandbox for which to experiment interactively with Guix, but has the same problems as the Alpine image, and is even a little worse because there are no “anchor” binaries inside the container, not even `/bin/sh`, until the container init process establishes symlinks into the `/gnu/store` tree. If you try to use it in scripts or CI/CD pipelines, they often fail due to race conditions with the init script, and without `/bin/sh` it’s difficult to reliably assert any pre-conditions inside the container. So it’s a real pain to work with, and I think of little utility for scripting as it currently stands. Anyway, hopefully this provides some more insight, and good luck!
Re: Guix Docker image inflation
On Sun, May 31, 2020 at 2:51 PM zimoun wrote: > Maybe the explosion of size would be slower. If you do, please report > here the number after say 12 generations; I am really interesting. ;-) > Now I'm confused - in your reply to Vincent, it seemed that there were still problems with the GC removing dead store items even after you did an export/re-import with the entire image on a single Docker layer? Or did I misread it? > All the question seems to be: > - what is the purpose of such Docker image? Which usage? > - what infrastructure do you have at hand to build the Docker images? > Well, Guix is interesting, and there aren't ready-made containers for it out there like there are for Ubuntu, Fedora, etc. if you have a need to do some task in that kind of environment, or just to play around, or see how the system is evolving. Also, I have been playing around with Guile lately and I thought Guix might be a better fit for that kind of work than other environments where Guile is largely neglected (Guix is *written* in Guile, after all). And I happened to be learning GitLab CI/CD around the same time, and it seemed like a good opportunity to experiment with both at once, so I thought, why not? :-) Which infrastructure - well, GitLab CI/CD, with fixed compute limits :-)
Re: Guix Docker image inflation
On Sun, May 31, 2020 at 5:37 AM zimoun wrote: > No, it is how Docker is designed. Maybe the terminology "layer" is > not the Docker one but when the images are chained, one cannot remove > the data of the previous layer of the total image. > I'm not disagreeing with that, but IF any of the store files resulting from `guix pull` are ephemeral (i.e. intermediate build results not anchored to a profile) AND guix GC worked inside the container, my approach might still work - yes there would be image and layers growth but it might be small enough not to care between periodic image rebases. But I'm starting to doubt that, or at least it is difficult to quantify with the GC issues. > Because if you run Guix outside an Docker container, you will not have > the issue. The main issue is how the Docker "filesystem" is designed. > Actually, there might be another way around this, still avoiding the need for a custom Runner, for example mounting /var/guix and /gnu/store into the container instead of belonging to it. If done that way, layer accumulation wouldn't be an issue, and maybe GC between layers neither.
Re: Guix Docker image inflation
On Sun, May 31, 2020 at 12:31 AM Chris Marusich wrote: > > Also, layers are helpful in the case of someone pulling down daily > > Guix Docker images on a frequent basis, because then only the new, > > ideally small layers need to be downloaded, whereas if you rebase for > > every image build, you'd have to download the entire image every day. > > That is true, but suppose I have the following 3 images: > > - Image A: A base image created in January 2020. > - Image B: Based on A, and I ran "guix pull" in February 2020. > - Image C: Based on A, and I ran "guix pull" in June 2020. > > I would guess that the size difference between A and B is approximately > the same as the difference between A and C. It'll be different, of > course, but generally the size difference between A and C should not > grow linearly with time, since "guix pull" is only going to install at > most the total closure of things necessary to build and run Guix, which > doesn't increase much in size as time goes on. However, when you > daisy-chain the images every day, the image size will grow linearly with > time because the contents of all the previous layers is carried forward. > > > My build script issues several `docker exec ` > > sequences, followed by a `docker commit `. Intermediate > > changes to the container file system prior to the commit do not > > generate layers, only the net changes after the commit. > > There are two problems here. One is that the image size grows without > bound. The other is that guix-daemon is failing to GC store items in > the Docker container. Although they are both concerning, the latter is > not the cause of the former. > > If you install new store items (e.g., via "guix pull"), make them dead, > and then GC them, all in the same container before running "docker > commit", then I agree: those GC'd store items would not persist in a > layer anywhere. However, I don't think that's what's happening here. > Sure, there might be a few store items like this, but in practice, there > will be many store items from the previous image which began live but > became dead when you ran "guix pull" and deleted your old profile > generations. It is those store items that are adding the most space to > your image. > Yes, I get this. I never expected the container to stay constant in size, but I was hoping daily pulls would result in relatively low image growth. It's not clear to me if any of the items which should get GC'd but don't are just ephemeral build results, in which case growth might be tolerable with an occasional rebase (perhaps monthly or bi-monthly). But I'm now starting to doubt my whole approach because it seems like there are some fundamental GC problems with running a live Guix system inside a container. > Besides store items, I noticed two other things about your images: > > - The contents of /var is growing slowly without bound, but it isn't > nearly as bad as the contents of /gnu/store. This is probably due to > log files; consider pruning them. > These are presumably OK to delete, without any special handling for Guix? > - Your script runs "docker commit" while guix-daemon (and other > programs) are still running. To ensure the guix-daemon's database (or > other things) does not become corrupt, consider terminating all > processes before committing the new image. > `docker commit` pauses the container (unless you tell it not to) ... although I guess that could still cause problems if Guix store writes aren't implemented in an atomic way. Thanks, -SS
Re: Guix Docker image inflation
On Fri, May 29, 2020 at 7:55 PM zimoun wrote: > Thank you for the explanation. The issue is these layers. When I > wrote [1], it was not clear for me because I am not enough familiar > with Docker, but with your explanations, it is clear now. :-) > > [1] http://issues.guix.gnu.org/41607#1 > No, it is not layers - they are a symptom, not the cause. See my reply to Chris. The problem is clearly that Guix isn't deleting garbage files ... which may have something to do with how Guix interacts with files in the file system and differences in Docker environments (no idea, I don't know how Guix works, but perhaps it needs some special privilege enabled when it runs inside Docker containers?), but layers themselves do not prevent file deletion inside a container. > > FYI, Guix itself can build Docker images from scratch - no base image > > required! It can even build a Docker image of a full-blown Guix System > > from scratch. Sorry if you already knew that - I just wanted to point > > it out in case you didn't! > > I think the idea is to use GitlabCI to build the Docker images > containing Guix materials. And AFAIK, GitlabCI does not provide Guix > related tools, isn't it? I mean there is no gitlab-runner able to run > guix-daemon. If I remember well, we discussed about this topic at > FOSDEM, it should be awesome. :-) > It is possible to host your own external Runners, and have them utilized by CI/CD jobs running inside the GitLab cloud service. You could install Guix on them and configure your CI/CD pipeline to require execution of certain jobs on these custom runners. But I'm not sure I see why that would help?
Re: Guix Docker image inflation
On Fri, May 29, 2020 at 7:31 PM Chris Marusich wrote: > > Could it be that you are accumulating layers without bound? > > > https://developers.redhat.com/blog/2016/03/09/more-about-docker-images-size/ > > Since Docker images are built up of immutable layers, if you build your > image from an existing base image, I'm not sure that it's possible to > produce a new image that is smaller than the base image. Basically, > even if you run "guix gc" to remove dead store items, they will still > exist on a prior layer, so the size of the new image won't decrease. > And since you're installing new things, the size will actually increase. > If you repeat this process by using the new image as an input for yet > another build, I think you will accumulate layers and storage space > without bound. > Layers certainly add some image size overhead, but I don't think that is the culprit here. And producing a smaller image isn't really the goal, it's just to keep image growth reasonable between each incremental guix pull. Dead store items would only exist on previous layers if they make it there in the first place. As has been demonstrated on previous posts in the thread, I believe the problem is some guix bug which prevents deletion of garbage-collected store items. What is reasonable growth? That is hard to answer, but I would expect it be roughly proportional to the growth of a guix installation over time in a non-Docker environment, taking some constant amount of layer overhead as a given. I don't really know what `guix pull` does, but I think it's something along these lines: 1) the global package index is brought up-to-date; 2) Any packages which are installed in the profile doing the pull are upgraded to newer versions if they've been updated. So day-to-day, particularly in the case where there have been no updates to packages installed in the profile, size growth should be very small. Periodic "rebasing" of incremental Docker images might still be helpful from time to time using one of the layer squashing tools out there, but I don't think it should be necessary on a daily basis. Also, layers are helpful in the case of someone pulling down daily Guix Docker images on a frequent basis, because then only the new, ideally small layers need to be downloaded, whereas if you rebase for every image build, you'd have to download the entire image every day. The boundless layer accumulation you point out shouldn't be a problem with the way that I'm building the images. When you do a `RUN ` inside a Dockerfile, it is essentially doing `docker exec ` followed by `docker commit `. It is the commit step which produces a new layer. You can think of a RUN command inside a Dockerfile as kind of a single-step transaction, which incorporates the net file system changes into the image. My build script issues several `docker exec ` sequences, followed by a `docker commit `. Intermediate changes to the container file system prior to the commit do not generate layers, only the net changes after the commit. You can convince yourself of this by doing something like the following: docker run docker exec dd if=/dev/urandom of=/RANDOM-DATA bs=1048576 count=1024 docker commit docker exec rm /RANDOM-DATA docker commit You'll end up with two new images - the first one should be about 1 GB larger than the base image, the second one the same size. FYI, Guix itself can build Docker images from scratch - no base image > required! It can even build a Docker image of a full-blown Guix System > from scratch. Sorry if you already knew that - I just wanted to point > it out in case you didn't! > Yes, thanks, I know - if you read through the thread you'll see that I make reference to `guix system docker-image [...]`. -SS
Re: Guix Docker image inflation
On Fri, May 29, 2020 at 5:54 PM zimoun wrote: > Do you have '/var/' in your Docker image? Because it looks like the same > than: > Yes: root@guix ~# ls -la /var/guix total 44 drwxr-xr-x 1 root root 4096 May 16 19:36 ./ drwxr-xr-x 1 root root 4096 May 29 22:02 ../ drwxr-xr-x 1 root root 4096 May 29 22:02 daemon-socket/ drwxr-xr-x 1 root root 4096 May 27 00:34 db/ -rw--- 1 root root0 May 16 19:35 gc.lock drwxr-xr-x 1 root root 4096 May 16 19:57 gcroots/ drwxr-xr-x 1 root root 4096 Jan 1 1970 profiles/ drwxr-xr-x 1 root root 4096 May 16 19:35 substitute/ drwxr-xr-x 1 root root 4096 May 27 00:34 temproots/ drwxr-xr-x 1 root root 4096 May 16 19:36 userpool/ If you'd like, you can fetch the exact same image and look around yourself: docker pull singularsyntax/guix:master-a5374cd # same as singularsyntax/guix:latest CONTAINER=`docker run --detach --tty --privileged singularsyntax/guix:master-a5374cd` docker exec --interactive --tty $CONTAINER /run/current-system/profile/bin/bash --login
Re: Guix Docker image inflation
On Fri, May 29, 2020 at 2:44 PM zimoun wrote: > On Fri, 29 May 2020 at 20:37, Leo Famulari wrote: > > > > On Fri, May 29, 2020 at 08:21:08PM +0200, Marius Bakke wrote: > > > Leo Famulari writes: > > > > --branch and --commit would be passed to `guix pull`, and then you'd > run > > > > `guix system docker-image` based on that. > > > > > > There is also 'guix time-machine --commit=abc123 -- system > docker-image'. > > > > Right, that's probably more efficient than creating lots of `guix pull` > > generations. > > Yes, but it is hard to apriori know the forward commit. > Yes, and also, does a Docker image created by `guix pull` followed by `guix system docker-image [...]` in fact really inherit the Guix snapshot from the system that creates it? Here's what I get on a freshly minted image made that way: root@guix /# guix pull --list-generations guix pull: error: profile '/var/guix/profiles/per-user/root/current-guix' does not exist root@guix /# guix describe guix describe: error: failed to determine origin hint: Perhaps this `guix' command was not obtained with `guix pull'? Its version string is 1.1.0-4.bdc801e. root@guix /# guix package --list-generations guix package: error: profile '/var/guix/profiles/per-user/root/guix-profile' does not exist But here's `guix describe` output from the parent system: root@localhost /# guix describe Generation 13 May 29 2020 19:28:11 (current) guix 41a2d6a repository URL: https://git.savannah.gnu.org/git/guix.git branch: master commit: 41a2d6a8b9294a6eb8e97aaefd569e755f5f461e Until a fresh `guix pull` is run on the new image, it isn't functional and there's no apparent way to confirm its actual commit hash, so I don't really see what advantage it offers over the incremental method I'm using (and it's unfeasibly slow, about 10-15 minutes for an incremental pull compared to over an hour to finish `guix system docker-image`).
Re: Guix Docker image inflation
On Fri, May 29, 2020 at 4:02 PM zimoun wrote: > Well, could you try > >guix system delete-generations >guix gc > root@guix /# guix system list-generations guix system: error: open-file: No such file or directory: "/var/guix/profiles/system-1-link/parameters" root@guix /# guix system delete-generations Backtrace: 1 (primitive-load "/root/.config/guix/current/bin/guix") In guix/ui.scm: 1936:12 0 (run-guix-command _ . _) guix/ui.scm:1936:12: In procedure run-guix-command: In procedure struct-vtable: Wrong type argument in position 1 (expecting struct): #f
Re: Guix Docker image inflation
On Fri, May 29, 2020 at 2:08 PM zimoun wrote: > How the initial Docker image > singularsyntax/guix-bootstrap:1.1.0-alpine-3.11 is built? > To understand, you use the Docker image > singularsyntax/guix-bootstrap:1.1.0-alpine-3.11 to build another > Docker image namely guix-docker-image.tar using Guix, right? > Well, that is not the point neither the issue. :-) > You can look at the Dockerfile here: https://gitlab.com/singularsyntax-docker-hub/guix-bootstrap It's pretty close to exactly the manual instructions for installing Guix on a "foreign" distro on top of Alpine Linux. Not the point, no, but how else do I obtain a seed Guix Docker image, which I can use to birth clean, pristine "baby" images of Guix's own making? It would be really nice if the Guix project itself provided such an image! > could you try that > > --8<---cut here---start->8--- > GUIX_PATH=/root/.config/guix/current/bin > $GUIX_PATH/guix pull --branch=$CI_COMMIT_REF_NAME--fallback > /root/.config/guix/current/bin/guix pull -d > /root/.config/guix/current/bin/guix package -d > /root/.config/guix/current/bin/guix gc > docker commit > /root/.config/guix/current/bin/guix package --install --fallback jq > --8<---cut here---end--->8--- > > Last, you could try to see what "guix package --list-profiles" says > and then "guix gc --list-dead". root@localhost /gnu/store# guix pull -d root@localhost /gnu/store# guix package --list-profiles /root/.config/guix/current root@localhost /gnu/store# guix package -d guix package: error: profile '/var/guix/profiles/per-user/root/guix-profile' does not exist root@localhost /gnu/store# guix package --list-profiles /root/.config/guix/current root@localhost /gnu/store# du -hs . 4.3G . root@localhost /gnu/store# guix gc finding garbage collector roots... deleting garbage... [0 MiB] deleting '/gnu/store/x7ns2xcp8lfg24zq7gr3y8ffczn1nsxp-guix-d79c917f2-modules' [0 MiB] deleting '/gnu/store/slwkzcmg6r1lr9a16x3krd2ax384p8wr-guix-system' [0 MiB] deleting '/gnu/store/dzifisbdk1gwy2fw2hwzgvdnjak22awl-guix-extra' deleting `/gnu/store/trash' deleting unused links... note: currently hard linking saves 1181.82 MiB guix gc: freed 0.636 MiBs root@localhost /gnu/store# du -hs . 4.3G . root@localhost /gnu/store# guix gc --list-dead finding garbage collector roots... determining live/dead paths... /gnu/store/0bm8h4ns6bymc7q24vhfr0dnb7qab729-guix-cli /gnu/store/0hjjj9dppc5xvq3bfjwbsygrfyqn0rlv-guix-cli /gnu/store/0m0xx2958fgyz8kk093afik5cn4rhrc1-guix-cli-modules /gnu/store/0pi2jhn3a778gc3fm1l31sh07fik4zwa-guix-system-tests-modules /gnu/store/0vwg9aqzs5xrk10vcs4dl105s3f42ilf-guix-b1affd477-modules # Lots more listed...
Re: Guix Docker image inflation
On Fri, May 29, 2020 at 2:02 PM Leo Famulari wrote: > Okay. For debugging, can you try garbage collecting those modules > directories? And if the garbage collector refuses, you can investigate > why with the 3 R's of Guix garbage collection, --referrers, > --references, and --requisites. > # Hmm... root@localhost /gnu/store# guix gc --references /gnu/store/x7ns2xcp8lfg24zq7gr3y8ffczn1nsxp-guix-d79c917f2-modules guix gc: error: path `/gnu/store/x7ns2xcp8lfg24zq7gr3y8ffczn1nsxp-guix-d79c917f2-modules' is not valid # Hmm... root@localhost /gnu/store# guix gc --requisites /gnu/store/x7ns2xcp8lfg24zq7gr3y8ffczn1nsxp-guix-d79c917f2-modules guix gc: error: path `/gnu/store/x7ns2xcp8lfg24zq7gr3y8ffczn1nsxp-guix-d79c917f2-modules' is not valid # Hmm... this one is different - no output root@localhost /gnu/store# guix gc --referrers /gnu/store/x7ns2xcp8lfg24zq7gr3y8ffczn1nsxp-guix-d79c917f2-modules # Now try to delete it... root@localhost /gnu/store# guix gc --delete /gnu/store/x7ns2xcp8lfg24zq7gr3y8ffczn1nsxp-guix-d79c917f2-modules finding garbage collector roots... [0 MiB] deleting '/gnu/store/x7ns2xcp8lfg24zq7gr3y8ffczn1nsxp-guix-d79c917f2-modules' deleting `/gnu/store/trash' deleting unused links... note: currently hard linking saves 1181.36 MiB # Still there... root@localhost /gnu/store# du -hs /gnu/store/x7ns2xcp8lfg24zq7gr3y8ffczn1nsxp-guix-d79c917f2-modules 210M /gnu/store/x7ns2xcp8lfg24zq7gr3y8ffczn1nsxp-guix-d79c917f2-modules
Re: Guix Docker image inflation
On Fri, May 29, 2020 at 1:08 PM Leo Famulari wrote: > I'm still not quite sure what you are doing (or what Docker does) so > please bear with me. > > > root@localhost /# du -h --max-depth=1 /gnu/store | egrep > > "guix-system$|guix-packages-base$|guix-[0-9a-f]*-modules$" > [...] > > 191M > /gnu/store/l3amdz5xyhflg5wdzlxr2685dq5glic2-guix-527ab3125-modules > > 201M > /gnu/store/5mhn1ynxvy7jihsknsnv3yspkkvc0r5s-guix-2e59ae238-modules > > If I understand correctly, you should not need both of these directories > in a Guix VM image. The latter hashes are truncated guix.git commit > hashes and a VM image would only be based on a single one. > Exactly, I agree (to the extent that I understand Guix). I recommend looking into why all these directories are being copied into > your images. > Whatever is in /gnu/store (as managed by Guix) goes into the image, nothing more and nothing less. > > I figure you'd want to create each image with *only* the things > corresponding to the Git commit it's based on, but it sounds like they > are being created by copying the entire host image, which doesn't seem > right. > > If the Docker images are being created by simply snapshotting the file > system of a non-ephemeral Guix system, that's probably not the right way > to do it. Is that what's going on? > Yes, as I said, the image is created from a file system snapshot, after Guix is brought up to date via `guix pull` and those various Guix garbage collection operations are run. However, it's not quite "non-ephmeral" as each Guix operation is run as an atomic command inside the Docker container, with nothing else running (except for guix-daemon, which has to always be running for Guix to operate to the best of my understanding, and a couple other Guix System daemons which anyway would be equivalent to the situation to any Guix installation running outside of a Docker container). How else would you suggest that it be done? It would be nice if `guix system docker-image` took `--branch` and `--commit` options to build a container from a well-defined Guix check-in state, but that doesn't seem to be the case. And in any case - too slow. The point here is to leverage daily incremental pulls to keep data transfer and build times down.
Re: Guix Docker image inflation
On Fri, May 29, 2020 at 1:12 PM zimoun wrote: > Dear Stephen, > > I am not sure to follow all the Docker dance. Well, if I understand > correctly, you did: > > guix package --delete-generations > guix gc > > which remove all except the current profile, i.e., ~/.guix-profile. > However, there is another profile '~/.config/guix/current' which is > the profile used when guix pull. Therefore, you have to clean the > generations here too: > >guix pull --delete-generations >guix gc > > Does it reduce the size? > root@localhost /# du -hs /gnu/store 4.3G /gnu/store ### only one generation ### root@localhost /# guix pull --list-generations Generation 12 May 28 2020 20:45:30 (current) guix a5374cd repository URL: https://git.savannah.gnu.org/git/guix.git branch: master commit: a5374cde918cfeae5c16b43b9f2dd2b24bc3564d root@localhost /# guix pull --delete-generations ### still only one generation ### root@localhost /# guix pull --list-generations Generation 12 May 28 2020 20:45:30 (current) guix a5374cd repository URL: https://git.savannah.gnu.org/git/guix.git branch: master commit: a5374cde918cfeae5c16b43b9f2dd2b24bc3564d root@localhost /# guix gc finding garbage collector roots... deleting garbage... [0 MiB] deleting '/gnu/store/slwkzcmg6r1lr9a16x3krd2ax384p8wr-guix-system' [0 MiB] deleting '/gnu/store/dzifisbdk1gwy2fw2hwzgvdnjak22awl-guix-extra' [0 MiB] deleting '/gnu/store/rqz825cwaf4866d2aljwkk9qq0g7rmzm-module-import' ### Many more store files (all 0 MiB) elided... deleting `/gnu/store/trash' deleting unused links... note: currently hard linking saves 1181.36 MiB guix gc: freed 0 MiBs ### no space recovered ### root@localhost /# du -hs /gnu/store 4.3G /gnu/store
Re: Guix Docker image inflation
On Thu, May 28, 2020 at 3:33 PM Leo Famulari wrote: > I'm not familiar with Docker so I'm not sure exactly what you are doing. > Specifically, I can't tell if you are creating new Docker images from > scratch each day, or if you are continuing to use the same one from day > to day. The previous day's Docker image is used as the base for the new one being built - the image is pulled from Docker Hub, `guix pull` is run inside it, and a new image is "committed" (Docker terminology for creating a new image from a file system snapshot). BTW, I posted an incorrect internal link - the actual Docker images are available here if you'd like to try them out: https://hub.docker.com/r/singularsyntax/guix/tags > I'm also not sure which image is growing each day... The daily Docker images described above. > In general, the parameters --delete-generations and --collect-garbage > are supposed to be passed values like a reference to a profile or an > amount of data to delete, respectively. Are you doing that? `guix gc --delete-generations` without a parameter causes all preceding pull and package generations to be deleted. > Are you removing / invalidating old generations before attempting to > garbage collect them? The store items they refer to cannot be deleted > until the generations themselves are no longer registered. Yes, `guix gc --delete-generations`, `guix gc --collect-garbage`, and `guix gc --optimize` are run in the order given. Note that passing a specific amount parameter to `--collect-garbage` makes no difference. > Usually, these old profiles are responsible for most of the disk usage > in /gnu/store. Indeed. It's clear what's taking up the space, but I don't understand why it does not get garbage collected: root@localhost /# guix pull --list-generations Generation 12 May 28 2020 20:45:30 (current) guix a5374cd repository URL: https://git.savannah.gnu.org/git/guix.git branch: master commit: a5374cde918cfeae5c16b43b9f2dd2b24bc3564d root@localhost /# guix package --list-generations guix package: error: profile '/var/guix/profiles/per-user/root/guix-profile' does not exist root@localhost /# du -h --max-depth=1 /gnu/store | egrep "guix-system$|guix-packages-base$|guix-[0-9a-f]*-modules$" 44M /gnu/store/slwkzcmg6r1lr9a16x3krd2ax384p8wr-guix-system 44M /gnu/store/zf67wb6c0s97vwmywjq09hy9jq0w5mmi-guix-system 107M /gnu/store/plaay02w581vx9ilyiv93sl1lw54n7h5-guix-packages-base 44M /gnu/store/qhbk7g8z97m37iak1s1yn2my82gv0lj5-guix-system 103M /gnu/store/2qcfl7h10dynjlifyvqwh9iiic52q5x6-guix-packages-base 107M /gnu/store/m0fv2xmfif5pxnfb1bscfvgyfx0x6xdc-guix-packages-base 90M /gnu/store/hz2rn2l0jixg91q4rsdcwc489y71ll29-guix-05e1edf22-modules 41M /gnu/store/w47fgv8p2hvaqdwywymwvm0qlh4gw0ih-guix-system 191M /gnu/store/l3amdz5xyhflg5wdzlxr2685dq5glic2-guix-527ab3125-modules 201M /gnu/store/5mhn1ynxvy7jihsknsnv3yspkkvc0r5s-guix-2e59ae238-modules 44M /gnu/store/dzc16sv8jv831m0jkk5llc2ws1a3mk0z-guix-system 44M /gnu/store/9a2hr5lh15vxqa7bjih8w47wr6hr11nv-guix-system 103M /gnu/store/1lwdys51wi08r5an2rr6sqk9kbgr7qip-guix-packages-base 44M /gnu/store/c3spiv1c0fg83j7d99mjwk0s6fw77wl5-guix-system 44M /gnu/store/vwzk618h1wxy6z9i06xnhnxj4gvhkiss-guix-system 6.7M /gnu/store/a5xsqxr04pwnyni5x2gqjnishzq80cbw-guix-packages-base 14M /gnu/store/mych9fchln22pbhpc5syxyymx4hz496y-guix-8bd0b533b-modules 35M /gnu/store/brbwlbnx56ms50kklyqk9fsf0xkwjjf9-guix-498e2e669-modules 3.2M /gnu/store/dirpwhdr7h4nyphy4ncxqi4f2njv3rsh-guix-packages-base 35M /gnu/store/d3h4b7nvnms8d03ddi9b481dlxpykl7l-guix-5e3d16994-modules 5.8M /gnu/store/n339sr8c63f0nzja6yl8zfwy1jklj19j-guix-packages-base 25M /gnu/store/0vwg9aqzs5xrk10vcs4dl105s3f42ilf-guix-b1affd477-modules 41M /gnu/store/pwr8ab20xa1whxag689lsz82l2na08x0-guix-system 6.5M /gnu/store/6sggbpgg0zkbgxwf3wa2j15dis8z7cr1-guix-packages-base 57M /gnu/store/8z9qc2bvq8azc08p4miq77yf2agk07aq-guix-843e77205-modules 71M /gnu/store/ibgjq1ampj8bldrabbsnwik2sr0gg3as-guix-a43fe7acd-modules 37M /gnu/store/x7ns2xcp8lfg24zq7gr3y8ffczn1nsxp-guix-d79c917f2-modules 18M /gnu/store/i72b4biraw6bhy1v7ly46kwyaacvfa28-guix-system 178M /gnu/store/47aack48aczpzm635axsy4jf2pvmwrv0-guix-ef1d475b0-modules 15M /gnu/store/77sxajrwigsdnyr4l4jq4pk6v5kwbm59-guix-system
Guix Docker image inflation
Hello, As an exercise, I set up daily Guix System Docker image builds using GitLab and Docker Hub, here: https://hub.docker.com/repository/registry-1.docker.io/singularsyntax/guix/tags?page=1 The build process works as follows: if an existing `latest` image does not exist for a given branch (master, 1.1.0, etc.), then bootstrap an image by running `guix system docker-image` inside an Alpine Linux Docker container with a fresh Guix installation. Using this image as a seed, `guix pull` is run for the desired branch, and the resulting image is committed to the Docker repository. If a "latest" image does exist, it is used instead as the base from which to run `guix pull`. Daily images are thus built incrementally from the previous day's build. For anybody curious about the process, the build script can be browsed here: https://gitlab.com/singularsyntax-docker-hub/guix/-/blob/master/.gitlab-ci.yml It works pretty well, except that I'm observing substantial image size inflation day-over-day, starting at ~197 MB from the seed image, now up to 1.71 GB eleven days later despite running `guix gc --delete-generations`, `guix gc --collect-garbage`, and `guix gc --optimize` after pulling prior to committing each new image. I'm wondering if there is some other Guix GC operation or option I'm missing, or any other suggestions which could stop this unsustainable image bloat from occurring. I really do doubt that the Guix System itself is growing this quickly.
Problem with `guix package --install-from-file` and certs on Guix System
Hello,
I'm trying to build a package locally - perl-net-bgp.scm was generated via
`guix import cpan Net::BGP`:
/# guix package --install-from-file=perl-net-bgp.scm
Backtrace:
In ice-9/boot-9.scm:
829:9 19 (catch _ _ # ?)
In guix/scripts/package.scm:
943:10 18 (_)
In guix/status.scm:
768:4 17 (call-with-status-report _ _)
In guix/scripts/package.scm:
951:14 16 (_)
In guix/build/syscalls.scm:
1142:4 15 (call-with-file-lock/no-wait _ _ _)
In guix/scripts/package.scm:
906:8 14 (_)
In guix/ui.scm:
1048:4 13 (show-manifest-transaction # ?)
In srfi/srfi-1.scm:
628:19 12 (mapn ("perl-net-bgp") (("0.17") ("out") (#)) 1)
In guix/ui.scm:
1053:23 11 (_ "perl-net-bgp" "0.17" "out" #)
In guix/packages.scm:
1308:13 10 (package-output # ?)
948:16 9 (cache! # # ?)
1267:22 8 (thunk)
1200:25 7 (bag->derivation # ?)
In srfi/srfi-1.scm:
592:29 6 (map1 (("perl" #) ?))
592:17 5 (map1 (("source" #) ?))
In ice-9/boot-9.scm:
829:9 4 (catch srfi-34 # ?)
In guix/packages.scm:
1015:18 3 (_)
In guix/store.scm:
1841:24 2 (run-with-store # ?)
1699:13 1 (_ _)
In guix/build/download.scm:
745:0 0 (url-fetch _ _ #:timeout _ #:verify-certificate? _ # _ # ?)
guix/build/download.scm:745:0: In procedure url-fetch:
Invalid keyword: #vu8(190 12 104 102 229 3 248 226 122 163 239 14 65 53 203
251 61 122 70 201 237 92 18 150 86 80 216 98 89 234 72 125)
This looks to be an issue verifying a TLS cert for whatever CPAN mirror the
package is choosing to download the CPAN module from.
Is there something special that needs to be done on Guix to install root
certs? I've tried adding the nss-certs, le-certs, and guile2.0-gnutls
packages, but they don't help.
Thanks.
Re: Docker image not working
I think the problem is that I was using an outdated version of `guix`. I
did a `guix pull` on my Guix system, the re-ran the command to build the
Docker image, and it works now.
One thing I noticed is that I have to pass the `--privileged` flag, or
subsequent invocations of `guix pull` fail with some error about failure to
clone the process:
$ docker run -it --privileged --tty guix:latest
making '/gnu/store/dn2247g52nv2vc3y578fm2ksrh8s00hn-system' the current
system...
setting up setuid programs in '/run/setuid-programs'...
populating /etc from /gnu/store/c3z6qji00a216xrjrf10sd3vr97ajr60-etc...
error in finalization thread: Success
error in finalization thread: Bad file descriptor
error in finalization thread: Bad file descriptor
error in finalization thread: Bad file descriptor
I'm not sure why the finalization errors are reported, but they don't seem
to cause problems. I can then do e.g.
$ docker exec -it /run/current-system/profile/bin/bash
--login
root@guix /#
Thanks for your help!
On Thu, Nov 28, 2019 at 8:35 PM Maxim Cournoyer
wrote:
> Hello Stephen,
>
> Stephen Scheck writes:
>
> > I don't think the docs are quite up to date, or perhaps refer to an older
> > version of Docker. If I do:
> >
> > $ docker create system:0qjxd5ljsh316ki7wqkk2xz9b68lynh2
> > Error response from daemon: No command specified
> >
> > Looks like `docker create` wants the entry point, hence why I tried to
> use
> > `docker run` to fire up the container ... so what should it be to kick
> off
> > the Shepherd service you referred to?
> >
> > Thanks.
>
> The documentation works for me, with a minor hiccup: the docker load
> doesn't return *just* the image ID: the first two words must be cut out
> (by piping it to awk '{ print $3 }' for example).
>
> --8<---cut here---start->8---
> guix system docker-image some-system-with-python.scm
> [...]
> /gnu/store/5ddyv2m6kfjhm0fc3v181ahs35hkf6j6-guix-docker-image.tar.gz
>
> image_id=`docker load <
> /gnu/store/5ddyv2m6kfjhm0fc3v181ahs35hkf6j6-guix-docker-image.tar.gz`
> $ echo $image_id
> Loaded image: guix:latest
> $ container_id=`docker create $image_id`
> invalid reference format: repository name must be lowercase
> $ image_id="guix:latest"
> $ container_id=`docker create $image_id`
> $ echo $container_id
> 9eabc3cae4bba25848c7f1ced15086232a5b19a2e5d775374e842855890d51f3
> $ docker start $container_id
> 9eabc3cae4bba25848c7f1ced15086232a5b19a2e5d775374e842855890d51f3
> $ docker exec -ti $container_id /run/current-system/profile/bin/bash
> --login
> root@9eabc3cae4bb /# which python
> which python
> /run/current-system/profile/bin/python
> root@9eabc3cae4bb /# python --version
> python --version
> Python 3.7.4
> --8<---cut here---end--->8---
>
> I also tested with your operating system declaration and it also worked:
>
> --8<---cut here---start->8---
> docker load <
> /gnu/store/49cgpsq6978j7f9l7fk3pw8f1dldlyv3-guix-docker-image.tar.gz
> c2cd1fc572fa: Loading layer
> [==>]631MB/631MB
> The image guix:latest already exists, renaming the old one with ID
> sha256:fb0dc0872f71e5051f316b36f1f12ef71a7eb88a77b16fbf86fff66258cce885 to
> empty string
> Loaded image: guix:latest
> $ ~/src/guix-master [env]$ docker create guix:latest
> 25c6b88a578270aec4bd20f7050f591b9f971397f77282ae826fe6f488b125c6
> $ docker start
> 25c6b88a578270aec4bd20f7050f591b9f971397f77282ae826fe6f488b125c6
> 25c6b88a578270aec4bd20f7050f591b9f971397f77282ae826fe6f488b125c6
> $ docker exec -ti
> 25c6b88a578270aec4bd20f7050f591b9f971397f77282ae826fe6f488b125c6
> /run/current-system/profile/bin/bash --login
> root@25c6b88a5782 /#
> --8<---cut here---end--->8---
>
> Sorry, I'm not sure what is wrong on your side. I've seen this error
> before but can't remember what the solution was :-/. Perhaps something
> doesn't work right with your Docker daemon?
>
> HTH,
>
> Maxim
>
Re: Docker image not working
I don't think the docs are quite up to date, or perhaps refer to an older version of Docker. If I do: $ docker create system:0qjxd5ljsh316ki7wqkk2xz9b68lynh2 Error response from daemon: No command specified Looks like `docker create` wants the entry point, hence why I tried to use `docker run` to fire up the container ... so what should it be to kick off the Shepherd service you referred to? Thanks. On Thu, Nov 28, 2019 at 11:09 AM Maxim Cournoyer wrote: > Hello Stephen, > > Stephen Scheck writes: > > > Hello, > > > > I'm trying to use the `guix system` command to create a Docker image as > > documented here: > > > > > > > https://guix.gnu.org/manual/en/html_node/Invoking-guix-system.html#Invoking-guix-system > > > > However, the created image does not work: > > > > $ docker run -it system:0qjxd5ljsh316ki7wqkk2xz9b68lynh2 > > /run/current-system/profile/bin/bash --login > > docker: Error response from daemon: OCI runtime create failed: > > container_linux.go:348: starting container process caused "exec: > > \"/run/current-system/profile/bin/bash\": stat > > /run/current-system/profile/bin/bash: no such file or directory": > unknown. > > > > This is the command I invoked to create the image: > > > > guix system init --no-bootloader --skip-checks --system=x86_64-linux > > guix-docker.scm /tmp/guix/docker-image > > > > And here is the system configuration I used: > > > > (use-modules (gnu)) > > (use-package-modules admin base bash less linux) > > > > (operating-system > > (host-name "guix") > > (timezone "UTC") > > (locale "en_US.utf8") > > > > (bootloader (bootloader-configuration > > (bootloader grub-bootloader) > > (target "/dev/null"))) > > (file-systems (cons (file-system > > (device (file-system-label > "guix-system-dummy")) > > (mount-point "/") > > (type "ext4")) > > %base-file-systems)) > > > > (packages (append (list bash coreutils-minimal inetutils less > procps > > which) %base-packages))) > > > > Am I missing something? > > Yes! The /run/current-system/profile/bin/bash symlink you are trying to > invoke is setup by one of the Shepherd services when the Guix system is > initialized. Here the system hasn't booted up yet. Currently the > system is initialized as part of the default entry point produced by > Guix, but such initialization only spawns Shepherd as PID 1 and leaves > you with a useless, non-interactive session that is not useful when > simply running 'docker run -it $your-image'. > > The only useful way to use a docker-image currently is to "start" the > container, using > > docker start $container_id > > And then attaching to it with docker exec > > docker exec -ti $container_id /run/current-system/profile/bin/bash > --login > > This is explained in the documentation. > > I don't find this really convenient, and intend to modify the default > entry point at some point to allow running commands directly from > 'docker run', but currently it is the way it works. > > What I can recommend for very simple use cases (starting a script, bash, > etc), is to use 'docker pack -f docker' instead to produce the Docker > image. > > This command allows you to produce symlinks in the generated image, for > example: > > --8<---cut here---start->8--- > guix pack --manifest=your-manifest.scm \ >-f docker \ >-S /etc/profile=etc/profile \ >-S /bin=bin > --8<---cut here---end--->8--- > > Will set the /etc/profile, /bin and /sbin links in the target to that of > the profile generated from your-manifest.scm. > > You could then override the default entry point of the docker image with > a command such as: > > docker run -it $your_image /bin/bash --login > > This bash session should source /etc/profile and make all of your > manifest installed software available to experiment with. > > HTH! > > Maxim >
Re: Docker image not working
BTW, I saw your comments in the Docker/Guix relationship thread about using `guix pack` as an alternative means to produce a Docker image. I tried this also, but I had a few issues: 1) You have to pass it `-S /bin=bin -S /sbin=sbin` flags so that command binaries can be conveniently located. This is not really an issue, but I think it does "lock" the resulting image to a particular snapshot of command binaries, which might be precisely what you want for an immutable container for a production application. But for other use cases, for example, for throw-away containers for development and testing, you might want the capability of being able to upgrade packages, and it's not clear to me if it's just as simple as replacing the symlinks to the `bin` and `sbin` directories of a new profile. And also, I guess `guix environment` addresses that use case. 2) In order to get a lot of basic functionality working, I had to install a Guix System into a VM from the distribution ISO, then copy various bits of its `/etc` (protocols, services, nsswitch.conf, pam.d, etc.) tree into an image to get network diagnostics (ping, etc.) and user management commands, among others, working. It seems like leveraging the `guix system` infrastructure to build a working, reproducible, and standard `/etc` tree is the way to go, but as already noted I'm not having any luck with the Docker images it produces. On Wed, Nov 27, 2019 at 11:40 PM Stephen Scheck wrote: > No, it does not: > > $ docker run -it system:0qjxd5ljsh316ki7wqkk2xz9b68lynh2 bash > docker: Error response from daemon: OCI runtime create failed: > container_linux.go:348: starting container process caused "exec: \"bash\": > executable file not found in $PATH": unknown. > > It seems like some things which get set up when creating a regular system > via `guix system` do not when specifying output as a Docker image. > Unfortunately, the documentation doesn't explain much about how the system > profile works, and I'm not knowledgeable enough about Guix to make an > informed guess. > > On Wed, Nov 27, 2019 at 9:39 AM zimoun wrote: > >> Hi, >> >> On Sun, 17 Nov 2019 at 18:17, Stephen Scheck >> wrote: >> >> > $ docker run -it system:0qjxd5ljsh316ki7wqkk2xz9b68lynh2 >> > /run/current-system/profile/bin/bash --login >> > docker: Error response from daemon: OCI runtime create failed: >> > container_linux.go:348: starting container process caused "exec: >> > \"/run/current-system/profile/bin/bash\": stat >> > /run/current-system/profile/bin/bash: no such file or directory": >> unknown. >> >> Does the command >> >> docker run -it system:0qjxd5ljsh316ki7wqkk2xz9b68lynh2 bash >> >> work? >> >> >> All the best, >> simon >> >
Re: Docker image not working
No, it does not: $ docker run -it system:0qjxd5ljsh316ki7wqkk2xz9b68lynh2 bash docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"bash\": executable file not found in $PATH": unknown. It seems like some things which get set up when creating a regular system via `guix system` do not when specifying output as a Docker image. Unfortunately, the documentation doesn't explain much about how the system profile works, and I'm not knowledgeable enough about Guix to make an informed guess. On Wed, Nov 27, 2019 at 9:39 AM zimoun wrote: > Hi, > > On Sun, 17 Nov 2019 at 18:17, Stephen Scheck > wrote: > > > $ docker run -it system:0qjxd5ljsh316ki7wqkk2xz9b68lynh2 > > /run/current-system/profile/bin/bash --login > > docker: Error response from daemon: OCI runtime create failed: > > container_linux.go:348: starting container process caused "exec: > > \"/run/current-system/profile/bin/bash\": stat > > /run/current-system/profile/bin/bash: no such file or directory": > unknown. > > Does the command > > docker run -it system:0qjxd5ljsh316ki7wqkk2xz9b68lynh2 bash > > work? > > > All the best, > simon >
Re: Docker image not working
> This is the command I invoked to create the image: > >guix system init --no-bootloader --skip-checks --system=x86_64-linux > guix-docker.scm /tmp/guix/docker-image Err, paste mistake, actually this: guix system docker-image --no-bootloader --skip-checks --system=x86_64-linux guix-docker.scm
Docker image not working
Hello, I'm trying to use the `guix system` command to create a Docker image as documented here: https://guix.gnu.org/manual/en/html_node/Invoking-guix-system.html#Invoking-guix-system However, the created image does not work: $ docker run -it system:0qjxd5ljsh316ki7wqkk2xz9b68lynh2 /run/current-system/profile/bin/bash --login docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"/run/current-system/profile/bin/bash\": stat /run/current-system/profile/bin/bash: no such file or directory": unknown. This is the command I invoked to create the image: guix system init --no-bootloader --skip-checks --system=x86_64-linux guix-docker.scm /tmp/guix/docker-image And here is the system configuration I used: (use-modules (gnu)) (use-package-modules admin base bash less linux) (operating-system (host-name "guix") (timezone "UTC") (locale "en_US.utf8") (bootloader (bootloader-configuration (bootloader grub-bootloader) (target "/dev/null"))) (file-systems (cons (file-system (device (file-system-label "guix-system-dummy")) (mount-point "/") (type "ext4")) %base-file-systems)) (packages (append (list bash coreutils-minimal inetutils less procps which) %base-packages))) Am I missing something?
