Re: Building a Docker image for GitLab-CI

[Andreas Enge]

Am Thu, Jun 06, 2024 at 11:23:20AM +0200 schrieb Ludovic Courtès:
> (It’s possible to sidestep that problem for instance by having GitLab-CI
> offload to a runner in a machine that you control, where Guix is
> installed.  But I’m looking for a solution that could work
> out-of-the-box on any GitLab-CI or similar instance.)

It turns out that our Gitlab instance enables docker containers in
privileged mode, so it just happened to work in my case...

Andreas

Re: Building a Docker image for GitLab-CI

[Ludovic Courtès]

Reza Housseini  skribis:

> Ludovic Courtès  writes:
>
>> That’s nice, but unless I’m mistaken, Guix is missing from the image,
>> right?
>>
>> My goal would be to be able to use Guix within the image, so I can have
>> GitLab-CI spawn ‘guix build’ commands (or similar).
>
> Oh sorry this was a misunderstanding from my side. Have you tried to use
> the guix system image command to create a docker container?

I did, but that’s not suitable:

  https://mail.gnu.org/archive/html/help-guix/2024-02/msg00066.html

Ludo’.

Re: Building a Docker image for GitLab-CI

[Reza Housseini]

Ludovic Courtès  writes:

> That’s nice, but unless I’m mistaken, Guix is missing from the image,
> right?
>
> My goal would be to be able to use Guix within the image, so I can have
> GitLab-CI spawn ‘guix build’ commands (or similar).

Oh sorry this was a misunderstanding from my side. Have you tried to use
the guix system image command to create a docker container?

Best,
Reza

Re: Building a Docker image for GitLab-CI

[Ludovic Courtès]

Hi,

Andreas Enge  skribis:

> Am Tue, Jun 04, 2024 at 01:29:22PM +0200 schrieb Ludovic Courtès:
>> My goal would be to be able to use Guix within the image, so I can have
>> GitLab-CI spawn ‘guix build’ commands (or similar).
>
> with a colleague we have set up such a system. He has started from a Debian
> image and written a docker script to install Guix "manually", which provides
> our base image. Then Gitlab CI creates a new image from a channels file in
> the git repository we want to monitor. The command we use is
> "guix system image -t docker" and not "guix pack";

To be able to do that, you need to have Guix already up and running in
the Docker image you pass to GitLab-CI: that’s the problem I’m trying to
solve.

Or did I misunderstand?

(It’s possible to sidestep that problem for instance by having GitLab-CI
offload to a runner in a machine that you control, where Guix is
installed.  But I’m looking for a solution that could work
out-of-the-box on any GitLab-CI or similar instance.)

Thanks,
Ludo’.

Re: Building a Docker image for GitLab-CI

[Andreas Enge]

Hello,

Am Tue, Jun 04, 2024 at 01:29:22PM +0200 schrieb Ludovic Courtès:
> My goal would be to be able to use Guix within the image, so I can have
> GitLab-CI spawn ‘guix build’ commands (or similar).

with a colleague we have set up such a system. He has started from a Debian
image and written a docker script to install Guix "manually", which provides
our base image. Then Gitlab CI creates a new image from a channels file in
the git repository we want to monitor. The command we use is
"guix system image -t docker" and not "guix pack"; I am not really familiar
with the second command, and as far as I understand the first one has the
difference of running the shepherd as the docker command, which in our case
starts the guix-daemon and a guix-build-coordinator-agent.
(One addition: We use the patch in
   https://issues.guix.gnu.org/70933
to enable chroot in the container and then run the container in privileged
mode.)

Maybe we should write up a little blog post once everything is settled.

Andreas

Re: Building a Docker image for GitLab-CI

[Ludovic Courtès]

Hi Reza,

Reza Housseini  skribis:

>> Has anyone succeeded in building a Docker image suitable for use in
>> GitLab-CI?
>
> I normally do the following and it seems to work fine with our gitlab
> instance:
>
> registry=registry.gitlab.ost.ch:45023/sciceg/teaching/eeu_mlds
> archive=$(guix time-machine -C channels.scm -- pack -f docker -S /bin=bin -S 
> /lib=lib -S /share=share -m manifest.scm)
> tag=$(docker load -i $archive)
> docker tag ${tag##*"Loaded image: "} $registry
> docker push $registry
>
> what seems to be crucial is to add the following packages to the
> manifest file:
>
> "bash" 
> "coreutils"
> "git"

That’s nice, but unless I’m mistaken, Guix is missing from the image,
right?

My goal would be to be able to use Guix within the image, so I can have
GitLab-CI spawn ‘guix build’ commands (or similar).

> PS: It would be really nice if one could provide a docker tag directly
> to `guix pack -f docker`

I think it’s already possible, see ‘guix pack --help-docker-format’.

Thanks,
Ludo’.

Re: Building a Docker image for GitLab-CI

[Reza Housseini]

Ludovic Courtès  writes:

sorry forgot to include the list...

Hi Ludo

> Has anyone succeeded in building a Docker image suitable for use in
> GitLab-CI?

I normally do the following and it seems to work fine with our gitlab
instance:

registry=registry.gitlab.ost.ch:45023/sciceg/teaching/eeu_mlds
archive=$(guix time-machine -C channels.scm -- pack -f docker -S /bin=bin -S 
/lib=lib -S /share=share -m manifest.scm)
tag=$(docker load -i $archive)
docker tag ${tag##*"Loaded image: "} $registry
docker push $registry

what seems to be crucial is to add the following packages to the
manifest file:

"bash" 
"coreutils"
"git"

than I can use the image in the gitlab-ci.yml file

generate-exercises:
  image: registry.gitlab.ost.ch:45023/sciceg/teaching/eeu_mlds:latest
  ...

Hope that helps!

Best,
Reza

PS: It would be really nice if one could provide a docker tag directly
to `guix pack -f docker`

Re: Building a Docker image for GitLab-CI

[Ludovic Courtès]

Hi,

Efraim Flashner  skribis:

> In the past I used a script to install guix using the shell script and
> then ran guix pull before building my package.  I suppose you could use
> a Debian image and run 'guix pull' first before building something.

I could… but that’d be cheating.  :-)

Ludo’.

Re: Building a Docker image for GitLab-CI

[Efraim Flashner]

On Tue, Feb 13, 2024 at 11:31:28AM +0100, Ludovic Courtès wrote:
> Hello Guix!
> 
> Has anyone succeeded in building a Docker image suitable for use in
> GitLab-CI?  I haven’t.  Here’s what I tried.

In the past I used a script to install guix using the shell script and
then ran guix pull before building my package.  I suppose you could use
a Debian image and run 'guix pull' first before building something.

-- 
Efraim Flashner  רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted


signature.asc
Description: PGP signature

Re: Building a Docker image for GitLab-CI

[Andreas Enge]

Am Tue, Feb 13, 2024 at 11:31:28AM +0100 schrieb Ludovic Courtès:
> Has anyone succeeded in building a Docker image suitable for use in
> GitLab-CI?  I haven’t.  Here’s what I tried.

A colleague of mine just found this:
   https://gitlab.com/daym/guix-on-docker/

:-)

Andreas

Re: Building a Docker image for GitLab-CI

[Suhail]

Ludovic Courtès  writes:

> Initially, I built an image with ‘guix system image -t docker …’ but
> that doesn’t work because then the image’s “entry point” is shepherd,
> but shepherd never returns.

Did you try resetting the entrypoint in .gitlab-ci.yml using the
image:entrypoint keyword? [1]

[1]: 

-- 
Suhail

This email is not an offer capable of acceptance, does not evidence an
intention to enter into an agreement, has no operative effect until a
definitive agreement is signed in writing by both parties, and that no
party should act in reliance on the email or any representations of the
sender until a definitive agreement is signed in writing by both
parties.

Building a Docker image for GitLab-CI

[Ludovic Courtès]

Hello Guix!

Has anyone succeeded in building a Docker image suitable for use in
GitLab-CI?  I haven’t.  Here’s what I tried.

Initially, I built an image with ‘guix system image -t docker …’ but
that doesn’t work because then the image’s “entry point” is shepherd,
but shepherd never returns.  Thus, GitLab-CI would spawn the image and
eventually time out.

So I tried this instead:

  guix pack guix bash-minimal coreutils-minimal grep net-base \
--save-provenance -S /bin=bin -S /share=share -S /etc=etc \
-f docker --max-layers=100

… with ‘.gitlab-ci.yml’ doing something like this:

--8<---cut here---start->8---
build:
  image: registry.gitlab.inria.fr/…
  tags: ["ci.inria.fr", "linux"]
  before_script:
- echo "nameserver 10.0.2.3 # XXX" > /etc/resolv.conf
- guix archive --authorize < /share/guix/ci.guix.gnu.org.pub
- guix archive --authorize < /share/guix/bordeaux.guix.gnu.org.pub
- guix-daemon --disable-chroot &
  script:
- guix shell -m manifest.scm -- rubber --pdf article.tex
  artifacts:
paths:
  - article.pdf
--8<---cut here---end--->8---

Problem is, name resolution appears to fail in the container image; the
‘resolv.conf’ trick was a crude attempt to work around it, but it
failed.  I guess the problem is that I don’t know how GitLab-CI or
Docker is supposed to set up networking inside those containers.

Thoughts?

Neat tip to upload your Guix-built image to a registry: use Skopeo.

  guix shell skopeo -- skopeo login registry.gitlab.inria.fr
  guix shell skopeo -- skopeo copy \
docker-archive:///gnu/store/…-docker-image.tar.gz \
docker://registry.gitlab.inria.fr/… \
--insecure-policy

(“Insecure policy”, what could possibly go wrong?)

Ludo’.

Re: guix docker on gitlab-ci

[Graham Addis]

Hi Wolf,

Patch is now submitted, see following for details.
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=64171
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=64173

Now we find out what I forgot. ;-)

Thanks again for your help,

Graham

On Tue, 13 Jun 2023 at 17:56, Graham Addis  wrote:
>
> Hi Wolf,
>
> Well I managed to sort it out last weekend.
>
> I have added a --docker-entry-point option which can be invoked
> multiple times to provide the docker EntryPoint value in exec form.
>
> I have left the --entry-point behaviour alone except the
> --docker-entry-point takes precedence.
>
> I still have the docs to update, and to test the some of the different
> cases, but it all seems to work.
>
> Thanks again,
>
> Graham
>
> On Mon, 5 Jun 2023 at 22:38, Graham Addis  wrote:
> >
> > Hi Wolf,
> >
> > Similar to what I was trying, which didn't work. I'll try to have a go
> > one evening and try and work out where I went wrong.
> >
> > Thanks again. Graham
> >
> > On Mon, 5 Jun 2023 at 18:35, wolf  wrote:
> > >
> > > On 2023-06-05 16:37:50 +0100, Graham Addis wrote:
> > > > Hi Wolf,
> > > >
> > > > Not a particularly successful weekend...
> > > >
> > > > As --entry-point was used by other pack methods I thought the best
> > > > option would be to add a --docker-entry-point just for the docker
> > > > format and pass it into the build as an extra-option like for deb and
> > > > rpm formats.
> > > >
> > > > However I couldn't work out how to pass in a list via the
> > > > extra-options, so I'm a bit stuck.
> > > >
> > > > If there is anyone who knows their way around the pack scripts and can
> > > > point me in the right direction, that would help, but other than that
> > > > I'll probably get some more time next weekend.
> > >
> > > I did not try to implement this, so my guess might be completely off, but
> > > looking at how -S is implemented, I would suggest trying something like:
> > >
> > > 1. Introducing new %docker-format-options and friends (similar to already
> > >existing %deb-format-options and friends), providing the 
> > > --entry-point-arg
> > >options, that would be specific to a docker format (although I am not 
> > > sure if
> > >it needs to be specific, maybe some other formats support arguments as
> > >well?).
> > > 2. Writing entry-point-arg-spec-option-parser, while taking inspiration 
> > > from
> > >symlink-spec-option-parser.  Yours would be much simpler, since you 
> > > would be
> > >just accumulating values into a list.
> > >
> > > As for the extra-options, I guess modifying current code (by adding the 
> > > 'docker
> > > option) into something like:
> > >
> > > (extra-options (match pack-format
> > >  ('deb
> > >   (list #:control-file
> > > (process-file-arg opts 'control-file)
> > > #:postinst-file
> > > (process-file-arg opts 'postinst-file)
> > > #:triggers-file
> > > (process-file-arg opts 'triggers-file)))
> > >  ('docker
> > >   (list #:entry-point-args
> > > (process-file-arg opts 'entry-point-arg)))
> > >  ('rpm
> > >   (list #:relocatable? relocatable?
> > > #:prein-file
> > > (process-file-arg opts 'prein-file)
> > > #:postin-file
> > > (process-file-arg opts 'postin-file)
> > > #:preun-file
> > > (process-file-arg opts 'preun-file)
> > > #:postun-file
> > > (process-file-arg opts 'postun-file)))
> > >  (_ '(
> > >
> > > could work?  build-docker-image already accepts a list as an 
> > > #:entry-point, so
> > > it should be just a matter of properly composing the list.
> > >
> > > Not sure if this is helpful.
> > >
> > > W.
> > >
> > > >
> > > > Graham
> > > >
> > > > On Fri, 2 Jun 2023 at 09:13, Graham Addis  
> > > > wrote:
> > > > >
> > > > > Hi Wolf,
> > > > >
> > > > > On Thu, 1 Jun 2023 at 22:55, wolf  wrote:
> > > > > >
> > > > > > On 2023-05-31 18:47:03 +0100, Graham Addis wrote:
> > > > > > > I could use the equivalent format for --entry-point
> > > > > > >
> > > > > > > --entry-point executable --entry-point param1 --entry-point param2
> > > > > > >
> > > > > > I think that is a reasonable idea.  Only downside is that it would 
> > > > > > not be
> > > > > > backwards compatible (since currently last --entry-point wins), 
> > > > > > however I am not
> > > > > > sure if someone actually relies on that behavior.
> > > > > >
> > > > > > Backwards compatible way would be keeping --entry-point as it is 
> > > > > > and introducing
> > > > > > new argument (--entry-point-arg) that could be used to build the 
> > > 

Re: guix docker on gitlab-ci

[Graham Addis]

Hi Wolf,

Well I managed to sort it out last weekend.

I have added a --docker-entry-point option which can be invoked
multiple times to provide the docker EntryPoint value in exec form.

I have left the --entry-point behaviour alone except the
--docker-entry-point takes precedence.

I still have the docs to update, and to test the some of the different
cases, but it all seems to work.

Thanks again,

Graham

On Mon, 5 Jun 2023 at 22:38, Graham Addis  wrote:
>
> Hi Wolf,
>
> Similar to what I was trying, which didn't work. I'll try to have a go
> one evening and try and work out where I went wrong.
>
> Thanks again. Graham
>
> On Mon, 5 Jun 2023 at 18:35, wolf  wrote:
> >
> > On 2023-06-05 16:37:50 +0100, Graham Addis wrote:
> > > Hi Wolf,
> > >
> > > Not a particularly successful weekend...
> > >
> > > As --entry-point was used by other pack methods I thought the best
> > > option would be to add a --docker-entry-point just for the docker
> > > format and pass it into the build as an extra-option like for deb and
> > > rpm formats.
> > >
> > > However I couldn't work out how to pass in a list via the
> > > extra-options, so I'm a bit stuck.
> > >
> > > If there is anyone who knows their way around the pack scripts and can
> > > point me in the right direction, that would help, but other than that
> > > I'll probably get some more time next weekend.
> >
> > I did not try to implement this, so my guess might be completely off, but
> > looking at how -S is implemented, I would suggest trying something like:
> >
> > 1. Introducing new %docker-format-options and friends (similar to already
> >existing %deb-format-options and friends), providing the 
> > --entry-point-arg
> >options, that would be specific to a docker format (although I am not 
> > sure if
> >it needs to be specific, maybe some other formats support arguments as
> >well?).
> > 2. Writing entry-point-arg-spec-option-parser, while taking inspiration from
> >symlink-spec-option-parser.  Yours would be much simpler, since you 
> > would be
> >just accumulating values into a list.
> >
> > As for the extra-options, I guess modifying current code (by adding the 
> > 'docker
> > option) into something like:
> >
> > (extra-options (match pack-format
> >  ('deb
> >   (list #:control-file
> > (process-file-arg opts 'control-file)
> > #:postinst-file
> > (process-file-arg opts 'postinst-file)
> > #:triggers-file
> > (process-file-arg opts 'triggers-file)))
> >  ('docker
> >   (list #:entry-point-args
> > (process-file-arg opts 'entry-point-arg)))
> >  ('rpm
> >   (list #:relocatable? relocatable?
> > #:prein-file
> > (process-file-arg opts 'prein-file)
> > #:postin-file
> > (process-file-arg opts 'postin-file)
> > #:preun-file
> > (process-file-arg opts 'preun-file)
> > #:postun-file
> > (process-file-arg opts 'postun-file)))
> >  (_ '(
> >
> > could work?  build-docker-image already accepts a list as an #:entry-point, 
> > so
> > it should be just a matter of properly composing the list.
> >
> > Not sure if this is helpful.
> >
> > W.
> >
> > >
> > > Graham
> > >
> > > On Fri, 2 Jun 2023 at 09:13, Graham Addis  
> > > wrote:
> > > >
> > > > Hi Wolf,
> > > >
> > > > On Thu, 1 Jun 2023 at 22:55, wolf  wrote:
> > > > >
> > > > > On 2023-05-31 18:47:03 +0100, Graham Addis wrote:
> > > > > > I could use the equivalent format for --entry-point
> > > > > >
> > > > > > --entry-point executable --entry-point param1 --entry-point param2
> > > > > >
> > > > > I think that is a reasonable idea.  Only downside is that it would 
> > > > > not be
> > > > > backwards compatible (since currently last --entry-point wins), 
> > > > > however I am not
> > > > > sure if someone actually relies on that behavior.
> > > > >
> > > > > Backwards compatible way would be keeping --entry-point as it is and 
> > > > > introducing
> > > > > new argument (--entry-point-arg) that could be used to build the 
> > > > > argument list,
> > > > > but I might be overthinking it :).
> > > >
> > > > I'll go with extending --entry-point to accept multiple values and use
> > > > all of them for --format=docker and simply use the last one provided
> > > > for the other formats.
> > > >
> > > > I'll try to put a patch together at the weekend.
> > > >
> > > > Thanks for all your input, it really helps.
> > > >
> > > > Graham
> > >
> >
> > --
> > There are only two hard things in Computer Science:
> > cache invalidation, naming things and 

Re: guix docker on gitlab-ci

[Graham Addis]

Hi Wolf,

Similar to what I was trying, which didn't work. I'll try to have a go
one evening and try and work out where I went wrong.

Thanks again. Graham

On Mon, 5 Jun 2023 at 18:35, wolf  wrote:
>
> On 2023-06-05 16:37:50 +0100, Graham Addis wrote:
> > Hi Wolf,
> >
> > Not a particularly successful weekend...
> >
> > As --entry-point was used by other pack methods I thought the best
> > option would be to add a --docker-entry-point just for the docker
> > format and pass it into the build as an extra-option like for deb and
> > rpm formats.
> >
> > However I couldn't work out how to pass in a list via the
> > extra-options, so I'm a bit stuck.
> >
> > If there is anyone who knows their way around the pack scripts and can
> > point me in the right direction, that would help, but other than that
> > I'll probably get some more time next weekend.
>
> I did not try to implement this, so my guess might be completely off, but
> looking at how -S is implemented, I would suggest trying something like:
>
> 1. Introducing new %docker-format-options and friends (similar to already
>existing %deb-format-options and friends), providing the --entry-point-arg
>options, that would be specific to a docker format (although I am not sure 
> if
>it needs to be specific, maybe some other formats support arguments as
>well?).
> 2. Writing entry-point-arg-spec-option-parser, while taking inspiration from
>symlink-spec-option-parser.  Yours would be much simpler, since you would 
> be
>just accumulating values into a list.
>
> As for the extra-options, I guess modifying current code (by adding the 
> 'docker
> option) into something like:
>
> (extra-options (match pack-format
>  ('deb
>   (list #:control-file
> (process-file-arg opts 'control-file)
> #:postinst-file
> (process-file-arg opts 'postinst-file)
> #:triggers-file
> (process-file-arg opts 'triggers-file)))
>  ('docker
>   (list #:entry-point-args
> (process-file-arg opts 'entry-point-arg)))
>  ('rpm
>   (list #:relocatable? relocatable?
> #:prein-file
> (process-file-arg opts 'prein-file)
> #:postin-file
> (process-file-arg opts 'postin-file)
> #:preun-file
> (process-file-arg opts 'preun-file)
> #:postun-file
> (process-file-arg opts 'postun-file)))
>  (_ '(
>
> could work?  build-docker-image already accepts a list as an #:entry-point, so
> it should be just a matter of properly composing the list.
>
> Not sure if this is helpful.
>
> W.
>
> >
> > Graham
> >
> > On Fri, 2 Jun 2023 at 09:13, Graham Addis  
> > wrote:
> > >
> > > Hi Wolf,
> > >
> > > On Thu, 1 Jun 2023 at 22:55, wolf  wrote:
> > > >
> > > > On 2023-05-31 18:47:03 +0100, Graham Addis wrote:
> > > > > I could use the equivalent format for --entry-point
> > > > >
> > > > > --entry-point executable --entry-point param1 --entry-point param2
> > > > >
> > > > I think that is a reasonable idea.  Only downside is that it would not 
> > > > be
> > > > backwards compatible (since currently last --entry-point wins), however 
> > > > I am not
> > > > sure if someone actually relies on that behavior.
> > > >
> > > > Backwards compatible way would be keeping --entry-point as it is and 
> > > > introducing
> > > > new argument (--entry-point-arg) that could be used to build the 
> > > > argument list,
> > > > but I might be overthinking it :).
> > >
> > > I'll go with extending --entry-point to accept multiple values and use
> > > all of them for --format=docker and simply use the last one provided
> > > for the other formats.
> > >
> > > I'll try to put a patch together at the weekend.
> > >
> > > Thanks for all your input, it really helps.
> > >
> > > Graham
> >
>
> --
> There are only two hard things in Computer Science:
> cache invalidation, naming things and off-by-one errors.

Re: guix docker on gitlab-ci

[wolf]

On 2023-06-05 16:37:50 +0100, Graham Addis wrote:
> Hi Wolf,
> 
> Not a particularly successful weekend...
> 
> As --entry-point was used by other pack methods I thought the best
> option would be to add a --docker-entry-point just for the docker
> format and pass it into the build as an extra-option like for deb and
> rpm formats.
> 
> However I couldn't work out how to pass in a list via the
> extra-options, so I'm a bit stuck.
> 
> If there is anyone who knows their way around the pack scripts and can
> point me in the right direction, that would help, but other than that
> I'll probably get some more time next weekend.

I did not try to implement this, so my guess might be completely off, but
looking at how -S is implemented, I would suggest trying something like:

1. Introducing new %docker-format-options and friends (similar to already
   existing %deb-format-options and friends), providing the --entry-point-arg
   options, that would be specific to a docker format (although I am not sure if
   it needs to be specific, maybe some other formats support arguments as
   well?).
2. Writing entry-point-arg-spec-option-parser, while taking inspiration from
   symlink-spec-option-parser.  Yours would be much simpler, since you would be
   just accumulating values into a list.

As for the extra-options, I guess modifying current code (by adding the 'docker
option) into something like:

(extra-options (match pack-format
 ('deb
  (list #:control-file
(process-file-arg opts 'control-file)
#:postinst-file
(process-file-arg opts 'postinst-file)
#:triggers-file
(process-file-arg opts 'triggers-file)))
 ('docker
  (list #:entry-point-args
(process-file-arg opts 'entry-point-arg)))
 ('rpm
  (list #:relocatable? relocatable?
#:prein-file
(process-file-arg opts 'prein-file)
#:postin-file
(process-file-arg opts 'postin-file)
#:preun-file
(process-file-arg opts 'preun-file)
#:postun-file
(process-file-arg opts 'postun-file)))
 (_ '(

could work?  build-docker-image already accepts a list as an #:entry-point, so
it should be just a matter of properly composing the list.

Not sure if this is helpful.

W.

> 
> Graham
> 
> On Fri, 2 Jun 2023 at 09:13, Graham Addis  wrote:
> >
> > Hi Wolf,
> >
> > On Thu, 1 Jun 2023 at 22:55, wolf  wrote:
> > >
> > > On 2023-05-31 18:47:03 +0100, Graham Addis wrote:
> > > > I could use the equivalent format for --entry-point
> > > >
> > > > --entry-point executable --entry-point param1 --entry-point param2
> > > >
> > > I think that is a reasonable idea.  Only downside is that it would not be
> > > backwards compatible (since currently last --entry-point wins), however I 
> > > am not
> > > sure if someone actually relies on that behavior.
> > >
> > > Backwards compatible way would be keeping --entry-point as it is and 
> > > introducing
> > > new argument (--entry-point-arg) that could be used to build the argument 
> > > list,
> > > but I might be overthinking it :).
> >
> > I'll go with extending --entry-point to accept multiple values and use
> > all of them for --format=docker and simply use the last one provided
> > for the other formats.
> >
> > I'll try to put a patch together at the weekend.
> >
> > Thanks for all your input, it really helps.
> >
> > Graham
> 

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.


signature.asc
Description: PGP signature

Re: guix docker on gitlab-ci

[Graham Addis]

Hi Wolf,

Not a particularly successful weekend...

As --entry-point was used by other pack methods I thought the best
option would be to add a --docker-entry-point just for the docker
format and pass it into the build as an extra-option like for deb and
rpm formats.

However I couldn't work out how to pass in a list via the
extra-options, so I'm a bit stuck.

If there is anyone who knows their way around the pack scripts and can
point me in the right direction, that would help, but other than that
I'll probably get some more time next weekend.

Graham

On Fri, 2 Jun 2023 at 09:13, Graham Addis  wrote:
>
> Hi Wolf,
>
> On Thu, 1 Jun 2023 at 22:55, wolf  wrote:
> >
> > On 2023-05-31 18:47:03 +0100, Graham Addis wrote:
> > > I could use the equivalent format for --entry-point
> > >
> > > --entry-point executable --entry-point param1 --entry-point param2
> > >
> > I think that is a reasonable idea.  Only downside is that it would not be
> > backwards compatible (since currently last --entry-point wins), however I 
> > am not
> > sure if someone actually relies on that behavior.
> >
> > Backwards compatible way would be keeping --entry-point as it is and 
> > introducing
> > new argument (--entry-point-arg) that could be used to build the argument 
> > list,
> > but I might be overthinking it :).
>
> I'll go with extending --entry-point to accept multiple values and use
> all of them for --format=docker and simply use the last one provided
> for the other formats.
>
> I'll try to put a patch together at the weekend.
>
> Thanks for all your input, it really helps.
>
> Graham

Re: guix docker on gitlab-ci

[Graham Addis]

Hi Wolf,

On Thu, 1 Jun 2023 at 22:55, wolf  wrote:
>
> On 2023-05-31 18:47:03 +0100, Graham Addis wrote:
> > I could use the equivalent format for --entry-point
> >
> > --entry-point executable --entry-point param1 --entry-point param2
> >
> I think that is a reasonable idea.  Only downside is that it would not be
> backwards compatible (since currently last --entry-point wins), however I am 
> not
> sure if someone actually relies on that behavior.
>
> Backwards compatible way would be keeping --entry-point as it is and 
> introducing
> new argument (--entry-point-arg) that could be used to build the argument 
> list,
> but I might be overthinking it :).

I'll go with extending --entry-point to accept multiple values and use
all of them for --format=docker and simply use the last one provided
for the other formats.

I'll try to put a patch together at the weekend.

Thanks for all your input, it really helps.

Graham

Re: guix docker on gitlab-ci

[Graham Addis]

On Thu, 1 Jun 2023 at 23:04, wolf  wrote:
> I did not know Guix does not currently support the shell form.  In that light 
> I
> think it should not be implemented, since once your idea (arguments for entry
> point) is implemented, it will be trivial for end-user to emulate it if so
> desired.

That was my thinking too.

> > One strange thing, I couldn't see the need for prefixing the profile
> > to the ENTRYPOINT command:
> > https://git.savannah.gnu.org/cgit/guix.git/tree/guix/scripts/pack.scm#n670
> > I took it out and everything seems to work, so I'm not sure what
> > problem it is fixing. Anybody any idea?
>
> Wild guess, but it might depend on your container runtime (whether it uses 
> execv
> or execvp). Absolute path feels somewhat more robust.

Ah. I'll look into that, and not touch it just yet...

Re: guix docker on gitlab-ci

[wolf]

On 2023-05-30 07:52:57 +0100, Graham Addis wrote:
> Hi Worf,
> 
> Thanks for the response, see below.
> 
> On Mon, 29 May 2023 at 20:41, wolf  wrote:
> >
> > On 2023-05-24 18:04:47 +0100, Graham Addis wrote:
> > > Dear people,
> > >
> > > I tried to create a docker image to use in a gitlab-ci instance but it
> > > failed because I couldn't use --entry-point="bin/sh -l -c" or
> > > equivalent, basically the gitlab-runner complains that it can't run
> > > binaries.
> >
> > Would this be better using just bin/sh for the entry point and passing the 
> > -l
> > and -c as an arguments?
> 
> Probably, but I don't think that's an option in gitlab ci and anyway
> it would be nice to support the docker options.
> 
> > > I've managed to get it working by making some changes to 
> > > guix/scripts/pack.scm
> > >
> > > Adding a fn in docker-image, just before the call to
> > > build-docker-image, to create a list from the string passed in from
> > > --entry-point="bin/sh -l -c"
> > >
> > > (define (make-docker-exec-form prefix value)
> > >   (cond
> > >((equal? value '())
> > > '())
> > >((equal? prefix '())
> > > (string-split value #\space))
> > >(else
> > > (let ((values (string-split value #\space)))
> > >   (cons
> > >(string-append prefix "/" (car values))
> > >(cdr values))
> >
> > If I read this right (sorry, still somewhat new to guile), you basically 
> > split
> > the --entry-point argument on spaces and use those parts as separate values 
> > to
> > invoke, is that correct?  If so, how would you pass a binary that has space 
> > in
> > the name (joke example: `/bin/ba sh') into the entry-point?
> 
> Basically, yes, and you are right about the problem.
> 
> I looked through all the guix documentation I could find and the only
> other place I saw that a list was passed in an option was for URLs and
> they were separated by spaces.
> 
> > > And replacing the setting of entry-point in the build-docker-image call 
> > > to:
> > >
> > > #:entry-point (make-docker-exec-form
> > > #$profile #$entry-point)
> > >
> > > The call to build-docker-image takes a list for entry-point, and it
> > > all works fine as far as I can tell.
> > >
> > > Before I send in a patch, some questions:
> > >
> > > Am I missing something?
> > >
> > > Am I on the right track?
> >
> > In my opinion (which you are free to disagree with :) ), I think it would be
> > better to either have /bin/sh as an entry-point (and pass -l -c as arguments
> > when starting the container, if required) or create a wrapper script 
> > /bin/shlc
> > that would exec /bin/sh with correct arguments.
> 
> Yep, lots of possible workarounds, but it seems to me that it would be
> better spending the time adjusting the pack command to fit the spec.
> 
> > Few random ideas: Maybe the same format Containerfiles use for cmd and
> > entrypoint directives could be used?  Maybe the --entry-point could also (in
> > addition to a string) accept a list of strings (LISP list)?
> 
> Sounds good to me. Do you have a reference for the json for this? (Not
> a big deal as I think I've worked it out from the code, but it's
> always nice to have the specs...)
> 
> From the Dockerfile reference for ENTRYPOINT
> https://docs.docker.com/engine/reference/builder/#entrypoint there are
> two fomrs:
> 
> ENTRYPOINT ["executable", "param1", "param2"] # The exec form, which
> is the preferred form:
> 
> ENTRYPOINT command param1 param2 # The shell form:
> 
> To implement the shell form I'd need to update build-docker-image in
> guix/docker.scm
> https://git.savannah.gnu.org/cgit/guix.git/tree/guix/docker.scm#n139
> to take a string instead of/ as well as the list it currently takes.
> Then update docker-image in guix/scripts/pack.scm
> https://git.savannah.gnu.org/cgit/guix.git/tree/guix/scripts/pack.scm#n592
> 
> Invocation would then simply be --entry-point="command param1 param2"
> 
> To implement the exec form (preferred according to docker) I wouldn't
> need to touch guix/docker.scm, but I would probably need to change the
> parsing for --entry-point as well as updating docker-iimge.

I did not kno

Re: guix docker on gitlab-ci

[wolf]

On 2023-05-31 18:47:03 +0100, Graham Addis wrote:
> Hi Wolf,
> 
> On Tue, 30 May 2023 at 07:52, Graham Addis  wrote:
> >
> > ENTRYPOINT ["executable", "param1", "param2"] # The exec form, which
> > is the preferred form:
> 
> I realised that pack takes multiple symlink(s) using
> 
> --symlink /bin=bin --symlink /opt=opt
> 
> I could use the equivalent format for --entry-point
> 
> --entry-point executable --entry-point param1 --entry-point param2
> 
> This seems the most consistent approach with the current implementation so 
> far.
> 
> Thoughts?
> 

I think that is a reasonable idea.  Only downside is that it would not be
backwards compatible (since currently last --entry-point wins), however I am not
sure if someone actually relies on that behavior.

Backwards compatible way would be keeping --entry-point as it is and introducing
new argument (--entry-point-arg) that could be used to build the argument list,
but I might be overthinking it :).

W.

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.


signature.asc
Description: PGP signature

Re: guix docker on gitlab-ci

[Graham Addis]

Hi Wolf,

On Tue, 30 May 2023 at 07:52, Graham Addis  wrote:
>
> ENTRYPOINT ["executable", "param1", "param2"] # The exec form, which
> is the preferred form:

I realised that pack takes multiple symlink(s) using

--symlink /bin=bin --symlink /opt=opt

I could use the equivalent format for --entry-point

--entry-point executable --entry-point param1 --entry-point param2

This seems the most consistent approach with the current implementation so far.

Thoughts?

Re: guix docker on gitlab-ci

[Graham Addis]

Hi Worf,

Thanks for the response, see below.

On Mon, 29 May 2023 at 20:41, wolf  wrote:
>
> On 2023-05-24 18:04:47 +0100, Graham Addis wrote:
> > Dear people,
> >
> > I tried to create a docker image to use in a gitlab-ci instance but it
> > failed because I couldn't use --entry-point="bin/sh -l -c" or
> > equivalent, basically the gitlab-runner complains that it can't run
> > binaries.
>
> Would this be better using just bin/sh for the entry point and passing the -l
> and -c as an arguments?

Probably, but I don't think that's an option in gitlab ci and anyway
it would be nice to support the docker options.

> > I've managed to get it working by making some changes to 
> > guix/scripts/pack.scm
> >
> > Adding a fn in docker-image, just before the call to
> > build-docker-image, to create a list from the string passed in from
> > --entry-point="bin/sh -l -c"
> >
> > (define (make-docker-exec-form prefix value)
> >   (cond
> >((equal? value '())
> > '())
> >((equal? prefix '())
> > (string-split value #\space))
> >(else
> > (let ((values (string-split value #\space)))
> >   (cons
> >(string-append prefix "/" (car values))
> >(cdr values))
>
> If I read this right (sorry, still somewhat new to guile), you basically split
> the --entry-point argument on spaces and use those parts as separate values to
> invoke, is that correct?  If so, how would you pass a binary that has space in
> the name (joke example: `/bin/ba sh') into the entry-point?

Basically, yes, and you are right about the problem.

I looked through all the guix documentation I could find and the only
other place I saw that a list was passed in an option was for URLs and
they were separated by spaces.

> > And replacing the setting of entry-point in the build-docker-image call to:
> >
> > #:entry-point (make-docker-exec-form
> > #$profile #$entry-point)
> >
> > The call to build-docker-image takes a list for entry-point, and it
> > all works fine as far as I can tell.
> >
> > Before I send in a patch, some questions:
> >
> > Am I missing something?
> >
> > Am I on the right track?
>
> In my opinion (which you are free to disagree with :) ), I think it would be
> better to either have /bin/sh as an entry-point (and pass -l -c as arguments
> when starting the container, if required) or create a wrapper script /bin/shlc
> that would exec /bin/sh with correct arguments.

Yep, lots of possible workarounds, but it seems to me that it would be
better spending the time adjusting the pack command to fit the spec.

> Few random ideas: Maybe the same format Containerfiles use for cmd and
> entrypoint directives could be used?  Maybe the --entry-point could also (in
> addition to a string) accept a list of strings (LISP list)?

Sounds good to me. Do you have a reference for the json for this? (Not
a big deal as I think I've worked it out from the code, but it's
always nice to have the specs...)

>From the Dockerfile reference for ENTRYPOINT
https://docs.docker.com/engine/reference/builder/#entrypoint there are
two fomrs:

ENTRYPOINT ["executable", "param1", "param2"] # The exec form, which
is the preferred form:

ENTRYPOINT command param1 param2 # The shell form:

To implement the shell form I'd need to update build-docker-image in
guix/docker.scm
https://git.savannah.gnu.org/cgit/guix.git/tree/guix/docker.scm#n139
to take a string instead of/ as well as the list it currently takes.
Then update docker-image in guix/scripts/pack.scm
https://git.savannah.gnu.org/cgit/guix.git/tree/guix/scripts/pack.scm#n592

Invocation would then simply be --entry-point="command param1 param2"

To implement the exec form (preferred according to docker) I wouldn't
need to touch guix/docker.scm, but I would probably need to change the
parsing for --entry-point as well as updating docker-iimge.

I prefer the second option, for which all I need is some guidance on
the option syntax

.e.g. --entry-point=["command", "param1", "param2"]

Suggestions please. :)

I could implement both and test for a string or a list and choose
between the shell and exec forms from there, but to be consistent with
the existing implementation.

Once I'm clear about the best approach for this, I could add the CMD
too, if that would be useful.
https://docs.docker.com/engine/reference/builder/#cmd

One strange thing, I couldn't see the need for prefixing the profile
to the ENTRYPOINT command:
https://git.savannah.gnu.org/cgit/guix.git/tree/guix/scripts/pack.scm#n670
I took it out and everything seems to work, so I'm not sure what
problem it is fixing. Anybody any idea?

Thanks,

Graham

Re: guix docker on gitlab-ci

[wolf]

On 2023-05-24 18:04:47 +0100, Graham Addis wrote:
> Dear people,
> 
> I tried to create a docker image to use in a gitlab-ci instance but it
> failed because I couldn't use --entry-point="bin/sh -l -c" or
> equivalent, basically the gitlab-runner complains that it can't run
> binaries.

Would this be better using just bin/sh for the entry point and passing the -l
and -c as an arguments?

> 
> I've managed to get it working by making some changes to guix/scripts/pack.scm
> 
> Adding a fn in docker-image, just before the call to
> build-docker-image, to create a list from the string passed in from
> --entry-point="bin/sh -l -c"
> 
> (define (make-docker-exec-form prefix value)
>   (cond
>((equal? value '())
> '())
>((equal? prefix '())
> (string-split value #\space))
>(else
> (let ((values (string-split value #\space)))
>   (cons
>(string-append prefix "/" (car values))
>(cdr values))

If I read this right (sorry, still somewhat new to guile), you basically split
the --entry-point argument on spaces and use those parts as separate values to
invoke, is that correct?  If so, how would you pass a binary that has space in
the name (joke example: `/bin/ba sh') into the entry-point?

> 
> And replacing the setting of entry-point in the build-docker-image call to:
> 
> #:entry-point (make-docker-exec-form
> #$profile #$entry-point)
> 
> The call to build-docker-image takes a list for entry-point, and it
> all works fine as far as I can tell.
> 
> Before I send in a patch, some questions:
> 
> Am I missing something?
> 
> Am I on the right track?

In my opinion (which you are free to disagree with :) ), I think it would be
better to either have /bin/sh as an entry-point (and pass -l -c as arguments
when starting the container, if required) or create a wrapper script /bin/shlc
that would exec /bin/sh with correct arguments.

Few random ideas: Maybe the same format Containerfiles use for cmd and
entrypoint directives could be used?  Maybe the --entry-point could also (in
addition to a string) accept a list of strings (LISP list)?

> 
> Should I be splitting this out before the call to docker-image?
> 
> Thanks,
> 
> Graham
> 

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.


signature.asc
Description: PGP signature

guix docker on gitlab-ci

[Graham Addis]

Dear people,

I tried to create a docker image to use in a gitlab-ci instance but it
failed because I couldn't use --entry-point="bin/sh -l -c" or
equivalent, basically the gitlab-runner complains that it can't run
binaries.

I've managed to get it working by making some changes to guix/scripts/pack.scm

Adding a fn in docker-image, just before the call to
build-docker-image, to create a list from the string passed in from
--entry-point="bin/sh -l -c"

(define (make-docker-exec-form prefix value)
  (cond
   ((equal? value '())
'())
   ((equal? prefix '())
(string-split value #\space))
   (else
(let ((values (string-split value #\space)))
  (cons
   (string-append prefix "/" (car values))
   (cdr values))

And replacing the setting of entry-point in the build-docker-image call to:

#:entry-point (make-docker-exec-form
#$profile #$entry-point)

The call to build-docker-image takes a list for entry-point, and it
all works fine as far as I can tell.

Before I send in a patch, some questions:

Am I missing something?

Am I on the right track?

Should I be splitting this out before the call to docker-image?

Thanks,

Graham

Re: Gitlab CI

[Stephen Scheck]

Ludovic Courtès writes:

> Hello! I’m late to the party but I’m interested in this discussion… :-)

> GitLab CI can run jobs directly in a Docker image:
>   https://docs.gitlab.com/ce/ci/docker/using_docker_images.html

> So it would be interesting to have any easy way for people to create a Docker 
> image for this purpose.

I’ve updated my Alpine-based Docker image, and I think it should work now for 
many common use cases,
but it would be great if it saw some more testing. Details here:

   https://hub.docker.com/r/singularsyntax/guix-bootstrap

Cheers -ss

Re: Gitlab CI

[divoplade]

Le lundi 27 juillet 2020 à 09:46 +0200, Ludovic Courtès a écrit :
> > I prefer the solution in which the pack is a .tar.gz that is
> > directly
> > untarred to /; this way it can be run on a shared unprivileged
> > runner
> > and the storage can be achieved in the runner itself, via the
> > gitlab CI
> > cache.
> 
> Ah yes, I guess that would also work, and it’s in fact less tedious.
> That amounts to running https://guix.gnu.org/install.sh every time
> though, no?

Pretty much, except "every time" is really less often than expected,
because the gitlab runner can cache files between jobs. So with an
appropriate tar command, it is possible to pack the whole store plus
the /var/guix and untar it at the beginning of the job (or re-run the
guix install if the cache has been dropped). Or run "guix pack".

There is a very unfortunate problem that running "guix pull" or guix
installing something will require the "ntp" debian package, so each job
requires apt-get update and apt-get install -y ntp... I am
investigating how to cache the ntp package (and its dependencies) to
avoid running apt-get update, but that's quite disappointing.

divoplade

Re: Gitlab CI

[Ludovic Courtès]

Hi,

divoplade  skribis:

> Le vendredi 24 juillet 2020 à 16:30 +0200, Ludovic Courtès a écrit :
>> 
>> So presumably a better option is to use an image created like this:
>> 
>>   guix pack -f docker -S /bin=bin guix --localstatedir
>> 
> As you may know, most gitlab server will not let you use the provided
> docker registry, and so the only option will be to push to docker hub.

Yes.  What I had in mind was that you would create that image, upload it
to a registry GitLab-CI can pull from, and then refer to that image in
your ‘.gitlab-ci.yml’.

> I prefer the solution in which the pack is a .tar.gz that is directly
> untarred to /; this way it can be run on a shared unprivileged runner
> and the storage can be achieved in the runner itself, via the gitlab CI
> cache.

Ah yes, I guess that would also work, and it’s in fact less tedious.
That amounts to running https://guix.gnu.org/install.sh every time
though, no?

Thanks,
Ludo’.

Re: Gitlab CI

[divoplade]

Hello,

Le vendredi 24 juillet 2020 à 16:30 +0200, Ludovic Courtès a écrit :
> 
> So presumably a better option is to use an image created like this:
> 
>   guix pack -f docker -S /bin=bin guix --localstatedir
> 
As you may know, most gitlab server will not let you use the provided
docker registry, and so the only option will be to push to docker hub.
Moreover, running this command as part of a CI job just to push it to
docker hub will require a privileged container. So you will need to
either provide your own runner, or provide you own docker registry
(docker hub or else).

I prefer the solution in which the pack is a .tar.gz that is directly
untarred to /; this way it can be run on a shared unprivileged runner
and the storage can be achieved in the runner itself, via the gitlab CI
cache.

divoplade

Re: Gitlab CI

[Ludovic Courtès]

Hello!

I’m late to the party but I’m interested in this discussion…  :-)

Jérémy Korwin-Zmijowski  skribis:

> I am experimenting with Gitlab CI for a Guile project I am working on.
> It is managed with Hall and I would like to benefit the Guix features.

GitLab CI can run jobs directly in a Docker image:

  https://docs.gitlab.com/ce/ci/docker/using_docker_images.html

So it would be interesting to have any easy way for people to create a
Docker image for this purpose.

Colleagues of mine tried ‘guix system docker-image’ for that but that
turns out to not quite work (IIRC gitlab-runner tries to do ‘docker exec
your build command’, which fails here because those images have shepherd
as their entry point).

So presumably a better option is to use an image created like this:

  guix pack -f docker -S /bin=bin guix --localstatedir

and then, in the image, you first need to spawn guix-daemon, perhaps
with ‘--disable-chroot’ because the image lacks build users.

It would be useful also to set the user under which commands run in the
image, similar to the ‘USER’ directive in Dockerfiles.

Thoughts?

It would be nice to make that as smooth as possible and to add a recipe
in the cookbook!

Ludo’.

Re: Gitlab CI

[Stephen Scheck]

Jérémy Korwin-Zmijowski  writes:

> Hey Guixters !
>
> I am experimenting with Gitlab CI for a Guile project I am working on.
> It is managed with Hall and I would like to benefit the Guix features.
>
> Does anyone has done that before ?
> Is there a way to get a Guix docker image instead of the Debian one ?

Hello Jérémy,

I have been trying to accomplish a similar goal - using Guix in GitLab CI for a 
Guile project. To that end, I’ve built some Docker images you might find 
interesting:

Guix in Alpine Linux
https://hub.docker.com/r/singularsyntax/guix-bootstrap/tags
https://gitlab.com/singularsyntax-docker-hub/guix-bootstrap (the Dockerfile and 
pipeline for deployment to Docker Hub)

My goal is to enable typical Docker workflows with Guix, for example something 
like this Dockerfile:

FROM gnu/guix:1.1.0
RUN guix package —install 

Then use the resulting Docker image in the build, test, etc. stages of a GitLab 
CI/CD pipeline.

Unfortunately, Guix does not have a natural fit with Docker containerization 
since `guix-daemon` must be running for any Guix commands to work. I haven’t 
been able to get any Dockerfile workflows like above working, since `docker 
build` seems to skip execution of the regular image entry point command which 
starts `guix-daemon`. I think to make it work, you’d need to provide some kind 
of shell wrapper around the `guix` command which first starts `guix-daemon`, 
waits for it to be ready, then runs Guix commands. But you can use the Alpine 
image for Docker-in-Docker workflows where you first start a container with 
`docker run`, then issue successive `docker exec` commands to perform work 
inside the container, then extract the results from the container for use in 
other stages of a CI/CD pipeline.

I also attempted to build a “pure” Guix container not requiring installation on 
top of another Linux distribution by using `guix system docker-image <…>`. I 
even had a GitLab pipeline making daily builds which you might find interesting 
(but I recently disabled it as I’ll explain below):

https://hub.docker.com/r/singularsyntax/guix/tags
https://gitlab.com/singularsyntax-docker-hub/guix (GitLab CI/CD pipeline for 
deployment to Docker Hub)

It works fine for obtaining a sandbox for which to experiment interactively 
with Guix, but has the same problems as the Alpine image, and is even a little 
worse because there are no “anchor” binaries inside the container, not even 
`/bin/sh`, until the container init process establishes symlinks into the 
`/gnu/store` tree. If you try to use it in scripts or CI/CD pipelines, they 
often fail due to race conditions with the init script, and without `/bin/sh` 
it’s difficult to reliably assert any pre-conditions inside the container. So 
it’s a real pain to work with, and I think of little utility for scripting as 
it currently stands.

Anyway, hopefully this provides some more insight, and good luck!

Re: Gitlab CI

[Pierre Neidhardt]

Jérémy Korwin-Zmijowski  writes:

> I the meantime I would like to try to put a Guix's docker image on
> dockerhub and use it in Framagit/Gitlab if it is possible...

This would be fantastic, please let us know about your progress :)

-- 
Pierre Neidhardt
https://ambrevar.xyz/


signature.asc
Description: PGP signature

Re: Debian package with Gitlab CI

[divoplade]

Hello,

Le lundi 06 juillet 2020 à 14:25 +0200, Jérémy Korwin-Zmijowski a
écrit :
> Thank you for the new yaml file ! Unfortunately, I didn't manage to
> use
> this one yet. I have to do more experiments I guess.

I do, too. In the mean time, I think we would be very happy to have a
debian package. There is some work done for that (
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=850644), but I'm not
familiar enough with the debian process to understand if we can use
anything of it yet. In any case, there is no guix package in the docker
image debian:experimental for now...

The best thing would be to just apt-get install guix (caching the
packages so as not to download it again) and simply start the daemon.

Re: Gitlab CI

[Jérémy Korwin-Zmijowski]

Le jeudi 25 juin 2020 à 23:32 +0200, divoplade a écrit :
> Hello Jérémy,
> 
> I have a new solution (it is quite different from the previous one)
> in
> which it is possible to run guix commands within a job, while still
> caching most of the files (see attached). I am sorry, my fork of your
> project is already a mess so I prefer to put the file in its entirety
> here.
> 
> Anyways, everything will be way easier once we have a proper debian
> package for guix:
> 
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=850644
> 
> divoplade

Hi divoplade !

Thank you for the new yaml file ! Unfortunately, I didn't manage to use
this one yet. I have to do more experiments I guess.

I the meantime I would like to try to put a Guix's docker image on
dockerhub and use it in Framagit/Gitlab if it is possible...

Cheers

Jérémy

Re: Gitlab CI

[divoplade]

Hello Jérémy,

I have a new solution (it is quite different from the previous one) in
which it is possible to run guix commands within a job, while still
caching most of the files (see attached). I am sorry, my fork of your
project is already a mess so I prefer to put the file in its entirety
here.

Anyways, everything will be way easier once we have a proper debian
package for guix:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=850644

divoplade

Le samedi 20 juin 2020 à 10:42 +0200, Jérémy Korwin-Zmijowski a écrit :
> Hey Guixters !
> 
> I am experimenting with Gitlab CI for a Guile project I am working
> on.
> It is managed with Hall and I would like to benefit the Guix
> features.
> 
> Currently, I struggle to run Guix inside the default Debian stable
> docker image (framagit.org).
> 
> Here is my .gitlab-ci.yml :
> 
>before_script:
> - apt -qq update
> - apt -qq install -y wget gpg xz-utils
> - wget 
>
> https://git.savannah.gnu.org/cgit/guix.git/plain/etc/guix-install.sh
>  - chmod +x guix-install.sh
>  - wget https://sv.gnu.org/people/viewgpg.php?user_id=15145 -qO - |
> gpg --import -
>  - printf '\n' | ./guix-install.sh
>  - /root/.config/guix/current/bin/guix-daemon --build-users-
> group=guixbuild
>  - guix pull
> 
> stages:
>   - bootstrap
>   - configure
>   - build
>   - test
> 
> job:bootstrap:
>   stage: bootstrap
>   script: autoreconf -vfi
> 
> What I get :
> 
>[1592641832.155]: [ FAIL ] unsupported init system; run the daemon
>manually:
>   /root/.config/guix/current/bin/guix-daemon --build-users-
>group=guixbuild
>[1592641832.155]: [ INFO ] making the guix command available to
>other users
> 
> Does anyone has done that before ?
> Is there a way to get a Guix docker image instead of the Debian one ?
> 
> Have a nice weekend !
> 
> Jérémy
> 
> 


.gitlab-ci.yml
Description: application/yaml

Re: Gitlab CI

[Jérémy Korwin-Zmijowski]

Hi Zimoun !

Thank you for taking time to help over here !

Thanks to divoplade, we succedded to make something up for relative
simple projects (I don't know how it goes for more complicated ones).
If anybody feel interested :
https://framagit.org/Jeko/guile-dummy

Cheers,
Jérémy

Re: Gitlab CI

[zimoun]

Hi Jérémy,

On Sat, 20 Jun 2020 at 10:42, Jérémy Korwin-Zmijowski 
 wrote:

>  - printf '\n' | ./guix-install.sh

[...]

>[1592641832.155]: [ FAIL ] unsupported init system; run the daemon
>manually:
>   /root/.config/guix/current/bin/guix-daemon --build-users-
>group=guixbuild
>[1592641832.155]: [ INFO ] making the guix command available to
>other users
>
> Does anyone has done that before ?

It is because the guix-install.sh script does not recognize the init
system of the Docker image, see 'sys_enable_guix_daemon()'.  Roughly
speaking, it does not fall into upstart) neither systemd) neither
sysv-init) so it falls into NA|*).

I do not know how the Docker image you use is built.

> Is there a way to get a Guix docker image instead of the Debian one ?

The easiest is, IMHO, instead of "./guix-install.sh", you should run all
the 8 steps described in [1] adapting the step #5.

1: https://guix.gnu.org/manual/devel/en/guix.html#Binary-Installation


Cheers,
simon

Re: Gitlab CI

[Jérémy Korwin-Zmijowski]

I tried to apply the caching as you did, but I error during project
configuration.

https://framagit.org/Jeko/guile-dummy/-/jobs/1029866

Jérémy

Le mardi 23 juin 2020 à 12:15 +0200, Jérémy Korwin-Zmijowski a écrit :
> Hey divoplade !
> 
> Thank you very much for sharing your hack ! The caching section is
> very
> interesting... Does it save time between runs ?
> 
> After a lot of attempts, I ended up with a green pipeline but it
> takes
> 43 minute to complete haha 
> https://framagit.org/Jeko/guile-dummy/-/pipelines
> 
> Jérémy
> 
> 
> 
> 
> 
> 

Re: Gitlab CI

[Jérémy Korwin-Zmijowski]

Hey divoplade !

Thank you very much for sharing your hack ! The caching section is very
interesting... Does it save time between runs ?

After a lot of attempts, I ended up with a green pipeline but it takes
43 minute to complete haha 
https://framagit.org/Jeko/guile-dummy/-/pipelines

Jérémy

Re: Gitlab CI

[Jérémy Korwin-Zmijowski]

Le samedi 20 juin 2020 à 15:40 +0200, Ricardo Wurmus a écrit :
> 
> Can you specify a different Docker image?  Then you could create one
> containing Guix with “guix pack -f docker”.
> 

I am not the owmer of the Gitlab instance, so I think I can't... But I
can ask to the admin haha. Worth the shot !

Thank you Ricardo !

Re: Gitlab CI

[divoplade]

Hello Jérémy,

The problem is that guix-install.sh asks you whether you want to
authorize the substitutes, but it can't because it is not used
interactively, so it fails.

There are other traps to avoid (do not forget to install ntp, call
guix-daemon with --disable-chroot because you are in an unprivileged
container, cache things to avoid pulling the world at each commit). I
have a solution based on caching a guix pack containing all your
packages: https://framagit.org/divoplade/guix-project

variables:
  GUIX_REF: "--branch=master"
  PACKAGES: "guile-hall hello"

before_script:
  - test -f cache.tar.gz || apt -qq update
  - test -f cache.tar.gz || apt -qq install -y wget gpg xz-utils ntp
  - test -f cache.tar.gz || wget 
https://sv.gnu.org/people/viewgpg.php?user_id=15145 -qO - | gpg --
import -
  - test -f cache.tar.gz || wget 
https://git.savannah.gnu.org/cgit/guix.git/plain/etc/guix-install.sh
  - test -f cache.tar.gz || chmod ugo+x guix-install.sh
  - test -f cache.tar.gz || (echo "" ; echo "y") | ./guix-install.sh ||
echo "Assume OK"
  - test -f cache.tar.gz || source
/root/.config/guix/current/etc/profile
  - test -f cache.tar.gz || guix-daemon --build-users-group=guixbuild
--disable-chroot &
  - test -f cache.tar.gz || guix pull $GUIX_REF
  - test -f cache.tar.gz || cp $(guix pack -C gzip $PACKAGES
-S/opt/bin=bin) cache.tar.gz
  - cp cache.tar.gz /
  - (cd / && tar xf cache.tar.gz)
  - export PATH=/opt/bin:$PATH

cache:
  key: "$GUIX_REF $PACKAGES"
  paths:
- cache.tar.gz

job:
  script:
- hello
- hall --help

After a very long set up (1/2 hour with a somewhat fast runner), you
will leverage the cached guix pack to run the jobs instantly, with the
default gitlab runner (unprivileged, debian-based).

If you want to update, clear the runner caches, or change the GUIX_REF
variable. You can pin a specific commit (better for reproducibility,
not so great for releases) with --commit=COMMIT.

I could not get guix to run within the job, even with "guix" added to
PACKAGES: it would need this in the script:

- apt -qq update
- apt -qq install -y ntp
- (groupadd --system guixbuild && for i in $(seq 1 10) ; do useradd
-g guixbuild -G guixbuild -d /var/empty -s $(which nologin) -c "Guix
build user $i" --system guixbuilder$i; done) || true
- guix-daemon --build-users-group=guixbuild --disable-chroot &
- guix archive --authorize < $(dirname $(readlink
/usr/local/bin/guix))/../share/guix/ci.guix.info.pub

However, if you try it, you will notice a crash.

I am looking forward to having guix shipped as a debian package, it
will be way easier.

divoplade

Le samedi 20 juin 2020 à 10:42 +0200, Jérémy Korwin-Zmijowski a écrit :
> Hey Guixters !
> 
> I am experimenting with Gitlab CI for a Guile project I am working
> on.
> It is managed with Hall and I would like to benefit the Guix
> features.
> 
> Currently, I struggle to run Guix inside the default Debian stable
> docker image (framagit.org).
> 
> Here is my .gitlab-ci.yml :
> 
>before_script:
> - apt -qq update
> - apt -qq install -y wget gpg xz-utils
> - wget 
>
> https://git.savannah.gnu.org/cgit/guix.git/plain/etc/guix-install.sh
>  - chmod +x guix-install.sh
>  - wget https://sv.gnu.org/people/viewgpg.php?user_id=15145 -qO - |
> gpg --import -
>  - printf '\n' | ./guix-install.sh
>  - /root/.config/guix/current/bin/guix-daemon --build-users-
> group=guixbuild
>  - guix pull
> 
> stages:
>   - bootstrap
>   - configure
>   - build
>   - test
> 
> job:bootstrap:
>   stage: bootstrap
>   script: autoreconf -vfi
> 
> What I get :
> 
>[1592641832.155]: [ FAIL ] unsupported init system; run the daemon
>manually:
>   /root/.config/guix/current/bin/guix-daemon --build-users-
>group=guixbuild
>[1592641832.155]: [ INFO ] making the guix command available to
>other users
> 
> Does anyone has done that before ?
> Is there a way to get a Guix docker image instead of the Debian one ?
> 
> Have a nice weekend !
> 
> Jérémy
> 
> 

Re: Gitlab CI

[Ricardo Wurmus]

Jérémy Korwin-Zmijowski  writes:

> Hey Guixters !
>
> I am experimenting with Gitlab CI for a Guile project I am working on.
> It is managed with Hall and I would like to benefit the Guix features.
>
> Currently, I struggle to run Guix inside the default Debian stable
> docker image (framagit.org).

Can you specify a different Docker image?  Then you could create one
containing Guix with “guix pack -f docker”.

-- 
Ricardo

Gitlab CI

[Jérémy Korwin-Zmijowski]

Hey Guixters !

I am experimenting with Gitlab CI for a Guile project I am working on.
It is managed with Hall and I would like to benefit the Guix features.

Currently, I struggle to run Guix inside the default Debian stable
docker image (framagit.org).

Here is my .gitlab-ci.yml :

   before_script:
- apt -qq update
- apt -qq install -y wget gpg xz-utils
- wget 
   https://git.savannah.gnu.org/cgit/guix.git/plain/etc/guix-install.sh
 - chmod +x guix-install.sh
 - wget https://sv.gnu.org/people/viewgpg.php?user_id=15145 -qO - |
gpg --import -
 - printf '\n' | ./guix-install.sh
 - /root/.config/guix/current/bin/guix-daemon --build-users-
group=guixbuild
 - guix pull

stages:
  - bootstrap
  - configure
  - build
  - test

job:bootstrap:
  stage: bootstrap
  script: autoreconf -vfi

What I get :

   [1592641832.155]: [ FAIL ] unsupported init system; run the daemon
   manually:
  /root/.config/guix/current/bin/guix-daemon --build-users-
   group=guixbuild
   [1592641832.155]: [ INFO ] making the guix command available to
   other users

Does anyone has done that before ?
Is there a way to get a Guix docker image instead of the Debian one ?

Have a nice weekend !

Jérémy