Re: [arch-dev-public] Arch Linux Docker / Vagrant: Current situation

[Pierre Schmitz]

Great. I tried to define it briefly in the README:
https://github.com/archlinux/archlinux-docker#purpose I am open to any
suggestions though.

side note: The "release process" is described within a Makefile at
https://github.com/pierres/archiso-manager

On Mon, Mar 12, 2018 at 7:06 PM, Santiago Torres-Arias via
arch-dev-public  wrote:
> Understood.
>
> Although I don't exactly know what's the "original purpose" I'll try to
> make sure no big radical changes are made without consensus from the
> community :)
>
> Thanks both of you!
> -Santiago.
>
> On Mon, Mar 12, 2018 at 03:41:56PM +0100, Pierre Schmitz wrote:
>> Yeah, that was the "and please don't get it the wrong way" part which
>> obviously did not work. I thought I just put this out there as I
>> already got PRs and mails from different people who wanted to make the
>> image more minimal by removing files from packages or provide
>> different images for all kind of uses cases.
>>
>> I just wanted to avoid frustration beforehand but not discourage your
>> work. I probably failed on this one :-)
>>
>> Greetings,
>>
>> Pierre
>>
>> On Mon, Mar 12, 2018 at 8:19 AM, Bartłomiej Piotrowski via
>> arch-dev-public  wrote:
>> > On 2018-03-12 05:33, Pierre Schmitz wrote:
>> >> Thanks for digging this up again. You may use the github issue or
>> >> project system to plan the different steps. Also (and please don't get
>> >> it the wrong way) let's keep the purpose I intended for our Docker
>> >> image intact.
>> >
>> > No one has suggested changing "the purpose". I'm not sure what is
>> > unclear in Santiago's mail.
>> >
>> > Bartłomiej

Re: [arch-dev-public] Arch Linux Docker / Vagrant: Current situation

[Santiago Torres-Arias via arch-dev-public]

Understood.

Although I don't exactly know what's the "original purpose" I'll try to
make sure no big radical changes are made without consensus from the
community :)

Thanks both of you!
-Santiago.

On Mon, Mar 12, 2018 at 03:41:56PM +0100, Pierre Schmitz wrote:
> Yeah, that was the "and please don't get it the wrong way" part which
> obviously did not work. I thought I just put this out there as I
> already got PRs and mails from different people who wanted to make the
> image more minimal by removing files from packages or provide
> different images for all kind of uses cases.
> 
> I just wanted to avoid frustration beforehand but not discourage your
> work. I probably failed on this one :-)
> 
> Greetings,
> 
> Pierre
> 
> On Mon, Mar 12, 2018 at 8:19 AM, Bartłomiej Piotrowski via
> arch-dev-public  wrote:
> > On 2018-03-12 05:33, Pierre Schmitz wrote:
> >> Thanks for digging this up again. You may use the github issue or
> >> project system to plan the different steps. Also (and please don't get
> >> it the wrong way) let's keep the purpose I intended for our Docker
> >> image intact.
> >
> > No one has suggested changing "the purpose". I'm not sure what is
> > unclear in Santiago's mail.
> >
> > Bartłomiej


signature.asc
Description: PGP signature

Re: [arch-dev-public] Arch Linux Docker / Vagrant: Current situation

[Pierre Schmitz]

Yeah, that was the "and please don't get it the wrong way" part which
obviously did not work. I thought I just put this out there as I
already got PRs and mails from different people who wanted to make the
image more minimal by removing files from packages or provide
different images for all kind of uses cases.

I just wanted to avoid frustration beforehand but not discourage your
work. I probably failed on this one :-)

Greetings,

Pierre

On Mon, Mar 12, 2018 at 8:19 AM, Bartłomiej Piotrowski via
arch-dev-public  wrote:
> On 2018-03-12 05:33, Pierre Schmitz wrote:
>> Thanks for digging this up again. You may use the github issue or
>> project system to plan the different steps. Also (and please don't get
>> it the wrong way) let's keep the purpose I intended for our Docker
>> image intact.
>
> No one has suggested changing "the purpose". I'm not sure what is
> unclear in Santiago's mail.
>
> Bartłomiej

Re: [arch-dev-public] Arch Linux Docker / Vagrant: Current situation

[Bartłomiej Piotrowski via arch-dev-public]

On 2018-03-12 05:33, Pierre Schmitz wrote:
> Thanks for digging this up again. You may use the github issue or
> project system to plan the different steps. Also (and please don't get
> it the wrong way) let's keep the purpose I intended for our Docker
> image intact.

No one has suggested changing "the purpose". I'm not sure what is
unclear in Santiago's mail.

Bartłomiej

Re: [arch-dev-public] Arch Linux Docker / Vagrant: Current situation

[Pierre Schmitz]

Thanks for digging this up again. You may use the github issue or
project system to plan the different steps. Also (and please don't get
it the wrong way) let's keep the purpose I intended for our Docker
image intact.

Greetings,

Pierre

On Mon, Mar 12, 2018 at 12:01 AM, Santiago Torres-Arias via
arch-dev-public  wrote:
> On Fri, Mar 09, 2018 at 09:46:48PM +0100, Bartłomiej Piotrowski via 
> arch-dev-public wrote:
>> On 2018-01-29 20:29, Pierre Schmitz wrote:
>> > * I did not look into the details of how we exactly need to proceed
>> > with making an "official" image. A few pull requests or some kind of
>> > setp-by-step plan (wiki or github) would help.
>>
>> Necrobumping. You actually quoted the message from Santiago that
>> describes the process. I'm going to grant him and Christian write
>> permissions so they can start working on this on separate branch.
>
> Thanks!
>
> I'll dig this up from my pile of things to do now.
>
> -Santiago.

Re: [arch-dev-public] Arch Linux Docker / Vagrant: Current situation

[Santiago Torres-Arias via arch-dev-public]

On Fri, Mar 09, 2018 at 09:46:48PM +0100, Bartłomiej Piotrowski via 
arch-dev-public wrote:
> On 2018-01-29 20:29, Pierre Schmitz wrote:
> > * I did not look into the details of how we exactly need to proceed
> > with making an "official" image. A few pull requests or some kind of
> > setp-by-step plan (wiki or github) would help.
> 
> Necrobumping. You actually quoted the message from Santiago that
> describes the process. I'm going to grant him and Christian write
> permissions so they can start working on this on separate branch.

Thanks!

I'll dig this up from my pile of things to do now.

-Santiago.


signature.asc
Description: PGP signature

Re: [arch-dev-public] Arch Linux Docker / Vagrant: Current situation

[Bartłomiej Piotrowski via arch-dev-public]

On 2018-01-29 20:29, Pierre Schmitz wrote:
> * I did not look into the details of how we exactly need to proceed
> with making an "official" image. A few pull requests or some kind of
> setp-by-step plan (wiki or github) would help.

Necrobumping. You actually quoted the message from Santiago that
describes the process. I'm going to grant him and Christian write
permissions so they can start working on this on separate branch.

Bartłomiej

Re: [arch-dev-public] Arch Linux Docker / Vagrant: Current situation

[Pierre Schmitz]

About the ISO bus factor:
* I just recently put the whole process into a simple script to make
it easier for anybody else to build ISOs. Unfortunately at least the
signing process requires some manual work. See
https://github.com/pierres/archiso-manager

About the official Docker Image:
* The docker image archlinux/base is pretty young and I did some
significant changes to its content lately. I am pretty happy with the
result right now but some more field testing shouldn't hurt. On my
virtual todo list is also figuring out whether we could reuse the
created archlinux.tar for other containers as well
* I did not look into the details of how we exactly need to proceed
with making an "official" image. A few pull requests or some kind of
setp-by-step plan (wiki or github) would help.

I am glad the interest in Arch within Docker and other containers is increasing.

Greetings,

Pierre

On Mon, Jan 29, 2018 at 8:20 PM, Santiago Torres-Arias via
arch-dev-public  wrote:
>> > The official images projects info is on [1] and [2] if you want to read
>> > more in-depth/updated information. I'll summarize here though:
>> >
>> > 1) A TU/Arch Linux "affiliate" submits a PR to the official images
>> > repository, which basically contains the following:
>> > 1. A tag name/image name
>> > 2. A sha256/ref of a commit/tag containig the image's information 
>> > on
>> > *another* repository (in this case, our official dockerr image 
>> > repo)
>> > 3. Image building instructions.
>>
>> A PR to this repository is also required, not sure if you mentioned it
>> :) [1]
>
> Right, I omitted that one for the sake of brevity.
>
> Thanks!
> -Santiago.

Re: [arch-dev-public] Arch Linux Docker / Vagrant: Current situation

[Santiago Torres-Arias via arch-dev-public]

> > The official images projects info is on [1] and [2] if you want to read
> > more in-depth/updated information. I'll summarize here though:
> > 
> > 1) A TU/Arch Linux "affiliate" submits a PR to the official images
> > repository, which basically contains the following:
> > 1. A tag name/image name
> > 2. A sha256/ref of a commit/tag containig the image's information on
> > *another* repository (in this case, our official dockerr image repo)
> > 3. Image building instructions.
> 
> A PR to this repository is also required, not sure if you mentioned it
> :) [1]

Right, I omitted that one for the sake of brevity.

Thanks!
-Santiago.


signature.asc
Description: PGP signature

Re: [arch-dev-public] Arch Linux Docker / Vagrant: Current situation

[Jelle van der Waa]

On 01/29/18 at 12:31pm, Santiago Torres-Arias via arch-dev-public wrote:
> Hi,
> 
> Sorry I've been quite sick (to the point of barely having energy to look
> at the computer). I'm back on my feet now though :)
> 
> > > Sangy/Santiago[3] was so nice to speak with the docker guys. They said
> > > they would approve our docker image and we could move it to the other
> > > official images[4]. But for this we need to do some changes on our
> > > docker repository on github. (As long I understood sangy correct it
> > > would be just some new branches).
> >
> > Can you actually give more details how it's going to look like?
> >
> 
> The official images projects info is on [1] and [2] if you want to read
> more in-depth/updated information. I'll summarize here though:
> 
> 1) A TU/Arch Linux "affiliate" submits a PR to the official images
> repository, which basically contains the following:
> 1. A tag name/image name
> 2. A sha256/ref of a commit/tag containig the image's information on
> *another* repository (in this case, our official dockerr image repo)
> 3. Image building instructions.

A PR to this repository is also required, not sure if you mentioned it
:) [1]

-- 
Jelle van der Waa

[1] https://github.com/docker-library/docs


signature.asc
Description: PGP signature

Re: [arch-dev-public] Arch Linux Docker / Vagrant: Current situation

[Santiago Torres-Arias via arch-dev-public]

Hi,

Sorry I've been quite sick (to the point of barely having energy to look
at the computer). I'm back on my feet now though :)

> > Sangy/Santiago[3] was so nice to speak with the docker guys. They said
> > they would approve our docker image and we could move it to the other
> > official images[4]. But for this we need to do some changes on our
> > docker repository on github. (As long I understood sangy correct it
> > would be just some new branches).
>
> Can you actually give more details how it's going to look like?
>

The official images projects info is on [1] and [2] if you want to read
more in-depth/updated information. I'll summarize here though:

1) A TU/Arch Linux "affiliate" submits a PR to the official images
repository, which basically contains the following:
1. A tag name/image name
2. A sha256/ref of a commit/tag containig the image's information on
*another* repository (in this case, our official dockerr image repo)
3. Image building instructions.
2) In parallel, we put this information on our repository. At least, a
rootfs and a Dockerfile (as otherdistros do).
3) once the PR is updated, it will fetch our rootfs and Dockerfile (and
other relevant info), build the docker image, and perform some quality
checks on it.
4) The image is published as an "official image" on the dockerhub.

The benefits from this is that industry/paranoid users often don't trust
non-official images to build upon. Also, if I recall correctly, official
images are periodically scanned for vulnerabilities, and (IIRC)
signed with the docker-controlled signing keys, so they can be used with
docker content trust[3].

I think it'd be not too difficult to schedule script the rootfs build
process in the same way we do with boxes right now, publish these as
tags and then update the official dockerfile repositories.

Sorry for the delay.

Cheers!
-Santiago.

[1] https://docs.docker.com/docker-hub/official_repos/
[2] https://github.com/docker-library/official-images/
[3] https://blog.docker.com/2015/08/content-trust-docker-1-8/


signature.asc
Description: PGP signature

Re: [arch-dev-public] Arch Linux Docker / Vagrant: Current situation

[Florian Pritz via arch-dev-public]

On 25.01.2018 02:02, Christian Rebischke wrote:
> Isn't this more like a workaround?
> I would really appreciate automated ISO image builds or at least the
> option that more than one person is able to generate these images.

Reducing the bus factor should really be a goal for us, so yes, it's a
workaround.

Florian



signature.asc
Description: OpenPGP digital signature

Re: [arch-dev-public] Arch Linux Docker / Vagrant: Current situation

[Christian Rebischke via arch-dev-public]

On Tue, Jan 23, 2018 at 10:58:48AM +0100, Public mailing list for Arch Linux 
development wrote:
> On 21.01.2018 20:50, Christian Rebischke via arch-dev-public wrote:
> > I would definitly
> > prefer to have the vagrant boxes around the same date as we release our
> > ISO images.
> 
> How about letting it run every day and adding a quick check to the
> script that only starts the build when the iso is available and no image
> has been built yet?
> 
> Florian
> 

Isn't this more like a workaround?
I would really appreciate automated ISO image builds or at least the
option that more than one person is able to generate these images.


Chris


signature.asc
Description: PGP signature

Re: [arch-dev-public] Arch Linux Docker / Vagrant: Current situation

[Florian Pritz via arch-dev-public]

On 21.01.2018 20:50, Christian Rebischke via arch-dev-public wrote:
> I would definitly
> prefer to have the vagrant boxes around the same date as we release our
> ISO images.

How about letting it run every day and adding a quick check to the
script that only starts the build when the iso is available and no image
has been built yet?

Florian



signature.asc
Description: OpenPGP digital signature

Re: [arch-dev-public] Arch Linux Docker / Vagrant: Current situation

[Eli Schwartz via arch-dev-public]

On 01/21/2018 02:39 PM, Christian Rebischke via arch-dev-public wrote:
> No idea about the bootstrap image. Is there a big difference between the
> bootstrap image and `pacstrap` in some random directory?

There is no difference, the bootstrap image is what you use to pacstrap.

The issue is merely that getting a pacman binary compiled for some other
linux distro (in case you wish to install Arch from within your
currently existing Linux distribution that isn't Arch) is not
necessarily easy, and in the event that it isn't, the bootstrap image
can be used instead as the bare minimum environment to enter a chroot
and run the Arch pacman package.

Well, and it is easy enough to pacstrap from, say, Gentoo, which I know
for a fact has a pacman ebuild for managing Arch installations from
Gentoo: https://packages.gentoo.org/packages/sys-apps/pacman

-- 
Eli Schwartz
Bug Wrangler and Trusted User



signature.asc
Description: OpenPGP digital signature

Re: [arch-dev-public] Arch Linux Docker / Vagrant: Current situation

[Christian Rebischke via arch-dev-public]

On Sun, Jan 21, 2018 at 07:41:53PM +0100, Public mailing list for Arch Linux 
development wrote:
> It would be probably seen as more "official" if it was mentioned on our
> website.

Absolutly! I will see what I can do to bring that on our website.

> > My first goal has been to add some hypervisors, but due to
> > the fact that we have only libvirt and virtualbox in our repositories I
> > have dismissed this plan. The automated build process works fine so far
> > (except some issues with qemu[2] and the dependency on punctual iso
> > releases on soyuz. The latter should we definitly fix. Currently the iso
> > images are build manually. Can we automate this somehow? I really rely
> > on punctual releases otherwise the automated build will fail and
> > somebody needs to trigger the build again manually. Happened about 1-2
> > times..)
> 
> This is something that would require a virtually offline box to be fully
> automated. For the time being, it sounds better to upgrade your vagrant
> image after first week of the month.

Actually I do this already. The systemd timer for the automated builds
is set on the 5th of each month (round about a week). Normally the ISO
was there on 1st or 2nd each month, but in the last months its more like
6th or 9th. I could move that timer on 14th, but I would definitly
prefer to have the vagrant boxes around the same date as we release our
ISO images.

> 
> Also, any plans to expand the scope to support OpenStack, AWS and GCE?
> At the moment the scope sounds rather limited.

It's possible to build images for OpenStack, AWS and GCE, but as far as
I understand the docs for it we would need an OpenStack cloud for
OpenStack images, an AWS account for generating Amazon EC2 Images and a
Google Account for generating GCE Images[1], because the packer connects
to these services to generate the image.

> Can you actually give more details how it's going to look like?
Sangy worked on this. He explained it one day, but It's already some
months ago. I hope he will attend our discussion here and give us a
short summary about that process.



signature.asc
Description: PGP signature

Re: [arch-dev-public] Arch Linux Docker / Vagrant: Current situation

[Christian Rebischke via arch-dev-public]

On Sun, Jan 21, 2018 at 03:23:44PM +0100, Jelle van der Waa wrote:
> I'm all for automatic builds, and the improvement where multiple people
> know how the ISO's are build & released. How would we however sign these
> builds? And is the bootstrap image also generated in the same manner?
Well, we have two options:

Option 1: More people can build ISOs and more people can sign the ISO.

Option 2: We automate the process and sign the ISO on our server with a
new public key. Thats how nixOS or voidlinux do this.


No idea about the bootstrap image. Is there a big difference between the
bootstrap image and `pacstrap` in some random directory?

> I or others can grant write access to the repository, see the owners on
> the Github organization. I would however first want to ask pierres about
> it since he irregurarly is on IRC.
> Are these docker builds going to be automated as well btw? Oh and
> testing images :-)

I would love to automate that as well, but I don't know a good process
for it yet. My first thought was that we automate docker via packer as
well, but unfortunately packer doesn't support docker builds. It
shouldn't be that difficult to automate it with some bash magic..

> This would be nice, how do we share the credentials to the docker login
> though? Or can we make multiple people owners?

Just register at dockerhub and I can add you to the Arch Linux Group.
(this applies to every other developer as well. If you want access just
register an account and message me)


signature.asc
Description: PGP signature

Re: [arch-dev-public] Arch Linux Docker / Vagrant: Current situation

[Bartłomiej Piotrowski via arch-dev-public]

On 2018-01-20 20:19, Christian Rebischke via arch-dev-public wrote:
> The Arch Linux Vagrant images are currently be build for libvirt and
> virtualbox. We have over 3800 downloads at the moment and slowly
> catching up to the community based arch linux vagrant images.[1]

It would be probably seen as more "official" if it was mentioned on our
website.

> My first goal has been to add some hypervisors, but due to
> the fact that we have only libvirt and virtualbox in our repositories I
> have dismissed this plan. The automated build process works fine so far
> (except some issues with qemu[2] and the dependency on punctual iso
> releases on soyuz. The latter should we definitly fix. Currently the iso
> images are build manually. Can we automate this somehow? I really rely
> on punctual releases otherwise the automated build will fail and
> somebody needs to trigger the build again manually. Happened about 1-2
> times..)

This is something that would require a virtually offline box to be fully
automated. For the time being, it sounds better to upgrade your vagrant
image after first week of the month.

Also, any plans to expand the scope to support OpenStack, AWS and GCE?
At the moment the scope sounds rather limited.

> Sangy/Santiago[3] was so nice to speak with the docker guys. They said
> they would approve our docker image and we could move it to the other
> official images[4]. But for this we need to do some changes on our
> docker repository on github. (As long I understood sangy correct it
> would be just some new branches).

Can you actually give more details how it's going to look like?

Bartłomiej

Re: [arch-dev-public] Arch Linux Docker / Vagrant: Current situation

[Jelle van der Waa]

On 01/20/18 at 08:19pm, Christian Rebischke via arch-dev-public wrote:
> Hello Everybody,
> It's now over a half year ago that I've started working together with
> sangy and pierre on our vagrant and docker images. I would like to give
> you a short update on this topic.
> 
> 
> The Arch Linux Vagrant images are currently be build for libvirt and
> virtualbox. We have over 3800 downloads at the moment and slowly
> catching up to the community based arch linux vagrant images.[1]
> 
> My first goal has been to add some hypervisors, but due to
> the fact that we have only libvirt and virtualbox in our repositories I
> have dismissed this plan.

Sounds good enough, this is also easier to manage.

> The automated build process works fine so far
> (except some issues with qemu[2] and the dependency on punctual iso
> releases on soyuz. The latter should we definitly fix. Currently the iso
> images are build manually. Can we automate this somehow? I really rely
> on punctual releases otherwise the automated build will fail and
> somebody needs to trigger the build again manually. Happened about 1-2
> times..)

I'm all for automatic builds, and the improvement where multiple people
know how the ISO's are build & released. How would we however sign these
builds? And is the bootstrap image also generated in the same manner?

> The other topic is the docker image. Has anybody of you contact to
> pierre? He doesn't answer my mails and he is the only one with access to
> the repository on github.

I or others can grant write access to the repository, see the owners on
the Github organization. I would however first want to ask pierres about
it since he irregurarly is on IRC.

Are these docker builds going to be automated as well btw? Oh and
testing images :-)
 
> Sangy/Santiago[3] was so nice to speak with the docker guys. They said
> they would approve our docker image and we could move it to the other
> official images[4]. But for this we need to do some changes on our
> docker repository on github. (As long I understood sangy correct it
> would be just some new branches).

This would be nice, how do we share the credentials to the docker login
though? Or can we make multiple people owners?

[1] https://github.com/orgs/archlinux/people

-- 
Jelle van der Waa


signature.asc
Description: PGP signature

[arch-dev-public] Arch Linux Docker / Vagrant: Current situation

[Christian Rebischke via arch-dev-public]

Hello Everybody,
It's now over a half year ago that I've started working together with
sangy and pierre on our vagrant and docker images. I would like to give
you a short update on this topic.


The Arch Linux Vagrant images are currently be build for libvirt and
virtualbox. We have over 3800 downloads at the moment and slowly
catching up to the community based arch linux vagrant images.[1]

My first goal has been to add some hypervisors, but due to
the fact that we have only libvirt and virtualbox in our repositories I
have dismissed this plan. The automated build process works fine so far
(except some issues with qemu[2] and the dependency on punctual iso
releases on soyuz. The latter should we definitly fix. Currently the iso
images are build manually. Can we automate this somehow? I really rely
on punctual releases otherwise the automated build will fail and
somebody needs to trigger the build again manually. Happened about 1-2
times..)


The other topic is the docker image. Has anybody of you contact to
pierre? He doesn't answer my mails and he is the only one with access to
the repository on github.

Sangy/Santiago[3] was so nice to speak with the docker guys. They said
they would approve our docker image and we could move it to the other
official images[4]. But for this we need to do some changes on our
docker repository on github. (As long I understood sangy correct it
would be just some new branches).


best regards,

chris / shibumi



[1] https://app.vagrantup.com/archlinux/boxes/archlinux
[2] https://github.com/hashicorp/packer/issues/5769
[3] https://www.archlinux.org/people/support-staff/#sangy
[4] https://hub.docker.com/explore/


signature.asc
Description: PGP signature