Re: [arch-projects] [arch-devops] [dbscripts] [PATCH 2/4] Add reproducible archive of packages.
On Wed, Dec 05, 2018 at 10:49:44AM +0100, Florian Pritz via arch-devops wrote: > Also thinking about this, it would be great if we could skip the pkg > symlinks for each day's archive and only copy the db itself. All we'd > need is to have a dedicated PackageServer= setting (like Server=, but > only for packages, not for the database) for pacman to find the > packages, but I'm not sure if Allan would like that. That setting would > also have to support the pkgname substring and the pkgname obviously. As discussed on IRC, we don't actually need support in pacman here. We can just set up a rewrite in nginx so that when pacman tries to download a package file, nginx maps the path correctly. So nginx would rewrite /$repo/os/$arch/package-tar.xz to /packages/p/package/pacakge-tar.xz. Pacman would still work then and the only difference would be that we don't have directory listings with the packages of each day, but who needs those anyways. You can get all that info from the dbs themselves. Florian signature.asc Description: PGP signature
Re: [arch-projects] [arch-devops] [dbscripts] [PATCH 2/4] Add reproducible archive of packages.
On 12/12/18 3:55 AM, Jelle van der Waa wrote: > On 12/04/18 at 01:15pm, Eli Schwartz via arch-devops wrote: >> On 12/4/18 1:09 PM, Eli Schwartz wrote: >>> Whenever adding new package files to the pool of distributed packages, >>> hardlink a copy of every package it was built with, into a >>> "reproducible" pool, and log which file required it. > > Does this also clean up the archive? As in remove packages which are not > required for reproducible builds? Since now our archive server is almost > running out of space again. Patch 4/4 will do so, but I only cc'ed patch 2/4 to the devops list. -- Eli Schwartz Bug Wrangler and Trusted User signature.asc Description: OpenPGP digital signature
Re: [arch-projects] [arch-devops] [dbscripts] [PATCH 2/4] Add reproducible archive of packages.
On 12/04/18 at 01:15pm, Eli Schwartz via arch-devops wrote:
> On 12/4/18 1:09 PM, Eli Schwartz wrote:
> > Whenever adding new package files to the pool of distributed packages,
> > hardlink a copy of every package it was built with, into a
> > "reproducible" pool, and log which file required it.
Does this also clean up the archive? As in remove packages which are not
required for reproducible builds? Since now our archive server is almost
running out of space again.
> The question becomes, where can I store these? As-is, this will burden
> the mirror network as well. Unsure how to handle this. Could this be
> configurable by the mirror, as ISOs are now? Should we exclusively
> self-host this, and if so, where?
>
> archive.archlinux.org is managed by another service with its own
> exclusively writable location.
>
> > Signed-off-by: Eli Schwartz
> > ---
> > config | 1 +
> > config.local.svn-community | 1 +
> > config.local.svn-packages | 1 +
> > db-functions | 49 +++---
> > db-update | 4
> > 5 files changed, 48 insertions(+), 8 deletions(-)
> >
> > diff --git a/config b/config
> > index 1cfe11f4..5144fca7 100644
> > --- a/config
> > +++ b/config
> > @@ -3,6 +3,7 @@
> > FTP_BASE="/srv/ftp"
> > PKGREPOS=()
> > PKGPOOL=''
> > +EXTRA_PKGPOOLS=()
> > SRCPOOL=''
> > TESTING_REPO=''
> > STABLE_REPOS=()
> > diff --git a/config.local.svn-community b/config.local.svn-community
> > index 5d61b5ea..15bcc17f 100644
> > --- a/config.local.svn-community
> > +++ b/config.local.svn-community
> > @@ -2,6 +2,7 @@
> >
> > PKGREPOS=('community' 'community-testing' 'community-staging' 'multilib'
> > 'multilib-testing' 'multilib-staging')
> > PKGPOOL='pool/community'
> > +EXTRA_PKGPOOLS=('pool/packages')
> > SRCPOOL='sources/community'
> > SVNREPO='file:///srv/repos/svn-community/svn'
> > SVNUSER='svn-community'
> > diff --git a/config.local.svn-packages b/config.local.svn-packages
> > index 34aab35c..75986b65 100644
> > --- a/config.local.svn-packages
> > +++ b/config.local.svn-packages
> > @@ -2,6 +2,7 @@
> >
> > PKGREPOS=('core' 'extra' 'testing' 'staging' 'kde-unstable'
> > 'gnome-unstable')
> > PKGPOOL='pool/packages'
> > +EXTRA_PKGPOOLS=('pool/community')
> > SRCPOOL='sources/packages'
> > SVNREPO='file:///srv/repos/svn-packages/svn'
> > SVNUSER='svn-packages'
> > diff --git a/db-functions b/db-functions
> > index 7aeedced..2b1ae87a 100644
> > --- a/db-functions
> > +++ b/db-functions
> > @@ -165,20 +165,23 @@ repo_unlock () { #repo_unlock
> > fi
> > }
> >
> > +# usage: _grep_all_info pkgfile infofile key
> > +_grep_all_info() {
> > + local _ret=()
> > +
> > + mapfile -t _ret < <(/usr/bin/bsdtar -xOqf "$1" "${2}" | grep "^${3} = ")
> > +
> > + printf '%s\n' "${_ret[@]#${3} = }"
> > +}
> > +
> > # usage: _grep_pkginfo pkgfile pattern
> > _grep_pkginfo() {
> > - local _ret
> > -
> > - _ret="$(/usr/bin/bsdtar -xOqf "$1" .PKGINFO | grep "^${2} = " | tail
> > -1)"
> > - echo "${_ret#${2} = }"
> > + _grep_all_info "${1}" .PKGINFO "${2}" | tail -1
> > }
> >
> > # usage: _grep_buildinfo pkgfile pattern
> > _grep_buildinfo() {
> > - local _ret
> > -
> > - _ret="$(/usr/bin/bsdtar -xOqf "$1" .BUILDINFO | grep "^${2} = " | tail
> > -1)"
> > - echo "${_ret#${2} = }"
> > + _grep_all_info "${1}" .BUILDINFO "${2}" | tail -1
> > }
> >
> > # Get the package base or name as fallback
> > @@ -444,4 +447,34 @@ arch_repo_modify() {
> > REPO_MODIFIED=1
> > }
> >
> > +# Build an index of dependent packages needed by a given pkgfile
> > +# usage: make_reproducible pkgfile [check]
> > +make_reproducible() {
> > + local pkg dir pkgs=() pkgfile pkgfiles=()
> > +
> > + mapfile -t pkgs < <(_grep_all_info "${1}" .BUILDINFO installed)
> > +
> > + for pkg in "${pkgs[@]}"; do
> > + for dir in "${FTP_BASE}/${PKGPOOL}"
> > "${EXTRA_PKGPOOLS[@]/#/${FTP_BASE}/}" "${STAGING}"/**/; do
> > + if pkgfile="$(getpkgfile "${dir}/${pkg}"${PKGEXTS}
> > 2>/dev/null)"; then
> > + pkgfiles+=("${pkgfile}")
> > + continue 2
> > + fi
> > + done
> > + error "could not find existing package for %s" "${pkg}"
> > + return 1
> > + done
> > +
> > + if [[ ${2} = check ]]; then
> > + return 0
> > + fi
> > +
> > + for pkg in "${pkgfiles[@]}"; do
> > + if [[ ! -f ${FTP_BASE}/${PKGPOOL}-reproducible/${pkg##*/} ]];
> > then
> > + ln -L "${pkg}"
> > "${FTP_BASE}/${PKGPOOL}-reproducible/${pkg##*/}"
> > + fi
> > + echo "${1}" >>
> > "${FTP_BASE}/${PKGPOOL}-reproducible/${pkg##*/}.buildlinks"
> > + done
> > +}
> > +
> > . "$(dirname "$(readlink -e "${BASH_SOURCE[0]}")")/db-functions-${VCS}"
> > diff --git a/db-update b/db-update
> > index 313fb999..11ec185f 100755
> > --- a/db-update
> > +++ b/db-update
> > @@
Re: [arch-projects] [arch-devops] [dbscripts] [PATCH 2/4] Add reproducible archive of packages.
On Tue, Dec 04, 2018 at 01:15:20PM -0500, Eli Schwartz via arch-devops
wrote:
> On 12/4/18 1:09 PM, Eli Schwartz wrote:
> > Whenever adding new package files to the pool of distributed packages,
> > hardlink a copy of every package it was built with, into a
> > "reproducible" pool, and log which file required it.
>
> The question becomes, where can I store these? As-is, this will burden
> the mirror network as well. Unsure how to handle this. Could this be
> configurable by the mirror, as ISOs are now? Should we exclusively
> self-host this, and if so, where?
I'm not a fan of adding this pool to the mirror root for multiple
reasons:
- Most mirrors would likely want to avoid mirroring it because it can
become quite large and we told them that we only need around 100GB.
If everyone wants to exclude it that requires action by every admin.
Not ideal.
- I'm not sure if all of our mirrors have hardlink support. We don't
currently ask for it even though we suggest the -H rsync option. Also
the current repos use symlinks for the packages instead of hardlinks.
That said, I'm not even sure if rsync can detect hardlinks across
directories. It can't even detect renames/moves across directories...
- I don't expect that we need to mirror it because we don't even get
that many requests to our current archive. If we ever need to mirror
it, we can worry about that later I'd say since moving it to the
mirror root should be rather simple.
I'd suggest to make the base path of the repro pool configurable so that
we can keep it out of the mirror root. For now I'd suggest something
like this:
REPRO_BASE="/srv/reproducible-archive/"
pkgname="foo"
pkgfile="foo-1.0-1.pkg.tar.xz"
dest="$REPRO_BASE/packages/${pkgname:0:1}/$pkgname/$pkgfile"
ln .. "$dest"
Also note that this does intentionally not include $PKGPOOL any more
even though you include it in your patch. The archive doesn't have it
and I don't think it really helps anyone. It will just cause confusion
if packages are moved between repos and it makes using the archive more
difficult because the user would have to check all possible pool names
or know which one to check.
Ideally I'd like to later extend this to also include the current
archive's features and from the looks of it, storing the packages like
this is the first step. Then we just need to copy the repos (dbs and pkg
symlinks) once a day and archive the ISOs.
Also thinking about this, it would be great if we could skip the pkg
symlinks for each day's archive and only copy the db itself. All we'd
need is to have a dedicated PackageServer= setting (like Server=, but
only for packages, not for the database) for pacman to find the
packages, but I'm not sure if Allan would like that. That setting would
also have to support the pkgname substring and the pkgname obviously.
Comments/thoughts/patches/... welcome.
Florian
signature.asc
Description: PGP signature
