Re: [arch-projects] [dbscripts] Licensing of the dbscripts repository under the GPL-2.0-or-later
I hereby confirm that I would like my work on dbscripts to be licensed under the terms of the GPL-2.0 or any later version of the license. Florian On Fri, Jun 05, 2020 at 12:34:45AM -0400, Eli Schwartz wrote: > Hi all, > > I'm one of the current maintainers of the Arch Linux dbscripts [1]. > Unfortunately, it did not used to have a license, and to fix that, I > have agreed with Pierre Schmitz, Aaron Griffin, Judd Vinet, and Dan > McGee (who between us cover 581/715 of the commits in the repository) to > license it under the terms of the GPL-2.0-or-later [2]. > > If you're getting this email (which is cross-posted to the arch-projects > mailing list) then you've contributed changes in the past as well, which > I would appreciate a positive affirmation that you consent to your > contribution(s) being distributed under this license as well. > > (If there is a different license you'd prefer, then I will have to see > what we can do, but I'm hoping that won't be necessary...) > > Here's the (deduplicated) list of `git shortlog` committers: > > Abhishek Dasgupta > Allan McRae > Andrea Scarpino > Daniel J Griffiths > Dave Reisner > Eric Bélanger > Florian Pritz > François Charette > Henning Garus > Jan Alexander Steffens (heftig) > Jan de Groot > Jelle van der Waa > Johannes Löthberg > Levente Polyak > Luke Shumaker > Rémy Oudompheng > Scott Horowitz > Simo Leone > Thomas Bächler > Travis Willard > Xavier Chantry > > > -- > Eli Schwartz > Bug Wrangler and Trusted User > > > [1] https://git.archlinux.org/dbscripts.git/about/ > https://github.com/archlinux/dbscripts > > [2] https://spdx.org/licenses/GPL-2.0-or-later.html > > > > > signature.asc Description: PGP signature
[arch-projects] [dbscripts] [GIT] Official repo DB scripts annotated tag 20190713 created. 20190713
This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "Official repo DB scripts". The annotated tag, 20190713 has been created at 00331c0ff67496af5279413da029ade368f30e88 (tag) tagging 0706db69d48e8fa9423587f0d11727075b0d3d5f (commit) replaces 20190113 tagged by Florian Pritz on Sat Jul 13 21:02:28 2019 +0200 - Log - Hotfix for reproducibility errors -BEGIN PGP SIGNATURE- iQIzBAABCgAdFiEEz6avFeXHQUn8HYwIbRZVwUzhwT4FAl0qKt8ACgkQbRZVwUzh wT7cexAAh/0WcvDpTqXj8EHtfRpjhdcbg+hi+IGZH5kHmNCYe3GigzBMYJHtNOj1 wmSd8330t8xagSJUL6Rb/50FNiAzmfnCFOgbfxpvoYFrk1LliBdk5iYld36xtk0k garw6SanlUiwFlr+G/RrXapiQfLBTzxMtJIW3S7HAFBv1xLStuQo/9cw5r2RNv6Z ohad/W8qm3Dtg6+TOxaKWSV2rZ2i3FJ8bbBJS/UtSjcXukz72sAMLJu51R8Nx4oe Y3lXbs7wZmTrjQ2NY9zPhXYUIKn702OcXrY0eu/2iKZKoatyfwRgE3JedU3mFhN4 Tvriu84viLjP+kcHJwdycltZwXekDFZzncVO3xPV1lB5JBMfZ7ac8cWLG4ncsvC9 hnu7s03qmYCx5DLS5p+AqEhhhOz9N8Zb2lwBvk0sx6F04VpWD6+EqYWTxl0AziIM oHxFfWDhtYWD1lyIhVOpvlgWAapE1SEbMxKh2rMIlSVV4QyaZyn86uwe0zCbgwnD aS8xCNFiPlSTPgh8lpQpNEvwki+teeq0tgDA72r5rkQLjMjLAgjpdhbQqYwGYJri vbzqsjwpXwSvKWszxEWT7a7Br+o0pjjQG6Q68pS8kOiWVJ36wj5sv3chxSQt2S3K 8mY/iupXkUHr4Rp6npdpmO95P2aCih13Gb6A3VO8W5WHqkpNZF8= =ILOb -END PGP SIGNATURE- Florian Pritz (1): Look for repro packages in live repo pool too --- hooks/post-receive -- Official repo DB scripts
[arch-projects] [dbscripts] [GIT] Official repo DB scripts branch master updated. 20190713
This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "Official repo DB scripts". The branch, master has been updated via 0706db69d48e8fa9423587f0d11727075b0d3d5f (commit) from f2e8d57e7fde40fd5a3e80e1f152ae850ac3c369 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log - commit 0706db69d48e8fa9423587f0d11727075b0d3d5f Author: Florian Pritz Date: Sat Jul 13 20:58:07 2019 +0200 Look for repro packages in live repo pool too I've cleaned older packages from the archive, but sometimes we do not rebuild packages in a long long time. We still keep have them in the repository, but this check does not look for the package there, thus when trying to db-update, the user sees an error. We fix this by also looking at currently live packages instead of only relying on the archive. This is mostly a hotfix until a better solution is created. Depending on when/how the ftpdir-cleanup cronjob removes such packages, users may still see errors when an old package is updated and the cronjob removes it from the repository before db-update is run. Signed-off-by: Florian Pritz --- Summary of changes: db-functions | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) hooks/post-receive -- Official repo DB scripts
Re: [arch-projects] [netctl] Extended testing for 1.20
On Tue, Feb 26, 2019 at 11:16:22AM -0500, Eli Schwartz via arch-projects wrote: > On 2/26/19 11:07 AM, Jouke Witteveen via arch-projects wrote: > > I have not seen any new bug reports over the last two weeks. I suggest > > we move 1.20 to [core]. > > We also have five signoffs from the > https://wiki.archlinux.org/index.php/Arch_Testing_Team Moved. Thanks! Florian signature.asc Description: PGP signature
Re: [arch-projects] [netctl] Extended testing for 1.20
On Tue, Feb 12, 2019 at 05:20:13PM +0100, Jouke Witteveen wrote: > I just tagged 1.20, which switches from using wpa_actiond to using wpa_cli. > Let's keep this in [testing] for at least two weeks to see whether > there is anything noticeable from this change. Sounds good. Since I'm not really in the loop, please ping me when you think it's time to move netctl to core and I'll do it (assuming I don't hear about breakage). Thanks for the CC! Florian signature.asc Description: PGP signature
Re: [arch-projects] [dbscripts] [PATCH v3 2/2] Add reproducible archive of packages.
On Wed, Jan 09, 2019 at 05:01:21PM -0500, Eli Schwartz via arch-projects wrote: > v3: string changes, consistently use BASH_SOURCE Looks good to me. Thanks! Florian signature.asc Description: PGP signature
Re: [arch-projects] [dbscripts] [PATCH v2 2/2] Add reproducible archive of packages.
On Wed, Jan 09, 2019 at 10:45:12AM -0500, Eli Schwartz via arch-projects
wrote:
> diff --git a/db-update b/db-update
> index 313fb999..04a29bf3 100755
> --- a/db-update
> +++ b/db-update
> @@ -61,6 +61,9 @@ for repo in "${repos[@]}"; do
> if ! check_builddir "${pkg}"; then
> die "Package %s was not built in a
> chroot" "$repo/${pkg##*/}"
> fi
> +if ! check_reproducible "${pkg}"; then
> +die "Package %s is not reproducible"
> "${pkg}"
> >>>
> >>> Same as above. I'd suggest something like this:
> >>>
> >>> "Package %s depends on packages that are missing in the reproducibility
> >>> archive and your staging directory. Ensure that all dependencies either
> >>> exist in the repositories or reproducibility archive already or that
> >>> they are added together with the package in a single call to db-update."
>
> Hmm, what about:
>
> Package %s is not reproducible. Ensure that all dependencies are
> available in the repositories or are added in the same db-update.
Fine by me.
Florian
signature.asc
Description: PGP signature
Re: [arch-projects] [dbscripts] [PATCH v2 2/2] Add reproducible archive of packages.
On Wed, Jan 09, 2019 at 09:49:26AM -0500, Eli Schwartz via arch-projects
wrote:
> >> diff --git a/db-functions b/db-functions
> >> index 7aeedced..b8a00b90 100644
> >> --- a/db-functions
> >> +++ b/db-functions
> >> @@ -444,4 +447,24 @@ arch_repo_modify() {
> >>REPO_MODIFIED=1
> >> }
> >>
> >> +# Verify the existence of dependent packages needed by a given pkgfile
> >> +# usage: check_reproducible pkgfile
> >> +check_reproducible() {
> >> + local pkg dir pkgs=() pkgfile pkgfiles=()
> >> +
> >> + mapfile -t pkgs < <(_grep_all_info "${1}" .BUILDINFO installed)
> >> +
> >> + for pkg in "${pkgs[@]}"; do
> >> + local pkgname=${pkg%-*-*-*}
> >> + for dir in "${ARCHIVE_BASE}/packages/${pkgname:0:1}/${pkgname}"
> >> "${STAGING}"/**/; do
> >> + if pkgfile="$(getpkgfile "${dir}/${pkg}"${PKGEXTS}
> >> 2>/dev/null)"; then
> >> + pkgfiles+=("${pkgfile}")
> >> + continue 2
> >> + fi
> >> + done
> >> + error "could not find existing package for %s" "${pkg}"
> >
> >
> > I imagine that I'd be confused if I ever saw this error. How about
> > clarifying it like this? "could not find package for dependency %s in
> > reproducibility archive or your staging directory"
>
> Maybe "existing or staged package for dependency %s"?
"could not find existing or staged package for dependency %s" is fine by
me.
>
> >> + return 1
> >> + done
> >> +}
> >> +
> >> . "$(dirname "$(readlink -e "${BASH_SOURCE[0]}")")/db-functions-${VCS}"
> >> diff --git a/db-update b/db-update
> >> index 313fb999..04a29bf3 100755
> >> --- a/db-update
> >> +++ b/db-update
> >> @@ -61,6 +61,9 @@ for repo in "${repos[@]}"; do
> >>if ! check_builddir "${pkg}"; then
> >>die "Package %s was not built in a chroot"
> >> "$repo/${pkg##*/}"
> >>fi
> >> + if ! check_reproducible "${pkg}"; then
> >> + die "Package %s is not reproducible" "${pkg}"
> >
> > Same as above. I'd suggest something like this:
> >
> > "Package %s depends on packages that are missing in the reproducibility
> > archive and your staging directory. Ensure that all dependencies either
> > exist in the repositories or reproducibility archive already or that
> > they are added together with the package in a single call to db-update."
>
> The two errors will only be called together. I think expanding the
> message when printing the missing dependency should be enough.
I get that, but I think that a user that sees these two message may not
understand that a missing dependency is related to the package being
reproducible. To be honest, I actually expected db-update to run all
checks and show all errors at once instead of terminating after the
first one. I now know that this is not the case. I'm pretty sure that I
would have treat these two messages as separate errors and that I'd then
be confused as to what the "second error" is actually about.
I think that it may save a lot of time and confusion if this error
message is clear about what is wrong and how it can be fixed. Most of
the other error messages in db-update are rather clear about the actual
problem. Maybe not as clear as the message I propose here, but clearer
than "Package %s is not reproducible". Apart from that, does it really
hurt to have a more verbose error message? It will only be shown if
there is an actual error and it doesn't influence normal usage. I'd say
we can afford to be more verbose in that case.
If you still think that this message should not be made more verbose,
I'd argue that it should be removed entirely. If we have just the
message about a dependency not being found, it is quite clear to a user
what is wrong and how they could fix the error. I'd say that is much
less confusing than if there were a second message about reproducibility
that some people may or may not consider to be a different, additional
error as I've explained above.
Florian
signature.asc
Description: PGP signature
Re: [arch-projects] [dbscripts] [PATCH v2 2/2] Add reproducible archive of packages.
On Tue, Jan 08, 2019 at 06:40:37PM -0500, Eli Schwartz via arch-projects
wrote:
> diff --git a/db-archive b/db-archive
> new file mode 100755
> index ..5680b9de
> --- /dev/null
> +++ b/db-archive
> @@ -0,0 +1,21 @@
> +#!/bin/bash
> +
> +. "$(dirname "$(readlink -e "$0")")/config"
This uses $0 (see below).
> +
> +if (( $# != 1 )); then
> + echo "usage: %s " "${0##*/}"
> + exit 1
> +fi
> +
> +if [[ -n ${ARCHIVEUSER} ]]; then
> + exec sudo -u "${ARCHIVEUSER}" bash "${BASH_SOURCE[0]}" "${@}"
This uses $BASH_SOURCE instead of $0 as used above. Is this intentional,
if so why? I'd argue that this should also use $0, but maybe I'm missing
something?
> +fi
> +
> +pkgfile=${1##*/}
> +pkgname=${pkgfile%-*-*-*}
> +archive_dir="${ARCHIVE_BASE}/packages/${pkgname:0:1}/${pkgname}"
> +
> +if [[ ! -f ${archive_dir}/${pkgfile} ]]; then
> + mkdir -p "${archive_dir}"
> + cp -np "${1}"{,.sig} "${archive_dir}/"
> +fi
> diff --git a/db-functions b/db-functions
> index 7aeedced..b8a00b90 100644
> --- a/db-functions
> +++ b/db-functions
> @@ -444,4 +447,24 @@ arch_repo_modify() {
> REPO_MODIFIED=1
> }
>
> +# Verify the existence of dependent packages needed by a given pkgfile
> +# usage: check_reproducible pkgfile
> +check_reproducible() {
> + local pkg dir pkgs=() pkgfile pkgfiles=()
> +
> + mapfile -t pkgs < <(_grep_all_info "${1}" .BUILDINFO installed)
> +
> + for pkg in "${pkgs[@]}"; do
> + local pkgname=${pkg%-*-*-*}
> + for dir in "${ARCHIVE_BASE}/packages/${pkgname:0:1}/${pkgname}"
> "${STAGING}"/**/; do
> + if pkgfile="$(getpkgfile "${dir}/${pkg}"${PKGEXTS}
> 2>/dev/null)"; then
> + pkgfiles+=("${pkgfile}")
> + continue 2
> + fi
> + done
> + error "could not find existing package for %s" "${pkg}"
I imagine that I'd be confused if I ever saw this error. How about
clarifying it like this? "could not find package for dependency %s in
reproducibility archive or your staging directory"
> + return 1
> + done
> +}
> +
> . "$(dirname "$(readlink -e "${BASH_SOURCE[0]}")")/db-functions-${VCS}"
> diff --git a/db-update b/db-update
> index 313fb999..04a29bf3 100755
> --- a/db-update
> +++ b/db-update
> @@ -61,6 +61,9 @@ for repo in "${repos[@]}"; do
> if ! check_builddir "${pkg}"; then
> die "Package %s was not built in a chroot"
> "$repo/${pkg##*/}"
> fi
> + if ! check_reproducible "${pkg}"; then
> + die "Package %s is not reproducible" "${pkg}"
Same as above. I'd suggest something like this:
"Package %s depends on packages that are missing in the reproducibility
archive and your staging directory. Ensure that all dependencies either
exist in the repositories or reproducibility archive already or that
they are added together with the package in a single call to db-update."
Florian
signature.asc
Description: PGP signature
Re: [arch-projects] [arch-devops] [dbscripts] [PATCH 2/4] Add reproducible archive of packages.
On Wed, Dec 05, 2018 at 10:49:44AM +0100, Florian Pritz via arch-devops wrote: > Also thinking about this, it would be great if we could skip the pkg > symlinks for each day's archive and only copy the db itself. All we'd > need is to have a dedicated PackageServer= setting (like Server=, but > only for packages, not for the database) for pacman to find the > packages, but I'm not sure if Allan would like that. That setting would > also have to support the pkgname substring and the pkgname obviously. As discussed on IRC, we don't actually need support in pacman here. We can just set up a rewrite in nginx so that when pacman tries to download a package file, nginx maps the path correctly. So nginx would rewrite /$repo/os/$arch/package-tar.xz to /packages/p/package/pacakge-tar.xz. Pacman would still work then and the only difference would be that we don't have directory listings with the packages of each day, but who needs those anyways. You can get all that info from the dbs themselves. Florian signature.asc Description: PGP signature
Re: [arch-projects] [arch-devops] [dbscripts] [PATCH 2/4] Add reproducible archive of packages.
On Tue, Dec 04, 2018 at 01:15:20PM -0500, Eli Schwartz via arch-devops
wrote:
> On 12/4/18 1:09 PM, Eli Schwartz wrote:
> > Whenever adding new package files to the pool of distributed packages,
> > hardlink a copy of every package it was built with, into a
> > "reproducible" pool, and log which file required it.
>
> The question becomes, where can I store these? As-is, this will burden
> the mirror network as well. Unsure how to handle this. Could this be
> configurable by the mirror, as ISOs are now? Should we exclusively
> self-host this, and if so, where?
I'm not a fan of adding this pool to the mirror root for multiple
reasons:
- Most mirrors would likely want to avoid mirroring it because it can
become quite large and we told them that we only need around 100GB.
If everyone wants to exclude it that requires action by every admin.
Not ideal.
- I'm not sure if all of our mirrors have hardlink support. We don't
currently ask for it even though we suggest the -H rsync option. Also
the current repos use symlinks for the packages instead of hardlinks.
That said, I'm not even sure if rsync can detect hardlinks across
directories. It can't even detect renames/moves across directories...
- I don't expect that we need to mirror it because we don't even get
that many requests to our current archive. If we ever need to mirror
it, we can worry about that later I'd say since moving it to the
mirror root should be rather simple.
I'd suggest to make the base path of the repro pool configurable so that
we can keep it out of the mirror root. For now I'd suggest something
like this:
REPRO_BASE="/srv/reproducible-archive/"
pkgname="foo"
pkgfile="foo-1.0-1.pkg.tar.xz"
dest="$REPRO_BASE/packages/${pkgname:0:1}/$pkgname/$pkgfile"
ln .. "$dest"
Also note that this does intentionally not include $PKGPOOL any more
even though you include it in your patch. The archive doesn't have it
and I don't think it really helps anyone. It will just cause confusion
if packages are moved between repos and it makes using the archive more
difficult because the user would have to check all possible pool names
or know which one to check.
Ideally I'd like to later extend this to also include the current
archive's features and from the looks of it, storing the packages like
this is the first step. Then we just need to copy the repos (dbs and pkg
symlinks) once a day and archive the ISOs.
Also thinking about this, it would be great if we could skip the pkg
symlinks for each day's archive and only copy the db itself. All we'd
need is to have a dedicated PackageServer= setting (like Server=, but
only for packages, not for the database) for pacman to find the
packages, but I'm not sure if Allan would like that. That setting would
also have to support the pkgname substring and the pkgname obviously.
Comments/thoughts/patches/... welcome.
Florian
signature.asc
Description: PGP signature
Re: [arch-projects] [netctl][PATCH] Makefile: Update upload URL
On 31.07.2017 15:05, Jouke Witteveen via arch-projects wrote: > Thanks! Now that systemd has a recent enough version in [core], netctl > 1.13 can move too (I am not aware of blocker bugs). Moved, thanks! Florian signature.asc Description: OpenPGP digital signature
Re: [arch-projects] [netctl][PATCH] Makefile: Update upload URL
On 14.03.2017 23:08, Jouke Witteveen wrote: > I noted that you did not use the generated PKGBUILD, but manually > updated the existing one. In doing so you missed that netctl now > depends on systemd>=233. Sorry about that. Fixed now Florian signature.asc Description: OpenPGP digital signature
[arch-projects] [netctl][PATCH] Makefile: Update upload URL
Signed-off-by: Florian Pritz --- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 702fb53..9418056 100644 --- a/Makefile +++ b/Makefile @@ -56,7 +56,7 @@ PKGBUILD: netctl-$(VERSION).tar.xz contrib/PKGBUILD.in $(lastword $^) > $@ upload: netctl-$(VERSION).tar.xz - scp $< $<.sig nymeria.archlinux.org:/srv/ftp/other/packages/netctl + scp $< $<.sig sources.archlinux.org:/srv/ftp/other/packages/netctl clean: $(MAKE) -C docs clean -- 2.12.0
Re: [arch-projects] [netctl] ftp.archlinux.org
On 04.06.2016 17:59, Jouke Witteveen via arch-projects wrote: > I cannot find much information about sources.archlinux.org and have > several questions about it. > > 1. > Will it still exist in a couple of years? It would be nice to have the > sources at a somewhat fixed location. Yes, that URL is hosted our infrastructure and will stay around. > 2. > What is the meaning of the "other/packages" part of te path? Why can > we not use sources.archlinux.org/netctl/ or > sources.archlinux.org/other/netctl/? I could not find any > documentation about this. For some packages it is difficult to find a stable or publicly available URL for the source tarballs. Since we need a reachable URL in our PKGBUILDs we mirror such tarballs in the /other/ subtree. netctl doesn't have a dedicated website and no dedicated source tarball hosting so it falls into this category. We used to have such packages directly in /other/, but I'm not entirely sure why that was abandoned and things moved to subdirectories. I guess it probably has something to do with the distinction between TUs and devs since TUs can not access the /other/packages/ directory. They can only access /other/community/. Besides the /other directory, there is also /sources which contains sourcepackages (makepkg --source) that we are required to provide thanks to the GPL. The sources.archlinux.org subdomain came into existence because nymeria has a rather slow network connection and I didn't want to enable public access there for all those files. I also didn't want to reuse ftp.archlinux.org so that I don't cause confusion about what this domain is used for. I also like using different subdomains for different things so it is easier to move them around in the future without breaking old URLs. Is there any reason why you'd want a different URL or is it just a matter of understanding the reasoning here? In the second case, I hope my explanation helps a bit. > 3. > Uploading the sources happens to nymeria.archlinux.org:/srv/ftp/[...]. > Shouldn't this be renamed (linked for compatibility) to (from) > nymeria.archlinux.org:/srv/sources/[...] for consistency? /srv/ftp/ is the place where pretty much all of our repo related data is located. This includes the sources, but also the packages and our ISOs. Changing this is pretty complex and /srv/ftp is also the root directory for our mirrors. Most choose not to mirror the sources and other directories though. While the "ftp" part is slightly misleading, I don't see the value in investing time into changing it. Given the sources are not the only content /srv/sources would be just as misleading as /srv/ftp is now. Florian signature.asc Description: OpenPGP digital signature
