Re: [gentoo-dev] Proposal to undeprecate EGO_SUM
On Wed, Sep 28, 2022 at 06:31:39PM +0200, Ulrich Mueller wrote: > > On Wed, 28 Sep 2022, Florian Schmaus wrote: > > > I would like to continue discussing whether we should entirely > > deprecate EGO_SUM without the desire to offend anyone. Don't worry, I am not offended. I just haven't found a simple way to do this. Sure, I will continue the discussion. > > We now have a pending GitHub PR that bumps restic to 0.14 [1]. Restic > > is a very popular backup software written in Go. The PR drops EGO_SUM > > in favor of a vendor tarball created by the proxied maintainer. > > However, I am unaware of any tool that lets you practically audit the > > 35 MiB source contained in the tarball. And even if such a tool > > exists, this would mean another manual step is required, which is, > > potentially, skipped most of the time, weakening our user's security. > > This is because I believe neither our tooling, e.g., go-mod.eclass, > > nor any Golang tooling, does authenticate the contents of the vendor > > tarball against upstream's go.sum. But please correct me if I am > > wrong. I don't know for certain about a vendor tarball, but I do know there are instances where a vendor tarball wouldn't work. app-containers/containerd is a good example of this, That is why the vendor tarball idea was dropped. Go modules are verified by go tooling. That is why I went with a dependency tarball. > > I wonder if we can reach consensus around un-depreacting EGO_SUM, but > > discouraging its usage in certain situations. That is, provide EGO_SUM > > as option but disallow its use if > > 1.) *upstream* provides a vendor tarball Upstream doesn't need to provide a tarball, just an up-to-date "vendor" directory at the top level of the project. Two examples that do this are docker and kubernetes. If the "vendor" directory is in the project, EGO_SUM should not be used. This is already documented in the eclass. > > 2.) the number of EGO_SUM entries exceeds 1000 and a Gentoo developer > > maintains the package > > 3.) the number of EGO_SUM entries exceeds 1500 and a proxied > > maintainer maintains the package > > These numbers seem quite large, compared to the mean number of 3.4 > distfiles for packages in the Gentoo repository. (The median and the > 99-percentile are 1 and 22, respectively.) There is no way from within portage to tell whether a proxied maintainer or a developer maintains the package, and I don't think we should care. We don't want different qa standards for packages in the tree based on who maintains them. I think we should settle on one limit. I could check for that limit inside the eclass and make the ebuild process die if the limit is not observed. The concern, as I understand it, is about the sizes of the ebuilds and manifests for go software. Since the number of distfiles was mentioned, I will add it here and show it in my example numbers below. To stay with your example, restic has a 300k manifest, multiple 30k+ ebuilds and897 distfiles. I'm thinking the limit would have to be much lower. Say, around 256 entries in EGO_SUM_SRC_URI. William signature.asc Description: PGP signature
[gentoo-dev] [PATCH] eclass/tests/unpacker.sh: Add online tests for makeself
Signed-off-by: Michał Górny
---
eclass/tests/unpacker.sh | 105 +++
1 file changed, 105 insertions(+)
diff --git a/eclass/tests/unpacker.sh b/eclass/tests/unpacker.sh
index ea9e64d0a4c7..ef17e724a851 100755
--- a/eclass/tests/unpacker.sh
+++ b/eclass/tests/unpacker.sh
@@ -223,6 +223,67 @@ test_reject_junk() {
rm -f "${archive}" || die
}
+test_online() {
+ local url=${1}
+ local b2sum=${2}
+ local unpacked=${3}
+ local unp_b2sum=${4}
+
+ local filename=${url##*/}
+ local archive=${DISTDIR}/${filename}
+
+ if [[ ! -f ${archive} ]]; then
+ if [[ ${UNPACKER_TESTS_ONLINE} != 1 ]]; then
+ ewarn "Skipping ${filename} test, distfile not found"
+ return
+ fi
+
+ if ! wget -O "${archive}" "${url}"; then
+ die "Fetching ${archive} failed"
+ fi
+ fi
+
+ local real_sum=$(b2sum "${archive}" | cut -d' ' -f1)
+ if [[ ${real_sum} != ${b2sum} ]]; then
+ eerror "Incorrect b2sum on ${filename}"
+ eerror " expected: ${b2sum}"
+ eerror " found: ${real_sum}"
+ die "Incorrect b2sum on ${filename}"
+ fi
+
+ rm -rf testdir || die
+ mkdir -p testdir || die
+
+ tbegin "unpacking ${filename}"
+ cd testdir || die
+
+ ln -s "${archive}" "${filename}" || die
+
+ local out
+ out=$(
+ _unpacker "${archive}" 2>&1
+ )
+ ret=$?
+ if [[ ${ret} -eq 0 ]]; then
+ if [[ ! -f ${unpacked} ]]; then
+ eerror "${unpacked} not found after unpacking"
+ ret=1
+ else
+ real_sum=$(b2sum "${unpacked}" | cut -d' ' -f1)
+ if [[ ${real_sum} != ${unp_b2sum} ]]; then
+ eerror "Incorrect b2sum on unpacked file
${unpacked}"
+ eerror " expected: ${unp_b2sum}"
+ eerror " found: ${real_sum}"
+ ret=1
+ fi
+ fi
+ fi
+ [[ ${ret} -ne 0 ]] && echo "${out}" >&2
+ tend ${ret}
+
+ cd .. || die
+}
+
test_compressed_file .bz2 bzip2
test_compressed_file .Z compress
test_compressed_file .gz gzip
@@ -322,4 +383,48 @@ test_reject_junk .rar
test_reject_junk .lha
test_reject_junk .lzh
+DISTDIR=$(portageq envvar DISTDIR)
+if [[ -n ${DISTDIR} ]]; then
+ einfo "Using DISTDIR: ${DISTDIR}"
+ if [[ ${UNPACKER_TESTS_ONLINE} != 1 ]]; then
+ ewarn "Online tests will be skipped if distfiles are not found
already."
+ ewarn "Set UNPACKER_TESTS_ONLINE=1 to enable fetching."
+ fi
+
+ # NB: a good idea to list the last file in the archive (to avoid
+ # passing on partial unpack)
+
+ # TODO: find test cases for makeself 2.0/2.0.1, 2.1.1, 2.1.2, 2.1.3
+
+ # makeself 1.5.4, gzip
+ test_online \
+ http://updates.lokigames.com/sof/sof-1.06a-cdrom-x86.run \
+
f76f605af08a19b77548455c0101e03aca7cae69462914e47911da2fadd6d4f3b766e1069556ead0d06c757b179ae2e8105e76ea37852f17796b47b4712aec87
\
+ update.sh \
+
ba7a3f8fa79bbed8ca3a34ead957aeaa308c6e6d6aedd603098aa9867ca745983ff98c83d65572e507f2c3c4e0778ae4984f8b69d2b8279741b06064253c5788
+
+ # makeself 1.6.0-nv*, xz
+ test_online \
+
https://download.nvidia.com/XFree86/Linux-x86/390.154/NVIDIA-Linux-x86-390.154.run
\
+
083d9dd234a37ec39a703ef7e0eb6ec165c24d2fcb5e92ca987c33df643d0604319eb65ef152c861acacd5a41858ab6b82c45c2c8ff270efc62b07727666daae
\
+ libEGL_nvidia.so.390.154 \
+
6665804947e71fb583dc7d5cc3a6f4514f612503000b0a9dbd8da5c362d3c2dcb2895d8cbbf5700a6f0e24cca9b0dd9c2cf5763d6fbb037f55257ac5af7d6084
+
+ # makeself 2.3.0, gzip
+ test_online \
+
http://www.sdrplay.com/software/SDRplay_RSP_API-Linux-3.07.1.run \
+
059d9a5fbd14c0e7ecb969cd3e5afe8e3f42896175b443bdaa9f9108302a1c9ef5ad9769e62f824465611d74f67191fff71cc6dbe297e399e5b2f6824c650112
\
+ i686/sdrplay_apiService \
+
806393c310d7e60dca7b8afee225bcc50c0d5771bdd04c3fa575eda2e687dc5c888279a7404316438b633fb91565a49899cf634194d43981151a12c6c284a162
+
+ # makeself 2.4.0, gzip
+ test_online \
+
http://www.sdrplay.com/software/SDRplay_RSP_API-Linux-2.13.1.run \
+
7eff1aa35190db1ead5b1d96994d24ae2301e3a765d6701756c6304a1719aa32125fedacf6a6859d89b89db5dd6956ec0e8c7e814dbd6242db5614a53e89efb3
\
+ sdrplay_license.txt \
+
041edb26ffb75b6b59e7a3514c6f81b05b06e0efe61cc56117d24f59733a6a6b1bca73a57dd11e0774ec443740ca55e6938cf6594a032ab4f14b23f2e732a3f2
+else
+ ewarn "Unable to obtain DISTDIR from port
