* "Robin H. Johnson" <robbat2-20220608T184338-394361540Z @orbis-terrarum.net> : Wrote on Wed, 8 Jun 2022 20:42:48 +0000: > EGO_SUM vs dependency tarballs: > - bloats ebuilds > - bloats Manifests > - bloats metadata/md5-cache/ (SRC_URI etc) > - doesn't bloat mirrors with gentoo-unique distfiles > - EGO_SUM is verifiable/reproducible from Upstream Go systems > - less downloads on upgrades (only changed Go deps, not entire dep tarballs) > > EGO_SUM data right now adds, to every user's system: > - 2.6MB of text to ebuilds (340k after de-dupe) > - 7MB of text to Manifests (2M after de-dupe) > - 6.4MB+ of text to metadata/md5-cache (I don't have a easy way to > calc deduped amount here) > On the server side: > - The sum total of Go distfiles mirrored on Gentoo mirrors right now > is only 3.4GB. > - less downloads > > Dependency tarballs: > - Right now ~15GiB on each mirror, plus storage of the primary copy > somewhere (dev.g.o right now, but not great) > - Conservatively if the remaining EGO_SUM packages converted to Dep > tarballs, it would need another 8GB each of primary location and > mirrors. > - larger downloads for users who DO want to upgrade a Go package (all > new deps tarball even if only one or two deps changed) > - must be preserved much longer, unless we can introduce a guaranteed > way to regenerate them for any prior ebuild. > > I was trying to introduce a third option, but I haven't had the time to > write an entire GLEP. > > The TL;DR is introducing a 2nd-level Manifest+metadata file, that tries > to move just the metadata out of the tree, in a way that can be > regenerated (specifically, a 1:1 reproducible creation from a given go.sum). > It DOES need to contain slightly more data than the present Manifest, > specifically a full SRC_URI entry for each file (upstream URI plus what > to rename it to on Gentoo side) > > The 2nd-level Manifest would be listed as SRC_URI, and be handled in > src_fetch/src_unpack. Download & verify the extra distfiles, against the > Manifest checksum data (and for Golang against go.sum checksums). > > The Portage mirrordist code needs the most work in this case, as it > would need to fetch the 2nd-level Manifests so it can populate Gentoo > mastermirror with the distfiles mirrored from upstream. > > The storage costs for the proposed idea: > - same 1:1 base distfile storage as EGO_SUM (e.g. upstream distfiles are > mirrored 1:1 content, just different naming) > - Probably 1 Metadata-Manifest file per ebuild $PVR (conceptually it > could be split more or shared between some ebuilds/packages) > - Main tree Manifests: 1 DIST entry per Metadata-Manifest in a given package > - Main tree ebuilds: 1 line for the Metadata-Manifest in the ebuild. > - metadata/md5-cache: 1 src_uri line! > - mirrors: add the Metadata-Manifest
[Without claiming to have fully understood the proposal above: around Apr 15th 22 I tried suggesting to WilliamH on IRC that perhaps portage should implement the dirhash approach that go has taken to solve the problem of upstream sources when they invented go.sum. from hash.go in sources go/src/cmd/vendor/golang.org/x/mod/sumdb/dirhash/hash.go // Hash1 is "h1:" followed by the base64-encoded SHA-256 hash of a summary prepared as if by the Unix command:find . -type f | sort | sha256sum loosely speaking the "manifest" could publish this dirhash of contents of go-mod/cache (which would have been bundled in the -deps.tar.xz) The immediate motivation was to avoid the network when I already had the sources locally: instead of downloading a -deps.tar.xz I could create it locally and dump it in distdir. portage would check the (hypothetically) published dirhash and let it through. the local timestamps and uid in my tarball and the upstream tarball wouldn't upset it. One unchecked assumption is that go-mod/cache can be recreated by unpacking sources. If so then with a notion of a "second level manifest" (the equivalent of go.sum) the contents can be assembled without having to store or download the actual -deps tarball. I didn't get very far in convincing WilliamH of my need so I dropped the idea. (I'm not sure if I'm being any clearer, if I'm missing something, do let me know)