On Sat, Oct 19, 2019 at 4:24 PM Joshua Kinard <ku...@gentoo.org> wrote:
> On 10/18/2019 09:41, Michał Górny wrote: > > Hi, everybody. > > > > It is my pleasure to announce that yesterday (EU) evening we've switched > > to a new distfile mirror layout. Users will be switching to the new > > layout either as they upgrade Portage to 2.3.77 or -- if they upgraded > > already -- as their caches expire (24hrs). > > > > The new layout is mostly a bow towards mirror admins, for some of whom > > having a 60000+ files in a single directory have been a problem. > > However, I suppose some of you also found e.g. the directory index > > hardly usable due to its size. > > > > Throughout a transitional period (whose exact length hasn't been decided > > yet), both layouts will be available. Afterwards, the old layout will > > be removed from mirrors. This has a few implications: > > > > 1. Users who don't upgrade their package managers in time will lose > > the ability of fetching from Gentoo mirrors. This shouldn't be that > > much of a problem given that the core software needed to upgrade Portage > > should all have reliable upstream SRC_URIs. > > > > 2. mirror://gentoo/file URIs will stop working. While technically you > > could use mirror://gentoo/XX/file, I'd rather recommend finally > > discarding its usage and moving distfiles to devspace. > > > > 3. Directly fetching files from distfiles.gentoo.org will become > > a little harder. To fetch a distfile named 'foo-1.tar.gz', you'd have > > to use something like: > > > > $ printf '%s' foo-1.tar.gz | b2sum | cut -c1-2 > > 1b > > $ wget http://distfiles.gentoo.org/distfiles/1b/foo-1.tar.gz > > ... > > > > > > Alternatively, you can: > > > > $ wget http://distfiles.gentoo.org/distfiles/INDEX > > > > and grep for the right path there. This INDEX is also a more > > lightweight alternative to HTML indexes generated by the servers. > > > > > > If you're interested in more background details and some plots, see [1]. > > > > [1] > https://dev.gentoo.org/~mgorny/articles/improving-distfile-mirror-structure.html > > > > So the answer I didn't really see directly stated here is, where do new > distfiles need to go //now//? E.g., if on woodpecker, I currently cp a > distfile to /space/distfiles-local. What is the new directory I need to > use? And if mirror://gentoo/${FOO} is going away, for the new distfiles > target, what would be the applicable prefix to use? > > > Directly using devspace seems like a bad idea, IMHO. Once long ago, we all > got chastised for doing exactly that. Too much possibility of > fragmentation > as devs retire or package maintainership changes hands. > > I looked at the whitepaper'ish-like writeup, and I kinda don't like using a > hash-based naming scheme on the new distfiles layout. I really kind prefer > breaking the directories up based on the first letter of the distfiles in > question, factoring case-sensitivity in (so you'd have 52 top-level > directories for A-Z and a-z, plus 10 more for 0-9). Under each of those > directories, additional subdirectories for the next few letters (say, > letters 2-3). Yes, this leads to some orphan cases where a distfile might > live on its own, but from a direct navigation standpoint, it's easy to find > for someone browsing the distfiles server and easy to predict where a > distfile is at. > > No math, statistical analysis, or deep-rooted knowledge of filesystems > behind that paragraph. Just a plain old unfiltered opinion. Sometimes, I > need to go get a distfile off the Gentoo mirrors, and being able to quickly > find it in the mirror root is great. Having to do hash calculations to > work > out the file path will be *really* annoying. > So if you want a tool that "downloads a distfile off of the mirrors" we should be able to build such a utility. I'm not really sure why that tool needs to be: *copy DISTFILENAME* wget distilfes.gentoo.org/$PASTE It could just `ebuild portageq download $DISTFILENAME or similar.` -A > > -- > Joshua Kinard > Gentoo/MIPS > ku...@gentoo.org > rsa6144/5C63F4E3F5C6C943 2015-04-27 > 177C 1972 1FB8 F254 BAD0 3E72 5C63 F4E3 F5C6 C943 > > "The past tempts us, the present confuses us, the future frightens us. And > our lives slip away, moment by moment, lost in that vast, terrible > in-between." > > --Emperor Turhan, Centauri Republic > >
