Re: [gentoo-dev] New distfile mirror layout

2019-10-29 Thread Kent Fredric
On Wed, 23 Oct 2019 01:16:51 -0400 Joshua Kinard wrote: > And for Perl or Python, I think we should be making an effort to leverage > their respective mirroring systems first before putting their distfiles onto > our mirrors. Perl's got CPAN, and Python has pypi. For things that don't > exist

Re: [gentoo-dev] New distfile mirror layout

2019-10-29 Thread Fabian Groffen
On 29-10-2019 15:45:34 +0100, Michał Górny wrote: > On Tue, 2019-10-29 at 15:33 +0100, Fabian Groffen wrote: > > In addition, there are currently files there that aren't referenced from > > ebuilds. Prefix uses these files during bootstrap, local mirrors are > > often much faster than dev.g.o. >

Re: [gentoo-dev] New distfile mirror layout

2019-10-29 Thread Michał Górny
On Tue, 2019-10-29 at 15:33 +0100, Fabian Groffen wrote: > On 29-10-2019 15:17:38 +0100, Ulrich Mueller wrote: > > > > > > > On Tue, 29 Oct 2019, Michał Górny wrote: > > > On Tue, 2019-10-29 at 14:09 +0100, Ulrich Mueller wrote: > > > > > What if the file is hosted at a non-standard tcp port

Re: [gentoo-dev] New distfile mirror layout

2019-10-29 Thread Fabian Groffen
On 29-10-2019 15:17:38 +0100, Ulrich Mueller wrote: > > On Tue, 29 Oct 2019, Michał Górny wrote: > > > On Tue, 2019-10-29 at 14:09 +0100, Ulrich Mueller wrote: > >> > What if the file is hosted at a non-standard tcp port upstream > >> > (like http://example.org:8080/)? The devmanual says that

Re: [gentoo-dev] New distfile mirror layout

2019-10-29 Thread Ulrich Mueller
> On Tue, 29 Oct 2019, Michał Górny wrote: > On Tue, 2019-10-29 at 14:09 +0100, Ulrich Mueller wrote: >> > What if the file is hosted at a non-standard tcp port upstream >> > (like http://example.org:8080/)? The devmanual says that it _must_ >> > be manually uploaded to

Re: [gentoo-dev] New distfile mirror layout

2019-10-29 Thread Michał Górny
On Tue, 2019-10-29 at 14:09 +0100, Ulrich Mueller wrote: > > > > > > On Tue, 29 Oct 2019, Ulrich Mueller wrote: > > > > > > On Tue, 29 Oct 2019, Michał Górny wrote: > > > The file should be placed in SRC_URI, and emirrordist will take care > > > of fetching it. > > What if the file is hosted at a

Re: [gentoo-dev] New distfile mirror layout

2019-10-29 Thread Michał Górny
On Tue, 2019-10-29 at 14:03 +0100, Ulrich Mueller wrote: > > > > > > On Tue, 29 Oct 2019, Michał Górny wrote: > > On Tue, 2019-10-29 at 13:23 +0100, Ulrich Mueller wrote: > > > So, what has to be be done to have it appear in the proper place? > > > Should the file be placed in a subdir of

Re: [gentoo-dev] New distfile mirror layout

2019-10-29 Thread Ulrich Mueller
> On Tue, 29 Oct 2019, Ulrich Mueller wrote: > On Tue, 29 Oct 2019, Michał Górny wrote: >> The file should be placed in SRC_URI, and emirrordist will take care >> of fetching it. > What if the file is hosted at a non-standard tcp port upstream (like > http://example.org:8080/)? The

Re: [gentoo-dev] New distfile mirror layout

2019-10-29 Thread Ulrich Mueller
> On Tue, 29 Oct 2019, Michał Górny wrote: > On Tue, 2019-10-29 at 13:23 +0100, Ulrich Mueller wrote: >> So, what has to be be done to have it appear in the proper place? >> Should the file be placed in a subdir of /space/distfiles-local/? >> That seems to be error prone, and certainly could

Re: [gentoo-dev] New distfile mirror layout

2019-10-29 Thread Michał Górny
On Tue, 2019-10-29 at 13:23 +0100, Ulrich Mueller wrote: > > > > > > On Tue, 29 Oct 2019, Michał Górny wrote: > > Dnia October 29, 2019 9:34:01 AM UTC, Fabian Groffen > > napisał(a): > > > /space/distfiles-local is no longer copied to the mirrors? or just > > > not copied in the

Re: [gentoo-dev] New distfile mirror layout

2019-10-29 Thread Ulrich Mueller
> On Tue, 29 Oct 2019, Michał Górny wrote: > Dnia October 29, 2019 9:34:01 AM UTC, Fabian Groffen > napisał(a): >> /space/distfiles-local is no longer copied to the mirrors? or just >> not copied in the subdir-hierarchy? > The latter. So, what has to be be done to have it appear in the

Re: [gentoo-dev] New distfile mirror layout

2019-10-29 Thread Michał Górny
Dnia October 29, 2019 9:34:01 AM UTC, Fabian Groffen napisał(a): >On 29-10-2019 05:27:37 +0100, Michał Górny wrote: >> On Tue, 2019-10-29 at 00:24 +0100, Chí-Thanh Christopher Nguyễn >wrote: >> > Hi! >> > >> > > Today you get chastised for using /space/distfiles-local and not >> > > following

Re: [gentoo-dev] New distfile mirror layout

2019-10-29 Thread Fabian Groffen
On 29-10-2019 05:27:37 +0100, Michał Górny wrote: > On Tue, 2019-10-29 at 00:24 +0100, Chí-Thanh Christopher Nguyễn wrote: > > Hi! > > > > > Today you get chastised for using /space/distfiles-local and not > > > following policy changes. The devmanual states that it's deprecated > > > since at

Re: [gentoo-dev] New distfile mirror layout

2019-10-28 Thread Michał Górny
On Tue, 2019-10-29 at 00:24 +0100, Chí-Thanh Christopher Nguyễn wrote: > Hi! > > > Today you get chastised for using /space/distfiles-local and not > > following policy changes. The devmanual states that it's deprecated > > since at least 2011, and talks of using d.g.o [1]. > > [1] >

Re: [gentoo-dev] New distfile mirror layout

2019-10-28 Thread Chí-Thanh Christopher Nguyễn
Hi! Today you get chastised for using /space/distfiles-local and not following policy changes. The devmanual states that it's deprecated since at least 2011, and talks of using d.g.o [1]. > [1] https://devmanual.gentoo.org/general-concepts/mirrors/index.html#suitable-download-hosts Sorry

Re: [gentoo-dev] New distfile mirror layout

2019-10-23 Thread Michał Górny
On Wed, 2019-10-23 at 17:04 -0500, William Hubbs wrote: > On Wed, Oct 23, 2019 at 01:18:02AM -0400, Joshua Kinard wrote: > > On 10/21/2019 19:36, Matt Turner wrote: > > > On Mon, Oct 21, 2019 at 9:42 AM Richard Yao wrote: > > > > Also, another idea is to use a cheap hash function (e.g. fletcher)

Re: ext4 readdir performance - was Re: [gentoo-dev] New distfile mirror layout

2019-10-23 Thread Richard Yao
> On Oct 23, 2019, at 7:48 PM, Richard Yao wrote: > > On 10/22/19 2:51 AM, Jaco Kroon wrote: >> Hi All, >> >> >>> On 2019/10/21 18:42, Richard Yao wrote: >>> >>> If we consider the access frequency, it might actually not be that >>> bad. Consider a simple example with 500 files and two

ext4 readdir performance - was Re: [gentoo-dev] New distfile mirror layout

2019-10-23 Thread Richard Yao
On 10/22/19 2:51 AM, Jaco Kroon wrote: > Hi All, > > > On 2019/10/21 18:42, Richard Yao wrote: >> >> If we consider the access frequency, it might actually not be that >> bad. Consider a simple example with 500 files and two directory >> buckets. If we have 250 in each, then the size of the

Re: [gentoo-dev] New distfile mirror layout

2019-10-23 Thread William Hubbs
On Wed, Oct 23, 2019 at 01:18:02AM -0400, Joshua Kinard wrote: > On 10/21/2019 19:36, Matt Turner wrote: > > On Mon, Oct 21, 2019 at 9:42 AM Richard Yao wrote: > >> Also, another idea is to use a cheap hash function (e.g. fletcher) and > >> just have the mirrors do the hashing behind the scenes.

Re: [gentoo-dev] New distfile mirror layout

2019-10-23 Thread William Hubbs
On Wed, Oct 23, 2019 at 12:06:24PM -0500, William Hubbs wrote: > On Wed, Oct 23, 2019 at 01:18:02AM -0400, Joshua Kinard wrote: > > On 10/21/2019 19:36, Matt Turner wrote: > > > On Mon, Oct 21, 2019 at 9:42 AM Richard Yao wrote: > > >> Also, another idea is to use a cheap hash function (e.g.

Re: [gentoo-dev] New distfile mirror layout

2019-10-23 Thread William Hubbs
On Wed, Oct 23, 2019 at 01:18:02AM -0400, Joshua Kinard wrote: > On 10/21/2019 19:36, Matt Turner wrote: > > On Mon, Oct 21, 2019 at 9:42 AM Richard Yao wrote: > >> Also, another idea is to use a cheap hash function (e.g. fletcher) and > >> just have the mirrors do the hashing behind the scenes.

Re: [gentoo-dev] New distfile mirror layout

2019-10-22 Thread Joshua Kinard
On 10/21/2019 19:36, Matt Turner wrote: > On Mon, Oct 21, 2019 at 9:42 AM Richard Yao wrote: >> Also, another idea is to use a cheap hash function (e.g. fletcher) and just >> have the mirrors do the hashing behind the scenes. Then we would have the >> best of both worlds. > > It probably would

Re: [gentoo-dev] New distfile mirror layout

2019-10-22 Thread Joshua Kinard
On 10/21/2019 06:13, Kent Fredric wrote: > On Sun, 20 Oct 2019 16:57:54 -0400 > Joshua Kinard wrote: > >> I know we've got a ton of Perl packages for the core set of Perl modules, >> but doesn't the CPAN eclass also have the capability to auto-generate an >> ebuild package for virtually any Perl

Re: [gentoo-dev] New distfile mirror layout

2019-10-22 Thread Rich Freeman
On Mon, Oct 21, 2019 at 12:42 PM Richard Yao wrote: > > Also, another idea is to use a cheap hash function (e.g. fletcher) and just > have the mirrors do the hashing behind the scenes. Then we would have the > best of both worlds. I think something that is getting missed in this discussion is

Re: [gentoo-dev] New distfile mirror layout

2019-10-22 Thread Jaco Kroon
Hi, On 2019/10/22 10:43, Ulrich Mueller wrote: On Tue, 22 Oct 2019, Jaco Kroon wrote: I also agree with others that it used to be easy to get distfiles as and when needed, so an alternative structure could mirror that of the portage tree itself, in other words "cat/pkg/distfile". Not a good

Re: [gentoo-dev] New distfile mirror layout

2019-10-22 Thread Ulrich Mueller
> On Tue, 22 Oct 2019, Jaco Kroon wrote: > I also agree with others that it used to be easy to get distfiles as > and when needed, so an alternative structure could mirror that of the > portage tree itself, in other words "cat/pkg/distfile". Not a good idea, because some distfiles are shared

Re: [gentoo-dev] New distfile mirror layout

2019-10-22 Thread Jaco Kroon
Hi All, On 2019/10/21 18:42, Richard Yao wrote: If we consider the access frequency, it might actually not be that bad. Consider a simple example with 500 files and two directory buckets. If we have 250 in each, then the size of the directory is always 250. However, if 50 files are

Re: [gentoo-dev] New distfile mirror layout

2019-10-21 Thread James Cloos
> "RY" == Richard Yao writes: RY> ext4 is probably okay, but don’t quote me on that. Ext4 works fine here for a local distfiles mirror. -JimC -- James Cloos OpenPGP: 0x997A9F17ED7DAEA6

Re: [gentoo-dev] New distfile mirror layout

2019-10-21 Thread Matt Turner
On Mon, Oct 21, 2019 at 9:42 AM Richard Yao wrote: > Also, another idea is to use a cheap hash function (e.g. fletcher) and just > have the mirrors do the hashing behind the scenes. Then we would have the > best of both worlds. It probably would have been better to make these suggestions when

Re: [gentoo-dev] New distfile mirror layout

2019-10-21 Thread Mikle Kolyada
On 21.10.2019 3:05, Joshua Kinard wrote: > So looking at texlive-latexextra-2019-r2.ebuild, it defines three variables: > > - TEXLIVE_MODULE_CONTENTS, with 1,241 space-delimited module names > - TEXLIVE_MODULE_DOC_CONTENTS, with 1,227 space-delimited doc names > -

Re: [gentoo-dev] New distfile mirror layout

2019-10-21 Thread Richard Yao
> On Oct 20, 2019, at 2:51 AM, Michał Górny wrote: > > On Sat, 2019-10-19 at 19:24 -0400, Joshua Kinard wrote: >>> On 10/18/2019 09:41, Michał Górny wrote: >>> Hi, everybody. >>> >>> It is my pleasure to announce that yesterday (EU) evening we've switched >>> to a new distfile mirror

Re: [gentoo-dev] New distfile mirror layout

2019-10-21 Thread Kent Fredric
On Sun, 20 Oct 2019 20:05:40 -0400 Joshua Kinard wrote: > Longer-term, I think this entire approach should be revisited by the TeX > team to make it behave more like Perl or Python packages by having discrete > ebuilds for these modules. That's not exactly a small undertaking, but > this

Re: [gentoo-dev] New distfile mirror layout

2019-10-21 Thread Kent Fredric
On Sun, 20 Oct 2019 16:57:54 -0400 Joshua Kinard wrote: > I know we've got a ton of Perl packages for the core set of Perl modules, > but doesn't the CPAN eclass also have the capability to auto-generate an > ebuild package for virtually any Perl package distributed via CPAN? Can > that logic

Re: [gentoo-dev] New distfile mirror layout

2019-10-20 Thread Ulrich Mueller
> On Mon, 21 Oct 2019, Joshua Kinard wrote: > - altfont is licensed under "GNU General Public License" (version ??) > - achemso is licensed under "The LaTeX Project Public License 1.3c" > - arraysort is licensed under "The LaTeX Project Public License 1.2" > - amsfonts is licensed

Re: [gentoo-dev] New distfile mirror layout

2019-10-20 Thread Joshua Kinard
On 10/20/2019 16:57, Joshua Kinard wrote:> On 10/20/2019 05:44, Michal Górny wrote: >> On Sun, 2019-10-20 at 05:21 -0400, Joshua Kinard wrote: >>> On 10/20/2019 04:32, Michal Górny wrote: [snip] >> You believe it to be a problem. Don't expect others to bother upstream >> with your preferences. >

Re: [gentoo-dev] New distfile mirror layout

2019-10-20 Thread Joshua Kinard
On 10/20/2019 05:44, Michał Górny wrote: > On Sun, 2019-10-20 at 05:21 -0400, Joshua Kinard wrote: >> On 10/20/2019 04:32, Michał Górny wrote: >>> On Sun, 2019-10-20 at 04:25 -0400, Joshua Kinard wrote: Why is having a max ~24k files in a directory a bad idea? Modern filesystems are

Re: [gentoo-dev] New distfile mirror layout

2019-10-20 Thread Matt Turner
On Sun, Oct 20, 2019 at 1:25 AM Joshua Kinard wrote: > In any event, I still think using devspace is a bad idea. A centralized > distfiles repo is what most other distros use, and it's what we should use. I agree, but let's discuss that in a separate topic.

Re: [gentoo-dev] New distfile mirror layout

2019-10-20 Thread Michał Górny
On Sun, 2019-10-20 at 05:21 -0400, Joshua Kinard wrote: > On 10/20/2019 04:32, Michał Górny wrote: > > On Sun, 2019-10-20 at 04:25 -0400, Joshua Kinard wrote: > > > Why is having a max ~24k files in a directory a bad idea? Modern > > > filesystems are more than capable of handling that. > > > >

Re: [gentoo-dev] New distfile mirror layout

2019-10-20 Thread Joshua Kinard
On 10/20/2019 04:32, Michał Górny wrote: > On Sun, 2019-10-20 at 04:25 -0400, Joshua Kinard wrote: >> On 10/20/2019 02:51, Michał Górny wrote: >>> On Sat, 2019-10-19 at 19:24 -0400, Joshua Kinard wrote: On 10/18/2019 09:41, Michał Górny wrote: > Hi, everybody. > > It is my

Re: [gentoo-dev] New distfile mirror layout

2019-10-20 Thread Michał Górny
On Sun, 2019-10-20 at 04:25 -0400, Joshua Kinard wrote: > On 10/20/2019 02:51, Michał Górny wrote: > > On Sat, 2019-10-19 at 19:24 -0400, Joshua Kinard wrote: > > > On 10/18/2019 09:41, Michał Górny wrote: > > > > Hi, everybody. > > > > > > > > It is my pleasure to announce that yesterday (EU)

Re: [gentoo-dev] New distfile mirror layout

2019-10-20 Thread Joshua Kinard
On 10/20/2019 02:51, Michał Górny wrote: > On Sat, 2019-10-19 at 19:24 -0400, Joshua Kinard wrote: >> On 10/18/2019 09:41, Michał Górny wrote: >>> Hi, everybody. >>> >>> It is my pleasure to announce that yesterday (EU) evening we've switched >>> to a new distfile mirror layout. Users will be

Re: [gentoo-dev] New distfile mirror layout

2019-10-20 Thread Michał Górny
On Sat, 2019-10-19 at 19:24 -0400, Joshua Kinard wrote: > On 10/18/2019 09:41, Michał Górny wrote: > > Hi, everybody. > > > > It is my pleasure to announce that yesterday (EU) evening we've switched > > to a new distfile mirror layout. Users will be switching to the new > > layout either as they

Re: [gentoo-dev] New distfile mirror layout

2019-10-19 Thread Joshua Kinard
On 10/19/2019 19:57, Alec Warner wrote: > On Sat, Oct 19, 2019 at 4:24 PM Joshua Kinard wrote: > >> On 10/18/2019 09:41, Michał Górny wrote: >>> Hi, everybody. >>> >>> It is my pleasure to announce that yesterday (EU) evening we've switched >>> to a new distfile mirror layout. Users will be

Re: [gentoo-dev] New distfile mirror layout

2019-10-19 Thread Alec Warner
On Sat, Oct 19, 2019 at 4:24 PM Joshua Kinard wrote: > On 10/18/2019 09:41, Michał Górny wrote: > > Hi, everybody. > > > > It is my pleasure to announce that yesterday (EU) evening we've switched > > to a new distfile mirror layout. Users will be switching to the new > > layout either as they

Re: [gentoo-dev] New distfile mirror layout

2019-10-19 Thread Joshua Kinard
On 10/18/2019 09:41, Michał Górny wrote: > Hi, everybody. > > It is my pleasure to announce that yesterday (EU) evening we've switched > to a new distfile mirror layout. Users will be switching to the new > layout either as they upgrade Portage to 2.3.77 or -- if they upgraded > already -- as

Re: [gentoo-dev] New distfile mirror layout

2019-10-19 Thread Richard Yao
> On Oct 19, 2019, at 4:03 PM, Michał Górny wrote: > > On Sat, 2019-10-19 at 15:26 -0400, Richard Yao wrote: On Oct 18, 2019, at 9:10 PM, Richard Yao wrote: >>> >>>  > On Oct 18, 2019, at 4:49 PM, Michał Górny wrote: On Fri, 2019-10-18 at 15:53 -0400, Richard Yao wrote:

Re: [gentoo-dev] New distfile mirror layout

2019-10-19 Thread Michał Górny
On Sat, 2019-10-19 at 15:26 -0400, Richard Yao wrote: > > On Oct 18, 2019, at 9:10 PM, Richard Yao wrote: > > > >  > > > > On Oct 18, 2019, at 4:49 PM, Michał Górny wrote: > > > On Fri, 2019-10-18 at 15:53 -0400, Richard Yao wrote: > > > > > > > > > On Oct 18, 2019, at 9:42 AM, Michał Górny

Re: [gentoo-dev] New distfile mirror layout

2019-10-19 Thread Richard Yao
> On Oct 18, 2019, at 9:10 PM, Richard Yao wrote: > >  >>> On Oct 18, 2019, at 4:49 PM, Michał Górny wrote: >> On Fri, 2019-10-18 at 15:53 -0400, Richard Yao wrote: On Oct 18, 2019, at 9:42 AM, Michał Górny wrote: >>> Hi, everybody. >>> It is my pleasure to announce that

Re: [gentoo-dev] New distfile mirror layout

2019-10-19 Thread Michał Górny
On Sat, 2019-10-19 at 15:31 +0200, Fabian Groffen wrote: > Hi, > > On 18-10-2019 15:41:32 +0200, Michał Górny wrote: > > 3. Directly fetching files from distfiles.gentoo.org will become > > a little harder. To fetch a distfile named 'foo-1.tar.gz', you'd have > > to use something like: > > > >

Re: [gentoo-dev] New distfile mirror layout

2019-10-19 Thread Fabian Groffen
Hi, On 18-10-2019 15:41:32 +0200, Michał Górny wrote: > 3. Directly fetching files from distfiles.gentoo.org will become > a little harder. To fetch a distfile named 'foo-1.tar.gz', you'd have > to use something like: > > $ printf '%s' foo-1.tar.gz | b2sum | cut -c1-2 > 1b > $ wget

Re: [gentoo-dev] New distfile mirror layout

2019-10-19 Thread Richard Yao
> On Oct 19, 2019, at 2:17 AM, Michał Górny wrote: > > On Fri, 2019-10-18 at 21:09 -0400, Richard Yao wrote: On Oct 18, 2019, at 4:49 PM, Michał Górny wrote: >>> >>> On Fri, 2019-10-18 at 15:53 -0400, Richard Yao wrote: On Oct 18, 2019, at 9:42 AM, Michał Górny wrote:

Re: [gentoo-dev] New distfile mirror layout

2019-10-19 Thread Michał Górny
On Fri, 2019-10-18 at 21:09 -0400, Richard Yao wrote: > > On Oct 18, 2019, at 4:49 PM, Michał Górny wrote: > > > > On Fri, 2019-10-18 at 15:53 -0400, Richard Yao wrote: > > > > > > > On Oct 18, 2019, at 9:42 AM, Michał Górny > > > > > > > wrote: > > > > > > Hi, everybody. > > > > > > It is

Re: [gentoo-dev] New distfile mirror layout

2019-10-18 Thread Richard Yao
> On Oct 18, 2019, at 4:49 PM, Michał Górny wrote: > > On Fri, 2019-10-18 at 15:53 -0400, Richard Yao wrote: >> On Oct 18, 2019, at 9:42 AM, Michał Górny wrote: > Hi, everybody. > It is my pleasure to announce that yesterday (EU) evening we've switched > to a new distfile

Re: [gentoo-dev] New distfile mirror layout

2019-10-18 Thread Michał Górny
On Fri, 2019-10-18 at 15:53 -0400, Richard Yao wrote: > > On Oct 18, 2019, at 9:42 AM, Michał Górny wrote: > > > > Hi, everybody. > > > > It is my pleasure to announce that yesterday (EU) evening we've switched > > to a new distfile mirror layout. Users will be switching to the new > > layout

Re: [gentoo-dev] New distfile mirror layout

2019-10-18 Thread Richard Yao
> On Oct 18, 2019, at 9:42 AM, Michał Górny wrote: > > Hi, everybody. > > It is my pleasure to announce that yesterday (EU) evening we've switched > to a new distfile mirror layout. Users will be switching to the new > layout either as they upgrade Portage to 2.3.77 or -- if they upgraded

[gentoo-dev] New distfile mirror layout

2019-10-18 Thread Michał Górny
Hi, everybody. It is my pleasure to announce that yesterday (EU) evening we've switched to a new distfile mirror layout. Users will be switching to the new layout either as they upgrade Portage to 2.3.77 or -- if they upgraded already -- as their caches expire (24hrs). The new layout is mostly