Re: Proposed F13 feature: drop separate updates repository
Matthew Booth wrote: The separate updates directory has been a pain for as long as I've been using RHL/Fedora Core/Fedora. It means you have two places to look when searching for packages manually, and twice as much to configure when you're configuring yum. It has never benefitted me, or anybody I know, but it has caught me out on any number of occasions. What's more, nobody really seems to know why it's like that: it seems it's always been that way, and nobody ever bother to fix it. So lets fix it. The package set at release time is only interesting to historians. If any of them are really that bothered, I'm sure somebody can come up with a yum module which finds the oldest available version of a package in a repo. Matt Would not this also provide the minor added benefit that there could now be a drpm for the first update for a package? -J -- in your fear, seek only peace in your fear, seek only love -d. bowie -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On 12/02/2009 03:39 PM, Matthew Booth wrote: The separate updates directory has been a pain for as long as I've been using RHL/Fedora Core/Fedora. It means you have two places to look when searching for packages manually, and twice as much to configure when you're configuring yum. It has never benefitted me, or anybody I know, but it has caught me out on any number of occasions. What's more, nobody really seems to know why it's like that: it seems it's always been that way, and nobody ever bother to fix it. So lets fix it. The package set at release time is only interesting to historians. If any of them are really that bothered, I'm sure somebody can come up with a yum module which finds the oldest available version of a package in a repo. So your proposal is to drop updates, but to add updates to "Everything"? In this case, I am all for it. Ralf -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On 12/02/2009 10:30 AM, Jon Ciesla wrote: Matthew Booth wrote: The separate updates directory has been a pain for as long as I've been using RHL/Fedora Core/Fedora. It means you have two places to look when searching for packages manually, and twice as much to configure when you're configuring yum. It has never benefitted me, or anybody I know, but it has caught me out on any number of occasions. What's more, nobody really seems to know why it's like that: it seems it's always been that way, and nobody ever bother to fix it. So lets fix it. The package set at release time is only interesting to historians. If any of them are really that bothered, I'm sure somebody can come up with a yum module which finds the oldest available version of a package in a repo. Matt Would not this also provide the minor added benefit that there could now be a drpm for the first update for a package? I like the proposal, the only drawback I found is that when a new Fedora release is out we mirror (local) the updates repository with priority, and the big Everything repository is mirrored using a small bandwidth setting because it is not needed normally for the first updates (updates that add new dependecies are not common the first weeks) -J -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, Dec 02, 2009 at 09:00:53AM -0600, Jon Ciesla wrote: > Matthew Booth wrote: >> The separate updates directory has been a pain for as long as I've >> been using RHL/Fedora Core/Fedora. It means you have two places to >> look when searching for packages manually, and twice as much to >> configure when you're configuring yum. It has never benefitted me, or >> anybody I know, but it has caught me out on any number of occasions. >> What's more, nobody really seems to know why it's like that: it seems >> it's always been that way, and nobody ever bother to fix it. >> >> So lets fix it. The package set at release time is only interesting to >> historians. If any of them are really that bothered, I'm sure somebody >> can come up with a yum module which finds the oldest available version >> of a package in a repo. >> >> Matt > Would not this also provide the minor added benefit that there could now > be a drpm for the first update for a package? We already have that if the update is done after GA. josh -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, Dec 02, 2009 at 02:39:30PM +, Matthew Booth wrote: > The separate updates directory has been a pain for as long as I've been > using RHL/Fedora Core/Fedora. It means you have two places to look when > searching for packages manually, and twice as much to configure when > you're configuring yum. It has never benefitted me, or anybody I know, > but it has caught me out on any number of occasions. What's more, nobody > really seems to know why it's like that: it seems it's always been that > way, and nobody ever bother to fix it. > > So lets fix it. And how do you propose to do that? josh -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On 02/12/09 15:26, Josh Boyer wrote: On Wed, Dec 02, 2009 at 02:39:30PM +, Matthew Booth wrote: The separate updates directory has been a pain for as long as I've been using RHL/Fedora Core/Fedora. It means you have two places to look when searching for packages manually, and twice as much to configure when you're configuring yum. It has never benefitted me, or anybody I know, but it has caught me out on any number of occasions. What's more, nobody really seems to know why it's like that: it seems it's always been that way, and nobody ever bother to fix it. So lets fix it. And how do you propose to do that? Unfortunately I'm not intimately familiar with Fedora infrastructure under-the-hood. However, I would hope that, technically at least, it wouldn't be much more than a systematic removal of all updates repositories and redirecting files which would have gone into them into the main repositories instead. This stems from the observation that if you copied everything from the main repository into updates you would have created the repository which people unfamiliar with the history would expect. The infrastructure side of this, on the face of it, sounds very simple. More involved would be: 1. Updating all packages which expect 2 repos to only expect 1. 2. Communicating the change effectively. Matt -- Matthew Booth, RHCA, RHCSS Red Hat Engineering, Virtualisation Team M: +44 (0)7977 267231 GPG ID: D33C3490 GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490 -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
Josh Boyer wrote: On Wed, Dec 02, 2009 at 09:00:53AM -0600, Jon Ciesla wrote: Matthew Booth wrote: The separate updates directory has been a pain for as long as I've been using RHL/Fedora Core/Fedora. It means you have two places to look when searching for packages manually, and twice as much to configure when you're configuring yum. It has never benefitted me, or anybody I know, but it has caught me out on any number of occasions. What's more, nobody really seems to know why it's like that: it seems it's always been that way, and nobody ever bother to fix it. So lets fix it. The package set at release time is only interesting to historians. If any of them are really that bothered, I'm sure somebody can come up with a yum module which finds the oldest available version of a package in a repo. Matt Would not this also provide the minor added benefit that there could now be a drpm for the first update for a package? We already have that if the update is done after GA. josh If that's the case, then good. In that case, I see no huge benefit to leaving it or changing it. -- in your fear, seek only peace in your fear, seek only love -d. bowie -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, Dec 02, 2009 at 02:39:30PM +, Matthew Booth wrote: > The separate updates directory has been a pain for as long as I've been > using RHL/Fedora Core/Fedora. It means you have two places to look when > searching for packages manually, and twice as much to configure when you're > configuring yum. It has never benefitted me, or anybody I know, but it has > caught me out on any number of occasions. What's more, nobody really seems > to know why it's like that: it seems it's always been that way, and nobody > ever bother to fix it. > The only downside to merging updates into the main repository is that network installs from the repository will no longer install the "release" they will install the updated release. QA that goes into that first impression is no longer there, and your installs are not repeatable because installing system A on one day and system B on another could end up with different versions of packages. Of course you can always install from the ISOs to avoid these problems. As things are, you have the choice of the "release" as it was, or the updated release at install time. I am not saying that this is a bad idea, in fact I rather like it, but there are things to consider. Justin -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, Dec 02, 2009 at 03:44:08PM +, Matthew Booth wrote: > On 02/12/09 15:26, Josh Boyer wrote: >> On Wed, Dec 02, 2009 at 02:39:30PM +, Matthew Booth wrote: >>> The separate updates directory has been a pain for as long as I've been >>> using RHL/Fedora Core/Fedora. It means you have two places to look when >>> searching for packages manually, and twice as much to configure when >>> you're configuring yum. It has never benefitted me, or anybody I know, >>> but it has caught me out on any number of occasions. What's more, nobody >>> really seems to know why it's like that: it seems it's always been that >>> way, and nobody ever bother to fix it. >>> >>> So lets fix it. >> >> And how do you propose to do that? > > Unfortunately I'm not intimately familiar with Fedora infrastructure > under-the-hood. However, I would hope that, technically at least, it > wouldn't be much more than a systematic removal of all updates > repositories and redirecting files which would have gone into them into > the main repositories instead. This stems from the observation that if > you copied everything from the main repository into updates you would > have created the repository which people unfamiliar with the history > would expect. The infrastructure side of this, on the face of it, sounds > very simple. What you describe is effectively how the development repository is built, so it's certainly a technical possibility. It does have implications though. Off the top of my head, I can think of: 1) Composing a new everything tree for updates would lead to larger compose times. That could possibly mean that getting updates out would take > 1 day per 'push'. We've been trying to improve updates push times so it would be a bit detrimental to that goal. 2) There might be GPL compliance issues 3) You would still need an 'updates-testing' repository given that this is a supposedly stable release. So there is still going to be at least one additional repo regardless. However, other than 'browsing manually for packages', I'm not really sure what problem you are trying to solve by getting rid of the updates repository. It would seem like this has quite a bit of cost for relatively little to no real gain? josh -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
Matthew Booth (mbo...@redhat.com) said: > The separate updates directory has been a pain for as long as I've > been using RHL/Fedora Core/Fedora. It means you have two places to > look when searching for packages manually, and twice as much to > configure when you're configuring yum. It has never benefitted me, > or anybody I know, but it has caught me out on any number of > occasions. What's more, nobody really seems to know why it's like > that: it seems it's always been that way, and nobody ever bother to > fix it. > > So lets fix it. The package set at release time is only interesting > to historians. If any of them are really that bothered, I'm sure > somebody can come up with a yum module which finds the oldest > available version of a package in a repo. The separate Everything tree that does not get obsoleted is required in some form for GPL compliance, with respect to the ISO images that we ship. Any new solution would have to preserve this. Bill -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
2009/12/2 Justin M. Forbes > The only downside to merging updates into the main repository... I would also assume that the repo data will need to be regenerated and often be much larger than the one that is for the updates only repository, so there will be acost to end users with this proposed change? -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, Dec 02, 2009 at 10:01:51AM -0600, Justin M. Forbes wrote: > On Wed, Dec 02, 2009 at 02:39:30PM +, Matthew Booth wrote: > > The separate updates directory has been a pain for as long as I've been > > using RHL/Fedora Core/Fedora. It means you have two places to look when > > searching for packages manually, and twice as much to configure when you're > > configuring yum. It has never benefitted me, or anybody I know, but it has > > caught me out on any number of occasions. What's more, nobody really seems > > to know why it's like that: it seems it's always been that way, and nobody > > ever bother to fix it. > > > The only downside to merging updates into the main repository is that > network installs from the repository will no longer install the "release" > they will install the updated release. QA that goes into that first > impression is no longer there, and your installs are not repeatable because > installing system A on one day and system B on another could end up with > different versions of packages. Of course you can always install from the > ISOs to avoid these problems. As things are, you have the choice of the > "release" as it was, or the updated release at install time. I am not > saying that this is a bad idea, in fact I rather like it, but there are > things to consider. I think this is quite a compelling reason not to touch it. Much as we'd like the updates repository to always be perfectly stable, there are still plenty of occassions when we hit problems with updates. Being able to reliably install the release at all times is pretty critical & saying we should install off ISO if we want a reliable install is not really an acceptable alternative since many people do all their installs straight off the network. Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :| -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On 12/02/2009 05:09 PM, Bill Nottingham wrote: Matthew Booth (mbo...@redhat.com) said: The separate updates directory has been a pain for as long as I've been using RHL/Fedora Core/Fedora. It means you have two places to look when searching for packages manually, and twice as much to configure when you're configuring yum. It has never benefitted me, or anybody I know, but it has caught me out on any number of occasions. What's more, nobody really seems to know why it's like that: it seems it's always been that way, and nobody ever bother to fix it. So lets fix it. The package set at release time is only interesting to historians. If any of them are really that bothered, I'm sure somebody can come up with a yum module which finds the oldest available version of a package in a repo. The separate Everything tree that does not get obsoleted is required in some form for GPL compliance, with respect to the ISO images that we ship. Isn't this the "Fedora" repo? To my knowledge the "Fedora" repos corresponds 1:1 to the isos. Ralf -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, Dec 02, 2009 at 11:09:41AM -0500, Bill Nottingham wrote: > Matthew Booth (mbo...@redhat.com) said: > > The separate updates directory has been a pain for as long as I've > > been using RHL/Fedora Core/Fedora. It means you have two places to > > look when searching for packages manually, and twice as much to > > configure when you're configuring yum. It has never benefitted me, > > or anybody I know, but it has caught me out on any number of > > occasions. What's more, nobody really seems to know why it's like > > that: it seems it's always been that way, and nobody ever bother to > > fix it. > > > > So lets fix it. The package set at release time is only interesting > > to historians. If any of them are really that bothered, I'm sure > > somebody can come up with a yum module which finds the oldest > > available version of a package in a repo. > > The separate Everything tree that does not get obsoleted is required > in some form for GPL compliance, with respect to the ISO images that > we ship. Any new solution would have to preserve this. Might there also be export compliance implications too? -- Paul W. Frieldshttp://paul.frields.org/ gpg fingerprint: 3DA6 A0AC 6D58 FEC4 0233 5906 ACDB C937 BD11 3717 http://redhat.com/ - - - - http://pfrields.fedorapeople.org/ irc.freenode.net: stickster @ #fedora-docs, #fedora-devel, #fredlug -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, 02 Dec 2009 17:27:17 +0100 Ralf Corsepius wrote: > On 12/02/2009 05:09 PM, Bill Nottingham wrote: > > Matthew Booth (mbo...@redhat.com) said: > >> The separate updates directory has been a pain for as long as I've > >> been using RHL/Fedora Core/Fedora. It means you have two places to > >> look when searching for packages manually, and twice as much to > >> configure when you're configuring yum. It has never benefitted me, > >> or anybody I know, but it has caught me out on any number of > >> occasions. What's more, nobody really seems to know why it's like > >> that: it seems it's always been that way, and nobody ever bother to > >> fix it. > >> > >> So lets fix it. The package set at release time is only interesting > >> to historians. If any of them are really that bothered, I'm sure > >> somebody can come up with a yum module which finds the oldest > >> available version of a package in a repo. > > > > The separate Everything tree that does not get obsoleted is required > > in some form for GPL compliance, with respect to the ISO images that > > we ship. > Isn't this the "Fedora" repo? > > To my knowledge the "Fedora" repos corresponds 1:1 to the isos. To the DVD iso, yes. To any of the spins/desktop/live... nope. They use packages from Everything/ kevin signature.asc Description: PGP signature -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On 12/02/2009 05:32 PM, Kevin Fenzi wrote: On Wed, 02 Dec 2009 17:27:17 +0100 Ralf Corsepius wrote: On 12/02/2009 05:09 PM, Bill Nottingham wrote: Matthew Booth (mbo...@redhat.com) said: The separate updates directory has been a pain for as long as I've been using RHL/Fedora Core/Fedora. It means you have two places to look when searching for packages manually, and twice as much to configure when you're configuring yum. It has never benefitted me, or anybody I know, but it has caught me out on any number of occasions. What's more, nobody really seems to know why it's like that: it seems it's always been that way, and nobody ever bother to fix it. So lets fix it. The package set at release time is only interesting to historians. If any of them are really that bothered, I'm sure somebody can come up with a yum module which finds the oldest available version of a package in a repo. The separate Everything tree that does not get obsoleted is required in some form for GPL compliance, with respect to the ISO images that we ship. Isn't this the "Fedora" repo? To my knowledge the "Fedora" repos corresponds 1:1 to the isos. To the DVD iso, yes. To any of the spins/desktop/live... nope. They use packages from Everything/ A question however remains: Does the FSF/GPL demand to keep a repo around for ISOs? A "rolling Everything" would not touch the ISOs. They would still be around. Ralf -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
Ralf Corsepius (rc040...@freenet.de) said: > Does the FSF/GPL demand to keep a repo around for ISOs? > A "rolling Everything" would not touch the ISOs. They would still be around. The LiveCD/spins satisfy their source requirements via the source repositories; they do not compose separate live source images. Bill -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, Dec 02, 2009 at 11:28:24AM -0500, Paul W. Frields wrote: > > > > The separate Everything tree that does not get obsoleted is required > > in some form for GPL compliance, with respect to the ISO images that > > we ship. Any new solution would have to preserve this. > > Might there also be export compliance implications too? What manner of export issues? This change doesn't, afaik, affect where export-restricted software appears in any way. --CJD -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, Dec 02, 2009 at 11:06:22AM -0500, Josh Boyer wrote: > 1) Composing a new everything tree for updates would lead to larger > compose times. That could possibly mean that getting updates out would > take > 1 day per 'push'. We've been trying to improve updates push > times so it would be a bit detrimental to that goal. > This might be a good opportunity to look at some way of pushing incremental sets of packages rather than re-building the entire yum repo each time. Mashing is not a cheap operation. > 2) There might be GPL compliance issues > > 3) You would still need an 'updates-testing' repository given that this > is a supposedly stable release. So there is still going to be at least > one additional repo regardless. > We have tags for updates that mark them as bugfix/feature/security. Maybe we could have one for "testing" which would keep yum from installing the package unless explicitly asked or specially configured. > However, other than 'browsing manually for packages', I'm not really > sure what problem you are trying to solve by getting rid of the > updates repository. It would seem like this has quite a bit of cost > for relatively little to no real gain? > This is true. I don't know that manual package browsing is a use case we've ever been interested in, and if we're going to be interested in it there's probably a better way (search engine in the Fedora community portal comes to mind). > josh You owe me $5. --CJD -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, 2 Dec 2009, Paul W. Frields wrote: On Wed, Dec 02, 2009 at 11:09:41AM -0500, Bill Nottingham wrote: we ship. Any new solution would have to preserve this. Might there also be export compliance implications too? A larger isssue is constantly having the repodata for the everything repo be: 1. growing 2. changing So now instead of having a 15K pkg repo that never changes you have an ever-growing repo that never shrinks in size that changes what? 3 times a week? -sv -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On 12/02/2009 06:01 PM, Casey Dahlin wrote: On Wed, Dec 02, 2009 at 11:06:22AM -0500, Josh Boyer wrote: However, other than 'browsing manually for packages', I'm not really sure what problem you are trying to solve by getting rid of the updates repository. It would seem like this has quite a bit of cost for relatively little to no real gain? This is true. It is not true. * It shifts "costs" from "users" to "vendor" and from "mirrors" to "master". * It helps users who are using networked installs to spare bandwidth (avoids downloading obsolete packages from "Everything"/"Fedora"). Admitted, for most users, it would change almost nothing. Ralf -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On 02/12/09 16:01, Justin M. Forbes wrote: On Wed, Dec 02, 2009 at 02:39:30PM +, Matthew Booth wrote: The separate updates directory has been a pain for as long as I've been using RHL/Fedora Core/Fedora. It means you have two places to look when searching for packages manually, and twice as much to configure when you're configuring yum. It has never benefitted me, or anybody I know, but it has caught me out on any number of occasions. What's more, nobody really seems to know why it's like that: it seems it's always been that way, and nobody ever bother to fix it. The only downside to merging updates into the main repository is that network installs from the repository will no longer install the "release" they will install the updated release. QA that goes into that first impression is no longer there, and your installs are not repeatable because installing system A on one day and system B on another could end up with different versions of packages. Of course you can always install from the ISOs to avoid these problems. As things are, you have the choice of the "release" as it was, or the updated release at install time. I am not saying that this is a bad idea, in fact I rather like it, but there are things to consider. That's not a bug, it's a feature! Seriously, wasn't it a specific F10 or F11 feature that anaconda allowed the user to specify that updates be installed at installation time? Certainly I used to have Debianites crow at me that my installation was vulnerable until I updated it. Not only that, but I had to download updated packages twice. These days I always install with updates. I find the installation argument dubious at best. Still, we could rename the main repo the 'installation' repo, and nothing but anaconda would ever look at it. Everything else would look at the main repo, which would be the same as the current updates repo except with everything in it from the start. Matt -- Matthew Booth, RHCA, RHCSS Red Hat Engineering, Virtualisation Team M: +44 (0)7977 267231 GPG ID: D33C3490 GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490 -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, 2009-12-02 at 14:39 +, Matthew Booth wrote: > The separate updates directory has been a pain for as long as I've been > using RHL/Fedora Core/Fedora. It means you have two places to look when > searching for packages manually, and twice as much to configure when > you're configuring yum. It has never benefitted me, or anybody I know, > but it has caught me out on any number of occasions. What's more, nobody > really seems to know why it's like that: it seems it's always been that > way, and nobody ever bother to fix it. > > So lets fix it. The package set at release time is only interesting to > historians. If any of them are really that bothered, I'm sure somebody > can come up with a yum module which finds the oldest available version > of a package in a repo. > This runs us into a few different problems. 1) current compose tools do a fresh compose of a tag every time, into a fresh directory. It is not easy to just 'add' newer things to a directory and only keep X number around. 2) the metadata generation step for the Everything repo is long, and very resource intensive. Doing it every time we push updates would cause unreasonable stress on the infrastructure and the people doing the pushes. 3) the repodata for the Everything repo is huge. Forcing users to download the entire thing every day or so as updates are pushed would cause unreasonable stress on the users's bandwidth, local machine resources to parse it, and the servers offering it for download. I think there are more, but those above are enough for me to not persue this avenue currently. The pros of this change, which seem small to me, don't outweigh the above cons, and we have far more other issues we could be spending time and resources on. -- Jesse Keating Fedora -- Freedom² is a feature! identi.ca: http://identi.ca/jkeating signature.asc Description: This is a digitally signed message part -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On 02/12/09 16:09, Bill Nottingham wrote: Matthew Booth (mbo...@redhat.com) said: The separate updates directory has been a pain for as long as I've been using RHL/Fedora Core/Fedora. It means you have two places to look when searching for packages manually, and twice as much to configure when you're configuring yum. It has never benefitted me, or anybody I know, but it has caught me out on any number of occasions. What's more, nobody really seems to know why it's like that: it seems it's always been that way, and nobody ever bother to fix it. So lets fix it. The package set at release time is only interesting to historians. If any of them are really that bothered, I'm sure somebody can come up with a yum module which finds the oldest available version of a package in a repo. The separate Everything tree that does not get obsoleted is required in some form for GPL compliance, with respect to the ISO images that we ship. Any new solution would have to preserve this. This argument sounds weird to me. However, there no reason you can't keep around as many repositories as you like as long as the One True Repository exists and people can use it exclusively. Currently it doesn't exist. Matt -- Matthew Booth, RHCA, RHCSS Red Hat Engineering, Virtualisation Team M: +44 (0)7977 267231 GPG ID: D33C3490 GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490 -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, 2009-12-02 at 18:09 +0100, Ralf Corsepius wrote: > > * It shifts "costs" from "users" to "vendor" > and from "mirrors" to "master". > * It helps users who are using networked installs to spare bandwidth > (avoids downloading obsolete packages from "Everything"/"Fedora"). > > Admitted, for most users, it would change almost nothing. > > People doing network installs can either add the updates repo to their kickstart, or check the box in the anaconda UI, so that the updates repos are considered at install time. No download of duplicate data. In fact, having separate repos would likely cost less bandwidth. If we only had one combined repo, there would be many duplicate packages, especially if we went the route of having updates-testing mixed in and only marked by some update tag. We'd have to keep sets of what's in updates-testing, updates, and the GA release set, and all of that would be in one repodata set. Everybody doing a network install, whether they wanted updates, updates-testing, or not would have to download and consume that larger repodata, introducing a higher cost for them. -- Jesse Keating Fedora -- Freedom² is a feature! identi.ca: http://identi.ca/jkeating signature.asc Description: This is a digitally signed message part -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, Dec 2, 2009 at 5:37 PM, Matthew Booth wrote: > > That's not a bug, it's a feature! Seriously, wasn't it a specific F10 or F11 > feature that anaconda allowed the user to specify that updates be installed > at installation time? Certainly I used to have Debianites crow at me that my > installation was vulnerable until I updated it. Not only that, but I had to > download updated packages twice. These days I always install with updates. I > find the installation argument dubious at best. This could be equivalently fixed by have the desktop check on initial login if there are any updates available before letting you launch say the web browser. However - we have bigger issues related to updates to fix. -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On 02/12/09 16:06, Josh Boyer wrote: On Wed, Dec 02, 2009 at 03:44:08PM +, Matthew Booth wrote: On 02/12/09 15:26, Josh Boyer wrote: On Wed, Dec 02, 2009 at 02:39:30PM +, Matthew Booth wrote: The separate updates directory has been a pain for as long as I've been using RHL/Fedora Core/Fedora. It means you have two places to look when searching for packages manually, and twice as much to configure when you're configuring yum. It has never benefitted me, or anybody I know, but it has caught me out on any number of occasions. What's more, nobody really seems to know why it's like that: it seems it's always been that way, and nobody ever bother to fix it. So lets fix it. And how do you propose to do that? Unfortunately I'm not intimately familiar with Fedora infrastructure under-the-hood. However, I would hope that, technically at least, it wouldn't be much more than a systematic removal of all updates repositories and redirecting files which would have gone into them into the main repositories instead. This stems from the observation that if you copied everything from the main repository into updates you would have created the repository which people unfamiliar with the history would expect. The infrastructure side of this, on the face of it, sounds very simple. What you describe is effectively how the development repository is built, so it's certainly a technical possibility. It does have implications though. Off the top of my head, I can think of: 1) Composing a new everything tree for updates would lead to larger compose times. That could possibly mean that getting updates out would take> 1 day per 'push'. We've been trying to improve updates push times so it would be a bit detrimental to that goal. I can't comment on this. 2) There might be GPL compliance issues This line of reasoning sounds bizarre to me. You can publish things in multiple places. 3) You would still need an 'updates-testing' repository given that this is a supposedly stable release. So there is still going to be at least one additional repo regardless. Indeed, however that would only be testers. Most people can ignore it. However, other than 'browsing manually for packages', I'm not really sure what problem you are trying to solve by getting rid of the updates repository. It would seem like this has quite a bit of cost for relatively little to no real gain? Any tool which deals with repositories requires the repo to be configured twice. Off the top of my head I can think of: 1. yum separate repo and updates-repo in yum.conf. 2. livecd tools two repos in kickstart 3. revisor two repos in kickstart 4. febootstrap two repos on command line Given that you almost always want updates, it would an improvement if all of the above could be replaced with a single configuration. Matt -- Matthew Booth, RHCA, RHCSS Red Hat Engineering, Virtualisation Team M: +44 (0)7977 267231 GPG ID: D33C3490 GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490 -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, 2009-12-02 at 14:39 +, Matthew Booth wrote: > The separate updates directory has been a pain for as long as I've been > using RHL/Fedora Core/Fedora. It means you have two places to look when > searching for packages manually, and twice as much to configure when > you're configuring yum. It has never benefitted me, or anybody I know, > but it has caught me out on any number of occasions. What's more, nobody > really seems to know why it's like that: it seems it's always been that > way, and nobody ever bother to fix it. If the real motivation is "I want to manually browse a single directory for all the packages" I have an alternate proposal. createrepo has the ability to take a list of packages (as paths) for input. I hypothesize that we could place all rpms for a given release in a single directory (seth will hate this as he wants to split them up based on first letter of their name for better filesystem performance), yet still maintain different repo paths for the different logical separation of rpms. One path would be the "Everything" repo, which would have repodata for the GA versions of the packages. Another path would be the "updates" repo, which has repodata for the current set of stable updates, and a third path would be the "updates-testing" repo, which has repodata for the current set of testing updates. All the repodata would reference files in a central directory (or directory tree). You could achieve a single place to manually look for packages, whilst users would still have logically separated repository metadata. Of course, I'm papering over the amount of work it would take to modify our compose tools to perform in this way, and the added work mirrors would have (they don't normally have to scan the Everything tree for changes, but now they'd have to scan a giant tree of rpms every rsync to see if anything changed), and the added complexity of trying to keep track of which packages are active in the repos and which aren't, and keeping the central directory pruned of obsolete packages. I'm certainly not signing up to work on this, but I am offering this as an alternative approach. -- Jesse Keating Fedora -- Freedom² is a feature! identi.ca: http://identi.ca/jkeating signature.asc Description: This is a digitally signed message part -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, 02 Dec 2009 12:47:17 -0500, Matthew Booth wrote: Any tool which deals with repositories requires the repo to be configured twice. Off the top of my head I can think of: 1. yum separate repo and updates-repo in yum.conf. 2. livecd tools two repos in kickstart 3. revisor two repos in kickstart 4. febootstrap two repos on command line Given that you almost always want updates, it would an improvement if all of the above could be replaced with a single configuration. Matt So, create a meta-repo that just says "I'm the combination of these several repos over here"? Of course, it'd require modification of all the programs that know how to talk to repos... Personally, I think the current repo setup is just fine as it is. -- James Cassell -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On 02/12/09 17:40, Jesse Keating wrote: On Wed, 2009-12-02 at 18:09 +0100, Ralf Corsepius wrote: * It shifts "costs" from "users" to "vendor" and from "mirrors" to "master". * It helps users who are using networked installs to spare bandwidth (avoids downloading obsolete packages from "Everything"/"Fedora"). Admitted, for most users, it would change almost nothing. People doing network installs can either add the updates repo to their kickstart, or check the box in the anaconda UI, so that the updates repos are considered at install time. No download of duplicate data. In fact, having separate repos would likely cost less bandwidth. If we only had one combined repo, there would be many duplicate packages, especially if we went the route of having updates-testing mixed in and only marked by some update tag. We'd have to keep sets of what's in updates-testing, updates, and the GA release set, and all of that would be in one repodata set. Everybody doing a network install, whether they wanted updates, updates-testing, or not would have to download and consume that larger repodata, introducing a higher cost for them. There's an argument for separate updates-testing. However, my original point was that nobody is interested in the original GA release set except historians. People should, and we rightly help them, be installing updates frequently. So, people doing network installs fall into: People who want to install GA only: Hopefully nobody. The only good reason to do this is if the distro is broken. People who want to install GA+updates: Everybody except testers. People who want GA+updates+testing: testers. The GA package could be kept around as a separate, static repo nobody uses under normal circumstances. Combining GA+updates into a single repo would not consume additional bandwidth for anybody at all, and only testers would have to do any additional configuration. Matt -- Matthew Booth, RHCA, RHCSS Red Hat Engineering, Virtualisation Team M: +44 (0)7977 267231 GPG ID: D33C3490 GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490 -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, Dec 02, 2009 at 05:47:17PM +, Matthew Booth wrote: > On 02/12/09 16:06, Josh Boyer wrote: > >On Wed, Dec 02, 2009 at 03:44:08PM +, Matthew Booth wrote: > >>On 02/12/09 15:26, Josh Boyer wrote: > >>>On Wed, Dec 02, 2009 at 02:39:30PM +, Matthew Booth wrote: > The separate updates directory has been a pain for as long as I've been > using RHL/Fedora Core/Fedora. It means you have two places to look when > searching for packages manually, and twice as much to configure when > you're configuring yum. It has never benefitted me, or anybody I know, > but it has caught me out on any number of occasions. What's more, nobody > really seems to know why it's like that: it seems it's always been that > way, and nobody ever bother to fix it. > > So lets fix it. > >>> > >>>And how do you propose to do that? > >> > >>Unfortunately I'm not intimately familiar with Fedora infrastructure > >>under-the-hood. However, I would hope that, technically at least, it > >>wouldn't be much more than a systematic removal of all updates > >>repositories and redirecting files which would have gone into them into > >>the main repositories instead. This stems from the observation that if > >>you copied everything from the main repository into updates you would > >>have created the repository which people unfamiliar with the history > >>would expect. The infrastructure side of this, on the face of it, sounds > >>very simple. > > > >What you describe is effectively how the development repository is built, > >so it's certainly a technical possibility. It does have implications > >though. Off the top of my head, I can think of: > > > >1) Composing a new everything tree for updates would lead to larger > >compose times. That could possibly mean that getting updates out would > >take> 1 day per 'push'. We've been trying to improve updates push > >times so it would be a bit detrimental to that goal. > > I can't comment on this. > > >2) There might be GPL compliance issues > > This line of reasoning sounds bizarre to me. You can publish things in > multiple places. > > >3) You would still need an 'updates-testing' repository given that this > >is a supposedly stable release. So there is still going to be at least > >one additional repo regardless. > > Indeed, however that would only be testers. Most people can ignore it. > > >However, other than 'browsing manually for packages', I'm not really > >sure what problem you are trying to solve by getting rid of the > >updates repository. It would seem like this has quite a bit of cost > >for relatively little to no real gain? > > Any tool which deals with repositories requires the repo to be > configured twice. Off the top of my head I can think of: > > 1. yum > separate repo and updates-repo in yum.conf. > 2. livecd tools > two repos in kickstart > 3. revisor > two repos in kickstart > 4. febootstrap > two repos on command line > > Given that you almost always want updates, it would an improvement if > all of the above could be replaced with a single configuration. That sounds like something for a yum plugin to solve. eg Have a URL you can query to get a list of valid repos for a particular release. Have a YUM plugin implements a virtual repo, and this repo simply fetches that URL, and then exposes a package set which is the union of all repos listed. That way we still have separate YUM repos & separate metadata hosted on all the servers, but the clients can all be configured with this single virtual repo Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :| -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, 2 Dec 2009, James Cassell wrote: On Wed, 02 Dec 2009 12:47:17 -0500, Matthew Booth wrote: So, create a meta-repo that just says "I'm the combination of these several repos over here"? Of course, it'd require modification of all the programs that know how to talk to repos... it wouldn't require any code changes at all. createrepo/mergerepos can do this now and yum interacts with them fine - EXCEPT it gets into mirroring issues. :-/ -sv -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, 2009-12-02 at 11:06 -0500, Josh Boyer wrote: > so it's certainly a technical possibility. It does have implications > though. Off the top of my head, I can think of: > > 1) Composing a new everything tree for updates would lead to larger > compose times. That could possibly mean that getting updates out would > take > 1 day per 'push'. We've been trying to improve updates push > times so it would be a bit detrimental to that goal. > > 2) There might be GPL compliance issues > > 3) You would still need an 'updates-testing' repository given that this > is a supposedly stable release. So there is still going to be at least > one additional repo regardless. 4) makes it much harder to recover from broken updates by simply pulling in the original release version of the package. Yes, the correct fix to this is not to ship broken updates, but we _do_ ship broken updates, and at least people have this option available currently. (And we have 'yum downgrade' which mostly automates the procedure if you're lucky.) -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org http://www.happyassassin.net -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, 2009-12-02 at 17:58 +, Matthew Booth wrote: > The GA package could be kept around as a separate, static repo nobody > uses under normal circumstances. Combining GA+updates into a single repo > would not consume additional bandwidth for anybody at all, and only > testers would have to do any additional configuration. After initial install, a GA+updates repo would cause consumption of more bandwidth, as the repodata would change on a nearly daily basis causing rather large repodata files to be downloaded over and over and over again. Instead what we have now is a static GA repo that is downloaded usually once, and a much much much smaller updates repo that is downloaded more frequently, but since it is much smaller the impact is much lower. -- Jesse Keating Fedora -- Freedom² is a feature! identi.ca: http://identi.ca/jkeating signature.asc Description: This is a digitally signed message part -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, 2 Dec 2009, Jesse Keating wrote: On Wed, 2009-12-02 at 14:39 +, Matthew Booth wrote: The separate updates directory has been a pain for as long as I've been using RHL/Fedora Core/Fedora. It means you have two places to look when searching for packages manually, and twice as much to configure when you're configuring yum. It has never benefitted me, or anybody I know, but it has caught me out on any number of occasions. What's more, nobody really seems to know why it's like that: it seems it's always been that way, and nobody ever bother to fix it. If the real motivation is "I want to manually browse a single directory for all the packages" I have an alternate proposal. createrepo has the ability to take a list of packages (as paths) for input. I hypothesize that we could place all rpms for a given release in a single directory (seth will hate this as he wants to split them up based on first letter of their name for better filesystem performance), And better webbrowser and webserver performance since having A GIANT list eats my webbrowser often. yet still maintain different repo paths for the different logical separation of rpms. One path would be the "Everything" repo, which would have repodata for the GA versions of the packages. Another path would be the "updates" repo, which has repodata for the current set of stable updates, and a third path would be the "updates-testing" repo, which has repodata for the current set of testing updates. All the repodata would reference files in a central directory (or directory tree). the paths don't matter - you just need to generate the lists and feed that into SOME sort of metadata. Of course, I'm papering over the amount of work it would take to modify our compose tools to perform in this way, and the added work mirrors would have (they don't normally have to scan the Everything tree for changes, but now they'd have to scan a giant tree of rpms every rsync to see if anything changed), and the added complexity of trying to keep track of which packages are active in the repos and which aren't, and keeping the central directory pruned of obsolete packages. The compose tools would just have to make a list of pkgs in what dirs - a list they already have. I guess what I don't see is what the net benefit is here. the merger of repos is already happening at the yum layer. -sv -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, 2009-12-02 at 13:09 -0500, Seth Vidal wrote: > I guess what I don't see is what the net benefit is here. From what I can gather... A) a single path to look at when manually looking for rpms. B) only one config entry to mess with when modifying repos for a release (my alternate proposal does not solve this one) > > the merger of repos is already happening at the yum layer. -- Jesse Keating Fedora -- Freedom² is a feature! identi.ca: http://identi.ca/jkeating signature.asc Description: This is a digitally signed message part -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
Since people are posting wishes, here is mine: 1. stop shuffling packages from directory to directory as they get promoted/demoted from release to release 2. have a single authoritative URL for each package 3. replace static mirrors with proxy-ing of kojipkgs.fedoraproject.org (make sure it works with web infrastructure instead of fighting it) 4. generate indexes of the kojipkgs.fedoraproject.org tree that represent Fedora X, Fedora X + updates, Fedora X + testing, etc. -- Nicolas Mailhot signature.asc Description: Ceci est une partie de message numériquement signée -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
Once upon a time, Matthew Booth said: > The separate updates directory has been a pain for as long as I've been > using RHL/Fedora Core/Fedora. It means you have two places to look when > searching for packages manually, and twice as much to configure when > you're configuring yum. Are these really significant issues for a significant number of users? Not many people go looking manually for packages, since there are many tools to do it easier (yum, PK, repoquery, etc.). The repos are automatically configured with fedora-release; how often are you configuring yum? The only time I have to care about this is if I'm writing a kickstart file, but it is one extra line (and then I copy the same kickstart base over and over). -- Chris Adams Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble. -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, 2 Dec 2009, Jesse Keating wrote: On Wed, 2009-12-02 at 13:09 -0500, Seth Vidal wrote: I guess what I don't see is what the net benefit is here. From what I can gather... A) a single path to look at when manually looking for rpms. Isn't this, eventually, what the packagedb is supposed to be able to do? B) only one config entry to mess with when modifying repos for a release (my alternate proposal does not solve this one) This one seems pretty minor - you only need to do this once. and most people will do it never. -sv -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, 2 Dec 2009, Nicolas Mailhot wrote: Since people are posting wishes, here is mine: 1. stop shuffling packages from directory to directory as they get promoted/demoted from release to release we sort of do this now with hardlinks - the problem is when we have to resign the pkgs. 2. have a single authoritative URL for each package packagedb... 3. replace static mirrors with proxy-ing of kojipkgs.fedoraproject.org (make sure it works with web infrastructure instead of fighting it) I don't think that would work fine with a lot of our mirrors. 4. generate indexes of the kojipkgs.fedoraproject.org tree that represent Fedora X, Fedora X + updates, Fedora X + testing, etc. this is intriguing but expensive on kojipkgs. -sv -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, 2009-12-02 at 15:52 -0500, Seth Vidal wrote: > Isn't this, eventually, what the packagedb is supposed to be able to > do? I gather it's a "ls in a directory" kind of thing, not an interface to one tool or another kind of thing. But I could be wrong. -- Jesse Keating Fedora -- Freedom² is a feature! identi.ca: http://identi.ca/jkeating signature.asc Description: This is a digitally signed message part -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
Dne 2.12.2009 17:06, Josh Boyer napsal(a): However, other than 'browsing manually for packages', I'm not really sure what problem you are trying to solve by getting rid of the updates repository. It would seem like this has quite a bit of cost for relatively little to no real gain? I am usually too lazy, so I use yumdownloader anyway. So, I really don't see any real use case covered. Matěj -- http://www.ceplovi.cz/matej/, Jabber: mceplceplovi.cz GPG Finger: 89EF 4BC6 288A BF43 1BAB 25C3 E09F EF25 D964 84AC If Patrick Henry thought that taxation without representation was bad, he should see how bad it is with representation. -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
(on my on tangent...) On 12/02/2009 12:48 PM, Jesse Keating wrote: > I hypothesize that we could place all rpms for a given release > in a single directory (seth will hate this as he wants to split them up > based on first letter of their name for better filesystem performance), Ugh, first letter isn't really a great plan anyway. First (few) letters of a hash of the filename is much better, but obviously hurts browsability. Next best is probably something like how a dead-tree dictionary index works; list everything, split the list up by starting letters evenly, so the directories (given a really unlikely hypothetical package set) are 0/ # contains packages named 0 through 3* 4/ # 4 through 9* a/ # a through ay* az/ # az through bw* bx/ # bx through cz* da/ # da through whatever's next ... so that every directory has about the same number of things. -- Peter I'd like to start a religion. That's where the money is. -- L. Ron Hubbard to Lloyd Eshbach, in 1949; quoted by Eshbach in _Over My Shoulder_. -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On 12/02/2009 05:03 PM, Jesse Keating wrote: > On Wed, 2009-12-02 at 15:52 -0500, Seth Vidal wrote: >> Isn't this, eventually, what the packagedb is supposed to be able to >> do? > > I gather it's a "ls in a directory" kind of thing, not an interface to > one tool or another kind of thing. But I could be wrong. Sure, but the better optimization for this is "don't do it". -- Peter I'd like to start a religion. That's where the money is. -- L. Ron Hubbard to Lloyd Eshbach, in 1949; quoted by Eshbach in _Over My Shoulder_. -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On 12/02/2009 03:53 PM, Seth Vidal wrote: > On Wed, 2 Dec 2009, Nicolas Mailhot wrote: >> 3. replace static mirrors with proxy-ing of kojipkgs.fedoraproject.org >> (make sure it works with web infrastructure instead of fighting it) > > I don't think that would work fine with a lot of our mirrors. I think it probably /could/ if we put some (relatively major) work in on the server side to make it look like the directories it isn't. Not that I'm endorsing such a plan. -- Peter I'd like to start a religion. That's where the money is. -- L. Ron Hubbard to Lloyd Eshbach, in 1949; quoted by Eshbach in _Over My Shoulder_. -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, Dec 02, 2009 at 02:03:51PM -0800, Jesse Keating wrote: > On Wed, 2009-12-02 at 15:52 -0500, Seth Vidal wrote: > > Isn't this, eventually, what the packagedb is supposed to be able to > > do? > > I gather it's a "ls in a directory" kind of thing, not an interface to > one tool or another kind of thing. But I could be wrong. We're at a point where 'ls in a directory' is becoming difficult even; you can't glob 15k package names in a shell. find . is your friend. -- Matt Domsch Technology Strategist, Dell Office of the CTO linux.dell.com & www.dell.com/linux -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, Dec 02, 2009 at 07:35:03PM +0100, Nicolas Mailhot wrote: > 3. replace static mirrors with proxy-ing of kojipkgs.fedoraproject.org > (make sure it works with web infrastructure instead of fighting it) Sorry, I don't understand this. Can you elaborate? That would help me scope the impact to MirrorManager. Thanks, Matt -- Matt Domsch Technology Strategist, Dell Office of the CTO linux.dell.com & www.dell.com/linux -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, 2 Dec 2009, Peter Jones wrote: (on my on tangent...) On 12/02/2009 12:48 PM, Jesse Keating wrote: I hypothesize that we could place all rpms for a given release in a single directory (seth will hate this as he wants to split them up based on first letter of their name for better filesystem performance), Ugh, first letter isn't really a great plan anyway. First (few) letters of a hash of the filename is much better, but obviously hurts browsability. Next best is probably something like how a dead-tree dictionary index works; list everything, split the list up by starting letters evenly, so the directories (given a really unlikely hypothetical package set) are 0/ # contains packages named 0 through 3* 4/ # 4 through 9* a/ # a through ay* az/ # az through bw* bx/ # bx through cz* da/ # da through whatever's next ... so that every directory has about the same number of things. If you're looking for perfect division, sure - but the reality is this: 19K items in a single dir and ext3 and nfs and many many other things crap themselves returning that list. If you make 36 subdirs (26+10) performance gets DRAMATICALLY better for producing the same list of files. I tested it on our backend to be sure. getting the complete pkglist goes from taking 5 minutes to take 30s. yes, I said 5 minutes. -sv -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, Dec 02, 2009 at 11:09:41AM -0500, Bill Nottingham wrote: > The separate Everything tree that does not get obsoleted is required > in some form for GPL compliance, with respect to the ISO images that > we ship. Any new solution would have to preserve this. ? We provide Fedora-*-source-disc*.iso and Fedora-*-source-DVD.iso images on the mirrors already. -- Matt Domsch Technology Strategist, Dell Office of the CTO linux.dell.com & www.dell.com/linux -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, 2 Dec 2009, Peter Jones wrote: On 12/02/2009 03:53 PM, Seth Vidal wrote: On Wed, 2 Dec 2009, Nicolas Mailhot wrote: 3. replace static mirrors with proxy-ing of kojipkgs.fedoraproject.org (make sure it works with web infrastructure instead of fighting it) I don't think that would work fine with a lot of our mirrors. I think it probably /could/ if we put some (relatively major) work in on the server side to make it look like the directories it isn't. Not that I'm endorsing such a plan. we'd have to make all our mirrors use that. not gonna fly. -sv -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On 12/02/2009 05:58 PM, Seth Vidal wrote: > > > On Wed, 2 Dec 2009, Peter Jones wrote: > >> On 12/02/2009 03:53 PM, Seth Vidal wrote: >>> On Wed, 2 Dec 2009, Nicolas Mailhot wrote: 3. replace static mirrors with proxy-ing of kojipkgs.fedoraproject.org (make sure it works with web infrastructure instead of fighting it) >>> >>> I don't think that would work fine with a lot of our mirrors. >> >> I think it probably /could/ if we put some (relatively major) work in >> on the server side to make it look like the directories it isn't. Not >> that I'm endorsing such a plan. > > we'd have to make all our mirrors use that. > > not gonna fly. Well, you just wouldn't get the benefits of this if you're using a mirror. Which, yes, is pretty lame. -- Peter I'd like to start a religion. That's where the money is. -- L. Ron Hubbard to Lloyd Eshbach, in 1949; quoted by Eshbach in _Over My Shoulder_. -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On 12/02/2009 05:58 PM, Seth Vidal wrote: > > > On Wed, 2 Dec 2009, Peter Jones wrote: > >> (on my on tangent...) >> >> On 12/02/2009 12:48 PM, Jesse Keating wrote: >>> I hypothesize that we could place all rpms for a given release >>> in a single directory (seth will hate this as he wants to split them up >>> based on first letter of their name for better filesystem performance), >> >> Ugh, first letter isn't really a great plan anyway. First (few) letters >> of a hash of the filename is much better, but obviously hurts >> browsability. >> Next best is probably something like how a dead-tree dictionary index >> works; >> list everything, split the list up by starting letters evenly, so the >> directories (given a really unlikely hypothetical package set) are >> >> 0/# contains packages named 0 through 3* >> 4/# 4 through 9* >> a/# a through ay* >> az/# az through bw* >> bx/# bx through cz* >> da/# da through whatever's next >> ... >> >> so that every directory has about the same number of things. > > If you're looking for perfect division, sure - but the reality is this: > > 19K items in a single dir and ext3 and nfs and many many other things > crap themselves returning that list. > > If you make 36 subdirs (26+10) performance gets DRAMATICALLY better for > producing the same list of files. > > I tested it on our backend to be sure. getting the complete pkglist goes > from taking 5 minutes to take 30s. > > yes, I said 5 minutes. Yeah, I buy that. -- Peter I'd like to start a religion. That's where the money is. -- L. Ron Hubbard to Lloyd Eshbach, in 1949; quoted by Eshbach in _Over My Shoulder_. -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, 2 Dec 2009, Peter Jones wrote: On 12/02/2009 05:58 PM, Seth Vidal wrote: On Wed, 2 Dec 2009, Peter Jones wrote: On 12/02/2009 03:53 PM, Seth Vidal wrote: On Wed, 2 Dec 2009, Nicolas Mailhot wrote: 3. replace static mirrors with proxy-ing of kojipkgs.fedoraproject.org (make sure it works with web infrastructure instead of fighting it) I don't think that would work fine with a lot of our mirrors. I think it probably /could/ if we put some (relatively major) work in on the server side to make it look like the directories it isn't. Not that I'm endorsing such a plan. we'd have to make all our mirrors use that. not gonna fly. Well, you just wouldn't get the benefits of this if you're using a mirror. Which, yes, is pretty lame. we won't get the benefits in fedora then. Our mirror infrastructure is a HUGE reason why we can do what we do. If we can't farm things out to our mirrors then we might as well go home b/c our users will NEVER be able to get our bits. -sv -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, 2009-12-02 at 17:46 -0500, Peter Jones wrote: > (on my on tangent...) > > On 12/02/2009 12:48 PM, Jesse Keating wrote: > > I hypothesize that we could place all rpms for a given release > > in a single directory (seth will hate this as he wants to split them up > > based on first letter of their name for better filesystem performance), > > Ugh, first letter isn't really a great plan anyway. It's not "great" but it's better than the current "all in one dir." and easy to do, and I haven't heard of any downsides (that aren't applicable to "all in one dir."). > First (few) letters > of a hash of the filename is much better, but obviously hurts browsability. Right, which is important enough to not do that IMO. I'm sure Jesse wouldn't like finding packages to mean using find. > Next best is probably something like how a dead-tree dictionary index works; > list everything, split the list up by starting letters evenly, so the > directories (given a really unlikely hypothetical package set) are > > 0/# contains packages named 0 through 3* > 4/# 4 through 9* > a/# a through ay* > az/ # az through bw* > bx/ # bx through cz* > da/ # da through whatever's next > ... > > so that every directory has about the same number of things. This should be fairly easy to code, but has a big downside: Packages will move directories. 1. This will upset yum (old repo. MD == no packages download). 2. This might upset mirrors. 3. This will probably upset users: i. URLs will only be safe until the next mash, they won't even be able to bookmark /y if they just want to see yum each time. ii. Users have to look/parse the directory layout each time they visit to see what "blah" is. -- James Antill - ja...@fedoraproject.org http://yum.baseurl.org/wiki/releases http://yum.baseurl.org/wiki/whatsnew/3.2.25 http://yum.baseurl.org/wiki/YumMultipleMachineCaching -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On 12/02/2009 06:05 PM, James Antill wrote: > On Wed, 2009-12-02 at 17:46 -0500, Peter Jones wrote: >> so that every directory has about the same number of things. > > This should be fairly easy to code, but has a big downside: > > Packages will move directories. > > 1. This will upset yum (old repo. MD == no packages download). > 2. This might upset mirrors. > 3. This will probably upset users: > i. URLs will only be safe until the next mash, they > won't even be able to bookmark /y if they just > want to see yum each time. > ii. Users have to look/parse the directory layout each > time they visit to see what "blah" is. Yeah. Seth makes a good point - first letter is such vast improvement that we probably don't need to worry about doing better - at least for now. -- Peter I'd like to start a religion. That's where the money is. -- L. Ron Hubbard to Lloyd Eshbach, in 1949; quoted by Eshbach in _Over My Shoulder_. -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, 2009-12-02 at 16:58 -0600, Matt Domsch wrote: > On Wed, Dec 02, 2009 at 11:09:41AM -0500, Bill Nottingham wrote: > > The separate Everything tree that does not get obsoleted is required > > in some form for GPL compliance, with respect to the ISO images that > > we ship. Any new solution would have to preserve this. > > ? We provide Fedora-*-source-disc*.iso and Fedora-*-source-DVD.iso > images on the mirrors already. > As stated elsewhere, that doesn't carry the packages found on all the live images we produce. -- Jesse Keating Fedora -- Freedom² is a feature! identi.ca: http://identi.ca/jkeating signature.asc Description: This is a digitally signed message part -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, Dec 02, 2009 at 03:08:06PM -0800, Jesse Keating wrote: > On Wed, 2009-12-02 at 16:58 -0600, Matt Domsch wrote: > > On Wed, Dec 02, 2009 at 11:09:41AM -0500, Bill Nottingham wrote: > > > The separate Everything tree that does not get obsoleted is required > > > in some form for GPL compliance, with respect to the ISO images that > > > we ship. Any new solution would have to preserve this. > > > > ? We provide Fedora-*-source-disc*.iso and Fedora-*-source-DVD.iso > > images on the mirrors already. > > > > As stated elsewhere, that doesn't carry the packages found on all the > live images we produce. All the more reason to put more effort into http://git.fedorahosted.org/git/correspondingsource.git -- Matt Domsch Technology Strategist, Dell Office of the CTO linux.dell.com & www.dell.com/linux -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, Dec 02, 2009 at 17:58:24 -0500, Seth Vidal wrote: > > I tested it on our backend to be sure. getting the complete pkglist > goes from taking 5 minutes to take 30s. > > yes, I said 5 minutes. Have you tried any of the tunning knobs to have the directory cache be alotted more space or given higher priority? For example: vfs_cache_pressure -- Controls the tendency of the kernel to reclaim the memory which is used for caching of directory and inode objects. At the default value of vfs_cache_pressure=100 the kernel will attempt to reclaim dentries and inodes at a "fair" rate with respect to pagecache and swapcache reclaim. Decreasing vfs_cache_pressure causes the kernel to prefer to retain dentry and inode caches. Increasing vfs_cache_pressure beyond 100 causes the kernel to prefer to reclaim dentries and inodes. -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, 2 Dec 2009, Bruno Wolff III wrote: On Wed, Dec 02, 2009 at 17:58:24 -0500, Seth Vidal wrote: I tested it on our backend to be sure. getting the complete pkglist goes from taking 5 minutes to take 30s. yes, I said 5 minutes. Have you tried any of the tunning knobs to have the directory cache be alotted more space or given higher priority? For example: vfs_cache_pressure -- Controls the tendency of the kernel to reclaim the memory which is used for caching of directory and inode objects. At the default value of vfs_cache_pressure=100 the kernel will attempt to reclaim dentries and inodes at a "fair" rate with respect to pagecache and swapcache reclaim. Decreasing vfs_cache_pressure causes the kernel to prefer to retain dentry and inode caches. Increasing vfs_cache_pressure beyond 100 causes the kernel to prefer to reclaim dentries and inodes. Not tried that - worth giving it a shot - but it still won't help the 'holy crap there are 15K items on this webpage and it won't render' problem. -sv -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
Seth Vidal wrote: > If you're looking for perfect division, sure - but the reality is this: > > 19K items in a single dir and ext3 and nfs and many many other things crap > themselves returning that list. > > If you make 36 subdirs (26+10) performance gets DRAMATICALLY better for > producing the same list of files. The problem is, a few of those starting letters still correspond to A LOT of packages, e.g. p* matches perl-*, python-* and php-* and that's a lot of stuff (especially Perl and Python). And adding python3-* (and perl6-*, or are we going to use the rakudo-* namespace there?) won't make it any less. Kevin Kofler -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Thu, 3 Dec 2009, Kevin Kofler wrote: Seth Vidal wrote: If you're looking for perfect division, sure - but the reality is this: 19K items in a single dir and ext3 and nfs and many many other things crap themselves returning that list. If you make 36 subdirs (26+10) performance gets DRAMATICALLY better for producing the same list of files. The problem is, a few of those starting letters still correspond to A LOT of packages, e.g. p* matches perl-*, python-* and php-* and that's a lot of stuff (especially Perl and Python). And adding python3-* (and perl6-*, or are we going to use the rakudo-* namespace there?) won't make it any less. And yet, when tested, just making those 36 subdirs made a HUGE difference. What I've found is getting any one dir down below 3K entries makes things faster, by a lot. -sv -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, Dec 02, 2009 at 12:01:49PM -0500, Casey Dahlin wrote: > >You owe me $5. It hasn't been a week and you haven't sent me your address. I did notice though, so keep up the good work ;) josh -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, 2 Dec 2009, Bruno Wolff III wrote: > On Wed, Dec 02, 2009 at 17:58:24 -0500, > Seth Vidal wrote: > > > > I tested it on our backend to be sure. getting the complete pkglist > > goes from taking 5 minutes to take 30s. > > > > yes, I said 5 minutes. > > Have you tried any of the tunning knobs to have the directory cache be > alotted more space or given higher priority? For example: > > vfs_cache_pressure > -- > > Controls the tendency of the kernel to reclaim the memory which is used for > caching of directory and inode objects. > > At the default value of vfs_cache_pressure=100 the kernel will attempt to > reclaim dentries and inodes at a "fair" rate with respect to pagecache and > swapcache reclaim. Decreasing vfs_cache_pressure causes the kernel to prefer > to retain dentry and inode caches. Increasing vfs_cache_pressure beyond 100 > causes the kernel to prefer to reclaim dentries and inodes. > We have experimented with this for the purposes of rsync on our mirrors but haven't experimented for this specific issue. -Mike -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On 12/02/2009 06:40 PM, Jesse Keating wrote: On Wed, 2009-12-02 at 18:09 +0100, Ralf Corsepius wrote: * It shifts "costs" from "users" to "vendor" and from "mirrors" to "master". * It helps users who are using networked installs to spare bandwidth (avoids downloading obsolete packages from "Everything"/"Fedora"). Admitted, for most users, it would change almost nothing. People doing network installs can either add the updates repo to their kickstart, or check the box in the anaconda UI, so that the updates repos are considered at install time. No download of duplicate data. Yes, for people who are doing "full featured networked installs" w/ custom kickstart files. I've never met such a person. In fact, having separate repos would likely cost less bandwidth. If we only had one combined repo, there would be many duplicate packages, Where? Unlike now, where you have each package twice (in Everything and "updates"), you would have each package only once: In Everything. It would help people like me, who are locally reusing downloaded packages in a networked environment, instead of letting each machine accesses the original repos directly. especially if we went the route of having updates-testing mixed in and only marked by some update tag. Updates-testing is an add-on repo to "Everything+updates". A merged new Everything would not change anything wrt. "updates-testing". The only difference would be packages being pushed to "Everything" instead of "updates". We'd have to keep sets of what's in updates-testing, updates, and the GA release set, and all of that would be in one repodata set. Everybody doing a network install, whether they wanted updates, updates-testing, or not would have to download and consume that larger repodata, introducing a higher cost for them. Your concern is the bigger repodata? Reality check: # ls -h -s releases/11/Everything/x86_64/os/repodata/*.sqlite.bz2 updates/11/x86_64/repodata/*.sqlite.bz2 16M releases/11/Everything/x86_64/os/repodata/0301ed1cbf01c00cdf9c42b71df6c74947d9c76a3c9767e8f85a99ef6fdccb86-filelists.sqlite.bz2 11M releases/11/Everything/x86_64/os/repodata/6a8bfab8ebcbc79f9827f5b16bc1bd1573c068f141bf47c6f216e72dd8b60ff0-primary.sqlite.bz2 5.8M releases/11/Everything/x86_64/os/repodata/d3405c970a0c27e9dc3fd3ed179af33fee9a72bf704f45f1ed96a9063b40d7c7-other.sqlite.bz2 9.0M updates/11/x86_64/repodata/3a725e07adff9d7e56e7fd8bd3091f38107723987b3b614220df72494dc6814d-filelists.sqlite.bz2 6.0M updates/11/x86_64/repodata/54ca116c2a00e87eaca6491adb9e077f4d3f0d5b20e7cbab984774aefb042d0f-primary.sqlite.bz2 3.2M updates/11/x86_64/repodata/646eb24cfe113de569853a4e05f55773b3ad9531dee646e7d843b8dc4a849780-other.sqlite.bz2 => An estimate for the increase in downloaded files's sizes you are talking about is ca. factor 2, from 18.2M (current "updates") to 32.8M+ (current "Everything"+"newly introduced packages) Whether this increase in download-size is "significant" is up to the beholder. For me, it gets lost in the noise of accessing a "good" or a "bad" connection to a mirror and in the time required to download packages from mirrors. On the other hand, the content (the packages referenced inside) of "updates" overlaps with packages in Everything => The number of packages to be considered in dep-computation should become much (?) smaller. However, I am not knowledgable enough with yum's internals to estimate the impact this would have on turnaround-times, memory and diskspace requirements this would have on yum. Ralf -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On 12/02/2009 07:09 PM, Seth Vidal wrote: the merger of repos is already happening at the yum layer. On the client's side - With a combined Everything+updates, this would happen on the server side. It's one of the aspects which made me said "a combined Everything+updates shifts costs from client to server". Ralf -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Thu, 3 Dec 2009, Ralf Corsepius wrote: On 12/02/2009 07:09 PM, Seth Vidal wrote: the merger of repos is already happening at the yum layer. On the client's side - With a combined Everything+updates, this would happen on the server side. It's one of the aspects which made me said "a combined Everything+updates shifts costs from client to server". except they wouldn't be merged on the server side -it would just be additive. We wouldn't be talking about removing the original GA set - just adding updated pkgs into the path. So you'd still have the number of pkgs -just all in one repo, that you have to download all of the metadata for all of the more often, despite that 15K of them don't CHANGE. this is why it is less good for our users. -sv -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On 12/03/2009 06:32 AM, Seth Vidal wrote: On Thu, 3 Dec 2009, Ralf Corsepius wrote: On 12/02/2009 07:09 PM, Seth Vidal wrote: the merger of repos is already happening at the yum layer. On the client's side - With a combined Everything+updates, this would happen on the server side. It's one of the aspects which made me said "a combined Everything+updates shifts costs from client to server". except they wouldn't be merged on the server side -it would just be additive. We wouldn't be talking about removing the original GA set - just adding updated pkgs into the path. Woa!!! With all due respect, but this would seem an stupid and silly plan to me. So you'd still have the number of pkgs -just all in one repo, that you have to download all of the metadata for all of the more often, despite that 15K of them don't CHANGE. this is why it is less good for our users. Yes - cf. above. I have nothing to add. Ralf -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
Ralf Corsepius wrote: Your concern is the bigger repodata? My download of repodata towards the end of a release, or from rawhide, is usually bigger than my download of packages. So yes, this would make a difference. On the otherhand, there probably could be a repodiff that would alleviate a lot of this problem. -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Thu, 2009-12-03 at 00:32 -0500, Seth Vidal wrote: > We wouldn't be talking about removing the original GA set - just adding > updated pkgs into the path. So you'd still have the number of pkgs -just > all in one repo, that you have to download all of the metadata for all of > the more often, despite that 15K of them don't CHANGE. I don't think that was actually made clear in the initial proposal. I'd been assuming that the proposal was _exactly_ to remove the GA set. Usually, when a newer package shows up in any given repository, we don't keep the previous version of the package, do we? So I assumed the proposal was expecting that behaviour for the combined repository. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org http://www.happyassassin.net -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
* Adam Williamson [03/12/2009 10:10] : > > I don't think that was actually made clear in the initial proposal. I'd > been assuming that the proposal was _exactly_ to remove the GA set. No can do. People who install from the netinst CD or do PXE installs without adding the updates repo during the installation would be unable to finish it. Emmanuel -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
Le Mer 2 décembre 2009 23:56, Matt Domsch a écrit : > > On Wed, Dec 02, 2009 at 07:35:03PM +0100, Nicolas Mailhot wrote: >> 3. replace static mirrors with proxy-ing of kojipkgs.fedoraproject.org >> (make sure it works with web infrastructure instead of fighting it) > > Sorry, I don't understand this. Can you elaborate? That would help > me scope the impact to MirrorManager. Right now the same package moves from master URL to master URL as it is pushed from testing to updates to GA to whatever. That means the same package gets downloaded many times over because it changed URL (and browsers, proxies, etc understand new url = new file) My proposal is to never move a package, put it in a single unchanging place after a koji build, and just index it or not in the relevant repodata when it gets promoted or demoted. -- Nicolas Mailhot -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
Matt Domsch writes: > We're at a point where 'ls in a directory' is becoming difficult even; > you can't glob 15k package names in a shell. Recent kernels have no argv limit any more. Andreas. -- Andreas Schwab, sch...@redhat.com GPG Key fingerprint = D4E8 DBE3 3813 BB5D FA84 5EC7 45C6 250E 6F00 984E "And now for something completely different." -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Thu, 3 Dec 2009, Adam Williamson wrote: On Thu, 2009-12-03 at 00:32 -0500, Seth Vidal wrote: We wouldn't be talking about removing the original GA set - just adding updated pkgs into the path. So you'd still have the number of pkgs -just all in one repo, that you have to download all of the metadata for all of the more often, despite that 15K of them don't CHANGE. I don't think that was actually made clear in the initial proposal. I'd been assuming that the proposal was _exactly_ to remove the GA set. Usually, when a newer package shows up in any given repository, we don't keep the previous version of the package, do we? So I assumed the proposal was expecting that behaviour for the combined repository. From a QA standpoint I'd think you'd want at least one known-installable set of pkgs. Since everything after the original GA set is a giant questionmark. Not to mention that removing all the old pkgs would more or less make deltarpms very difficult. -sv -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Thu, 2009-12-03 at 10:29 +0100, Nicolas Mailhot wrote: > > Le Mer 2 décembre 2009 23:56, Matt Domsch a écrit : > > > > On Wed, Dec 02, 2009 at 07:35:03PM +0100, Nicolas Mailhot wrote: > >> 3. replace static mirrors with proxy-ing of kojipkgs.fedoraproject.org > >> (make sure it works with web infrastructure instead of fighting it) > > > > Sorry, I don't understand this. Can you elaborate? That would help > > me scope the impact to MirrorManager. > > Right now the same package moves from master URL to master URL as it is pushed > from testing to updates to GA to whatever. That means the same package gets > downloaded many times over because it changed URL (and browsers, proxies, etc > understand new url = new file) It only moves once, at least in the vast majority of cases. > My proposal is to never move a package, put it in a single unchanging place > after a koji build, and just index it or not in the relevant repodata when it > gets promoted or demoted. One problem is that atm. when it moves from testing to updates, or rawhide to GA, it also gets resigned with a different key ... so the bits are not the same. Having the URL be the same will not be helpful until that changes. -- James Antill - ja...@fedoraproject.org http://yum.baseurl.org/wiki/releases http://yum.baseurl.org/wiki/whatsnew/3.2.25 http://yum.baseurl.org/wiki/YumMultipleMachineCaching -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
I think that idea maybe isn't benefit with repository. If updates repository is merged into Everything repository, Will metadata files become too large? I know that the size of metadatas on updates and everything are more than 30 megabytes. If these two repositories compose, We will need download more than 30MB files before updating. I believe that it will decrease the advantage of delrarpm update. It will increase more time to download the files. Is there any way to make createrepo generate delta metadata files? Just like delta rpm, we just need to get what meta files are changed, then yum generates the entire metadata files. We exactly not need and want to download the big files. Additionally, if an user is far away from repository server, though ISP provides quite a good 10Mbps connection, it is still quite slow to install some big applications, such OpenOffice.org, Eclipse, Netbeans, KOffice and some 3D games. Does yum allow to download more than one rpm files with only one thread per files after checking dependencies? Or using a yum plugin to replace yum default downloader by axel to download one rpm files with multi threads. I ever found out this kind of plugin, but I can assure it can run in Fedora 12. -- My Page: http://www.liangsuilong.info Fedora Project Contributor -- Packager && Ambassador https://fedoraproject.org/wiki/User:Liangsuilong -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Dec 3, 2009, at 5:28, James Antill wrote: On Thu, 2009-12-03 at 10:29 +0100, Nicolas Mailhot wrote: Le Mer 2 décembre 2009 23:56, Matt Domsch a écrit : On Wed, Dec 02, 2009 at 07:35:03PM +0100, Nicolas Mailhot wrote: 3. replace static mirrors with proxy-ing of kojipkgs.fedoraproject.org (make sure it works with web infrastructure instead of fighting it) Sorry, I don't understand this. Can you elaborate? That would help me scope the impact to MirrorManager. Right now the same package moves from master URL to master URL as it is pushed from testing to updates to GA to whatever. That means the same package gets downloaded many times over because it changed URL (and browsers, proxies, etc understand new url = new file) It only moves once, at least in the vast majority of cases. My proposal is to never move a package, put it in a single unchanging place after a koji build, and just index it or not in the relevant repodata when it gets promoted or demoted. One problem is that atm. when it moves from testing to updates, or rawhide to GA, it also gets resigned with a different key ... so the bits are not the same. Having the URL be the same will not be helpful until that changes. -- That is no longer true. We used a single key for all of f11 and a single key for all of f12. -- Jes -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Thu, 2009-12-03 at 08:20 -0500, Seth Vidal wrote: > >> We wouldn't be talking about removing the original GA set - just adding > >> updated pkgs into the path. So you'd still have the number of pkgs -just > >> all in one repo, that you have to download all of the metadata for all of > >> the more often, despite that 15K of them don't CHANGE. > > > > I don't think that was actually made clear in the initial proposal. I'd > > been assuming that the proposal was _exactly_ to remove the GA set. > > Usually, when a newer package shows up in any given repository, we don't > > keep the previous version of the package, do we? So I assumed the > > proposal was expecting that behaviour for the combined repository. > > >From a QA standpoint I'd think you'd want at least one known-installable > set of pkgs. Since everything after the original GA set is a giant > questionmark. > > Not to mention that removing all the old pkgs would more or less make > deltarpms very difficult. I'm not saying I support the proposal, I don't, I think it's a waste of effort for no benefit. I was just clarifying the initial characterization. Actually I think the initial proposer _was_ expecting to remove initial packages when updates become available, because they keep saying stuff like 'only historians are interested in the GA packages'. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org http://www.happyassassin.net -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On 12/03/2009 12:24 AM, Ralf Corsepius wrote: > On 12/02/2009 06:40 PM, Jesse Keating wrote: >> People doing network installs can either add the updates repo to their >> kickstart, or check the box in the anaconda UI, so that the updates >> repos are considered at install time. No download of duplicate data. > Yes, for people who are doing "full featured networked installs" w/ > custom kickstart files. I've never met such a person. Really? This very much seems to me like it's a vast majority of kickstarts that happen. -- Peter Power corrupts. Absolute power is kind of neat. -- John Lehman, Secretary of the Navy, 1981-1987 -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On 12/03/2009 08:20 AM, Seth Vidal wrote: > > > On Thu, 3 Dec 2009, Adam Williamson wrote: > >> On Thu, 2009-12-03 at 00:32 -0500, Seth Vidal wrote: >> >>> We wouldn't be talking about removing the original GA set - just adding >>> updated pkgs into the path. So you'd still have the number of pkgs -just >>> all in one repo, that you have to download all of the metadata for >>> all of >>> the more often, despite that 15K of them don't CHANGE. >> >> I don't think that was actually made clear in the initial proposal. I'd >> been assuming that the proposal was _exactly_ to remove the GA set. >> Usually, when a newer package shows up in any given repository, we don't >> keep the previous version of the package, do we? So I assumed the >> proposal was expecting that behaviour for the combined repository. > >> From a QA standpoint I'd think you'd want at least one known-installable > set of pkgs. Since everything after the original GA set is a giant > questionmark. > > Not to mention that removing all the old pkgs would more or less make > deltarpms very difficult. Well, if we're talking about removing them from Fedora/ but leaving them in Everything/ (am I understanding the current form of the proposal correctly?), then it's not really significantly more difficult, but it is one more process that needs modification in order to enact such a plan. -- Peter If the facts don't fit the theory, change the facts. -- Einstein -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On 12/02/2009 09:12 PM, Kevin Kofler wrote: > Seth Vidal wrote: >> If you're looking for perfect division, sure - but the reality is this: >> >> 19K items in a single dir and ext3 and nfs and many many other things crap >> themselves returning that list. >> >> If you make 36 subdirs (26+10) performance gets DRAMATICALLY better for >> producing the same list of files. > > The problem is, a few of those starting letters still correspond to A LOT of > packages, e.g. p* matches perl-*, python-* and php-* and that's a lot of > stuff (especially Perl and Python). And adding python3-* (and perl6-*, or > are we going to use the rakudo-* namespace there?) won't make it any less. Even still, separating "p" out to a subdirectory means that subdirectory has an order of magnitude fewer files than the previous way. That's a really big difference. -- Peter If the facts don't fit the theory, change the facts. -- Einstein -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Thu, 2009-12-03 at 06:51 +0100, Ralf Corsepius wrote: > > We wouldn't be talking about removing the original GA set - just adding > > updated pkgs into the path. > > Woa!!! With all due respect, but this would seem an stupid and silly > plan to me. The only way not to do that would be to maintain yet another directory of packages that matches the GA set, for GPL compliance. Now we're just getting silly. -- Jesse Keating Fedora -- Freedom² is a feature! identi.ca: http://identi.ca/jkeating signature.asc Description: This is a digitally signed message part -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Thu, 2009-12-03 at 06:24 +0100, Ralf Corsepius wrote: > > People doing network installs can either add the updates repo to their > > kickstart, or check the box in the anaconda UI, so that the updates > > repos are considered at install time. No download of duplicate data. > Yes, for people who are doing "full featured networked installs" w/ > custom kickstart files. I've never met such a person. Really? I meet people who use kickstart all the time. Any sizable deployment of Fedora/RHEL uses or should use kickstart. And those that don't aren't afraid to check that little 'updates' box at the software selection screen. You seemed to have ignored that part of my point. > > > In fact, having separate repos would likely cost less bandwidth. If we > > only had one combined repo, there would be many duplicate packages, > Where? Unlike now, where you have each package twice (in Everything and > "updates"), you would have each package only once: In Everything. That assumes we purge anything but the latest version of a package, which as noted in other parts of this thread gets complicated with GPL compliance. > > It would help people like me, who are locally reusing downloaded > packages in a networked environment, instead of letting each machine > accesses the original repos directly. Surely a caching proxy server doesn't care what folder a file comes from, it'll cache it. Whether the new file shows up in Everything/ or the new file shows up in updates/ shouldn't matter. > > > especially if we went the route of having updates-testing mixed in and > > only marked by some update tag. > Updates-testing is an add-on repo to "Everything+updates". > > A merged new Everything would not change anything wrt. > "updates-testing". The only difference would be packages being pushed to > "Everything" instead of "updates". > > > We'd have to keep sets of what's in > > updates-testing, updates, and the GA release set, and all of that would > > be in one repodata set. Everybody doing a network install, whether they > > wanted updates, updates-testing, or not would have to download and > > consume that larger repodata, introducing a higher cost for them. > Your concern is the bigger repodata? > > Reality check: > # ls -h -s releases/11/Everything/x86_64/os/repodata/*.sqlite.bz2 > updates/11/x86_64/repodata/*.sqlite.bz2 > > 16M > releases/11/Everything/x86_64/os/repodata/0301ed1cbf01c00cdf9c42b71df6c74947d9c76a3c9767e8f85a99ef6fdccb86-filelists.sqlite.bz2 > 11M > releases/11/Everything/x86_64/os/repodata/6a8bfab8ebcbc79f9827f5b16bc1bd1573c068f141bf47c6f216e72dd8b60ff0-primary.sqlite.bz2 > 5.8M > releases/11/Everything/x86_64/os/repodata/d3405c970a0c27e9dc3fd3ed179af33fee9a72bf704f45f1ed96a9063b40d7c7-other.sqlite.bz2 > > 9.0M > updates/11/x86_64/repodata/3a725e07adff9d7e56e7fd8bd3091f38107723987b3b614220df72494dc6814d-filelists.sqlite.bz2 > 6.0M > updates/11/x86_64/repodata/54ca116c2a00e87eaca6491adb9e077f4d3f0d5b20e7cbab984774aefb042d0f-primary.sqlite.bz2 > 3.2M > updates/11/x86_64/repodata/646eb24cfe113de569853a4e05f55773b3ad9531dee646e7d843b8dc4a849780-other.sqlite.bz2 > > => An estimate for the increase in downloaded files's sizes you are > talking about is ca. factor 2, from 18.2M (current "updates") > to 32.8M+ (current "Everything"+"newly introduced packages) > > > Whether this increase in download-size is "significant" is up to the > beholder. For me, it gets lost in the noise of accessing a "good" or a > "bad" connection to a mirror and in the time required to download > packages from mirrors. 33~ megs downloaded every single time an update is pushed is a significant hit for a fair number of people. That was why it was important to make yum not re-fetch that repodata every time, and use a cached version of it. -- Jesse Keating Fedora -- Freedom² is a feature! identi.ca: http://identi.ca/jkeating signature.asc Description: This is a digitally signed message part -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On 12/03/2009 07:22 AM, Jesse Keating wrote: On Thu, 2009-12-03 at 06:24 +0100, Ralf Corsepius wrote: People doing network installs can either add the updates repo to their kickstart, or check the box in the anaconda UI, so that the updates repos are considered at install time. No download of duplicate data. Yes, for people who are doing "full featured networked installs" w/ custom kickstart files. I've never met such a person. Really? I meet people who use kickstart all the time. May-be internal at RH? Any sizable deployment of Fedora/RHEL uses or should use kickstart. And those that don't aren't afraid to check that little 'updates' box at the software selection screen. You seemed to have ignored that part of my point. No, I didn't. It's just that unless this "little check button" is the default, many users will ignore it or as in my case ... I am normally "yum-upgrading" between distros and rarely use anaconda. In fact, having separate repos would likely cost less bandwidth. If we only had one combined repo, there would be many duplicate packages, Where? Unlike now, where you have each package twice (in Everything and "updates"), you would have each package only once: In Everything. That assumes we purge anything but the latest version of a package, Correct. which as noted in other parts of this thread gets complicated with GPL compliance. Not necessarily - E.g. it would be GPL compliant to store "purged packages" outside of the "current" repos. And whether "spins" and the way they currently are implemented is "good"/"feasible"/"reasonable" is a different question. => An estimate for the increase in downloaded files's sizes you are talking about is ca. factor 2, from 18.2M (current "updates") to 32.8M+ (current "Everything"+"newly introduced packages) Whether this increase in download-size is "significant" is up to the beholder. For me, it gets lost in the noise of accessing a "good" or a "bad" connection to a mirror and in the time required to download packages from mirrors. 33~ megs downloaded every single time an update is pushed is a significant hit for a fair number of people. Yes, but ... some more figures: The same figures as above for FC10: => Everything: 25.8M => updates: 18.5M => A rough estimate for sizes of repodata for a "near EOL'ed" Fedora: 70% of the size of "Everything's repodata". I.e. should this estimate apply to later Fedoras, Fedora 11 users are likely to see 70% of 33MB = 23MB near EOL of Fedora 11. That was why it was important to make yum not re-fetch that repodata every time, and use a cached version of it. Yes, the keys to minimize bandwidth demands would be * to shink the size of the repodata/-files * to shink the size of "changes" to them. Besides obvious solutions, such as using a different compression (e.g. xz instead of bz2) and minimizing their contents, one could ship repodata/-files in "chunks". What I mean: In theory, one could * update repodata/-files incrementally by shipping some kind of "deltas". * split repodata/-files into several, e.g. sorted by "some other criteria", i.e. to provide several sets of *-[primary,filelist,other] files. One "package push", then would only affect a subset of the files, but not all. - This is very similar to what (IIRC) Seth had proposed (Split the repo into several repos, alphabetically), except that the "split" would happen inside of repodata and thus be transparent to users. I am not sure how difficult to implement this would be. Ralf -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Fri, December 4, 2009 9:20 pm, Ralf Corsepius wrote: > On 12/03/2009 07:22 AM, Jesse Keating wrote: >> On Thu, 2009-12-03 at 06:24 +0100, Ralf Corsepius wrote: >>> Yes, for people who are doing "full featured networked installs" w/ >>> custom kickstart files. I've never met such a person. >> >> Really? I meet people who use kickstart all the time. > May-be internal at RH? I do every install via kickstart - small company with 30-50 machines. Been doing fedora+everything+updates installs for many releases now. In fact - every "upgrade" is a fresh kickstart install + restore critical files from backup. -- Orion Poplawski Technical Manager 303-415-9701 x222 NWRA/CoRA DivisionFAX: 303-415-9702 3380 Mitchell Lane or...@cora.nwra.com Boulder, CO 80301 http://www.cora.nwra.com -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On 12/05/2009 06:22 AM, Orion Poplawski wrote: On Fri, December 4, 2009 9:20 pm, Ralf Corsepius wrote: On 12/03/2009 07:22 AM, Jesse Keating wrote: On Thu, 2009-12-03 at 06:24 +0100, Ralf Corsepius wrote: Yes, for people who are doing "full featured networked installs" w/ custom kickstart files. I've never met such a person. Really? I meet people who use kickstart all the time. May-be internal at RH? I do every install via kickstart - small company with 30-50 machines. Been doing fedora+everything+updates installs for many releases now. OK, then it's likely a "full time" professional admins vs. "part time"/"side-job" admins and "home-users" thing. Ralf -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
