Re: Proposed F13 feature: drop separate updates repository
On Thu, 2009-12-03 at 06:51 +0100, Ralf Corsepius wrote: We wouldn't be talking about removing the original GA set - just adding updated pkgs into the path. Woa!!! With all due respect, but this would seem an stupid and silly plan to me. The only way not to do that would be to maintain yet another directory of packages that matches the GA set, for GPL compliance. Now we're just getting silly. -- Jesse Keating Fedora -- Freedom² is a feature! identi.ca: http://identi.ca/jkeating signature.asc Description: This is a digitally signed message part -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On 12/03/2009 07:22 AM, Jesse Keating wrote: On Thu, 2009-12-03 at 06:24 +0100, Ralf Corsepius wrote: People doing network installs can either add the updates repo to their kickstart, or check the box in the anaconda UI, so that the updates repos are considered at install time. No download of duplicate data. Yes, for people who are doing full featured networked installs w/ custom kickstart files. I've never met such a person. Really? I meet people who use kickstart all the time. May-be internal at RH? Any sizable deployment of Fedora/RHEL uses or should use kickstart. And those that don't aren't afraid to check that little 'updates' box at the software selection screen. You seemed to have ignored that part of my point. No, I didn't. It's just that unless this little check button is the default, many users will ignore it or as in my case ... I am normally yum-upgrading between distros and rarely use anaconda. In fact, having separate repos would likely cost less bandwidth. If we only had one combined repo, there would be many duplicate packages, Where? Unlike now, where you have each package twice (in Everything and updates), you would have each package only once: In Everything. That assumes we purge anything but the latest version of a package, Correct. which as noted in other parts of this thread gets complicated with GPL compliance. Not necessarily - E.g. it would be GPL compliant to store purged packages outside of the current repos. And whether spins and the way they currently are implemented is good/feasible/reasonable is a different question. = An estimate for the increase in downloaded files's sizes you are talking about is ca. factor 2, from 18.2M (current updates) to 32.8M+ (current Everything+newly introduced packages) Whether this increase in download-size is significant is up to the beholder. For me, it gets lost in the noise of accessing a good or a bad connection to a mirror and in the time required to download packages from mirrors. 33~ megs downloaded every single time an update is pushed is a significant hit for a fair number of people. Yes, but ... some more figures: The same figures as above for FC10: = Everything: 25.8M = updates: 18.5M = A rough estimate for sizes of repodata for a near EOL'ed Fedora: 70% of the size of Everything's repodata. I.e. should this estimate apply to later Fedoras, Fedora 11 users are likely to see 70% of 33MB = 23MB near EOL of Fedora 11. That was why it was important to make yum not re-fetch that repodata every time, and use a cached version of it. Yes, the keys to minimize bandwidth demands would be * to shink the size of the repodata/-files * to shink the size of changes to them. Besides obvious solutions, such as using a different compression (e.g. xz instead of bz2) and minimizing their contents, one could ship repodata/-files in chunks. What I mean: In theory, one could * update repodata/-files incrementally by shipping some kind of deltas. * split repodata/-files into several, e.g. sorted by some other criteria, i.e. to provide several sets of *-[primary,filelist,other] files. One package push, then would only affect a subset of the files, but not all. - This is very similar to what (IIRC) Seth had proposed (Split the repo into several repos, alphabetically), except that the split would happen inside of repodata and thus be transparent to users. I am not sure how difficult to implement this would be. Ralf -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Fri, December 4, 2009 9:20 pm, Ralf Corsepius wrote: On 12/03/2009 07:22 AM, Jesse Keating wrote: On Thu, 2009-12-03 at 06:24 +0100, Ralf Corsepius wrote: Yes, for people who are doing full featured networked installs w/ custom kickstart files. I've never met such a person. Really? I meet people who use kickstart all the time. May-be internal at RH? I do every install via kickstart - small company with 30-50 machines. Been doing fedora+everything+updates installs for many releases now. In fact - every upgrade is a fresh kickstart install + restore critical files from backup. -- Orion Poplawski Technical Manager 303-415-9701 x222 NWRA/CoRA DivisionFAX: 303-415-9702 3380 Mitchell Lane or...@cora.nwra.com Boulder, CO 80301 http://www.cora.nwra.com -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On 12/05/2009 06:22 AM, Orion Poplawski wrote: On Fri, December 4, 2009 9:20 pm, Ralf Corsepius wrote: On 12/03/2009 07:22 AM, Jesse Keating wrote: On Thu, 2009-12-03 at 06:24 +0100, Ralf Corsepius wrote: Yes, for people who are doing full featured networked installs w/ custom kickstart files. I've never met such a person. Really? I meet people who use kickstart all the time. May-be internal at RH? I do every install via kickstart - small company with 30-50 machines. Been doing fedora+everything+updates installs for many releases now. OK, then it's likely a full time professional admins vs. part time/side-job admins and home-users thing. Ralf -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Thu, 2009-12-03 at 00:32 -0500, Seth Vidal wrote: We wouldn't be talking about removing the original GA set - just adding updated pkgs into the path. So you'd still have the number of pkgs -just all in one repo, that you have to download all of the metadata for all of the more often, despite that 15K of them don't CHANGE. I don't think that was actually made clear in the initial proposal. I'd been assuming that the proposal was _exactly_ to remove the GA set. Usually, when a newer package shows up in any given repository, we don't keep the previous version of the package, do we? So I assumed the proposal was expecting that behaviour for the combined repository. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org http://www.happyassassin.net -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
* Adam Williamson [03/12/2009 10:10] : I don't think that was actually made clear in the initial proposal. I'd been assuming that the proposal was _exactly_ to remove the GA set. No can do. People who install from the netinst CD or do PXE installs without adding the updates repo during the installation would be unable to finish it. Emmanuel -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
Le Mer 2 décembre 2009 23:56, Matt Domsch a écrit : On Wed, Dec 02, 2009 at 07:35:03PM +0100, Nicolas Mailhot wrote: 3. replace static mirrors with proxy-ing of kojipkgs.fedoraproject.org (make sure it works with web infrastructure instead of fighting it) Sorry, I don't understand this. Can you elaborate? That would help me scope the impact to MirrorManager. Right now the same package moves from master URL to master URL as it is pushed from testing to updates to GA to whatever. That means the same package gets downloaded many times over because it changed URL (and browsers, proxies, etc understand new url = new file) My proposal is to never move a package, put it in a single unchanging place after a koji build, and just index it or not in the relevant repodata when it gets promoted or demoted. -- Nicolas Mailhot -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Thu, 3 Dec 2009, Adam Williamson wrote: On Thu, 2009-12-03 at 00:32 -0500, Seth Vidal wrote: We wouldn't be talking about removing the original GA set - just adding updated pkgs into the path. So you'd still have the number of pkgs -just all in one repo, that you have to download all of the metadata for all of the more often, despite that 15K of them don't CHANGE. I don't think that was actually made clear in the initial proposal. I'd been assuming that the proposal was _exactly_ to remove the GA set. Usually, when a newer package shows up in any given repository, we don't keep the previous version of the package, do we? So I assumed the proposal was expecting that behaviour for the combined repository. From a QA standpoint I'd think you'd want at least one known-installable set of pkgs. Since everything after the original GA set is a giant questionmark. Not to mention that removing all the old pkgs would more or less make deltarpms very difficult. -sv -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Thu, 2009-12-03 at 10:29 +0100, Nicolas Mailhot wrote: Le Mer 2 décembre 2009 23:56, Matt Domsch a écrit : On Wed, Dec 02, 2009 at 07:35:03PM +0100, Nicolas Mailhot wrote: 3. replace static mirrors with proxy-ing of kojipkgs.fedoraproject.org (make sure it works with web infrastructure instead of fighting it) Sorry, I don't understand this. Can you elaborate? That would help me scope the impact to MirrorManager. Right now the same package moves from master URL to master URL as it is pushed from testing to updates to GA to whatever. That means the same package gets downloaded many times over because it changed URL (and browsers, proxies, etc understand new url = new file) It only moves once, at least in the vast majority of cases. My proposal is to never move a package, put it in a single unchanging place after a koji build, and just index it or not in the relevant repodata when it gets promoted or demoted. One problem is that atm. when it moves from testing to updates, or rawhide to GA, it also gets resigned with a different key ... so the bits are not the same. Having the URL be the same will not be helpful until that changes. -- James Antill - ja...@fedoraproject.org http://yum.baseurl.org/wiki/releases http://yum.baseurl.org/wiki/whatsnew/3.2.25 http://yum.baseurl.org/wiki/YumMultipleMachineCaching -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
I think that idea maybe isn't benefit with repository. If updates repository is merged into Everything repository, Will metadata files become too large? I know that the size of metadatas on updates and everything are more than 30 megabytes. If these two repositories compose, We will need download more than 30MB files before updating. I believe that it will decrease the advantage of delrarpm update. It will increase more time to download the files. Is there any way to make createrepo generate delta metadata files? Just like delta rpm, we just need to get what meta files are changed, then yum generates the entire metadata files. We exactly not need and want to download the big files. Additionally, if an user is far away from repository server, though ISP provides quite a good 10Mbps connection, it is still quite slow to install some big applications, such OpenOffice.org, Eclipse, Netbeans, KOffice and some 3D games. Does yum allow to download more than one rpm files with only one thread per files after checking dependencies? Or using a yum plugin to replace yum default downloader by axel to download one rpm files with multi threads. I ever found out this kind of plugin, but I can assure it can run in Fedora 12. -- My Page: http://www.liangsuilong.info Fedora Project Contributor -- Packager Ambassador https://fedoraproject.org/wiki/User:Liangsuilong -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Dec 3, 2009, at 5:28, James Antill ja...@fedoraproject.org wrote: On Thu, 2009-12-03 at 10:29 +0100, Nicolas Mailhot wrote: Le Mer 2 décembre 2009 23:56, Matt Domsch a écrit : On Wed, Dec 02, 2009 at 07:35:03PM +0100, Nicolas Mailhot wrote: 3. replace static mirrors with proxy-ing of kojipkgs.fedoraproject.org (make sure it works with web infrastructure instead of fighting it) Sorry, I don't understand this. Can you elaborate? That would help me scope the impact to MirrorManager. Right now the same package moves from master URL to master URL as it is pushed from testing to updates to GA to whatever. That means the same package gets downloaded many times over because it changed URL (and browsers, proxies, etc understand new url = new file) It only moves once, at least in the vast majority of cases. My proposal is to never move a package, put it in a single unchanging place after a koji build, and just index it or not in the relevant repodata when it gets promoted or demoted. One problem is that atm. when it moves from testing to updates, or rawhide to GA, it also gets resigned with a different key ... so the bits are not the same. Having the URL be the same will not be helpful until that changes. -- That is no longer true. We used a single key for all of f11 and a single key for all of f12. -- Jes -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Thu, 2009-12-03 at 08:20 -0500, Seth Vidal wrote: We wouldn't be talking about removing the original GA set - just adding updated pkgs into the path. So you'd still have the number of pkgs -just all in one repo, that you have to download all of the metadata for all of the more often, despite that 15K of them don't CHANGE. I don't think that was actually made clear in the initial proposal. I'd been assuming that the proposal was _exactly_ to remove the GA set. Usually, when a newer package shows up in any given repository, we don't keep the previous version of the package, do we? So I assumed the proposal was expecting that behaviour for the combined repository. From a QA standpoint I'd think you'd want at least one known-installable set of pkgs. Since everything after the original GA set is a giant questionmark. Not to mention that removing all the old pkgs would more or less make deltarpms very difficult. I'm not saying I support the proposal, I don't, I think it's a waste of effort for no benefit. I was just clarifying the initial characterization. Actually I think the initial proposer _was_ expecting to remove initial packages when updates become available, because they keep saying stuff like 'only historians are interested in the GA packages'. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org http://www.happyassassin.net -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On 12/03/2009 12:24 AM, Ralf Corsepius wrote: On 12/02/2009 06:40 PM, Jesse Keating wrote: People doing network installs can either add the updates repo to their kickstart, or check the box in the anaconda UI, so that the updates repos are considered at install time. No download of duplicate data. Yes, for people who are doing full featured networked installs w/ custom kickstart files. I've never met such a person. Really? This very much seems to me like it's a vast majority of kickstarts that happen. -- Peter Power corrupts. Absolute power is kind of neat. -- John Lehman, Secretary of the Navy, 1981-1987 -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On 12/03/2009 08:20 AM, Seth Vidal wrote: On Thu, 3 Dec 2009, Adam Williamson wrote: On Thu, 2009-12-03 at 00:32 -0500, Seth Vidal wrote: We wouldn't be talking about removing the original GA set - just adding updated pkgs into the path. So you'd still have the number of pkgs -just all in one repo, that you have to download all of the metadata for all of the more often, despite that 15K of them don't CHANGE. I don't think that was actually made clear in the initial proposal. I'd been assuming that the proposal was _exactly_ to remove the GA set. Usually, when a newer package shows up in any given repository, we don't keep the previous version of the package, do we? So I assumed the proposal was expecting that behaviour for the combined repository. From a QA standpoint I'd think you'd want at least one known-installable set of pkgs. Since everything after the original GA set is a giant questionmark. Not to mention that removing all the old pkgs would more or less make deltarpms very difficult. Well, if we're talking about removing them from Fedora/ but leaving them in Everything/ (am I understanding the current form of the proposal correctly?), then it's not really significantly more difficult, but it is one more process that needs modification in order to enact such a plan. -- Peter If the facts don't fit the theory, change the facts. -- Einstein -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On 12/02/2009 09:12 PM, Kevin Kofler wrote: Seth Vidal wrote: If you're looking for perfect division, sure - but the reality is this: 19K items in a single dir and ext3 and nfs and many many other things crap themselves returning that list. If you make 36 subdirs (26+10) performance gets DRAMATICALLY better for producing the same list of files. The problem is, a few of those starting letters still correspond to A LOT of packages, e.g. p* matches perl-*, python-* and php-* and that's a lot of stuff (especially Perl and Python). And adding python3-* (and perl6-*, or are we going to use the rakudo-* namespace there?) won't make it any less. Even still, separating p out to a subdirectory means that subdirectory has an order of magnitude fewer files than the previous way. That's a really big difference. -- Peter If the facts don't fit the theory, change the facts. -- Einstein -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Proposed F13 feature: drop separate updates repository
The separate updates directory has been a pain for as long as I've been using RHL/Fedora Core/Fedora. It means you have two places to look when searching for packages manually, and twice as much to configure when you're configuring yum. It has never benefitted me, or anybody I know, but it has caught me out on any number of occasions. What's more, nobody really seems to know why it's like that: it seems it's always been that way, and nobody ever bother to fix it. So lets fix it. The package set at release time is only interesting to historians. If any of them are really that bothered, I'm sure somebody can come up with a yum module which finds the oldest available version of a package in a repo. Matt -- Matthew Booth, RHCA, RHCSS Red Hat Engineering, Virtualisation Team M: +44 (0)7977 267231 GPG ID: D33C3490 GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490 -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
Matthew Booth wrote: The separate updates directory has been a pain for as long as I've been using RHL/Fedora Core/Fedora. It means you have two places to look when searching for packages manually, and twice as much to configure when you're configuring yum. It has never benefitted me, or anybody I know, but it has caught me out on any number of occasions. What's more, nobody really seems to know why it's like that: it seems it's always been that way, and nobody ever bother to fix it. So lets fix it. The package set at release time is only interesting to historians. If any of them are really that bothered, I'm sure somebody can come up with a yum module which finds the oldest available version of a package in a repo. Matt Would not this also provide the minor added benefit that there could now be a drpm for the first update for a package? -J -- in your fear, seek only peace in your fear, seek only love -d. bowie -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On 12/02/2009 03:39 PM, Matthew Booth wrote: The separate updates directory has been a pain for as long as I've been using RHL/Fedora Core/Fedora. It means you have two places to look when searching for packages manually, and twice as much to configure when you're configuring yum. It has never benefitted me, or anybody I know, but it has caught me out on any number of occasions. What's more, nobody really seems to know why it's like that: it seems it's always been that way, and nobody ever bother to fix it. So lets fix it. The package set at release time is only interesting to historians. If any of them are really that bothered, I'm sure somebody can come up with a yum module which finds the oldest available version of a package in a repo. So your proposal is to drop updates, but to add updates to Everything? In this case, I am all for it. Ralf -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, Dec 02, 2009 at 09:00:53AM -0600, Jon Ciesla wrote: Matthew Booth wrote: The separate updates directory has been a pain for as long as I've been using RHL/Fedora Core/Fedora. It means you have two places to look when searching for packages manually, and twice as much to configure when you're configuring yum. It has never benefitted me, or anybody I know, but it has caught me out on any number of occasions. What's more, nobody really seems to know why it's like that: it seems it's always been that way, and nobody ever bother to fix it. So lets fix it. The package set at release time is only interesting to historians. If any of them are really that bothered, I'm sure somebody can come up with a yum module which finds the oldest available version of a package in a repo. Matt Would not this also provide the minor added benefit that there could now be a drpm for the first update for a package? We already have that if the update is done after GA. josh -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, Dec 02, 2009 at 02:39:30PM +, Matthew Booth wrote: The separate updates directory has been a pain for as long as I've been using RHL/Fedora Core/Fedora. It means you have two places to look when searching for packages manually, and twice as much to configure when you're configuring yum. It has never benefitted me, or anybody I know, but it has caught me out on any number of occasions. What's more, nobody really seems to know why it's like that: it seems it's always been that way, and nobody ever bother to fix it. So lets fix it. And how do you propose to do that? josh -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On 02/12/09 15:26, Josh Boyer wrote: On Wed, Dec 02, 2009 at 02:39:30PM +, Matthew Booth wrote: The separate updates directory has been a pain for as long as I've been using RHL/Fedora Core/Fedora. It means you have two places to look when searching for packages manually, and twice as much to configure when you're configuring yum. It has never benefitted me, or anybody I know, but it has caught me out on any number of occasions. What's more, nobody really seems to know why it's like that: it seems it's always been that way, and nobody ever bother to fix it. So lets fix it. And how do you propose to do that? Unfortunately I'm not intimately familiar with Fedora infrastructure under-the-hood. However, I would hope that, technically at least, it wouldn't be much more than a systematic removal of all updates repositories and redirecting files which would have gone into them into the main repositories instead. This stems from the observation that if you copied everything from the main repository into updates you would have created the repository which people unfamiliar with the history would expect. The infrastructure side of this, on the face of it, sounds very simple. More involved would be: 1. Updating all packages which expect 2 repos to only expect 1. 2. Communicating the change effectively. Matt -- Matthew Booth, RHCA, RHCSS Red Hat Engineering, Virtualisation Team M: +44 (0)7977 267231 GPG ID: D33C3490 GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490 -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
Josh Boyer wrote: On Wed, Dec 02, 2009 at 09:00:53AM -0600, Jon Ciesla wrote: Matthew Booth wrote: The separate updates directory has been a pain for as long as I've been using RHL/Fedora Core/Fedora. It means you have two places to look when searching for packages manually, and twice as much to configure when you're configuring yum. It has never benefitted me, or anybody I know, but it has caught me out on any number of occasions. What's more, nobody really seems to know why it's like that: it seems it's always been that way, and nobody ever bother to fix it. So lets fix it. The package set at release time is only interesting to historians. If any of them are really that bothered, I'm sure somebody can come up with a yum module which finds the oldest available version of a package in a repo. Matt Would not this also provide the minor added benefit that there could now be a drpm for the first update for a package? We already have that if the update is done after GA. josh If that's the case, then good. In that case, I see no huge benefit to leaving it or changing it. -- in your fear, seek only peace in your fear, seek only love -d. bowie -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, Dec 02, 2009 at 03:44:08PM +, Matthew Booth wrote: On 02/12/09 15:26, Josh Boyer wrote: On Wed, Dec 02, 2009 at 02:39:30PM +, Matthew Booth wrote: The separate updates directory has been a pain for as long as I've been using RHL/Fedora Core/Fedora. It means you have two places to look when searching for packages manually, and twice as much to configure when you're configuring yum. It has never benefitted me, or anybody I know, but it has caught me out on any number of occasions. What's more, nobody really seems to know why it's like that: it seems it's always been that way, and nobody ever bother to fix it. So lets fix it. And how do you propose to do that? Unfortunately I'm not intimately familiar with Fedora infrastructure under-the-hood. However, I would hope that, technically at least, it wouldn't be much more than a systematic removal of all updates repositories and redirecting files which would have gone into them into the main repositories instead. This stems from the observation that if you copied everything from the main repository into updates you would have created the repository which people unfamiliar with the history would expect. The infrastructure side of this, on the face of it, sounds very simple. What you describe is effectively how the development repository is built, so it's certainly a technical possibility. It does have implications though. Off the top of my head, I can think of: 1) Composing a new everything tree for updates would lead to larger compose times. That could possibly mean that getting updates out would take 1 day per 'push'. We've been trying to improve updates push times so it would be a bit detrimental to that goal. 2) There might be GPL compliance issues 3) You would still need an 'updates-testing' repository given that this is a supposedly stable release. So there is still going to be at least one additional repo regardless. However, other than 'browsing manually for packages', I'm not really sure what problem you are trying to solve by getting rid of the updates repository. It would seem like this has quite a bit of cost for relatively little to no real gain? josh -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
2009/12/2 Justin M. Forbes jmfor...@linuxtx.org The only downside to merging updates into the main repository... I would also assume that the repo data will need to be regenerated and often be much larger than the one that is for the updates only repository, so there will be acost to end users with this proposed change? -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, Dec 02, 2009 at 10:01:51AM -0600, Justin M. Forbes wrote: On Wed, Dec 02, 2009 at 02:39:30PM +, Matthew Booth wrote: The separate updates directory has been a pain for as long as I've been using RHL/Fedora Core/Fedora. It means you have two places to look when searching for packages manually, and twice as much to configure when you're configuring yum. It has never benefitted me, or anybody I know, but it has caught me out on any number of occasions. What's more, nobody really seems to know why it's like that: it seems it's always been that way, and nobody ever bother to fix it. The only downside to merging updates into the main repository is that network installs from the repository will no longer install the release they will install the updated release. QA that goes into that first impression is no longer there, and your installs are not repeatable because installing system A on one day and system B on another could end up with different versions of packages. Of course you can always install from the ISOs to avoid these problems. As things are, you have the choice of the release as it was, or the updated release at install time. I am not saying that this is a bad idea, in fact I rather like it, but there are things to consider. I think this is quite a compelling reason not to touch it. Much as we'd like the updates repository to always be perfectly stable, there are still plenty of occassions when we hit problems with updates. Being able to reliably install the release at all times is pretty critical saying we should install off ISO if we want a reliable install is not really an acceptable alternative since many people do all their installs straight off the network. Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :| -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, Dec 02, 2009 at 11:09:41AM -0500, Bill Nottingham wrote: Matthew Booth (mbo...@redhat.com) said: The separate updates directory has been a pain for as long as I've been using RHL/Fedora Core/Fedora. It means you have two places to look when searching for packages manually, and twice as much to configure when you're configuring yum. It has never benefitted me, or anybody I know, but it has caught me out on any number of occasions. What's more, nobody really seems to know why it's like that: it seems it's always been that way, and nobody ever bother to fix it. So lets fix it. The package set at release time is only interesting to historians. If any of them are really that bothered, I'm sure somebody can come up with a yum module which finds the oldest available version of a package in a repo. The separate Everything tree that does not get obsoleted is required in some form for GPL compliance, with respect to the ISO images that we ship. Any new solution would have to preserve this. Might there also be export compliance implications too? -- Paul W. Frieldshttp://paul.frields.org/ gpg fingerprint: 3DA6 A0AC 6D58 FEC4 0233 5906 ACDB C937 BD11 3717 http://redhat.com/ - - - - http://pfrields.fedorapeople.org/ irc.freenode.net: stickster @ #fedora-docs, #fedora-devel, #fredlug -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, 02 Dec 2009 17:27:17 +0100 Ralf Corsepius rc040...@freenet.de wrote: On 12/02/2009 05:09 PM, Bill Nottingham wrote: Matthew Booth (mbo...@redhat.com) said: The separate updates directory has been a pain for as long as I've been using RHL/Fedora Core/Fedora. It means you have two places to look when searching for packages manually, and twice as much to configure when you're configuring yum. It has never benefitted me, or anybody I know, but it has caught me out on any number of occasions. What's more, nobody really seems to know why it's like that: it seems it's always been that way, and nobody ever bother to fix it. So lets fix it. The package set at release time is only interesting to historians. If any of them are really that bothered, I'm sure somebody can come up with a yum module which finds the oldest available version of a package in a repo. The separate Everything tree that does not get obsoleted is required in some form for GPL compliance, with respect to the ISO images that we ship. Isn't this the Fedora repo? To my knowledge the Fedora repos corresponds 1:1 to the isos. To the DVD iso, yes. To any of the spins/desktop/live... nope. They use packages from Everything/ kevin signature.asc Description: PGP signature -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
Ralf Corsepius (rc040...@freenet.de) said: Does the FSF/GPL demand to keep a repo around for ISOs? A rolling Everything would not touch the ISOs. They would still be around. The LiveCD/spins satisfy their source requirements via the source repositories; they do not compose separate live source images. Bill -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, Dec 02, 2009 at 11:28:24AM -0500, Paul W. Frields wrote: The separate Everything tree that does not get obsoleted is required in some form for GPL compliance, with respect to the ISO images that we ship. Any new solution would have to preserve this. Might there also be export compliance implications too? What manner of export issues? This change doesn't, afaik, affect where export-restricted software appears in any way. --CJD -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, Dec 02, 2009 at 11:06:22AM -0500, Josh Boyer wrote: 1) Composing a new everything tree for updates would lead to larger compose times. That could possibly mean that getting updates out would take 1 day per 'push'. We've been trying to improve updates push times so it would be a bit detrimental to that goal. This might be a good opportunity to look at some way of pushing incremental sets of packages rather than re-building the entire yum repo each time. Mashing is not a cheap operation. 2) There might be GPL compliance issues 3) You would still need an 'updates-testing' repository given that this is a supposedly stable release. So there is still going to be at least one additional repo regardless. We have tags for updates that mark them as bugfix/feature/security. Maybe we could have one for testing which would keep yum from installing the package unless explicitly asked or specially configured. However, other than 'browsing manually for packages', I'm not really sure what problem you are trying to solve by getting rid of the updates repository. It would seem like this has quite a bit of cost for relatively little to no real gain? This is true. I don't know that manual package browsing is a use case we've ever been interested in, and if we're going to be interested in it there's probably a better way (search engine in the Fedora community portal comes to mind). josh You owe me $5. --CJD -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, 2 Dec 2009, Paul W. Frields wrote: On Wed, Dec 02, 2009 at 11:09:41AM -0500, Bill Nottingham wrote: we ship. Any new solution would have to preserve this. Might there also be export compliance implications too? A larger isssue is constantly having the repodata for the everything repo be: 1. growing 2. changing So now instead of having a 15K pkg repo that never changes you have an ever-growing repo that never shrinks in size that changes what? 3 times a week? -sv -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On 12/02/2009 06:01 PM, Casey Dahlin wrote: On Wed, Dec 02, 2009 at 11:06:22AM -0500, Josh Boyer wrote: However, other than 'browsing manually for packages', I'm not really sure what problem you are trying to solve by getting rid of the updates repository. It would seem like this has quite a bit of cost for relatively little to no real gain? This is true. It is not true. * It shifts costs from users to vendor and from mirrors to master. * It helps users who are using networked installs to spare bandwidth (avoids downloading obsolete packages from Everything/Fedora). Admitted, for most users, it would change almost nothing. Ralf -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On 02/12/09 16:01, Justin M. Forbes wrote: On Wed, Dec 02, 2009 at 02:39:30PM +, Matthew Booth wrote: The separate updates directory has been a pain for as long as I've been using RHL/Fedora Core/Fedora. It means you have two places to look when searching for packages manually, and twice as much to configure when you're configuring yum. It has never benefitted me, or anybody I know, but it has caught me out on any number of occasions. What's more, nobody really seems to know why it's like that: it seems it's always been that way, and nobody ever bother to fix it. The only downside to merging updates into the main repository is that network installs from the repository will no longer install the release they will install the updated release. QA that goes into that first impression is no longer there, and your installs are not repeatable because installing system A on one day and system B on another could end up with different versions of packages. Of course you can always install from the ISOs to avoid these problems. As things are, you have the choice of the release as it was, or the updated release at install time. I am not saying that this is a bad idea, in fact I rather like it, but there are things to consider. That's not a bug, it's a feature! Seriously, wasn't it a specific F10 or F11 feature that anaconda allowed the user to specify that updates be installed at installation time? Certainly I used to have Debianites crow at me that my installation was vulnerable until I updated it. Not only that, but I had to download updated packages twice. These days I always install with updates. I find the installation argument dubious at best. Still, we could rename the main repo the 'installation' repo, and nothing but anaconda would ever look at it. Everything else would look at the main repo, which would be the same as the current updates repo except with everything in it from the start. Matt -- Matthew Booth, RHCA, RHCSS Red Hat Engineering, Virtualisation Team M: +44 (0)7977 267231 GPG ID: D33C3490 GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490 -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On 02/12/09 16:09, Bill Nottingham wrote: Matthew Booth (mbo...@redhat.com) said: The separate updates directory has been a pain for as long as I've been using RHL/Fedora Core/Fedora. It means you have two places to look when searching for packages manually, and twice as much to configure when you're configuring yum. It has never benefitted me, or anybody I know, but it has caught me out on any number of occasions. What's more, nobody really seems to know why it's like that: it seems it's always been that way, and nobody ever bother to fix it. So lets fix it. The package set at release time is only interesting to historians. If any of them are really that bothered, I'm sure somebody can come up with a yum module which finds the oldest available version of a package in a repo. The separate Everything tree that does not get obsoleted is required in some form for GPL compliance, with respect to the ISO images that we ship. Any new solution would have to preserve this. This argument sounds weird to me. However, there no reason you can't keep around as many repositories as you like as long as the One True Repository exists and people can use it exclusively. Currently it doesn't exist. Matt -- Matthew Booth, RHCA, RHCSS Red Hat Engineering, Virtualisation Team M: +44 (0)7977 267231 GPG ID: D33C3490 GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490 -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
Once upon a time, Matthew Booth mbo...@redhat.com said: The separate updates directory has been a pain for as long as I've been using RHL/Fedora Core/Fedora. It means you have two places to look when searching for packages manually, and twice as much to configure when you're configuring yum. Are these really significant issues for a significant number of users? Not many people go looking manually for packages, since there are many tools to do it easier (yum, PK, repoquery, etc.). The repos are automatically configured with fedora-release; how often are you configuring yum? The only time I have to care about this is if I'm writing a kickstart file, but it is one extra line (and then I copy the same kickstart base over and over). -- Chris Adams cmad...@hiwaay.net Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble. -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, 2 Dec 2009, Nicolas Mailhot wrote: Since people are posting wishes, here is mine: 1. stop shuffling packages from directory to directory as they get promoted/demoted from release to release we sort of do this now with hardlinks - the problem is when we have to resign the pkgs. 2. have a single authoritative URL for each package packagedb... 3. replace static mirrors with proxy-ing of kojipkgs.fedoraproject.org (make sure it works with web infrastructure instead of fighting it) I don't think that would work fine with a lot of our mirrors. 4. generate indexes of the kojipkgs.fedoraproject.org tree that represent Fedora X, Fedora X + updates, Fedora X + testing, etc. this is intriguing but expensive on kojipkgs. -sv -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, 2009-12-02 at 15:52 -0500, Seth Vidal wrote: Isn't this, eventually, what the packagedb is supposed to be able to do? I gather it's a ls in a directory kind of thing, not an interface to one tool or another kind of thing. But I could be wrong. -- Jesse Keating Fedora -- Freedom² is a feature! identi.ca: http://identi.ca/jkeating signature.asc Description: This is a digitally signed message part -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
Dne 2.12.2009 17:06, Josh Boyer napsal(a): However, other than 'browsing manually for packages', I'm not really sure what problem you are trying to solve by getting rid of the updates repository. It would seem like this has quite a bit of cost for relatively little to no real gain? I am usually too lazy, so I use yumdownloader anyway. So, I really don't see any real use case covered. Matěj -- http://www.ceplovi.cz/matej/, Jabber: mceplatceplovi.cz GPG Finger: 89EF 4BC6 288A BF43 1BAB 25C3 E09F EF25 D964 84AC If Patrick Henry thought that taxation without representation was bad, he should see how bad it is with representation. -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
(on my on tangent...) On 12/02/2009 12:48 PM, Jesse Keating wrote: I hypothesize that we could place all rpms for a given release in a single directory (seth will hate this as he wants to split them up based on first letter of their name for better filesystem performance), Ugh, first letter isn't really a great plan anyway. First (few) letters of a hash of the filename is much better, but obviously hurts browsability. Next best is probably something like how a dead-tree dictionary index works; list everything, split the list up by starting letters evenly, so the directories (given a really unlikely hypothetical package set) are 0/ # contains packages named 0 through 3* 4/ # 4 through 9* a/ # a through ay* az/ # az through bw* bx/ # bx through cz* da/ # da through whatever's next ... so that every directory has about the same number of things. -- Peter I'd like to start a religion. That's where the money is. -- L. Ron Hubbard to Lloyd Eshbach, in 1949; quoted by Eshbach in _Over My Shoulder_. -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On 12/02/2009 05:03 PM, Jesse Keating wrote: On Wed, 2009-12-02 at 15:52 -0500, Seth Vidal wrote: Isn't this, eventually, what the packagedb is supposed to be able to do? I gather it's a ls in a directory kind of thing, not an interface to one tool or another kind of thing. But I could be wrong. Sure, but the better optimization for this is don't do it. -- Peter I'd like to start a religion. That's where the money is. -- L. Ron Hubbard to Lloyd Eshbach, in 1949; quoted by Eshbach in _Over My Shoulder_. -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On 12/02/2009 03:53 PM, Seth Vidal wrote: On Wed, 2 Dec 2009, Nicolas Mailhot wrote: 3. replace static mirrors with proxy-ing of kojipkgs.fedoraproject.org (make sure it works with web infrastructure instead of fighting it) I don't think that would work fine with a lot of our mirrors. I think it probably /could/ if we put some (relatively major) work in on the server side to make it look like the directories it isn't. Not that I'm endorsing such a plan. -- Peter I'd like to start a religion. That's where the money is. -- L. Ron Hubbard to Lloyd Eshbach, in 1949; quoted by Eshbach in _Over My Shoulder_. -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, Dec 02, 2009 at 02:03:51PM -0800, Jesse Keating wrote: On Wed, 2009-12-02 at 15:52 -0500, Seth Vidal wrote: Isn't this, eventually, what the packagedb is supposed to be able to do? I gather it's a ls in a directory kind of thing, not an interface to one tool or another kind of thing. But I could be wrong. We're at a point where 'ls in a directory' is becoming difficult even; you can't glob 15k package names in a shell. find . is your friend. -- Matt Domsch Technology Strategist, Dell Office of the CTO linux.dell.com www.dell.com/linux -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, Dec 02, 2009 at 07:35:03PM +0100, Nicolas Mailhot wrote: 3. replace static mirrors with proxy-ing of kojipkgs.fedoraproject.org (make sure it works with web infrastructure instead of fighting it) Sorry, I don't understand this. Can you elaborate? That would help me scope the impact to MirrorManager. Thanks, Matt -- Matt Domsch Technology Strategist, Dell Office of the CTO linux.dell.com www.dell.com/linux -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, 2 Dec 2009, Peter Jones wrote: (on my on tangent...) On 12/02/2009 12:48 PM, Jesse Keating wrote: I hypothesize that we could place all rpms for a given release in a single directory (seth will hate this as he wants to split them up based on first letter of their name for better filesystem performance), Ugh, first letter isn't really a great plan anyway. First (few) letters of a hash of the filename is much better, but obviously hurts browsability. Next best is probably something like how a dead-tree dictionary index works; list everything, split the list up by starting letters evenly, so the directories (given a really unlikely hypothetical package set) are 0/ # contains packages named 0 through 3* 4/ # 4 through 9* a/ # a through ay* az/ # az through bw* bx/ # bx through cz* da/ # da through whatever's next ... so that every directory has about the same number of things. If you're looking for perfect division, sure - but the reality is this: 19K items in a single dir and ext3 and nfs and many many other things crap themselves returning that list. If you make 36 subdirs (26+10) performance gets DRAMATICALLY better for producing the same list of files. I tested it on our backend to be sure. getting the complete pkglist goes from taking 5 minutes to take 30s. yes, I said 5 minutes. -sv -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, Dec 02, 2009 at 11:09:41AM -0500, Bill Nottingham wrote: The separate Everything tree that does not get obsoleted is required in some form for GPL compliance, with respect to the ISO images that we ship. Any new solution would have to preserve this. ? We provide Fedora-*-source-disc*.iso and Fedora-*-source-DVD.iso images on the mirrors already. -- Matt Domsch Technology Strategist, Dell Office of the CTO linux.dell.com www.dell.com/linux -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, 2 Dec 2009, Peter Jones wrote: On 12/02/2009 03:53 PM, Seth Vidal wrote: On Wed, 2 Dec 2009, Nicolas Mailhot wrote: 3. replace static mirrors with proxy-ing of kojipkgs.fedoraproject.org (make sure it works with web infrastructure instead of fighting it) I don't think that would work fine with a lot of our mirrors. I think it probably /could/ if we put some (relatively major) work in on the server side to make it look like the directories it isn't. Not that I'm endorsing such a plan. we'd have to make all our mirrors use that. not gonna fly. -sv -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On 12/02/2009 05:58 PM, Seth Vidal wrote: On Wed, 2 Dec 2009, Peter Jones wrote: On 12/02/2009 03:53 PM, Seth Vidal wrote: On Wed, 2 Dec 2009, Nicolas Mailhot wrote: 3. replace static mirrors with proxy-ing of kojipkgs.fedoraproject.org (make sure it works with web infrastructure instead of fighting it) I don't think that would work fine with a lot of our mirrors. I think it probably /could/ if we put some (relatively major) work in on the server side to make it look like the directories it isn't. Not that I'm endorsing such a plan. we'd have to make all our mirrors use that. not gonna fly. Well, you just wouldn't get the benefits of this if you're using a mirror. Which, yes, is pretty lame. -- Peter I'd like to start a religion. That's where the money is. -- L. Ron Hubbard to Lloyd Eshbach, in 1949; quoted by Eshbach in _Over My Shoulder_. -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On 12/02/2009 05:58 PM, Seth Vidal wrote: On Wed, 2 Dec 2009, Peter Jones wrote: (on my on tangent...) On 12/02/2009 12:48 PM, Jesse Keating wrote: I hypothesize that we could place all rpms for a given release in a single directory (seth will hate this as he wants to split them up based on first letter of their name for better filesystem performance), Ugh, first letter isn't really a great plan anyway. First (few) letters of a hash of the filename is much better, but obviously hurts browsability. Next best is probably something like how a dead-tree dictionary index works; list everything, split the list up by starting letters evenly, so the directories (given a really unlikely hypothetical package set) are 0/# contains packages named 0 through 3* 4/# 4 through 9* a/# a through ay* az/# az through bw* bx/# bx through cz* da/# da through whatever's next ... so that every directory has about the same number of things. If you're looking for perfect division, sure - but the reality is this: 19K items in a single dir and ext3 and nfs and many many other things crap themselves returning that list. If you make 36 subdirs (26+10) performance gets DRAMATICALLY better for producing the same list of files. I tested it on our backend to be sure. getting the complete pkglist goes from taking 5 minutes to take 30s. yes, I said 5 minutes. Yeah, I buy that. -- Peter I'd like to start a religion. That's where the money is. -- L. Ron Hubbard to Lloyd Eshbach, in 1949; quoted by Eshbach in _Over My Shoulder_. -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, 2 Dec 2009, Peter Jones wrote: On 12/02/2009 05:58 PM, Seth Vidal wrote: On Wed, 2 Dec 2009, Peter Jones wrote: On 12/02/2009 03:53 PM, Seth Vidal wrote: On Wed, 2 Dec 2009, Nicolas Mailhot wrote: 3. replace static mirrors with proxy-ing of kojipkgs.fedoraproject.org (make sure it works with web infrastructure instead of fighting it) I don't think that would work fine with a lot of our mirrors. I think it probably /could/ if we put some (relatively major) work in on the server side to make it look like the directories it isn't. Not that I'm endorsing such a plan. we'd have to make all our mirrors use that. not gonna fly. Well, you just wouldn't get the benefits of this if you're using a mirror. Which, yes, is pretty lame. we won't get the benefits in fedora then. Our mirror infrastructure is a HUGE reason why we can do what we do. If we can't farm things out to our mirrors then we might as well go home b/c our users will NEVER be able to get our bits. -sv -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, 2009-12-02 at 17:46 -0500, Peter Jones wrote: (on my on tangent...) On 12/02/2009 12:48 PM, Jesse Keating wrote: I hypothesize that we could place all rpms for a given release in a single directory (seth will hate this as he wants to split them up based on first letter of their name for better filesystem performance), Ugh, first letter isn't really a great plan anyway. It's not great but it's better than the current all in one dir. and easy to do, and I haven't heard of any downsides (that aren't applicable to all in one dir.). First (few) letters of a hash of the filename is much better, but obviously hurts browsability. Right, which is important enough to not do that IMO. I'm sure Jesse wouldn't like finding packages to mean using find. Next best is probably something like how a dead-tree dictionary index works; list everything, split the list up by starting letters evenly, so the directories (given a really unlikely hypothetical package set) are 0/# contains packages named 0 through 3* 4/# 4 through 9* a/# a through ay* az/ # az through bw* bx/ # bx through cz* da/ # da through whatever's next ... so that every directory has about the same number of things. This should be fairly easy to code, but has a big downside: Packages will move directories. 1. This will upset yum (old repo. MD == no packages download). 2. This might upset mirrors. 3. This will probably upset users: i. URLs will only be safe until the next mash, they won't even be able to bookmark prefix/y if they just want to see yum each time. ii. Users have to look/parse the directory layout each time they visit to see what blah is. -- James Antill - ja...@fedoraproject.org http://yum.baseurl.org/wiki/releases http://yum.baseurl.org/wiki/whatsnew/3.2.25 http://yum.baseurl.org/wiki/YumMultipleMachineCaching -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On 12/02/2009 06:05 PM, James Antill wrote: On Wed, 2009-12-02 at 17:46 -0500, Peter Jones wrote: so that every directory has about the same number of things. This should be fairly easy to code, but has a big downside: Packages will move directories. 1. This will upset yum (old repo. MD == no packages download). 2. This might upset mirrors. 3. This will probably upset users: i. URLs will only be safe until the next mash, they won't even be able to bookmark prefix/y if they just want to see yum each time. ii. Users have to look/parse the directory layout each time they visit to see what blah is. Yeah. Seth makes a good point - first letter is such vast improvement that we probably don't need to worry about doing better - at least for now. -- Peter I'd like to start a religion. That's where the money is. -- L. Ron Hubbard to Lloyd Eshbach, in 1949; quoted by Eshbach in _Over My Shoulder_. -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, 2009-12-02 at 16:58 -0600, Matt Domsch wrote: On Wed, Dec 02, 2009 at 11:09:41AM -0500, Bill Nottingham wrote: The separate Everything tree that does not get obsoleted is required in some form for GPL compliance, with respect to the ISO images that we ship. Any new solution would have to preserve this. ? We provide Fedora-*-source-disc*.iso and Fedora-*-source-DVD.iso images on the mirrors already. As stated elsewhere, that doesn't carry the packages found on all the live images we produce. -- Jesse Keating Fedora -- Freedom² is a feature! identi.ca: http://identi.ca/jkeating signature.asc Description: This is a digitally signed message part -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, Dec 02, 2009 at 03:08:06PM -0800, Jesse Keating wrote: On Wed, 2009-12-02 at 16:58 -0600, Matt Domsch wrote: On Wed, Dec 02, 2009 at 11:09:41AM -0500, Bill Nottingham wrote: The separate Everything tree that does not get obsoleted is required in some form for GPL compliance, with respect to the ISO images that we ship. Any new solution would have to preserve this. ? We provide Fedora-*-source-disc*.iso and Fedora-*-source-DVD.iso images on the mirrors already. As stated elsewhere, that doesn't carry the packages found on all the live images we produce. All the more reason to put more effort into http://git.fedorahosted.org/git/correspondingsource.git -- Matt Domsch Technology Strategist, Dell Office of the CTO linux.dell.com www.dell.com/linux -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, Dec 02, 2009 at 17:58:24 -0500, Seth Vidal skvi...@fedoraproject.org wrote: I tested it on our backend to be sure. getting the complete pkglist goes from taking 5 minutes to take 30s. yes, I said 5 minutes. Have you tried any of the tunning knobs to have the directory cache be alotted more space or given higher priority? For example: vfs_cache_pressure -- Controls the tendency of the kernel to reclaim the memory which is used for caching of directory and inode objects. At the default value of vfs_cache_pressure=100 the kernel will attempt to reclaim dentries and inodes at a fair rate with respect to pagecache and swapcache reclaim. Decreasing vfs_cache_pressure causes the kernel to prefer to retain dentry and inode caches. Increasing vfs_cache_pressure beyond 100 causes the kernel to prefer to reclaim dentries and inodes. -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, 2 Dec 2009, Bruno Wolff III wrote: On Wed, Dec 02, 2009 at 17:58:24 -0500, Seth Vidal skvi...@fedoraproject.org wrote: I tested it on our backend to be sure. getting the complete pkglist goes from taking 5 minutes to take 30s. yes, I said 5 minutes. Have you tried any of the tunning knobs to have the directory cache be alotted more space or given higher priority? For example: vfs_cache_pressure -- Controls the tendency of the kernel to reclaim the memory which is used for caching of directory and inode objects. At the default value of vfs_cache_pressure=100 the kernel will attempt to reclaim dentries and inodes at a fair rate with respect to pagecache and swapcache reclaim. Decreasing vfs_cache_pressure causes the kernel to prefer to retain dentry and inode caches. Increasing vfs_cache_pressure beyond 100 causes the kernel to prefer to reclaim dentries and inodes. Not tried that - worth giving it a shot - but it still won't help the 'holy crap there are 15K items on this webpage and it won't render' problem. -sv -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
Seth Vidal wrote: If you're looking for perfect division, sure - but the reality is this: 19K items in a single dir and ext3 and nfs and many many other things crap themselves returning that list. If you make 36 subdirs (26+10) performance gets DRAMATICALLY better for producing the same list of files. The problem is, a few of those starting letters still correspond to A LOT of packages, e.g. p* matches perl-*, python-* and php-* and that's a lot of stuff (especially Perl and Python). And adding python3-* (and perl6-*, or are we going to use the rakudo-* namespace there?) won't make it any less. Kevin Kofler -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Thu, 3 Dec 2009, Kevin Kofler wrote: Seth Vidal wrote: If you're looking for perfect division, sure - but the reality is this: 19K items in a single dir and ext3 and nfs and many many other things crap themselves returning that list. If you make 36 subdirs (26+10) performance gets DRAMATICALLY better for producing the same list of files. The problem is, a few of those starting letters still correspond to A LOT of packages, e.g. p* matches perl-*, python-* and php-* and that's a lot of stuff (especially Perl and Python). And adding python3-* (and perl6-*, or are we going to use the rakudo-* namespace there?) won't make it any less. And yet, when tested, just making those 36 subdirs made a HUGE difference. What I've found is getting any one dir down below 3K entries makes things faster, by a lot. -sv -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, Dec 02, 2009 at 12:01:49PM -0500, Casey Dahlin wrote: You owe me $5. It hasn't been a week and you haven't sent me your address. I did notice though, so keep up the good work ;) josh -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Wed, 2 Dec 2009, Bruno Wolff III wrote: On Wed, Dec 02, 2009 at 17:58:24 -0500, Seth Vidal skvi...@fedoraproject.org wrote: I tested it on our backend to be sure. getting the complete pkglist goes from taking 5 minutes to take 30s. yes, I said 5 minutes. Have you tried any of the tunning knobs to have the directory cache be alotted more space or given higher priority? For example: vfs_cache_pressure -- Controls the tendency of the kernel to reclaim the memory which is used for caching of directory and inode objects. At the default value of vfs_cache_pressure=100 the kernel will attempt to reclaim dentries and inodes at a fair rate with respect to pagecache and swapcache reclaim. Decreasing vfs_cache_pressure causes the kernel to prefer to retain dentry and inode caches. Increasing vfs_cache_pressure beyond 100 causes the kernel to prefer to reclaim dentries and inodes. We have experimented with this for the purposes of rsync on our mirrors but haven't experimented for this specific issue. -Mike -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On 12/02/2009 06:40 PM, Jesse Keating wrote: On Wed, 2009-12-02 at 18:09 +0100, Ralf Corsepius wrote: * It shifts costs from users to vendor and from mirrors to master. * It helps users who are using networked installs to spare bandwidth (avoids downloading obsolete packages from Everything/Fedora). Admitted, for most users, it would change almost nothing. People doing network installs can either add the updates repo to their kickstart, or check the box in the anaconda UI, so that the updates repos are considered at install time. No download of duplicate data. Yes, for people who are doing full featured networked installs w/ custom kickstart files. I've never met such a person. In fact, having separate repos would likely cost less bandwidth. If we only had one combined repo, there would be many duplicate packages, Where? Unlike now, where you have each package twice (in Everything and updates), you would have each package only once: In Everything. It would help people like me, who are locally reusing downloaded packages in a networked environment, instead of letting each machine accesses the original repos directly. especially if we went the route of having updates-testing mixed in and only marked by some update tag. Updates-testing is an add-on repo to Everything+updates. A merged new Everything would not change anything wrt. updates-testing. The only difference would be packages being pushed to Everything instead of updates. We'd have to keep sets of what's in updates-testing, updates, and the GA release set, and all of that would be in one repodata set. Everybody doing a network install, whether they wanted updates, updates-testing, or not would have to download and consume that larger repodata, introducing a higher cost for them. Your concern is the bigger repodata? Reality check: # ls -h -s releases/11/Everything/x86_64/os/repodata/*.sqlite.bz2 updates/11/x86_64/repodata/*.sqlite.bz2 16M releases/11/Everything/x86_64/os/repodata/0301ed1cbf01c00cdf9c42b71df6c74947d9c76a3c9767e8f85a99ef6fdccb86-filelists.sqlite.bz2 11M releases/11/Everything/x86_64/os/repodata/6a8bfab8ebcbc79f9827f5b16bc1bd1573c068f141bf47c6f216e72dd8b60ff0-primary.sqlite.bz2 5.8M releases/11/Everything/x86_64/os/repodata/d3405c970a0c27e9dc3fd3ed179af33fee9a72bf704f45f1ed96a9063b40d7c7-other.sqlite.bz2 9.0M updates/11/x86_64/repodata/3a725e07adff9d7e56e7fd8bd3091f38107723987b3b614220df72494dc6814d-filelists.sqlite.bz2 6.0M updates/11/x86_64/repodata/54ca116c2a00e87eaca6491adb9e077f4d3f0d5b20e7cbab984774aefb042d0f-primary.sqlite.bz2 3.2M updates/11/x86_64/repodata/646eb24cfe113de569853a4e05f55773b3ad9531dee646e7d843b8dc4a849780-other.sqlite.bz2 = An estimate for the increase in downloaded files's sizes you are talking about is ca. factor 2, from 18.2M (current updates) to 32.8M+ (current Everything+newly introduced packages) Whether this increase in download-size is significant is up to the beholder. For me, it gets lost in the noise of accessing a good or a bad connection to a mirror and in the time required to download packages from mirrors. On the other hand, the content (the packages referenced inside) of updates overlaps with packages in Everything = The number of packages to be considered in dep-computation should become much (?) smaller. However, I am not knowledgable enough with yum's internals to estimate the impact this would have on turnaround-times, memory and diskspace requirements this would have on yum. Ralf -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On 12/02/2009 07:09 PM, Seth Vidal wrote: the merger of repos is already happening at the yum layer. On the client's side - With a combined Everything+updates, this would happen on the server side. It's one of the aspects which made me said a combined Everything+updates shifts costs from client to server. Ralf -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On Thu, 3 Dec 2009, Ralf Corsepius wrote: On 12/02/2009 07:09 PM, Seth Vidal wrote: the merger of repos is already happening at the yum layer. On the client's side - With a combined Everything+updates, this would happen on the server side. It's one of the aspects which made me said a combined Everything+updates shifts costs from client to server. except they wouldn't be merged on the server side -it would just be additive. We wouldn't be talking about removing the original GA set - just adding updated pkgs into the path. So you'd still have the number of pkgs -just all in one repo, that you have to download all of the metadata for all of the more often, despite that 15K of them don't CHANGE. this is why it is less good for our users. -sv -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
On 12/03/2009 06:32 AM, Seth Vidal wrote: On Thu, 3 Dec 2009, Ralf Corsepius wrote: On 12/02/2009 07:09 PM, Seth Vidal wrote: the merger of repos is already happening at the yum layer. On the client's side - With a combined Everything+updates, this would happen on the server side. It's one of the aspects which made me said a combined Everything+updates shifts costs from client to server. except they wouldn't be merged on the server side -it would just be additive. We wouldn't be talking about removing the original GA set - just adding updated pkgs into the path. Woa!!! With all due respect, but this would seem an stupid and silly plan to me. So you'd still have the number of pkgs -just all in one repo, that you have to download all of the metadata for all of the more often, despite that 15K of them don't CHANGE. this is why it is less good for our users. Yes - cf. above. I have nothing to add. Ralf -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Proposed F13 feature: drop separate updates repository
Ralf Corsepius wrote: Your concern is the bigger repodata? My download of repodata towards the end of a release, or from rawhide, is usually bigger than my download of packages. So yes, this would make a difference. On the otherhand, there probably could be a repodiff that would alleviate a lot of this problem. -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
