Re: yum-presto not on by default
Eric Springer wrote: Would it be possible to have a diff on the filelist db? It seems like a very large download for something that would change very little. Also the rawhide db itself. A guaranteed 8-12 megabyte download usually swamps out any saving from presto (barring changes to eclipse, kernel and openoffice). -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
Also the rawhide db itself. A guaranteed 8-12 megabyte download usually swamps out any saving from presto (barring changes to eclipse, kernel and openoffice). Why aren't we rsync'ing that 12MB db, instead of re-downloading ? Wasn't there some web friendly rsync fork -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
Ahmed Kamal (email.ahmedka...@googlemail.com) said: Also the rawhide db itself. A guaranteed 8-12 megabyte download usually swamps out any saving from presto (barring changes to eclipse, kernel and openoffice). Why aren't we rsync'ing that 12MB db, instead of re-downloading ? Wasn't there some web friendly rsync fork rsync over http/ftp? We don't really have that. Bill -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On Mon, 2009-10-05 at 20:32 +0200, Ahmed Kamal wrote: Also the rawhide db itself. A guaranteed 8-12 megabyte download usually swamps out any saving from presto (barring changes to eclipse, kernel and openoffice). Why aren't we rsync'ing that 12MB db, instead of re-downloading ? Wasn't there some web friendly rsync fork zsync, see the recent thread for why it isn't in Fedora. And I doubt they'd be much savings, even if it worked (and you'd have to be clever due to the names changing). Adding delta MD support would be easier, and by easier I mean significant amounts of code and significant problems on the f-i side. -- James Antill ja...@fedoraproject.org Fedora -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On Wed, 2009-09-23 at 22:05 -0700, Bill Nottingham wrote: So... just set the xz compression level to 2, let it be that way for future builds, and go about our business? Quick follow-up on this issue: I heard that this part at least has been done, and it certainly seems to have done the trick. Delta rebuild speeds have increased approx ten-fold here, and are now faster than my (pretty quick) download speeds. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org http://www.happyassassin.net -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On Mon, Oct 5, 2009 at 5:05 AM, Adam Williamson awill...@redhat.com wrote: Quick follow-up on this issue: I heard that this part at least has been done, and it certainly seems to have done the trick. Delta rebuild speeds have increased approx ten-fold here, and are now faster than my (pretty quick) download speeds. Is it possible to do the rebuilds in parallel? I noticed that only one of my four cores was used. And that on smolt 63% of people have a dual-core or greater, so it could lead to massive speed-ups as well. Although, personally I would be happy even if the rebuild was a tenth the current speed -- as for me the priority is reducing the load on the network link. Which brings me to the next point: Would it be possible to have a diff on the filelist db? It seems like a very large download for something that would change very little. -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On Sun, Oct 4, 2009 at 10:42 PM, Eric Springer erik...@gmail.com wrote: On Mon, Oct 5, 2009 at 5:05 AM, Adam Williamson awill...@redhat.com wrote: Quick follow-up on this issue: I heard that this part at least has been done, and it certainly seems to have done the trick. Delta rebuild speeds have increased approx ten-fold here, and are now faster than my (pretty quick) download speeds. Is it possible to do the rebuilds in parallel? I noticed that only one of my four cores was used. And that on smolt 63% of people have a dual-core or greater, so it could lead to massive speed-ups as well. xz is not threaded yet, I did some benchmark on my core i7 systems: http://193.200.113.196/apache2-default/res pbzip2 and pigz are threaded (compare them with bzip2 and gzip in the output to see what kind of speedup is possible). -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
Is it possible to do the rebuilds in parallel? I noticed that only one of my four cores was used. And that on smolt 63% of people have a dual-core or greater, so it could lead to massive speed-ups as well. The speedup would be noticeably less than the number of cores. xz uses a history+search table that is significantly larger than dcache. There is competition for memory bandwidth, not just CPU+cache cycles. Would it be possible to have a diff on the filelist db? It seems like a very large download for something that would change very little. Agreed, this would be a 99% savings. -- -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On 10/04/2009 04:45 PM, drago01 wrote: On Sun, Oct 4, 2009 at 10:42 PM, Eric Springererik...@gmail.com wrote: On Mon, Oct 5, 2009 at 5:05 AM, Adam Williamsonawill...@redhat.com wrote: Quick follow-up on this issue: I heard that this part at least has been done, and it certainly seems to have done the trick. Delta rebuild speeds have increased approx ten-fold here, and are now faster than my (pretty quick) download speeds. Is it possible to do the rebuilds in parallel? I noticed that only one of my four cores was used. And that on smolt 63% of people have a dual-core or greater, so it could lead to massive speed-ups as well. xz is not threaded yet, I did some benchmark on my core i7 systems: Although that isn't a problem if multiple packages are reconstructed simultaneously. Warren -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 John Reiser wrote: Would it be possible to have a diff on the filelist db? It seems like a very large download for something that would change very little. Agreed, this would be a 99% savings. Relatedly, I am amused every time the presto metadata is larger than the resultant package itself (usually 700k or less). Maybe there could/should be some heuristic for this? - --Ben -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) iQIcBAEBCAAGBQJKyTjqAAoJEKaxavVX4C1X7SAQAJTHRkYs0/GIQBuCcR5AiTmr UitxucY0VVVWHh7aw/4lWnfH0hhIsEEP95f9TbY3I0Vf6pL0N67xm0SO54/mbXxk FeAyJFdDyyAZziOrdmjw93Y3lLs8n65j1MX5IrwOrhKAa1NOOORS/IXdVGkVWtrj xIWB8MZ2NEr5rlBv1vzuBV33hzEq1pfG9BRKsCIpRUVzPbiq/ealoqXxPwg/XlF9 pLV7rwRU606LfTQsDUl11P01Zdu651hRPikfasUeJzJY8QcrZ0fLCkmDmZcUXpoa UIWfa9yhnQ5QHwZTyQXtM36vOE5pG/90/NgZlzii+dC/ZQuX+O8hbEEbhs5tAs3I qGKXfBUYI+EYuI30CdUiA5CIyHTyYPCRqKjVtZ31RY8fxgMO2Sa3W/67vRy93v8M 5boUOWZya4SxY2UoktARs/3tVP2N3QMHAqdWcLMtjsiuUPpQCUq0XFWr33QRvB03 nsYD6kkywqk1hp3CmMBEWy2RmkeuwXAQKuPpePvgxHq9lUoQ7+MQeP1EqLXukZqe Q5SlUgrBsM6NLf71Anb1OktKfl3xxcqU5LJaTpYZ/VjqtTIHSUvmUrZO5OcWy+pH m/Y9tg9e4WTrwrZYxKAo5v16mfIO8QzvazpvF4wSPXLHxTUjQoGw2XFJD/K827BU 4kamNtc34AWmoGskoOzk =cpwh -END PGP SIGNATURE- -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On Sat, 2009-09-26 at 11:17 +0200, nicolas.mail...@laposte.net wrote: On 9/25/2009 1:00 PM, Adam Williamson wrote: So how do you get those 'others' to read the instructions? Linux used to basically require that you do that. Things had to be found in the system and then configured by hand. In the rush to make things auto-magic, which is not a bad thing IMO, when something does not 'just work' users are lost. They seem to have forgotten how to read. Or perhaps comprehend what they read. It's not like that. It's more like, the complexity of a Fedora system went I did not write the stuff you quoted. Please be careful to attribute accurately. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org http://www.happyassassin.net -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
Seth Vidal wrote: Seriously? We have lots of documentation that is run this one command. We're a bleeding edge distro - that's our VAUNTED mission - shouldn't we be able to assume that users of a BLEEDING EDGE, LATEST PACKAGE distro be able to run a single command? I think if you assume too little of your users you end up with bad users. +1 I think this is also a big problem with how the download page is being simplified. Call me elitist, but frankly I think we DO NOT WANT that kind of users. They have little to no chance of ever becoming contributors and they'll strain our support channels (IRC, fedora-list, they even flood this list with off- topic user questions!) with their dumb questions and take away valuable contributor time. Kevin Kofler -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
Bill Nottingham wrote: So... just set the xz compression level to 2, let it be that way for future builds, and go about our business? We still need to fix the noarch bug (i.e. xz compression being endianness- specific and breaking deltas of noarch packages). Kevin Kofler -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
Alex Hudson wrote: I'd be interested in trying to fix xz (I disagree entirely that this is an xz bug; it's a bug in the assumptions of those whose used it for the wrong purpose), but presumably it means rebuilding packages built on certain arches anyway - and if that's not easy to determine, does that imply a rebuild anyway? Basically, the safest solution would be to fix xz, then rebuild all noarch packages. At least those noarch packages which happened to be built on a PPC builder will need rebuilding after xz gets fixed. Please note that this probably also affects noarch subpackages. Kevin Kofler -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
Rahul Sundaram wrote: Since it has been added the gnome-desktop (by me) and kde-desktop (by the KDE SIG) package groups in Fedora 12, majority of users will get it by default. But if the consensus is that it shouldn't be the default, we (KDE SIG) will follow that. Kevin Kofler -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On Fri, 25 Sep 2009, Alexander Boström wrote: tor 2009-09-24 klockan 16:31 -0400 skrev Seth Vidal: We're a bleeding edge distro - that's our VAUNTED mission - shouldn't we be able to assume that users of a BLEEDING EDGE, LATEST PACKAGE distro be able to run a single command? Though Fedora currently might not currently be an optimal just works experience, I think that should be the design goal. I want ordinary people to be able to use free software. (And not just use.) Wouldn't you rather lead the other distributions who already have that project goal? Let us lead, invent, whatever you want to call it. Let them refine and present to the 'ordinary people'. Let us play to our strengths, let them play to theirs. We win, they win, users win. IMHO RHL/Fedora has always been targeting a just works experience and that should stay the goal. Note that this is very different from targeting ordinary people as ordinary people are too often taken as a synonym for not you, if you can use bugzilla you're not in the target audience and as a pretext for coders to do whatever they want without having to justify themselves or process bug reports. I had a very unpleasant example of such sophistry (you're not the target audience, the target audience supports me, and I have nothing to show for it, because if I did, it wouldn't be the target audience) recently. The gold standard should stay an argument needs to be justified technically, if you can't argue for it on technical grounds, do not drag mythical target audience in the debate. That's the only way to make objective assessments. And if you don't strive to stay objective, you'll just create a rotten from the core community -- Nicolas Mailhot -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
drago01 wrote: As for the GPG signature ... can't the drpm itself be signed? If the metadata is getting signed, it basically is already. The metadata contains a checksum of the DRPM, so if the metadata passes the signature check and the DRPM matches the checksum, the DRPM's integrity and uncompromisedness is verified. So I think it's safe to disable the checksum check for the rebuilt RPMs entirely. Kevin Kofler -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On 9/25/2009 1:00 PM, Adam Williamson wrote: So how do you get those 'others' to read the instructions? Linux used to basically require that you do that. Things had to be found in the system and then configured by hand. In the rush to make things auto-magic, which is not a bad thing IMO, when something does not 'just work' users are lost. They seem to have forgotten how to read. Or perhaps comprehend what they read. It's not like that. It's more like, the complexity of a Fedora system went over the roof in the past years, and has little in common with old limited *nix systems, so users now refuse to be dragged in reading the documentation, because they know that if they let themselves fooled by just a few apps, they'll saturate. Remember, human brains do not follow Moore's law, if we want to be able to continue growing the distribution, individual packages need to get simpler to use. And simpler does not mean I'll assume everything is a laptop, and everyone is a 'normal' user. Simpler means my app is smart enough to manage all cases alone without human intervention -- Nicolas Mailhot -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
I wrote: Basically, the safest solution would be to fix xz, then rebuild all noarch packages. At least those noarch packages which happened to be built on a PPC builder will need rebuilding after xz gets fixed. Please note that this probably also affects noarch subpackages. PS: If we don't care about F11-F12 and Rawhide-F12 deltas, then we don't have to rebuild anything, just fix xz before the release. Kevin Kofler -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On 9/25/2009 1:00 PM, Adam Williamson wrote: On Fri, 2009-09-25 at 12:43 -0400, David Boles wrote: 'lurker mode off' Comment from a 'lurker' You developers and all do a fine job with this. *But* you all seem to have the same wrong idea about *most* regular Linux users. At least the ones that I see on the help lists. *Almost none of them read that darn DOCs.* Nor do they read the release Notes. Nor do they read the FAQ's. Nor do they read they Known Problems. That's, um, pretty much what I was saying. Really? They don't read the instructions and hints *before* they destroy their working systems with an update? Or with a release upgrade? Really? Not the Newbies? Not the 'I think that I am an Expert' people? Not the 'I have much Linux experience' people? Really? Sure. Just follow the questions asked when something well written about pops up as a *surprise*. sigh I know this. It is, however, fair to point out that you're using an inherently skewed sample - the ones that I see on the help lists'. All other things being equal, the people who bother to read the documentation naturally show up far less often on help lists. =) Agreed. So how do you get those 'others' to read the instructions? Linux used to basically require that you do that. Things had to be found in the system and then configured by hand. In the rush to make things auto-magic, which is not a bad thing IMO, when something does not 'just work' users are lost. They seem to have forgotten how to read. Or perhaps comprehend what they read. And those users will always be Linux's bane. -- David signature.asc Description: OpenPGP digital signature -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On Fri, 25 Sep 2009, Alexander Boström wrote: tor 2009-09-24 klockan 16:31 -0400 skrev Seth Vidal: We're a bleeding edge distro - that's our VAUNTED mission - shouldn't we be able to assume that users of a BLEEDING EDGE, LATEST PACKAGE distro be able to run a single command? Though Fedora currently might not currently be an optimal just works experience, I think that should be the design goal. I want ordinary people to be able to use free software. (And not just use.) Wouldn't you rather lead the other distributions who already have that project goal? Let us lead, invent, whatever you want to call it. Let them refine and present to the 'ordinary people'. Let us play to our strengths, let them play to theirs. We win, they win, users win. -Mike-- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On Fri, Sep 25, 2009 at 5:55 PM, Mike McGrath mmcgr...@redhat.com wrote: Wouldn't you rather lead the other distributions who already have that project goal? Let us lead, invent, whatever you want to call it. Let them refine and present to the 'ordinary people'. Let us play to our strengths, let them play to theirs. We win, they win, users win. I don't think that works; in general doing infrastructure work without reference to a user experience is going to result in a big mismatch between the desired UI and what's available. In this particular case it seems to me we want it to be a dynamic property; e.g. if I start an update while on my mobile broadband card, suspend in the middle of downloading, go to an office where I have a local mirror, well ideally I'd be able to check a box in the UI to toggle it (if it really mattered), or even better the system would use some heuristics. But obviously, you can't disable delta compression by doing a yum remove in the middle of your download... -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
Hi. On Fri, 25 Sep 2009 19:08:25 +, Colin Walters wrote In this particular case it seems to me we want it to be a dynamic property; e.g. if I start an update while on my mobile broadband card, suspend in the middle of downloading, go to an office where I have a local mirror, well ideally I'd be able to check a box in the UI to toggle it (if it really mattered), or even better the system would use some heuristics. I really think that in this, rather special and constructed-sounding case, you may survive if the rest of the transaction ran on using delta-rpms instead of dynamically disabling them in the middle. -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On 09/25/2009 12:43 AM, Andre Robatino wrote:
On 09/24/2009 11:31 PM, Seth Vidal wrote:
On Thu, 24 Sep 2009, Ben Boeckel wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Seth Vidal wrote:
Does yum-presto make use of multiple cores when rebuilding the
RPMs?
(My machines only have one, so I can't tell.)
other than the xz compression part I believe most of the time is
disk bound, not cpu bound.
-sv
The rpmrebuild operation has saturated one of my 3.0GHz Intel Core
2 Duo cores for at least a minute on large updates, so my guess is
that that part is not multithreaded and CPU bound.
right, currently - using xz -6 - which is using more cpu.
Running {make,apply}deltaiso always maxes out my 2.7 GHz CPU, even
between ISOs with RPMs using Gzip. I'll check what happens next time
it's doing a delta rebuild during a F11 update.
As near as I can tell, applydeltarpm maxes out the CPU while it's
running during a F11 update, even though the RPMs use Gzip.
signature.asc
Description: OpenPGP digital signature
--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
setting xz level to 2 seems reasonable but how about thinking about a better solution: Q: what put the gpg signatures ? A: rpmbuild (or something that call it like mock) so how about to make rpmbuild generate gpg and checksums on both uncompressed and uncompressed payloads so every rpm would carry the signatures and checksums if its payload is decompressed beside the typical signatures and checksums maybe this is not for F12 as it could require some changes in rpm, yum, deltarpm, ...etc. but this is the right thing http://fedoratux.blogspot.com/2009/05/xz-compression.html the default is -6 we are talking about -2 could you please do your benchmarks at that level please -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 shmuel siegel wrote: The article also hints at our problem. We ARE doing the compression on the end user side. So the compression is costing us 3 minutes to save 24 megabytes of transmission. This actually slows things down for most broadband users. Since when was yum-presto about time? I thought it was about bandwidth usage. Here, the dorm connections are capped at 600kb/s (well, not a hard cap, but it can be annoying anyways). At one university I know (with over 20k students on the main campus), there is (or was last year at least) a cap at 2GB / week. Go over and you're capped at 56k for the rest of the semester. I can't imagine Fedora on such a restriction (and I have 4 machines to update, 2 with largely non-overlapping package sets, the other 2 are similar and a caching server would help) and that's a lot of students that would be hard pressed to use Fedora at college. CPU time is cheaper than bandwidth these days. Maybe I'm mistaken about what yum-presto was aiming to solve? - --Ben -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) iQIcBAEBCAAGBQJKu3NrAAoJEKaxavVX4C1XRJIP/0ELUuNd/35Q+CXePgeJR2HN UnUdp2OyApqTwqWtTQ4sPcICKAcJKo8wct0G3JK11t1x4FOZA/O26Hs815DpzWkj kHzV02MRQpa0B4ZKnMlZ/WDpVrPehi6nwqnxjL5k8g+OuVdSaEXGlQQTG+LK0KER DdRMKKnop/fZEWbTYj1OMf8Nv7KeXW92EPdyaHVJGKOSW+SVi7scXbHHxsSkhqe6 7zG6HvnUt7aNAcH7NMM3dlNDMKJwiU1Q3r266XBcVtu4inogkZ9xwMkmKqr2QMel aReynuAg8tKk6M8Drwbgj7VWLtjZRMjKuICslw3lj+2w0bXMJ20P7g7gTcObp8ki KfjeHTXQZHnQpCAbOsRCUrmNSz3hfyfB3HsaB7DI9Txa05aJva6pis8ajanc0zSe Ob8kvduhDriEdk1sp/3UEng5AYfggr2O5+uTnWsc4pfqJVNwZvPOT0GBnG7XaY/+ YsOo7k2JM0vbL/FEgJPL9Y/h0jgJpGWhZPdVWC+iPOg6uRau7Npso3mbenJXpOz4 9MLb/58HRmoPDTo5PKuWQnenbNzgVyJA3TNUavhihIoVrIhCCnzj5Sh9sharzuVX UKe7K0Xd0mf2s6GCfGqf5f4j14UdySxEC/PT0NP+U18SRWqwKq+llE+iv1gEpIJH ipdaPtQVekcIoM7LKJo4 =it9Q -END PGP SIGNATURE- -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On Thu, 24 Sep 2009, Ben Boeckel wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 shmuel siegel wrote: The article also hints at our problem. We ARE doing the compression on the end user side. So the compression is costing us 3 minutes to save 24 megabytes of transmission. This actually slows things down for most broadband users. Since when was yum-presto about time? I thought it was about bandwidth usage. Here, the dorm connections are capped at 600kb/s (well, not a hard cap, but it can be annoying anyways). At one university I know (with over 20k students on the main campus), there is (or was last year at least) a cap at 2GB / week. Go over and you're capped at 56k for the rest of the semester. I can't imagine Fedora on such a restriction (and I have 4 machines to update, 2 with largely non-overlapping package sets, the other 2 are similar and a caching server would help) and that's a lot of students that would be hard pressed to use Fedora at college. CPU time is cheaper than bandwidth these days. Maybe I'm mistaken about what yum-presto was aiming to solve? it's not about local cpu But if we make presto be on by default and the local performance is so bad for people with fast connections that it is almost unusuable then we have a problem. So the idea is: 1. make performance not suck 2. maybe not make presto the default anyway. Now #1 is obvious, I think :) #2 is about the way someone would use the system. If I'm a place where I know the bandwidth is questionable then I figure immediately after install I can run: yum install yum-presto and be ready to go. Or, we install yum-presto by default but disable it. So the first thing someone with bandwidth issues does is enable the plugin. i think that's what this is all about. -sv -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On Thu, 2009-09-24 at 16:00 -0400, Seth Vidal wrote: #2 is about the way someone would use the system. If I'm a place where I know the bandwidth is questionable then I figure immediately after install I can run: yum install yum-presto and be ready to go. Or, we install yum-presto by default but disable it. So the first thing someone with bandwidth issues does is enable the plugin. Neither of these will happen because they require esoteric knowledge of yum plugins that users don't have. So if we turn it off by default, it will not be used by a significant percentage of the people for whom it is beneficial. And all the infrastructure cost we put into maintaining delta rpms is effectively wasted... -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On Thu, 2009-09-24 at 16:22 -0400, Seth Vidal wrote: I don't consider the single command a significant barrier, either way. Maybe not for people who maintain yum... -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On 09/24/2009 04:22 PM, Seth Vidal wrote: Not really and you're being a little dramatic, I think. If we don't install it but have its default state be 'on' then they can just do: yum install yum-presto and then they're done. On the other hand someone who has good bandwidth can easily also do: yum remove yum-presto and it is off. As I said when asked the last time - I don't have a preference if it is installed and on or not installed. I don't consider the single command a significant barrier, either way. I don't think it is the end of the world, either way. The inconvenience for a high-bandwidth user of having to wait a little longer for the update to complete is minor compared to someone on dialup having to spend hours updating (or days, if they've just installed and are unaware of yum-presto), or someone with bandwidth caps having to pay for the download. Besides, the high-bandwidth users can keep themselves busy using the connection for something else while the update finishes. And even they might want to help reduce the load on the mirrors. signature.asc Description: OpenPGP digital signature -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On Thu, 24 Sep 2009, Matthias Clasen wrote: On Thu, 2009-09-24 at 16:22 -0400, Seth Vidal wrote: I don't consider the single command a significant barrier, either way. Maybe not for people who maintain yum... Seriously? We have lots of documentation that is run this one command. We're a bleeding edge distro - that's our VAUNTED mission - shouldn't we be able to assume that users of a BLEEDING EDGE, LATEST PACKAGE distro be able to run a single command? I think if you assume too little of your users you end up with bad users. I assume they are capable of a single command to either help their bandwidth (in the case where presto is not installed and they have poor bandwidth) or speed up the update process (in the case where presto is installed and they have good bandwidth). Seriously.. -sv -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
Once upon a time, Matthias Clasen mcla...@redhat.com said: Neither of these will happen because they require esoteric knowledge of yum plugins that users don't have. So if we turn it off by default, it will not be used by a significant percentage of the people for whom it is beneficial. If it impacts update performance by default, then it should be off, or we'll have another thing added to the oft-repeated Fedora fixes list (like yum remove pulseaudio, turn off SELinux, etc.). Lots of users are connected to high-speed Internet, so their update performance should not be impacted to help those that are not as well connected. -- Chris Adams cmad...@hiwaay.net Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble. -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On 09/25/2009 01:59 AM, Andre Robatino wrote: The inconvenience for a high-bandwidth user of having to wait a little longer for the update to complete is minor compared to someone on dialup having to spend hours updating (or days, if they've just installed and are unaware of yum-presto), or someone with bandwidth caps having to pay for the download. Besides, the high-bandwidth users can keep themselves busy using the connection for something else while the update finishes. And even they might want to help reduce the load on the mirrors. Since it has been added the gnome-desktop (by me) and kde-desktop (by the KDE SIG) package groups in Fedora 12, majority of users will get it by default. Rahul -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On Thu, 24 Sep 2009, Adam Williamson wrote: On Thu, 2009-09-24 at 15:32 -0500, Chris Adams wrote: Lots of users are connected to high-speed Internet, so their update performance should not be impacted to help those that are not as well connected. if we fix xz to use a sensible compression level, it won't be. Well, unless you mean 'positively impacted'. :) Up to a point - it'll still churn your disks a bit. -sv -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On Thu, Sep 24, 2009 at 04:25:46PM -0400, Matthias Clasen wrote: On Thu, 2009-09-24 at 16:22 -0400, Seth Vidal wrote: I don't consider the single command a significant barrier, either way. Maybe not for people who maintain yum... I get the feeling that either 1) you think our users have no idea what the command line is or 2) hate the command line with a passion or 3) don't believe there is no suitable graphical way to install this or generally that you find our users to be unintelligent enough to work out how to get a piece of software onto their systems. I find that to be rather wrong, and I don't think you have to be the maintainer of yum to understand how to install something. Now, I could be very very wrong in these feelings and if so, great. But you didn't reply with much so I had to work with what I had, inferences and all. josh -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On Thu, 24 Sep 2009, Adam Williamson wrote: On Thu, 2009-09-24 at 17:21 -0400, Matthias Clasen wrote: On Thu, 2009-09-24 at 16:52 -0400, Josh Boyer wrote: or generally that you find our users to be unintelligent enough to work out how to get a piece of software onto their systems. I find that to be rather wrong, and I don't think you have to be the maintainer of yum to understand how to install something. There is a difference between saying someone is 'unintelligent' and not expecting them to figure out on their own that in order to solve their bandwidth problem they should install something called yum-presto. Right. The issue isn't whether people would be able to install the package, it's whether they'd be aware that they ought to be installing it in the first place. which is why, I thought, we have documentation. -sv -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On Thu, 2009-09-24 at 17:46 -0400, Seth Vidal wrote: Right. The issue isn't whether people would be able to install the package, it's whether they'd be aware that they ought to be installing it in the first place. which is why, I thought, we have documentation. oh, you optimist :) -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org http://www.happyassassin.net -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On Thu, Sep 24, 2009 at 11:46 PM, Seth Vidal skvi...@fedoraproject.org wrote: On Thu, 24 Sep 2009, Adam Williamson wrote: On Thu, 2009-09-24 at 17:21 -0400, Matthias Clasen wrote: On Thu, 2009-09-24 at 16:52 -0400, Josh Boyer wrote: or generally that you find our users to be unintelligent enough to work out how to get a piece of software onto their systems. I find that to be rather wrong, and I don't think you have to be the maintainer of yum to understand how to install something. There is a difference between saying someone is 'unintelligent' and not expecting them to figure out on their own that in order to solve their bandwidth problem they should install something called yum-presto. Right. The issue isn't whether people would be able to install the package, it's whether they'd be aware that they ought to be installing it in the first place. which is why, I thought, we have documentation. OK lets read up docs on how can I let updates not require that much bandwith ... well how many users think that way? -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On Thu, 2009-09-24 at 23:13 -0400, Seth Vidal wrote: On Fri, 25 Sep 2009, drago01 wrote: which is why, I thought, we have documentation. OK lets read up docs on how can I let updates not require that much bandwith ... well how many users think that way? How many uses who are using a bleeding edge advanced distro? I hope most of them. I know that when I have problems with something I google for the answer. But they wouldn't be having any problems. It's not as if anything doesn't work if you're not using yum-presto. They'd just be missing out on a possible improvement. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org http://www.happyassassin.net -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On 09/24/2009 11:31 PM, Seth Vidal wrote:
On Thu, 24 Sep 2009, Ben Boeckel wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Seth Vidal wrote:
Does yum-presto make use of multiple cores when rebuilding the
RPMs?
(My machines only have one, so I can't tell.)
other than the xz compression part I believe most of the time is
disk bound, not cpu bound.
-sv
The rpmrebuild operation has saturated one of my 3.0GHz Intel Core
2 Duo cores for at least a minute on large updates, so my guess is
that that part is not multithreaded and CPU bound.
right, currently - using xz -6 - which is using more cpu.
Running {make,apply}deltaiso always maxes out my 2.7 GHz CPU, even
between ISOs with RPMs using Gzip. I'll check what happens next time
it's doing a delta rebuild during a F11 update.
signature.asc
Description: OpenPGP digital signature
--
fedora-devel-list mailing list
fedora-devel-list@redhat.com
https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On Wed, Sep 23, 2009 at 10:51 AM, Michal Schmidt mschm...@redhat.com wrote: Dne Wed, 23 Sep 2009 07:04:23 +0300 Jonathan Dieter napsal(a): https://bugzilla.redhat.com/show_bug.cgi?id=524720 https://bugzilla.redhat.com/show_bug.cgi?id=524982 ... The second one has to do with the fact that when rebuilding the rpms, we have to recompress the data, and xz compression is over 10x slower than gzip. Do I understand it right that yum-presto compresses the data and then passes them to rpm which decompresses them back again? Why? Is it because it's currently the only way to verify checksums/signatures? We had a IRC discussion about this yesterday ... it is not yum-presto but delta rpm and it does not make sense at all. It should just create uncompressed rpms (assuming rpm can handle them which it should) ...according to Seth yum does not care whether the rpms are compressed or not. So yes the compression is a useless step here. -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On Wed, 2009-09-23 at 10:49 +0200, drago01 wrote: On Wed, Sep 23, 2009 at 10:51 AM, Michal Schmidt mschm...@redhat.com wrote: Dne Wed, 23 Sep 2009 07:04:23 +0300 Jonathan Dieter napsal(a): https://bugzilla.redhat.com/show_bug.cgi?id=524720 https://bugzilla.redhat.com/show_bug.cgi?id=524982 ... The second one has to do with the fact that when rebuilding the rpms, we have to recompress the data, and xz compression is over 10x slower than gzip. Do I understand it right that yum-presto compresses the data and then passes them to rpm which decompresses them back again? Why? Is it because it's currently the only way to verify checksums/signatures? We had a IRC discussion about this yesterday ... it is not yum-presto but delta rpm and it does not make sense at all. It should just create uncompressed rpms (assuming rpm can handle them which it should) ...according to Seth yum does not care whether the rpms are compressed or not. So yes the compression is a useless step here. As I think may have been mentioned elsewhere, the *only* problem is that the rpm signatures must match and the signatures are over the *compressed* rpm. I would *love* to see deltarpm rebuilding uncompressed rpms, but that will require storing two signatures per rpm in the metadata (compressed and uncompressed sha256), and either modifying yum to check the appropriate one, or deltarpm to change the rpm's signature to the uncompressed one. I don't think we want to go down the road of having deltarpm-rebuilt rpms not having their signature checked at all. Jonathan signature.asc Description: This is a digitally signed message part -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On Wed, 23 Sep 2009, Jonathan Dieter wrote: As I think may have been mentioned elsewhere, the *only* problem is that the rpm signatures must match and the signatures are over the *compressed* rpm. I would *love* to see deltarpm rebuilding uncompressed rpms, but that will require storing two signatures per rpm in the metadata (compressed and uncompressed sha256), and either modifying yum to check the appropriate one, or deltarpm to change the rpm's signature to the uncompressed one. I don't think we want to go down the road of having deltarpm-rebuilt rpms not having their signature checked at all. I agree about having to deal with the signatures but as a minor mitigating circumstance - we're going to be gpg signing the repomd.xml - which should at least help you keep a safe infrastructure down to the deltarpm metadata. -sv -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On Wed, 2009-09-23 at 15:22 +0300, Jonathan Dieter wrote: On Wed, 2009-09-23 at 10:49 +0200, drago01 wrote: On Wed, Sep 23, 2009 at 10:51 AM, Michal Schmidt mschm...@redhat.com wrote: Dne Wed, 23 Sep 2009 07:04:23 +0300 Jonathan Dieter napsal(a): https://bugzilla.redhat.com/show_bug.cgi?id=524720 https://bugzilla.redhat.com/show_bug.cgi?id=524982 ... The second one has to do with the fact that when rebuilding the rpms, we have to recompress the data, and xz compression is over 10x slower than gzip. Do I understand it right that yum-presto compresses the data and then passes them to rpm which decompresses them back again? Why? Is it because it's currently the only way to verify checksums/signatures? We had a IRC discussion about this yesterday ... it is not yum-presto but delta rpm and it does not make sense at all. It should just create uncompressed rpms (assuming rpm can handle them which it should) ...according to Seth yum does not care whether the rpms are compressed or not. So yes the compression is a useless step here. As I think may have been mentioned elsewhere, the *only* problem is that the rpm signatures must match and the signatures are over the *compressed* rpm. No, we have at least 3 problems I think: 1. Nobody wants to download uncompressed rpms, if they don't have presto. 2. gig signature is over the rpm data (and thus. is over compressed data). 3. createrepo sha256 data is over the entire rpm (and thus. is over compressed data). ...but to me this is all a _problem_in_xz_, not presto/deltarpms. If nobody can fix xz before F12 GA then IMNSO we should revert the compression to something that works ... the minor savings in xz compression isn't worth as much as delta's. -- James Antill - ja...@fedoraproject.org I'd just like to see a realistic approach to updates via packages. -- Les Mikesell -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On Wed, 23 Sep 2009, James Antill wrote: No, we have at least 3 problems I think: 1. Nobody wants to download uncompressed rpms, if they don't have presto. 2. gig signature is over the rpm data (and thus. is over compressed data). 3. createrepo sha256 data is over the entire rpm (and thus. is over compressed data). ...but to me this is all a _problem_in_xz_, not presto/deltarpms. If nobody can fix xz before F12 GA then IMNSO we should revert the compression to something that works ... the minor savings in xz compression isn't worth as much as delta's. agreed. -sv -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On Wed, Sep 23, 2009 at 10:23:31AM -0400, Seth Vidal wrote: On Wed, 23 Sep 2009, James Antill wrote: No, we have at least 3 problems I think: 1. Nobody wants to download uncompressed rpms, if they don't have presto. 2. gig signature is over the rpm data (and thus. is over compressed data). 3. createrepo sha256 data is over the entire rpm (and thus. is over compressed data). ...but to me this is all a _problem_in_xz_, not presto/deltarpms. If nobody can fix xz before F12 GA then IMNSO we should revert the compression to something that works ... the minor savings in xz compression isn't worth as much as delta's. agreed. The xz feature is among the Fedora 12 talking points, so I've made a note to track this going forward. https://fedoraproject.org/wiki/Fedora_12_Talking_Points We'll need a heads-up to the Marketing list if there's a reversion that affects F12 GA. -- Paul W. Frieldshttp://paul.frields.org/ gpg fingerprint: 3DA6 A0AC 6D58 FEC4 0233 5906 ACDB C937 BD11 3717 http://redhat.com/ - - - - http://pfrields.fedorapeople.org/ irc.freenode.net: stickster @ #fedora-docs, #fedora-devel, #fredlug -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On Wed, Sep 23, 2009 at 10:20:21AM -0400, James Antill wrote: ...but to me this is all a _problem_in_xz_, not presto/deltarpms. If nobody can fix xz before F12 GA then IMNSO we should revert the compression to something that works ... the minor savings in xz compression isn't worth as much as delta's. I guess there won't be any deltarpms to update from F11 to F12, so afaics it would be enough to only switch back to gzip payload for everthing that is going into F12 updates(-testing). Is this true? Then we would still save space on the iso images and for the Everything Repository, but deltarpms would still be possible. Regards Till pgpFb7V9FfCR7.pgp Description: PGP signature -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On Wed, 23 Sep 2009, Till Maas wrote: On Wed, Sep 23, 2009 at 10:20:21AM -0400, James Antill wrote: ...but to me this is all a _problem_in_xz_, not presto/deltarpms. If nobody can fix xz before F12 GA then IMNSO we should revert the compression to something that works ... the minor savings in xz compression isn't worth as much as delta's. I guess there won't be any deltarpms to update from F11 to F12, so afaics it would be enough to only switch back to gzip payload for everthing that is going into F12 updates(-testing). Is this true? Then we would still save space on the iso images and for the Everything Repository, but deltarpms would still be possible. We'd need a mass rebuild to all pkgs in rawhide/f12-candidate to shake Xz out of the payload compression. That's non-trivial. -sv -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On Wed, 23 Sep 2009, Paul W. Frields wrote: The xz feature is among the Fedora 12 talking points, so I've made a note to track this going forward. https://fedoraproject.org/wiki/Fedora_12_Talking_Points We'll need a heads-up to the Marketing list if there's a reversion that affects F12 GA. We don't have anything sexier than 'We changed the compression system'? in F12? I'm sure we can come up with something more interesting. Maybe we changed the background of something.. That's more interesting than this, right? :) -sv -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On Wed, Sep 23, 2009 at 4:20 PM, James Antill ja...@fedoraproject.org wrote: On Wed, 2009-09-23 at 15:22 +0300, Jonathan Dieter wrote: On Wed, 2009-09-23 at 10:49 +0200, drago01 wrote: On Wed, Sep 23, 2009 at 10:51 AM, Michal Schmidt mschm...@redhat.com wrote: Dne Wed, 23 Sep 2009 07:04:23 +0300 Jonathan Dieter napsal(a): https://bugzilla.redhat.com/show_bug.cgi?id=524720 https://bugzilla.redhat.com/show_bug.cgi?id=524982 ... The second one has to do with the fact that when rebuilding the rpms, we have to recompress the data, and xz compression is over 10x slower than gzip. Do I understand it right that yum-presto compresses the data and then passes them to rpm which decompresses them back again? Why? Is it because it's currently the only way to verify checksums/signatures? We had a IRC discussion about this yesterday ... it is not yum-presto but delta rpm and it does not make sense at all. It should just create uncompressed rpms (assuming rpm can handle them which it should) ...according to Seth yum does not care whether the rpms are compressed or not. So yes the compression is a useless step here. As I think may have been mentioned elsewhere, the *only* problem is that the rpm signatures must match and the signatures are over the *compressed* rpm. No, we have at least 3 problems I think: 1. Nobody wants to download uncompressed rpms, if they don't have presto. 2. gig signature is over the rpm data (and thus. is over compressed data). 3. createrepo sha256 data is over the entire rpm (and thus. is over compressed data). ...but to me this is all a _problem_in_xz_, not presto/deltarpms. If nobody can fix xz before F12 GA then IMNSO we should revert the compression to something that works ... the minor savings in xz compression isn't worth as much as delta's. Does not matter which compression algorithm we use creating a compressed rpm just to uncompressed it again shortly after that is a waste of cycles/power/time. As for the GPG signature ... can't the drpm itself be signed? So we would only need to check that, rather than the rebuilt rpm if we don't trust the files on the disk we already lost anyway (box is compromised). -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On Wed, 23 Sep 2009, Paul W. Frields wrote: The xz feature is among the Fedora 12 talking points, so I've made a note to track this going forward. https://fedoraproject.org/wiki/Fedora_12_Talking_Points We'll need a heads-up to the Marketing list if there's a reversion that affects F12 GA. We don't have anything sexier than 'We changed the compression system'? in F12? I'm sure we can come up with something more interesting. Maybe we changed the background of something.. That's more interesting than this, right? :) I'd like to say we fixed the packages that include fonts as we were supposed to do for F11 but sadly many maintainers didn't change a dot in their spec (and even added more un-reviewed fonts in the distro, like xine-ui-skins for example) -- Nicolas Mailhot -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On Wed, 23 Sep 2009, drago01 wrote: Does not matter which compression algorithm we use creating a compressed rpm just to uncompressed it again shortly after that is a waste of cycles/power/time. As for the GPG signature ... can't the drpm itself be signed? We'd need to do that signing which would take, umm, forever. -sv -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On Wed, Sep 23, 2009 at 5:11 PM, Seth Vidal skvi...@fedoraproject.org wrote: On Wed, 23 Sep 2009, drago01 wrote: Does not matter which compression algorithm we use creating a compressed rpm just to uncompressed it again shortly after that is a waste of cycles/power/time. As for the GPG signature ... can't the drpm itself be signed? We'd need to do that signing which would take, umm, forever. What? You mean at compose time? (Signing on the client side would not make much sense) -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On Wed, 23 Sep 2009, drago01 wrote: On Wed, Sep 23, 2009 at 5:11 PM, Seth Vidal skvi...@fedoraproject.org wrote: On Wed, 23 Sep 2009, drago01 wrote: Does not matter which compression algorithm we use creating a compressed rpm just to uncompressed it again shortly after that is a waste of cycles/power/time. As for the GPG signature ... can't the drpm itself be signed? We'd need to do that signing which would take, umm, forever. What? You mean at compose time? (Signing on the client side would not make much sense) I mean on the server/repo side. The steps we'd need to do for a full release would be: 1. compose tree 2. sign pkgs in tree 3. make deltarpms of pkgs vs older tree 4. sign deltarpms 5. generate repository metadata that would take a long time. -sv -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On Wed, Sep 23, 2009 at 10:58:23AM -0400, Seth Vidal wrote: On Wed, 23 Sep 2009, Till Maas wrote: On Wed, Sep 23, 2009 at 10:20:21AM -0400, James Antill wrote: ...but to me this is all a _problem_in_xz_, not presto/deltarpms. If nobody can fix xz before F12 GA then IMNSO we should revert the compression to something that works ... the minor savings in xz compression isn't worth as much as delta's. I guess there won't be any deltarpms to update from F11 to F12, so afaics it would be enough to only switch back to gzip payload for everthing that is going into F12 updates(-testing). Is this true? Then we would still save space on the iso images and for the Everything Repository, but deltarpms would still be possible. We'd need a mass rebuild to all pkgs in rawhide/f12-candidate to shake Xz out of the payload compression. That's non-trivial. Why do we need a mass rebuild? Afaics it is only needed to change the compression back to gzip at the time of the final freeze, so that all newly build packages (which are the ones going to F12-updates(-testing) are build with gzip compression. Then delta rpms can be created from F12 Everything to F12 updates-testing. For packages that are built to break the freeze, there could be a seperate build target tag, that still uses the xz compression. This is all under the assumption, that delta rpm creation from a xz compressed rpm to a gzip compressed rpm works. Regards Till pgpVHZ63y5dmX.pgp Description: PGP signature -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On Wed, Sep 23, 2009 at 5:14 PM, Seth Vidal skvi...@fedoraproject.org wrote: On Wed, 23 Sep 2009, drago01 wrote: On Wed, Sep 23, 2009 at 5:11 PM, Seth Vidal skvi...@fedoraproject.org wrote: On Wed, 23 Sep 2009, drago01 wrote: Does not matter which compression algorithm we use creating a compressed rpm just to uncompressed it again shortly after that is a waste of cycles/power/time. As for the GPG signature ... can't the drpm itself be signed? We'd need to do that signing which would take, umm, forever. What? You mean at compose time? (Signing on the client side would not make much sense) I mean on the server/repo side. The steps we'd need to do for a full release would be: 1. compose tree 2. sign pkgs in tree 3. make deltarpms of pkgs vs older tree 4. sign deltarpms 5. generate repository metadata that would take a long time. Yeah but if you take into account the time saved on x clients it would be worth it (assume x is very high). How long would the extra signing process take? -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On Wed, 23 Sep 2009, drago01 wrote: Yeah but if you take into account the time saved on x clients it would be worth it (assume x is very high). How long would the extra signing process take? figure at leastas long as it takes to gpg sign the distro now - probably longer b/c there are potentially MORE drpms. Finally, this would mean we're generating rpms from the deltarpms that do not match the original rpms. That may or may not be fatal but it is feels weird. -sv -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
Seth Vidal, Wed, 23 Sep 2009 11:00:16 -0400: We don't have anything sexier than 'We changed the compression system'? in F12? I'm sure we can come up with something more interesting. Maybe we changed the background of something.. That's more interesting than this, right? :) We made it much easier for you to file duplicate bugs? Matěj P.S.: Sorry for ranting but I spent yesterday fighting with Firefox abrt crashes. -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On Wed, 23 Sep 2009, Matej Cepl wrote: Seth Vidal, Wed, 23 Sep 2009 11:00:16 -0400: We don't have anything sexier than 'We changed the compression system'? in F12? I'm sure we can come up with something more interesting. Maybe we changed the background of something.. That's more interesting than this, right? :) We made it much easier for you to file duplicate bugs? :) -sv -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On Wed, 2009-09-23 at 10:23 -0400, Seth Vidal wrote: ...but to me this is all a _problem_in_xz_, not presto/deltarpms. If nobody can fix xz before F12 GA then IMNSO we should revert the compression to something that works ... the minor savings in xz compression isn't worth as much as delta's. agreed. Dropping xz at this point would mean another full mass rebuild. Do-able but needs to happen very very soon. -- Jesse Keating Fedora -- Freedom² is a feature! identi.ca: http://identi.ca/jkeating signature.asc Description: This is a digitally signed message part -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On 23/09/09 17:00, Jesse Keating wrote: On Wed, 2009-09-23 at 10:23 -0400, Seth Vidal wrote: ...but to me this is all a _problem_in_xz_, not presto/deltarpms. If nobody can fix xz before F12 GA then IMNSO we should revert the compression to something that works agreed. Dropping xz at this point would mean another full mass rebuild. Do-able but needs to happen very very soon. I'd be interested in trying to fix xz (I disagree entirely that this is an xz bug; it's a bug in the assumptions of those whose used it for the wrong purpose), but presumably it means rebuilding packages built on certain arches anyway - and if that's not easy to determine, does that imply a rebuild anyway? Cheers Alex. -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On Wed, Sep 23, 2009 at 09:00:37AM -0700, Jesse Keating wrote: On Wed, 2009-09-23 at 10:23 -0400, Seth Vidal wrote: ...but to me this is all a _problem_in_xz_, not presto/deltarpms. If nobody can fix xz before F12 GA then IMNSO we should revert the compression to something that works ... the minor savings in xz compression isn't worth as much as delta's. agreed. Dropping xz at this point would mean another full mass rebuild. Do-able but needs to happen very very soon. I'm still wondering why it is not possible to use xz for the Everything/ and iso rpm and gz for the rpms going to updates. Regards Till pgpwmkKjQqeKh.pgp Description: PGP signature -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
Alex Hudson wrote: I'd be interested in trying to fix xz (I disagree entirely that this is an xz bug; it's a bug in the assumptions of those whose used it for the wrong purpose), If I understand correctly what the problem is - that successfully xz-uncompressing a file requires being on the same endian arch as the one it was compressed on - that's just wrong. It shouldn't be platform-dependent at all. I checked Bugzilla for xz bugs and this isn't listed. Should someone file it? signature.asc Description: OpenPGP digital signature -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
Andre Robatino (an...@bwh.harvard.edu) said: If I understand correctly what the problem is - that successfully xz-uncompressing a file requires being on the same endian arch as the one it was compressed on - that's just wrong. It shouldn't be platform-dependent at all. That is not the issue. There are two issues: - Compressing a file on different arches, while it produces output that any arch can decompress, does produce *different* output. - Compression of xz is not fast. Bill -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On 09/23/2009 07:17 PM, Bill Nottingham wrote: Andre Robatino (an...@bwh.harvard.edu) said: If I understand correctly what the problem is - that successfully xz-uncompressing a file requires being on the same endian arch as the one it was compressed on - that's just wrong. It shouldn't be platform-dependent at all. That is not the issue. There are two issues: - Compressing a file on different arches, while it produces output that any arch can decompress, does produce *different* output. - Compression of xz is not fast. Ah, OK. But hasn't openSUSE been using xz in their deltarpms since 11.0 (over a year ago)? What are they doing differently so they aren't affected? signature.asc Description: OpenPGP digital signature -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On Wed, Sep 23, 2009 at 07:38:24PM -0400, Andre Robatino wrote: On 09/23/2009 07:17 PM, Bill Nottingham wrote: Andre Robatino (an...@bwh.harvard.edu) said: If I understand correctly what the problem is - that successfully xz-uncompressing a file requires being on the same endian arch as the one it was compressed on - that's just wrong. It shouldn't be platform-dependent at all. That is not the issue. There are two issues: - Compressing a file on different arches, while it produces output that any arch can decompress, does produce *different* output. - Compression of xz is not fast. Ah, OK. But hasn't openSUSE been using xz in their deltarpms since 11.0 (over a year ago)? What are they doing differently so they aren't affected? 1) Maybe they don't build noarch rpms on ppc 2) They use a faster compression, that creates bigger output. At least I remember something like this mentioned in this thread or some other discussion. Regards Till pgpGmcA0YAMxU.pgp Description: PGP signature -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On 09/23/2009 07:17 PM, Bill Nottingham wrote: Andre Robatino (an...@bwh.harvard.edu) said: If I understand correctly what the problem is - that successfully xz-uncompressing a file requires being on the same endian arch as the one it was compressed on - that's just wrong. It shouldn't be platform-dependent at all. That is not the issue. There are two issues: - Compressing a file on different arches, while it produces output that any arch can decompress, does produce *different* output. - Compression of xz is not fast. Bill man xz When compressing, xz automatically adjusts the compression settings downwards if the memory usage limit would be exceeded, so it is safe to specify a high preset level even on systems that don’t have lots of RAM. I hope we turned off this feature in rpm's use of xz as well. Warren -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On Wed, 2009-09-23 at 17:22 +0200, Till Maas wrote: This is all under the assumption, that delta rpm creation from a xz compressed rpm to a gzip compressed rpm works. Yeh, I don't know the answer to that. I'd _guess_ that it would work, but someone needs to try it. This would mean that drpms on rawhide will still suck upto F12, but I could live with that :). I assume we don't do F11 = F12 drpms? On the other side of it, does anyone have any stats. on how much was saved by using Xz instead of bzip2? -- Ie. what we'd lose if we did a mass rebuild. -- James Antill ja...@fedoraproject.org Fedora -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On 09/23/2009 09:46 PM, James Antill wrote: On Wed, 2009-09-23 at 17:22 +0200, Till Maas wrote: This is all under the assumption, that delta rpm creation from a xz compressed rpm to a gzip compressed rpm works. Yeh, I don't know the answer to that. I'd _guess_ that it would work, but someone needs to try it. This would mean that drpms on rawhide will still suck upto F12, but I could live with that :). I assume we don't do F11 = F12 drpms? On the other side of it, does anyone have any stats. on how much was saved by using Xz instead of bzip2? -- Ie. what we'd lose if we did a mass rebuild. AFAIK we used gzip (well, libz really) before, not bzip2. If we'd used bzip2, switching to xz might even have resulted in a speedup. -Toshio -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
2009/9/24 Bill Nottingham nott...@redhat.com: That is not the issue. There are two issues: - Compressing a file on different arches, while it produces output that any arch can decompress, does produce *different* output. - Compression of xz is not fast. So long as RPM is not rebuilt on client's computer, xz is definitely a favorable choice: less download, fast decompression. What about to rebuild unzipped RPM on client's computer, and compare its signature with pre-calculated one? It should be fast on server side (pre-calculate signature for all drpm is too slow, as seth said.) -- bbbush ^_^ -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
Jonathan Dieter (jdie...@gmail.com) said: The second one has to do with the fact that when rebuilding the rpms, we have to recompress the data, and xz compression is over 10x slower than gzip. Stats of the day... I took an existing xz-compressed RPM, and built a newer release of that package with varying xz levels. The numbers are: - the new package size at that XZ compression level - the time to create a new deltarpm from a delta to that new version level 1 1416k real0m0.935s user0m0.893s sys 0m0.036s level 2 952k real0m0.839s user0m0.790s sys 0m0.048s level 3 848k real0m2.762s user0m2.702s sys 0m0.055s level 4 832k real0m2.902s user0m2.817s sys 0m0.084s level 5 824k real0m3.269s user0m3.131s sys 0m0.128s level 6 812k real0m4.364s user0m4.026s sys 0m0.164s level 7 (what we do now) 812k real0m4.698s user0m4.516s sys 0m0.176s So... just set the xz compression level to 2, let it be that way for future builds, and go about our business? Bill -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
On Thu, 2009-09-24 at 00:46 -0400, James Antill wrote: On Wed, 2009-09-23 at 17:22 +0200, Till Maas wrote: This is all under the assumption, that delta rpm creation from a xz compressed rpm to a gzip compressed rpm works. Yeh, I don't know the answer to that. I'd _guess_ that it would work, but someone needs to try it. Tried it and it works correctly for both directions (old rpm gzip, new rpm xz and vice versa). This would mean that drpms on rawhide will still suck upto F12, but I could live with that :). I assume we don't do F11 = F12 drpms? On the other side of it, does anyone have any stats. on how much was saved by using Xz instead of bzip2? -- Ie. what we'd lose if we did a mass rebuild. Here's one I came across: http://fedoratux.blogspot.com/2009/05/xz-compression.html Jonathan signature.asc Description: This is a digitally signed message part -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: yum-presto not on by default
Jonathan Dieter wrote: Tried it and it works correctly for both directions (old rpm gzip, new rpm xz and vice versa). This would mean that drpms on rawhide will still suck upto F12, but I could live with that :). I assume we don't do F11 = F12 drpms? On the other side of it, does anyone have any stats. on how much was saved by using Xz instead of bzip2? -- Ie. what we'd lose if we did a mass rebuild. Here's one I came across: http://fedoratux.blogspot.com/2009/05/xz-compression.html Jonathan The article also hints at our problem. We ARE doing the compression on the end user side. So the compression is costing us 3 minutes to save 24 megabytes of transmission. This actually slows things down for most broadband users. -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
