Re: fedup performance
On 07/03/2013 07:42 AM, Alex G. wrote: On 07/02/2013 08:28 PM, Neal Becker wrote: Not d/l speed related. I just want to share. I update a very fast 8 core server, with a conventional disk drive. Took 2-3 hours, not including d/l. I update my laptop which has an ssd (and MORE packages). Took 10-15 minutes. I think this might simply have to do with rpm running ldconfig (a very disk IO expensive operation) for a large number of packages. I'm not sure yum/rpm has deferred ldconfig processing. DISCLAIMER: I may be very wrong. Please don't quote me on this. ldconfig gets run a lot yes, but its also really fast these days. fdatasync() which gets called even more (a lot more at that) seems like a more likely painpoint on upgrades. - Panu - -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: fedup performance
On 07/03/2013 09:59 AM, Panu Matilainen wrote: On 07/03/2013 07:42 AM, Alex G. wrote: On 07/02/2013 08:28 PM, Neal Becker wrote: Not d/l speed related. I just want to share. I update a very fast 8 core server, with a conventional disk drive. Took 2-3 hours, not including d/l. I update my laptop which has an ssd (and MORE packages). Took 10-15 minutes. I think this might simply have to do with rpm running ldconfig (a very disk IO expensive operation) for a large number of packages. I'm not sure yum/rpm has deferred ldconfig processing. DISCLAIMER: I may be very wrong. Please don't quote me on this. ldconfig gets run a lot yes, but its also really fast these days. fdatasync() which gets called even more (a lot more at that) seems like a more likely painpoint on upgrades. Oh and here are some numbers for your entertainment. This is a 185 core package install to empty chroot on my laptop with a conventional disk, with the two worst script-offenders (kernel and selinux-policy-targeted have) taken out of the picture as they'd very much dominate the running time on a set this small: fdatasync, no scripts 1m16s fdatasync, scripts 1m29s no fdatasync, no scripts 16s no fdatasync, scripts 25s When fdatasync() is disabled (on initial install where there's no data to lose), sure all the scripts start taking a considerable portion of the running time. But for normal operation (such as upgrades), fdatasync() is where the vast majority of time gets spent. Of course on real-world upgrades there are many many more things at play than in the simple test-case above, but to improve performance you need to figure out where the time is getting spent, guessing gets you nowhere. - Panu - - Panu - -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: fedup performance
On 07/03/2013 03:23 AM, Panu Matilainen wrote: On 07/03/2013 09:59 AM, Panu Matilainen wrote: On 07/03/2013 07:42 AM, Alex G. wrote: On 07/02/2013 08:28 PM, Neal Becker wrote: Not d/l speed related. I just want to share. I update a very fast 8 core server, with a conventional disk drive. Took 2-3 hours, not including d/l. I update my laptop which has an ssd (and MORE packages). Took 10-15 minutes. I think this might simply have to do with rpm running ldconfig (a very disk IO expensive operation) for a large number of packages. I'm not sure yum/rpm has deferred ldconfig processing. DISCLAIMER: I may be very wrong. Please don't quote me on this. ldconfig gets run a lot yes, but its also really fast these days. fdatasync() which gets called even more (a lot more at that) seems like a more likely painpoint on upgrades. Oh and here are some numbers for your entertainment. This is a 185 core package install to empty chroot on my laptop with a conventional disk, with the two worst script-offenders (kernel and selinux-policy-targeted have) taken out of the picture as they'd very much dominate the running time on a set this small: fdatasync, no scripts 1m16s fdatasync, scripts 1m29s no fdatasync, no scripts 16s no fdatasync, scripts 25s When fdatasync() is disabled (on initial install where there's no data to lose), sure all the scripts start taking a considerable portion of the running time. But for normal operation (such as upgrades), fdatasync() is where the vast majority of time gets spent. Of course on real-world upgrades there are many many more things at play than in the simple test-case above, but to improve performance you need to figure out where the time is getting spent, guessing gets you nowhere. Guessing gets other people to research the matter. A great way to get others to work for you for free. :p Alex - Panu - - Panu - -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: fedup performance
On 07/03/2013 11:29 AM, Alex G. wrote: On 07/03/2013 03:23 AM, Panu Matilainen wrote: On 07/03/2013 09:59 AM, Panu Matilainen wrote: On 07/03/2013 07:42 AM, Alex G. wrote: On 07/02/2013 08:28 PM, Neal Becker wrote: Not d/l speed related. I just want to share. I update a very fast 8 core server, with a conventional disk drive. Took 2-3 hours, not including d/l. I update my laptop which has an ssd (and MORE packages). Took 10-15 minutes. I think this might simply have to do with rpm running ldconfig (a very disk IO expensive operation) for a large number of packages. I'm not sure yum/rpm has deferred ldconfig processing. DISCLAIMER: I may be very wrong. Please don't quote me on this. ldconfig gets run a lot yes, but its also really fast these days. fdatasync() which gets called even more (a lot more at that) seems like a more likely painpoint on upgrades. Oh and here are some numbers for your entertainment. This is a 185 core package install to empty chroot on my laptop with a conventional disk, with the two worst script-offenders (kernel and selinux-policy-targeted have) taken out of the picture as they'd very much dominate the running time on a set this small: fdatasync, no scripts 1m16s fdatasync, scripts 1m29s no fdatasync, no scripts 16s no fdatasync, scripts 25s When fdatasync() is disabled (on initial install where there's no data to lose), sure all the scripts start taking a considerable portion of the running time. But for normal operation (such as upgrades), fdatasync() is where the vast majority of time gets spent. Of course on real-world upgrades there are many many more things at play than in the simple test-case above, but to improve performance you need to figure out where the time is getting spent, guessing gets you nowhere. Guessing gets other people to research the matter. A great way to get others to work for you for free. :p Sometimes maybe, but it can also be a great way to irritate people who have done their research a long time ago. - Panu - -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: fedup performance
On Wed, Jul 03, 2013 at 12:25:19AM -0500, Alex G. wrote: aptitude has something called deferred ldconfig processing, and annoyingly, aptitude updates faster than yum. I've always wondered how yum/rpm can be smartized to speed things up this way. But this discussion is for a brighter day. Well, this is how dpkg works. It downloads all packages, then unpacks it on filesystem and then configures them all, which means running post-scripts. Personally, I dislike this design... -- Regards,-- Sir Raorn. --- http://thousandsofhate.blogspot.com/ signature.asc Description: Digital signature -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Fedup Performance
I am doing an upgrade from F18 to F19 using fedup network and performance is very slow - 3+ hours on a 20 Mbit connection. There seems to be about a 10-15 second delay between package downloads. Is there a reason for this delay? -- Mark Bidewell http://www.linkedin.com/in/markbidewell -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Fedup Performance
On 2013-07-02 15:00, Mark Bidewell wrote: I am doing an upgrade from F18 to F19 using fedup network and performance is very slow - 3+ hours on a 20 Mbit connection. There seems to be about a 10-15 second delay between package downloads. Is there a reason for this delay? Well, it's *release day*. Tends to be a busy time, mirror-wise. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Fedup Performance
On Tue, 2013-07-02 at 18:00 -0400, Mark Bidewell wrote: I am doing an upgrade from F18 to F19 using fedup network and performance is very slow - 3+ hours on a 20 Mbit connection. There seems to be about a 10-15 second delay between package downloads. Is there a reason for this delay? Actually, I sort of knew that she had something to do with food, had diabetes, and was on TV from the covers of women's magazines on display at the checkout counter at the Rite Aid. That was pretty much it. When the story broke, however, I did some research. Her brother maintains separate entrances and bathrooms for white and black employees? She must think she's shooting craps, but those look like hand grenades to me. I caught a couple of minutes of her cracker act, and act it certainly is. She brought this all on her head. And the repercussions! Companies are still dumping her, weeks later. What a scam she was running! I fully expect to hear that Alice Chalmers and Anaconda Copper will be the next to drop her. -- Herbert Rutledge, aka Train | To post is human; to flame, divine. Reply-to: tr...@voicenet.com | ---Alexander Pope -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: Fedup Performance
On Tue, 2013-07-02 at 18:13 -0400, Herbert Rutledge wrote: Actually, I sort of knew that she had something to do with food, had diabetes, and was on TV from the covers of women's magazines on display at the checkout counter at the Rite Aid. That was pretty much it. A thousand apologies. I accidentally replied to the wrong message. I haven't done that in nearly twenty years. To get back on topic, I actually used fedora-upgrade. It was reasonably fast and the upgrade went off without a hitch. -- Herbert Rutledge, aka Train | To post is human; to flame, divine. Reply-to: tr...@voicenet.com | ---Alexander Pope -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
fedup performance
Not d/l speed related. I just want to share. I update a very fast 8 core server, with a conventional disk drive. Took 2-3 hours, not including d/l. I update my laptop which has an ssd (and MORE packages). Took 10-15 minutes. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: fedup performance
On 07/02/2013 08:28 PM, Neal Becker wrote: Not d/l speed related. I just want to share. I update a very fast 8 core server, with a conventional disk drive. Took 2-3 hours, not including d/l. I update my laptop which has an ssd (and MORE packages). Took 10-15 minutes. I think this might simply have to do with rpm running ldconfig (a very disk IO expensive operation) for a large number of packages. I'm not sure yum/rpm has deferred ldconfig processing. DISCLAIMER: I may be very wrong. Please don't quote me on this. Alex -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: fedup performance
On 2013-07-02 21:42, Alex G. wrote: On 07/02/2013 08:28 PM, Neal Becker wrote: Not d/l speed related. I just want to share. I update a very fast 8 core server, with a conventional disk drive. Took 2-3 hours, not including d/l. I update my laptop which has an ssd (and MORE packages). Took 10-15 minutes. I think this might simply have to do with rpm running ldconfig (a very disk IO expensive operation) for a large number of packages. I'm not sure yum/rpm has deferred ldconfig processing. rpm has the concept of %posttrans . Stuff in %posttrans is run after *the entire transaction* has completed, not after *the specific package install* has completed. However, I think we can't put ldconfig in %posttrans, because what happens if a package installed later relies on the ldconfig being correct for a package installed earlier in the same transaction? DISCLAIMER: I may be very wrong. Please don't quote me on this. Seconded =) -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: fedup performance
On 07/03/2013 12:15 AM, Adam Williamson wrote: On 2013-07-02 21:42, Alex G. wrote: On 07/02/2013 08:28 PM, Neal Becker wrote: Not d/l speed related. I just want to share. I update a very fast 8 core server, with a conventional disk drive. Took 2-3 hours, not including d/l. I update my laptop which has an ssd (and MORE packages). Took 10-15 minutes. I think this might simply have to do with rpm running ldconfig (a very disk IO expensive operation) for a large number of packages. I'm not sure yum/rpm has deferred ldconfig processing. rpm has the concept of %posttrans . Stuff in %posttrans is run after *the entire transaction* has completed, not after *the specific package install* has completed. However, I think we can't put ldconfig in %posttrans, because what happens if a package installed later relies on the ldconfig being correct for a package installed earlier in the same transaction? aptitude has something called deferred ldconfig processing, and annoyingly, aptitude updates faster than yum. I've always wondered how yum/rpm can be smartized to speed things up this way. But this discussion is for a brighter day. DISCLAIMER: I may be very wrong. Please don't quote me on this. Seconded =) My name is Alexandru Gagniuc, and I approve this message. :) Alex -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: fedup performance
On 2013-07-02 22:25, Alex G. wrote: On 07/03/2013 12:15 AM, Adam Williamson wrote: On 2013-07-02 21:42, Alex G. wrote: On 07/02/2013 08:28 PM, Neal Becker wrote: Not d/l speed related. I just want to share. I update a very fast 8 core server, with a conventional disk drive. Took 2-3 hours, not including d/l. I update my laptop which has an ssd (and MORE packages). Took 10-15 minutes. I think this might simply have to do with rpm running ldconfig (a very disk IO expensive operation) for a large number of packages. I'm not sure yum/rpm has deferred ldconfig processing. rpm has the concept of %posttrans . Stuff in %posttrans is run after *the entire transaction* has completed, not after *the specific package install* has completed. However, I think we can't put ldconfig in %posttrans, because what happens if a package installed later relies on the ldconfig being correct for a package installed earlier in the same transaction? aptitude has something called deferred ldconfig processing, and annoyingly, aptitude updates faster than yum. *mumble mumble* correlation *mumble* causation *mumble* -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
