Re: [Distutils] idea for Distribute: make unzipped eggs be the default
On Wed, Jul 29, 2009 at 9:51 AM, Lennart Regebro wrote: > 2009/7/29 Leonardo Santagada : >> I think P.J. Eby is right, we need to test it before analising the results >> we don't have. > > No we don't, because zipped eggs are such a pain that speed is > irrelevant in comparison. Those who *do* care about minor improvements > in startup speed can test. And then they can set their settings to > install eggs zipped by default. +1 for unzipped eggs by default. I've accepted the #33 in our tracker and I propose to fix it for 0.7 http://bitbucket.org/tarek/distribute/issue/19 +1 for creating a performance test. If we can improve the lookup speed of applications that heavily rely on eggs, that would be nice. I think a Plone-based application is a good sandbox for this test, because it uses hundreds of eggs. If someone wants to give it a shot. Cheers Tarek > > -- > Lennart Regebro: Python, Zope, Plone, Grok > http://regebro.wordpress.com/ > +33 661 58 14 64 > ___ > Distutils-SIG maillist - distutils-...@python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -- Tarek Ziadé | http://ziade.org ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] idea for Distribute: make unzipped eggs be the default
2009/7/29 Leonardo Santagada : > I think P.J. Eby is right, we need to test it before analising the results > we don't have. No we don't, because zipped eggs are such a pain that speed is irrelevant in comparison. Those who *do* care about minor improvements in startup speed can test. And then they can set their settings to install eggs zipped by default. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] idea for Distribute: make unzipped eggs be the default
On Jul 28, 2009, at 9:38 PM, David Lyon wrote: I think it might improve performance a bit. (Yes, that's right -- we haven't done a real measurement of performance, but the few times that people briefly glanced at performance it seemed like zipping the eggs made them slower to load, not faster.) It's totally true. unzipped is faster because decompression adds another processing layer which consumes more time than doing the dma transfer of the extra bytes. For those still using Z-80s or other 8-bit processors along with C or assembler this isn't the case, but it will be true for any 32- bit and above architecture running an elephant interpretor like python. I think P.J. Eby is right, we need to test it before analising the results we don't have. Anyway even with a minor decrease in performance using unzipped eggs makes things easier. -- Leonardo Santagada santagada at gmail.com ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] idea for Distribute: make unzipped eggs be the default
>> I think it might improve >> performance >> a bit. (Yes, that's right -- we haven't done a real measurement of >> performance, but the few times that people briefly glanced at performance >> it >> seemed like zipping the eggs made them slower to load, not faster.) It's totally true. unzipped is faster because decompression adds another processing layer which consumes more time than doing the dma transfer of the extra bytes. For those still using Z-80s or other 8-bit processors along with C or assembler this isn't the case, but it will be true for any 32- bit and above architecture running an elephant interpretor like python. +1 from me David ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] idea for Distribute: make unzipped eggs be the default
On Tue, Jul 28, 2009 at 10:23 AM, Zooko Wilcox-O'Hearn wrote: > Folks: > > I have been keeping a running tally of people who have trouble with zipped > eggs and the trouble goes away when they have unzipped eggs instead. I just > added the newest entry: someone was trying to build a package with py2exe, > and configuring their distutils config file to always unzip eggs fixed the > problem: > > http://bugs.python.org/setuptools/issue33 # zipped eggs cause various > problems -- perhaps set default setting to unzipped? > > Perhaps the Distribute project should consider installing eggs unzipped by > default. It would avoid a slew of problems (see that ticket for a catalog > of problems), would be fully backwards compatible (it is what many people > already do, per the suggested workaround, and I've never heard a report of a > problem due to an *unzipped* egg), and I think it might improve performance > a bit. (Yes, that's right -- we haven't done a real measurement of > performance, but the few times that people briefly glanced at performance it > seemed like zipping the eggs made them slower to load, not faster.) +1 Jim -- Jim Fulton ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] idea for Distribute: make unzipped eggs be the default
At 08:23 AM 7/28/2009 -0600, Zooko Wilcox-O'Hearn wrote: I think it might improve performance a bit. (Yes, that's right -- we haven't done a real measurement of performance, but the few times that people briefly glanced at performance it seemed like zipping the eggs made them slower to load, not faster.) You're trading startup time for import time, actually. Unzipped eggs force OS level stat calls during importing of every module, whereas zipped eggs force directory reads at startup. In both cases, the time cost mostly occurs at startup; the difference is in whether you have more imports or more eggs. Each zipped egg adds a one-time performance hit; each unzipped egg adds a smaller -- but *N-times* as often -- performance hit. So the optimum performance tradeoff depends on how many imports you have *and* how many eggs you have on sys.path. If you have lots of eggs and few imports, unzipped ones will probably be faster. If you have lots of eggs and *lots* of imports, zipped ones will probably be faster. To really know what the tradeoff is, some actual measurements are needed. (And they need to measure both startup overhead and import overhead.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] idea for Distribute: make unzipped eggs be the default
2009/7/28 Zooko Wilcox-O'Hearn : > Perhaps the Distribute project should consider installing eggs unzipped by > default. +1. Zipped packages are apparently only useful on specialized systems where the filesystem has limitations, maybe it's slow or can't handle many files, like embedded systems or similar. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] idea for Distribute: make unzipped eggs be the default
On Tue, Jul 28, 2009 at 4:23 PM, Zooko Wilcox-O'Hearn wrote: > http://bugs.python.org/setuptools/issue33 # zipped eggs cause various > problems -- perhaps set default setting to unzipped? +1 > Perhaps the Distribute project should consider installing eggs unzipped by > default. It would avoid a slew of problems (see that ticket for a catalog > of problems), would be fully backwards compatible (it is what many people > already do, per the suggested workaround, and I've never heard a report of a > problem due to an *unzipped* egg), and I think it might improve performance > a bit. (Yes, that's right -- we haven't done a real measurement of > performance, but the few times that people briefly glanced at performance it > seemed like zipping the eggs made them slower to load, not faster.) The measurements I know of suggest that having lots of zipped eggs for each package doesn't provide any benefit whatsoever and is usually worse than normal unzipped eggs. Only if you combine a whole lot of packages and code into one giant manually crafted egg, you sometimes get benefits from it. But that's a specific deployment optimization that should be left to specific situations. Hanno ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
