Re: [Distutils] extends-cache
Rafael Monnerat wrote: I found that the file http://url/to/sample1.cfg is downloaded twice in the same run when I use buildout or bootstrap.py, even if the file is already present into the extends-cache. This is probably because the code that downloads that config file uses the cache in fall-back mode, i.e. it always tries to get a fresh version of the file (and update the cache with it) and uses a cached copy only when offline. 1) Is there a way to avoid download twice? Since the download utility doesn't know about the needs of the code calling it (which is a good thing in terms of orthogonality), the part of the buildout code which recursively evaluates the config files that extend each other would have to be changed to remember which files it already downloaded. It could then either reuse them instead of calling the download utility again, or use the download utility's regular cache mode for any further downloads of the same file. PS.: If it looks like a bug, I'm volunteer to fix it. If you implement the above (or something better) on a branch, I'd be happy to review it. BTW, this raises the question whether the fall-back mode of the cache should be made selectable with each call to a download utility. Currently, fall-back mode can only be configured when first instantiating a download utility (or by overriding its fallback attribute later, which would be ugly as a usage pattern). -- Thomas ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] zc.buildout: download utility vs. local resources
Hi all, I'm currently second-guessing a detail of the zc.buildout's download utility I wrote last year (*) and I want to ask for others' opinions. What I question is how the settings for cache usage and download destination end up determining whether the copy of the resource that client code receives is private to the buildout, or shared. The current logic involves an optimisation that tries to avoid copying files in the file system by creating hard links instead if possible. (**) I'd like to make the decision about using a private or shared copy more explicit and foreseeable. The obvious way to do this is adding another keyword parameter to the download call that expresses whether hard-linking should be attempted. I'm however not at all clear on what would be a sensible default value: Using a shared copy may not be desirable if client code is going to modify the file after downloading, so attempting to create hard links by default is the more dangerous behaviour. Always copying, on the other hand, means that when using a cache (or downloading a file-system resource) every download creates multiple copies of potentially large files in the file system, which is unnecessary in what I think is the majority of cases. It would be very helpful if someone could offer some input. Thank you. Thomas (*) The download utility is an API inside zc.buildout that can be used by recipes or zc.buildout itself to download HTTP or file-system resources. It can be configured to use a download cache and to put the downloaded resource in a particular place. In any case, it returns the file-system path of that copy of the downloaded resource which is to be used by the client code. (**) If an HTTP resource is downloaded, * having neither a cache nor a download destination puts the resource at a temporary path unique to the current download call, * using a cache but no download destination results in accessing the (shared) file inside the cache, * if using a download destination but no cache, a private copy of the resource is put in the destination, * if using both a cache and a download destination, the utility tries to hard-link the cached file to the destination and failing that, copies it, so the result may be either a private copy or a shared one. If the resource comes from the file system, * using no download destination results in shared access to either the original or cached resource, * specifying a download destination results in accessing either a shared or private copy of the resource depending on whether hard links from its original or cached path to the download destination are possible. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] extends-cache
Hi, I posted the bugs (I think are 2 diferent) into launchpad with my patches which I made on friday: - https://bugs.launchpad.net/zc.buildout/+bug/717730 - https://bugs.launchpad.net/zc.buildout/+bug/717802 You are right about fall-back, I used the ugly way to not change download utility API. Feel free to comment, I can update my patch. Regards, Rafael Monnerat On 13-02-2011 08:05, Thomas Lotze wrote: Rafael Monnerat wrote: I found that the file http://url/to/sample1.cfg is downloaded twice in the same run when I use buildout or bootstrap.py, even if the file is already present into the extends-cache. This is probably because the code that downloads that config file uses the cache in fall-back mode, i.e. it always tries to get a fresh version of the file (and update the cache with it) and uses a cached copy only when offline. 1) Is there a way to avoid download twice? Since the download utility doesn't know about the needs of the code calling it (which is a good thing in terms of orthogonality), the part of the buildout code which recursively evaluates the config files that extend each other would have to be changed to remember which files it already downloaded. It could then either reuse them instead of calling the download utility again, or use the download utility's regular cache mode for any further downloads of the same file. PS.: If it looks like a bug, I'm volunteer to fix it. If you implement the above (or something better) on a branch, I'd be happy to review it. BTW, this raises the question whether the fall-back mode of the cache should be made selectable with each call to a download utility. Currently, fall-back mode can only be configured when first instantiating a download utility (or by overriding its fallback attribute later, which would be ugly as a usage pattern). ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Testing modules from PyPI
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 02/12/2011 08:19 AM, cool-RR wrote: What would be a good workflow to do post-release testing of my PyPI packages? That is, I make a new release on PyPI, and I want to install it on a VM and run the tests on the installed version. What would be a good way to do it? I use virtualenv, something like this:: $ export WHERE=/tmp/test-yourpackage $ /path/to/python/bin/virtualenv --no-site-packages $WHERE $ $WHERE/bin/easy_install nose coverage $ $WHERE/easy_install --always-unzip yourpackage $ export TARGET=$WHERE/lib/python2.x/site-packages $ $WHERE/bin/nosetests --where=$TARGET/yourpackage-x.y-python2.x.egg Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk1Yn68ACgkQ+gerLs4ltQ6iVQCg0SxbRTQMUMRJA9frxpZqqa6i IWsAn0NyCeQDQs4sGupc/M7OKL6OeIzr =kA8g -END PGP SIGNATURE- ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] extends-cache
Rafael Monnerat wrote: I posted the bugs (I think are 2 diferent) into launchpad with my patches which I made on friday: Thank you. - https://bugs.launchpad.net/zc.buildout/+bug/717730 As far as the cache issue is concerned, this patch looks fine by me, but of course, there should be a test for the changed behaviour. OTOH, the patch reminds me of a different issue which I've been running into with buildout a couple of times now: there should be an API somewhere in zc.buildout that helps resolve string values from config files into boolean values. I'll remember to bring this up as a separate issue. - https://bugs.launchpad.net/zc.buildout/+bug/717802 * A test for the changed behaviour would be in order in this case as well. * A set might be a more appropriate data structure than a list for collecting the file names seen. * The empty set of file names should be passed into the first call to _open to avoid a mutable object as the default value of downloaded_list. * The whole logic of the nested if-else blocks might be refactored to get rid of code duplication (which has been there all along and becomes even more apparent by your patch). Maybe this also makes the appropriate fallback value known at the time of instantiating the download utility. -- Thomas ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
