Re: [Distutils] pkg_resources get_distribution - zipfile support?
On Tue, Oct 1, 2013 at 7:10 PM, Nick Coghlan wrote: > > On 2 Oct 2013 07:12, "Paul Moore" wrote: >> >> On 1 October 2013 21:32, PJ Eby wrote: >> > On Tue, Oct 1, 2013 at 1:51 PM, Daniel Holth wrote: >> >> pkg_resources only finds distributions inside .egg and inside sys.path >> >> entries that are filesystem directories. >> > >> > Actually it looks in zipfiles (or subdirectory paths thereof) or >> > filesystem directories, and can spot zipfile subdirectories named >> > '.egg' inside zipfiles (or subdirectories thereof). >> >> But not dist-info? I thought setuptools supported dist-info formats >> these days. Is this somewhere that got missed? Or is it more subtle >> than that? > > I believe it's not recognising "*.whl" as interesting, so it isn't even > looking inside the nested wheel to check for dist-info. That would only be relevant if the .whl weren't on sys.path. Since it's on sys.path, it's processed by importer, not by filename. (It's just that there's no .dist-info detection in the zipimporter handler.) The .whl extension is only relevant for discovery of nested .whl's (wheels within wheels... or within eggs or plain zipfiles...), or .whl's in directories. There isn't any good terminology for describing this at the moment, which makes all this sound much more complicated than it actually is. ;-) Making up some new terminology, suppose we have foo.egg, bar.egg-info, baz.dist-info, and spam.whl in site-packages. Then the bar and baz distributions are "mounted" at the '.../site-packages' location string, but the foo and spam distributions are merely "discoverable" at that same location string. (They are *mounted* at '.../site-packages/foo.egg' and '.../site-packages/spam.whl', respectively.) That is, to be mounted at a given location string means "to be importable if that location string is on sys.path", and to be discoverable at a given location means "to be available for dynamic dependency resolution (e.g. via require()) if that location string is on sys.path". Determining what is mounted or discoverable at a given sys.path location is the job of the find_distributions() API. If the 'only' flag is true, it yields only mounted distributions at the given location string. If false (the default), it yields both mounted and discoverable distributions. Behind the scenes, this is implemented by finding a handler for the importer that the PEP 302 import protocol would use to look for modules at the given location string, and then delegating the operation to that handler. The handler then has to look at the location string and figure out what distributions are mounted and/or discoverable there. To find mounted distributions, the directory handler (find_on_path()) checks whether the directory string itself ends in '.egg' (and could theoretically do the same for .whl), and also looks for contained .dist-info and .egg-info files or directories. To find mountable distributions, it checks for files or directories ending in '.egg' (and could theoretically do the same for .whl). The zipfile handler (find_in_zip()) doesn't actually bother checking for an .egg extension; instead it checks for an EGG-INFO/PKG-INFO and assumes it'll be able to figure things out from that. And it checks for nested .eggs if it's looking for discoverables. So, what it's missing to support Paul's use case is a check for .dist-info/METADATA, analagous to the EGG-INFO/PKG-INFO check. It should be relatively simple to add, if somebody wants to do that. (It can even be done from code outside pkg_resources, as there is a 'register_finder()' API that can be called to register a replacement handler.) In some ways, I'm finding the code structure irritating now, because the one abstraction I *didn't* build into the original framework was a concept that there would ever be competing formats to .egg and .egg-info, so implementing .dist-info and .whl requires annoying repetitions of code at the moment. But it's probably not worth refactoring to make this cleaner, because the odds that there will be a *third* file format needing to be supported any time soon are hopefully quite small. ;-) ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] File names with spaces
Hello setuptools developers, I'm attempting to package the newest setuptools package on Solaris 9 and 10. One of the limitations of the Solaris package manager (the old one, pkgadd/pkgrm), is that it is unable to handle file names with spaces. Would you mind renaming "script template.py" to something like "script_template.py"? Regards, Maciej ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] File names with spaces
Hello setuptools developers, I'm attempting to package the newest setuptools version (1.1.6) on Solaris 9 and 10. One of the limitations of the Solaris package manager (the old one, pkgadd/pkgrm), is that it is unable to handle file names with spaces. Would you mind renaming "script template.py" to something like "script_template.py"? Maciej ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pkg_resources get_distribution - zipfile support?
On Tue, Oct 1, 2013 at 5:11 PM, Paul Moore wrote:
> On 1 October 2013 21:32, PJ Eby wrote:
>> On Tue, Oct 1, 2013 at 1:51 PM, Daniel Holth wrote:
>>> pkg_resources only finds distributions inside .egg and inside sys.path
>>> entries that are filesystem directories.
>>
>> Actually it looks in zipfiles (or subdirectory paths thereof) or
>> filesystem directories, and can spot zipfile subdirectories named
>> '.egg' inside zipfiles (or subdirectories thereof).
>
> But not dist-info? I thought setuptools supported dist-info formats
> these days. Is this somewhere that got missed? Or is it more subtle
> than that?
The person or persons who implemented dist-info either didn't know
where to put in the support, or didn't consider it a priority to also
support zipped dist-info.
A cursory glance at the DistinfoDistribution class, however, suggests
that it should work fine as long as it's paired with an appropriate
metadata provider. Something like:
class WheelMetadata(EggMetadata):
def _setup_prefix(self):
# like EggProvider._setup_prefix, except w/".whl" and
# ".dist-info" instead of ".egg" and "EGG-INFO"
# ... or refactor EggProvider to parameterize these as
# class attrs and override here
and then make find_in_zip look more like:
def find_in_zip(importer, path_item, only=False):
for (meta_factory, fn, dist_factory) in [
(EggMetadata, 'PKG-INFO', Distribution),
(WheelMetadata, 'METADATA', DistInfoDistribution),
]:
metadata = meta_factory(importer)
if metadata.has_metadata(fn):
yield dist_factory.from_filename(path_item, metadata=metadata)
if only:
return # don't yield nested distros
for subitem in metadata.resource_listdir('/'):
if subitem.endswith('.egg') or subitem.endswith('.whl'):
subpath = os.path.join(path_item, subitem)
for dist in find_in_zip(zipimport.zipimporter(subpath), subpath):
yield dist
So it's not a huge deal to implement, just a bit of tedium. Under the
hood, there's little difference between wheels and eggs besides naming
conventions.
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pkg_resources get_distribution - zipfile support?
On 2 Oct 2013 07:12, "Paul Moore" wrote: > > On 1 October 2013 21:32, PJ Eby wrote: > > On Tue, Oct 1, 2013 at 1:51 PM, Daniel Holth wrote: > >> pkg_resources only finds distributions inside .egg and inside sys.path > >> entries that are filesystem directories. > > > > Actually it looks in zipfiles (or subdirectory paths thereof) or > > filesystem directories, and can spot zipfile subdirectories named > > '.egg' inside zipfiles (or subdirectories thereof). > > But not dist-info? I thought setuptools supported dist-info formats > these days. Is this somewhere that got missed? Or is it more subtle > than that? I believe it's not recognising "*.whl" as interesting, so it isn't even looking inside the nested wheel to check for dist-info. Cheers, Nick. > Paul > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pkg_resources get_distribution - zipfile support?
On 1 October 2013 21:32, PJ Eby wrote: > On Tue, Oct 1, 2013 at 1:51 PM, Daniel Holth wrote: >> pkg_resources only finds distributions inside .egg and inside sys.path >> entries that are filesystem directories. > > Actually it looks in zipfiles (or subdirectory paths thereof) or > filesystem directories, and can spot zipfile subdirectories named > '.egg' inside zipfiles (or subdirectories thereof). But not dist-info? I thought setuptools supported dist-info formats these days. Is this somewhere that got missed? Or is it more subtle than that? Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pkg_resources get_distribution - zipfile support?
On Tue, Oct 1, 2013 at 1:51 PM, Daniel Holth wrote:
> pkg_resources only finds distributions inside .egg and inside sys.path
> entries that are filesystem directories.
Actually it looks in zipfiles (or subdirectory paths thereof) or
filesystem directories, and can spot zipfile subdirectories named
'.egg' inside zipfiles (or subdirectories thereof).
It will also look in any other sort of sys.path entry, if you register
a handler for the importer type.
> You might be able to manually
> add a new Distribution instance to working_set or start looking for a
> place to add the feature here:
>
> https://bitbucket.org/pypa/setuptools/src/27df3c725f9696ba730456f3f444cc2fb5271d4b/pkg_resources.py?at=default#cl-2560
>
> _distributionImpl = {
> '.egg': Distribution,
> '.egg-info': Distribution,
> '.dist-info': DistInfoDistribution,
> }
>
> https://bitbucket.org/pypa/setuptools/src/27df3c725f9696ba730456f3f444cc2fb5271d4b/pkg_resources.py?at=default#cl-2219
>
> (Distribution.from_location())
Actually, you would add the feature *here*:
https://bitbucket.org/pypa/setuptools/src/27df3c725f9696ba730456f3f444cc2fb5271d4b/pkg_resources.py?at=default#cl-1810
That is, in the find_in_zip() function, which is used to identify
distributions contained in zip files (or subdirectories thereof) on
sys.path.
(This feature needs to get added at some point anyway, for
pkg_resources to support mountable wheels.)
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pkg_resources get_distribution - zipfile support?
On 1 October 2013 18:51, Daniel Holth wrote: > pkg_resources only finds distributions inside .egg and inside sys.path > entries that are filesystem directories. You might be able to manually > add a new Distribution instance to working_set or start looking for a > place to add the feature here: OK, I thought that might be the case. Context - you can't run pip/setuptools from uninstalled wheels to do pip.main(['install', '--use-wheel', '']) because --use-wheel checks for a suitable version of setuptools using find_distribution. So the check fails. This is for virtualenv bundling wheels rather than sdists for faster creation. Looks like I'll have to work out a patch for pip to allow disabling that check somehow, or modifying it to use setuptools.__version__ rather than find_distribution. Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pkg_resources get_distribution - zipfile support?
On Tue, Oct 1, 2013 at 12:59 PM, Paul Moore wrote:
> I'm having trouble using pkg_resources.get_distribution, and I'm not
> sure if it's because what I'm trying to do is unsupported (likely) or
> if I'm doing something wrong (also possible :-()
>
> I have a wheel file - essentially a zip file containing a
> distribution, plus a dist-info directory. If I put that zipfile on
> sys.path, I can import from it with no issues. However, if I try to do
> pkg_resources.get_distribution to find the distribution contained in
> that wheel/zipfile I get nothing back.
>
> Is this a bug, or an unsupported feature, or should I do something
> particular to make it work?
>
> Thanks,
> Paul
> ___
> Distutils-SIG maillist - Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
pkg_resources only finds distributions inside .egg and inside sys.path
entries that are filesystem directories. You might be able to manually
add a new Distribution instance to working_set or start looking for a
place to add the feature here:
https://bitbucket.org/pypa/setuptools/src/27df3c725f9696ba730456f3f444cc2fb5271d4b/pkg_resources.py?at=default#cl-2560
_distributionImpl = {
'.egg': Distribution,
'.egg-info': Distribution,
'.dist-info': DistInfoDistribution,
}
https://bitbucket.org/pypa/setuptools/src/27df3c725f9696ba730456f3f444cc2fb5271d4b/pkg_resources.py?at=default#cl-2219
(Distribution.from_location())
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] pkg_resources get_distribution - zipfile support?
I'm having trouble using pkg_resources.get_distribution, and I'm not sure if it's because what I'm trying to do is unsupported (likely) or if I'm doing something wrong (also possible :-() I have a wheel file - essentially a zip file containing a distribution, plus a dist-info directory. If I put that zipfile on sys.path, I can import from it with no issues. However, if I try to do pkg_resources.get_distribution to find the distribution contained in that wheel/zipfile I get nothing back. Is this a bug, or an unsupported feature, or should I do something particular to make it work? Thanks, Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
