Re: [Distutils] platform_python_implementation not implemented.
On Thu, Sep 10, 2015 at 11:34 AM, Daniel Holth wrote: > That is too bad. markerlib was added to pkg_resources as _markerlib in 2012. > It is used for .dist-info metadata as present in wheel. Then, only to > implement markers to setup.py or .egg-info style metadata, pkg_resources > gains its inline markers implementation in 2013, including its own > definitions of the key/values used in environment marker comparisons. Dots > switched to underscores in revision aa74cf234684 in the inline > implementation and in revision 1fc8a2d94f9f for setuptools' vendored > _markerlib. To clarify: the Distribute fork of setuptools added _markerlib, but Distribute didn't support older version of Python as the official setuptools 0.6 did, which is why I added the inline version there. There was also active discussion at the time about changing the markers spec, as Nick was working on PEP 426. Previously, there were two other PEPs, 345 and 390, with their own not-quite compatible specs, and more recently there is now a PEP 496. So, there has never really been a stable standard for environment markers; all of the previous specs have had various ambiguities, underspecification, or excessive lenience. My hope at the time of the PEP 426 discussion was that we could define a *strict* grammar in place of a loose pseudo-grammar so that the spec would be robust to multiple implementations rather than forcing everyone to copy the quirks of one particular implementation (e.g. markerlib). Unfortunately, even PEP 496 is still a little underspecified: it doesn't even say what kinds of string literals are allowed, address encodings or character sets, etc. Is r"foo" a legal string expression? Can you use double-quotes? Backslash escaping? Adjacent string literal concatentation? But it's better than the previous versions, since the mini-language is no longer an ambiguously-defined subset of Python and thus can no longer be parsed using Python's built-in grammar, and probably not its lexer either. > I would prefer to see _markerlib._VARS used everywhere, the inline > pkg_resources implementation deleted, markerlib improved if necessary, and > no backwards compatibility with unspecified environment marker variables. My > hunch is that no one needs the backwards compatibility. It may have changed since when I added markers to the official setuptools, but I intentionally did not document the marker feature; it was experimental and added mainly to support setuptools' internal use case of including SSL backports on older Python versions to support easy_install checking SSL certs. So, anybody using it was (and perhaps is; I haven't looked lately) using an undocumented experimental feature rather than an officially supported one. In any case, if compatibility is broken, it should probably be done by switching to the much-stricter PEP 496, rather than one of its even-more-ambiguous predecessors. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Proper handling of PEP420 namespace packages with setuptools and pip
On Wed, Apr 22, 2015 at 5:20 PM, Robert Collins
wrote:
> Ah, ok so I think this is the crux - I'm arguing that Python version
> isn't a big enough check. Because anything installed with a current
> version of setuptools, or any wheel built likewise, is going to not
> have that per-Python-version check.
>
> And it seems to me that that implies that bringing in a
> per-Python-version check in a new release of setuptools or pip is
> going to result in mixed mode installs:
>
> install name.A with setuptools-X [legacy]
> upgrade setuptools
> install name.B with setuptools-Y [does a Python version check]
> -> boom
>
> But perhaps sufficient glue can be written to make it all work.
When I wrote PEP 402 (the precursor to 420), the idea was that in a
mixed environment you would need to:
1. Change pkg_resources' namespace system to support non-__init__
directories (and likewise pkgutil.extend_path)
2. Change easy_install's .pth generation magic to not do any magic or
import a PEP 420 emulation module on old systems
3. Change package building tools to stop injecting __init__.py files
This basically solves the mixed installation problem because if you
have an __init__.py that uses existing magic, the empty dirs get
folded in. You basically have a transitional state with mixed
__init__ and non-__init__ stuff. If there happens to be an
__init__.py, then as long as it declares the namespace then the local
*runtime* takes care of making the runtime environment work. The
*installation* tools don't have to manage mixed modes, they should
just blindly install whatever package they have, and over the long
term the packages all end up shipped without __init__.py's, but the
__init__.py approach will continue to work basically forever.
> My personal preferred migration strategy is:
> - have a flag day amongst the cooperating packages that make up the namespace
> - release versions that are all in the new layout in a batch to PyPI.
>
> It would be nice if PEP-426 had a conflicts stanza, so you could say
> conflicts: [name.A < version_with_new_X] without that implying that
> name.A *should* be installed.
This is all *really* unnecessary. Setuptools has essentially *always*
built non-egg, non-exe binary distributions in a PEP 420-compatible
way (i.e., without __init__.py). And pkg_resources already builds a
namespace path by asking importers if they can import a package at
that location. So if PEP 420 importers say "yes" when asked to
find_module('somepkg') in the case of an __init__-less subdirectory
named 'somepkg', then pkg_resources *already* supports mixed-mode
installations under PEP 420, and doesn't even need to be updated!
I haven't checked whether that's the case, but if it is, then the only
thing that setuptools neds to change is its .pth generation magic, to
not do the magic if it's on a PEP 420 platform at runtime, and to stop
including __init__.py's for namespace packages.
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Installing namespace packages with pip inserts strange modules into sys.modules
On Fri, Sep 12, 2014 at 3:55 PM, Erik Bray wrote: > Unfortunately there are some dark corners of setuptools I've > encountered where namespace packages don't work properly during > installation *unless* they were installed in the old-fashioned > setuptools way. I'll have to see if I can dig up what those cases > are, because they should be fixed. I don't know if this is what you had in mind, but, the main problem I know of with changing setuptools to not install the .pth files, is that if you already have any __init__.py's installed (and possibly, any namespace .pth files to go with them), then the newly-installed package isn't going to work. PEP 420 only takes effect if there are no existing __init__.py files for the namespace. In other words, it's not just a matter of changing how things are installed, it's also a matter of upgrading existing environments with things installed by older installation tools. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 470 Round 2 - Using Multi Index Support for External to PyPI Package File Hosting
On Fri, Jun 6, 2014 at 10:25 AM, Donald Stufft wrote: > I expected more people to move to safe external vs staying with the unsafe > external. > Is there a tool that makes this *easy*? I'm not aware of one. (Ideally, something like a replacement for setup.py upload that generates the download URLs and sends them off to PyPI, so that all one needs is a setup_requires for the tool, a setup.cfg with the hosting prefix, and a run of "setup.py register bdist_whatever uplink" to get the links set up.) ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What is the syntax for passing conditional non-extra dependencies in setuptools?
On Mon, Apr 7, 2014 at 4:03 PM, Daniel Holth wrote:
> OK, it does in fact work that way; empty-string-before-colon specifies
> a default requirement with a marker.
>
> Parses out to the following _dep_map with 'None' representing 'not an
> extra dependency':
>
> {None: [Requirement.parse('coding'), Requirement.parse('stuff')],
> 'quux': [Requirement.parse('more-stuff')]}
>
> requirements.txt:
>
> coding
>
> [:sys_platform=='linux2']
> stuff
>
> [quux:sys_platform=='linux2']
> more_stuff
>
>
Huh. Either I did that intentionally and forgot about it, or it just fell
out as a brilliant accidental side effect of the orthogonal design of the
parsing functions. Either way, I feel like I did something smart there.
;-)
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What is the syntax for passing conditional non-extra dependencies in setuptools?
On Wed, Mar 26, 2014 at 11:29 PM, Daniel Holth wrote:
> How do I specify a conditional (marker-guarded) non-extra dependency
> in setuptools? The syntax for a conditional extra dependency is
> currently:
>
> extras_require = {
> "ssl:sys_platform=='win32'": "wincertstore==0.2",
> "certs": "certifi==1.0.1",
> },
>
I only implemented support via extras, and the feature wasn't officially
supported (still isn't, I don't think) because the PEP specifying the
syntax wasn't fully baked yet. I figured that if *only* setuptools itself
used it, then if the syntax changed only setuptools would break... but fix
itself at the same time.
The same cannot be said for any other package, so use at your own risk. Or
better yet, don't use it. ;-)
(At least, not until it's a documented feature w/a PEP-approved syntax.)
___
> Distutils-SIG maillist - Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Import-SIG] nspkg.pth files break $PYTHONPATH overrides
On Wed, Mar 26, 2014 at 11:55 PM, Eric Snow wrote: > In 3.4 it's called _NamespaceLoader, but in 3.3 it's NamespaceLoader. > > Ouch. That is going to be a really *long* bit of code. Not like it isn't already, though. By "available on the platform" do you mean "Python 3.3+ or has a > backport installed"? Otherwise you'd just check sys.version*. > I just hate doing version checks when a feature check ought to be possible. You never know when it's going to cause trouble. But yes, certainly being able to handle the backport case would be nice. When I wrote PEP 402, I was thinking in terms of transitioning the .pth files to importing a compatibility module. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] nspkg.pth files break $PYTHONPATH overrides
On Tue, Mar 25, 2014 at 3:50 PM, Barry Warsaw wrote:
> On Mar 25, 2014, at 03:35 PM, PJ Eby wrote:
>
> >I think the correct fix would be to change the nspkg.pth magic to check
> for
> >PEP 420 support, but unfortunately it seems we may have to use version
> >checking: on rereading PEP 420 I see there's no 'sys.namespace_packages'
> or
> >similar object that can be directly checked for feature support. :-(
>
> There is. It's *pronounced* sys.namespace_packages, but it's spelled
> importlib._bootstrap._NamespaceLoader ;)
>
Yeah, well that's not exactly a public attribute, so it's not necessarily a
great way to do it. But if we did use that, presumably it'd add something
like:
hnp =
hasattr(sys.modules.get('importlib._bootstrap',None),'_NamespaceLoader');
To the front of the magic, and then prefix all subsequent expression values
with 'not hnp and' in order to prevent them executing if PEP 420 support is
available. (Note: it's checking sys.modules since on any interpreter where
PEP 420 is natively available, the module should *already* be loaded by the
time site.py does its thing.)
And we should probably think about adding sys.namespace_packages or
something of the sort, or at least a proper flag for whether PEP 420
support is available on the platform.
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] nspkg.pth files break $PYTHONPATH overrides
On Mon, Mar 24, 2014 at 6:09 PM, Barry Warsaw wrote: > On Mar 24, 2014, at 05:53 PM, Donald Stufft wrote: > > >See also https://github.com/pypa/pip/issues/3 > > Hah, yeah. I didn't realize --single-version-externally-managed is > implied by > --root, and in fact our Debian build scripts often (either explicitly or > implicitly) define --root, as is the case in lazr.uri. > > >Basically prior to PEP420 namespace packages were bad and using them > results > >in pain sooner or later :( I'm not sure if a good solution yet, perhaps we > >can backport PEP420 to PyPI and have namespace packages depend on that? > > Or maybe do a version check before installing this file? Python 3.3 and > newer > will have PEP 420 support, and this file makes no sense in that case. > That won't *quite* work, because the .pth files are generated for other types of direct-install distributions. I think the correct fix would be to change the nspkg.pth magic to check for PEP 420 support, but unfortunately it seems we may have to use version checking: on rereading PEP 420 I see there's no 'sys.namespace_packages' or similar object that can be directly checked for feature support. :-( ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] setuptools tracker
On Mon, Mar 24, 2014 at 8:46 AM, Nick Coghlan wrote: > On 24 March 2014 20:53, "Martin v. Löwis" wrote: > > Both problems would be resolved by setting the tracker to read-only; > > shutting it down is actually not necessary (although it would slightly > > reduce our maintenance efforts). > > That sounds good to me. > > I've also filed https://bitbucket.org/pypa/setuptools/issue/174/ to > decide on a longer term solution. If Jason decides to review/migrate > issues, it may be necessary to turn developer write access back on to > allow issues to be marked as closed once they have been dealt with > appropriately. > Yep, looks like Jason came to the same conclusion(s) independently, but also wants better banners on the old tracker to alert people to the move. I guess we should move any further discussion to that ticket, since Jason's response time is quicker there than here. ;-) ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] setuptools tracker
I think it would be a good idea to check with Jason and other PyPA volunteers to see if there is anyone else who can handle the moves. I'd prefer we didn't lose the history, since my comments on those issues (and the closed ones, too) often contain key information about use cases and design decisions that may not be available elsewhere, even from my memory. ;-) But, since I'm no longer in the lead on development, I think it would be better for someone closer to the future of things to do the prioritizing of what, if anything, to transfer as an issue or keep as documentation. On Sun, Mar 23, 2014 at 2:00 PM, "Martin v. Löwis" wrote: > Am 23.03.14 18:30, schrieb PJ Eby: > > On Sun, Mar 23, 2014 at 4:55 AM, "Martin v. Löwis" > <mailto:mar...@v.loewis.de>> wrote: > > > > We are still hosting a roundup installation for setuptools, > > at http://bugs.python.org/setuptools/. > > > > Is this still needed? If not: what should we do with it? > > > > > > I think probably the remaining issues need to be moved to Bitbucket > > (unless they're already addressed in later setuptools versions), and the > > tracker closed. At this point, I think it's safe to say that the 0.6 > > line isn't getting any more changes; persons and organizations using > > older versions of Python will have to take 0.6 as it is, or upgrade. > > Would you volunteer to move them? Alternatively, I could close them all > with an automatic message saying that they should re-report them if the > issue still exists. > > Regards, > Martin > > > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] setuptools tracker
On Sun, Mar 23, 2014 at 4:55 AM, "Martin v. Löwis" wrote: > We are still hosting a roundup installation for setuptools, > at http://bugs.python.org/setuptools/. > > Is this still needed? If not: what should we do with it? > > I think probably the remaining issues need to be moved to Bitbucket (unless they're already addressed in later setuptools versions), and the tracker closed. At this point, I think it's safe to say that the 0.6 line isn't getting any more changes; persons and organizations using older versions of Python will have to take 0.6 as it is, or upgrade. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Redirects for old distribute documentation?
I was browsing through some docs on a database API for Python today when I noticed that they had a link to http://pythonhosted.org/distribute/easy_install.html -- which is defunct, along with most everything else under pythonhosted.org/distribute/. Perhaps it would be a good idea for the missing pages to redirect to some active documentation instead? ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Is there any sense to put setuptools as a requirement?
On Sat, Mar 1, 2014 at 4:47 PM, Jim Fulton wrote: > On Thu, Feb 27, 2014 at 6:49 AM, Piotr Dobrogost > wrote: > > Hi! > > > > I've seen people putting 'setuptools' in 'install_requires' in > > setup.py starting with import from setuptools like this: > > from setuptools import setup, find_packages > > > > Does it make any sense? > > In what circumstances should 'setuptools' be listed in > > 'install_requires' or 'setup_requires'? > > It makes sense when you use setuptools to implement namespace packages. > > So, for example in zope packages, the __init__.py in the zope namespace > package typically has: > > import pkg_resources > pkg_resources.declare_namespace(__name__) > It also makes sense if you use entry points, the plugin finder, or any other pkg_resources features, for that matter. ;-) Or if you reuse setuptools functionality (e.g. easy_install) to install plugins for your app. But in general, it only makes sense to depend on setuptools if your actual project contents (aside from setup.py) are importing things from pkg_resources or setuptools. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Setuptools 3.0b1 available for preview
On Wed, Feb 12, 2014 at 11:11 AM, Sebastien Douche wrote:
> On Wed, Feb 12, 2014 at 5:09 AM, Jason R. Coombs wrote:
>
> Hi Jason
>
>> This backward-incompatible release contains the changes detailed in the
>> CHANGES.txt file:
>
> Issue #148: When building (bdist_egg), setuptools no longer adds
> ``__init__.py`` files to namespace packages. Any packages that rely on this
> behavior will need to create ``__init__.py`` files and include the
> ``declare_namespace()``.
>
> Thing like:
>
> try:
> __import__('pkg_resources').declare_namespace(__name__)
> except ImportError:
> from pkgutil import extend_path
> __path__ = extend_path(__path__, __name__)
> __init__.py (END)
>
> are theses lines good?
Yes. The main idea is just that the __init__.py must set up the
namespace package, and not rely on setuptools doing the declaration
implicitly/automatically.
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] File names with spaces
On Tue, Jan 7, 2014 at 4:07 AM, Maciej (Matchek) Bliziński wrote: > 2013/10/1 Maciej (Matchek) Bliziński >> >> Hello setuptools developers, >> >> I'm attempting to package the newest setuptools version (1.1.6) >> on Solaris 9 and 10. One of the limitations of the Solaris package manager >> (the old one, pkgadd/pkgrm), is that it is unable to handle file names >> with >> spaces. Would you mind renaming "script template.py" to something like >> "script_template.py"? Probably yes, they would mind. ;-) I believe the reason there's a space there is so that the file is not importable. (It should never be imported; it's a data file rather than a python module.) I suspect you'll have better luck with a suggestion like 'script-template.py', since that would still not be importable. > I've created a pull request with the change, do you have any advice who > should I talk to? > > https://bitbucket.org/pypa/setuptools/pull-request/33/rename-script-templatepy-to/diff I've added a comment there about the underscore vs. dash; it looks from the other pull requests like Jason wants issues filed describing the problems that a pull request is intended to fix, and that may mean he hasn't seen the pull request yet. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Major update to the metadata 2.0 draft PEPs
On Wed, Jan 1, 2014 at 9:39 AM, Nick Coghlan wrote: > On 1 Jan 2014 10:33, "PJ Eby" wrote: >> I have only been skimming these so far, will comment more later, but I >> just want to mention that for standard extensions, what is the >> rationale for not defining e.g. commands as exports? Or for that >> matter, defining hooks as exports? Both commands and hooks have a >> payload that's the same as an export payload, i.e., a reference to an >> importable object. I think it'd be good for them to be defined in >> terms of exports in order to reduce duplication of concepts and >> implementations, as well as providing a PEP-defined example of how to >> use export groups and export names effectively. > > I believe it was due to the extra layer of nesting they needed - using > multiple parallel export groups with different final elements in their > dotted names didn't feel right. > > I guess that indicates a flaw in my initial design for the export > definitions though - I agree it would be better if commands and hooks could > be cleanly defined within the exports mechanism, rather than needing > separate custom metadata extensions. If it's a flaw, I'd say it's in my original design of entry points. ;-) Basically, I wanted a way to do -- without XML -- what Eclipse does with its "extensions" and "extension points" machinery. I went with a "flat (with dots) is better than nested" concept. To me, though, this doesn't look terribly complicated (using entry points syntax): [python.exports.after_install] ComfyChair.plugins = ComfyChair.plugins:install_hook [python.exports.after_uninstall] ComfyChair.plugins = ComfyChair.plugins:install_hook Nor this: [python.extensions.after_install] python.exports = pip.export_group_hooks:run_install_hooks python.commands = pip.command_hook:install_wrapper_scripts [python.extensions.after_uninstall] python.exports = pip.export_group_hooks:run_uninstall_hooks (Also, adding hooks to *validate* extensions and exports at build and/or install time might be handy.) Finally, note that if the typical usecase is to define *both* an install and uninstall hook, then it might be simpler to just define the hook once, as an object with 'after_install' and 'after_uninstall' methods. This avoids the need to register a hook in two groups, and in the simplest case people can just make them both module-level functions and list the module as the export. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Major update to the metadata 2.0 draft PEPs
On Sat, Dec 21, 2013 at 9:46 AM, Nick Coghlan wrote: > I've just published the first draft of the metadata 2.0 spec that > moves all of the fields that aren't part of the core metadata or > potentially needed for dependency resolution out to a separate > "standard metadata extensions" PEP. Yay! > I think this makes PEP 426 itself substantially less overwhelming, and > also provides convenient names to refer to the various other subsets > of the metadata. Indeed! > Metadata 2.0: http://www.python.org/dev/peps/pep-0426/ > Versioning: http://www.python.org/dev/peps/pep-0440/ > Standard Extensions: http://www.python.org/dev/peps/pep-0459/ I have only been skimming these so far, will comment more later, but I just want to mention that for standard extensions, what is the rationale for not defining e.g. commands as exports? Or for that matter, defining hooks as exports? Both commands and hooks have a payload that's the same as an export payload, i.e., a reference to an importable object. I think it'd be good for them to be defined in terms of exports in order to reduce duplication of concepts and implementations, as well as providing a PEP-defined example of how to use export groups and export names effectively. (Also, noting the mapping of current script export namespaces to the new metadata would be helpful for people implementing translation tools from setuptools metadata.) Last but not least, some of the export examples should use dotted post-module names (e.g. "some.module:RandomClass.a_class_method") to highlight that qualified names can be used there. (Yes, the spec *says* qualified names, but it's the sort of thing that people overlook when the examples are all of unqualified simple names.) ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Distribution format for Python3 libraries
On Tue, Nov 19, 2013 at 5:24 PM, Nick Coghlan wrote: > Perhaps it would be worth breaking the environment markers out as their own > PEP? As I said back then, I think *everything* in PEP 426 is worth breaking out into its own PEP. ;-) Less jokingly, I think that the scope of PEP 426 includes many things that have different (if overlapping) audiences, and splitting it into smaller pieces will help those audiences concentrate on the bits relevant to them. The audience for markers is a subset of the audience for requirement specs, which is a subset of the audience for version syntax. Lots of other subtopics are of interest mainly to build tool and distro makers, so there's little point in making everybody read everything. (Also, a lot of these smaller things are (hopefully) easier to reach consensus on, at which point the PEPs are settled and allow moving discussion on to the more open issues.) So, yes, good idea to break out environment markers. ;-) ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Distribution format for Python3 libraries
On Tue, Nov 19, 2013 at 6:22 AM, Nick Coghlan wrote: > On 19 November 2013 15:09, PJ Eby wrote: >> On Sun, Nov 17, 2013 at 8:44 AM, Nick Coghlan wrote: >>> More accurately, it appears setuptools *does* support PEP 345 style >>> environment markers, it just isn't documented at >>> http://pythonhosted.org/setuptools/setuptools.html#declaring-dependencies >>> *sigh* >> >> That's because it's technically an experimental feature I hacked into >> 0.6c12 development shortly before the distribute merge in order to >> handle setuptools' own requirements for SSL, so that setuptools could >> ship just one platform-independent egg and still have >> platform-specific dependencies. ;-) >> >> I think Jason et al may have since upgraded it to a supported feature, >> but last I looked I think there may have still been issues with Jython >> support for the feature. >> >> So, the lack of documentation may still be a feature rather than a bug ATM. >> ;-) > > A fair point :) > > Regardless, it highlights the fact I need to ensure PEP 426 preserves > the legacy spellings for the markers that have dots in their names in > metadata 1.2 (https://bitbucket.org/pypa/pypi-metadata-formats/issue/12). Eh? I implemented PEP 426 (draft-at-the-time) markers in the feature I mentioned above. Hm. Actually, looking over it now, I see the real reason it's experimental: it was implemented based on consensus-in-progress for PEP 426, so not a finalized PEP. It also sort-of handles PEP 345. And the Jython fallback, IIRC, was maybe only doing PEP 345. (And setuptools itself only needed the Python version and platform name.) So it's actually a bit of a mess -- which is why it wasn't documented, in order to avoid people forcing the new PEP's hand with another de facto standard. I just figured it would get us most of the way there and when the PEP dust settled, the tests and code could be tweaked a bit to match, then documented with reference to the PEP. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Distribution format for Python3 libraries
On Sun, Nov 17, 2013 at 8:44 AM, Nick Coghlan wrote: > More accurately, it appears setuptools *does* support PEP 345 style > environment markers, it just isn't documented at > http://pythonhosted.org/setuptools/setuptools.html#declaring-dependencies > *sigh* That's because it's technically an experimental feature I hacked into 0.6c12 development shortly before the distribute merge in order to handle setuptools' own requirements for SSL, so that setuptools could ship just one platform-independent egg and still have platform-specific dependencies. ;-) I think Jason et al may have since upgraded it to a supported feature, but last I looked I think there may have still been issues with Jython support for the feature. So, the lack of documentation may still be a feature rather than a bug ATM. ;-) ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Egg name computation
On Mon, Oct 28, 2013 at 4:29 AM, Martin Fiers wrote: > I guess we'll have to > rename them manually after the setup() function, unless there's a way to > 'force' setup() to 'think' it has compiled extensions in them? You could include a dummy extension that does nothing, I suppose. Or which controls the building of your actual extensions. Setuptools has long supported Pyrex and I think that Cython might also work, i.e., that you could just specify your cython modules as extensions in setup.py to start with. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Egg name computation
On Tue, Oct 15, 2013 at 8:07 AM, Martin Fiers wrote: > So the platform argument now is > self.distribution.has_ext_modules() and self.plat_name > Shouldn't it just be > self.plat_name > ? No. The platform name is only included if the distribution has extension modules, because extension modules are what make the egg platform-specific. If there is only Python code and data, then the egg is considered platform independent. > Would there be a workaround? What do you want to work around this *for*? If the egg doesn't contain extension modules, what is platform-specific about it? ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Warehouse - Simple API
On Wed, Oct 2, 2013 at 10:57 PM, Donald Stufft wrote: > I've recently deployed the current Warehouse code. So far the > simple API has been implemented and it would be great if > people could point their installers to it and report back to me > if they run into any problems. > > The current PyPI simple api is not documented at all so I've had > to reverse engineer it going from what I know, pip, etc. Ideally > if i've missed anything it will be found out with testing from y'all. FYI, compare: https://preview-pypi.python.org/simple/setuptools/ and: https://pypi.python.org/simple/setuptools/ The former is missing the #egg URLs, which are required to install SVN versions. In general, extracted URLs don't seem to be working on the preview site. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pkg_resources get_distribution - zipfile support?
On Tue, Oct 1, 2013 at 7:10 PM, Nick Coghlan wrote: > > On 2 Oct 2013 07:12, "Paul Moore" wrote: >> >> On 1 October 2013 21:32, PJ Eby wrote: >> > On Tue, Oct 1, 2013 at 1:51 PM, Daniel Holth wrote: >> >> pkg_resources only finds distributions inside .egg and inside sys.path >> >> entries that are filesystem directories. >> > >> > Actually it looks in zipfiles (or subdirectory paths thereof) or >> > filesystem directories, and can spot zipfile subdirectories named >> > '.egg' inside zipfiles (or subdirectories thereof). >> >> But not dist-info? I thought setuptools supported dist-info formats >> these days. Is this somewhere that got missed? Or is it more subtle >> than that? > > I believe it's not recognising "*.whl" as interesting, so it isn't even > looking inside the nested wheel to check for dist-info. That would only be relevant if the .whl weren't on sys.path. Since it's on sys.path, it's processed by importer, not by filename. (It's just that there's no .dist-info detection in the zipimporter handler.) The .whl extension is only relevant for discovery of nested .whl's (wheels within wheels... or within eggs or plain zipfiles...), or .whl's in directories. There isn't any good terminology for describing this at the moment, which makes all this sound much more complicated than it actually is. ;-) Making up some new terminology, suppose we have foo.egg, bar.egg-info, baz.dist-info, and spam.whl in site-packages. Then the bar and baz distributions are "mounted" at the '.../site-packages' location string, but the foo and spam distributions are merely "discoverable" at that same location string. (They are *mounted* at '.../site-packages/foo.egg' and '.../site-packages/spam.whl', respectively.) That is, to be mounted at a given location string means "to be importable if that location string is on sys.path", and to be discoverable at a given location means "to be available for dynamic dependency resolution (e.g. via require()) if that location string is on sys.path". Determining what is mounted or discoverable at a given sys.path location is the job of the find_distributions() API. If the 'only' flag is true, it yields only mounted distributions at the given location string. If false (the default), it yields both mounted and discoverable distributions. Behind the scenes, this is implemented by finding a handler for the importer that the PEP 302 import protocol would use to look for modules at the given location string, and then delegating the operation to that handler. The handler then has to look at the location string and figure out what distributions are mounted and/or discoverable there. To find mounted distributions, the directory handler (find_on_path()) checks whether the directory string itself ends in '.egg' (and could theoretically do the same for .whl), and also looks for contained .dist-info and .egg-info files or directories. To find mountable distributions, it checks for files or directories ending in '.egg' (and could theoretically do the same for .whl). The zipfile handler (find_in_zip()) doesn't actually bother checking for an .egg extension; instead it checks for an EGG-INFO/PKG-INFO and assumes it'll be able to figure things out from that. And it checks for nested .eggs if it's looking for discoverables. So, what it's missing to support Paul's use case is a check for .dist-info/METADATA, analagous to the EGG-INFO/PKG-INFO check. It should be relatively simple to add, if somebody wants to do that. (It can even be done from code outside pkg_resources, as there is a 'register_finder()' API that can be called to register a replacement handler.) In some ways, I'm finding the code structure irritating now, because the one abstraction I *didn't* build into the original framework was a concept that there would ever be competing formats to .egg and .egg-info, so implementing .dist-info and .whl requires annoying repetitions of code at the moment. But it's probably not worth refactoring to make this cleaner, because the odds that there will be a *third* file format needing to be supported any time soon are hopefully quite small. ;-) ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pkg_resources get_distribution - zipfile support?
On Tue, Oct 1, 2013 at 5:11 PM, Paul Moore wrote:
> On 1 October 2013 21:32, PJ Eby wrote:
>> On Tue, Oct 1, 2013 at 1:51 PM, Daniel Holth wrote:
>>> pkg_resources only finds distributions inside .egg and inside sys.path
>>> entries that are filesystem directories.
>>
>> Actually it looks in zipfiles (or subdirectory paths thereof) or
>> filesystem directories, and can spot zipfile subdirectories named
>> '.egg' inside zipfiles (or subdirectories thereof).
>
> But not dist-info? I thought setuptools supported dist-info formats
> these days. Is this somewhere that got missed? Or is it more subtle
> than that?
The person or persons who implemented dist-info either didn't know
where to put in the support, or didn't consider it a priority to also
support zipped dist-info.
A cursory glance at the DistinfoDistribution class, however, suggests
that it should work fine as long as it's paired with an appropriate
metadata provider. Something like:
class WheelMetadata(EggMetadata):
def _setup_prefix(self):
# like EggProvider._setup_prefix, except w/".whl" and
# ".dist-info" instead of ".egg" and "EGG-INFO"
# ... or refactor EggProvider to parameterize these as
# class attrs and override here
and then make find_in_zip look more like:
def find_in_zip(importer, path_item, only=False):
for (meta_factory, fn, dist_factory) in [
(EggMetadata, 'PKG-INFO', Distribution),
(WheelMetadata, 'METADATA', DistInfoDistribution),
]:
metadata = meta_factory(importer)
if metadata.has_metadata(fn):
yield dist_factory.from_filename(path_item, metadata=metadata)
if only:
return # don't yield nested distros
for subitem in metadata.resource_listdir('/'):
if subitem.endswith('.egg') or subitem.endswith('.whl'):
subpath = os.path.join(path_item, subitem)
for dist in find_in_zip(zipimport.zipimporter(subpath), subpath):
yield dist
So it's not a huge deal to implement, just a bit of tedium. Under the
hood, there's little difference between wheels and eggs besides naming
conventions.
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pkg_resources get_distribution - zipfile support?
On Tue, Oct 1, 2013 at 1:51 PM, Daniel Holth wrote:
> pkg_resources only finds distributions inside .egg and inside sys.path
> entries that are filesystem directories.
Actually it looks in zipfiles (or subdirectory paths thereof) or
filesystem directories, and can spot zipfile subdirectories named
'.egg' inside zipfiles (or subdirectories thereof).
It will also look in any other sort of sys.path entry, if you register
a handler for the importer type.
> You might be able to manually
> add a new Distribution instance to working_set or start looking for a
> place to add the feature here:
>
> https://bitbucket.org/pypa/setuptools/src/27df3c725f9696ba730456f3f444cc2fb5271d4b/pkg_resources.py?at=default#cl-2560
>
> _distributionImpl = {
> '.egg': Distribution,
> '.egg-info': Distribution,
> '.dist-info': DistInfoDistribution,
> }
>
> https://bitbucket.org/pypa/setuptools/src/27df3c725f9696ba730456f3f444cc2fb5271d4b/pkg_resources.py?at=default#cl-2219
>
> (Distribution.from_location())
Actually, you would add the feature *here*:
https://bitbucket.org/pypa/setuptools/src/27df3c725f9696ba730456f3f444cc2fb5271d4b/pkg_resources.py?at=default#cl-1810
That is, in the find_in_zip() function, which is used to identify
distributions contained in zip files (or subdirectories thereof) on
sys.path.
(This feature needs to get added at some point anyway, for
pkg_resources to support mountable wheels.)
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] unparseable sdist filenames
On Mon, Sep 30, 2013 at 6:06 PM, Marcus Smith wrote: >> >> > how will context decide between the version being "dev" or "xdist-dev"? >> >> By whether you asked to install "pytest-xdist" or "pytest", and by >> whether "dev" or "xdist-dev" match your version requirements. In >> practice this tends to only be a problem if you are: >> >> 1. Installing the two packages during the same run, and >> 2. Aren't using version specifiers. > > > #2 is a big deal. the pip issue (#1192) that motivated this was a #2. local > find-links with non-versioned requirements. I should probably add a #3 to that list: it's also rarely a problem because usually there's at least one *numbered* version of the desired package available, which will always prioritize newer than an arbitrary alphabetic version like "xdist", even if you haven't requested a specific version. IOW, you need to not only not request a version and come across the other package in the same run, but you *also* need for there not to be any valid versions of your intended target present. > so, suppose you have "pytest-xdist-dev.tar.gz" in a find-links location. > whether you're trying to install "pytest" or "pytest-xdist" doesn't help the > installer determine how to parse that archive. In easy_install's case, it will take the one with the highest version number, which means it'll try to install pytest-xdist-dev.tar.gz, on the theory that 'xdist-dev' is a newer version than 'dev'. This is kind of silly, but it was the Simplest Possible Thing That Could Work, and I figured I'd change it when there was enough feedback/field experience to decide what to change it to. ;-) Probably the "right" thing to do would be to warn or decline on ambiguous filenames in the future, since it is quite possible to detect if a filename has more than one possible interpretation. Another possibility would be to treat them analagously to stable/unstable versions, prioritize them below regular matches, etc. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] unparseable sdist filenames
On Mon, Sep 30, 2013 at 4:11 PM, Marcus Smith wrote: > >> The setuptools.package_index API, however, *does* support parsing >> sdist names, it's just that it generates a *list* of possibilities, > > > oh, ok, "setuptools.package_index.distros_for_url" > >> >> Thus, tools using this API can contextually decide which to consider >> the "real" interpretation, based on context. This method is used by >> easy_install; I don't know if pip does as well. > > > how will context decide between the version being "dev" or "xdist-dev"? By whether you asked to install "pytest-xdist" or "pytest", and by whether "dev" or "xdist-dev" match your version requirements. In practice this tends to only be a problem if you are: 1. Installing the two packages during the same run, and 2. Aren't using version specifiers. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] unparseable sdist filenames
On Mon, Sep 30, 2013 at 1:55 PM, Marcus Smith wrote: > so, take a case like so "pytest-xdist-dev.tar.gz" (or any sdist with "-" in > the project name, and a version starting with a string) > > I think it's like so: > - pkg_resources.Distribution.from_location will treat "xdist-dev" as the > version. > - distlib.util.split_filename won't parse it because versions have to start > with [0-9]. For these APIs, these are reasonable defaults, because they are intended only to parse *unambiguous* filenames. Neither API should be used with sdist files, because sdist filenames are ambiguous and cannot therefore be parsed to a single name. The setuptools.package_index API, however, *does* support parsing sdist names, it's just that it generates a *list* of possibilities, rather than a single unambiguous interpretation of the filename. If you ask it to parse the example you gave, you will get a list of: * Project "pydist", version "xdist-dev" * Project "pydist-xdist", version "dev" * Project "pydist-xdist-dev", no version Thus, tools using this API can contextually decide which to consider the "real" interpretation, based on context. This method is used by easy_install; I don't know if pip does as well. > - pip will accept this as a "pytest" archive and install it potentially if > no other version matches greater. Does it also accept it as "pytest-xdist"? > what's the right answer? It depends on what you want, and whether what you want is compatible with reality. ;-) > The historical-compatible answer is to be confused when projects have "-". > so stay confused? or get rigid like distlib? The future sdist format should not be ambiguous like this. Dealing with ambiguity in past packages is for the moment something of a requirement if you are making a general-use tool. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] README file format and pypi.python.org
On Sun, Sep 22, 2013 at 9:01 AM, Paul G. wrote: > 1. What format should I use in my README.txt file for my package's content > to be displayed on its package page? It's not the README file; it's the package's "long_description" keyword, as specified in your setup.py setup() call. And the format is reStructuredText > 2. Do I have to use a different extension for the README? It doesn't matter, since the README is not read by PyPI. (You can put code in your setup.py to read the file into the long_description field, though. Take a look at other packages' setup.py files to see how they do it.) ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Deviation between distutils and setuptools over package_data
On Fri, Sep 20, 2013 at 8:15 AM, Jim Fulton wrote: > It appears that MANIFEST.in is ignored if setuptools recognizes your > VCS. Not in setuptools 0.6 it isn't. However, it's really only useful for *adding* files not picked up by revision control; it can't *remove* files found by revision control. Manifest and revision control support are semi-independent, as it's quite possible to have files you don't put in revision control but which nonetheless need to be included in an sdist. (For example, you might want to have Cython/Pyrex-generated C code, or compiled documentation files.) ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Deviation between distutils and setuptools over package_data
On Wed, Sep 18, 2013 at 5:07 PM, Benjamin Root wrote: > In creating a source distribution, I have found a disparity between the > behaviors of distutils and setuptools with respect to package_data. As of > python Issue 2279: http://bugs.python.org/issue2279, entries listed in > package_data are used when building an sdist. I have verified that this > works in a simple example when setup() is imported from distutils.core. > However, if I import setup() from setuptools, it does not pull in the data > as listed in package_data (and presumedly data_files). You need to use the include_package_data = True flag. > P.S. - on a related note, for a package "foo", foo.egg-info directory is > created with a SOURCES.txt file. I have found that under certain situations, > it seems that a successful install would result in a fully listed > SOURCES.txt, and then subsequent calls to sdist seems to use that > information to build seemingly correct archives. A co-working > double-checking a deployment process I made did an sdist and created a > source distribution without the package_data when he did a fresh checkout, > but whenever I did it from my development branch, the source distribution > worked fine. I haven't figured out exactly how this came about, but it seems > to be tied to the SOURCES.txt file. SOURCES.txt mostly exists so that you can safely build an sdist from an sdist, as is required by e.g. bdist_rpm, without having any revision control data on hand to guide the process. Setuptools also can insert a possibly-modified setup.cfg into an sdist for the same reason, so that if you used revision control tags to specify the version when building the sdist, any sdists rebuilt from that sdist will have the same version tags. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip does not work as expected from a zip
On Thu, Sep 12, 2013 at 2:14 PM, Daniel Holth wrote: > I'm suggesting it might be a bug in pkg_resources, or it might be > something pkg_resources can already do, if pip.zip is added to > PYTHONPATH while executing setup.py in a subprocess. pkg_resources can detect subdirectories in a zip that are named "whatever-version.egg/", and treat them like egg files or directories. I don't recall whether it can detect .egg-info in .zip, though; I don't think I specifically implemented that, so if it works, it's by virtue of a good orthogonal design rather than any specific intent for it to work. ;-) My guess, though, is that the "basket" support (i.e. zipped .egg subdirectories) is the only way at the moment to bundle multiple distributions in a .pyz or other archive. (Note: I don't mean putting an .egg file in the .zip, just zipping up an .egg directory and including it as a subdirectory inside the main .zip/.pyz/whatever.) ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Specification for a local PyPI simple index
On Mon, Sep 9, 2013 at 10:54 AM, Paul Moore wrote: > Is the spec at > http://peak.telecommunity.com/DevCenter/EasyInstall#package-index-api > still the definitive version of what must be provided for a local PyPI > index (for use by something like pip)? Or is there a more up to date > version anywhere? That spec is for setuptools 0.6. Later versions should have changed this documentation (in the PyPA repository) if they changed the protocol, but I don't know if anybody's actually keeping tabs on that. > For example, are MD5 signatures still the only supported version? I > thought we were moving away from MD5. Hm. Yeah, a quick glance at https://bitbucket.org/pypa/setuptools/src shows the docs unchanged, so whoever added the non-MD5 support forgot to check the docs for references to md5. > And while I haven't really > followed the offsite hosting changes, are the > rel="homepage"/rel="download" links still as stated? (I think I'd want > rel="download" on everything as I only expect to provide URLs for > actual package content). The meaning of re="downloadl" values is, "spider this page for download links", not "this is a link to download".Links to download are identified by inspecting a link, not retrieving it. The only reason the rel tags exist is to mark URLs as spiderable. > Also, how definitive is item 7, which states that the root URL must > result in a page containing all projects, but it can be omitted if > case-insensitive safe_name() matching of projects is implemented? It's definitive for easy_install. The only reason easy_install retrieves the root URL is if a requested package isn't found; the reason it does this is to catch alternative spellings due to case-sensitivity and/or differences in punctuation folding. If you can interpret easy_install's initial GET as a package requirement string (w/case- and punctuation-insensitivity via pkg_resources.safe_name()) rather than as an exact string match, then failure to match would produce the same failure to match on a full package listing, so there's no point having the full listing appear. It's strictly a fallback intended for "dumb" package indexes that simply consist of a directory tree and a web server providing directory listings. (I think it can even work with an FTP site, but it's been a while since I worked on that code.) > The > reason I ask is that providing a full listing will be somewhat costly > in my application, but providing case-insensitive matching should be > doable (actually, I'm not sure yet what's feasible, but I want to know > whether it's worth my time even investigating). I don't know what pip does, but I assume that it's probably true of all package managers that either their targeted request succeeds or fails, and then they either request the full listing or they don't. So... the only possible way not providing the full list would be if some (foolish) package manager always began by requesting a full package listing. It's possible there are tools that wish to obtain a full listing and use the base URL for that... but AFAICT it would be a foolish thing to do if you're just trying to install packages. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Distributable binary with dependencies
On Mon, Aug 26, 2013 at 2:25 PM, bharath ravi kumar wrote: > Carl, Eby, > > Thanks for taking time to suggest various alternatives. Considering that the > deployment hosts are identical in every as[ect, the approach of moving > virtualenv's with packages pip-installed at build time appears the simplest, > low-overhead approach that can be implemented without hacking the > environment or resorting to custom scripts. I'll go ahead with that option. What hacking the environment or custom scripts? I'm confused, because AFAIK there are actually more steps to pip-install a virtualenv and copy it to different machines, than there are involved in using easy_install to create a portable installation. In both cases, you end up with a directory to archive and copy, so the only difference is in the commands used to build that directory, and the layout of the directory afterwards. Perhaps you misunderstood my post as meaning that you had to run easy_install on the target system? (I don't have any particular stake in what you do for your own system, but I'm curious, both for the future reference of folks reading this thread by way of Googling this question, and in case there is something for me to learn or that I'm mistaken about, in relation either to pip/virtualenv or your use case. And certainly if you are more familiar with pip+virtualenv, that would actually be sufficient reason in this case to use it. But I'd prefer future readers of this thread not to be under an erroneous impression that easy_install involves more steps, scripts, or environment changes in order to implement this use case. Thanks.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Multi-version import support for wheel files
On Tue, Aug 27, 2013 at 3:01 AM, Paul Moore wrote: > On 27 August 2013 00:15, PJ Eby wrote: > None of these things is wrong. It is *spreading* FUD (and in particular, > doing so cynically to undermine a proposal) that is wrong, and I hope I > didn't do that - I certainly did not intend to and I'm a bit unhappy about > the implication that I might have. Sorry for the implication; it was not intended. I did not think you had any intent to make other people share your doubts or had any desire to shoot down the proposal. As I said, the real intent of my (clearly, in retrospect, very poorly-worded) side-remark was that I thought 90% of the objections to Nick's proposals were based on fear, uncertainty, and doubt rather than any actual issues with the proposals themselves. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Multi-version import support for wheel files
On Mon, Aug 26, 2013 at 5:59 PM, Donald Stufft wrote: > I think you're confused. The only comments I see in this thread are people > doing > due diligence to ensure that Nick's proposal *didn't* include the parts of > setuptools > that we felt were incurring a cost against people not using the feature and > expressing > a desire *not* to attach it to the Wheel format and instead attach it to > another format > on it's own. I mean the person you originally quoted even explicitly said "I > have no > objection to this proposal" sans some stuff about not wanting it to be > attached to > Wheels. So I'm not sure how you can take someone saying they have no objection > to the proposal and translate it to people are shooting down Nick's proposal. FUD stands for fear, uncertainty, doubt. My comment was that a lot of the original objections to Nick's proposal seemed fearful, uncertain, and doubting, specifically because they were thinking the proposal was proposing things it wasn't. It was you who brought up the idea of persecution; my response was that I don't think anybody's persecuting setuptools, only giving unnecessary levels of doubt to Nick's proposal due to confusion about how it relates (i.e. mostly doesn't) to setuptools. You pounced on a tiny piece of my email to Paul, in which I mainly expressed confusion about his statements about "cost". I was having trouble understanding what sort of "costs" he meant, and in subsequent discussion realized that it's because he and others appeared to have conflated setuptools' default-version issues, with Nick's proposal for handling non-default versions. My comment was that 90% of the thread appeared to stem from this fear, uncertainty, and doubt, based on this misunderstanding, although more precisely worded, what I actually meant was that 90% of the *objections* raised to Nick's proposal were based on the aforementioned fear, uncertainty, and doubt -- i.e., that the objections had nothing to do with that which was being proposed. At one point this weekend, I had intended to write a detailed rebuttal to all of the points that had been raised, but by the time I had time to do so, the discussion was mostly settled and the issue mostly moot... but the impression that 90% of the original objections were misunderstanding-based remained, which led to my (perhaps poorly-phrased) 90% remark. All that being said, I'm not sure why you pounced on that side-comment in the first place; did you think I was personally insulting you or accusing you of something? ISTM that you are making an awfully big deal out of an incidental remark that had very little to do with the main point of the email, and framing it as though I am the one who is making a big deal of something. If you hadn't intervened, I don't see any reason why the conversation wouldn't have reached a peaceable conclusion, and am still puzzled as to why you felt the need to intervene. Your initial email began by disputing facts that you now appear to accept, in that you did not reply to any of my rebuttals to your assertions. But instead of admitting your assertions were in error, you're asserting that I'm the one who's confused. Well, I wasn't before, but I sure am *now*. ;-) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to detect a namespace packages?
On Mon, Aug 26, 2013 at 6:36 AM, Hartmut Goebel wrote: > Hi, > > I'm one of the developers of www.pyinstaller.org, a tool for creating > stand-alone executables. > > We need to reliable detect if a package is a namespace package (nspkg). > For each namespace, we need to add an empty fake-module into our > executable to keep the import mechanism working. This has to work in all > versions of Python starting with 2.4. > > nspkgs set up via a nspkg.pth-file are detected by being in sys.modules, > but imp.find_module() files. > > For nspkgs using __init__.py-files (which use > pkg_resources.declare_namespace() or pkgutil.extend_path()) I have no > clue how to detect them. > > I tried to query meta-information using pkgresources, but I did not find > a solution. > > Any help? Setuptools package metadata includes a namespace_packages.txt file with this information: http://peak.telecommunity.com/DevCenter/EggFormats#namespace-packages-txt-namespace-package-metadata This won't help you with PEP 420 namespace packages (3.3+), unless someone declares them, and likewise it won't help if somebody uses the dynamic APIs without any declaration. But at least it'll give you the declared ones. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Multi-version import support for wheel files
On Mon, Aug 26, 2013 at 1:48 PM, Chris Barker - NOAA Federal wrote: > Just to add a bit more "FUD" ;-) > > i.e. if a user has only needs one version of a given package > installed, lets not have much overhead there to support that, and > let's not require much run-time support at all. Nick's proposal does that. What I mean by FUD in this context is that a lot of the thread is discussing fears that *are* relevant to setuptools, but *not* to Nick's proposal. It doesn't mean I think anyone's use cases or needs are irrelevant or dumb; I'm just saying that a lack of understanding of Nick's proposal is causing people to equate it with problems in setuptools that relate to *default* versions, not to making alternate versions available. Nick's proposal doesn't involve any weirdness for packages that aren't *already* using pkg_resources or which require the use of a non-default version. Under his proposal, default versions don't behave any differently than stock Python and pip, and nobody "pays" any cost for something they're not actually using. If you never need a non-default version, his proposal affects nothing on that system. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Multi-version import support for wheel files
On Mon, Aug 26, 2013 at 11:15 AM, Donald Stufft wrote: > There is always a cost. In this case mostly in complexity and start up time. > > As you mentioned originally the cost to multi version support was the need > to use a require() function and when people complained about that you > added the .pth files which imposed another sort of cost to people using > multi versioned installs. See, this is exactly what I'm talking about: you've got this 100% backwards: .pth files are for people who *aren't* using multi-version imports. They're for *default* versions, not alternate versions! And they're utterly unnecessary for Nick's proposal. > You claim it is part of core Python but it's really not, if it was it > wouldn't require > importing pkg_resources of the .pth files to make it work. As I pointed out in the email you apparently didn't read, along with multiple emails from Jim: pkg_resources isn't necessary for alternate-version support. All that's required for alternate versions is to add them to sys.path, which buildout does just fine *without pkg_resources*. > I find it ridiculous that you'd call this thread 90% FUD when the vast bulk > of the > thread has been trying to determine if there were any reasonable concerns > with the approach and upon examination determined that the biggest problem > with it was attaching it to Wheel and not the multi version support at all What I'm referring to as the FUD is that people have been confusing what Nick proposed with what setuptools does, and getting *both* of them wrong in the details. Nick's proposal was not to mimic setuptools' multi-version support, but rather to provide something else: let's call it "alternate version support", to separate it from what setuptools does. In Nick's AVS proposal, there is *no* overhead for anything that doesn't need a non-default version, and it's 100% opt-in, used only for things that need *non-default* versions. Note, by the way, that since these *non-default* packages aren't on sys.path by default, *there is no overhead and no .pth files are involved*. They are effectively invisible and irrelevant for anything that doesn't use them. The only place where there's overhead is in the script that needs the alternative version(s), and its sys.path is lengthened only by those items that it can't obtain from the default sys.path. And if you use buildout's approach of simply adding: sys.path[0:0] = [path1,...] to the head of a script, then *pkg_resources isn't involved either*. This is bog-standard stock Python. So the FUD part I was referring to is all the "oh no, setuptools is complicated" in response to Nick's perfectly reasonable idea *which doesn't involve any of setuptools' complexity*, because it's doing something completely different. > I realize > setuptools and easy_install are your baby but the persecution complex doesn't > help to win people over to your side of things. I think you're confused here. I don't think setuptools is being persecuted, I think *Nick's idea* is being misunderstood, and being construed as almost the exact *opposite* of what it is. All the stuff people bitch about that relates to multi-versions in setuptools are actually issues with setuptools' implementation of *default* versions, not *alternative* versions. So to look at Nick's proposal and think it's going to have the same problems is completely ludicrous - it's 180 degrees opposite of what setuptools does, because for setuptools, *default versions* are the special case -- they're what cause 90% of the complexity in pkg_resources' manipulation of sys.path, and they're the main reason .pth files are ever used. So it's crazy-making to see people thinking Nick's proposal is going to bring all that crap along, when that's the exact *opposite* of the situation. > In my experience setuptools has a lot of good ideas but they are wrapped in > bad > ideas or implementations that obscure the fact that there *are* good ideas > there. > I do not believe it to be unreasonable for people to want to make sure that > we're > standardizing around one of the *good* ideas instead of one of the bad ideas. It would help if people understood the actual facts, then. AFAICT, Nick's proposal doesn't do any of the things that people are worried about, or at the very least does not *require* them. As Jim and I have pointed out more than once, pkg_resources is not a runtime requirement to allow alternative versions to be importable by code that wants them. It would really be a shame to shoot down Nick's idea based on a vague misunderstanding of it. It's a good proposal, and has far less to do with setuptools than most people in the thread seem to think. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Multi-version import support for wheel files
On Mon, Aug 26, 2013 at 5:20 AM, Paul Moore wrote: > On 25 August 2013 23:14, PJ Eby wrote: >> Thus, you don't have to know you have multiple versions installed; it >> can trivially happen by way of dependencies you aren't paying >> attention to. The more things you install, the more likely it is you >> have two versions hanging around. > > > OK, I see. But I'm not sure if we're agreeing or disagreeing over the > result. To me, this is a bad thing on the principle that there is a cost to > multiversion support (it's not part of core Python, so you have to do > *something* to make it work) Seriously? The basic functionality of using sys.path to have multiple versions *is* part of core Python, and has been since 1.5.2 (16 years ago), and probably longer than that. In the days before easy_install and virtualenv, if you needed different versions of things, you used "setup.py install" to different directories (assuming distutils was involved, otherwise you just copied files) and either put your scripts in the same directories, or used PYTHONPATH or explicit sys.path manipulation. That is all easy_install does: add a naming convention for the directories, and automate the sys.path manipulation. Buildout does the same thing, it just writes the sys.path manipulation into the scripts statically, instead of using pkg_resources at runtime. So the notion of "cost" doesn't make any sense. Tools like easy_install and buildout *reduce* the management cost, they don't add anything to core Python. (Now, if you're talking about the .pth files from easy_install, those are something that I added because people complained about having to use require(), and wanted to have a default version available in the interpreter.) > and so having people inadvertently pay that > cost to use a feature that they don't actually *need* is wrong. What cost are you talking about here? Given that most people don't even know they *have* multiple versions installed or care, how is a cost being imposed upon them? Are you talking about disk storage? > One other point, just as a matter of curiosity (because it's not relevant to > the current discussion): in your explanation above, there doesn't seem to be > any step that says the user normally uses CherryPy 3 (so that would be the > one they would get automatically at the interactive interpreter). If they easy_install that version, sure, that's what they'll get as a default version. > For me, > that's really the only use case I'd have for multi-versioning - 99% of the > time I use a particular version of a project, but I have one particular > application that can't work with the version I prefer. Yes, and that's the sort of scenario Nick was proposing pip support, that you have an explicit "install me a different version for my other app" capability -- such that that app's script wrapper adds its alternate version to sys.path ahead of the default one. So it would have been opt-in and impose the "cost" of a slightly longer sys.path and increased disk space usage only on those who ask for it. (Honestly, 90% of this entire thread has sounded like complete FUD to me, i.e. fear based on a lack of understanding that there actually isn't anything magical about multi-version support. As Jim has pointed out, buildout does multi-version support without even using pkg_resources. And before all these tools existed, people just installed things in different directories and used either adjacent scripts, PYTHONPATH, or explicit sys.path manipulation. There is nothing magical whatsoever about having multiple versions of a thing installed on your system; all the tools do is add naming conventions for where stuff is installed... and having such naming conventions is a *good* thing, compared to the old days.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Multi-version import support for wheel files
On Sun, Aug 25, 2013 at 4:32 PM, Paul Moore wrote: > On 25 August 2013 20:53, PJ Eby wrote: >> >> FWIW, I would also note that if you use easy_install to install >> anything, you are quite possibly using multi-version installs without >> realizing it. (The __main__.__requires__ API is used in >> easy_install-generated script wrappers, so there isn't any way you'd >> know about it without paying specific attention.) > > > Unless I'm missing something, I suspect that this over-counts the number of > people using multi-version, in the sense that many (the majority?) of > wrapper scripts using multi-version do not actually need to,because the > users never install more than one version. And quite likely don't even know > that they could. That's just it: if you install two programs, one of which needs CherryPy 2 and the other CherryPy 3, then with easy_install this just works, without you having any idea that you even have more than one version installed, unless you for some reason choose to look into it. Thus, you don't have to know you have multiple versions installed; it can trivially happen by way of dependencies you aren't paying attention to. The more things you install, the more likely it is you have two versions hanging around. (The main limiting factor on conflicts isn't a choice to install multiple versions, it's the relative dearth of pinned versions and upper limits on version numbers. If everything just specifies minimum versions, you'll end up using the latest version for everything as the default version. It's only if a package pins or limits a dependency that any conflict is possible to begin with.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Multi-version import support for wheel files
On Sun, Aug 25, 2013 at 12:58 PM, Jim Fulton wrote: > On Sun, Aug 25, 2013 at 3:06 AM, Nick Coghlan wrote: >> The clumsiness of the __main__.__requires__ workaround aside, the main >> advantage this offers is that it *should* result in a relatively >> straightforward addition to pkg_resources to make it work with wheel >> files as well as eggs. That's important, because anyone that is >> currently doing side-by-side multi-versioning in Python is using the >> pkg_resources API to do it, since that's the only option currently >> available. > > No. It isn't. Buildout doesn't use pks_resources to do it. > (Buildout used pkg_resources at build time to manage package meta > data, but I think that's orthogonal to what you're talking about.) > > I'd also hazard to guess that most of the folks with multi-version > installs are using buildout to do it, as buildout does have a > fair number of users. FWIW, I would also note that if you use easy_install to install anything, you are quite possibly using multi-version installs without realizing it. (The __main__.__requires__ API is used in easy_install-generated script wrappers, so there isn't any way you'd know about it without paying specific attention.) I don't know how big the "buildout users w/known multi-version" vs. "easy_install users w/implicit multi-version" groups are, but I imagine the combined group has got to be pretty darn big. ;-) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Distributable binary with dependencies
On Fri, Aug 23, 2013 at 6:45 AM, bharath ravi kumar wrote: > I'm looking to package an application with all its dependencies for > deployment on multiple hosts. I'd like to ensure that there is no > compilation or setup step before starting the application in production. An > nice to have ability would be to isolate base library dependencies per > application (like virtualenv does). Ideally, the development -> deployment > lifecycle would involve: (a) Build an application archive with all its > dependencies baked in (b) Copy archive to a host in production. (c) Unwrap > archive (d) Start services. (Note that the build host & production hosts are > identical in architecture, OS patch level and python version). You can use "easy_install -Zmad deployment_dir application", then archive deployment_dir and extract it on the target machines. (Note: "application" must be a setuptools-packaged project with its dependencies declared, for easy_install to know what to build and deploy.) The "Z" option means "unzip eggs", "m" means "don't worry about the target being on sys.path; we're not trying to install a default version", "a" means "copy all dependencies, even if locally installed already", and "d" means "install libraries and scripts to the following directory". So, the scripts will be put inside deployment_dir with a bunch of adjacent subdirectories containing all the compiled and ready-to-use libraries. The resulting directory is a portable installation of "application": as long as the entire subdirectory is copied to the target machines, everything should work just fine. None of the dependencies or the app itself will interfere with other Python code installed on the target system; it is in a sense a minimal virtualenv which will run whatever scripts that easy_install puts in that directory. One note: the target machines *will* need pkg_resources installed, and it will not be included in the directory by default. If they don't have local copies installed (due to e.g. setuptools, distribute, etc. being installed), you can manually copy a pkg_resources.py to the deployment directory, and it will be used by whatever scripts are in that directory. While there may be other tools available that support this kind of thing, I don't think any of them can do it quite this simply. This deployment scenario was actually a key use case for the original design of easy_install and eggs, so it actually works pretty decently for this. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Wed, Aug 21, 2013 at 9:24 AM, Donald Stufft wrote: > An example is the wsgiref from the standard library. It's an example, alright, but not for your side. ;-) The wsgiref library doesn't just implement the spec, it implements a ton of utility classes for use with the spec. The validator was almost an afterthought grafted on later, borrowed from another project. It implements a framework with all sorts of features that are not technically part of the spec, but are just useful if you want to implement the spec. Very few of the classes, methods, etc. in the entire package are specified by the spec, except in the sense that many of them match a calling signature defined in the PEP. (The PEP doesn't specify any method names, except for things like read() on file-like objects.) IOW, wsgiref is a collection of generally useful tools for anybody doing things with the spec, as an combination of "examples of how to do this" and "ready-to-use code for working with the spec". Personally, I'm very happy to see Vinay's extensions, because they are IMO important validations of whether the new specs are likely to be useful for replacing all of setuptools' functionality. There are people who need to mount eggs and have their extensions run, so if it wasn't possible to build tools that support them under the new specs (whether that support is required by the spec or not), that would still be a reason to use setuptools -- meaning, IMO, that the new spec effort is failing to create a unified packaging world. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Tue, Aug 20, 2013 at 12:39 PM, Thomas Heller wrote: > Ok, now I understand. But the zipfile could contain a loader-module > for each extension which does something like this (this example extracts > and loads 'bz2.pyd'): > ... > > (py2exe for Python 3, which is work in progress, uses this approach) Setuptools has also done this since the egg format was developed, but it has some well-known problems, which unfortunately your example has worse versions of. ;-) Setuptools takes the approach of keeping a per-user cache directory (so that cleanup isn't required, and so there are no security issues where somebody can replace a tempfile between you writing it and importing it), and it uses a unique subdirectory per egg so that different (say) bz2.pyd files can't conflict with each other. Even with these adjustments, Unix users frequently run into issues where the user a process is running as doesn't have access to a suitable cache directory, and so it's a common complaint about the use of zipped eggs. I thought that at one point you (Thomas) had come up with a way to load modules into memory from a zipfile without needing to extract them. Was that you? If so, how did that work out? (ISTR that there was some sort of licensing issue, too.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Changing the "install hooks" mechanism for PEP 426
On Fri, Aug 16, 2013 at 8:04 AM, Nick Coghlan wrote: > Concrete extension use cases I have in mind that don't fit in the > exports/entry-point data model: > > - the mapping of prebuilt executable names to wheel contents > - platform specific external dependencies and other hints for conversion to > platform specific formats (e.g. Windows GUIDs) > - metadata for build purposes (e.g. the working directory to use when > building from a VCS checkout, so you can have multiple projects in the same > repo) > - project specific metadata (e.g. who actually did the release) > - security metadata (e.g. security contact address and email GPG > fingerprint) > > This is why extensions/exports were originally separate, and may still > remain that way. But exports should still be used for hooks defined by the spec; they are the Obvious Way to specify importable hooks, and *not* dogfooding there is still a bad idea. (To be clear, I was never exactly *enthused* about the idea of merging extensions and exports, just *unenthused* about the idea of the spec using extensions or custom metadata to do things that could be perfectly well expressed as exports from a reserved namespace.) I'm kind of toying with the idea that the core metadata itself could be carved into extension namespaces, have the core itself be just another extension, rather than nesting extensions and exports inside the core, so that the entire spec is just a relatively-flat collection of namespaces, in more human-digestible groups. There are some conceptual and practical advantages to that, at least in principle, but until I've played around with actually expressing some concepts from the PEP that way, I won't know whether it would actually pay off in practice. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Changing the "install hooks" mechanism for PEP 426
On Fri, Aug 16, 2013 at 6:21 AM, Vinay Sajip wrote:
> PJ Eby telecommunity.com> writes:
>
>> I guess I didn't explain it very well, because that's roughly what I
>> meant: a single namespace for all "extensions", structured as a
>> mapping from group names to submappings whose keys can be arbitrary
>> values, but whose values must then be either a string or a JSON
>> object, and if it's a string, then it should be an export specifier.
>
> Why should the keys be completely arbitrary?
By "arbitrary" I mean only that the PEP doesn't place syntactical
restrictions on them.
> I can't see a reason for this;
> the current constraint of the "prefixed name" seems sufficient. What would
> relaxing this constraint make possible which otherwise isn't?
I think you're thinking I'm describing a single level namespace; I'm
referring to something like this:
{group1: {anykey: export_or_mapping}}
"anykey" is not validated by the spec, only by registered validators
for "group1". Of course it has to have some meaning that is
interpretable by consumers of group1. The point is that the *spec*
only defines the syntax of group names and export strings, and it's
left to specific groups to define the syntax/semantics of the keys.
> On the values: an export specifier is just a more human-friendly version of
> a dict with module/content/extra keys. While of course the uses of
> importables in this area is well established, what specific use cases are we
> enabling by allowing arbitrary JSON? It certainly would clutter the metadata
> and render it less human-readable, and the only thing it provides is a dict
> which could be expressed in an importable form
I gave one example already: i18n/l10n information that's about files
contained in the distribution's data. It's quite possible to have
distributions without any code, only data of that kind. A requirement
to create code, just to specify the data seems rather pointless. In
Nick's reply, he's listed other use cases.
The main question is, should exports and extensions be treated
separately? Nick originally proposed merging the concepts and using
arbitrary JSON. My counterproposal was to say, let's distinguish
exports and extensions by restricting the spec to something which
spells out the distinction.
After this further discussion, I think that the use cases we're
discussing really boil back down to exports vs. metadata extensions,
and that maybe we should stick to them being separate.
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Changing the "install hooks" mechanism for PEP 426
On Thu, Aug 15, 2013 at 7:16 PM, Nick Coghlan wrote: > But if we're only going to validate it via hooks, why not have the "mapping > of names to export specifiers" just be a recommended convention for > extensions rather than a separate exports field? I guess I didn't explain it very well, because that's roughly what I meant: a single namespace for all "extensions", structured as a mapping from group names to submappings whose keys can be arbitrary values, but whose values must then be either a string or a JSON object, and if it's a string, then it should be an export specifier. To put it another way, I'm saying something slightly stronger than a recommended convention: making it a requirement that strings at that level be import specifiers, and only allowing mappings as an alternative. In that way, there is a minimum level of validation possible for the majority of extensions *by default*, without needing an explicit validator declared. To put it another way, it ensures that there's a kind of lingua franca or lowest-common denominator that lets somebody understand what's going on in most extensions, without having to understand a new *structural* schema for every extension group. (Just a *syntactical* one) > As an extension, pydist.extension_hooks would also be non-conventional, > since it would define a new namespace, where extension names map to an > export group of hooks. A separate export group per hook would be utterly > unreadable. If you already know what keys go in an entry point group, there's a good chance you're doing it wrong. Normally, the whole point of the group is that the keys are defined by the publisher, not the consumer. The normal pattern is that the consumer names the group (representing a hook), and the publishers name the extensions (representing implementations for the hook). I don't see how it makes it unreadable, but then I think in terms of the ini syntax or setup.py syntax for defining entry points, which is all very flat. IMO, creating a second-level data structure for this doesn't make a whole lot of sense, because now you're nesting something. I'm not even clear why you need separate registrations for the different hooks anyway; ISTM a single hook with an event parameter is sufficient. Even if it weren't, I'd be inclined to just make the information part of the key in that case, e.g. [pydist.extension_listeners] preinstall:foo.bar = some.module:hook This sort of thing is very flat and easy to express in a simple configuration syntax, which we really shouldn't lose sight of. It's just as easy to have write a syntax validator as a structure validator, but if you start with structures then you have to back-figure a syntax. I'd very much like it to be easy to define a simple flat syntax that's usable for 90%+ of extension use cases... which means I'd rather not see the PEP make up its own data structures when they're not actually needed. Don't get me wrong, I'm okay with allowing JSON structures for extensions in place of export strings, but I don't think there's been a single use case proposed as yet that actually *works better* as a data structure. If you need to do something like have a bunch of i18n/l10n resource definitions with locales and subpaths and stuff like that... awesome. That's something that might make a lot of sense for JSON. But when the ultimate point of the data structure is to define an importable entry point, and the information needed to identify it can be put into a relatively short human readable string, ISTM that the One Obvious Way to do it is something like a setuptools entry point -- i.e. a basic key-value pair in a consumer-defined namespace, mapping a semantically-valued name to an importable object. And *most* use cases for extensions, that I'm aware of, fit that bill. You have to be doing something pretty complex to need anything more complicated, *and* there has to be a possibility that you're going to avoid importing the related code or putting in on sys.path, or else you don't actually *save* anything by putting it in the metadata. IOW, if you're going to have to import it anyway, there is no point to putting it in the metadata; you might as well import it. The only things that make sense to put in metadata for these things are data that tells you whether or not you need to import it. Generally, this means keys, not values, in other words. (Which is why l10n and scripts make sense to not be entry points: at the time you use them, you're not importing 'em.) >That's why I'm still inclined to make this one a separate top > level field: *installers* have to know how to bootstrap the hook system, and > I like the symmetry of separate, relatively flat, publication and > subscription interfaces. I don't really see the value of a separate top-level field, but then that's because I don't see anything at all special about these hooks that demands something more sophisticated than common entry points. AFAICT it's a YAGNI
Re: [Distutils] Changing the "install hooks" mechanism for PEP 426
On Thu, Aug 15, 2013 at 12:36 PM, Vinay Sajip wrote: > PJ Eby telecommunity.com> writes: >> than nested.) So I would suggest that an export can either be an >> import identifier string, *or* a JSON object with arbitrary contents. > [snip] >> Given how many use cases are already met today by providing >> import-based exports, ISTM that they are the 20% that provides 80% of >> the value; arbitrary JSON is the 80% that only provides 20%, and so >> should not be the entry point (no pun intended) for people dealing >> with extensions. > > The above two statements seem to be contradictory as to the value of > arbitrary JSON. I don't see a contradiction. I said that the majority of use cases (the figurative 80% of value) can be met with just a string (20% of complexity), and that a minority of use cases (20% of value) would be met by JSON (80% of complexity). This is consistent with STASCTAP, i.e., simple things are simple, complex things are possible. To be clear: I am *against* arbitrary JSON as the core protocol; it should be only for "complex things are possible" and only used when absolutely required. I think we are in agreement on this. > I think the metadata format is a communication tool between > developers as much as anything else (though intended to be primarily > consumed by software), so I think KISS and YAGNI should be our watch-words > (in terms of what the PEP allows), until specific uses have been identified. +100. >> That would make it easier, I think, to implement both a full-featured >> replacement for setuptools entry point API, and allow simple > > What do you feel is missing in terms of functionality? What I was saying is that starting from a base of arbitrary JSON (as Nick seemed to be proposing) would make it *harder* to provide the simple functionality. Not that adding JSON is needed to support setuptools functionality. Setuptools does just fine with plain export strings! I don't want to lose that simplicity; the "export string or JSON" suggestion was a compromise counterproposal to Nick's "let's just use arbitrary JSON structures". > I think the thing here is to identify what the components in the build > system would be (as an abstraction), how they would interact etc. If we look > at how the build side of distutils works, it's all pretty much hardcoded > once you specify the inputs, without doing a lot of work to subclass, > monkey-patch etc. all over the place. It's unnecessarily hard to do even > simple stuff like "use this set of compilation flags for only this specific > set of sources in my extension". In any realistic build pipeline you'd need > to be able to insert components into the pipeline, sometimes to augment the > work of other components, sometimes to replace it etc. and ISTM we don't > really know how any of that would work (at a meta level, I mean). I was assuming that we leave build tools to build tool developers. If somebody wants to create a pipelined or meta-tool system, then projects that want to use that can just say, "I use the foobar metabuild system". For installer-tool purposes, it suffices to say what system will be responsible, and have a standard for how to invoke build systems and get wheels or the raw materials from which the wheel should be created. *How* this build system gets the raw materials and does the build is its own business. It might use extensions, or it might be setup.py based, or Makefile based, or who knows whatever else. That's none of the metadata PEP's business, really. Just how to invoke the builder and get the outputs. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Changing the "install hooks" mechanism for PEP 426
On Thu, Aug 15, 2013 at 9:21 AM, Nick Coghlan wrote: > > On 15 Aug 2013 00:39, "Vinay Sajip" wrote: >> >> PJ Eby telecommunity.com> writes: >> >> > The build system *should* reserve at least one (subdivisible) >> > namespace for itself, and use that mechanism for its own extension, >> >> +1 - dog-food :-) > > Sounds fair - let's use "pydist", since we want these definitions to be > somewhat independent of their reference implementation in distlib :) I think that as part of the spec, we should either reserve multiple prefixes for Python/stdlib use, or have a single, always-reserved top-level prefix like 'py.' that can be subdivided in the future. Extensions are a honking great idea, so the stdlib will probably do more of them in the future. Likewise, future standards and informational PEPs will likely document specific extension protocols of general and specialized interest. (Notice, for example, that extensions could be used to publicize what database drivers are installed and available on a system.) > Based on PJE's feedback, I'm also starting to think that the > exports/extensions split is artificial and we should drop it. Instead, there > should be a "validate" export hook that build tools can call to check for > export validity, and the contents of an export group be permitted to be > arbitrary JSON. I think there is still something to be said for STASCTAP: simple things are simple, complex things are possible. (Also, flat is better than nested.) So I would suggest that an export can either be an import identifier string, *or* a JSON object with arbitrary contents. That would make it easier, I think, to implement both a full-featured replacement for setuptools entry point API, and allow simple extensions to be simple. It means, too, that simple exports can be defined with a flatter syntax (ala setuptools' ini format) in tools that generate the JSON. Given how many use cases are already met today by providing import-based exports, ISTM that they are the 20% that provides 80% of the value; arbitrary JSON is the 80% that only provides 20%, and so should not be the entry point (no pun intended) for people dealing with extensions. Removing the extension/export split also raises a somewhat different question, which is what to *call* them. I'm sort of leaning towards "extensions" as the general category, with "exports" being extensions that consist of an importable object, and "JSON extensions" for ones that are a JSON mapping object. So the terminology would be: Extension group - package like names, subdivisible as a namespace, should have a prefix associated with a project that defines the semantics of the extension group; analagous to Eclipse's notion of an "extension point" Extension name - arbitrary string, unique per distribution for a given group, but not required to be globally unique even for the group. Specific names or specific syntax for names may be specified by the creators of the group, and may optionally be validated. Extension object - either an "export string" specifying an importable object, or a JSON object. If a string, must be syntactically valid as an export; it is not, however, required to reference a module in the distribution that exports it; it *should* be in that distribution or one of its dependencies, however. So, an extension is machine-usable metadata published by a distribution in order to be (optionally) consumed by other distributions. It can be either static JSON metadata, or an importable object. The semantics of an extension are defined by its group, and other extensions can be used to validate those semantics. Any project that wants to be able to use plugins or extensions of some kind, can define its own groups, and publish extensions for validating them. Python itself will reserve and define a group namespace for extending the build and installation system, including a sub-namespace where the validators can be declared. > So we would have "pydist.commands" and "pydist.export_hooks" as export > groups, with "distlib" used as an example of how to define handlers for > them. Is 'commands' for scripts, or something else? Following "flat is better than nested", I would suggest not using arbitrary JSON for these when it's easy to define new dotted groups. (Keeping to such a style will make it easier for humans to define this stuff in the first place, before it's turned into JSON.) (Note, btw, that having more dots in a name does not necessarily equal "nested", whereas replacing those dots with nested JSON structures most definitely *is* "nested"!) Similarly, I'd just as soon see e.g. pydist.hooks.* subgroups, rather than a dedicated data structu
Re: [Distutils] How to handle launcher script importability?
On Wed, Aug 14, 2013 at 2:14 PM, Paul Moore wrote: > .pyw files can be imported as modules, just like .py, Darn. Okay, so another extension *is* needed if you want to be able to make non-console apps runnable-but-not-importable. IIUC it should by '.pywa' rather than '.pya', though, because of the issue with only the first three characters of an extension working in PowerShell, which means it would be executed by the wrong PyLauncher under some circumstances. (Honestly, though, I'm not sure whether anybody cares about PATH/PATHEXT in relation to GUI apps; ISTM they'd mostly be invoked by shortcuts, and there's also a registry mechanism that's supposed to be used for these things nowadays, rather than PATH... but I think it only works for .exe's, so there we go again back to the land of .exe's just plain Work Better On Windows.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Changing the "install hooks" mechanism for PEP 426
On Wed, Aug 14, 2013 at 3:14 PM, Nick Coghlan wrote: > On 14 August 2013 14:00, PJ Eby wrote: >> On Wed, Aug 14, 2013 at 11:36 AM, Nick Coghlan wrote: >>> * group - name of the export group to hook >>> * preupdate - export to call prior to installing/updating/removing a >>> distribution that exports this export group >>> * postupdate - export to call after installing/updating/removing a >>> distribution that exports this export group >>> * refresh - export to call to resynchronise any caches with the system >>> state. This will be invoked for every distribution on the system that >>> exports this export group any time the distribution defining the >>> export hook is itself installed or upgraded >> >> I've reread your post a few times and I'm not sure I understand it. >> Let me try and spell out a scenario to see if I've got it: >> >> * Distribution A defines a refresh hook for group 'foo.bar' -- but >> doesn't export anything in that group >> * Distribution B defines an *export* (fka "entry point") -- any export >> -- in export group 'foo.bar' -- but doesn't define any hooks >> * Distribution A's refresh hook will be notified when B is installed, >> updated, or removed > > No, A's preupdate and postupdate hooks would fire when B (or any other > distro exporting the "foo.bar" group) is installed/updated/removed. > refresh() would fire only when A was installed or updated. Huh? So refresh is only relevant to the package itself? I guess I don't understand the point of that, since you get the same info from postupdate then, no? > I realised that my proposed signature for the refresh() hook is wrong, > though, since it doesn't deal with catching up on *removed* > distributions properly. Rather than being called multiple times, > refresh() instead needs to be called with an iterable providing the > metadata for all currently installed distributions that export that > group. Ah. But then why is it called for A, instead of.. oh, I think I see now. Gotcha. This is the sort of thing that examples are really made for, so you can see the use cases for the different hooks. >> If so, my confusion is probably because of overloading of the term >> "export" in this context; among other things, it's unclear whether >> this is a separate data structure from exports themselves... and if >> so, why? > > Where "exports" is about publishing entries into an export group, the > new "export_hooks" field would be about *subscribing* to an export > group and being told about changes to it. That's not actually a justification for not using exports. > While you could use a naming convention to defined these hooks > directly in "exports" without colliding with the export of the group > itself, but I think it's better to separate them out so you can do > stricter validation on the permitted keys and values (the rationale is > similar to that for separating out commands from more general exports, > and exports from arbitrary metadata extensions). The separation of commands is (just barely) justifiable because it's not a runtime use, it's installer use. Stricter validation, OTOH, is a completely bogus justification for not using exports, otherwise nobody would ever have any reason to use exports, everybody would have to define their own extensions so they could have stricter validation. ;-) The solution to providing more validation is to use *more* export groups, e.g.: [mebs.export_validators] mebs.refresh = module.that.validates.keys.in.the.refresh.group:somefunc (In other words, define hooks for validating export groups, the way setuptools uses an entry point group for validating setup keywords.) Of course, even without that possibility, the stricter validation concept is kind of bogus here: the only thing you can really validate is that syntactically valid group names are being used as export names, which isn't much of a validation. You can't *semantically* validate them, since there is no global registry of group names. So what's the point? The build system *should* reserve at least one (subdivisible) namespace for itself, and use that mechanism for its own extension, for two reasons: 1. Entities should not be multiplied beyond necessity, 2. It serves as an example of how exports are to be used, and 3. The API is reusable... No, three reasons! Wait, I'll come in again... the API is reusable, it serves as an example, no duplication, and namespaces are a good idea, let's do more of them... no, four reasons... chief amongst the reasonry... Seriously: I can *sort of* see a reason to keep comma
Re: [Distutils] Changing the "install hooks" mechanism for PEP 426
On Wed, Aug 14, 2013 at 11:36 AM, Nick Coghlan wrote: > * group - name of the export group to hook > * preupdate - export to call prior to installing/updating/removing a > distribution that exports this export group > * postupdate - export to call after installing/updating/removing a > distribution that exports this export group > * refresh - export to call to resynchronise any caches with the system > state. This will be invoked for every distribution on the system that > exports this export group any time the distribution defining the > export hook is itself installed or upgraded I've reread your post a few times and I'm not sure I understand it. Let me try and spell out a scenario to see if I've got it: * Distribution A defines a refresh hook for group 'foo.bar' -- but doesn't export anything in that group * Distribution B defines an *export* (fka "entry point") -- any export -- in export group 'foo.bar' -- but doesn't define any hooks * Distribution A's refresh hook will be notified when B is installed, updated, or removed Is that what this is for? If so, my confusion is probably because of overloading of the term "export" in this context; among other things, it's unclear whether this is a separate data structure from exports themselves... and if so, why? If I were doing something like this in the existing entry point system, I'd do something like: [mebs.refresh] foo.bar = my.hook.module:refresh i.e., just list the hooks in an export group, using the export name to designate what export group is being monitored. This approach leverages the fact that exports already need to be indexed, so why create a whole new sort of metadata just for the hooks? (But of course if I have misunderstood what you're trying to do in the first place, this and my other thoughts may be moot.) (Oh, and btw, if a distribution has hooks for itself, then how are you going to invoke two different versions of the code? Rollback sys.modules and reload? Spawn another process?) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Wed, Aug 14, 2013 at 10:33 AM, Nick Coghlan wrote: > If you prepend the shebang line, you get a ".pya" file (Note: I > suggested to Daniel that the extension be changed to "pya" for "Python > application", but the PEP hasn't been updated yet), if you prepend the > executable, you get an actual Windows executable. The shebang line, or some equivalent is still needed, even if you're prepending an .exe. (It does need to be able to find Python, after all.) The wrappers, I think, should also be updated to support the .deleteme protocol I described earlier, so that if, upon exiting, the program finds it is named .deleteme, it should respawn Python with a -c scriptlet to delete the .deleteme file, and immediately exit without waiting for the result. That way, pip and other installation tools can update their own .exe files, without leaving any garbage behind. With this overall approach, .exe's can remain the default choice of script wrapping, with .pya's available as an option for those who want them. PyLauncher should include the .pya association and PATHEXT, and people who want to write or edit scripts by hand can use that extension. (Of course, using .pya won't work with subprocess.call and other CreateProcess-y things, which incidentally reminds me that I've had some royal pains in the past trying to get other applications to invoke Python scripts or indeed *any* language scripts. For example the Calibre Windows app's "Open With..." plugin requires an .exe... and it's *written* in Python, for heaven's sakes! So .exe headers and extensions for .pya files really ought to be the default option on Windows, for the sake of users' sanity. Plain-text .pya files are a developer/quick-and-dirty feature that you don't use for scripts on Windows that are invoked by anything besides other scripts.) > That would also let > us avoid the need for a separate ".pyaw" extension - you would just > prepend a Windows GUI executable to handle that case, with .pya only > handling console applications. Shouldn't naming the file .pyw already work today for that case? Certainly, the .pyw extension is already suitable for manually creating GUI scripts in a text editor. Unless there's something special about how the 'pythonw' executable processes the command line, it should work just as well for a zipped archive. So, probably no need to have a separate extension in either case. (But maybe somebody should verify that 2.6+ on Windows does indeed run .zip files named with .pyw and double-clicked on.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Wed, Aug 14, 2013 at 9:58 AM, Vinay Sajip wrote:
> IIUC PEP 441 is about tooling to create archives; don't we just need a
> Python-compatible .zip (i.e. with a __main__.py)?
I meant that it has a #! line before the zip part, so that the
launcher knows what Python to invoke.
There are also some challenges for older Pythons to invoke __main__,
since the normal Python import machinery frowns on reloading __main__.
I expect the zip would need an extra __main.py stub to bootstrap the
loading of __main__, and then invoke python with something like '-c
"__import__('sys').path[0:0]=['/path/to','path/to/exe''];
__import__('__main').go()"'.
(It can't have the import run the app as a side effect, because
otherwise the import lock will be held, leading to Bad Things in
multi-threaded apps.)
> This is less helpful; one might have N scripts per project, no need to stick
> the whole project in with each one, or am I misunderstanding?
I just meant that for cases where there's only one script, or where
you are doing a custom-built application. This also becomes The One
Obvious Way to do py2exe-like things.
> How would such an offset be used? Are you saying the -c scriptlet would use
> that offset to extract the script? Or do you mean something else?
Extract the script by seeking to the offset and reading it. It's far
from ideal, though; the .zip is much better since everything back to
2.3 can support it in some fashion.
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Wed, Aug 14, 2013 at 7:34 AM, Vinay Sajip wrote: > Jason R. Coombs jaraco.com> writes: > >> This means that instead of installing, for example: >> >> Scripts\my-command.exe >> Scripts\my-command-script.py >> Scripts\my-command.exe.manifest >> > > Just to muddy the waters a little, I'd like to suggest an alternative > approach which doesn't appear to have been tried: > > 1. The installer just installs foo.exe on Windows for script 'foo', >where the foo.exe contains the actual script 'foo' as a resource. > > 2. The launcher, before looking for foo-script.py, examines its resources. >If a script resource is found, it is extracted and written to >foo-script.py in the same directory as foo.exe. If such a resource isn't >found, it continues to the next step. > > 3. The launcher looks for 'foo-script.py' in its directory, invokes it and >waits for it to complete. > > 4. If a 'foo-script.py' was written in step 2, it is deleted. > > The launcher comes with an embedded manifest, so no external manifest is > needed. Better suggestion: just append a PEP 441 .pyz to the .exe, and no extraction is necessary; the .exe just reads out the #! part. For Python 2.6 and up, the .exe can simply pass itself as argv[1] to the interpreter. (For older Pythons, a little -c and PYTHONPATH munging is required, but should be doable.) For bonus points, you can actually stick a compatibly-built wheel on the end of the .exe instead, and embed the entire relevant project. ;-) > Thoughts? Writing the script.py file means the current user needs write access to a program installation directory, which is probably not a good idea. Also, what if two instances are running, or you overwrite an existing script while it's being read by Python in another process? No, if you're taking the embedding route, it's got to be either a zipfile, or else you have to use -c and give Python an offset to seek to in the file. In any case, it'd probably be a good idea to offer some command line tools for manipulating such .exes, to e.g. show/change what Python it's set to use, extract/dump/replace the zip, etc. (As for ctypes, if that's needed for this approach (which I somewhat doubt), there are official Windows binaries available for 2.3 and 2.4.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Tue, Aug 13, 2013 at 12:33 PM, Paul Moore wrote: > This works, but is an ugly, fragile workaround. It's *not* a huge problem, > it's just how executables work on Windows, and all installers have to deal > with this dance (it's why a lot of things need a reboot to complete > installation - the "delete on next reboot" API). But it's not *nice*. > > I would never use exe wrappers for my own personal scripts - I *always* > write them as .py files and rely on PATHEXT. I only use exe wrappers for > commands installed as part of a Python package (pip.exe, nosetests.exe, > etc). That says something about how friendly they are as a general tool. In an ironic reversal, I use them for any command I plan to use frequently. In other words, if I use it often enough to care about how easy it is to use, I take the trouble to wrap it in a project and then use setup.py develop to create the script wrappers. From then on, I can edit the *source* scripts, and the wrappers run the right thing. (I don't edit the -script.py's directly, since they're not where the real code is.) > On another point you mention, Cygwin Python should be using Unix-style shell > script wrappers, not Windows-style exes, surely? The whole point of Cygwin > is that it emulates Unix, after all... So I don't see that as an argument > either way. I said I'm using *Windows* Python from the Cygwin shell. I often test my projects with Cygwin Python, to ensure coverage of Unixisms, but I only write dedicated Cygwin Python scripts if I need to use Cygwin paths or APIs, which is relatively infrequent. In any case, the use of .exe means that my invocation patterns are unchanged between commands I've implemented in Cygwin Python vs. Windows Python. If the Windows Python versions used a different extension, then I'd have to remember whether which language a specific command was written in in order to invoke it. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Tue, Aug 13, 2013 at 8:54 AM, Jason R. Coombs wrote:
> 1. Renames, deletes, and other actions must be synchronized.
Why are you manually deleting or altering executables? Why are you
renaming them at all?
I've been using .exe wrappers since they were written, and have never
had a single one of the issues you mention, because I never do any of
the things you mention by hand. IMO that's what tools are for.
Doesn't pip uninstall scripts?
I may be slightly biased in my preference for .exe, because files with
other extensions don't work with Cygwin (which doesn't support
PATHEXT), but I work primarily with Windows Python rather than Cygwin
Python. So, if there *has* to be a single file, I would greatly
prefer an .exe with the script embedded, rather than a non-.exe file.
It's a bit less discoverable, but at least it'll discourage anybody
from editing the contents. (Because nobody should be editing
generated scripts anyway.)
(Also relevant: not every situation where wrapper scripts are used is
going to be one where a PyLauncher install is possible. For example,
portable deployment of an app to USB stick with a bundled Python can't
assume PATHEXT and a globally-installed PyLauncher.)
> 4. Updates to the launcher won't apply to existing scripts. If the launcher is
> updated, the side-by-side versions will remain out-of-date until their scripts
> are re-installed.
This is kind of a bogus point; *any* update to how scripts are
generated isn't automatically applied to existing scripts; the format
in which they're written is of no relevance.
> 5. Files in use can't be replaced. Because a Windows executable that's in use
> is not allowed to be overwritten,
But they can be renamed, and deleted afterwards. For example, when
updating, you can do the simple dance of:
1. delete scriptname.exe.deleteme if it exists
2. rename scriptname.exe to scriptname.exe.deleteme
3. replace scriptname.exe
4. try to delete the .deleteme file created in step 2, ignoring errors.
And since this only needs to be done for the wrappers on installation
tools themselves (pip, easy_install, etc.), it's not like a lot of
people are going to have to write this code.
It can also be further enhanced, by having the .exe wrapper check (as
it exits) whether it was renamed, and if so, spin off a 'python -c
"import os, time; time.sleep(0.1); os.unlink('path to .deleteme')"'
and immediately exit. (Or use one of the other tricks from
http://www.catch22.net/tuts/self-deleting-executables -- but I think
this one is the simplest and best for our purposes, since the wrapper
already knows at this point it can invoke Python using the path it
previously found, and it's not doing anything questionable with
process invocations that might raise red flags with security tools.)
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Mon, Aug 12, 2013 at 4:29 PM, Donald Stufft wrote: > Hopefully this all will solve this problem, as it is right now if you use > setuptools entry points then Wheels erroneously pretend to be platform > agnostic. IMO it's okay to give up having ready-to-use scripts in a platform-agnostic wheel; scripts are sadly not a platform-agnostic thing. (If only MS-DOS had been a little bit more Unix, and a little less CP/M...) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Mon, Aug 12, 2013 at 4:04 PM, Paul Moore wrote: > I would still like to see the standard be registered .pye (I'm happy with a > bikeshed of this colour) and .pwe extensions which are added to PATHEXT and As long as we're discussing bikeshed colors, I'd like to counterpropose .pya and .pwa, to be registered in the Windows class registry as Python Console Application and Python Windowed Application, respectively. Since the *only* reason we need these extensions is for Windows (other OSes do fine at making things executable without an extension), and Windows calls things like these "Applications" or "Apps" in Explorer normally, I think it's better to call them what Windows calls them. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Mon, Aug 12, 2013 at 2:14 PM, Jason R. Coombs wrote: > > >> -Original Message- >> From: Distutils-SIG [mailto:distutils-sig- >> bounces+jaraco=jaraco@python.org] On Behalf Of PJ Eby >> Sent: Monday, 12 August, 2013 11:22 >> >> On Mon, Aug 12, 2013 at 10:32 AM, Paul Moore >> wrote: >> > On 12 August 2013 14:01, PJ Eby wrote: >> > >> > As far as zipped Python applications are concerned (pyz), these can be >> > created by just using a pys file containing a #! line prepended to the >> > zip file. Certainly, it's a binary file with a filename that would >> > normally indicate a text file format, but is that any less true on >> > Unix when users create these files? I don't know what the user >> > experience with zipped Python applications on Unix is like - I doubt >> > it's *that* much better than on Windows. Probably the reality is that >> > nobody uses zipped applications anyway, so the problems haven't been >> > identified yet. Maybe the pyz PEP would bet better rewritten to >> > propose providing tools to create and manage zipped Python >> > applications, but *not* to require new extensions, merely to reuse >> > existing ones (pys on Windows, no extension on Unix) with binary (zipped) >> content. >> >> Seems reasonable... but then somebody will need to write another PEP for >> the file extension(s) issue. > > My preference is to reject the idea of the side-by-side executable launcher. > There are several downsides that I'm trying to avoid by moving away from the > executable: > > 1. Disparity with Unix. Better parity means cleaner code, easier > documentation, and less confusion moving from platform to platform. > 2. Executables that look like installers. If a launcher executable is used and > Windows detects that it "looks like" an installer and it's a 32-bit executable > and it doesn't have a manifest to disable the functionality, Windows will > execute the file in a separate UAC context (often in a separate Window). > 3. Additional files are needed. In particular, due to (2), a manifest must be > provided for 32-bit executables. > 4. Word size accounting. It's not clear to me what word size is needed. 32-bit > may be sufficient, though 64-bit seem to have some advantages: a manifest is > not needed, and it can match the word size of the installed Python executable > (for consistency). Setuptools currently provides both (and installs the one > that matches the Python executable). > 5. Platform support. We're fortunate that Windows is one of the most stable > binary platforms out there. Nevertheless, Setuptools recently got support for > AMD binaries in the launcher. By relying on an external launcher, the launcher > becomes responsible for platform support. > 6. Two to three files to do the job of one. In fact, the "job" isn't much more > than to invoke code elsewhere, so it seems ugly to require as many as three > files to do the job. Then multiply that by the Python-specific version and you > have up to six files for a single script. > 7. Obfuscation of purpose. A single script pretty directly communicates its > purpose. When there are multiple files, it's not obvious why they exist or > what their purpose is. Indeed, I went years without realizing we had an open > issue in Distribute due to a missing manifest (which was fixed in Setuptools), > all because I used the 64-bit executable. While it may take some time for the > community to learn what a '.pyl' is, it's easily documented and simple to > grasp, unlike the subtle and sometimes implicit nuances (and fragility) of a > side-by-side executable. > 8. Unwanted content. Some Unix users have complained about finding Windows > executables in their Linux packages, so now Setuptools has special handling to > omit the launchers when installed on Unix systems. This is far from beautiful. > >> I think the issue of "too many extensions" vs. "source/binary confusion" is >> going to boil down to a BDFL judgment call, whether it's by Nick, Guido, or >> some more Windows-specific BDFL For One PEP. >> >> If we go with One Extension To Rule Them All, I would actually suggest >> '.pyl' >> (for PyLauncher), since really all that extension does is say, "hey, run >> this as a >> console app via PyLauncher", not that it's a "script" (which would be >> assumed to be text). And that all you can be sure of is that a .pyl files >> will >> start with a #! line, and launch whatever other program is specified there, >> on
Re: [Distutils] How to handle launcher script importability?
On Mon, Aug 12, 2013 at 10:32 AM, Paul Moore wrote: > On 12 August 2013 14:01, PJ Eby wrote: > > As far as zipped Python applications are concerned (pyz), these can be > created by just using a pys file containing a #! line prepended to the zip > file. Certainly, it's a binary file with a filename that would normally > indicate a text file format, but is that any less true on Unix when users > create these files? I don't know what the user experience with zipped Python > applications on Unix is like - I doubt it's *that* much better than on > Windows. Probably the reality is that nobody uses zipped applications > anyway, so the problems haven't been identified yet. Maybe the pyz PEP would > bet better rewritten to propose providing tools to create and manage zipped > Python applications, but *not* to require new extensions, merely to reuse > existing ones (pys on Windows, no extension on Unix) with binary (zipped) > content. Seems reasonable... but then somebody will need to write another PEP for the file extension(s) issue. I think the issue of "too many extensions" vs. "source/binary confusion" is going to boil down to a BDFL judgment call, whether it's by Nick, Guido, or some more Windows-specific BDFL For One PEP. If we go with One Extension To Rule Them All, I would actually suggest '.pyl' (for PyLauncher), since really all that extension does is say, "hey, run this as a console app via PyLauncher", not that it's a "script" (which would be assumed to be text). And that all you can be sure of is that a .pyl files will start with a #! line, and launch whatever other program is specified there, on the contents of the file -- which may actually be a zipfile. > PS Either the ref file marker approach, or a new Python command line > argument with appropriate behaviour, could avoid the need for even the > pys/pws extension, if people prefer to reduce the number of extensions > claimed still further. But those would only be available for future Python versions. A file extension would solve the problem upon installing PyLauncher and PATHEXT, at least for those OSes and shells that recognize PATHEXT. Hm, here's a side thought: what if PyLauncher added the ability to serve as a script wrapper, just like setuptools' existing wrappers? Then setuptools could just copy py.exe or pyw.exe alongside a .pyl or .pyw, and presto! No PATHEXT compatibility needed, but users could still opt out of using the .exe wrappers if they're sure their shell works right without it. (The wrapper facility would be implemented by simply checking for an adjacent file of matching filename and extension (.pyl for py.exe, .pyw for pyw.exe), and if found, insert that filename as argv[1] before proceeding with the normal launch process. For efficiency, the file check could be skipped if the executable has its original name, at the minor cost of it not being possible to name a console script 'py' or a windows app 'pyw'. But that's an optional tweak.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Mon, Aug 12, 2013 at 7:33 AM, Nick Coghlan wrote: > Having pys and pyz for "executable, but not importable" (source and zip > archive forms) could be quite clean. In effect, the pys extension would > bring windows to parity with *nix, where "no extension at all" has > traditionally served the purpose of making it impossible to import a script. Oh, that reminds me: IIUC, it's not necessary to *actually* zip a .pyz. Remember, Python doesn't use the extension to determine the contents, it sniffs for a zip trailer. Likewise, there was IIRC no plan for pylauncher to inspect zip contents -- it just reads the #! line and runs python on the file. This means that you can actually write source as a .pyz or .pwz file on Windows, and it would Just Work -- *without any sys.path modification*. For *all versions of Python*, provided PyLauncher is installed. (And setuptools could check the environment and registry for that, if you opt into the non-.exe scripts, and error out if you don't have them set up correctly.) IOW, implementing PEP 441 in PyLauncher gives us the "executable, not importable" format for free. (Granted, I can see reasons for not wanting to use the same extension for source and zipped versions, mostly in the area of tools other than pylauncher, but if you do have different extensions then there have to be *four*, due to console vs. windowed and source vs. zipped.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Sun, Aug 11, 2013 at 7:31 PM, Jason R. Coombs wrote: >> -Original Message- >> From: Nick Coghlan [mailto:ncogh...@gmail.com] >> Sent: Sunday, 11 August, 2013 17:14 >> >> We actually have a proposal on import-sig to allow module specific import >> path manipulation (including the ability to say "don't import this module >> from this directory, even though it looks like it is here"). I'd favour that >> mechanism over a new "not importable" file extension. > > I don't believe this mechanism would suffice. My previous example was > over-simplified to the general problem, which is that any script could > potentially be imported as a module of the same name. So if I were to launch > easy_install.py, it would set sys.path[0] to Scripts\ and if it were then to > import cython (which it does), it would import Scripts/cython.py as cython, > unless there were some way to globally declare all installed scripts somewhere > so they're excluded from import. Indeed. It really *does* need to be a "don't import this" extension, though it doesn't much matter what that extension is. Except on Windows, of course, where it has to be something associated with Python that also still works as a console app and is listed in PATHEXT. (As you surmised earlier, my choice of '-script.py' was indeed chosen to prevent accidental importing, as the '-' ensures it's not a valid module name.) >> If that doesn't make it into 3.4, the proposed zipapp extensions would also >> serve a similar purpose, with some straightforward sys.path manipulation in >> __main__.py (as PJE pointed out). > > Regardless what solution might be made available for Python 3.4, I'd prefer to > work toward a solution that leverages pylauncher under older Pythons. After > all, one of the huge benefits of pylauncher is that it supports multiple > Pythons. If zipapp is the preferred mechanism for that, then so be it. For 2.6+, zipapps would work as long as pylauncher supported them and put the requisite extensions in PATHEXT. > This approach also means that the script generation is not congruent with that > on Unix systems. Using a zipapp means that the whole script generation needs > to be special-cased for Windows. One of great benefits of using a simple > script was that the code becomes largely unified (only requiring appending of > an extension when on Windows). That is, unless zipapps can be made executable > on Unix as well. They're already executable on Unix (as of 2.6+), as they contain a #! line. And they don't need a special extension; on both Unix and Windows, Python detects zipapps by inspecting the tail signature, not by the extension. (Of course, you could just continue using the existing wrapper mechanism on Unix, which has the advantage of added transparency, at the cost of having two code paths.) > Given these obstacles, do you still feel that zipapp is the best approach? Long-term, yes. I would slightly prefer the "this is a script" extension, though, as it has the extra transparency benefit. (Still, nobody should be editing an installed script anyway.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Sun, Aug 11, 2013 at 1:58 PM, Jason R. Coombs wrote: > This sounds like a suitable idea, but as you mention in a subsequent > message, this format has issues with sys.path assumptions as well. Meh. It's basically, a one-line fix in the __main__.py, i.e.: import sys,os.path; sys.path[0] = os.path.dirname(sys.path[0]) Not exactly rocket science. ;-) > In this case, I'm inclined to suggest yet another option (7) - create another > extension to specifically represent executable but not importable scripts, > perhaps .pys/.pysw (or .pycs/.pygs to more closely match console script and > gui script). Probably .pys/.pyws or .pws would be needed, due to issues with some Windows shells using extensions longer than three characters. (This came up in PEP 441 discussions on Python-Dev.) > It sounds as if there is a fundamental need for Python to define an > extension that distinguishes a script from a module. Yep. > I am aware of the PATHEXT factor. I personally add .py and .pyw to the > PATHEXT (for all users) on my systems, so I was unsure if Python 3.3 did add > those or if pylauncher would add them (if installed separately). I was > _hoping_ that was the case, but it sounds like it is not. I did include in > the documentation notes about this requirement > (https://bitbucket.org/pypa/setuptools/src/1.0b1/docs/easy_install.txt#cl-98 > ). > > I do want to explore the possibility of setuptools facilitating this > configuration such that it's easy for a Windows user to enable these > settings even if Python does not. It would definitely make sense to have an installer that sets this up, but it would need to be a Windows installer, I think, not a Python program. That is, I don't think setuptools can really do anything about it directly. Personally, I'm not sure I see the point of pushing for early elimination of .exe's - they don't depend on the registry or environment variables or anything else, which makes them great for standalone applications, and they work across all Python versions. Meanwhile, the experimental nature of your change -- and its inability to be the default on versions below 3.4 -- means you're going to be maintaining two sets of code for a very long time. OTOH, implementing a way to deploy an app as a .pyz/.pwz file is a useful feature in its own right, so it might not be doubling up as much. ;-) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Sun, Aug 11, 2013 at 12:17 PM, PJ Eby wrote: > May I suggest an option 5 instead? Use the new .pyz (or .pyzw for > non-console apps) as a zipped Python application. .pyz files aren't > importable, but *are* executable. That's basically all that's needed > to prevent importing -- a file extension that's launchable but not > importable. (Details I forgot to mention: the script would be in __main__.py inside the zipped application file, and it would need to change sys.path[0], because sys.path[0] will be the .pyz file itself; it should replace it with the directory containing the .pyz file before doing anything else. That would be the correct way to simulate the existing .exe approach.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Sun, Aug 11, 2013 at 10:38 AM, Jason R. Coombs wrote: > In Setuptools 1.0 (currently in beta), I've added an experimental, opt-in > feature to install pure Python launcher scripts on Windows instead of > installing a launcher executable for each script, with the intention that > these scripts will be launched by pylauncher or Python directly, eventually > obviating the need for a launcher executable in setuptools at all. > > This means that instead of installing, for example: > > Scripts\my-command.exe > Scripts\my-command-script.py > Scripts\my-command.exe.manifest > > Instead Setuptools just installs: > > Scripts\my-command.py > > This technique is much like the scripts that get installed to bin/ on Unix, > except that due to the nature of launching commands on Windows, the .py > extension is essentially required. > > One problem with this technique is that if the script is also a valid module > name, it can be imported, and because Python puts the script's directory at > the top of sys.path, it _will_ be imported if that name is imported. > > This happens, for example, after installing Cython. Cython provides a > command, 'cython', and a (extension) module called 'cython'. If one launches > cython using the script launcher, the 'cython' module will not be importable > (because "import cython" will import the launcher script). Presumably, this > is why '-script' was added to the launcher scripts in previous versions. > > This is a rather unfortunate situation, and I'd like to solicit comments for > a way to avoid this situation. I see a few options: > > 1. Have the setuptools-generated launcher scripts del sys.path[0] before > launching. > 2. Accept the implementation and file bugs with the offending projects like > Cython to have them either rename their script or rename their internal > module. > 3. Continue to generate the script names with '-script.py' appended, > requiring invocation to always include -script.py on Windows. > 4. Continue to generate executables, duplicating the effort of pylauncher, > and dealing with the maintenance burden of that functionality. > > I don't see (2), (3), or (4) as really viable, so my proposal is to move > forward with (1) if there aren't any better suggestions. > > If we move forward with (1), there are a few concerns that come to mind. Here's another problem with #1: you will break single-directory standalone portable app installs, where you use "easy_install -mad somedir" to install all of an app's dependencies to a single directory that the app can then be run from (assuming Python is available). In order to work around this issue, you'd need to hardcode sys.path entries for the dependencies, or do something else more complicated in order to ensure that dependency resolution will pick up the adjacent distributions before searching anything else on sys.path. > Third, is it possible some users are depending on the presence of > sys.path[0] Absolutely. It's a documented feature of Python that the script directory is always first on sys.path, so that you can provide modules and packages adjacent to it. That's how portable app installs work with easy_install. May I suggest an option 5 instead? Use the new .pyz (or .pyzw for non-console apps) as a zipped Python application. .pyz files aren't importable, but *are* executable. That's basically all that's needed to prevent importing -- a file extension that's launchable but not importable. (There's also an option 6, which is to use import system hooks to prevent the script modules from being found in the sys.path[0] entry, but that's rather hairier.) Using option 5 means the feature can only work with versions of Python on Windows that install the necessary PATHEXT support to allow that extension to work, but you're kind of limited to that anyway, because by default .py files aren't findable via PATH on Windows. Your post doesn't make it clear whether you're aware of that, btw: IIUC, on most Windows setups, executing a .py file via PATH doesn't work unless you've set up PATHEXT to include .py. So your feature's going to break until that's fixed, and AFAIK there is *no* Windows Python that fixes this, with the possible exception of 3.4 alpha, possibly a future alpha that hasn't been released yet, because last I saw on Python-Dev it was still being discussed *how* to update PATHEXT safely from the installer. In short: dropping .exe wrappers is not supportable on *any* current version of Python for Windows, in the sense that anybody who uses it will not yet be able to execute the scripts if they are currently doing so via PATH (and haven't manually fixed their PATHEXT). (This was one of the main reasons for using .exe wrappers in the first place.) The .pyz approach of course has the same drawback, but at least it should be viable for future Python versions, and doesn't have the sys.path[0] problems. I think you are going to have to keep .exe wrappers the default for all Python versions < 3.4.
Re: [Distutils] Unexpected VersionConflict
On Fri, Aug 9, 2013 at 5:41 PM, Nick Coghlan wrote: > On 10 August 2013 04:06, PJ Eby wrote: >> Probably a better way would be to change the version resolution >> algorithm to be less "greedy", and simply rule out unacceptable >> versions as the process goes along, then picking the most recent >> versions left when everything necessary has been eliminated. >> (Ideally, such an algorithm would still track which distributions had >> the conflicting requirements, though.) > > The part I find most surprising is the fact that pkg_resources ignores > sys.path order entirely when choosing between multiple acceptable > versions. Technically, it doesn't ignore it: if a distribution is listed in sys.path, it takes precedence over any distribution listed later, or that has to be found *in* a directory on sys.path, and will in fact cause a VersionConflict if you ask for a version spec that it doesn't match. However, where the distributions aren't listed in sys.path, but merely *found in a directory on sys.path*, then sys.path has no bearing. It would make things a lot more complicated, and not just in an "implementation is hard to explain" kind of way. (In principle, you could write an Environment subclass that had a different precedence, but I'm not sure what benefit you would gain from the added complexity. The core version resolution algorithm wouldn't be affected, though, since it delegates the "find me something I haven't already got on sys.path" operation to an Environment instance's best_match() method.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Unexpected VersionConflict
On Fri, Aug 9, 2013 at 9:04 AM, Townsend, Scott E. (GRC-RTM0)[Vantage Partners, LLC] wrote: > That does indeed fix this problem, but requiring an egg writer to > interrogate all dependent packages (and their dependent packagesŠ) and > then hoist the dependencies up won't be robust if those dependent packages > change their requirements between the time the egg is written and the time > it's loaded. That's why it's generally left up to the application installer/integrator to address these sorts of conflicts, and why it's usually a bad idea for anybody to be requiring exact versions. (I'd suggest asking your dependency to not specify exact point releases, too.) There is one other possibility, though: have you tried reversing the list of your project's dependencies so that the more-specific project's dependencies are processed first? (i.e., so that 1.5.2 will be selected as "best" before the non-version-specific one is used) That might fix it without requiring you to pin a version yourself. > It seems to me that if a requirement has no version specified, then it > shouldn't have a way to cause a VersionConflict. One possible way of > implementing this would be to have resolve() only check that a > distribution exists if no version is specified, do not update 'best'. > 'to_activate' would need to be updated with 'generic' distributions only > if a requirement with a version specifier hadn't been seen. Thing is, the complete lack of a version requirement is pretty rare, AFAIK, and so is the exact version match that's causing your problem. The combination existing on the same library is therefore that much rarer, so such a change would just be something of a complex kludge that wouldn't improve any other use cases. Probably a better way would be to change the version resolution algorithm to be less "greedy", and simply rule out unacceptable versions as the process goes along, then picking the most recent versions left when everything necessary has been eliminated. (Ideally, such an algorithm would still track which distributions had the conflicting requirements, though.) That would be a pretty significant change, but potentially worth someone investigating. There are some big downsides, however: * It's not really a suitable algorithm for installation tools that don't have access to a universal dependency graph, because they can't tell what the next level of dependencies will be * Recursion causes a combinatorial explosion, because what if you select a different version and it has different dependencies (recursively)? Now you need backtracking, and there's a possibility that the algorithm will take a ridiculous amount of time to still conclude that there's nothing you can do about the conflict. These drawbacks are basically why I just wrote a simple greedy match algorithm in the first place, figuring it could always be improved later if it turned out to be needed in practice. There have been occasional comments over the last decade or so by people with ideas for better algorithms, but no actual code yet, as far as I know. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] help requested getting existing project "gv" into PyPI
On Fri, Aug 9, 2013 at 8:04 AM, Jess Austin wrote: > hi, > > I recently suggested that the "gv" package be added to PyPI. This is a > close-to-C Graphviz port distributed by the Graphviz graph visualization > project. They currently distribute this package in source, and in rpms, > debs, and the equivalents of those for Windows, Mac, and Solaris. It seemed > only natural that a small entry in PyPI be added so that "pip" would be able > to install this package in a straightforward way (i.e., without using "git" > references etc). > > The issue tracker url for this suggestion: > > http://www.graphviz.org/mantisbt/view.php?id=2324 > > As you can see at that link, the maintainer had some difficulty when trying > to create the project through the web form, culminating in a "Not found()" > error message. I unfortunately don't have a great deal of insight into this, > but I thought someone on this list might be able to help. > > Alternatively, please tell me if I'm making an unreasonable suggestion. It > was my understanding that "pip" is able to find installable files from even > quite rudimentary PyPI entries, even if they're hosted elsewhere. However, > if that's not the case, and the PyPI entry would require extensive ongoing > interaction from the "gv" maintainers, then this probably isn't going to > work. It would require ongoing interaction with *somebody*, in order to update the download links whenever a new version is released. Note, too, that unless there is a specific download for the Python binding that uses setup.py to invoke its build process, a PyPI listing won't make the binding pip-installable. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Unexpected VersionConflict
On Thu, Aug 8, 2013 at 3:19 PM, Townsend, Scott E. (GRC-RTM0)[Vantage
Partners, LLC] wrote:
> During easy_install of an egg where two versions of pyparsing were available
> (1.5.2 and 1.5.6), a VersionConflict was raised:
>
> pkg_resources.VersionConflict: (pyparsing 1.5.6
> (/usr/lib/python2.7/dist-packages), Requirement.parse('pyparsing==1.5.2'))
>
> This was unexpected since sys.path (via virtualenv) has version 1.5.2 before
> 1.5.6. And the system gets 1.5.2 from 'import pyparsing', not 1.5.6.
Have you tried declaring the 1.5.2 dependency from your main project?
IIRC, that should make it take precedence over either of the indirect
dependencies.
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Who administers bugs.python.org, and can we get a notice on the setuptools tracker?
On Mon, Aug 5, 2013 at 11:10 AM, Nick Coghlan wrote: > I filed the request on the metatracker: > http://psf.upfronthosting.co.za/roundup/meta/issue522 Thanks! That should keep me from having to keep telling people their princess is in another castle. ;-) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Who administers bugs.python.org, and can we get a notice on the setuptools tracker?
Hi. Not sure who this should go to, but it would be really good if we could get a prominent notice on the old setuptools tracker (at bugs.python.org), specifically on the issue creation screen, to inform people that this tracker is only for setuptools 0.6, and that issues for later versions should go to https://bitbucket.org/pypa/setuptools/issues instead (with a link, of course). Right now, what's happening is that, despite all the other prominent links to the correct tracker, there are still people going to the old tracker and posting issues for the newer versions of setuptools. This means they then have to wait for me to tell them they're in the wrong place, and then resubmit to the correct tracker. Setuptools 0.6 is still supported, so closing the tracker entirely isn't appropriate just yet, but putting up a prominent notice on the issue submission screen saying, "If you're reporting an issue for setuptools 0.7 or higher, please use". Thanks in advance! ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Last PEP 426 update for a while
On Sat, Aug 3, 2013 at 1:07 PM, Vinay Sajip wrote: > Nick Coghlan gmail.com> writes: > >> Just pushed these changes. I'm happy to leave the PEP alone for a >> while now, > > Thanks for doing these updates. Can I suggest the following corrections? > > 1. In the section "Exports", there is a dangling sentence which needs to be >completed: "The interpretation of the individual export keys is defined >by the distribution that i" > > 2. In the same section, it says "Export group names SHOULD correspond to >module names ..." and also "It is suggested that export groups be named >after distributions to help avoid name conflicts." It should be one of >these (I presume the former). And not quite the former, either; the same arguments about not splitting a distribution apply to modules as well. i.e., a single module might consume exports from more than one group, so saying they should correspond is too strong; I would say instead that export group names are dotted names that should be *prefixed* with the name of a package or module provided by the relevant distribution. (Of course, it's also perfectly fine for one to use a domain name or other similarly-unique prefix; the real point is just that top-level names should be reserved for groups defined by the stdlib and/or PEPs, and everybody else should be using unique prefixes that give some indication where one would look for a spec.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
On Tue, Jul 30, 2013 at 4:58 PM, Donald Stufft wrote: > Hrm. > > So I hear what you're saying and part of the problem is likely due to the > history > of where I tried to get a change through and then felt like all I was getting > was > stop energy and people wanting to keep the status quo which ultimately > ended up preventing changes has lead me to view distutils-sig in more of an > adversarial light then is probably appropriate for the distutils-sig of 2013 > (versus > the distutils-sig of 2011/2012). This is probably reflected in my tone and > likely > has others, such as yourself, respond similarly, pushing us further down that > path. My thought process has become "Ok here's something that needs to > happen, now how do I get distutils-sig not to prevent it from happening". Thanks for the thoughtful response. I appreciate it. I also want to just throw in one extra piece of information for you and anybody else reading this: 99% of "stop energy" doesn't happen because people actively want to prevent progress or frustrate other people. It simply happens when people notice a problem but don't have as much personal stake in your goal as they do in not experiencing the problem they will experience (or perceive they will), from the proposed change. When you look at it from this perspective, it's easier to understand that the way to fix this is with more engagement on their part, which can only be gotten by engagement on your part. When I first proposed WSGI, the initial reaction of Web-SIG was pretty negative. "Stop energy" if you will. Things only moved forward once I was able to channel the energy of objections into offering solutions. It's helpful to remember that asking, "okay, so how you would recommend I do it?" *doesn't* obligate you to actually follow all of the recommendations you get. (Especially since some of them will be mutually contradictory!) Anyway, I guess what I'm saying is that people lacking enthusiasm for your goals is not really them trying to stop you. In fact, objections are a positive thing: it means you got somebody's attention. The next step is to leverage that attention into actually getting help, or at least more constructive input. ;-) It's true that some individuals will never provide really helpful input. In the WSGI effort, there were people whose advice I never took, because their goals were directed entirely opposite to where I wanted to go. But I remained engaged until it was mutually clear (or at least I thought it was) that our goals were different, and didn't try to persuade them to go in the same direction. Such attempts at persuasion are pretty much a waste of time, and a big energy drain. Consensus-building is something that you do with people who have at least some goals in common, so it's best to focus on finding out what you *do* agree on. > This was again reflected in the Python 2.3 discussion because my immediate > reaction and impression was that you were attempting to block the move > from MD5 due to Python 2.3, which I felt 2.3 wasn't worth blocking > enhancements > to PyPI. The "snark" in my statements primarily came from that feeling of > someone was trying to "shut down" an enhancement. Right. In such a case, a question you could ask is, "Do you agree in general that we should move to a better hash at some point in the future?", because then the disagreement can be narrowed down to timeframe, migration or deprecation process, etc. The truth is, I had no intention of "blocking the move", I had concerns I wanted addressed about the impact, timing and process. (Actually, I originally just noticed a couple of errors in what you'd laid out as the current state of things, and wanted to make sure they were included in the assessment.) The point is, if somebody doesn't have *any* common ground with you, it's unlikely they're even talking to you. At the very least, if they're talking with you about PyPI, they must care about PyPI, even if they care about different things than you, or with different relative priorities. ;-) > As far as how to fix it I don't have a particularly magic answer. I will try > to be more > mindful of my tone and that distutils-sig is likely not my adversary anymore > as well > as try to ask questions instead of arguing the relevance immediately. Again, thank you. And hopefully, remember that probably nobody was intentionally being your adversary before, either. As the old adage says, the best way to destroy your enemies is to make friends with them. ;-) And we do that by focusing on common ground, and inviting participation. (This is again not to say that I've been 100% Mr. Wonderful myself; I know I haven't. But the community's best consensus-building happens when somebody is doing the tough work of engaging with all parties. Sometimes this doesn't happen, alas; back when I was developing setuptools there just weren't enough people interested in the problems available on Distutils-SIG to build a
Re: [Distutils] Last PEP 426 update for a while
On Fri, Aug 2, 2013 at 11:27 AM, Nick Coghlan wrote: > I pushed a version of PEP 426 with an initial sketch of an entry > points replacement design: http://hg.python.org/peps/rev/ea3d93e40e02 > > To give it a sensible home in the PEP, I ended up defining "modules" > and "namespaces" fields in addition to "commands" and "exports". The > overall section is called "Installed interfaces". > > I initially tried it with the unpacked multi-field mapping for export > specifiers, but ended up reverting to something closer to the > setuptools notation for readability purposes. For the moment, > "requires_extra" is included since it isn't that hard to explain. Thanks again for all the hard work in putting this together! Btw, under setuptools, entry point *group* names are restricted to valid Python module names, so this is a subset of valid distribution names. Conversely, entry *point* names are intentionally arbitrary and may contain anything that isn't an '=', as long as they don't start with a '#'. The reason for these choices is that entry point groups are used to ensure global uniqueness, but need a standard way for subdividing namespaces. (Notice that setuptools has groups like distutils.setup_arguments and distutils.setup_commands.) Conversely, individual entry point names have a free-form syntax so that they can carry additional structured information, like a converter specifying what it converts from and to, with a quality metric. The idea is to allow tools to build plugin registries from this kind of information without needing to import the modules. Basically, if you can fit it on one line, before the '=', in a reasonably human-readable way, and it saves you from having to import the module in order to figure out whether you wanted to import it in the first place, you can put it in the name. You might wish to make names a bit more restrictive than this, I suppose, but I'm not sure that all of the limitations of distribution names are actually appropriate here. In particular, restricting to alphanumerics, dots, and dashes is *way* too restrictive. Entry point names are sometimes used for human-readable command descriptions, e.g. this is a perfectly valid entry point definition in setuptools: wikiup: Upload documents to one or more wiki pages = some.module:entrypoint [extra1, extra2] Anyway, entry point group names are definitely *not* recommended to follow distribution names, as that makes them rather useless. Things that consume entry points will generally have more than one group, eventually, so at least one of those groups will then have to *not* be named after a distribution, unless you arbitrarily break up the project into multiple distributions so the group names match, which is kind of silly. ;-) Finally, it might be good to point out once again that extras are not so much "a set of dependencies that will be checked for at runtime" as a set of dependencies that are *needed* at runtime. This need may or may not be checked, and may or may not be satisfied dynamically at runtime; it depends on the API in use, and how/whether it is invoked. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
On Tue, Jul 30, 2013 at 2:04 PM, Donald Stufft wrote: > > On Jul 30, 2013, at 1:13 PM, PJ Eby wrote: > >> On Tue, Jul 30, 2013 at 4:14 AM, Donald Stufft wrote: >>> Heh, I'm pretty good at getting yelled at :) >> >> Nick is also pretty good at making people feel like he both knows and >> *cares* about their breakage, and isn't just dismissing their concerns >> as trivial or unimportant. Breakage isn't trivial or unimportant to >> the person who's yelling, so this is an important >> community-maintenance skill. It builds trust, and reduces the total >> amount of yelling. > > *shrug*, If I didn't care I would have made this change as soon as Nick > said it was ok. Instead I declared I was going to and waited to make sure > nobody else had any concerns. And once Holger said he did I said > ok I won't do it. Maybe my mannerisms give the impression I don't but > that's actually pretty far from the truth. I did say "feel like". ;-) Nick usually gives more of an impression that he's thought about concerns raised before rejecting them; your responses often sound like, "Who cares about that?" Asking for suggestions, for example, would be good. Nick also rarely seems irritated by people's concerns or problems, whereas you sometimes seem in a big hurry to fix something today or this week that's been broken for years, without giving folks a while to get used to the idea. Often your proposals seem less like proposals than, "I've decided to do this, so deal with it". I'm not saying all this because I want to complain or yell at you; I'm saying it because I think you do care enough to know how you're coming across, at least to me. Our discussions have gotten heated in the past because my impression of your reaction to the concerns I raise often seems like, "I don't care about supporting [group of people affected], so neither should you." Perhaps the issue is just one of confusion. When I raise an issue about, say, Python 2.3 users (who are still downloading setuptools 0.6 releases, and presumably also using them), it's not because I expect *you* to change your plans to support them, but because I need to know how *I* can, if the issue arises. So I don't actually expect you to care about Python 2.3 users (again, as an example), but I do expect you to care about *me* supporting them. In the most recent situation, you did in fact point me to your awesome hashlib port, so I do know you *do* care to at least that extent. But the rhetoric that you sent both before and after the helpful bit seemed on the inflammatory side, as though I were crazy to be thinking of Python 2.3. Whether or not this is true ;-) -- it's not especially *helpful* to the discussion. If I may offer a suggestion, asking questions in response to objections is generally more useful than immediately arguing the relevance of the objection. First, it tells the objector that you're interested in what they have to say, and second, it may well help the objector understand that there isn't actually any real problem, and gives them an easier path to backing down and/or compromising, whereas a frontal assault tends to focus people on responding to you instead of reconsidering their objection. On the hashlib issue, for example, it actually occurred to me later that it's completely a non-issue because the actual hash scenario I was most concerned about *can't actually happen*. (MD5 hashes in code or dependency_links, used e.g. by setuptools itself to secure its own downloads. Changing PyPI won't affect these, duh.) It might've occurred to me sooner, though, if you'd actually asked what scenario I was worried about, instead of arguing about the relevance. This isn't to say that you're responsible for what I do or don't figure out; my point is simply that asking questions and inviting suggestions in response to people's objections will generally get you more thoughtful responses and more trust, and resolve issues sooner, with less arguing. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
On Tue, Jul 30, 2013 at 4:14 AM, Donald Stufft wrote: > Heh, I'm pretty good at getting yelled at :) Nick is also pretty good at making people feel like he both knows and *cares* about their breakage, and isn't just dismissing their concerns as trivial or unimportant. Breakage isn't trivial or unimportant to the person who's yelling, so this is an important community-maintenance skill. It builds trust, and reduces the total amount of yelling. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Migrating Hashes from MD5 to SHA256
On Fri, Jul 26, 2013 at 3:14 PM, Donald Stufft wrote: > Does the hashlib backport I added to > setuptools 0.9 for Python 2.4 work on 2.3? It's a pure python > implementation of hashlib. Ah, didn't know about that! I can't imagine what problems there would be; not much changed in 2.4 that can't be emulated in 2.3. Anyway, I'll have a look. Thanks! > I don't have a Python 2.3 available to attempt to test. To be honest I've > never even used Python 2.3. Heh. Noob. ;-) (j/k) 2.3 is basically 2.4 minus decorators, generator expressions, various builtins and stdlib features. Unless you used set types, reversed(), or various other odds and ends, I should be able to backport it. > [stuff about RHEL support] If there's a 2.4 hashlib backport, that addresses my concerns just fine. If I need to, I'll backport it to setuptools 0.6. Thanks. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Migrating Hashes from MD5 to SHA256
On Fri, Jul 26, 2013 at 12:25 PM, Donald Stufft wrote: > Additionally there is no security list from setuptools versions earlier than > 0.7. Not true, actually. Setuptools 0.6 dev releases supported SSL verification since mid-May, but don't support any hashes besides MD5. Anybody who updated their setuptools between then and the release of 0.7 would have that version. Unfortunately, it's hard to tell how many people that is, though I could try and dig through my server logs to find out. There's also another issue with jumping to SHA256: Python prior to 2.5 didn't support it. Which brings up another point: the setuptools 0.6 series is the only setuptools available for Python 2.3. That's one of the reasons it's still available for download. If you want SSL verification on 2.3, it's the only thing available. (Meanwhile, a lot of people are still downloading 0.6c11; probably I should package up an 0.6c12 so those folks pick it up instead of 0.6c11.) Anyway, this is all somewhat moot since the hashes only matter when the download is hosted somewhere besides PyPI, since SSL verification is available for the PyPI part. Even so, I'd suggest that moving to SHA1 might be a good intermediate step: it's available on Python 2.3, so I could backport the relevant support to the 0.6 branch. (IIUC, Python 2.3 is still the default version for many Linux distros that have not reached end-of-life support.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On Mon, Jul 22, 2013 at 7:29 AM, Paul Moore wrote: > I'm still trying to be clear in my mind about what extras are, and how they > should work. From this description, it occurs to me to ask, what is the > difference between an extra and a (metadata only, empty) second distribution > that depends on the base project as well as the "additional set of > dependencies"? Is it just the admin overhead of registering a second > project? That's one way of looking at it. But it's not implemented that way; it's more like environment markers -- i.e., conditional dependencies -- based on whether you want support for certain features that are, well, "extra". ;-) > Looking at extras this way gives a possible way of generating scripts only > when the extras are present - just add the scripts to the dummy "extra" > distribution. Setuptools doesn't actually *have* a dummy distribution (just conditional requirements in the base), but I don't see a problem with only installing a script if you asked to install the extras that script needs. It probably would've been sensible to implement easy_install that way. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On Sun, Jul 21, 2013 at 6:44 PM, Nick Coghlan wrote: > > On 22 Jul 2013 01:46, "PJ Eby" wrote: >> >> Now that I'm thinking about it some more, one of the motivating use >> cases for extras in entry points was startup performance in >> plugin-heavy GUI applications like Chandler. The use of extras allows >> for late-loading of additions to sys.path. IOW, it's intended more >> for a situation where not only are the entry points imported late, but >> you also want as few plugins as possible on sys.path to start with, in >> order to have fast startup. > > I'm working with Eric Snow on a scheme that I hope will allow > module-specific path entries that aren't processed at interpreter startup > and never get added to sys.path at all (even if you import the module). > Assuming we can get it to work the way I hope (which is still a "maybe" at > this point in time), it should then be possible to backport it to earlier > versions as a metaimporter. I haven't had a chance to look at that proposal at more than surface depth, but my immediate concern with it is that it seems to be at the wrong level of abstraction for the packaging system, i.e., just because you can import a module, doesn't mean you can get at its project metadata (e.g., how would you find its exports, or even know what distribution it belonged to?). (Also, I don't actually see how it would be useful or relevant to the use case we're talking about; it seems maybe orthogonal at best.) > OK, so as Daniel suggested, it's more like an export/entry-point specific > "requires" field, but limited to the extras of the current distribution. Correct: at the time, it seemed a lot simpler to me than supporting arbitrary requirements, and allows for more DRY, since entry points might share some requirements. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On Sat, Jul 20, 2013 at 10:54 PM, Nick Coghlan wrote: > Extras should just be a way to ask "are these optional dependencies present > on this system?", without needing to worry about how they got there. Technically, they are a way to ask "can you get this for me?", since pkg_resources' API allows you to specify an installer callback when you ask to load an entry point. This means that an installer tool can dynamically obtain any extras it needs, not just check for their installation. To put it another way, it's not "exported only if extra is available", it's "exported, but make sure you have this first." A subtle difference, but important to the original use cases (see below). > For now, I'll switch export specifiers back to the concise > "modulename:qualname" entry point format and add "Do we need to > support the exported-only-if-extra-is-available feature?" as an open > question. My current thinking is that the point you made about script > wrappers (putting the wrapper in separate distribution and depending > on that from an extra) applies to other plugins as well. Now that I'm thinking about it some more, one of the motivating use cases for extras in entry points was startup performance in plugin-heavy GUI applications like Chandler. The use of extras allows for late-loading of additions to sys.path. IOW, it's intended more for a situation where not only are the entry points imported late, but you also want as few plugins as possible on sys.path to start with, in order to have fast startup. The other use case is similar, in that a plugin-heavy environment with self-upgrading abilities can defer *installation* of parts of a plug-in until it is actually used. (Which is why EntryPoint objects have a .require() method separate from .load() - you can loop over a relevant set of entry points to pre-test or pre-ensure that they're all available and dependencies are installed before importing any of them, even though .load() will also do that for a single entry point.) For the specific case of the meta build system itself, these use cases may be moot. For the overall use of exports, however, the use cases are still valuable for plugin-heavy apps. (Specifically, applications that use lots of plugins written by different people, and don't want to have to import everything at startup.) Indeed, this is the original use case for exports in the first place: it's a plugin system that doesn't require importing any plugins until you actually need a particular plugin's functionality. Extras just expand that slightly to "don't require installing things or putting them on sys.path until you need their functionality". Heck, if pip itself were split into two distributions, one of which were a command line script declared with an extra, pointing into the second distribution, it'd have dynamic bootstrapping. (Were it not for the part where it would need pip available to *do* the bootstrapping, of course. ;-) ) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On Sat, Jul 20, 2013 at 8:08 PM, Nick Coghlan wrote: > I see it as more useful for making an executable optional by defining a > "cli" extra. If your project just gets installed as a dependency, no wrapper > would get generated. > > Only if you went "pip install myproject[cli]" (or another project > specifically depended on the cli extra) would it be installed. Why stop there... how about environment markers for exports, too? ;-) And throw in an environment marker syntax for whether something was installed as a dependency or explicitly... ;-) (Btw, the above is a change from setuptools semantics, but I don't really see it as a problem; ISTM unlikely that anybody has used extras on a script wrapper. Extras on *other* entry points, however, *do* exist, at least IIRC. I'm pretty sure there was at least one concrete use case for them involving Chandler plugins when I originally implemented the feature. The possibility of having extras on a script is just a side effect, though, not an actually-intended feature; if you have the need, it actually makes more sense to just bundle the script in another package and require that pacakge from the extra, rather than putting it in the original package.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] entry points PEP
On Fri, Jul 19, 2013 at 2:09 PM, Joe Gordon wrote: > When I try importing pkg_resources in our development environment it is very > slow: Use zc.buildout to install the application you're invoking, and then it won't need to import pkg_resources. (Unless the actual app uses it.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On Fri, Jul 19, 2013 at 9:10 AM, Nick Coghlan wrote: > Right, I think the reasonable near term solutions are for pip to either: > > 1. generate zc.buildout style wrappers with absolute paths to avoid > the implied runtime dependency > 2. interpret use of script entry points as an implied dependency on > setuptools and install it even if not otherwise requested > > Either way, pip would need to do something about its *own* command > line script, which heavily favours option 1 Option 1 also would address some or all of the startup performance complaint. It occurs to me that it might actually be a good idea *not* to put the script wrappers in the standard entry points file, even if that's what setuptools does right now: if lots of packages use that approach, it'll slow down the effective indexing for code that's scanning multiple packages for something like a sqlalchemy adapter. (Alternately, we could use something like 'exports-some.group.name.json' so that each export group is a separate file; this would keep scripts separate from everything else, and optimize plugin searches falling in a particular group. In fact, the files needn't have any contents; it'd be okay to just parse the main .json for any distribution that has exports in the group you're looking for. i.e., the real purpose of the separation of entry points was always just to avoid loading metadata for distributions that don't have the kind of exports you're looking for. In the old world, few distributions exported anything, so just identifying whether a distribution had exports was sufficient. In the new world, more and more distributions over time will have some kind of export, so knowing *which* exports they have will become more important.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Replacing pip.exe with a Python script
On Tue, Jul 16, 2013 at 8:23 AM, Paul Moore wrote: > > On 16 July 2013 12:42, Oscar Benjamin wrote: >> >> I thought that 64 bit Windows could run 32 bit Windows .exe files >> (although I don't have a way to check this). > > > Yes, but there are 32-bit and 64-bit exe wrappers, which I suspect is > because a 32-bit exe can't load a 64-bit DLL (and may be vice versa). As I > said, I don't know for sure at the moment, but it needs investigating. That's not why they exist; the .exe's don't load the Python DLL, they just invoke python.exe. The existence of separate 32- and 64-bit .exe's is a Distribute innovation, actually; setuptools 0.6 doesn't use them. Instead, setuptools writes a manifest file to tell Windows that it doesn't need privilege escalation or to create a separate console. This meant that only one (32-bit) .exe was needed. I forget what happened with the Distribute approach or why the 64-bit version was kept after the merge; ISTM there was some other use for it, or at least Jason thought so. But DLL loading is not the reason. (Actually, after searching my email, my guess is that there actually *isn't* any need for the 64-bit .exe; ISTM that it was introduced solely as a false fix for the privilege escalation problem, that only fixes it for 64-bit Windows and doesn't help 32-bit.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On Thu, Jul 18, 2013 at 7:09 PM, Vinay Sajip wrote: > PJ Eby telecommunity.com> writes: >> While other strategies are definitely possible, distlib's approach is >> not backward-compatible, as it means installing new versions of a > > Correct, because distlib does not support multiple installed versions of the > same distribution, nor does it do the sys.path manipulations on the fly which > have caused many people to have a problem with setuptools. > > Do people see this as a problem? I would have thought that venvs would allow > people to deal with multiple versions in a less magical way. So does buildout, which doesn't need venvs; it just (if you configure it that way) puts all your eggs in a giant cache directory and writes scripts with hardcoded sys.path to include the right ones. This is actually more explicit than venvs, since it doesn't depend on environment variables or on installation state. IOW, there are other choices available besides "implicit environment-based path" and "dynamically generated path". Even setuptools doesn't require that you have a dynamic path. > If that is a real requirement which should be supported, shouldn't there be a > PEP for it, if it's coming into Python? It's not supported by distutils, and > it has been a point of contention. Distutils lets you install things wherever you want; in the naive case you could use install --root to install every package to a version-specific directory and then use something like Gnu Stow to create symlink farms. Python supports explicit sys.path construction and modification, and of course people certainly "vendor" (i.e. bundle) their dependencies directly in order to have a specific version of them. So, I don't think it's accurate to consider multi-version installation a totally new feature. (And AFAIK, the point of contention isn't that setuptools *supports* multi-version installation, it's that it's the *default* implementation.) In any event, wheels are designed to be able to be used in the same way as eggs for multi-version installs. The question of *how* has been brought up by Nick before, and I've thrown out some counter-proposals. It's still an open issue as to how much *active* support will be provided, but my impression of the discussion is that even if the stdlib isn't exactly *encouraging* multi-version installs, we don't want to *break* them. Hence my suggestion that if you want to drop pkg_resources use from generated scripts, you should use buildout's approach (explicit sys.path baked into the script) rather than distlib's current laissez-faire approach. Or you can just check versions, I'm not that picky. All I want is that if you install a new version of a package and still have an old copy of the script, the old script should still run the old version, or at least give you an error telling you the script wasn't updated, rather than silently running a different version. Buildout's approach accomplishes this by hardcoding egg paths, so as long as you don't delete the eggs, everything is fine, and if you do delete any of them, you can see what's wrong by looking at the script source. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On Thu, Jul 18, 2013 at 4:36 PM, Paul Moore wrote: > As regards console scripts, I think they should be rewritten to remove the > dependency on pkg_resources. That should be a setuptools fix, As others have already mentioned, this is not a bug but a feature. Setuptools-generated scripts are linked to a specific version of the project, which means that you can install more than one version by renaming the scripts or installing the scripts to different directories. While other strategies are definitely possible, distlib's approach is not backward-compatible, as it means installing new versions of a project will change *existing scripts'* semantics, even if you installed the previous version's scripts to different locations and intended them to remain accessible. If you want an example of doing it right, see buildout, which hardcodes the entire sys.path of a script to refer to the exact versions of all dependencies; while this has different failure modes (i.e., dependence on absolute paths), it is more stable as to script semantics even than setuptools' default behavior. > maybe triggered by a flag (possibly implied by > --single-version-externally-managed, as the pkg_resources complexity is only > needed when multi-versions are involved). That option does not preclude the existence of multiple versions, or the possibility of installing the same script to different directories for different installed versions. If you *must* do this, I suggest using buildout's approach of hardwiring sys.path in the script, only strengthened by checking for the actual existence and versions, rather than distlib's anything-goes approach. (Of course, as Donald points out, this won't do anything for those scripts that themselves make use of other packages' entry points: they will have a runtime dependency on pkg_resources anyway.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] entry points PEP
On Thu, Jul 18, 2013 at 1:50 PM, Vinay Sajip wrote: > Daniel Holth gmail.com> writes: >> For one thing you can have more than one mysql = in the same >> sqlalchemy.dialects. I think in this instance the string parsing is > > Don't you say in the PEP about the key that "It must be locally unique > within this distribution’s group."? Setuptools requires this per-distribution uniqueness, but note that uniqueness is not required across distributions. So more than one distribution can export a 'mysql' in the 'sqlalchemy.dialects' group. It's up to the application to decide how to handle multiple definitions; typically one either uses all of them or the first one found on sys.path, or some other tie-breaking mechanism. The pkg_resources entry point APIs just provide operations for iterating over entry points defined on either a single distribution, or across all distributions on a specified set of directories. (Via the WorkingSet API.) > Note that I don't see necessarily a connection between extras and flags, > though > you've mentioned that they're extras. Does setuptools store, against an > installed distribution, the extras it was installed with? AFAIK it doesn't. > (Even if it did, it would need to keep that updated if one of the extras' > dependencies were later uninstalled.) And if not, how would having extras in > the specification help, since you can't test the "must be installed" part? The pkg_resources implementation does a require() for the extras at the time the entry point is loaded, i.e., just before importing. This allows it to dynamically add requirements to sys.path, or alternatively raise an error to indicate the extras aren't available. In addition, various entry point API functions take an 'installer' keyword argument, specifying a callback to handle installation of missing extras. Setuptools uses this feature internally, so that if you use a setup.py command whose entry point needs additional dependencies, those will be fetched on-the-fly. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 439 and pip bootstrap updated
On Thu, Jul 11, 2013 at 10:20 AM, Brett Cannon wrote: > And if people want to promote the -m option then the executable scripts just > become a secondary convenience. Plus you can't exactly require setuptools to > create those scripts at install-time with Python if that's when they are > going to be installed. You don't need setuptools in order to include .exe wrappers, though: there's nothing setuptools-specific about the .exe files, they just run a matching, adjacent 'foo-script.py', which can contain whatever you want. Just take the appropriate wrapper .exe, and rename it to whatever 'foo' you want. IOW, if you want to ship a pip.exe on windows that just does "from pip import __main__; __main__()" (or whatever), you can do precisely that, no setuptools needed. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [issue153] Jython - easy_install -- ssl_support.py: module has no attribute 'CERT_REQUIRED'
On Sat, Jul 6, 2013 at 10:29 PM, Tom Brennan wrote: > > New submission from Tom Brennan: > > Can't install anything with easy_install or pip. Happens in at least v.0.8 > and 0.7.8. (stack trace attached is from 0.7.8) > > -- > files: easy_install_stack_trace.txt > messages: 729 > nosy: tjb1982 > priority: bug > status: unread > title: Jython - easy_install -- ssl_support.py: module has no attribute > 'CERT_REQUIRED' > Added file: > http://bugs.python.org/setuptools/file93/easy_install_stack_trace.txt It appears that Jython's ssl module is not 100% compatible with Python's. I'm not sure what to do about that. ( By the way, bugs for setuptools 0.7 and higher should be reported to the issue tracker at https://bitbucket.org/pypa/setuptools ) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Metadataformat PEP 426 on PyPI?
On Wed, Jul 3, 2013 at 2:34 PM, Donald Stufft wrote: > PEP426 does not support dependency_links. Right - I would've expected direct references in this scenario, assuming the PEP still has them. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Metadataformat PEP 426 on PyPI?
On Wed, Jul 3, 2013 at 10:51 AM, Vinay Sajip wrote:
> If you deserialize the JSON at an URL like the above into a dict, the PEP
> 426 metadata is available in the subdict at key "index-metadata" in the
> top-level dict. Example from setuptools 0.7.5:
>
> "index-metadata": {
>
> "name": "setuptools"
> },
>
> I expect this metadata to track the PEP as changes to it are published.
> Currently, the top-level dict contains some legacy representations of the
> metadata which will be removed in due course.
Just an FYI, not sure if this is an issue with your converter or with
the new spec, but the metadata shown for setuptools is missing
something important: 0.7.x pins specific distributions of its
dependencies using dependency_links URLs with #md5 hashes, so that SSL
support can be installed in a reasonably secure manner, as long as
you're starting from a trusted copy of the distribution. The
converted metadata you show lacks this pinning.
Granted, the pinning is somewhat kludged, and the specific need is
perhaps a rare use case outside of installer tools themselves. But I
thought it worth pointing out as a limitation of either the converter
or with the spec itself in relation to version support.
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Finding modules in an egg / distribution
On Tue, Jul 2, 2013 at 1:30 PM, Iwan Vosloo wrote: > On 02/07/2013 17:08, PJ Eby wrote: >> >> If you are targeting at least Python 2.5, see: >> http://docs.python.org/2/library/pkgutil.html#pkgutil.walk_packages > > > We're targeting Python 2.7. > > Trouble is that pkgutil.walk_packages needs a path to search from. > Distribution.location is always your site-packages directory once a > Distribution is installed, so walking that just gives ALL installed > packages. If your Distribution contains some sort of main package that > contains everything in it, you can use that package's .__path__, but then > you'd need to discover what that package is. > > Distributions could also contain more than one package next to each other, > and top-level modules. (The __path__ of a top-level module is also simply > the site-packages directory.) > > There is a metadata file top_level.txt which one could use to get the names > of top-level packages/modules in the Distribution. This can however contain > a namespace package too - and you don't want all the packages inside the > namespace package - just the bits inside the chosen Distribution... Ah, well in that case you'll have to inspect either .egg-info/SOURCES.txt or the PEP 376 installation manifest. I don't know of any reliable way to do what you want for system-installed packages at the moment. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Finding modules in an egg / distribution
On Tue, Jul 2, 2013 at 6:59 AM, Iwan Vosloo wrote: > Hi there, > > We have been struggling to find a nice way to list all the modules (or > packages) that are part of a particular Distribution (or egg). Nice should > also mean that it works when the egg is installed. We have a need to do some > introspection on the code shipped as an egg. > > Any ideas? If you are targeting at least Python 2.5, see: http://docs.python.org/2/library/pkgutil.html#pkgutil.walk_packages ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Download Counts on PyPI
On Thu, Jun 27, 2013 at 5:18 PM, Donald Stufft wrote: > Download counts have (kind of) been re-enabled on PyPI. > > The new download counts work as so: > > * At the bottom of a detail page there is rolling tallies for the last >day, the last week, and the last month[1] > * The API exposes total download counts per file still. > * The rolling counts update more or less in real time while the total > counts in the API update hourly. > * We no longer incorporate download counts from mirrors, making the > mirroring infrastructure a tree and not a federation. > Is there any plan for per-version/per-file stats? I notice that the current stat display is across all versions, which makes it harder to tell e.g. setuptools 0.7 uptake vs 0.6. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 440 and direct references
On Wed, Jun 26, 2013 at 1:40 PM, Vinay Sajip wrote: > (a) Repeatability (not possible with a generic "latest version" URL). ISTM that forcing repeatability in the spec implies the inability to do development with requirements that are also in development, unless there is also an out-of-band channel for communicating the URL. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
