Re: [Distutils] Proper handling of PEP420 namespace packages with setuptools and pip
On Wed, Apr 22, 2015 at 5:20 PM, Robert Collins
robe...@robertcollins.net wrote:
Ah, ok so I think this is the crux - I'm arguing that Python version
isn't a big enough check. Because anything installed with a current
version of setuptools, or any wheel built likewise, is going to not
have that per-Python-version check.
And it seems to me that that implies that bringing in a
per-Python-version check in a new release of setuptools or pip is
going to result in mixed mode installs:
install name.A with setuptools-X [legacy]
upgrade setuptools
install name.B with setuptools-Y [does a Python version check]
- boom
But perhaps sufficient glue can be written to make it all work.
When I wrote PEP 402 (the precursor to 420), the idea was that in a
mixed environment you would need to:
1. Change pkg_resources' namespace system to support non-__init__
directories (and likewise pkgutil.extend_path)
2. Change easy_install's .pth generation magic to not do any magic or
import a PEP 420 emulation module on old systems
3. Change package building tools to stop injecting __init__.py files
This basically solves the mixed installation problem because if you
have an __init__.py that uses existing magic, the empty dirs get
folded in. You basically have a transitional state with mixed
__init__ and non-__init__ stuff. If there happens to be an
__init__.py, then as long as it declares the namespace then the local
*runtime* takes care of making the runtime environment work. The
*installation* tools don't have to manage mixed modes, they should
just blindly install whatever package they have, and over the long
term the packages all end up shipped without __init__.py's, but the
__init__.py approach will continue to work basically forever.
My personal preferred migration strategy is:
- have a flag day amongst the cooperating packages that make up the namespace
- release versions that are all in the new layout in a batch to PyPI.
It would be nice if PEP-426 had a conflicts stanza, so you could say
conflicts: [name.A version_with_new_X] without that implying that
name.A *should* be installed.
This is all *really* unnecessary. Setuptools has essentially *always*
built non-egg, non-exe binary distributions in a PEP 420-compatible
way (i.e., without __init__.py). And pkg_resources already builds a
namespace path by asking importers if they can import a package at
that location. So if PEP 420 importers say yes when asked to
find_module('somepkg') in the case of an __init__-less subdirectory
named 'somepkg', then pkg_resources *already* supports mixed-mode
installations under PEP 420, and doesn't even need to be updated!
I haven't checked whether that's the case, but if it is, then the only
thing that setuptools neds to change is its .pth generation magic, to
not do the magic if it's on a PEP 420 platform at runtime, and to stop
including __init__.py's for namespace packages.
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Installing namespace packages with pip inserts strange modules into sys.modules
On Fri, Sep 12, 2014 at 3:55 PM, Erik Bray erik.m.b...@gmail.com wrote: Unfortunately there are some dark corners of setuptools I've encountered where namespace packages don't work properly during installation *unless* they were installed in the old-fashioned setuptools way. I'll have to see if I can dig up what those cases are, because they should be fixed. I don't know if this is what you had in mind, but, the main problem I know of with changing setuptools to not install the .pth files, is that if you already have any __init__.py's installed (and possibly, any namespace .pth files to go with them), then the newly-installed package isn't going to work. PEP 420 only takes effect if there are no existing __init__.py files for the namespace. In other words, it's not just a matter of changing how things are installed, it's also a matter of upgrading existing environments with things installed by older installation tools. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 470 Round 2 - Using Multi Index Support for External to PyPI Package File Hosting
On Fri, Jun 6, 2014 at 10:25 AM, Donald Stufft don...@stufft.io wrote: I expected more people to move to safe external vs staying with the unsafe external. Is there a tool that makes this *easy*? I'm not aware of one. (Ideally, something like a replacement for setup.py upload that generates the download URLs and sends them off to PyPI, so that all one needs is a setup_requires for the tool, a setup.cfg with the hosting prefix, and a run of setup.py register bdist_whatever uplink to get the links set up.) ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What is the syntax for passing conditional non-extra dependencies in setuptools?
On Mon, Apr 7, 2014 at 4:03 PM, Daniel Holth dho...@gmail.com wrote:
OK, it does in fact work that way; empty-string-before-colon specifies
a default requirement with a marker.
Parses out to the following _dep_map with 'None' representing 'not an
extra dependency':
{None: [Requirement.parse('coding'), Requirement.parse('stuff')],
'quux': [Requirement.parse('more-stuff')]}
requirements.txt:
coding
[:sys_platform=='linux2']
stuff
[quux:sys_platform=='linux2']
more_stuff
Huh. Either I did that intentionally and forgot about it, or it just fell
out as a brilliant accidental side effect of the orthogonal design of the
parsing functions. Either way, I feel like I did something smart there.
;-)
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [Import-SIG] nspkg.pth files break $PYTHONPATH overrides
On Wed, Mar 26, 2014 at 11:55 PM, Eric Snow ericsnowcurren...@gmail.comwrote: In 3.4 it's called _NamespaceLoader, but in 3.3 it's NamespaceLoader. ducks Ouch. That is going to be a really *long* bit of code. Not like it isn't already, though. By available on the platform do you mean Python 3.3+ or has a backport installed? Otherwise you'd just check sys.version*. I just hate doing version checks when a feature check ought to be possible. You never know when it's going to cause trouble. But yes, certainly being able to handle the backport case would be nice. When I wrote PEP 402, I was thinking in terms of transitioning the .pth files to importing a compatibility module. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What is the syntax for passing conditional non-extra dependencies in setuptools?
On Wed, Mar 26, 2014 at 11:29 PM, Daniel Holth dho...@gmail.com wrote:
How do I specify a conditional (marker-guarded) non-extra dependency
in setuptools? The syntax for a conditional extra dependency is
currently:
extras_require = {
ssl:sys_platform=='win32': wincertstore==0.2,
certs: certifi==1.0.1,
},
I only implemented support via extras, and the feature wasn't officially
supported (still isn't, I don't think) because the PEP specifying the
syntax wasn't fully baked yet. I figured that if *only* setuptools itself
used it, then if the syntax changed only setuptools would break... but fix
itself at the same time.
The same cannot be said for any other package, so use at your own risk. Or
better yet, don't use it. ;-)
(At least, not until it's a documented feature w/a PEP-approved syntax.)
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] nspkg.pth files break $PYTHONPATH overrides
On Mon, Mar 24, 2014 at 6:09 PM, Barry Warsaw ba...@python.org wrote: On Mar 24, 2014, at 05:53 PM, Donald Stufft wrote: See also https://github.com/pypa/pip/issues/3 Hah, yeah. I didn't realize --single-version-externally-managed is implied by --root, and in fact our Debian build scripts often (either explicitly or implicitly) define --root, as is the case in lazr.uri. Basically prior to PEP420 namespace packages were bad and using them results in pain sooner or later :( I'm not sure if a good solution yet, perhaps we can backport PEP420 to PyPI and have namespace packages depend on that? Or maybe do a version check before installing this file? Python 3.3 and newer will have PEP 420 support, and this file makes no sense in that case. That won't *quite* work, because the .pth files are generated for other types of direct-install distributions. I think the correct fix would be to change the nspkg.pth magic to check for PEP 420 support, but unfortunately it seems we may have to use version checking: on rereading PEP 420 I see there's no 'sys.namespace_packages' or similar object that can be directly checked for feature support. :-( ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] nspkg.pth files break $PYTHONPATH overrides
On Tue, Mar 25, 2014 at 3:50 PM, Barry Warsaw ba...@python.org wrote:
On Mar 25, 2014, at 03:35 PM, PJ Eby wrote:
I think the correct fix would be to change the nspkg.pth magic to check
for
PEP 420 support, but unfortunately it seems we may have to use version
checking: on rereading PEP 420 I see there's no 'sys.namespace_packages'
or
similar object that can be directly checked for feature support. :-(
There is. It's *pronounced* sys.namespace_packages, but it's spelled
importlib._bootstrap._NamespaceLoader ;)
Yeah, well that's not exactly a public attribute, so it's not necessarily a
great way to do it. But if we did use that, presumably it'd add something
like:
hnp =
hasattr(sys.modules.get('importlib._bootstrap',None),'_NamespaceLoader');
To the front of the magic, and then prefix all subsequent expression values
with 'not hnp and' in order to prevent them executing if PEP 420 support is
available. (Note: it's checking sys.modules since on any interpreter where
PEP 420 is natively available, the module should *already* be loaded by the
time site.py does its thing.)
And we should probably think about adding sys.namespace_packages or
something of the sort, or at least a proper flag for whether PEP 420
support is available on the platform.
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] setuptools tracker
On Mon, Mar 24, 2014 at 8:46 AM, Nick Coghlan ncogh...@gmail.com wrote: On 24 March 2014 20:53, Martin v. Löwis mar...@v.loewis.de wrote: Both problems would be resolved by setting the tracker to read-only; shutting it down is actually not necessary (although it would slightly reduce our maintenance efforts). That sounds good to me. I've also filed https://bitbucket.org/pypa/setuptools/issue/174/ to decide on a longer term solution. If Jason decides to review/migrate issues, it may be necessary to turn developer write access back on to allow issues to be marked as closed once they have been dealt with appropriately. Yep, looks like Jason came to the same conclusion(s) independently, but also wants better banners on the old tracker to alert people to the move. I guess we should move any further discussion to that ticket, since Jason's response time is quicker there than here. ;-) ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] setuptools tracker
On Sun, Mar 23, 2014 at 4:55 AM, Martin v. Löwis mar...@v.loewis.dewrote: We are still hosting a roundup installation for setuptools, at http://bugs.python.org/setuptools/. Is this still needed? If not: what should we do with it? I think probably the remaining issues need to be moved to Bitbucket (unless they're already addressed in later setuptools versions), and the tracker closed. At this point, I think it's safe to say that the 0.6 line isn't getting any more changes; persons and organizations using older versions of Python will have to take 0.6 as it is, or upgrade. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] setuptools tracker
I think it would be a good idea to check with Jason and other PyPA volunteers to see if there is anyone else who can handle the moves. I'd prefer we didn't lose the history, since my comments on those issues (and the closed ones, too) often contain key information about use cases and design decisions that may not be available elsewhere, even from my memory. ;-) But, since I'm no longer in the lead on development, I think it would be better for someone closer to the future of things to do the prioritizing of what, if anything, to transfer as an issue or keep as documentation. On Sun, Mar 23, 2014 at 2:00 PM, Martin v. Löwis mar...@v.loewis.dewrote: Am 23.03.14 18:30, schrieb PJ Eby: On Sun, Mar 23, 2014 at 4:55 AM, Martin v. Löwis mar...@v.loewis.de mailto:mar...@v.loewis.de wrote: We are still hosting a roundup installation for setuptools, at http://bugs.python.org/setuptools/. Is this still needed? If not: what should we do with it? I think probably the remaining issues need to be moved to Bitbucket (unless they're already addressed in later setuptools versions), and the tracker closed. At this point, I think it's safe to say that the 0.6 line isn't getting any more changes; persons and organizations using older versions of Python will have to take 0.6 as it is, or upgrade. Would you volunteer to move them? Alternatively, I could close them all with an automatic message saying that they should re-report them if the issue still exists. Regards, Martin ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Is there any sense to put setuptools as a requirement?
On Sat, Mar 1, 2014 at 4:47 PM, Jim Fulton j...@zope.com wrote: On Thu, Feb 27, 2014 at 6:49 AM, Piotr Dobrogost p...@google-groups-2014.dobrogost.net wrote: Hi! I've seen people putting 'setuptools' in 'install_requires' in setup.py starting with import from setuptools like this: from setuptools import setup, find_packages Does it make any sense? In what circumstances should 'setuptools' be listed in 'install_requires' or 'setup_requires'? It makes sense when you use setuptools to implement namespace packages. So, for example in zope packages, the __init__.py in the zope namespace package typically has: import pkg_resources pkg_resources.declare_namespace(__name__) It also makes sense if you use entry points, the plugin finder, or any other pkg_resources features, for that matter. ;-) Or if you reuse setuptools functionality (e.g. easy_install) to install plugins for your app. But in general, it only makes sense to depend on setuptools if your actual project contents (aside from setup.py) are importing things from pkg_resources or setuptools. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Setuptools 3.0b1 available for preview
On Wed, Feb 12, 2014 at 11:11 AM, Sebastien Douche sdou...@gmail.com wrote:
On Wed, Feb 12, 2014 at 5:09 AM, Jason R. Coombs jar...@jaraco.com wrote:
Hi Jason
This backward-incompatible release contains the changes detailed in the
CHANGES.txt file:
Issue #148: When building (bdist_egg), setuptools no longer adds
``__init__.py`` files to namespace packages. Any packages that rely on this
behavior will need to create ``__init__.py`` files and include the
``declare_namespace()``.
Thing like:
try:
__import__('pkg_resources').declare_namespace(__name__)
except ImportError:
from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)
__init__.py (END)
are theses lines good?
Yes. The main idea is just that the __init__.py must set up the
namespace package, and not rely on setuptools doing the declaration
implicitly/automatically.
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] File names with spaces
On Tue, Jan 7, 2014 at 4:07 AM, Maciej (Matchek) Bliziński mac...@opencsw.org wrote: 2013/10/1 Maciej (Matchek) Bliziński mac...@opencsw.org Hello setuptools developers, I'm attempting to package the newest setuptools version (1.1.6) on Solaris 9 and 10. One of the limitations of the Solaris package manager (the old one, pkgadd/pkgrm), is that it is unable to handle file names with spaces. Would you mind renaming script template.py to something like script_template.py? Probably yes, they would mind. ;-) I believe the reason there's a space there is so that the file is not importable. (It should never be imported; it's a data file rather than a python module.) I suspect you'll have better luck with a suggestion like 'script-template.py', since that would still not be importable. I've created a pull request with the change, do you have any advice who should I talk to? https://bitbucket.org/pypa/setuptools/pull-request/33/rename-script-templatepy-to/diff I've added a comment there about the underscore vs. dash; it looks from the other pull requests like Jason wants issues filed describing the problems that a pull request is intended to fix, and that may mean he hasn't seen the pull request yet. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Major update to the metadata 2.0 draft PEPs
On Wed, Jan 1, 2014 at 9:39 AM, Nick Coghlan ncogh...@gmail.com wrote: On 1 Jan 2014 10:33, PJ Eby p...@telecommunity.com wrote: I have only been skimming these so far, will comment more later, but I just want to mention that for standard extensions, what is the rationale for not defining e.g. commands as exports? Or for that matter, defining hooks as exports? Both commands and hooks have a payload that's the same as an export payload, i.e., a reference to an importable object. I think it'd be good for them to be defined in terms of exports in order to reduce duplication of concepts and implementations, as well as providing a PEP-defined example of how to use export groups and export names effectively. I believe it was due to the extra layer of nesting they needed - using multiple parallel export groups with different final elements in their dotted names didn't feel right. I guess that indicates a flaw in my initial design for the export definitions though - I agree it would be better if commands and hooks could be cleanly defined within the exports mechanism, rather than needing separate custom metadata extensions. If it's a flaw, I'd say it's in my original design of entry points. ;-) Basically, I wanted a way to do -- without XML -- what Eclipse does with its extensions and extension points machinery. I went with a flat (with dots) is better than nested concept. To me, though, this doesn't look terribly complicated (using entry points syntax): [python.exports.after_install] ComfyChair.plugins = ComfyChair.plugins:install_hook [python.exports.after_uninstall] ComfyChair.plugins = ComfyChair.plugins:install_hook Nor this: [python.extensions.after_install] python.exports = pip.export_group_hooks:run_install_hooks python.commands = pip.command_hook:install_wrapper_scripts [python.extensions.after_uninstall] python.exports = pip.export_group_hooks:run_uninstall_hooks (Also, adding hooks to *validate* extensions and exports at build and/or install time might be handy.) Finally, note that if the typical usecase is to define *both* an install and uninstall hook, then it might be simpler to just define the hook once, as an object with 'after_install' and 'after_uninstall' methods. This avoids the need to register a hook in two groups, and in the simplest case people can just make them both module-level functions and list the module as the export. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Major update to the metadata 2.0 draft PEPs
On Sat, Dec 21, 2013 at 9:46 AM, Nick Coghlan ncogh...@gmail.com wrote: I've just published the first draft of the metadata 2.0 spec that moves all of the fields that aren't part of the core metadata or potentially needed for dependency resolution out to a separate standard metadata extensions PEP. Yay! I think this makes PEP 426 itself substantially less overwhelming, and also provides convenient names to refer to the various other subsets of the metadata. Indeed! Metadata 2.0: http://www.python.org/dev/peps/pep-0426/ Versioning: http://www.python.org/dev/peps/pep-0440/ Standard Extensions: http://www.python.org/dev/peps/pep-0459/ I have only been skimming these so far, will comment more later, but I just want to mention that for standard extensions, what is the rationale for not defining e.g. commands as exports? Or for that matter, defining hooks as exports? Both commands and hooks have a payload that's the same as an export payload, i.e., a reference to an importable object. I think it'd be good for them to be defined in terms of exports in order to reduce duplication of concepts and implementations, as well as providing a PEP-defined example of how to use export groups and export names effectively. (Also, noting the mapping of current script export namespaces to the new metadata would be helpful for people implementing translation tools from setuptools metadata.) Last but not least, some of the export examples should use dotted post-module names (e.g. some.module:RandomClass.a_class_method) to highlight that qualified names can be used there. (Yes, the spec *says* qualified names, but it's the sort of thing that people overlook when the examples are all of unqualified simple names.) ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Distribution format for Python3 libraries
On Tue, Nov 19, 2013 at 6:22 AM, Nick Coghlan ncogh...@gmail.com wrote: On 19 November 2013 15:09, PJ Eby p...@telecommunity.com wrote: On Sun, Nov 17, 2013 at 8:44 AM, Nick Coghlan ncogh...@gmail.com wrote: More accurately, it appears setuptools *does* support PEP 345 style environment markers, it just isn't documented at http://pythonhosted.org/setuptools/setuptools.html#declaring-dependencies *sigh* That's because it's technically an experimental feature I hacked into 0.6c12 development shortly before the distribute merge in order to handle setuptools' own requirements for SSL, so that setuptools could ship just one platform-independent egg and still have platform-specific dependencies. ;-) I think Jason et al may have since upgraded it to a supported feature, but last I looked I think there may have still been issues with Jython support for the feature. So, the lack of documentation may still be a feature rather than a bug ATM. ;-) A fair point :) Regardless, it highlights the fact I need to ensure PEP 426 preserves the legacy spellings for the markers that have dots in their names in metadata 1.2 (https://bitbucket.org/pypa/pypi-metadata-formats/issue/12). Eh? I implemented PEP 426 (draft-at-the-time) markers in the feature I mentioned above. Hm. Actually, looking over it now, I see the real reason it's experimental: it was implemented based on consensus-in-progress for PEP 426, so not a finalized PEP. It also sort-of handles PEP 345. And the Jython fallback, IIRC, was maybe only doing PEP 345. (And setuptools itself only needed the Python version and platform name.) So it's actually a bit of a mess -- which is why it wasn't documented, in order to avoid people forcing the new PEP's hand with another de facto standard. I just figured it would get us most of the way there and when the PEP dust settled, the tests and code could be tweaked a bit to match, then documented with reference to the PEP. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Distribution format for Python3 libraries
On Tue, Nov 19, 2013 at 5:24 PM, Nick Coghlan ncogh...@gmail.com wrote: Perhaps it would be worth breaking the environment markers out as their own PEP? As I said back then, I think *everything* in PEP 426 is worth breaking out into its own PEP. ;-) Less jokingly, I think that the scope of PEP 426 includes many things that have different (if overlapping) audiences, and splitting it into smaller pieces will help those audiences concentrate on the bits relevant to them. The audience for markers is a subset of the audience for requirement specs, which is a subset of the audience for version syntax. Lots of other subtopics are of interest mainly to build tool and distro makers, so there's little point in making everybody read everything. (Also, a lot of these smaller things are (hopefully) easier to reach consensus on, at which point the PEPs are settled and allow moving discussion on to the more open issues.) So, yes, good idea to break out environment markers. ;-) ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Distribution format for Python3 libraries
On Sun, Nov 17, 2013 at 8:44 AM, Nick Coghlan ncogh...@gmail.com wrote: More accurately, it appears setuptools *does* support PEP 345 style environment markers, it just isn't documented at http://pythonhosted.org/setuptools/setuptools.html#declaring-dependencies *sigh* That's because it's technically an experimental feature I hacked into 0.6c12 development shortly before the distribute merge in order to handle setuptools' own requirements for SSL, so that setuptools could ship just one platform-independent egg and still have platform-specific dependencies. ;-) I think Jason et al may have since upgraded it to a supported feature, but last I looked I think there may have still been issues with Jython support for the feature. So, the lack of documentation may still be a feature rather than a bug ATM. ;-) ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Egg name computation
On Mon, Oct 28, 2013 at 4:29 AM, Martin Fiers martin.fi...@intec.ugent.be wrote: I guess we'll have to rename them manually after the setup() function, unless there's a way to 'force' setup() to 'think' it has compiled extensions in them? You could include a dummy extension that does nothing, I suppose. Or which controls the building of your actual extensions. Setuptools has long supported Pyrex and I think that Cython might also work, i.e., that you could just specify your cython modules as extensions in setup.py to start with. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Egg name computation
On Tue, Oct 15, 2013 at 8:07 AM, Martin Fiers martin.fi...@intec.ugent.be wrote: So the platform argument now is self.distribution.has_ext_modules() and self.plat_name Shouldn't it just be self.plat_name ? No. The platform name is only included if the distribution has extension modules, because extension modules are what make the egg platform-specific. If there is only Python code and data, then the egg is considered platform independent. Would there be a workaround? What do you want to work around this *for*? If the egg doesn't contain extension modules, what is platform-specific about it? ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Warehouse - Simple API
On Wed, Oct 2, 2013 at 10:57 PM, Donald Stufft don...@stufft.io wrote: I've recently deployed the current Warehouse code. So far the simple API has been implemented and it would be great if people could point their installers to it and report back to me if they run into any problems. The current PyPI simple api is not documented at all so I've had to reverse engineer it going from what I know, pip, etc. Ideally if i've missed anything it will be found out with testing from y'all. FYI, compare: https://preview-pypi.python.org/simple/setuptools/ and: https://pypi.python.org/simple/setuptools/ The former is missing the #egg URLs, which are required to install SVN versions. In general, extracted URLs don't seem to be working on the preview site. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pkg_resources get_distribution - zipfile support?
On Tue, Oct 1, 2013 at 1:51 PM, Daniel Holth dho...@gmail.com wrote:
pkg_resources only finds distributions inside .egg and inside sys.path
entries that are filesystem directories.
Actually it looks in zipfiles (or subdirectory paths thereof) or
filesystem directories, and can spot zipfile subdirectories named
'.egg' inside zipfiles (or subdirectories thereof).
It will also look in any other sort of sys.path entry, if you register
a handler for the importer type.
You might be able to manually
add a new Distribution instance to working_set or start looking for a
place to add the feature here:
https://bitbucket.org/pypa/setuptools/src/27df3c725f9696ba730456f3f444cc2fb5271d4b/pkg_resources.py?at=default#cl-2560
_distributionImpl = {
'.egg': Distribution,
'.egg-info': Distribution,
'.dist-info': DistInfoDistribution,
}
https://bitbucket.org/pypa/setuptools/src/27df3c725f9696ba730456f3f444cc2fb5271d4b/pkg_resources.py?at=default#cl-2219
(Distribution.from_location())
Actually, you would add the feature *here*:
https://bitbucket.org/pypa/setuptools/src/27df3c725f9696ba730456f3f444cc2fb5271d4b/pkg_resources.py?at=default#cl-1810
That is, in the find_in_zip() function, which is used to identify
distributions contained in zip files (or subdirectories thereof) on
sys.path.
(This feature needs to get added at some point anyway, for
pkg_resources to support mountable wheels.)
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pkg_resources get_distribution - zipfile support?
On Tue, Oct 1, 2013 at 5:11 PM, Paul Moore p.f.mo...@gmail.com wrote:
On 1 October 2013 21:32, PJ Eby p...@telecommunity.com wrote:
On Tue, Oct 1, 2013 at 1:51 PM, Daniel Holth dho...@gmail.com wrote:
pkg_resources only finds distributions inside .egg and inside sys.path
entries that are filesystem directories.
Actually it looks in zipfiles (or subdirectory paths thereof) or
filesystem directories, and can spot zipfile subdirectories named
'.egg' inside zipfiles (or subdirectories thereof).
But not dist-info? I thought setuptools supported dist-info formats
these days. Is this somewhere that got missed? Or is it more subtle
than that?
The person or persons who implemented dist-info either didn't know
where to put in the support, or didn't consider it a priority to also
support zipped dist-info.
A cursory glance at the DistinfoDistribution class, however, suggests
that it should work fine as long as it's paired with an appropriate
metadata provider. Something like:
class WheelMetadata(EggMetadata):
def _setup_prefix(self):
# like EggProvider._setup_prefix, except w/.whl and
# .dist-info instead of .egg and EGG-INFO
# ... or refactor EggProvider to parameterize these as
# class attrs and override here
and then make find_in_zip look more like:
def find_in_zip(importer, path_item, only=False):
for (meta_factory, fn, dist_factory) in [
(EggMetadata, 'PKG-INFO', Distribution),
(WheelMetadata, 'METADATA', DistInfoDistribution),
]:
metadata = meta_factory(importer)
if metadata.has_metadata(fn):
yield dist_factory.from_filename(path_item, metadata=metadata)
if only:
return # don't yield nested distros
for subitem in metadata.resource_listdir('/'):
if subitem.endswith('.egg') or subitem.endswith('.whl'):
subpath = os.path.join(path_item, subitem)
for dist in find_in_zip(zipimport.zipimporter(subpath), subpath):
yield dist
So it's not a huge deal to implement, just a bit of tedium. Under the
hood, there's little difference between wheels and eggs besides naming
conventions.
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pkg_resources get_distribution - zipfile support?
On Tue, Oct 1, 2013 at 7:10 PM, Nick Coghlan ncogh...@gmail.com wrote: On 2 Oct 2013 07:12, Paul Moore p.f.mo...@gmail.com wrote: On 1 October 2013 21:32, PJ Eby p...@telecommunity.com wrote: On Tue, Oct 1, 2013 at 1:51 PM, Daniel Holth dho...@gmail.com wrote: pkg_resources only finds distributions inside .egg and inside sys.path entries that are filesystem directories. Actually it looks in zipfiles (or subdirectory paths thereof) or filesystem directories, and can spot zipfile subdirectories named '.egg' inside zipfiles (or subdirectories thereof). But not dist-info? I thought setuptools supported dist-info formats these days. Is this somewhere that got missed? Or is it more subtle than that? I believe it's not recognising *.whl as interesting, so it isn't even looking inside the nested wheel to check for dist-info. That would only be relevant if the .whl weren't on sys.path. Since it's on sys.path, it's processed by importer, not by filename. (It's just that there's no .dist-info detection in the zipimporter handler.) The .whl extension is only relevant for discovery of nested .whl's (wheels within wheels... or within eggs or plain zipfiles...), or .whl's in directories. There isn't any good terminology for describing this at the moment, which makes all this sound much more complicated than it actually is. ;-) Making up some new terminology, suppose we have foo.egg, bar.egg-info, baz.dist-info, and spam.whl in site-packages. Then the bar and baz distributions are mounted at the '.../site-packages' location string, but the foo and spam distributions are merely discoverable at that same location string. (They are *mounted* at '.../site-packages/foo.egg' and '.../site-packages/spam.whl', respectively.) That is, to be mounted at a given location string means to be importable if that location string is on sys.path, and to be discoverable at a given location means to be available for dynamic dependency resolution (e.g. via require()) if that location string is on sys.path. Determining what is mounted or discoverable at a given sys.path location is the job of the find_distributions() API. If the 'only' flag is true, it yields only mounted distributions at the given location string. If false (the default), it yields both mounted and discoverable distributions. Behind the scenes, this is implemented by finding a handler for the importer that the PEP 302 import protocol would use to look for modules at the given location string, and then delegating the operation to that handler. The handler then has to look at the location string and figure out what distributions are mounted and/or discoverable there. To find mounted distributions, the directory handler (find_on_path()) checks whether the directory string itself ends in '.egg' (and could theoretically do the same for .whl), and also looks for contained .dist-info and .egg-info files or directories. To find mountable distributions, it checks for files or directories ending in '.egg' (and could theoretically do the same for .whl). The zipfile handler (find_in_zip()) doesn't actually bother checking for an .egg extension; instead it checks for an EGG-INFO/PKG-INFO and assumes it'll be able to figure things out from that. And it checks for nested .eggs if it's looking for discoverables. So, what it's missing to support Paul's use case is a check for .dist-info/METADATA, analagous to the EGG-INFO/PKG-INFO check. It should be relatively simple to add, if somebody wants to do that. (It can even be done from code outside pkg_resources, as there is a 'register_finder()' API that can be called to register a replacement handler.) In some ways, I'm finding the code structure irritating now, because the one abstraction I *didn't* build into the original framework was a concept that there would ever be competing formats to .egg and .egg-info, so implementing .dist-info and .whl requires annoying repetitions of code at the moment. But it's probably not worth refactoring to make this cleaner, because the odds that there will be a *third* file format needing to be supported any time soon are hopefully quite small. ;-) ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] unparseable sdist filenames
On Mon, Sep 30, 2013 at 1:55 PM, Marcus Smith qwc...@gmail.com wrote: so, take a case like so pytest-xdist-dev.tar.gz (or any sdist with - in the project name, and a version starting with a string) I think it's like so: - pkg_resources.Distribution.from_location will treat xdist-dev as the version. - distlib.util.split_filename won't parse it because versions have to start with [0-9]. For these APIs, these are reasonable defaults, because they are intended only to parse *unambiguous* filenames. Neither API should be used with sdist files, because sdist filenames are ambiguous and cannot therefore be parsed to a single name. The setuptools.package_index API, however, *does* support parsing sdist names, it's just that it generates a *list* of possibilities, rather than a single unambiguous interpretation of the filename. If you ask it to parse the example you gave, you will get a list of: * Project pydist, version xdist-dev * Project pydist-xdist, version dev * Project pydist-xdist-dev, no version Thus, tools using this API can contextually decide which to consider the real interpretation, based on context. This method is used by easy_install; I don't know if pip does as well. - pip will accept this as a pytest archive and install it potentially if no other version matches greater. Does it also accept it as pytest-xdist? what's the right answer? It depends on what you want, and whether what you want is compatible with reality. ;-) The historical-compatible answer is to be confused when projects have -. so stay confused? or get rigid like distlib? The future sdist format should not be ambiguous like this. Dealing with ambiguity in past packages is for the moment something of a requirement if you are making a general-use tool. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] unparseable sdist filenames
On Mon, Sep 30, 2013 at 4:11 PM, Marcus Smith qwc...@gmail.com wrote: The setuptools.package_index API, however, *does* support parsing sdist names, it's just that it generates a *list* of possibilities, oh, ok, setuptools.package_index.distros_for_url Thus, tools using this API can contextually decide which to consider the real interpretation, based on context. This method is used by easy_install; I don't know if pip does as well. how will context decide between the version being dev or xdist-dev? By whether you asked to install pytest-xdist or pytest, and by whether dev or xdist-dev match your version requirements. In practice this tends to only be a problem if you are: 1. Installing the two packages during the same run, and 2. Aren't using version specifiers. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] unparseable sdist filenames
On Mon, Sep 30, 2013 at 6:06 PM, Marcus Smith qwc...@gmail.com wrote: how will context decide between the version being dev or xdist-dev? By whether you asked to install pytest-xdist or pytest, and by whether dev or xdist-dev match your version requirements. In practice this tends to only be a problem if you are: 1. Installing the two packages during the same run, and 2. Aren't using version specifiers. #2 is a big deal. the pip issue (#1192) that motivated this was a #2. local find-links with non-versioned requirements. I should probably add a #3 to that list: it's also rarely a problem because usually there's at least one *numbered* version of the desired package available, which will always prioritize newer than an arbitrary alphabetic version like xdist, even if you haven't requested a specific version. IOW, you need to not only not request a version and come across the other package in the same run, but you *also* need for there not to be any valid versions of your intended target present. so, suppose you have pytest-xdist-dev.tar.gz in a find-links location. whether you're trying to install pytest or pytest-xdist doesn't help the installer determine how to parse that archive. In easy_install's case, it will take the one with the highest version number, which means it'll try to install pytest-xdist-dev.tar.gz, on the theory that 'xdist-dev' is a newer version than 'dev'. This is kind of silly, but it was the Simplest Possible Thing That Could Work, and I figured I'd change it when there was enough feedback/field experience to decide what to change it to. ;-) Probably the right thing to do would be to warn or decline on ambiguous filenames in the future, since it is quite possible to detect if a filename has more than one possible interpretation. Another possibility would be to treat them analagously to stable/unstable versions, prioritize them below regular matches, etc. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] README file format and pypi.python.org
On Sun, Sep 22, 2013 at 9:01 AM, Paul G. paul.l...@isrcomputing.com wrote: 1. What format should I use in my README.txt file for my package's content to be displayed on its package page? It's not the README file; it's the package's long_description keyword, as specified in your setup.py setup() call. And the format is reStructuredText 2. Do I have to use a different extension for the README? It doesn't matter, since the README is not read by PyPI. (You can put code in your setup.py to read the file into the long_description field, though. Take a look at other packages' setup.py files to see how they do it.) ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Deviation between distutils and setuptools over package_data
On Fri, Sep 20, 2013 at 8:15 AM, Jim Fulton j...@zope.com wrote: It appears that MANIFEST.in is ignored if setuptools recognizes your VCS. Not in setuptools 0.6 it isn't. However, it's really only useful for *adding* files not picked up by revision control; it can't *remove* files found by revision control. Manifest and revision control support are semi-independent, as it's quite possible to have files you don't put in revision control but which nonetheless need to be included in an sdist. (For example, you might want to have Cython/Pyrex-generated C code, or compiled documentation files.) ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Deviation between distutils and setuptools over package_data
On Wed, Sep 18, 2013 at 5:07 PM, Benjamin Root ben.v.r...@gmail.com wrote: In creating a source distribution, I have found a disparity between the behaviors of distutils and setuptools with respect to package_data. As of python Issue 2279: http://bugs.python.org/issue2279, entries listed in package_data are used when building an sdist. I have verified that this works in a simple example when setup() is imported from distutils.core. However, if I import setup() from setuptools, it does not pull in the data as listed in package_data (and presumedly data_files). You need to use the include_package_data = True flag. P.S. - on a related note, for a package foo, foo.egg-info directory is created with a SOURCES.txt file. I have found that under certain situations, it seems that a successful install would result in a fully listed SOURCES.txt, and then subsequent calls to sdist seems to use that information to build seemingly correct archives. A co-working double-checking a deployment process I made did an sdist and created a source distribution without the package_data when he did a fresh checkout, but whenever I did it from my development branch, the source distribution worked fine. I haven't figured out exactly how this came about, but it seems to be tied to the SOURCES.txt file. SOURCES.txt mostly exists so that you can safely build an sdist from an sdist, as is required by e.g. bdist_rpm, without having any revision control data on hand to guide the process. Setuptools also can insert a possibly-modified setup.cfg into an sdist for the same reason, so that if you used revision control tags to specify the version when building the sdist, any sdists rebuilt from that sdist will have the same version tags. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pip does not work as expected from a zip
On Thu, Sep 12, 2013 at 2:14 PM, Daniel Holth dho...@gmail.com wrote: I'm suggesting it might be a bug in pkg_resources, or it might be something pkg_resources can already do, if pip.zip is added to PYTHONPATH while executing setup.py in a subprocess. pkg_resources can detect subdirectories in a zip that are named whatever-version.egg/, and treat them like egg files or directories. I don't recall whether it can detect .egg-info in .zip, though; I don't think I specifically implemented that, so if it works, it's by virtue of a good orthogonal design rather than any specific intent for it to work. ;-) My guess, though, is that the basket support (i.e. zipped .egg subdirectories) is the only way at the moment to bundle multiple distributions in a .pyz or other archive. (Note: I don't mean putting an .egg file in the .zip, just zipping up an .egg directory and including it as a subdirectory inside the main .zip/.pyz/whatever.) ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Specification for a local PyPI simple index
On Mon, Sep 9, 2013 at 10:54 AM, Paul Moore p.f.mo...@gmail.com wrote: Is the spec at http://peak.telecommunity.com/DevCenter/EasyInstall#package-index-api still the definitive version of what must be provided for a local PyPI index (for use by something like pip)? Or is there a more up to date version anywhere? That spec is for setuptools 0.6. Later versions should have changed this documentation (in the PyPA repository) if they changed the protocol, but I don't know if anybody's actually keeping tabs on that. For example, are MD5 signatures still the only supported version? I thought we were moving away from MD5. Hm. Yeah, a quick glance at https://bitbucket.org/pypa/setuptools/src shows the docs unchanged, so whoever added the non-MD5 support forgot to check the docs for references to md5. And while I haven't really followed the offsite hosting changes, are the rel=homepage/rel=download links still as stated? (I think I'd want rel=download on everything as I only expect to provide URLs for actual package content). The meaning of re=downloadl values is, spider this page for download links, not this is a link to download.Links to download are identified by inspecting a link, not retrieving it. The only reason the rel tags exist is to mark URLs as spiderable. Also, how definitive is item 7, which states that the root URL must result in a page containing all projects, but it can be omitted if case-insensitive safe_name() matching of projects is implemented? It's definitive for easy_install. The only reason easy_install retrieves the root URL is if a requested package isn't found; the reason it does this is to catch alternative spellings due to case-sensitivity and/or differences in punctuation folding. If you can interpret easy_install's initial GET as a package requirement string (w/case- and punctuation-insensitivity via pkg_resources.safe_name()) rather than as an exact string match, then failure to match would produce the same failure to match on a full package listing, so there's no point having the full listing appear. It's strictly a fallback intended for dumb package indexes that simply consist of a directory tree and a web server providing directory listings. (I think it can even work with an FTP site, but it's been a while since I worked on that code.) The reason I ask is that providing a full listing will be somewhat costly in my application, but providing case-insensitive matching should be doable (actually, I'm not sure yet what's feasible, but I want to know whether it's worth my time even investigating). I don't know what pip does, but I assume that it's probably true of all package managers that either their targeted request succeeds or fails, and then they either request the full listing or they don't. So... the only possible way not providing the full list would be if some (foolish) package manager always began by requesting a full package listing. It's possible there are tools that wish to obtain a full listing and use the base URL for that... but AFAICT it would be a foolish thing to do if you're just trying to install packages. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Distributable binary with dependencies
On Mon, Aug 26, 2013 at 2:25 PM, bharath ravi kumar reachb...@outlook.com wrote: Carl, Eby, Thanks for taking time to suggest various alternatives. Considering that the deployment hosts are identical in every as[ect, the approach of moving virtualenv's with packages pip-installed at build time appears the simplest, low-overhead approach that can be implemented without hacking the environment or resorting to custom scripts. I'll go ahead with that option. What hacking the environment or custom scripts? I'm confused, because AFAIK there are actually more steps to pip-install a virtualenv and copy it to different machines, than there are involved in using easy_install to create a portable installation. In both cases, you end up with a directory to archive and copy, so the only difference is in the commands used to build that directory, and the layout of the directory afterwards. Perhaps you misunderstood my post as meaning that you had to run easy_install on the target system? (I don't have any particular stake in what you do for your own system, but I'm curious, both for the future reference of folks reading this thread by way of Googling this question, and in case there is something for me to learn or that I'm mistaken about, in relation either to pip/virtualenv or your use case. And certainly if you are more familiar with pip+virtualenv, that would actually be sufficient reason in this case to use it. But I'd prefer future readers of this thread not to be under an erroneous impression that easy_install involves more steps, scripts, or environment changes in order to implement this use case. Thanks.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Multi-version import support for wheel files
On Tue, Aug 27, 2013 at 3:01 AM, Paul Moore p.f.mo...@gmail.com wrote: On 27 August 2013 00:15, PJ Eby p...@telecommunity.com wrote: None of these things is wrong. It is *spreading* FUD (and in particular, doing so cynically to undermine a proposal) that is wrong, and I hope I didn't do that - I certainly did not intend to and I'm a bit unhappy about the implication that I might have. Sorry for the implication; it was not intended. I did not think you had any intent to make other people share your doubts or had any desire to shoot down the proposal. As I said, the real intent of my (clearly, in retrospect, very poorly-worded) side-remark was that I thought 90% of the objections to Nick's proposals were based on fear, uncertainty, and doubt rather than any actual issues with the proposals themselves. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Multi-version import support for wheel files
On Mon, Aug 26, 2013 at 5:20 AM, Paul Moore p.f.mo...@gmail.com wrote: On 25 August 2013 23:14, PJ Eby p...@telecommunity.com wrote: Thus, you don't have to know you have multiple versions installed; it can trivially happen by way of dependencies you aren't paying attention to. The more things you install, the more likely it is you have two versions hanging around. OK, I see. But I'm not sure if we're agreeing or disagreeing over the result. To me, this is a bad thing on the principle that there is a cost to multiversion support (it's not part of core Python, so you have to do *something* to make it work) Seriously? The basic functionality of using sys.path to have multiple versions *is* part of core Python, and has been since 1.5.2 (16 years ago), and probably longer than that. In the days before easy_install and virtualenv, if you needed different versions of things, you used setup.py install to different directories (assuming distutils was involved, otherwise you just copied files) and either put your scripts in the same directories, or used PYTHONPATH or explicit sys.path manipulation. That is all easy_install does: add a naming convention for the directories, and automate the sys.path manipulation. Buildout does the same thing, it just writes the sys.path manipulation into the scripts statically, instead of using pkg_resources at runtime. So the notion of cost doesn't make any sense. Tools like easy_install and buildout *reduce* the management cost, they don't add anything to core Python. (Now, if you're talking about the .pth files from easy_install, those are something that I added because people complained about having to use require(), and wanted to have a default version available in the interpreter.) and so having people inadvertently pay that cost to use a feature that they don't actually *need* is wrong. What cost are you talking about here? Given that most people don't even know they *have* multiple versions installed or care, how is a cost being imposed upon them? Are you talking about disk storage? One other point, just as a matter of curiosity (because it's not relevant to the current discussion): in your explanation above, there doesn't seem to be any step that says the user normally uses CherryPy 3 (so that would be the one they would get automatically at the interactive interpreter). If they easy_install that version, sure, that's what they'll get as a default version. For me, that's really the only use case I'd have for multi-versioning - 99% of the time I use a particular version of a project, but I have one particular application that can't work with the version I prefer. Yes, and that's the sort of scenario Nick was proposing pip support, that you have an explicit install me a different version for my other app capability -- such that that app's script wrapper adds its alternate version to sys.path ahead of the default one. So it would have been opt-in and impose the cost of a slightly longer sys.path and increased disk space usage only on those who ask for it. (Honestly, 90% of this entire thread has sounded like complete FUD to me, i.e. fear based on a lack of understanding that there actually isn't anything magical about multi-version support. As Jim has pointed out, buildout does multi-version support without even using pkg_resources. And before all these tools existed, people just installed things in different directories and used either adjacent scripts, PYTHONPATH, or explicit sys.path manipulation. There is nothing magical whatsoever about having multiple versions of a thing installed on your system; all the tools do is add naming conventions for where stuff is installed... and having such naming conventions is a *good* thing, compared to the old days.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Multi-version import support for wheel files
On Mon, Aug 26, 2013 at 11:15 AM, Donald Stufft don...@stufft.io wrote: There is always a cost. In this case mostly in complexity and start up time. As you mentioned originally the cost to multi version support was the need to use a require() function and when people complained about that you added the .pth files which imposed another sort of cost to people using multi versioned installs. See, this is exactly what I'm talking about: you've got this 100% backwards: .pth files are for people who *aren't* using multi-version imports. They're for *default* versions, not alternate versions! And they're utterly unnecessary for Nick's proposal. You claim it is part of core Python but it's really not, if it was it wouldn't require importing pkg_resources of the .pth files to make it work. As I pointed out in the email you apparently didn't read, along with multiple emails from Jim: pkg_resources isn't necessary for alternate-version support. All that's required for alternate versions is to add them to sys.path, which buildout does just fine *without pkg_resources*. I find it ridiculous that you'd call this thread 90% FUD when the vast bulk of the thread has been trying to determine if there were any reasonable concerns with the approach and upon examination determined that the biggest problem with it was attaching it to Wheel and not the multi version support at all What I'm referring to as the FUD is that people have been confusing what Nick proposed with what setuptools does, and getting *both* of them wrong in the details. Nick's proposal was not to mimic setuptools' multi-version support, but rather to provide something else: let's call it alternate version support, to separate it from what setuptools does. In Nick's AVS proposal, there is *no* overhead for anything that doesn't need a non-default version, and it's 100% opt-in, used only for things that need *non-default* versions. Note, by the way, that since these *non-default* packages aren't on sys.path by default, *there is no overhead and no .pth files are involved*. They are effectively invisible and irrelevant for anything that doesn't use them. The only place where there's overhead is in the script that needs the alternative version(s), and its sys.path is lengthened only by those items that it can't obtain from the default sys.path. And if you use buildout's approach of simply adding: sys.path[0:0] = [path1,...] to the head of a script, then *pkg_resources isn't involved either*. This is bog-standard stock Python. So the FUD part I was referring to is all the oh no, setuptools is complicated in response to Nick's perfectly reasonable idea *which doesn't involve any of setuptools' complexity*, because it's doing something completely different. I realize setuptools and easy_install are your baby but the persecution complex doesn't help to win people over to your side of things. I think you're confused here. I don't think setuptools is being persecuted, I think *Nick's idea* is being misunderstood, and being construed as almost the exact *opposite* of what it is. All the stuff people bitch about that relates to multi-versions in setuptools are actually issues with setuptools' implementation of *default* versions, not *alternative* versions. So to look at Nick's proposal and think it's going to have the same problems is completely ludicrous - it's 180 degrees opposite of what setuptools does, because for setuptools, *default versions* are the special case -- they're what cause 90% of the complexity in pkg_resources' manipulation of sys.path, and they're the main reason .pth files are ever used. So it's crazy-making to see people thinking Nick's proposal is going to bring all that crap along, when that's the exact *opposite* of the situation. In my experience setuptools has a lot of good ideas but they are wrapped in bad ideas or implementations that obscure the fact that there *are* good ideas there. I do not believe it to be unreasonable for people to want to make sure that we're standardizing around one of the *good* ideas instead of one of the bad ideas. It would help if people understood the actual facts, then. AFAICT, Nick's proposal doesn't do any of the things that people are worried about, or at the very least does not *require* them. As Jim and I have pointed out more than once, pkg_resources is not a runtime requirement to allow alternative versions to be importable by code that wants them. It would really be a shame to shoot down Nick's idea based on a vague misunderstanding of it. It's a good proposal, and has far less to do with setuptools than most people in the thread seem to think. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Multi-version import support for wheel files
On Mon, Aug 26, 2013 at 5:59 PM, Donald Stufft don...@stufft.io wrote: I think you're confused. The only comments I see in this thread are people doing due diligence to ensure that Nick's proposal *didn't* include the parts of setuptools that we felt were incurring a cost against people not using the feature and expressing a desire *not* to attach it to the Wheel format and instead attach it to another format on it's own. I mean the person you originally quoted even explicitly said I have no objection to this proposal sans some stuff about not wanting it to be attached to Wheels. So I'm not sure how you can take someone saying they have no objection to the proposal and translate it to people are shooting down Nick's proposal. FUD stands for fear, uncertainty, doubt. My comment was that a lot of the original objections to Nick's proposal seemed fearful, uncertain, and doubting, specifically because they were thinking the proposal was proposing things it wasn't. It was you who brought up the idea of persecution; my response was that I don't think anybody's persecuting setuptools, only giving unnecessary levels of doubt to Nick's proposal due to confusion about how it relates (i.e. mostly doesn't) to setuptools. You pounced on a tiny piece of my email to Paul, in which I mainly expressed confusion about his statements about cost. I was having trouble understanding what sort of costs he meant, and in subsequent discussion realized that it's because he and others appeared to have conflated setuptools' default-version issues, with Nick's proposal for handling non-default versions. My comment was that 90% of the thread appeared to stem from this fear, uncertainty, and doubt, based on this misunderstanding, although more precisely worded, what I actually meant was that 90% of the *objections* raised to Nick's proposal were based on the aforementioned fear, uncertainty, and doubt -- i.e., that the objections had nothing to do with that which was being proposed. At one point this weekend, I had intended to write a detailed rebuttal to all of the points that had been raised, but by the time I had time to do so, the discussion was mostly settled and the issue mostly moot... but the impression that 90% of the original objections were misunderstanding-based remained, which led to my (perhaps poorly-phrased) 90% remark. All that being said, I'm not sure why you pounced on that side-comment in the first place; did you think I was personally insulting you or accusing you of something? ISTM that you are making an awfully big deal out of an incidental remark that had very little to do with the main point of the email, and framing it as though I am the one who is making a big deal of something. If you hadn't intervened, I don't see any reason why the conversation wouldn't have reached a peaceable conclusion, and am still puzzled as to why you felt the need to intervene. Your initial email began by disputing facts that you now appear to accept, in that you did not reply to any of my rebuttals to your assertions. But instead of admitting your assertions were in error, you're asserting that I'm the one who's confused. Well, I wasn't before, but I sure am *now*. ;-) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Multi-version import support for wheel files
On Sun, Aug 25, 2013 at 12:58 PM, Jim Fulton j...@zope.com wrote: On Sun, Aug 25, 2013 at 3:06 AM, Nick Coghlan ncogh...@gmail.com wrote: The clumsiness of the __main__.__requires__ workaround aside, the main advantage this offers is that it *should* result in a relatively straightforward addition to pkg_resources to make it work with wheel files as well as eggs. That's important, because anyone that is currently doing side-by-side multi-versioning in Python is using the pkg_resources API to do it, since that's the only option currently available. No. It isn't. Buildout doesn't use pks_resources to do it. (Buildout used pkg_resources at build time to manage package meta data, but I think that's orthogonal to what you're talking about.) I'd also hazard to guess that most of the folks with multi-version installs are using buildout to do it, as buildout does have a fair number of users. FWIW, I would also note that if you use easy_install to install anything, you are quite possibly using multi-version installs without realizing it. (The __main__.__requires__ API is used in easy_install-generated script wrappers, so there isn't any way you'd know about it without paying specific attention.) I don't know how big the buildout users w/known multi-version vs. easy_install users w/implicit multi-version groups are, but I imagine the combined group has got to be pretty darn big. ;-) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Multi-version import support for wheel files
On Sun, Aug 25, 2013 at 4:32 PM, Paul Moore p.f.mo...@gmail.com wrote: On 25 August 2013 20:53, PJ Eby p...@telecommunity.com wrote: FWIW, I would also note that if you use easy_install to install anything, you are quite possibly using multi-version installs without realizing it. (The __main__.__requires__ API is used in easy_install-generated script wrappers, so there isn't any way you'd know about it without paying specific attention.) Unless I'm missing something, I suspect that this over-counts the number of people using multi-version, in the sense that many (the majority?) of wrapper scripts using multi-version do not actually need to,because the users never install more than one version. And quite likely don't even know that they could. That's just it: if you install two programs, one of which needs CherryPy 2 and the other CherryPy 3, then with easy_install this just works, without you having any idea that you even have more than one version installed, unless you for some reason choose to look into it. Thus, you don't have to know you have multiple versions installed; it can trivially happen by way of dependencies you aren't paying attention to. The more things you install, the more likely it is you have two versions hanging around. (The main limiting factor on conflicts isn't a choice to install multiple versions, it's the relative dearth of pinned versions and upper limits on version numbers. If everything just specifies minimum versions, you'll end up using the latest version for everything as the default version. It's only if a package pins or limits a dependency that any conflict is possible to begin with.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Distributable binary with dependencies
On Fri, Aug 23, 2013 at 6:45 AM, bharath ravi kumar reachb...@outlook.com wrote: I'm looking to package an application with all its dependencies for deployment on multiple hosts. I'd like to ensure that there is no compilation or setup step before starting the application in production. An nice to have ability would be to isolate base library dependencies per application (like virtualenv does). Ideally, the development - deployment lifecycle would involve: (a) Build an application archive with all its dependencies baked in (b) Copy archive to a host in production. (c) Unwrap archive (d) Start services. (Note that the build host production hosts are identical in architecture, OS patch level and python version). You can use easy_install -Zmad deployment_dir application, then archive deployment_dir and extract it on the target machines. (Note: application must be a setuptools-packaged project with its dependencies declared, for easy_install to know what to build and deploy.) The Z option means unzip eggs, m means don't worry about the target being on sys.path; we're not trying to install a default version, a means copy all dependencies, even if locally installed already, and d means install libraries and scripts to the following directory. So, the scripts will be put inside deployment_dir with a bunch of adjacent subdirectories containing all the compiled and ready-to-use libraries. The resulting directory is a portable installation of application: as long as the entire subdirectory is copied to the target machines, everything should work just fine. None of the dependencies or the app itself will interfere with other Python code installed on the target system; it is in a sense a minimal virtualenv which will run whatever scripts that easy_install puts in that directory. One note: the target machines *will* need pkg_resources installed, and it will not be included in the directory by default. If they don't have local copies installed (due to e.g. setuptools, distribute, etc. being installed), you can manually copy a pkg_resources.py to the deployment directory, and it will be used by whatever scripts are in that directory. While there may be other tools available that support this kind of thing, I don't think any of them can do it quite this simply. This deployment scenario was actually a key use case for the original design of easy_install and eggs, so it actually works pretty decently for this. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Wed, Aug 21, 2013 at 9:24 AM, Donald Stufft don...@stufft.io wrote: An example is the wsgiref from the standard library. It's an example, alright, but not for your side. ;-) The wsgiref library doesn't just implement the spec, it implements a ton of utility classes for use with the spec. The validator was almost an afterthought grafted on later, borrowed from another project. It implements a framework with all sorts of features that are not technically part of the spec, but are just useful if you want to implement the spec. Very few of the classes, methods, etc. in the entire package are specified by the spec, except in the sense that many of them match a calling signature defined in the PEP. (The PEP doesn't specify any method names, except for things like read() on file-like objects.) IOW, wsgiref is a collection of generally useful tools for anybody doing things with the spec, as an combination of examples of how to do this and ready-to-use code for working with the spec. Personally, I'm very happy to see Vinay's extensions, because they are IMO important validations of whether the new specs are likely to be useful for replacing all of setuptools' functionality. There are people who need to mount eggs and have their extensions run, so if it wasn't possible to build tools that support them under the new specs (whether that support is required by the spec or not), that would still be a reason to use setuptools -- meaning, IMO, that the new spec effort is failing to create a unified packaging world. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Tue, Aug 20, 2013 at 12:39 PM, Thomas Heller thel...@ctypes.org wrote: Ok, now I understand. But the zipfile could contain a loader-module for each extension which does something like this (this example extracts and loads 'bz2.pyd'): ... (py2exe for Python 3, which is work in progress, uses this approach) Setuptools has also done this since the egg format was developed, but it has some well-known problems, which unfortunately your example has worse versions of. ;-) Setuptools takes the approach of keeping a per-user cache directory (so that cleanup isn't required, and so there are no security issues where somebody can replace a tempfile between you writing it and importing it), and it uses a unique subdirectory per egg so that different (say) bz2.pyd files can't conflict with each other. Even with these adjustments, Unix users frequently run into issues where the user a process is running as doesn't have access to a suitable cache directory, and so it's a common complaint about the use of zipped eggs. I thought that at one point you (Thomas) had come up with a way to load modules into memory from a zipfile without needing to extract them. Was that you? If so, how did that work out? (ISTR that there was some sort of licensing issue, too.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Changing the install hooks mechanism for PEP 426
On Fri, Aug 16, 2013 at 6:21 AM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote:
PJ Eby pje at telecommunity.com writes:
I guess I didn't explain it very well, because that's roughly what I
meant: a single namespace for all extensions, structured as a
mapping from group names to submappings whose keys can be arbitrary
values, but whose values must then be either a string or a JSON
object, and if it's a string, then it should be an export specifier.
Why should the keys be completely arbitrary?
By arbitrary I mean only that the PEP doesn't place syntactical
restrictions on them.
I can't see a reason for this;
the current constraint of the prefixed name seems sufficient. What would
relaxing this constraint make possible which otherwise isn't?
I think you're thinking I'm describing a single level namespace; I'm
referring to something like this:
{group1: {anykey: export_or_mapping}}
anykey is not validated by the spec, only by registered validators
for group1. Of course it has to have some meaning that is
interpretable by consumers of group1. The point is that the *spec*
only defines the syntax of group names and export strings, and it's
left to specific groups to define the syntax/semantics of the keys.
On the values: an export specifier is just a more human-friendly version of
a dict with module/content/extra keys. While of course the uses of
importables in this area is well established, what specific use cases are we
enabling by allowing arbitrary JSON? It certainly would clutter the metadata
and render it less human-readable, and the only thing it provides is a dict
which could be expressed in an importable form
I gave one example already: i18n/l10n information that's about files
contained in the distribution's data. It's quite possible to have
distributions without any code, only data of that kind. A requirement
to create code, just to specify the data seems rather pointless. In
Nick's reply, he's listed other use cases.
The main question is, should exports and extensions be treated
separately? Nick originally proposed merging the concepts and using
arbitrary JSON. My counterproposal was to say, let's distinguish
exports and extensions by restricting the spec to something which
spells out the distinction.
After this further discussion, I think that the use cases we're
discussing really boil back down to exports vs. metadata extensions,
and that maybe we should stick to them being separate.
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Changing the install hooks mechanism for PEP 426
On Fri, Aug 16, 2013 at 8:04 AM, Nick Coghlan ncogh...@gmail.com wrote: Concrete extension use cases I have in mind that don't fit in the exports/entry-point data model: - the mapping of prebuilt executable names to wheel contents - platform specific external dependencies and other hints for conversion to platform specific formats (e.g. Windows GUIDs) - metadata for build purposes (e.g. the working directory to use when building from a VCS checkout, so you can have multiple projects in the same repo) - project specific metadata (e.g. who actually did the release) - security metadata (e.g. security contact address and email GPG fingerprint) This is why extensions/exports were originally separate, and may still remain that way. But exports should still be used for hooks defined by the spec; they are the Obvious Way to specify importable hooks, and *not* dogfooding there is still a bad idea. (To be clear, I was never exactly *enthused* about the idea of merging extensions and exports, just *unenthused* about the idea of the spec using extensions or custom metadata to do things that could be perfectly well expressed as exports from a reserved namespace.) I'm kind of toying with the idea that the core metadata itself could be carved into extension namespaces, have the core itself be just another extension, rather than nesting extensions and exports inside the core, so that the entire spec is just a relatively-flat collection of namespaces, in more human-digestible groups. There are some conceptual and practical advantages to that, at least in principle, but until I've played around with actually expressing some concepts from the PEP that way, I won't know whether it would actually pay off in practice. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Changing the install hooks mechanism for PEP 426
On Thu, Aug 15, 2013 at 9:21 AM, Nick Coghlan ncogh...@gmail.com wrote: On 15 Aug 2013 00:39, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: PJ Eby pje at telecommunity.com writes: The build system *should* reserve at least one (subdivisible) namespace for itself, and use that mechanism for its own extension, +1 - dog-food :-) Sounds fair - let's use pydist, since we want these definitions to be somewhat independent of their reference implementation in distlib :) I think that as part of the spec, we should either reserve multiple prefixes for Python/stdlib use, or have a single, always-reserved top-level prefix like 'py.' that can be subdivided in the future. Extensions are a honking great idea, so the stdlib will probably do more of them in the future. Likewise, future standards and informational PEPs will likely document specific extension protocols of general and specialized interest. (Notice, for example, that extensions could be used to publicize what database drivers are installed and available on a system.) Based on PJE's feedback, I'm also starting to think that the exports/extensions split is artificial and we should drop it. Instead, there should be a validate export hook that build tools can call to check for export validity, and the contents of an export group be permitted to be arbitrary JSON. I think there is still something to be said for STASCTAP: simple things are simple, complex things are possible. (Also, flat is better than nested.) So I would suggest that an export can either be an import identifier string, *or* a JSON object with arbitrary contents. That would make it easier, I think, to implement both a full-featured replacement for setuptools entry point API, and allow simple extensions to be simple. It means, too, that simple exports can be defined with a flatter syntax (ala setuptools' ini format) in tools that generate the JSON. Given how many use cases are already met today by providing import-based exports, ISTM that they are the 20% that provides 80% of the value; arbitrary JSON is the 80% that only provides 20%, and so should not be the entry point (no pun intended) for people dealing with extensions. Removing the extension/export split also raises a somewhat different question, which is what to *call* them. I'm sort of leaning towards extensions as the general category, with exports being extensions that consist of an importable object, and JSON extensions for ones that are a JSON mapping object. So the terminology would be: Extension group - package like names, subdivisible as a namespace, should have a prefix associated with a project that defines the semantics of the extension group; analagous to Eclipse's notion of an extension point Extension name - arbitrary string, unique per distribution for a given group, but not required to be globally unique even for the group. Specific names or specific syntax for names may be specified by the creators of the group, and may optionally be validated. Extension object - either an export string specifying an importable object, or a JSON object. If a string, must be syntactically valid as an export; it is not, however, required to reference a module in the distribution that exports it; it *should* be in that distribution or one of its dependencies, however. So, an extension is machine-usable metadata published by a distribution in order to be (optionally) consumed by other distributions. It can be either static JSON metadata, or an importable object. The semantics of an extension are defined by its group, and other extensions can be used to validate those semantics. Any project that wants to be able to use plugins or extensions of some kind, can define its own groups, and publish extensions for validating them. Python itself will reserve and define a group namespace for extending the build and installation system, including a sub-namespace where the validators can be declared. So we would have pydist.commands and pydist.export_hooks as export groups, with distlib used as an example of how to define handlers for them. Is 'commands' for scripts, or something else? Following flat is better than nested, I would suggest not using arbitrary JSON for these when it's easy to define new dotted groups. (Keeping to such a style will make it easier for humans to define this stuff in the first place, before it's turned into JSON.) (Note, btw, that having more dots in a name does not necessarily equal nested, whereas replacing those dots with nested JSON structures most definitely *is* nested!) Similarly, I'd just as soon see e.g. pydist.hooks.* subgroups, rather than a dedicated data structure. A 'pydist.validators' group would of course also be needed for syntax validation, with extension names in that group possibly allowing trailing '*' or '**' wildcards. (There will of course need to be a validation API, which is why I think that a separate PEP for the extensions system is probably going
Re: [Distutils] Changing the install hooks mechanism for PEP 426
On Thu, Aug 15, 2013 at 7:16 PM, Nick Coghlan ncogh...@gmail.com wrote: But if we're only going to validate it via hooks, why not have the mapping of names to export specifiers just be a recommended convention for extensions rather than a separate exports field? I guess I didn't explain it very well, because that's roughly what I meant: a single namespace for all extensions, structured as a mapping from group names to submappings whose keys can be arbitrary values, but whose values must then be either a string or a JSON object, and if it's a string, then it should be an export specifier. To put it another way, I'm saying something slightly stronger than a recommended convention: making it a requirement that strings at that level be import specifiers, and only allowing mappings as an alternative. In that way, there is a minimum level of validation possible for the majority of extensions *by default*, without needing an explicit validator declared. To put it another way, it ensures that there's a kind of lingua franca or lowest-common denominator that lets somebody understand what's going on in most extensions, without having to understand a new *structural* schema for every extension group. (Just a *syntactical* one) As an extension, pydist.extension_hooks would also be non-conventional, since it would define a new namespace, where extension names map to an export group of hooks. A separate export group per hook would be utterly unreadable. If you already know what keys go in an entry point group, there's a good chance you're doing it wrong. Normally, the whole point of the group is that the keys are defined by the publisher, not the consumer. The normal pattern is that the consumer names the group (representing a hook), and the publishers name the extensions (representing implementations for the hook). I don't see how it makes it unreadable, but then I think in terms of the ini syntax or setup.py syntax for defining entry points, which is all very flat. IMO, creating a second-level data structure for this doesn't make a whole lot of sense, because now you're nesting something. I'm not even clear why you need separate registrations for the different hooks anyway; ISTM a single hook with an event parameter is sufficient. Even if it weren't, I'd be inclined to just make the information part of the key in that case, e.g. [pydist.extension_listeners] preinstall:foo.bar = some.module:hook This sort of thing is very flat and easy to express in a simple configuration syntax, which we really shouldn't lose sight of. It's just as easy to have write a syntax validator as a structure validator, but if you start with structures then you have to back-figure a syntax. I'd very much like it to be easy to define a simple flat syntax that's usable for 90%+ of extension use cases... which means I'd rather not see the PEP make up its own data structures when they're not actually needed. Don't get me wrong, I'm okay with allowing JSON structures for extensions in place of export strings, but I don't think there's been a single use case proposed as yet that actually *works better* as a data structure. If you need to do something like have a bunch of i18n/l10n resource definitions with locales and subpaths and stuff like that... awesome. That's something that might make a lot of sense for JSON. But when the ultimate point of the data structure is to define an importable entry point, and the information needed to identify it can be put into a relatively short human readable string, ISTM that the One Obvious Way to do it is something like a setuptools entry point -- i.e. a basic key-value pair in a consumer-defined namespace, mapping a semantically-valued name to an importable object. And *most* use cases for extensions, that I'm aware of, fit that bill. You have to be doing something pretty complex to need anything more complicated, *and* there has to be a possibility that you're going to avoid importing the related code or putting in on sys.path, or else you don't actually *save* anything by putting it in the metadata. IOW, if you're going to have to import it anyway, there is no point to putting it in the metadata; you might as well import it. The only things that make sense to put in metadata for these things are data that tells you whether or not you need to import it. Generally, this means keys, not values, in other words. (Which is why l10n and scripts make sense to not be entry points: at the time you use them, you're not importing 'em.) That's why I'm still inclined to make this one a separate top level field: *installers* have to know how to bootstrap the hook system, and I like the symmetry of separate, relatively flat, publication and subscription interfaces. I don't really see the value of a separate top-level field, but then that's because I don't see anything at all special about these hooks that demands something more sophisticated than common entry points. AFAICT it's a
Re: [Distutils] How to handle launcher script importability?
On Wed, Aug 14, 2013 at 7:34 AM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: Jason R. Coombs jaraco at jaraco.com writes: This means that instead of installing, for example: Scripts\my-command.exe Scripts\my-command-script.py Scripts\my-command.exe.manifest Just to muddy the waters a little, I'd like to suggest an alternative approach which doesn't appear to have been tried: 1. The installer just installs foo.exe on Windows for script 'foo', where the foo.exe contains the actual script 'foo' as a resource. 2. The launcher, before looking for foo-script.py, examines its resources. If a script resource is found, it is extracted and written to foo-script.py in the same directory as foo.exe. If such a resource isn't found, it continues to the next step. 3. The launcher looks for 'foo-script.py' in its directory, invokes it and waits for it to complete. 4. If a 'foo-script.py' was written in step 2, it is deleted. The launcher comes with an embedded manifest, so no external manifest is needed. Better suggestion: just append a PEP 441 .pyz to the .exe, and no extraction is necessary; the .exe just reads out the #! part. For Python 2.6 and up, the .exe can simply pass itself as argv[1] to the interpreter. (For older Pythons, a little -c and PYTHONPATH munging is required, but should be doable.) For bonus points, you can actually stick a compatibly-built wheel on the end of the .exe instead, and embed the entire relevant project. ;-) Thoughts? Writing the script.py file means the current user needs write access to a program installation directory, which is probably not a good idea. Also, what if two instances are running, or you overwrite an existing script while it's being read by Python in another process? No, if you're taking the embedding route, it's got to be either a zipfile, or else you have to use -c and give Python an offset to seek to in the file. In any case, it'd probably be a good idea to offer some command line tools for manipulating such .exes, to e.g. show/change what Python it's set to use, extract/dump/replace the zip, etc. (As for ctypes, if that's needed for this approach (which I somewhat doubt), there are official Windows binaries available for 2.3 and 2.4.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Wed, Aug 14, 2013 at 9:58 AM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote:
IIUC PEP 441 is about tooling to create archives; don't we just need a
Python-compatible .zip (i.e. with a __main__.py)?
I meant that it has a #! line before the zip part, so that the
launcher knows what Python to invoke.
There are also some challenges for older Pythons to invoke __main__,
since the normal Python import machinery frowns on reloading __main__.
I expect the zip would need an extra __main.py stub to bootstrap the
loading of __main__, and then invoke python with something like '-c
__import__('sys').path[0:0]=['/path/to','path/to/exe''];
__import__('__main').go()'.
(It can't have the import run the app as a side effect, because
otherwise the import lock will be held, leading to Bad Things in
multi-threaded apps.)
This is less helpful; one might have N scripts per project, no need to stick
the whole project in with each one, or am I misunderstanding?
I just meant that for cases where there's only one script, or where
you are doing a custom-built application. This also becomes The One
Obvious Way to do py2exe-like things.
How would such an offset be used? Are you saying the -c scriptlet would use
that offset to extract the script? Or do you mean something else?
Extract the script by seeking to the offset and reading it. It's far
from ideal, though; the .zip is much better since everything back to
2.3 can support it in some fashion.
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Changing the install hooks mechanism for PEP 426
On Wed, Aug 14, 2013 at 11:36 AM, Nick Coghlan ncogh...@gmail.com wrote: * group - name of the export group to hook * preupdate - export to call prior to installing/updating/removing a distribution that exports this export group * postupdate - export to call after installing/updating/removing a distribution that exports this export group * refresh - export to call to resynchronise any caches with the system state. This will be invoked for every distribution on the system that exports this export group any time the distribution defining the export hook is itself installed or upgraded I've reread your post a few times and I'm not sure I understand it. Let me try and spell out a scenario to see if I've got it: * Distribution A defines a refresh hook for group 'foo.bar' -- but doesn't export anything in that group * Distribution B defines an *export* (fka entry point) -- any export -- in export group 'foo.bar' -- but doesn't define any hooks * Distribution A's refresh hook will be notified when B is installed, updated, or removed Is that what this is for? If so, my confusion is probably because of overloading of the term export in this context; among other things, it's unclear whether this is a separate data structure from exports themselves... and if so, why? If I were doing something like this in the existing entry point system, I'd do something like: [mebs.refresh] foo.bar = my.hook.module:refresh i.e., just list the hooks in an export group, using the export name to designate what export group is being monitored. This approach leverages the fact that exports already need to be indexed, so why create a whole new sort of metadata just for the hooks? (But of course if I have misunderstood what you're trying to do in the first place, this and my other thoughts may be moot.) (Oh, and btw, if a distribution has hooks for itself, then how are you going to invoke two different versions of the code? Rollback sys.modules and reload? Spawn another process?) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Changing the install hooks mechanism for PEP 426
On Wed, Aug 14, 2013 at 3:14 PM, Nick Coghlan ncogh...@gmail.com wrote: On 14 August 2013 14:00, PJ Eby p...@telecommunity.com wrote: On Wed, Aug 14, 2013 at 11:36 AM, Nick Coghlan ncogh...@gmail.com wrote: * group - name of the export group to hook * preupdate - export to call prior to installing/updating/removing a distribution that exports this export group * postupdate - export to call after installing/updating/removing a distribution that exports this export group * refresh - export to call to resynchronise any caches with the system state. This will be invoked for every distribution on the system that exports this export group any time the distribution defining the export hook is itself installed or upgraded I've reread your post a few times and I'm not sure I understand it. Let me try and spell out a scenario to see if I've got it: * Distribution A defines a refresh hook for group 'foo.bar' -- but doesn't export anything in that group * Distribution B defines an *export* (fka entry point) -- any export -- in export group 'foo.bar' -- but doesn't define any hooks * Distribution A's refresh hook will be notified when B is installed, updated, or removed No, A's preupdate and postupdate hooks would fire when B (or any other distro exporting the foo.bar group) is installed/updated/removed. refresh() would fire only when A was installed or updated. Huh? So refresh is only relevant to the package itself? I guess I don't understand the point of that, since you get the same info from postupdate then, no? I realised that my proposed signature for the refresh() hook is wrong, though, since it doesn't deal with catching up on *removed* distributions properly. Rather than being called multiple times, refresh() instead needs to be called with an iterable providing the metadata for all currently installed distributions that export that group. Ah. But then why is it called for A, instead of.. oh, I think I see now. Gotcha. This is the sort of thing that examples are really made for, so you can see the use cases for the different hooks. If so, my confusion is probably because of overloading of the term export in this context; among other things, it's unclear whether this is a separate data structure from exports themselves... and if so, why? Where exports is about publishing entries into an export group, the new export_hooks field would be about *subscribing* to an export group and being told about changes to it. That's not actually a justification for not using exports. While you could use a naming convention to defined these hooks directly in exports without colliding with the export of the group itself, but I think it's better to separate them out so you can do stricter validation on the permitted keys and values (the rationale is similar to that for separating out commands from more general exports, and exports from arbitrary metadata extensions). The separation of commands is (just barely) justifiable because it's not a runtime use, it's installer use. Stricter validation, OTOH, is a completely bogus justification for not using exports, otherwise nobody would ever have any reason to use exports, everybody would have to define their own extensions so they could have stricter validation. ;-) The solution to providing more validation is to use *more* export groups, e.g.: [mebs.export_validators] mebs.refresh = module.that.validates.keys.in.the.refresh.group:somefunc (In other words, define hooks for validating export groups, the way setuptools uses an entry point group for validating setup keywords.) Of course, even without that possibility, the stricter validation concept is kind of bogus here: the only thing you can really validate is that syntactically valid group names are being used as export names, which isn't much of a validation. You can't *semantically* validate them, since there is no global registry of group names. So what's the point? The build system *should* reserve at least one (subdivisible) namespace for itself, and use that mechanism for its own extension, for two reasons: 1. Entities should not be multiplied beyond necessity, 2. It serves as an example of how exports are to be used, and 3. The API is reusable... No, three reasons! Wait, I'll come in again... the API is reusable, it serves as an example, no duplication, and namespaces are a good idea, let's do more of them... no, four reasons... chief amongst the reasonry... Seriously: I can *sort of* see a reason to keep commands separate, but that's a meh. I admittedly just grabbed it as a handy way to shoehorn that functionality into setuptools. But keeping extensions to the build system itself in a separate place? No, a thousand times no. This sort of extensibility is *precisely* what the darn things are *for*. If the build system doesn't use them, what's the point? Mostly so you can validate them and display them differently, and avoid reserving any part
Re: [Distutils] How to handle launcher script importability?
On Wed, Aug 14, 2013 at 2:14 PM, Paul Moore p.f.mo...@gmail.com wrote: .pyw files can be imported as modules, just like .py, Darn. Okay, so another extension *is* needed if you want to be able to make non-console apps runnable-but-not-importable. IIUC it should by '.pywa' rather than '.pya', though, because of the issue with only the first three characters of an extension working in PowerShell, which means it would be executed by the wrong PyLauncher under some circumstances. (Honestly, though, I'm not sure whether anybody cares about PATH/PATHEXT in relation to GUI apps; ISTM they'd mostly be invoked by shortcuts, and there's also a registry mechanism that's supposed to be used for these things nowadays, rather than PATH... but I think it only works for .exe's, so there we go again back to the land of .exe's just plain Work Better On Windows.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Tue, Aug 13, 2013 at 8:54 AM, Jason R. Coombs jar...@jaraco.com wrote:
1. Renames, deletes, and other actions must be synchronized.
Why are you manually deleting or altering executables? Why are you
renaming them at all?
I've been using .exe wrappers since they were written, and have never
had a single one of the issues you mention, because I never do any of
the things you mention by hand. IMO that's what tools are for.
Doesn't pip uninstall scripts?
I may be slightly biased in my preference for .exe, because files with
other extensions don't work with Cygwin (which doesn't support
PATHEXT), but I work primarily with Windows Python rather than Cygwin
Python. So, if there *has* to be a single file, I would greatly
prefer an .exe with the script embedded, rather than a non-.exe file.
It's a bit less discoverable, but at least it'll discourage anybody
from editing the contents. (Because nobody should be editing
generated scripts anyway.)
(Also relevant: not every situation where wrapper scripts are used is
going to be one where a PyLauncher install is possible. For example,
portable deployment of an app to USB stick with a bundled Python can't
assume PATHEXT and a globally-installed PyLauncher.)
4. Updates to the launcher won't apply to existing scripts. If the launcher is
updated, the side-by-side versions will remain out-of-date until their scripts
are re-installed.
This is kind of a bogus point; *any* update to how scripts are
generated isn't automatically applied to existing scripts; the format
in which they're written is of no relevance.
5. Files in use can't be replaced. Because a Windows executable that's in use
is not allowed to be overwritten,
But they can be renamed, and deleted afterwards. For example, when
updating, you can do the simple dance of:
1. delete scriptname.exe.deleteme if it exists
2. rename scriptname.exe to scriptname.exe.deleteme
3. replace scriptname.exe
4. try to delete the .deleteme file created in step 2, ignoring errors.
And since this only needs to be done for the wrappers on installation
tools themselves (pip, easy_install, etc.), it's not like a lot of
people are going to have to write this code.
It can also be further enhanced, by having the .exe wrapper check (as
it exits) whether it was renamed, and if so, spin off a 'python -c
import os, time; time.sleep(0.1); os.unlink('path to .deleteme')'
and immediately exit. (Or use one of the other tricks from
http://www.catch22.net/tuts/self-deleting-executables -- but I think
this one is the simplest and best for our purposes, since the wrapper
already knows at this point it can invoke Python using the path it
previously found, and it's not doing anything questionable with
process invocations that might raise red flags with security tools.)
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Tue, Aug 13, 2013 at 12:33 PM, Paul Moore p.f.mo...@gmail.com wrote: This works, but is an ugly, fragile workaround. It's *not* a huge problem, it's just how executables work on Windows, and all installers have to deal with this dance (it's why a lot of things need a reboot to complete installation - the delete on next reboot API). But it's not *nice*. I would never use exe wrappers for my own personal scripts - I *always* write them as .py files and rely on PATHEXT. I only use exe wrappers for commands installed as part of a Python package (pip.exe, nosetests.exe, etc). That says something about how friendly they are as a general tool. In an ironic reversal, I use them for any command I plan to use frequently. In other words, if I use it often enough to care about how easy it is to use, I take the trouble to wrap it in a project and then use setup.py develop to create the script wrappers. From then on, I can edit the *source* scripts, and the wrappers run the right thing. (I don't edit the -script.py's directly, since they're not where the real code is.) On another point you mention, Cygwin Python should be using Unix-style shell script wrappers, not Windows-style exes, surely? The whole point of Cygwin is that it emulates Unix, after all... So I don't see that as an argument either way. I said I'm using *Windows* Python from the Cygwin shell. I often test my projects with Cygwin Python, to ensure coverage of Unixisms, but I only write dedicated Cygwin Python scripts if I need to use Cygwin paths or APIs, which is relatively infrequent. In any case, the use of .exe means that my invocation patterns are unchanged between commands I've implemented in Cygwin Python vs. Windows Python. If the Windows Python versions used a different extension, then I'd have to remember whether which language a specific command was written in in order to invoke it. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Mon, Aug 12, 2013 at 7:33 AM, Nick Coghlan ncogh...@gmail.com wrote: Having pys and pyz for executable, but not importable (source and zip archive forms) could be quite clean. In effect, the pys extension would bring windows to parity with *nix, where no extension at all has traditionally served the purpose of making it impossible to import a script. Oh, that reminds me: IIUC, it's not necessary to *actually* zip a .pyz. Remember, Python doesn't use the extension to determine the contents, it sniffs for a zip trailer. Likewise, there was IIRC no plan for pylauncher to inspect zip contents -- it just reads the #! line and runs python on the file. This means that you can actually write source as a .pyz or .pwz file on Windows, and it would Just Work -- *without any sys.path modification*. For *all versions of Python*, provided PyLauncher is installed. (And setuptools could check the environment and registry for that, if you opt into the non-.exe scripts, and error out if you don't have them set up correctly.) IOW, implementing PEP 441 in PyLauncher gives us the executable, not importable format for free. (Granted, I can see reasons for not wanting to use the same extension for source and zipped versions, mostly in the area of tools other than pylauncher, but if you do have different extensions then there have to be *four*, due to console vs. windowed and source vs. zipped.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Mon, Aug 12, 2013 at 10:32 AM, Paul Moore p.f.mo...@gmail.com wrote: On 12 August 2013 14:01, PJ Eby p...@telecommunity.com wrote: As far as zipped Python applications are concerned (pyz), these can be created by just using a pys file containing a #! line prepended to the zip file. Certainly, it's a binary file with a filename that would normally indicate a text file format, but is that any less true on Unix when users create these files? I don't know what the user experience with zipped Python applications on Unix is like - I doubt it's *that* much better than on Windows. Probably the reality is that nobody uses zipped applications anyway, so the problems haven't been identified yet. Maybe the pyz PEP would bet better rewritten to propose providing tools to create and manage zipped Python applications, but *not* to require new extensions, merely to reuse existing ones (pys on Windows, no extension on Unix) with binary (zipped) content. Seems reasonable... but then somebody will need to write another PEP for the file extension(s) issue. I think the issue of too many extensions vs. source/binary confusion is going to boil down to a BDFL judgment call, whether it's by Nick, Guido, or some more Windows-specific BDFL For One PEP. If we go with One Extension To Rule Them All, I would actually suggest '.pyl' (for PyLauncher), since really all that extension does is say, hey, run this as a console app via PyLauncher, not that it's a script (which would be assumed to be text). And that all you can be sure of is that a .pyl files will start with a #! line, and launch whatever other program is specified there, on the contents of the file -- which may actually be a zipfile. PS Either the ref file marker approach, or a new Python command line argument with appropriate behaviour, could avoid the need for even the pys/pws extension, if people prefer to reduce the number of extensions claimed still further. But those would only be available for future Python versions. A file extension would solve the problem upon installing PyLauncher and PATHEXT, at least for those OSes and shells that recognize PATHEXT. Hm, here's a side thought: what if PyLauncher added the ability to serve as a script wrapper, just like setuptools' existing wrappers? Then setuptools could just copy py.exe or pyw.exe alongside a .pyl or .pyw, and presto! No PATHEXT compatibility needed, but users could still opt out of using the .exe wrappers if they're sure their shell works right without it. (The wrapper facility would be implemented by simply checking for an adjacent file of matching filename and extension (.pyl for py.exe, .pyw for pyw.exe), and if found, insert that filename as argv[1] before proceeding with the normal launch process. For efficiency, the file check could be skipped if the executable has its original name, at the minor cost of it not being possible to name a console script 'py' or a windows app 'pyw'. But that's an optional tweak.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Mon, Aug 12, 2013 at 2:14 PM, Jason R. Coombs jar...@jaraco.com wrote: -Original Message- From: Distutils-SIG [mailto:distutils-sig- bounces+jaraco=jaraco@python.org] On Behalf Of PJ Eby Sent: Monday, 12 August, 2013 11:22 On Mon, Aug 12, 2013 at 10:32 AM, Paul Moore p.f.mo...@gmail.com wrote: On 12 August 2013 14:01, PJ Eby p...@telecommunity.com wrote: As far as zipped Python applications are concerned (pyz), these can be created by just using a pys file containing a #! line prepended to the zip file. Certainly, it's a binary file with a filename that would normally indicate a text file format, but is that any less true on Unix when users create these files? I don't know what the user experience with zipped Python applications on Unix is like - I doubt it's *that* much better than on Windows. Probably the reality is that nobody uses zipped applications anyway, so the problems haven't been identified yet. Maybe the pyz PEP would bet better rewritten to propose providing tools to create and manage zipped Python applications, but *not* to require new extensions, merely to reuse existing ones (pys on Windows, no extension on Unix) with binary (zipped) content. Seems reasonable... but then somebody will need to write another PEP for the file extension(s) issue. My preference is to reject the idea of the side-by-side executable launcher. There are several downsides that I'm trying to avoid by moving away from the executable: 1. Disparity with Unix. Better parity means cleaner code, easier documentation, and less confusion moving from platform to platform. 2. Executables that look like installers. If a launcher executable is used and Windows detects that it looks like an installer and it's a 32-bit executable and it doesn't have a manifest to disable the functionality, Windows will execute the file in a separate UAC context (often in a separate Window). 3. Additional files are needed. In particular, due to (2), a manifest must be provided for 32-bit executables. 4. Word size accounting. It's not clear to me what word size is needed. 32-bit may be sufficient, though 64-bit seem to have some advantages: a manifest is not needed, and it can match the word size of the installed Python executable (for consistency). Setuptools currently provides both (and installs the one that matches the Python executable). 5. Platform support. We're fortunate that Windows is one of the most stable binary platforms out there. Nevertheless, Setuptools recently got support for AMD binaries in the launcher. By relying on an external launcher, the launcher becomes responsible for platform support. 6. Two to three files to do the job of one. In fact, the job isn't much more than to invoke code elsewhere, so it seems ugly to require as many as three files to do the job. Then multiply that by the Python-specific version and you have up to six files for a single script. 7. Obfuscation of purpose. A single script pretty directly communicates its purpose. When there are multiple files, it's not obvious why they exist or what their purpose is. Indeed, I went years without realizing we had an open issue in Distribute due to a missing manifest (which was fixed in Setuptools), all because I used the 64-bit executable. While it may take some time for the community to learn what a '.pyl' is, it's easily documented and simple to grasp, unlike the subtle and sometimes implicit nuances (and fragility) of a side-by-side executable. 8. Unwanted content. Some Unix users have complained about finding Windows executables in their Linux packages, so now Setuptools has special handling to omit the launchers when installed on Unix systems. This is far from beautiful. I think the issue of too many extensions vs. source/binary confusion is going to boil down to a BDFL judgment call, whether it's by Nick, Guido, or some more Windows-specific BDFL For One PEP. If we go with One Extension To Rule Them All, I would actually suggest '.pyl' (for PyLauncher), since really all that extension does is say, hey, run this as a console app via PyLauncher, not that it's a script (which would be assumed to be text). And that all you can be sure of is that a .pyl files will start with a #! line, and launch whatever other program is specified there, on the contents of the file -- which may actually be a zipfile. If it's '.py*', I don't see why it's not reasonable to allow omission of the shebang, and assume the default python. After encountering and now understanding the subtle import semantics, I'm hoping that this new extension can also be used in my personal 'scripts' collection to serve the same purpose it does for setuptools console entry points. I guess one could require #!/usr/bin/python in each, but that seems superfluous on Windows. I don't feel at all strongly on this point. PS Either the ref file marker approach, or a new Python command line
Re: [Distutils] How to handle launcher script importability?
On Mon, Aug 12, 2013 at 4:04 PM, Paul Moore p.f.mo...@gmail.com wrote: I would still like to see the standard be registered .pye (I'm happy with a bikeshed of this colour) and .pwe extensions which are added to PATHEXT and As long as we're discussing bikeshed colors, I'd like to counterpropose .pya and .pwa, to be registered in the Windows class registry as Python Console Application and Python Windowed Application, respectively. Since the *only* reason we need these extensions is for Windows (other OSes do fine at making things executable without an extension), and Windows calls things like these Applications or Apps in Explorer normally, I think it's better to call them what Windows calls them. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Mon, Aug 12, 2013 at 4:29 PM, Donald Stufft don...@stufft.io wrote: Hopefully this all will solve this problem, as it is right now if you use setuptools entry points then Wheels erroneously pretend to be platform agnostic. IMO it's okay to give up having ready-to-use scripts in a platform-agnostic wheel; scripts are sadly not a platform-agnostic thing. (If only MS-DOS had been a little bit more Unix, and a little less CP/M...) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Sun, Aug 11, 2013 at 10:38 AM, Jason R. Coombs jar...@jaraco.com wrote: In Setuptools 1.0 (currently in beta), I've added an experimental, opt-in feature to install pure Python launcher scripts on Windows instead of installing a launcher executable for each script, with the intention that these scripts will be launched by pylauncher or Python directly, eventually obviating the need for a launcher executable in setuptools at all. This means that instead of installing, for example: Scripts\my-command.exe Scripts\my-command-script.py Scripts\my-command.exe.manifest Instead Setuptools just installs: Scripts\my-command.py This technique is much like the scripts that get installed to bin/ on Unix, except that due to the nature of launching commands on Windows, the .py extension is essentially required. One problem with this technique is that if the script is also a valid module name, it can be imported, and because Python puts the script's directory at the top of sys.path, it _will_ be imported if that name is imported. This happens, for example, after installing Cython. Cython provides a command, 'cython', and a (extension) module called 'cython'. If one launches cython using the script launcher, the 'cython' module will not be importable (because import cython will import the launcher script). Presumably, this is why '-script' was added to the launcher scripts in previous versions. This is a rather unfortunate situation, and I'd like to solicit comments for a way to avoid this situation. I see a few options: 1. Have the setuptools-generated launcher scripts del sys.path[0] before launching. 2. Accept the implementation and file bugs with the offending projects like Cython to have them either rename their script or rename their internal module. 3. Continue to generate the script names with '-script.py' appended, requiring invocation to always include -script.py on Windows. 4. Continue to generate executables, duplicating the effort of pylauncher, and dealing with the maintenance burden of that functionality. I don't see (2), (3), or (4) as really viable, so my proposal is to move forward with (1) if there aren't any better suggestions. If we move forward with (1), there are a few concerns that come to mind. Here's another problem with #1: you will break single-directory standalone portable app installs, where you use easy_install -mad somedir to install all of an app's dependencies to a single directory that the app can then be run from (assuming Python is available). In order to work around this issue, you'd need to hardcode sys.path entries for the dependencies, or do something else more complicated in order to ensure that dependency resolution will pick up the adjacent distributions before searching anything else on sys.path. Third, is it possible some users are depending on the presence of sys.path[0] Absolutely. It's a documented feature of Python that the script directory is always first on sys.path, so that you can provide modules and packages adjacent to it. That's how portable app installs work with easy_install. May I suggest an option 5 instead? Use the new .pyz (or .pyzw for non-console apps) as a zipped Python application. .pyz files aren't importable, but *are* executable. That's basically all that's needed to prevent importing -- a file extension that's launchable but not importable. (There's also an option 6, which is to use import system hooks to prevent the script modules from being found in the sys.path[0] entry, but that's rather hairier.) Using option 5 means the feature can only work with versions of Python on Windows that install the necessary PATHEXT support to allow that extension to work, but you're kind of limited to that anyway, because by default .py files aren't findable via PATH on Windows. Your post doesn't make it clear whether you're aware of that, btw: IIUC, on most Windows setups, executing a .py file via PATH doesn't work unless you've set up PATHEXT to include .py. So your feature's going to break until that's fixed, and AFAIK there is *no* Windows Python that fixes this, with the possible exception of 3.4 alpha, possibly a future alpha that hasn't been released yet, because last I saw on Python-Dev it was still being discussed *how* to update PATHEXT safely from the installer. In short: dropping .exe wrappers is not supportable on *any* current version of Python for Windows, in the sense that anybody who uses it will not yet be able to execute the scripts if they are currently doing so via PATH (and haven't manually fixed their PATHEXT). (This was one of the main reasons for using .exe wrappers in the first place.) The .pyz approach of course has the same drawback, but at least it should be viable for future Python versions, and doesn't have the sys.path[0] problems. I think you are going to have to keep .exe wrappers the default for all Python versions 3.4.
Re: [Distutils] How to handle launcher script importability?
On Sun, Aug 11, 2013 at 12:17 PM, PJ Eby p...@telecommunity.com wrote: May I suggest an option 5 instead? Use the new .pyz (or .pyzw for non-console apps) as a zipped Python application. .pyz files aren't importable, but *are* executable. That's basically all that's needed to prevent importing -- a file extension that's launchable but not importable. (Details I forgot to mention: the script would be in __main__.py inside the zipped application file, and it would need to change sys.path[0], because sys.path[0] will be the .pyz file itself; it should replace it with the directory containing the .pyz file before doing anything else. That would be the correct way to simulate the existing .exe approach.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Sun, Aug 11, 2013 at 1:58 PM, Jason R. Coombs jar...@jaraco.com wrote: This sounds like a suitable idea, but as you mention in a subsequent message, this format has issues with sys.path assumptions as well. Meh. It's basically, a one-line fix in the __main__.py, i.e.: import sys,os.path; sys.path[0] = os.path.dirname(sys.path[0]) Not exactly rocket science. ;-) In this case, I'm inclined to suggest yet another option (7) - create another extension to specifically represent executable but not importable scripts, perhaps .pys/.pysw (or .pycs/.pygs to more closely match console script and gui script). Probably .pys/.pyws or .pws would be needed, due to issues with some Windows shells using extensions longer than three characters. (This came up in PEP 441 discussions on Python-Dev.) It sounds as if there is a fundamental need for Python to define an extension that distinguishes a script from a module. Yep. I am aware of the PATHEXT factor. I personally add .py and .pyw to the PATHEXT (for all users) on my systems, so I was unsure if Python 3.3 did add those or if pylauncher would add them (if installed separately). I was _hoping_ that was the case, but it sounds like it is not. I did include in the documentation notes about this requirement (https://bitbucket.org/pypa/setuptools/src/1.0b1/docs/easy_install.txt#cl-98 ). I do want to explore the possibility of setuptools facilitating this configuration such that it's easy for a Windows user to enable these settings even if Python does not. It would definitely make sense to have an installer that sets this up, but it would need to be a Windows installer, I think, not a Python program. That is, I don't think setuptools can really do anything about it directly. Personally, I'm not sure I see the point of pushing for early elimination of .exe's - they don't depend on the registry or environment variables or anything else, which makes them great for standalone applications, and they work across all Python versions. Meanwhile, the experimental nature of your change -- and its inability to be the default on versions below 3.4 -- means you're going to be maintaining two sets of code for a very long time. OTOH, implementing a way to deploy an app as a .pyz/.pwz file is a useful feature in its own right, so it might not be doubling up as much. ;-) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] How to handle launcher script importability?
On Sun, Aug 11, 2013 at 7:31 PM, Jason R. Coombs jar...@jaraco.com wrote: -Original Message- From: Nick Coghlan [mailto:ncogh...@gmail.com] Sent: Sunday, 11 August, 2013 17:14 We actually have a proposal on import-sig to allow module specific import path manipulation (including the ability to say don't import this module from this directory, even though it looks like it is here). I'd favour that mechanism over a new not importable file extension. I don't believe this mechanism would suffice. My previous example was over-simplified to the general problem, which is that any script could potentially be imported as a module of the same name. So if I were to launch easy_install.py, it would set sys.path[0] to Scripts\ and if it were then to import cython (which it does), it would import Scripts/cython.py as cython, unless there were some way to globally declare all installed scripts somewhere so they're excluded from import. Indeed. It really *does* need to be a don't import this extension, though it doesn't much matter what that extension is. Except on Windows, of course, where it has to be something associated with Python that also still works as a console app and is listed in PATHEXT. (As you surmised earlier, my choice of '-script.py' was indeed chosen to prevent accidental importing, as the '-' ensures it's not a valid module name.) If that doesn't make it into 3.4, the proposed zipapp extensions would also serve a similar purpose, with some straightforward sys.path manipulation in __main__.py (as PJE pointed out). Regardless what solution might be made available for Python 3.4, I'd prefer to work toward a solution that leverages pylauncher under older Pythons. After all, one of the huge benefits of pylauncher is that it supports multiple Pythons. If zipapp is the preferred mechanism for that, then so be it. For 2.6+, zipapps would work as long as pylauncher supported them and put the requisite extensions in PATHEXT. This approach also means that the script generation is not congruent with that on Unix systems. Using a zipapp means that the whole script generation needs to be special-cased for Windows. One of great benefits of using a simple script was that the code becomes largely unified (only requiring appending of an extension when on Windows). That is, unless zipapps can be made executable on Unix as well. They're already executable on Unix (as of 2.6+), as they contain a #! line. And they don't need a special extension; on both Unix and Windows, Python detects zipapps by inspecting the tail signature, not by the extension. (Of course, you could just continue using the existing wrapper mechanism on Unix, which has the advantage of added transparency, at the cost of having two code paths.) Given these obstacles, do you still feel that zipapp is the best approach? Long-term, yes. I would slightly prefer the this is a script extension, though, as it has the extra transparency benefit. (Still, nobody should be editing an installed script anyway.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Unexpected VersionConflict
On Fri, Aug 9, 2013 at 5:41 PM, Nick Coghlan ncogh...@gmail.com wrote: On 10 August 2013 04:06, PJ Eby p...@telecommunity.com wrote: Probably a better way would be to change the version resolution algorithm to be less greedy, and simply rule out unacceptable versions as the process goes along, then picking the most recent versions left when everything necessary has been eliminated. (Ideally, such an algorithm would still track which distributions had the conflicting requirements, though.) The part I find most surprising is the fact that pkg_resources ignores sys.path order entirely when choosing between multiple acceptable versions. Technically, it doesn't ignore it: if a distribution is listed in sys.path, it takes precedence over any distribution listed later, or that has to be found *in* a directory on sys.path, and will in fact cause a VersionConflict if you ask for a version spec that it doesn't match. However, where the distributions aren't listed in sys.path, but merely *found in a directory on sys.path*, then sys.path has no bearing. It would make things a lot more complicated, and not just in an implementation is hard to explain kind of way. (In principle, you could write an Environment subclass that had a different precedence, but I'm not sure what benefit you would gain from the added complexity. The core version resolution algorithm wouldn't be affected, though, since it delegates the find me something I haven't already got on sys.path operation to an Environment instance's best_match() method.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] help requested getting existing project gv into PyPI
On Fri, Aug 9, 2013 at 8:04 AM, Jess Austin jess.aus...@gmail.com wrote: hi, I recently suggested that the gv package be added to PyPI. This is a close-to-C Graphviz port distributed by the Graphviz graph visualization project. They currently distribute this package in source, and in rpms, debs, and the equivalents of those for Windows, Mac, and Solaris. It seemed only natural that a small entry in PyPI be added so that pip would be able to install this package in a straightforward way (i.e., without using git references etc). The issue tracker url for this suggestion: http://www.graphviz.org/mantisbt/view.php?id=2324 As you can see at that link, the maintainer had some difficulty when trying to create the project through the web form, culminating in a Not found() error message. I unfortunately don't have a great deal of insight into this, but I thought someone on this list might be able to help. Alternatively, please tell me if I'm making an unreasonable suggestion. It was my understanding that pip is able to find installable files from even quite rudimentary PyPI entries, even if they're hosted elsewhere. However, if that's not the case, and the PyPI entry would require extensive ongoing interaction from the gv maintainers, then this probably isn't going to work. It would require ongoing interaction with *somebody*, in order to update the download links whenever a new version is released. Note, too, that unless there is a specific download for the Python binding that uses setup.py to invoke its build process, a PyPI listing won't make the binding pip-installable. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Unexpected VersionConflict
On Fri, Aug 9, 2013 at 9:04 AM, Townsend, Scott E. (GRC-RTM0)[Vantage Partners, LLC] scott.e.towns...@nasa.gov wrote: That does indeed fix this problem, but requiring an egg writer to interrogate all dependent packages (and their dependent packagesŠ) and then hoist the dependencies up won't be robust if those dependent packages change their requirements between the time the egg is written and the time it's loaded. That's why it's generally left up to the application installer/integrator to address these sorts of conflicts, and why it's usually a bad idea for anybody to be requiring exact versions. (I'd suggest asking your dependency to not specify exact point releases, too.) There is one other possibility, though: have you tried reversing the list of your project's dependencies so that the more-specific project's dependencies are processed first? (i.e., so that 1.5.2 will be selected as best before the non-version-specific one is used) That might fix it without requiring you to pin a version yourself. It seems to me that if a requirement has no version specified, then it shouldn't have a way to cause a VersionConflict. One possible way of implementing this would be to have resolve() only check that a distribution exists if no version is specified, do not update 'best'. 'to_activate' would need to be updated with 'generic' distributions only if a requirement with a version specifier hadn't been seen. Thing is, the complete lack of a version requirement is pretty rare, AFAIK, and so is the exact version match that's causing your problem. The combination existing on the same library is therefore that much rarer, so such a change would just be something of a complex kludge that wouldn't improve any other use cases. Probably a better way would be to change the version resolution algorithm to be less greedy, and simply rule out unacceptable versions as the process goes along, then picking the most recent versions left when everything necessary has been eliminated. (Ideally, such an algorithm would still track which distributions had the conflicting requirements, though.) That would be a pretty significant change, but potentially worth someone investigating. There are some big downsides, however: * It's not really a suitable algorithm for installation tools that don't have access to a universal dependency graph, because they can't tell what the next level of dependencies will be * Recursion causes a combinatorial explosion, because what if you select a different version and it has different dependencies (recursively)? Now you need backtracking, and there's a possibility that the algorithm will take a ridiculous amount of time to still conclude that there's nothing you can do about the conflict. These drawbacks are basically why I just wrote a simple greedy match algorithm in the first place, figuring it could always be improved later if it turned out to be needed in practice. There have been occasional comments over the last decade or so by people with ideas for better algorithms, but no actual code yet, as far as I know. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Unexpected VersionConflict
On Thu, Aug 8, 2013 at 3:19 PM, Townsend, Scott E. (GRC-RTM0)[Vantage
Partners, LLC] scott.e.towns...@nasa.gov wrote:
During easy_install of an egg where two versions of pyparsing were available
(1.5.2 and 1.5.6), a VersionConflict was raised:
pkg_resources.VersionConflict: (pyparsing 1.5.6
(/usr/lib/python2.7/dist-packages), Requirement.parse('pyparsing==1.5.2'))
This was unexpected since sys.path (via virtualenv) has version 1.5.2 before
1.5.6. And the system gets 1.5.2 from 'import pyparsing', not 1.5.6.
Have you tried declaring the 1.5.2 dependency from your main project?
IIRC, that should make it take precedence over either of the indirect
dependencies.
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Who administers bugs.python.org, and can we get a notice on the setuptools tracker?
Hi. Not sure who this should go to, but it would be really good if we could get a prominent notice on the old setuptools tracker (at bugs.python.org), specifically on the issue creation screen, to inform people that this tracker is only for setuptools 0.6, and that issues for later versions should go to https://bitbucket.org/pypa/setuptools/issues instead (with a link, of course). Right now, what's happening is that, despite all the other prominent links to the correct tracker, there are still people going to the old tracker and posting issues for the newer versions of setuptools. This means they then have to wait for me to tell them they're in the wrong place, and then resubmit to the correct tracker. Setuptools 0.6 is still supported, so closing the tracker entirely isn't appropriate just yet, but putting up a prominent notice on the issue submission screen saying, If you're reporting an issue for setuptools 0.7 or higher, please use. Thanks in advance! ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Who administers bugs.python.org, and can we get a notice on the setuptools tracker?
On Mon, Aug 5, 2013 at 11:10 AM, Nick Coghlan ncogh...@gmail.com wrote: I filed the request on the metatracker: http://psf.upfronthosting.co.za/roundup/meta/issue522 Thanks! That should keep me from having to keep telling people their princess is in another castle. ;-) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Last PEP 426 update for a while
On Sat, Aug 3, 2013 at 1:07 PM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: Nick Coghlan ncoghlan at gmail.com writes: Just pushed these changes. I'm happy to leave the PEP alone for a while now, Thanks for doing these updates. Can I suggest the following corrections? 1. In the section Exports, there is a dangling sentence which needs to be completed: The interpretation of the individual export keys is defined by the distribution that i 2. In the same section, it says Export group names SHOULD correspond to module names ... and also It is suggested that export groups be named after distributions to help avoid name conflicts. It should be one of these (I presume the former). And not quite the former, either; the same arguments about not splitting a distribution apply to modules as well. i.e., a single module might consume exports from more than one group, so saying they should correspond is too strong; I would say instead that export group names are dotted names that should be *prefixed* with the name of a package or module provided by the relevant distribution. (Of course, it's also perfectly fine for one to use a domain name or other similarly-unique prefix; the real point is just that top-level names should be reserved for groups defined by the stdlib and/or PEPs, and everybody else should be using unique prefixes that give some indication where one would look for a spec.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Last PEP 426 update for a while
On Fri, Aug 2, 2013 at 11:27 AM, Nick Coghlan ncogh...@gmail.com wrote: I pushed a version of PEP 426 with an initial sketch of an entry points replacement design: http://hg.python.org/peps/rev/ea3d93e40e02 To give it a sensible home in the PEP, I ended up defining modules and namespaces fields in addition to commands and exports. The overall section is called Installed interfaces. I initially tried it with the unpacked multi-field mapping for export specifiers, but ended up reverting to something closer to the setuptools notation for readability purposes. For the moment, requires_extra is included since it isn't that hard to explain. Thanks again for all the hard work in putting this together! Btw, under setuptools, entry point *group* names are restricted to valid Python module names, so this is a subset of valid distribution names. Conversely, entry *point* names are intentionally arbitrary and may contain anything that isn't an '=', as long as they don't start with a '#'. The reason for these choices is that entry point groups are used to ensure global uniqueness, but need a standard way for subdividing namespaces. (Notice that setuptools has groups like distutils.setup_arguments and distutils.setup_commands.) Conversely, individual entry point names have a free-form syntax so that they can carry additional structured information, like a converter specifying what it converts from and to, with a quality metric. The idea is to allow tools to build plugin registries from this kind of information without needing to import the modules. Basically, if you can fit it on one line, before the '=', in a reasonably human-readable way, and it saves you from having to import the module in order to figure out whether you wanted to import it in the first place, you can put it in the name. You might wish to make names a bit more restrictive than this, I suppose, but I'm not sure that all of the limitations of distribution names are actually appropriate here. In particular, restricting to alphanumerics, dots, and dashes is *way* too restrictive. Entry point names are sometimes used for human-readable command descriptions, e.g. this is a perfectly valid entry point definition in setuptools: wikiup: Upload documents to one or more wiki pages = some.module:entrypoint [extra1, extra2] Anyway, entry point group names are definitely *not* recommended to follow distribution names, as that makes them rather useless. Things that consume entry points will generally have more than one group, eventually, so at least one of those groups will then have to *not* be named after a distribution, unless you arbitrarily break up the project into multiple distributions so the group names match, which is kind of silly. ;-) Finally, it might be good to point out once again that extras are not so much a set of dependencies that will be checked for at runtime as a set of dependencies that are *needed* at runtime. This need may or may not be checked, and may or may not be satisfied dynamically at runtime; it depends on the API in use, and how/whether it is invoked. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
On Tue, Jul 30, 2013 at 4:58 PM, Donald Stufft don...@stufft.io wrote: Hrm. So I hear what you're saying and part of the problem is likely due to the history of where I tried to get a change through and then felt like all I was getting was stop energy and people wanting to keep the status quo which ultimately ended up preventing changes has lead me to view distutils-sig in more of an adversarial light then is probably appropriate for the distutils-sig of 2013 (versus the distutils-sig of 2011/2012). This is probably reflected in my tone and likely has others, such as yourself, respond similarly, pushing us further down that path. My thought process has become Ok here's something that needs to happen, now how do I get distutils-sig not to prevent it from happening. Thanks for the thoughtful response. I appreciate it. I also want to just throw in one extra piece of information for you and anybody else reading this: 99% of stop energy doesn't happen because people actively want to prevent progress or frustrate other people. It simply happens when people notice a problem but don't have as much personal stake in your goal as they do in not experiencing the problem they will experience (or perceive they will), from the proposed change. When you look at it from this perspective, it's easier to understand that the way to fix this is with more engagement on their part, which can only be gotten by engagement on your part. When I first proposed WSGI, the initial reaction of Web-SIG was pretty negative. Stop energy if you will. Things only moved forward once I was able to channel the energy of objections into offering solutions. It's helpful to remember that asking, okay, so how you would recommend I do it? *doesn't* obligate you to actually follow all of the recommendations you get. (Especially since some of them will be mutually contradictory!) Anyway, I guess what I'm saying is that people lacking enthusiasm for your goals is not really them trying to stop you. In fact, objections are a positive thing: it means you got somebody's attention. The next step is to leverage that attention into actually getting help, or at least more constructive input. ;-) It's true that some individuals will never provide really helpful input. In the WSGI effort, there were people whose advice I never took, because their goals were directed entirely opposite to where I wanted to go. But I remained engaged until it was mutually clear (or at least I thought it was) that our goals were different, and didn't try to persuade them to go in the same direction. Such attempts at persuasion are pretty much a waste of time, and a big energy drain. Consensus-building is something that you do with people who have at least some goals in common, so it's best to focus on finding out what you *do* agree on. This was again reflected in the Python 2.3 discussion because my immediate reaction and impression was that you were attempting to block the move from MD5 due to Python 2.3, which I felt 2.3 wasn't worth blocking enhancements to PyPI. The snark in my statements primarily came from that feeling of someone was trying to shut down an enhancement. Right. In such a case, a question you could ask is, Do you agree in general that we should move to a better hash at some point in the future?, because then the disagreement can be narrowed down to timeframe, migration or deprecation process, etc. The truth is, I had no intention of blocking the move, I had concerns I wanted addressed about the impact, timing and process. (Actually, I originally just noticed a couple of errors in what you'd laid out as the current state of things, and wanted to make sure they were included in the assessment.) The point is, if somebody doesn't have *any* common ground with you, it's unlikely they're even talking to you. At the very least, if they're talking with you about PyPI, they must care about PyPI, even if they care about different things than you, or with different relative priorities. ;-) As far as how to fix it I don't have a particularly magic answer. I will try to be more mindful of my tone and that distutils-sig is likely not my adversary anymore as well as try to ask questions instead of arguing the relevance immediately. Again, thank you. And hopefully, remember that probably nobody was intentionally being your adversary before, either. As the old adage says, the best way to destroy your enemies is to make friends with them. ;-) And we do that by focusing on common ground, and inviting participation. (This is again not to say that I've been 100% Mr. Wonderful myself; I know I haven't. But the community's best consensus-building happens when somebody is doing the tough work of engaging with all parties. Sometimes this doesn't happen, alas; back when I was developing setuptools there just weren't enough people interested in the problems available on Distutils-SIG to build any sort of consensus on
Re: [Distutils] a plea for backward-compatibility / smooth transitions
On Tue, Jul 30, 2013 at 4:14 AM, Donald Stufft don...@stufft.io wrote: Heh, I'm pretty good at getting yelled at :) Nick is also pretty good at making people feel like he both knows and *cares* about their breakage, and isn't just dismissing their concerns as trivial or unimportant. Breakage isn't trivial or unimportant to the person who's yelling, so this is an important community-maintenance skill. It builds trust, and reduces the total amount of yelling. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] a plea for backward-compatibility / smooth transitions
On Tue, Jul 30, 2013 at 2:04 PM, Donald Stufft don...@stufft.io wrote: On Jul 30, 2013, at 1:13 PM, PJ Eby p...@telecommunity.com wrote: On Tue, Jul 30, 2013 at 4:14 AM, Donald Stufft don...@stufft.io wrote: Heh, I'm pretty good at getting yelled at :) Nick is also pretty good at making people feel like he both knows and *cares* about their breakage, and isn't just dismissing their concerns as trivial or unimportant. Breakage isn't trivial or unimportant to the person who's yelling, so this is an important community-maintenance skill. It builds trust, and reduces the total amount of yelling. *shrug*, If I didn't care I would have made this change as soon as Nick said it was ok. Instead I declared I was going to and waited to make sure nobody else had any concerns. And once Holger said he did I said ok I won't do it. Maybe my mannerisms give the impression I don't but that's actually pretty far from the truth. I did say feel like. ;-) Nick usually gives more of an impression that he's thought about concerns raised before rejecting them; your responses often sound like, Who cares about that? Asking for suggestions, for example, would be good. Nick also rarely seems irritated by people's concerns or problems, whereas you sometimes seem in a big hurry to fix something today or this week that's been broken for years, without giving folks a while to get used to the idea. Often your proposals seem less like proposals than, I've decided to do this, so deal with it. I'm not saying all this because I want to complain or yell at you; I'm saying it because I think you do care enough to know how you're coming across, at least to me. Our discussions have gotten heated in the past because my impression of your reaction to the concerns I raise often seems like, I don't care about supporting [group of people affected], so neither should you. Perhaps the issue is just one of confusion. When I raise an issue about, say, Python 2.3 users (who are still downloading setuptools 0.6 releases, and presumably also using them), it's not because I expect *you* to change your plans to support them, but because I need to know how *I* can, if the issue arises. So I don't actually expect you to care about Python 2.3 users (again, as an example), but I do expect you to care about *me* supporting them. In the most recent situation, you did in fact point me to your awesome hashlib port, so I do know you *do* care to at least that extent. But the rhetoric that you sent both before and after the helpful bit seemed on the inflammatory side, as though I were crazy to be thinking of Python 2.3. Whether or not this is true ;-) -- it's not especially *helpful* to the discussion. If I may offer a suggestion, asking questions in response to objections is generally more useful than immediately arguing the relevance of the objection. First, it tells the objector that you're interested in what they have to say, and second, it may well help the objector understand that there isn't actually any real problem, and gives them an easier path to backing down and/or compromising, whereas a frontal assault tends to focus people on responding to you instead of reconsidering their objection. On the hashlib issue, for example, it actually occurred to me later that it's completely a non-issue because the actual hash scenario I was most concerned about *can't actually happen*. (MD5 hashes in code or dependency_links, used e.g. by setuptools itself to secure its own downloads. Changing PyPI won't affect these, duh.) It might've occurred to me sooner, though, if you'd actually asked what scenario I was worried about, instead of arguing about the relevance. This isn't to say that you're responsible for what I do or don't figure out; my point is simply that asking questions and inviting suggestions in response to people's objections will generally get you more thoughtful responses and more trust, and resolve issues sooner, with less arguing. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Migrating Hashes from MD5 to SHA256
On Fri, Jul 26, 2013 at 12:25 PM, Donald Stufft don...@stufft.io wrote: Additionally there is no security list from setuptools versions earlier than 0.7. Not true, actually. Setuptools 0.6 dev releases supported SSL verification since mid-May, but don't support any hashes besides MD5. Anybody who updated their setuptools between then and the release of 0.7 would have that version. Unfortunately, it's hard to tell how many people that is, though I could try and dig through my server logs to find out. There's also another issue with jumping to SHA256: Python prior to 2.5 didn't support it. Which brings up another point: the setuptools 0.6 series is the only setuptools available for Python 2.3. That's one of the reasons it's still available for download. If you want SSL verification on 2.3, it's the only thing available. (Meanwhile, a lot of people are still downloading 0.6c11; probably I should package up an 0.6c12 so those folks pick it up instead of 0.6c11.) Anyway, this is all somewhat moot since the hashes only matter when the download is hosted somewhere besides PyPI, since SSL verification is available for the PyPI part. Even so, I'd suggest that moving to SHA1 might be a good intermediate step: it's available on Python 2.3, so I could backport the relevant support to the 0.6 branch. (IIUC, Python 2.3 is still the default version for many Linux distros that have not reached end-of-life support.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Migrating Hashes from MD5 to SHA256
On Fri, Jul 26, 2013 at 3:14 PM, Donald Stufft don...@stufft.io wrote: Does the hashlib backport I added to setuptools 0.9 for Python 2.4 work on 2.3? It's a pure python implementation of hashlib. Ah, didn't know about that! I can't imagine what problems there would be; not much changed in 2.4 that can't be emulated in 2.3. Anyway, I'll have a look. Thanks! I don't have a Python 2.3 available to attempt to test. To be honest I've never even used Python 2.3. Heh. Noob. ;-) (j/k) 2.3 is basically 2.4 minus decorators, generator expressions, various builtins and stdlib features. Unless you used set types, reversed(), or various other odds and ends, I should be able to backport it. [stuff about RHEL support] If there's a 2.4 hashlib backport, that addresses my concerns just fine. If I need to, I'll backport it to setuptools 0.6. Thanks. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On Sat, Jul 20, 2013 at 10:54 PM, Nick Coghlan ncogh...@gmail.com wrote: Extras should just be a way to ask are these optional dependencies present on this system?, without needing to worry about how they got there. Technically, they are a way to ask can you get this for me?, since pkg_resources' API allows you to specify an installer callback when you ask to load an entry point. This means that an installer tool can dynamically obtain any extras it needs, not just check for their installation. To put it another way, it's not exported only if extra is available, it's exported, but make sure you have this first. A subtle difference, but important to the original use cases (see below). For now, I'll switch export specifiers back to the concise modulename:qualname entry point format and add Do we need to support the exported-only-if-extra-is-available feature? as an open question. My current thinking is that the point you made about script wrappers (putting the wrapper in separate distribution and depending on that from an extra) applies to other plugins as well. Now that I'm thinking about it some more, one of the motivating use cases for extras in entry points was startup performance in plugin-heavy GUI applications like Chandler. The use of extras allows for late-loading of additions to sys.path. IOW, it's intended more for a situation where not only are the entry points imported late, but you also want as few plugins as possible on sys.path to start with, in order to have fast startup. The other use case is similar, in that a plugin-heavy environment with self-upgrading abilities can defer *installation* of parts of a plug-in until it is actually used. (Which is why EntryPoint objects have a .require() method separate from .load() - you can loop over a relevant set of entry points to pre-test or pre-ensure that they're all available and dependencies are installed before importing any of them, even though .load() will also do that for a single entry point.) For the specific case of the meta build system itself, these use cases may be moot. For the overall use of exports, however, the use cases are still valuable for plugin-heavy apps. (Specifically, applications that use lots of plugins written by different people, and don't want to have to import everything at startup.) Indeed, this is the original use case for exports in the first place: it's a plugin system that doesn't require importing any plugins until you actually need a particular plugin's functionality. Extras just expand that slightly to don't require installing things or putting them on sys.path until you need their functionality. Heck, if pip itself were split into two distributions, one of which were a command line script declared with an extra, pointing into the second distribution, it'd have dynamic bootstrapping. (Were it not for the part where it would need pip available to *do* the bootstrapping, of course. ;-) ) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On Sun, Jul 21, 2013 at 6:44 PM, Nick Coghlan ncogh...@gmail.com wrote: On 22 Jul 2013 01:46, PJ Eby p...@telecommunity.com wrote: Now that I'm thinking about it some more, one of the motivating use cases for extras in entry points was startup performance in plugin-heavy GUI applications like Chandler. The use of extras allows for late-loading of additions to sys.path. IOW, it's intended more for a situation where not only are the entry points imported late, but you also want as few plugins as possible on sys.path to start with, in order to have fast startup. I'm working with Eric Snow on a scheme that I hope will allow module-specific path entries that aren't processed at interpreter startup and never get added to sys.path at all (even if you import the module). Assuming we can get it to work the way I hope (which is still a maybe at this point in time), it should then be possible to backport it to earlier versions as a metaimporter. I haven't had a chance to look at that proposal at more than surface depth, but my immediate concern with it is that it seems to be at the wrong level of abstraction for the packaging system, i.e., just because you can import a module, doesn't mean you can get at its project metadata (e.g., how would you find its exports, or even know what distribution it belonged to?). (Also, I don't actually see how it would be useful or relevant to the use case we're talking about; it seems maybe orthogonal at best.) OK, so as Daniel suggested, it's more like an export/entry-point specific requires field, but limited to the extras of the current distribution. Correct: at the time, it seemed a lot simpler to me than supporting arbitrary requirements, and allows for more DRY, since entry points might share some requirements. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On Sat, Jul 20, 2013 at 8:08 PM, Nick Coghlan ncogh...@gmail.com wrote: I see it as more useful for making an executable optional by defining a cli extra. If your project just gets installed as a dependency, no wrapper would get generated. Only if you went pip install myproject[cli] (or another project specifically depended on the cli extra) would it be installed. Why stop there... how about environment markers for exports, too? ;-) And throw in an environment marker syntax for whether something was installed as a dependency or explicitly... ;-) (Btw, the above is a change from setuptools semantics, but I don't really see it as a problem; ISTM unlikely that anybody has used extras on a script wrapper. Extras on *other* entry points, however, *do* exist, at least IIRC. I'm pretty sure there was at least one concrete use case for them involving Chandler plugins when I originally implemented the feature. The possibility of having extras on a script is just a side effect, though, not an actually-intended feature; if you have the need, it actually makes more sense to just bundle the script in another package and require that pacakge from the extra, rather than putting it in the original package.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On Fri, Jul 19, 2013 at 9:10 AM, Nick Coghlan ncogh...@gmail.com wrote: Right, I think the reasonable near term solutions are for pip to either: 1. generate zc.buildout style wrappers with absolute paths to avoid the implied runtime dependency 2. interpret use of script entry points as an implied dependency on setuptools and install it even if not otherwise requested Either way, pip would need to do something about its *own* command line script, which heavily favours option 1 Option 1 also would address some or all of the startup performance complaint. It occurs to me that it might actually be a good idea *not* to put the script wrappers in the standard entry points file, even if that's what setuptools does right now: if lots of packages use that approach, it'll slow down the effective indexing for code that's scanning multiple packages for something like a sqlalchemy adapter. (Alternately, we could use something like 'exports-some.group.name.json' so that each export group is a separate file; this would keep scripts separate from everything else, and optimize plugin searches falling in a particular group. In fact, the files needn't have any contents; it'd be okay to just parse the main .json for any distribution that has exports in the group you're looking for. i.e., the real purpose of the separation of entry points was always just to avoid loading metadata for distributions that don't have the kind of exports you're looking for. In the old world, few distributions exported anything, so just identifying whether a distribution had exports was sufficient. In the new world, more and more distributions over time will have some kind of export, so knowing *which* exports they have will become more important.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] entry points PEP
On Fri, Jul 19, 2013 at 2:09 PM, Joe Gordon joe.gord...@gmail.com wrote: When I try importing pkg_resources in our development environment it is very slow: Use zc.buildout to install the application you're invoking, and then it won't need to import pkg_resources. (Unless the actual app uses it.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] entry points PEP
On Thu, Jul 18, 2013 at 1:50 PM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: Daniel Holth dholth at gmail.com writes: For one thing you can have more than one mysql = in the same sqlalchemy.dialects. I think in this instance the string parsing is Don't you say in the PEP about the key that It must be locally unique within this distribution’s group.? Setuptools requires this per-distribution uniqueness, but note that uniqueness is not required across distributions. So more than one distribution can export a 'mysql' in the 'sqlalchemy.dialects' group. It's up to the application to decide how to handle multiple definitions; typically one either uses all of them or the first one found on sys.path, or some other tie-breaking mechanism. The pkg_resources entry point APIs just provide operations for iterating over entry points defined on either a single distribution, or across all distributions on a specified set of directories. (Via the WorkingSet API.) Note that I don't see necessarily a connection between extras and flags, though you've mentioned that they're extras. Does setuptools store, against an installed distribution, the extras it was installed with? AFAIK it doesn't. (Even if it did, it would need to keep that updated if one of the extras' dependencies were later uninstalled.) And if not, how would having extras in the specification help, since you can't test the must be installed part? The pkg_resources implementation does a require() for the extras at the time the entry point is loaded, i.e., just before importing. This allows it to dynamically add requirements to sys.path, or alternatively raise an error to indicate the extras aren't available. In addition, various entry point API functions take an 'installer' keyword argument, specifying a callback to handle installation of missing extras. Setuptools uses this feature internally, so that if you use a setup.py command whose entry point needs additional dependencies, those will be fetched on-the-fly. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On Thu, Jul 18, 2013 at 4:36 PM, Paul Moore p.f.mo...@gmail.com wrote: As regards console scripts, I think they should be rewritten to remove the dependency on pkg_resources. That should be a setuptools fix, As others have already mentioned, this is not a bug but a feature. Setuptools-generated scripts are linked to a specific version of the project, which means that you can install more than one version by renaming the scripts or installing the scripts to different directories. While other strategies are definitely possible, distlib's approach is not backward-compatible, as it means installing new versions of a project will change *existing scripts'* semantics, even if you installed the previous version's scripts to different locations and intended them to remain accessible. If you want an example of doing it right, see buildout, which hardcodes the entire sys.path of a script to refer to the exact versions of all dependencies; while this has different failure modes (i.e., dependence on absolute paths), it is more stable as to script semantics even than setuptools' default behavior. maybe triggered by a flag (possibly implied by --single-version-externally-managed, as the pkg_resources complexity is only needed when multi-versions are involved). That option does not preclude the existence of multiple versions, or the possibility of installing the same script to different directories for different installed versions. If you *must* do this, I suggest using buildout's approach of hardwiring sys.path in the script, only strengthened by checking for the actual existence and versions, rather than distlib's anything-goes approach. (Of course, as Donald points out, this won't do anything for those scripts that themselves make use of other packages' entry points: they will have a runtime dependency on pkg_resources anyway.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Q about best practices now (or near future)
On Thu, Jul 18, 2013 at 7:09 PM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: PJ Eby pje at telecommunity.com writes: While other strategies are definitely possible, distlib's approach is not backward-compatible, as it means installing new versions of a Correct, because distlib does not support multiple installed versions of the same distribution, nor does it do the sys.path manipulations on the fly which have caused many people to have a problem with setuptools. Do people see this as a problem? I would have thought that venvs would allow people to deal with multiple versions in a less magical way. So does buildout, which doesn't need venvs; it just (if you configure it that way) puts all your eggs in a giant cache directory and writes scripts with hardcoded sys.path to include the right ones. This is actually more explicit than venvs, since it doesn't depend on environment variables or on installation state. IOW, there are other choices available besides implicit environment-based path and dynamically generated path. Even setuptools doesn't require that you have a dynamic path. If that is a real requirement which should be supported, shouldn't there be a PEP for it, if it's coming into Python? It's not supported by distutils, and it has been a point of contention. Distutils lets you install things wherever you want; in the naive case you could use install --root to install every package to a version-specific directory and then use something like Gnu Stow to create symlink farms. Python supports explicit sys.path construction and modification, and of course people certainly vendor (i.e. bundle) their dependencies directly in order to have a specific version of them. So, I don't think it's accurate to consider multi-version installation a totally new feature. (And AFAIK, the point of contention isn't that setuptools *supports* multi-version installation, it's that it's the *default* implementation.) In any event, wheels are designed to be able to be used in the same way as eggs for multi-version installs. The question of *how* has been brought up by Nick before, and I've thrown out some counter-proposals. It's still an open issue as to how much *active* support will be provided, but my impression of the discussion is that even if the stdlib isn't exactly *encouraging* multi-version installs, we don't want to *break* them. Hence my suggestion that if you want to drop pkg_resources use from generated scripts, you should use buildout's approach (explicit sys.path baked into the script) rather than distlib's current laissez-faire approach. Or you can just check versions, I'm not that picky. All I want is that if you install a new version of a package and still have an old copy of the script, the old script should still run the old version, or at least give you an error telling you the script wasn't updated, rather than silently running a different version. Buildout's approach accomplishes this by hardcoding egg paths, so as long as you don't delete the eggs, everything is fine, and if you do delete any of them, you can see what's wrong by looking at the script source. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Replacing pip.exe with a Python script
On Tue, Jul 16, 2013 at 8:23 AM, Paul Moore p.f.mo...@gmail.com wrote: On 16 July 2013 12:42, Oscar Benjamin oscar.j.benja...@gmail.com wrote: I thought that 64 bit Windows could run 32 bit Windows .exe files (although I don't have a way to check this). Yes, but there are 32-bit and 64-bit exe wrappers, which I suspect is because a 32-bit exe can't load a 64-bit DLL (and may be vice versa). As I said, I don't know for sure at the moment, but it needs investigating. That's not why they exist; the .exe's don't load the Python DLL, they just invoke python.exe. The existence of separate 32- and 64-bit .exe's is a Distribute innovation, actually; setuptools 0.6 doesn't use them. Instead, setuptools writes a manifest file to tell Windows that it doesn't need privilege escalation or to create a separate console. This meant that only one (32-bit) .exe was needed. I forget what happened with the Distribute approach or why the 64-bit version was kept after the merge; ISTM there was some other use for it, or at least Jason thought so. But DLL loading is not the reason. (Actually, after searching my email, my guess is that there actually *isn't* any need for the 64-bit .exe; ISTM that it was introduced solely as a false fix for the privilege escalation problem, that only fixes it for 64-bit Windows and doesn't help 32-bit.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 439 and pip bootstrap updated
On Thu, Jul 11, 2013 at 10:20 AM, Brett Cannon br...@python.org wrote: And if people want to promote the -m option then the executable scripts just become a secondary convenience. Plus you can't exactly require setuptools to create those scripts at install-time with Python if that's when they are going to be installed. You don't need setuptools in order to include .exe wrappers, though: there's nothing setuptools-specific about the .exe files, they just run a matching, adjacent 'foo-script.py', which can contain whatever you want. Just take the appropriate wrapper .exe, and rename it to whatever 'foo' you want. IOW, if you want to ship a pip.exe on windows that just does from pip import __main__; __main__() (or whatever), you can do precisely that, no setuptools needed. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Metadataformat PEP 426 on PyPI?
On Wed, Jul 3, 2013 at 10:51 AM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote:
If you deserialize the JSON at an URL like the above into a dict, the PEP
426 metadata is available in the subdict at key index-metadata in the
top-level dict. Example from setuptools 0.7.5:
index-metadata: {
name: setuptools
},
I expect this metadata to track the PEP as changes to it are published.
Currently, the top-level dict contains some legacy representations of the
metadata which will be removed in due course.
Just an FYI, not sure if this is an issue with your converter or with
the new spec, but the metadata shown for setuptools is missing
something important: 0.7.x pins specific distributions of its
dependencies using dependency_links URLs with #md5 hashes, so that SSL
support can be installed in a reasonably secure manner, as long as
you're starting from a trusted copy of the distribution. The
converted metadata you show lacks this pinning.
Granted, the pinning is somewhat kludged, and the specific need is
perhaps a rare use case outside of installer tools themselves. But I
thought it worth pointing out as a limitation of either the converter
or with the spec itself in relation to version support.
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Metadataformat PEP 426 on PyPI?
On Wed, Jul 3, 2013 at 2:34 PM, Donald Stufft don...@stufft.io wrote: PEP426 does not support dependency_links. Right - I would've expected direct references in this scenario, assuming the PEP still has them. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Finding modules in an egg / distribution
On Tue, Jul 2, 2013 at 6:59 AM, Iwan Vosloo i...@reahl.org wrote: Hi there, We have been struggling to find a nice way to list all the modules (or packages) that are part of a particular Distribution (or egg). Nice should also mean that it works when the egg is installed. We have a need to do some introspection on the code shipped as an egg. Any ideas? If you are targeting at least Python 2.5, see: http://docs.python.org/2/library/pkgutil.html#pkgutil.walk_packages ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Finding modules in an egg / distribution
On Tue, Jul 2, 2013 at 1:30 PM, Iwan Vosloo i...@reahl.org wrote: On 02/07/2013 17:08, PJ Eby wrote: If you are targeting at least Python 2.5, see: http://docs.python.org/2/library/pkgutil.html#pkgutil.walk_packages We're targeting Python 2.7. Trouble is that pkgutil.walk_packages needs a path to search from. Distribution.location is always your site-packages directory once a Distribution is installed, so walking that just gives ALL installed packages. If your Distribution contains some sort of main package that contains everything in it, you can use that package's .__path__, but then you'd need to discover what that package is. Distributions could also contain more than one package next to each other, and top-level modules. (The __path__ of a top-level module is also simply the site-packages directory.) There is a metadata file top_level.txt which one could use to get the names of top-level packages/modules in the Distribution. This can however contain a namespace package too - and you don't want all the packages inside the namespace package - just the bits inside the chosen Distribution... Ah, well in that case you'll have to inspect either .egg-info/SOURCES.txt or the PEP 376 installation manifest. I don't know of any reliable way to do what you want for system-installed packages at the moment. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 440 and direct references
On Wed, Jun 26, 2013 at 1:40 PM, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: (a) Repeatability (not possible with a generic latest version URL). ISTM that forcing repeatability in the spec implies the inability to do development with requirements that are also in development, unless there is also an out-of-band channel for communicating the URL. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Setuptools 0.7.2 and Distribute 0.7 (compatibility wrapper) on PyPI
On Wed, Jun 19, 2013 at 4:50 PM, Jason R. Coombs jar...@jaraco.com wrote: What’s the next step to support this bootstrap script? Can you just upload it with the docs? ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] option #1 plus download_url scraping
On Wed, Jun 5, 2013 at 2:47 PM, Donald Stufft don...@stufft.io wrote: One of the big problems with download_url is that the data in setup.py is used in (and influences the content of) the final dist file. This means that inside of a setup.py you won't know what the hash of the final file is. So it's difficult for a setup.py based workflow with external urls to provide md5 sums for the files which means that pip and friends can't verify that no body modified the download in transit. Not if it's done in a setup.py command that runs after the distributions are built, akin to the way the upload command works now. If there were, say, an uplink command based on a modified version of upload, it could call the PyPI API to pass along hashed URLs. At some point I intend to write such a command so that my current snapshot scripts (which run on the server the downloads are hosted from) can update PyPI with properly hashed URLs. (But I'm not sure when some point will be, exactly, so if someone else writes it first I'll be a happy camper.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] A process for removal of PyPi entries
On Sat, Jun 1, 2013 at 4:29 PM, Lennart Regebro rege...@gmail.com wrote: On Sat, Jun 1, 2013 at 9:20 PM, Paul Moore p.f.mo...@gmail.com wrote: I'm -1 on anything that doesn't involve at least a minimal level of human involvement (possibly excepting an initial clean up exercise for projects with no author email) This is why I basically said I'm OK with automatic deletion after a time if there are no downloadable packages and no contact information. Otherwise the owner should be contacted. Some people are saying files uploaded vs. downloadable packages. I don't like the files uploaded criterion because IMO it's a perfectly valid use case to list a package on PyPI which is only available via external revision control. Heck, a project that only has planning documents and a reasonably active mailing list should still qualify for PyPI listing, else the original distutils-sig would not have qualified for reserving the name distutils on PyPI, before its first release. ;-) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Scripts\foo.exe on a python which doesn't have Distribute installed?
On Sat, May 25, 2013 at 4:06 AM, Matt Wilkie map...@gmail.com wrote: ... or use a script that doesn't depend on entry points not desirable, as we like the .exe entry points creates and don't want to use a batch file (the are you sure you want to quit? message on ctrl-c is annoying) , or copy your script to foo-script.py alongside the .exe launcher, and manually include those two files as scripts (not using entry points) in your setup() definition. Would you please expand on this? The .exe launcher seems to depend on entry points, or I'm doing something wrong (entirely likely). The foo.exe launcher doesn't use the entry point, all it does is run an adjacent foo-script.py. You can put anything you want in foo-script.py adjacent to that .exe, and it'll run. As for different versions of pkg_resources on sys.path, Python always puts the directory containing the script first. So if there's a pkg_resources.py adjacent to the script, that will take precedence over any other copy. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Announcing Setuptools 0.7b3
On Sat, May 25, 2013 at 4:40 PM, Jason R. Coombs jar...@jaraco.com wrote: https://bitbucket.org/pypa/setuptools/downloads You may be already aware of this, but the installation instructions don't match the available downloads. I was previously thinking we should just drop binary distributions of setuptools altogether anyway, and make ez_setup.py the standard way to install setuptools on all platforms. (Which would help address things like installation conflicts w/existing Distribute, etc.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Announcing Setuptools 0.7b3
On Sat, May 25, 2013 at 7:02 PM, Jason R. Coombs jar...@jaraco.com wrote: I heard in person by a newer Windows user recently, that they still hear that Setuptools is still considered friendlier than Distribute. Without knowing what exactly they considered friendlier, I'm not sure what to suggest. Even if you use a GUI to install setuptools, it's not going to do you much good without getting to a command line! ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Using Scripts\foo.exe on a python which doesn't have Distribute installed?
On Wed, May 22, 2013 at 8:12 PM, Matt Wilkie map...@gmail.com wrote: How do I get my installer to include Distribute so I don't have to tell our users before you install our program you have to go install this other program? Setuptools (which Distribute is based on) is designed for shipping libraries, not applications; it's developer installer, not primarily an end-user installer for applications. So you probably should be using py2exe instead. Alternatively, you can bundle a copy of pkg_resources.py, or use a script that doesn't depend on entry points, or copy your script to foo-script.py alongside the .exe launcher, and manually include those two files as scripts (not using entry points) in your setup() definition. Any of these approaches will solve the problem; it's mostly a matter of your preferences or other requirements. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 438 - Transition Phase 1
On Sun, May 19, 2013 at 7:26 PM, Richard Jones rich...@python.org wrote:
Donald wrote a handy script to help make this easier:
https://pypi.python.org/pypi/pypi-show-urls
Doesn't seem to work for me:
$ pypi-show-urls -u pje
Traceback (most recent call last):
File /usr/bin/pypi-show-urls, line 8, in module
load_entry_point('pypi-show-urls==2.1.1', 'console_scripts',
'pypi-show-urls')()
File build/bdist.linux-i686/egg/pkg_resources.py, line 318, in
load_entry_point
File build/bdist.linux-i686/egg/pkg_resources.py, line 2221, in
load_entry_point
File build/bdist.linux-i686/egg/pkg_resources.py, line 1954, in load
File build/bdist.cygwin-1.7.15-i686/egg/pypi_show_urls/__main__.py,
line 24, in module
ImportError: No module named pip.req
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] PEP 438 - Transition Phase 1
On Mon, May 20, 2013 at 1:03 PM, Lennart Regebro rege...@gmail.com wrote:
On Mon, May 20, 2013 at 6:05 PM, PJ Eby p...@telecommunity.com wrote:
On Sun, May 19, 2013 at 7:26 PM, Richard Jones rich...@python.org wrote:
Donald wrote a handy script to help make this easier:
https://pypi.python.org/pypi/pypi-show-urls
Doesn't seem to work for me:
$ pypi-show-urls -u pje
Traceback (most recent call last):
File /usr/bin/pypi-show-urls, line 8, in module
load_entry_point('pypi-show-urls==2.1.1', 'console_scripts',
'pypi-show-urls')()
File build/bdist.linux-i686/egg/pkg_resources.py, line 318, in
load_entry_point
File build/bdist.linux-i686/egg/pkg_resources.py, line 2221, in
load_entry_point
File build/bdist.linux-i686/egg/pkg_resources.py, line 1954, in load
File build/bdist.cygwin-1.7.15-i686/egg/pypi_show_urls/__main__.py,
line 24, in module
ImportError: No module named pip.req
Do you have pip installed?
No, but installing it didn't help; I got an ElementTree exception next:
$ pypi-show-urls -u pje
Download candidates for PEAK
Traceback (most recent call last):
File /usr/bin/pypi-show-urls, line 8, in module
load_entry_point('pypi-show-urls==2.1.1', 'console_scripts',
'pypi-show-urls')()
File build/bdist.cygwin-1.7.15-i686/egg/pypi_show_urls/__main__.py,
line 148, in main
File /usr/lib/python2.6/xml/etree/ElementPath.py, line 198, in findall
return _compile(path).findall(element)
File /usr/lib/python2.6/xml/etree/ElementPath.py, line 176, in _compile
p = Path(path)
File /usr/lib/python2.6/xml/etree/ElementPath.py, line 93, in __init__
expected path separator (%s) % (op or tag)
SyntaxError: expected path separator ([)
(And of course the package should specify that it has an install-time
requirement for pip.)
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
