Re: [Python-Dev] cpython: Add Py_RETURN_NOTIMPLEMENTED macro. Fixes #12724.
Le Thu, 11 Aug 2011 03:34:37 +0200, brian.curtin a écrit : > http://hg.python.org/cpython/rev/77a65b078852 > changeset: 71809:77a65b078852 > parent: 71803:1b4fae183da3 > user:Brian Curtin > date:Wed Aug 10 20:05:21 2011 -0500 > summary: > Add Py_RETURN_NOTIMPLEMENTED macro. Fixes #12724. It would sound more useful to have a generic Py_RETURN() macro rather than some specific forms for each and every common object. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving forward with the concurrent package
Le Thu, 11 Aug 2011 09:03:35 +1000, Nick Coghlan a écrit : > On Thu, Aug 11, 2011 at 4:55 AM, Brian Curtin > wrote: > > Now that we have concurrent.futures, is there any plan for > > multiprocessing to follow suit? PEP 3148 mentions a hope to add or move > > things in the future [0], which would be now. > > As Jesse said, moving multiprocessing or threading wholesale was never > part of the plan. The main motivator of that comment in PEP 3148 was > the idea of creating 'concurrent.pool', which would provide a > concurrent worker pool API modelled on multiprocessing.Pool that > supported either threads or processes as the back end, just like the > executor model in concurrent.futures. Executors *are* pools, so I don't know what you're talking about. Besides, multiprocessing.Pool is quite bloated and therefore difficult to improve. It should be slowly phased out in favour of concurrent.futures. In general, it would be nice if people wanting to improve the concurrent primitives made actual, concrete propositions. We've had lots of hand-waving in that area for years, to no effect. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving forward with the concurrent package
On Thu, Aug 11, 2011 at 5:07 PM, Antoine Pitrou wrote: > Le Thu, 11 Aug 2011 09:03:35 +1000, > Nick Coghlan a écrit : >> On Thu, Aug 11, 2011 at 4:55 AM, Brian Curtin >> wrote: >> > Now that we have concurrent.futures, is there any plan for >> > multiprocessing to follow suit? PEP 3148 mentions a hope to add or move >> > things in the future [0], which would be now. >> >> As Jesse said, moving multiprocessing or threading wholesale was never >> part of the plan. The main motivator of that comment in PEP 3148 was >> the idea of creating 'concurrent.pool', which would provide a >> concurrent worker pool API modelled on multiprocessing.Pool that >> supported either threads or processes as the back end, just like the >> executor model in concurrent.futures. > > Executors *are* pools, so I don't know what you're talking about. Yes, that's the point. A developer shouldn't be forced into using a particular invocation model (i.e. futures) just to get thread or process pool functionality - the pool should be a lower layer building block that's provided separately. As you say, though, nobody has stepped up for the task of actually defining that common, lower level interface. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving forward with the concurrent package
On Thu, Aug 11, 2011 at 10:56 AM, Nick Coghlan wrote: > On Thu, Aug 11, 2011 at 5:07 PM, Antoine Pitrou > wrote: > > Le Thu, 11 Aug 2011 09:03:35 +1000, > > Nick Coghlan a écrit : > >> On Thu, Aug 11, 2011 at 4:55 AM, Brian Curtin > >> wrote: > >> > Now that we have concurrent.futures, is there any plan for > >> > multiprocessing to follow suit? PEP 3148 mentions a hope to add or > move > >> > things in the future [0], which would be now. > >> > >> As Jesse said, moving multiprocessing or threading wholesale was never > >> part of the plan. The main motivator of that comment in PEP 3148 was > >> the idea of creating 'concurrent.pool', which would provide a > >> concurrent worker pool API modelled on multiprocessing.Pool that > >> supported either threads or processes as the back end, just like the > >> executor model in concurrent.futures. > > > > Executors *are* pools, so I don't know what you're talking about. > Also the Pool from multiprocessing "works" for threads and process: from multiprocessing.pool import Pool as ProcessPool from multiprocessing.dummy import Pool as ThreadPool ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython (2.7): Fix closes Issue12722 - link heapq source in the text format in the
Hi, > IIRC the reason why we don't do it on 2.x is because we don't have the > 'source' directive available in Sphinx and therefore we would have to > update all the links manually to link to h.p.o instead of s.p.o. In 3.2 and higher, there is a custom source role in Doc/tools/sphinxext/pyspecific.py. For 2.7, I volunteered to change all links manually (sed being, as usual, my friend) but just lacked time. Cheers ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEPs] Rebooting PEP 394 (aka Support the /usr/bin/python2 symlink upstream)
Hi, I’ve read the latest version of this PEP, as updated by Nick Coghlan in the Mercurial repo on July, 20th. Excuse me if I repeat old arguments, I did not reread all the threads. In summary, I don’t think the PEP is useful right now, nor that it will set a good practice for the future. > * Unix-like software distributions (including systems like Mac OS X and Minor: I call these “operating systems”. > * The Python 2.x ``idle``, ``pydoc``, and ``python-config`` commands should > likewise be available as ``idle2``, ``pydoc2``, and ``python2-config``, > with the original commands invoking these versions by default, but possibly > invoking the Python 3.x versions instead if configured to do so by the > system administrator. This item ignores that on some OSes, defining the default Python version is not a decision made by the sysadmin. The example I know is Debian (and derivatives): despite what one can read on the Web, it is not a good idea to change /usr/bin/python to point to the version you want; the decision affects all scripts used by the system itself, and is thus the call of the Debian Python maintainers. (FTR, Debian developers discussed adding /usr/bin/python2 at the latest DebConf and rejected it; I don’t know if the arguments raised are the same as mine, but maybe Piotr or someone else will chime in in this thread.) > This is needed as, even though the majority of distributions still alias the > ``python`` command to Python 2, some now alias it to Python 3. Some of > the former also do not provide a ``python2`` command; hence, there is > currently no way for Python 2 code (or any code that invokes the Python 2 > interpreter directly rather than via ``sys.executable``) to reliably run on > all Unix-like systems without modification, as the ``python`` command will > invoke the wrong interpreter version on some systems, and the ``python2`` > command will fail completely on others. The recommendations in this PEP > provide a very simple mechanism to restore cross-platform support, with > minimal additional work required on the part of distribution maintainers. I would like more data about this. How many OSes have moved their python executable to python2? How much people does that impact? Right now I think that there’s only Arch and Gentoo, which I would call minority platforms. (I’m aware that all UNIX-like free operating systems could be considered a minority OS all together, but we’re talking about UNIX-like OSes here :) Doing what the majority does is not always a good thing, but for this PEP I think that numbers can help us assess whether the trouble/benefit ratio is worth it. In my opinion, the current situation is clear: python is some python2.y, python3 is a python3.y, this is not ambiguous and will still work in ten years when we get python4. Thus, the previous decision of python-dev to use python3 indefinitely seems good to me. As a script/program author, if I use python2 in my shebangs now to support what appears to be minority platforms, I’m breaking compatibility with a huge number of systems. Therefore, I don’t see how this PEP makes the situation better. If one OS wants to change the meaning of the python command, then its packaging tools should adapt shebangs, and its users should be aware that the majority of existing Python 3 scripts will break. Therefore, I’m strongly -1 on this PEP: changing the meaning of python brings much trouble for little or no benefit, and adding python2 adds another compatibility trouble. It would be interesting to have feedback from people who lived the transition to Python 2. > * The ``pythonX.X`` (e.g. ``python2.6``) commands exist on some systems, on > which they invoke specific minor versions of the Python interpreter. It > can be useful for distribution-specific packages to take advantage of these > utilities if they exist, since it will prevent code breakage if the default > minor version of a given major version is changed. However, scripts > intending to be cross-platform should not rely on the presence of these > utilities, but rather should be tested on several recent minor versions of > the target major version, compensating, if necessary, for the small > differences that exist between minor versions. This prevents the need for > sysadmins to install many very similar versions of the interpreter. Here again I would be interested in more numbers. Pythons that people manually download and install using the provided makefile do have these pythonx.y executables, so I thought that all OSes did likewise. Moreover, I disagree about the implied assertion that the minor number hardly matters (I’m paraphrasing): Python 2.6 and 2.7 *are* different, not “very similar”. I don’t know very well the usages of the community, but in my experience moving from 2.x to 2.x+1, or even just checking that your code still works, is a Big Deal. I’d like this whole bullet item to be removed. > Impact on PYTHON* Environment Va
[Python-Dev] PEP 402: Simplified Package Layout and Partitioning
Hi, I’ve read PEP 402 and would like to offer comments. I know a bit about the import system, but not down to the nitty-gritty details of PEP 302 and __path__ computations and all this fun stuff (by which I mean, not fun at all). As such, I can’t find nasty issues in dark corners, but I can offer feedback as a user. I think it’s a very well-written explanation of a very useful feature: +1 from me. If it is accepted, the docs will certainly be much more concise, but the PEP as a thought process is a useful document to read. > When new users come to Python from other languages, they are often > confused by Python's packaging semantics. Minor: I would reserve “packaging” for packaging/distribution/installation/deployment matters, not Python modules. I suggest “Python package semantics”. > On the negative side, however, it is non-intuitive for beginners, and > requires a more complex step to turn a module into a package. If > ``Foo`` begins its life as ``Foo.py``, then it must be moved and > renamed to ``Foo/__init__.py``. Minor: In the UNIX world, or with version control tools, moving and renaming are the same one thing (hg mv spam.py spam/__init__.py for example). Also, if you turn a module into a package, you may want to move code around, change imports, etc., so I’m not sure the renaming part is such a big step. Anyway, if the import-sig people say that users think it’s a complex or costly operation, I can believe it. > (By the way, both of these additions to the import protocol (i.e. the > dynamically-added ``__path__``, and dynamically-created modules) > apply recursively to child packages, using the parent package's > ``__path__`` in place of ``sys.path`` as a basis for generating a > child ``__path__``. This means that self-contained and virtual > packages can contain each other without limitation, with the caveat > that if you put a virtual package inside a self-contained one, it's > gonna have a really short ``__path__``!) I don’t understand the caveat or its implications. > In other words, we don't allow pure virtual packages to be imported > directly, only modules and self-contained packages. (This is an > acceptable limitation, because there is no *functional* value to > importing such a package by itself. After all, the module object > will have no *contents* until you import at least one of its > subpackages or submodules!) > > Once ``zc.buildout`` has been successfully imported, though, there > *will* be a ``zc`` module in ``sys.modules``, and trying to import it > will of course succeed. We are only preventing an *initial* import > from succeeding, in order to prevent false-positive import successes > when clashing subdirectories are present on ``sys.path``. I find that limitation acceptable. After all, there is no zc project, and no zc module, just a zc namespace. I’ll just regret that it’s not possible to provide a module docstring to inform that this is a namespace package used for X and Y. > The resulting list (whether empty or not) is then stored in a > ``sys.virtual_package_paths`` dictionary, keyed by module name. This was probably said on import-sig, but here I go: yet another import artifact in the sys module! I hope we get ImportEngine in 3.3 to clean up all this. > * A new ``extend_virtual_paths(path_entry)`` function, to extend > existing, already-imported virtual packages' ``__path__`` attributes > to include any portions found in a new ``sys.path`` entry. This > function should be called by applications extending ``sys.path`` > at runtime, e.g. when adding a plugin directory or an egg to the > path. Let’s imagine my application Spam has a namespace spam.ext for plugins. To use a custom directory where plugins are stored, or a zip file with plugins (I don’t use eggs, so let me talk about zip files here), I’d have to call sys.path.append *and* pkgutil.extend_virtual_paths? > * ``ImpImporter.iter_modules()`` should be changed to also detect and > yield the names of modules found in virtual packages. Is there any value in providing an argument to get the pre-PEP behavior? Or to look at it from a different place, how can Python code know that some module is a virtual or pure virtual package, if that is even a useful thing to know? > Last, but not least, the ``imp`` module (or ``importlib``, if > appropriate) should expose the algorithm described in the `virtual > paths`_ section above, as a > ``get_virtual_path(modulename, parent_path=None)`` function, so that > creators of ``__import__`` replacements can use it. If I’m not mistaken, the rule of thumb these days is that imp is edited when it’s absolutely necessary, otherwise code goes into importlib (more easily written, read and maintained). I wonder if importlib.import_module could implement the new import semantics all by itself, so that we can benefit from this PEP in older Pythons (importlib is on PyPI). > * If you are changing a currently self-contained package into a > virtual one, it
Re: [Python-Dev] cpython: News item for #12724
On Thu, Aug 11, 2011 at 00:26, Georg Brandl wrote: > Am 11.08.2011 03:34, schrieb brian.curtin: > > http://hg.python.org/cpython/rev/3a6782f2a4a8 > > changeset: 71811:3a6782f2a4a8 > > user:Brian Curtin > > date:Wed Aug 10 20:32:10 2011 -0500 > > summary: > > News item for #12724 > > > > files: > > Misc/NEWS | 2 ++ > > 1 files changed, 2 insertions(+), 0 deletions(-) > > If it gets a NEWS entry, why isn't it documented? I left it out just to see if you were paying attention :) Now that I got caught, added in http://hg.python.org/cpython/rev/e88362fb4950 ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython (2.7): Fix closes Issue12722 - link heapq source in the text format in the
On Thu, Aug 11, 2011 at 16:33, Éric Araujo wrote: > Hi, > >> IIRC the reason why we don't do it on 2.x is because we don't have the >> 'source' directive available in Sphinx and therefore we would have to >> update all the links manually to link to h.p.o instead of s.p.o. > > In 3.2 and higher, there is a custom source role in > Doc/tools/sphinxext/pyspecific.py. For 2.7, I volunteered to change all > links manually (sed being, as usual, my friend) but just lacked time. Is there a reason we can't use the same sphinx role in 2.7 too? And also the same sphinx (thus sphinxext) versions on 2.7 and 3.x? that would probably help in keeping the diffs on the documentation smaller. Regards, -- Sandro Tosi (aka morph, morpheus, matrixhasu) My website: http://matrixhasu.altervista.org/ Me at Debian: http://wiki.debian.org/SandroTosi ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython (2.7): Fix closes Issue12722 - link heapq source in the text format in the
Le 11/08/2011 16:47, Sandro Tosi a écrit : > Is there a reason we can't use the same sphinx role in 2.7 too? And > also the same sphinx (thus sphinxext) versions on 2.7 and 3.x? that > would probably help in keeping the diffs on the documentation smaller. Even though the pyspecific module is wholly private and used only for our build process, Georg seems to follow the rule that we don’t add new features in stable branches. I think that’s why the new role was added in 3.2 when in was in dev phase but not to 2.7 (see #10334). We also use different versions of Sphinx. Regards ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEPs] Rebooting PEP 394 (aka Support the /usr/bin/python2 symlink upstream)
I think you missed the point of the PEP. The point is to create a new, python-dev-blessed standard that the distros will follow. The primary goal is so that a script can specify python2 or python3 in the #! line and expect that to work on all compliant linux systems, which we hope will be all of them. Everything else is just details. And yes, that distinction is much more important than the distinction between minor version numbers. That's the whole point of python3, after all. -- R. David Murray http://www.bitdance.com ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEPs] Rebooting PEP 394 (aka Support the /usr/bin/python2 symlink upstream)
Hi Devid, > I think you missed the point of the PEP. The point is to create a new, > python-dev-blessed standard that the distros will follow. The primary > goal is so that a script can specify python2 or python3 in the #! > line and expect that to work on all compliant linux systems, which we > hope will be all of them. Everything else is just details. I’m sorry if my opinion on that main point was lost among remarks on details. To rephrase one part of my reply: Right now, the de facto standard is that shebangs can use python to mean python2 and python3 to mean python3. Adding python2 to that and supporting making python ambiguous seems harmful to me. Regards ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 402: Simplified Package Layout and Partitioning
On Aug 11, 2011, at 04:39 PM, Éric Araujo wrote: >> * XXX what is the __file__ of a "pure virtual" package? ``None``? >> Some arbitrary string? The path of the first directory with a >> trailing separator? No matter what we put, *some* code is >> going to break, but the last choice might allow some code to >> accidentally work. Is that good or bad? >A pure virtual package having no source file, I think it should have no >__file__ at all. I don’t know if that would break more code than using >an empty string for example, but it feels righter. I agree that the empty string is the worst of the choices. no __file__ or __file__=None is better. -Barry ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 402: Simplified Package Layout and Partitioning
On Aug 11, 2011, at 11:39 AM, Barry Warsaw wrote: > On Aug 11, 2011, at 04:39 PM, Éric Araujo wrote: > >>> * XXX what is the __file__ of a "pure virtual" package? ``None``? >>> Some arbitrary string? The path of the first directory with a >>> trailing separator? No matter what we put, *some* code is >>> going to break, but the last choice might allow some code to >>> accidentally work. Is that good or bad? >> A pure virtual package having no source file, I think it should have no >> __file__ at all. I don’t know if that would break more code than using >> an empty string for example, but it feels righter. > > I agree that the empty string is the worst of the choices. no __file__ or > __file__=None is better. In some sense, I agree: hacks like empty strings are likely to lead to path-manipulation bugs where the wrong file gets opened (or worse, deleted, with predictable deleterious effects). But the whole "pure virtual" mechanism here seems to pile even more inconsistency on top of an already irritatingly inconsistent import mechanism. I was reasonably happy with my attempt to paper over PEP 302's weirdnesses from a user perspective: http://twistedmatrix.com/documents/11.0.0/api/twisted.python.modules.html (or https://launchpad.net/modules if you are not a Twisted user) Users of this API can traverse the module hierarchy with certain expectations; each module or package would have .pathEntry and .filePath attributes, each of which would refer to the appropriate place. Of course __path__ complicates things a bit, but so it goes. Now it seems like pure virtual packages are going to introduce a new type of special case into the hierarchy which have neither .pathEntry nor .filePath objects. Rather than a one-by-one ad-hoc consideration of which attribute should be set to None or empty strings or "" or what have you, I'd really like to see a discussion in the PEP saying what a package really is vs. what a module is, and what one can reasonably expect from it from an API and tooling perspective. Right now I have to puzzle out the intent of the final API from the problem/solution description and thought experiment. Despite authoring several namespace packages myself, I don't have any of the problems described in the PEP. I just want to know how to write correct tools given this new specification. I suspect that this PEP will be the only reference for how packages work for a long time coming (just as PEP 302 was before it) so it should really get this right.___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 402: Simplified Package Layout and Partitioning
On Thu, 11 Aug 2011 11:39:52 -0400 Barry Warsaw wrote: > On Aug 11, 2011, at 04:39 PM, Éric Araujo wrote: > > >> * XXX what is the __file__ of a "pure virtual" package? ``None``? > >> Some arbitrary string? The path of the first directory with a > >> trailing separator? No matter what we put, *some* code is > >> going to break, but the last choice might allow some code to > >> accidentally work. Is that good or bad? > >A pure virtual package having no source file, I think it should have no > >__file__ at all. I don’t know if that would break more code than using > >an empty string for example, but it feels righter. > > I agree that the empty string is the worst of the choices. no __file__ or > __file__=None is better. None should be the answer. It simplifies inspection of module data (repr(__file__) gives you something recognizable instead of raising) and makes semantically sense (!) since there is, indeed, no actual file backing the module. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython (2.7): Fix closes Issue12722 - link heapq source in the text format in the
Am 11.08.2011 17:01, schrieb Merwok: > Le 11/08/2011 16:47, Sandro Tosi a écrit : >> Is there a reason we can't use the same sphinx role in 2.7 too? And >> also the same sphinx (thus sphinxext) versions on 2.7 and 3.x? that >> would probably help in keeping the diffs on the documentation smaller. > > Even though the pyspecific module is wholly private and used only for > our build process, Georg seems to follow the rule that we don’t add new > features in stable branches. I think that’s why the new role was added > in 3.2 when in was in dev phase but not to 2.7 (see #10334). I think I just put it in default as a test, and intended to backport it later when it proved useful. You're welcome to do so now. > We also use different versions of Sphinx. That doesn't matter for this role. Georg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 402: Simplified Package Layout and Partitioning
At 04:39 PM 8/11/2011 +0200, Ãric Araujo wrote: Hi, I've read PEP 402 and would like to offer comments. Thanks. Minor: I would reserve "packaging" for packaging/distribution/installation/deployment matters, not Python modules. I suggest "Python package semantics". Changing to "Python package import semantics" to hopefully be even clearer. ;-) (Nitpick: I was somewhat intentionally ambiguous because we are talking here about how a package is physically implemented in the filesystem, and that actually *is* kind of a packaging issue. But it's not necessarily a *useful* intentional ambiguity, so I've no problem with removing it.) Minor: In the UNIX world, or with version control tools, moving and renaming are the same one thing (hg mv spam.py spam/__init__.py for example). Also, if you turn a module into a package, you may want to move code around, change imports, etc., so I'm not sure the renaming part is such a big step. Anyway, if the import-sig people say that users think it's a complex or costly operation, I can believe it. It's not that it's complex or costly in anything other than *mental* overhead -- you have to remember to do it and it's not particularly obvious. (But people on import-sig did mention this and other things covered by the PEP as being a frequent root cause of beginner inquiries on #python, Stackoverflow, et al.) > (By the way, both of these additions to the import protocol (i.e. the > dynamically-added ``__path__``, and dynamically-created modules) > apply recursively to child packages, using the parent package's > ``__path__`` in place of ``sys.path`` as a basis for generating a > child ``__path__``. This means that self-contained and virtual > packages can contain each other without limitation, with the caveat > that if you put a virtual package inside a self-contained one, it's > gonna have a really short ``__path__``!) I don't understand the caveat or its implications. Since each package's __path__ is the same length or shorter than its parent's by default, then if you put a virtual package inside a self-contained one, it will be functionally speaking no different than a self-contained one, in that it will have only one path entry. So, it's not really useful to put a virtual package inside a self-contained one, even though you can do it. (Apart form it letting you avoid a superfluous __init__ module, assuming it's indeed superfluous.) > In other words, we don't allow pure virtual packages to be imported > directly, only modules and self-contained packages. (This is an > acceptable limitation, because there is no *functional* value to > importing such a package by itself. After all, the module object > will have no *contents* until you import at least one of its > subpackages or submodules!) > > Once ``zc.buildout`` has been successfully imported, though, there > *will* be a ``zc`` module in ``sys.modules``, and trying to import it > will of course succeed. We are only preventing an *initial* import > from succeeding, in order to prevent false-positive import successes > when clashing subdirectories are present on ``sys.path``. I find that limitation acceptable. After all, there is no zc project, and no zc module, just a zc namespace. I'll just regret that it's not possible to provide a module docstring to inform that this is a namespace package used for X and Y. It *is* possible - you'd just have to put it in a "zc.py" file. IOW, this PEP still allows "namespace-defining packages" to exist, as was requested by early commenters on PEP 382. It just doesn't *require* them to exist in order for the namespace contents to be importable. > The resulting list (whether empty or not) is then stored in a > ``sys.virtual_package_paths`` dictionary, keyed by module name. This was probably said on import-sig, but here I go: yet another import artifact in the sys module! I hope we get ImportEngine in 3.3 to clean up all this. Well, I rather *like* having them there, personally, vs. having to learn yet another API, but oh well, whatever. AFAIK, ImportEngine isn't going to do away with the need for the global ones to live somewhere, at least not in 3.3. > * A new ``extend_virtual_paths(path_entry)`` function, to extend > existing, already-imported virtual packages' ``__path__`` attributes > to include any portions found in a new ``sys.path`` entry. This > function should be called by applications extending ``sys.path`` > at runtime, e.g. when adding a plugin directory or an egg to the > path. Let's imagine my application Spam has a namespace spam.ext for plugins. To use a custom directory where plugins are stored, or a zip file with plugins (I don't use eggs, so let me talk about zip files here), I'd have to call sys.path.append *and* pkgutil.extend_virtual_paths? As written in the current proposal, yes. There was some discussion on Python-Dev about having this happen automatically, and I proposed that i
Re: [Python-Dev] [PEPs] Rebooting PEP 394 (aka Support the /usr/bin/python2 symlink upstream)
On Thu, 11 Aug 2011 17:12:22 +0200, =?UTF-8?B?w4lyaWMgQXJhdWpv?= wrote: > Iâm sorry if my opinion on that main point was lost among remarks on > details. To rephrase one part of my reply: Right now, the de facto > standard is that shebangs can use python to mean python2 and python3 to > mean python3. Adding python2 to that and supporting making python > ambiguous seems harmful to me. OK. So you are -1 on the PEP. I'm a big +1. To address your argument briefly, *now* a minority of distros have python pointing to python2. We expect this to change. It may not happen for 5 years, but someday it will. So this PEP is about preparing for the future. Given that, I fail to see what harm having an additional symlink named python2 will do. And yes this was argued about earlier and should (in theory at least) be addressed by the PEP, which is why I'm concluding that you are -1 on the PEP :). -- R. David Murray http://www.bitdance.com ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GIL removal question
Den 09.08.2011 11:33, skrev Марк Коренберг: Probably I want to re-invent a bicycle. I want developers to say me why we can not remove GIL in that way: 1. Remove GIL completely with all current logick. 2. Add it's own RW-locking to all mutable objects (like list or dict) 3. Add RW-locks to every context instance 4. use RW-locks when accessing members of object instances Only one reason, I see, not do that -- is performance of singlethreaded applications. Why not to fix locking functions for this 4 cases to stubs when only one thread present? This has been discussed to death before, and is probably OT to this list. There is another reason than speed of single-threaded applications, but it is rather technical: As CPython uses reference counting for garbage collection, we would get "false sharing" of reference counts -- which would work as an "invisible GIL" (synchronization bottleneck) anyway. That is, if one processor writes to memory in a cache-line shared by another processor, they must stop whatever they are doing to synchronize the dirty cache lines with RAM. Thus, updating reference counts would flood the memory bus with traffic and be much worse than the GIL. Instead of doing useful work, the processors would be stuck synchronizing dirty cache lines. You can think of it as a severe traffic jam. To get rid of the GIL, CPython would either need (a) another GC method (e.g. similar to .NET or Java) or (b) another threading model (e.g. one interpreter per thread, as in Tcl, Erlang, or .NET app domains). As CPython has neither, we are better off with the GIL. Nobody likes the GIL, fork a project to write a GIL free CPython if you can. But note that: 1. With Cython, you have full manual control over the GIL. IronPython and Jython does not have a GIL at all. 2. Much of the FUD against the GIL is plain ignorance: The GIL slows down parallel computational code, but any serious number crunching should use numerical performance libraries (i.e. C extensions) anyway. Libraries are free to release the GIL or spawn threads internally. Also, the GIL does not matter for (a) I/O bound code such as network servers or clients and (b) background threads in GUI programs -- which are the two common use-cases for threads in Python programs. If the GIL bites you, it's most likely a warning that your program is badly written, independent of the GIL issue. There seems to be a common misunderstanding that Python threads work like fibers due to they GIL. They do not! Python threads are native OS threads and can do anything a thread can do, including executing library code in parallel. If one thread is blocking on I/O, the other threads can continue with their business. The only thing Python threads cannot do is access the Python interpreter concurrently. And the reason CPython needs that restriction is reference counting. Sturla ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of the PEP 400? (deprecate codecs.StreamReader/StreamWriter)
Le 29/07/2011 19:01, Guido van Rossum a écrit : I will add your alternative to the PEP (except if you would like to do that yourself?). If I understood correctly, you propose to: * rename codecs.open() to codecs.open_stream() * change codecs.open() to reuse open() (and so io.TextIOWrapper) (...) +1 Ok, most people prefer this option. Should I modify the PEP to "move" this option has the first/main proposition (move my proposition as an alternative?), or can the PEP be validated in the current state? Victor ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEPs] Rebooting PEP 394 (aka Support the /usr/bin/python2 symlink upstream)
On 8/11/2011 10:36 AM, Éric Araujo wrote: It would be interesting to have feedback from people who lived the transition to Python 2. There was no comparable transition. Python 2.0 was basically 1.6 renamed for a different distributor. I regard Python 2.2, which introduced new-style, as the beginning of Python 2 as something significantly different from Python 1. I suppose one could also point to the earlier intro of unicode. The new iterator protocol was also a major change. In any case, back compatibility was kept in all three respects (and others) until Python 3. -- Terry Jan Reedy ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of the PEP 400? (deprecate codecs.StreamReader/StreamWriter)
On 8/11/2011 3:31 PM, Victor Stinner wrote: Le 29/07/2011 19:01, Guido van Rossum a écrit : I will add your alternative to the PEP (except if you would like to do that yourself?). If I understood correctly, you propose to: * rename codecs.open() to codecs.open_stream() * change codecs.open() to reuse open() (and so io.TextIOWrapper) (...) +1 Ok, most people prefer this option. Should I modify the PEP to "move" this option has the first/main proposition (move my proposition as an alternative?), or can the PEP be validated in the current state? I would relabel the above as the Minimal Change Alternative or M.A.L. alternative or whatever and possibly move it but in any case note that Guido (and others) accepted that alternative with consideration of more drastic changes deferred to later. And add an explicit reference to the email you quoted. -- Terry Jan Reedy ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 402: Simplified Package Layout and Partitioning
On Fri, Aug 12, 2011 at 4:30 AM, P.J. Eby wrote: > At 04:39 PM 8/11/2011 +0200, Éric Araujo wrote: >> > The resulting list (whether empty or not) is then stored in a >> > ``sys.virtual_package_paths`` dictionary, keyed by module name. >> This was probably said on import-sig, but here I go: yet another import >> artifact in the sys module! I hope we get ImportEngine in 3.3 to clean >> up all this. > > Well, I rather *like* having them there, personally, vs. having to learn yet > another API, but oh well, whatever. AFAIK, ImportEngine isn't going to do > away with the need for the global ones to live somewhere, at least not in > 3.3. And likely not for the entire 3.x series - I shudder at the thought of the backwards incompatibility hell associated with trying to remove them... The point of the ImportEngine API is that the caching elements of the import state introduce cross dependencies between various global data structures. Code that manipulates those data structures needs to correctly invalidate or otherwise update the state as things change. I seem to recall a certain programming construct that is designed to make it easier to manage interdependent data structures... Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEPs] Rebooting PEP 394 (aka Support the /usr/bin/python2 symlink upstream)
On Fri, Aug 12, 2011 at 1:12 AM, Éric Araujo wrote: > I’m sorry if my opinion on that main point was lost among remarks on > details. To rephrase one part of my reply: Right now, the de facto > standard is that shebangs can use python to mean python2 and python3 to > mean python3. Adding python2 to that and supporting making python > ambiguous seems harmful to me. This PEP comes mainly out of the fact that we collectively think Arch (the case that prompted the original discussion) are making a mistake that will hurt their users in switching the default Python *right now*, so the PEP is first and foremost designed to record that consensus. However, their actions do mean that the 'python' name is *already* ambiguous, no matter what the mainstream distros think. The Debian maintainers may not care about that, but *I* do, as does anyone wanting to write distro-agnostic shebang lines. Given that some distros (large or small), along with some system administrators, are going to want to have python refer to python3, either now or at some point in the future, there are really only two options available to us here: 1. Accept the reality of that situation, and propose a mechanism that minimises the impact of the resulting ambiguity on end users of Python by allowing developers to be explicit about their target language. This is the approach advocated in PEP 394. 2. Tell the Arch developers (and anyone else inclined to point the python name at python3) that they're wrong, and the python symlink should, now and forever, always refer to a version of Python 2.x. It's worth noting that there has never been any previous python-dev consensus to use 'python3' indefinitely - the status quo came about because it makes sense for the moment even to those of us that *want* 'python' to eventually refer to 'python3', so there was no previous need for an explicit choice between the two alternatives. By migrating, Arch has forced us to choose between either supporting their action or else telling Python users "Don't blame us, blame the distros that pointed python at python3" when things break. I flat out disagree with the second approach - having to type 'python3' when a 3.x variant is the only version of Python installed would just be dumb, and I also think playing that kind of blame game in the face of inevitable cross-distro compatibility problems is disrespectful to our users. If you want to get Zen about it, 'practicality beats purity', 'explicit is better than implicit' and 'In the face of ambiguity, refuse the temptation to guess' all come down in favour of the approach in PEP 394. If I haven't persuaded you to adjust your view up to at least a -0 (i.e. don't entirely agree, but won't object to others moving forward with it) and you still wish to advocate for the second approach, then I suggest creating a competing PEP in order to provide a clear alternative proposal (with Guido or his appointed delegate having the final say, as usual) that explains the alternative recommendation for: - distros that have already switched their 'python' links to refer to python3 - Python developers wishing to write shebang lines that work on multiple 2.x versions and support both platforms that define 'python2' and those that only define 'python' FWIW, the closest historical precedent I can recall is Red Hat's issues when users switched the system Python from 1.5 to 2.2+, and the lesson learned from that exercise was that distro installed utilities should always reference a specific Python version rather than relying on the system administrator leaving the 'python' link alone. It sounds like Debian chose not to heed that lesson, which is unfortunate (although, to be honest, I'm not sure how well Fedora/Red Hat heed it, either). However, the commentary in PEP 394 based on that history (i.e. that distros really shouldn't care where the python name points) will remain in place. Regards, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of the PEP 400? (deprecate codecs.StreamReader/StreamWriter)
On Fri, Aug 12, 2011 at 8:21 AM, Terry Reedy wrote: > On 8/11/2011 3:31 PM, Victor Stinner wrote: >> Ok, most people prefer this option. Should I modify the PEP to "move" >> this option has the first/main proposition (move my proposition as an >> alternative?), or can the PEP be validated in the current state? > > I would relabel the above as the Minimal Change Alternative or M.A.L. > alternative or whatever and possibly move it but in any case note that Guido > (and others) accepted that alternative with consideration of more drastic > changes deferred to later. And add an explicit reference to the email you > quoted. Yeah, definitely retitle/rewrite/rearrange to be clear what Guido accepted and then state that any future deprecation of components in the codecs module will be dealt with as a new PEP. Regards, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
