[Python-Dev] Left the GSoC-mentors list
Hi, I've just left the soc2009-mentors list on request, as I'm not a mentor. So if you need my input on the mentor side regarding ideas I've contributed to [1] (struct, socket, core helper tools or Roundup), please CC me. Best regards, Daniel [1] http://wiki.python.org/moin/SummerOfCode/2009/Incoming ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's update CObject API so it is safe and regular!
Thanks Larry. I didn't notice the patch, or indeed the defect, hence my question. A clarification in the documentation that a string comparison is indeed used might be useful. As a user of CObject I appreciate this effort. K -Original Message- From: Larry Hastings [mailto:la...@hastings.org] A method for answering further such questions suggests itself, ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's update CObject API so it is safe and regular!
Jim Fulton wrote: The only type-safety mechanism for a CObject is it's identity. If you want to make sure you're using the foomodule api, make sure the address of the CObject is the same as the address of the api object exported by the module. I don't follow that. If you already have the address of the thing you want to use, you don't need a CObject. 2. Only code provided by the module provider should be accessing the CObject exported by the module. Not following that either. Without attaching some kind of metadata to a CObject, I don't see how you can know whether a CObject passed to you from Python code is one that you created yourself, or by some other unrelated piece of code. Attaching some kind of type info to a CObject and having an easy way of checking it makes sense to me. If the existing CObject API can't be changed, maybe a new enhanced one could be added. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] OSError.errno = exception hierarchy?
Apologies if this has already been discussed. I was expecting that by now, python 3.0, the following code: # clean the target dir import errno try: shutil.rmtree(trace_output_path) except OSError, ex: if ex.errno not in [errno.ENOENT]: raise Would have become something simpler, like this: # clean the target dir try: shutil.rmtree(trace_output_path) except OSErrorNoEntry: # or maybe os.ErrorNoEntry pass Apparently no one has bothered yet to turn OSError + errno into a hierarchy of OSError subclasses, as it should. What's the problem, no will to do it, or no manpower? Regards, -- Gustavo J. A. M. Carneiro INESC Porto, Telecommunications and Multimedia Unit The universe is always one step beyond logic. -- Frank Herbert ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's update CObject API so it is safe and regular!
Greg Ewing wrote: Attaching some kind of type info to a CObject and having an easy way of checking it makes sense to me. If the existing CObject API can't be changed, maybe a new enhanced one could be added. I thought the entire *point* of C object was that it's an opaque box without any info whatsoever, except that which is known and shared by its creator and its consumer. If we're adding type information, then please make it a Python object rather than a C string. That way the creator and the consumer can use a richer API to query the type, such as by calling its methods or by inspecting it in some other way. Instead of comparing strings with strcmp, it could use PyObject_RichCompareBool, which would allow a much more flexible way to define types. Using a PyObject also ensures that the lifecycle of the attached type is managed by the well-understood reference-counting mechanism. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's update CObject API so it is safe and regular!
On Apr 1, 2009, at 11:51 PM, Guido van Rossum wrote: ... Note also this cheap exported-vtable hack isn't the only use of CObjects; for example _ctypes uses them to wrap plenty of one-off objects which are never set as attributes of the _ctypes module. We'd like a solution that enforces some safety for those too, without creating spurious module attributes. Why would you care about safety for ctypes? It's about as unsafe as it gets anyway. Coredump emptor I say. At which point, I wonder why we worry so much about someone intentionally breaking a CObject as in Larry's example. Jim -- Jim Fulton Zope Corporation ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's update CObject API so it is safe and regular!
On Apr 2, 2009, at 7:28 AM, Greg Ewing wrote: Jim Fulton wrote: The only type-safety mechanism for a CObject is it's identity. If you want to make sure you're using the foomodule api, make sure the address of the CObject is the same as the address of the api object exported by the module. I don't follow that. If you already have the address of the thing you want to use, you don't need a CObject. I was refering to the identity of the CObject itself. 2. Only code provided by the module provider should be accessing the CObject exported by the module. Not following that either. Without attaching some kind of metadata to a CObject, I don't see how you can know whether a CObject passed to you from Python code is one that you created yourself, or by some other unrelated piece of code. The original use case for CObjects was to export an API from a module, in which case, you'd be importing the API from the module. The presence in the module indicates the type. Of course, this doesn't account for someone intentionally replacing the module's CObject with a fake. Attaching some kind of type info to a CObject and having an easy way of checking it makes sense to me. If the existing CObject API can't be changed, maybe a new enhanced one could be added. I don't think backward compatibility needs to be a consideration for Python 3 at this point. I don't see much advantage in the proposal, but I can live with it for Python 3. Jim -- Jim Fulton Zope Corporation ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] py3k regression tests on Windows
Hello there. Yesterday I created a number of defects for regression test failures on Windows: http://bugs.python.org/issue5646 : test_importlib fails for py3k on Windows http://bugs.python.org/issue5645 : test_memoryio fails for py3k on windows http://bugs.python.org/issue5643 : test__locale fails with RADIXCHAR on Windows Does anyone feel like taking a look? K ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 382: Namespace Packages
I propose the following PEP for inclusion to Python 3.1. Please comment. Regards, Martin Abstract Namespace packages are a mechanism for splitting a single Python package across multiple directories on disk. In current Python versions, an algorithm to compute the packages __path__ must be formulated. With the enhancement proposed here, the import machinery itself will construct the list of directories that make up the package. Terminology === Within this PEP, the term package refers to Python packages as defined by Python's import statement. The term distribution refers to separately installable sets of Python modules as stored in the Python package index, and installed by distutils or setuptools. The term vendor package refers to groups of files installed by an operating system's packaging mechanism (e.g. Debian or Redhat packages install on Linux systems). The term portion refers to a set of files in a single directory (possibly stored in a zip file) that contribute to a namespace package. Namespace packages today Python currently provides the pkgutil.extend_path to denote a package as a namespace package. The recommended way of using it is to put:: from pkgutil import extend_path __path__ = extend_path(__path__, __name__) int the package's ``__init__.py``. Every distribution needs to provide the same contents in its ``__init__.py``, so that extend_path is invoked independent of which portion of the package gets imported first. As a consequence, the package's ``__init__.py`` cannot practically define any names as it depends on the order of the package fragments on sys.path which portion is imported first. As a special feature, extend_path reads files named ``*.pkg`` which allow to declare additional portions. setuptools provides a similar function pkg_resources.declare_namespace that is used in the form:: import pkg_resources pkg_resources.declare_namespace(__name__) In the portion's __init__.py, no assignment to __path__ is necessary, as declare_namespace modifies the package __path__ through sys.modules. As a special feature, declare_namespace also supports zip files, and registers the package name internally so that future additions to sys.path by setuptools can properly add additional portions to each package. setuptools allows declaring namespace packages in a distribution's setup.py, so that distribution developers don't need to put the magic __path__ modification into __init__.py themselves. Rationale = The current imperative approach to namespace packages has lead to multiple slightly-incompatible mechanisms for providing namespace packages. For example, pkgutil supports ``*.pkg`` files; setuptools doesn't. Likewise, setuptools supports inspecting zip files, and supports adding portions to its _namespace_packages variable, whereas pkgutil doesn't. In addition, the current approach causes problems for system vendors. Vendor packages typically must not provide overlapping files, and an attempt to install a vendor package that has a file already on disk will fail or cause unpredictable behavior. As vendors might chose to package distributions such that they will end up all in a single directory for the namespace package, all portions would contribute conflicting __init__.py files. Specification = Rather than using an imperative mechanism for importing packages, a declarative approach is proposed here, as an extension to the existing ``*.pkg`` mechanism. The import statement is extended so that it directly considers ``*.pkg`` files during import; a directory is considered a package if it either contains a file named __init__.py, or a file whose name ends with .pkg. In addition, the format of the ``*.pkg`` file is extended: a line with the single character ``*`` indicates that the entire sys.path will be searched for portions of the namespace package at the time the namespace packages is imported. Importing a package will immediately compute the package's __path__; the ``*.pkg`` files are not considered anymore after the initial import. If a ``*.pkg`` package contains an asterisk, this asterisk is prepended to the package's __path__ to indicate that the package is a namespace package (and that thus further extensions to sys.path might also want to extend __path__). At most one such asterisk gets prepended to the path. extend_path will be extended to recognize namespace packages according to this PEP, and avoid adding directories twice to __path__. No other change to the importing mechanism is made; searching modules (including __init__.py) will continue to stop at the first module encountered. Discussion == With the addition of ``*.pkg`` files to the import mechanism, namespace packages can stop filling out the namespace package's __init__.py. As a consequence, extend_path and declare_namespace become obsolete. It is recommended that distributions put a file distribution.pkg into
Re: [Python-Dev] PEP 382: Namespace Packages
At 10:32 AM 4/2/2009 -0500, Martin v. Löwis wrote: I propose the following PEP for inclusion to Python 3.1. Please comment. An excellent idea. One thing I am not 100% clear on, is how to get additions to sys.path to work correctly with this. Currently, when pkg_resources adds a new egg to sys.path, it uses its existing registry of namespace packages in order to locate which packages need __path__ fixups. It seems under this proposal that it would have to scan sys.modules for objects with __path__ attributes that are lists that begin with a '*', instead... which is a bit troubling because sys.modules doesn't always only contain module objects. Many major frameworks place lazy module objects, and module proxies or wrappers of various sorts in there, so scanning through it arbitrarily is not really a good idea. Perhaps we could add something like a sys.namespace_packages that would be updated by this mechanism? Then, pkg_resources could check both that and its internal registry to be both backward and forward compatible. Apart from that, this mechanism sounds great! I only wish there was a way to backport it all the way to 2.3 so I could drop the messy bits from setuptools. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's update CObject API so it is safe and regular!
On Thu, Apr 2, 2009 at 6:22 AM, Jim Fulton j...@zope.com wrote: The original use case for CObjects was to export an API from a module, in which case, you'd be importing the API from the module. I consider this the *only* use case. What other use cases are there? The presence in the module indicates the type. Of course, this doesn't account for someone intentionally replacing the module's CObject with a fake. And that's the problem. I would like the following to hold: given a finite number of extension modules that I trust to be safe (i.e. excluding ctypes!), pure Python code should not be able to cause any of their CObjects to be passed off for another. Putting an identity string in the CObject and checking that string in PyCObject_Import() solves this. Adding actual information about what the CObject *means* is emphatically out of scope. Once a CObject is identified as having the correct module and name, I am okay with trusting it, because Python code has no way to create CObjects. I have to trust the extension that exports the CObject anyway, since after all it is C code that could do anything at all. But I need to be able to trust that the app cannot swap CObjects. Attaching some kind of type info to a CObject and having an easy way of checking it makes sense to me. If the existing CObject API can't be changed, maybe a new enhanced one could be added. I don't think backward compatibility needs to be a consideration for Python 3 at this point. I don't see much advantage in the proposal, but I can live with it for Python 3. Good. Let's solve this for 3.1, and figure out whether or how to backport later, since for 2.6 (and probably 2.7) binary backwards compatibility is most important. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's update CObject API so it is safe and regular!
Guido van Rossum guido at python.org writes: On Thu, Apr 2, 2009 at 6:22 AM, Jim Fulton jim at zope.com wrote: The original use case for CObjects was to export an API from a module, in which case, you'd be importing the API from the module. I consider this the *only* use case. What other use cases are there? I don't know if it is good style, but I could imagine it being used to accumulate non-PyObject data in a Python container (e.g. a list), without too much overhead. It is used in getargs.c to manage a list of destructors of temporarily created data for when a call to PyArg_Parse* fails. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's update CObject API so it is safe and regular!
On Thu, Apr 2, 2009 at 10:24 AM, Antoine Pitrou solip...@pitrou.net wrote: Guido van Rossum guido at python.org writes: On Thu, Apr 2, 2009 at 6:22 AM, Jim Fulton jim at zope.com wrote: The original use case for CObjects was to export an API from a module, in which case, you'd be importing the API from the module. I consider this the *only* use case. What other use cases are there? I don't know if it is good style, but I could imagine it being used to accumulate non-PyObject data in a Python container (e.g. a list), without too much overhead. It is used in getargs.c to manage a list of destructors of temporarily created data for when a call to PyArg_Parse* fails. Well, that sounds like it really just needs to manage a variable-length array of void pointers, and using PyList and PyCObject is just laziness (and perhaps the wrong kind -- I imagine I could write the same code without using Python objects and it would be cleaner *and* faster). So no, I don't consider that a valid use case, or at least not one we need to consider for backwards compatibility of the PyCObject design. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PyDict_SetItem hook
On Thu, Apr 2, 2009 at 04:16, John Ehresman j...@wingware.com wrote: Collin Winter wrote: Have you measured the impact on performance? I've tried to test using pystone, but am seeing more differences between runs than there is between python w/ the patch and w/o when there is no hook installed. The highest pystone is actually from the binary w/ the patch, which I don't really believe unless it's some low level code generation affect. The cost is one test of a global variable and then a switch to the branch that doesn't call the hooks. I'd be happy to try to come up with better numbers next week after I get home from pycon. Pystone is pretty much a useless benchmark. If it measures anything, it's the speed of the bytecode dispatcher (and it doesn't measure it particularly well.) PyBench isn't any better, in my experience. Collin has collected a set of reasonable benchmarks for Unladen Swallow, but they still leave a lot to be desired. From the discussions at the VM and Language summits before PyCon, I don't think anyone else has better benchmarks, though, so I would suggest using Unladen Swallow's: http://code.google.com/p/unladen-swallow/wiki/Benchmarks -- Thomas Wouters tho...@python.org Hi! I'm a .signature virus! copy me into your .signature file to help me spread! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3to2 Project
On Wed, Apr 1, 2009 at 12:50 PM, Ron DuPlain ron.dupl...@gmail.com wrote: On Mon, Mar 30, 2009 at 9:29 PM, Benjamin Peterson benja...@python.org wrote: 2009/3/30 Collin Winter coll...@gmail.com: If anyone is interested in working on this during the PyCon sprints or otherwise, here are some easy, concrete starter projects that would really help move this along: - The core refactoring engine needs to be broken out from 2to3. In particular, the tests/ and fixes/ need to get pulled up a directory, out of lib2to3/. - Once that's done, lib2to3 should then be renamed to something like librefactor or something else that indicates its more general nature. This will allow both 2to3 and 3to2 to more easily share the core components. FWIW, I think it is unfortunately too late to make this change. We've already released it as lib2to3 in the standard library and I have actually seen it used in other projects. (PythonScope, for example.) Paul Kippes and I have been sprinting on this. We put lib2to3 into a refactor package and kept a shell lib2to3 to support the old interface. We are able to run 2to3, 3to2, lib2to3 tests, and refactor tests. We only have a few simple 3to2 fixes now, but they should be easy to add. We kept the old lib2to3 tests to make sure we didn't break anything. As things settle down, I'd like to verify that our new lib2to3 is backward-compatible (since right now it points to the new refactor lib) with one of the external projects. We've been using hg to push changesets between each other, but we'll be committing to the svn sandbox before the week is out. I'm heading out today, but Paul is sticking around another day. It's a start, Ron See sandbox/trunk/refactor_pkg. More fixers to come... -Ron ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PyDict_SetItem hook
The measurements are just a distractor. We all already know that the hook is being added to a critical path. Everyone will pay a cost for a feature that few people will use. This is a really bad idea. It is not part of a thorough, thought-out framework of container hooks (something that would need a PEP at the very least).The case for how it helps us is somewhat thin. The case for DTrace hooks was much stronger. If something does go in, it should be #ifdef'd out by default. But then, I don't think it should go in at all. Raymond On Thu, Apr 2, 2009 at 04:16, John Ehresman j...@wingware.com wrote: Collin Winter wrote: Have you measured the impact on performance? I've tried to test using pystone, but am seeing more differences between runs than there is between python w/ the patch and w/o when there is no hook installed. The highest pystone is actually from the binary w/ the patch, which I don't really believe unless it's some low level code generation affect. The cost is one test of a global variable and then a switch to the branch that doesn't call the hooks. I'd be happy to try to come up with better numbers next week after I get home from pycon. Pystone is pretty much a useless benchmark. If it measures anything, it's the speed of the bytecode dispatcher (and it doesn't measure it particularly well.) PyBench isn't any better, in my experience. Collin has collected a set of reasonable benchmarks for Unladen Swallow, but they still leave a lot to be desired. From the discussions at the VM and Language summits before PyCon, I don't think anyone else has better benchmarks, though, so I would suggest using Unladen Swallow's: http://code.google.com/p/unladen-swallow/wiki/Benchmarks ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's update CObject API so it is safe and regular!
Guido van Rossum wrote: On Thu, Apr 2, 2009 at 6:22 AM, Jim Fulton j...@zope.com wrote: The original use case for CObjects was to export an API from a module, in which case, you'd be importing the API from the module. I consider this the *only* use case. What other use cases are there? Exporting a C/C++ data structure: http://wiki.cacr.caltech.edu/danse/index.php/Lots_more_details_on_writing_wrappers http://www.cacr.caltech.edu/projects/ARCS/array_kluge/array_klugemodule/html/misc_8h.html http://svn.xiph.org/trunk/vorbisfile-python/vorbisfile.c Some folks don't register a proper type; they just wrap their objects in CObjects and add module methods. The obscure method in the Robin package ( http://code.google.com/p/robin/ ) curiously wraps a *Python* object in a CObject: http://code.google.com/p/robin/source/browse/trunk/src/robin/frontends/python/module.cc I must admit I don't understand why this is a good idea. There are many more wild wooly use cases to be found if you Google for PyCObject_FromVoidPtr. Using CObject to exporting C APIs seems to be the minority, outside the CPython sources anyway. /larry/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's update CObject API so it is safe and regular!
Guido van Rossum wrote: OK, my proposal would be to agree on the value of this string too: module.variable. That's a fine idea for cases where the CObject is stored as an attribute of a module; my next update of my patch will change the existing uses to use that format. Why would you care about safety for ctypes? It's about as unsafe as it gets anyway. Coredump emptor I say. _ctypes and exporting C APIs are not the only use cases of CObjects in the wild. Please see, uh, that email I wrote like five minutes ago, also a reply to you. /larry/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
P.J. Eby wrote: Apart from that, this mechanism sounds great! I only wish there was a way to backport it all the way to 2.3 so I could drop the messy bits from setuptools. Maybe we could? :-) Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] OSError.errno = exception hierarchy?
2009/4/2 Gustavo Carneiro gjcarne...@gmail.com: Apologies if this has already been discussed. I don't believe it has ever been discussed to be implemented. Apparently no one has bothered yet to turn OSError + errno into a hierarchy of OSError subclasses, as it should. What's the problem, no will to do it, or no manpower? Python doesn't need any more builtin exceptions to clutter the namespace. Besides, what's wrong with just checking the errno? -- Regards, Benjamin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
On 2009-04-02 17:32, Martin v. Löwis wrote: I propose the following PEP for inclusion to Python 3.1. Thanks for picking this up. I'd like to extend the proposal to Python 2.7 and later. Please comment. Regards, Martin Specification = Rather than using an imperative mechanism for importing packages, a declarative approach is proposed here, as an extension to the existing ``*.pkg`` mechanism. The import statement is extended so that it directly considers ``*.pkg`` files during import; a directory is considered a package if it either contains a file named __init__.py, or a file whose name ends with .pkg. That's going to slow down Python package detection a lot - you'd replace an O(1) test with an O(n) scan. Alternative Approach: - Wouldn't it be better to stick with a simpler approach and look for __pkg__.py files to detect namespace packages using that O(1) check ? This would also avoid any issues you'd otherwise run into if you want to maintain this scheme in an importer that doesn't have access to a list of files in a package directory, but is well capable for the checking the existence of a file. Mechanism: -- If the import mechanism finds a matching namespace package (a directory with a __pkg__.py file), it then goes into namespace package scan mode and scans the complete sys.path for more occurrences of the same namespace package. The import loads all __pkg__.py files of matching namespace packages having the same package name during the search. One of the namespace packages, the defining namespace package, will have to include a __init__.py file. After having scanned all matching namespace packages and loading the __pkg__.py files in the order of the search, the import mechanism then sets the packages .__path__ attribute to include all namespace package directories found on sys.path and finally executes the __init__.py file. (Please let me know if the above is not clear, I will then try to follow up on it.) Discussion: --- The above mechanism allows the same kind of flexibility we already have with the existing normal __init__.py mechanism. * It doesn't add yet another .pth-style sys.path extension (which are difficult to manage in installations). * It always uses the same naive sys.path search strategy. The strategy is not determined by some file contents. * The search is only done once - on the first import of the package. * It's possible to have a defining package dir and add-one package dirs. * Namespace packages are easy to recognize by testing for a single resource. * Namespace __pkg__.py modules can provide extra meta-information, logging, etc. to simplify debugging namespace package setups. * It's possible to freeze such setups, to put them into ZIP files, or only have parts of it in a ZIP file and the other parts in the file-system. Caveats: * Changes to sys.path will not result in an automatic rescan for additional namespace packages, if the package was already loaded. However, we could have a function to make such a rescan explicit. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 02 2009) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2009-03-19: Released mxODBC.Connect 1.0.1 http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] OSError.errno = exception hierarchy?
On Thu, Apr 2, 2009 at 4:25 PM, Benjamin Peterson benja...@python.org wrote: 2009/4/2 Gustavo Carneiro gjcarne...@gmail.com: Apologies if this has already been discussed. I don't believe it has ever been discussed to be implemented. Apparently no one has bothered yet to turn OSError + errno into a hierarchy of OSError subclasses, as it should. What's the problem, no will to do it, or no manpower? Python doesn't need any more builtin exceptions to clutter the namespace. Besides, what's wrong with just checking the errno? The problem is manpower (this has been no ones itch). In order to have a hierarchy of OSError exceptions the underlying code would have to raise them. That means diving into all the C code that raises OSError and cleaning them up. I'm +1 on the idea but -1 on doing the work myself. -Jack ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] issue5578 - explanation
R. David Murray wrote: On Wed, 1 Apr 2009 at 13:12, Chris Withers wrote: Guido van Rossum wrote: Well hold on for a minute, I remember we used to have an exec statement in a class body in the standard library, to define some file methods in socket.py IIRC. But why an exec?! Surely there must be some other way to do this than an exec? Maybe, but this sure is gnarly code: _s = (def %s(self, *args): return self._sock.%s(*args)\n\n %s.__doc__ = _realsocket.%s.__doc__\n) for _m in _socketmethods: exec _s % (_m, _m, _m, _m) del _m, _s I played around with this and managed to rewrite it as: from functools import partial from new import instancemethod def meth(name,self,*args): return getattr(self._sock,name)(*args) for _m in _socketmethods: p = partial(meth,_m) p.__name__ = _m p.__doc__ = getattr(_realsocket,_m).__doc__ m = instancemethod(p,None,_socketobject) setattr(_socketobject,_m,m) Have I missed something or is that a suitable replacement that gets rid of the exec nastiness? Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] issue5578 - explanation
On Thu, Apr 2, 2009 at 2:16 PM, Chris Withers ch...@simplistix.co.uk wrote: R. David Murray wrote: On Wed, 1 Apr 2009 at 13:12, Chris Withers wrote: Guido van Rossum wrote: Well hold on for a minute, I remember we used to have an exec statement in a class body in the standard library, to define some file methods in socket.py IIRC. But why an exec?! Surely there must be some other way to do this than an exec? Maybe, but this sure is gnarly code: _s = (def %s(self, *args): return self._sock.%s(*args)\n\n %s.__doc__ = _realsocket.%s.__doc__\n) for _m in _socketmethods: exec _s % (_m, _m, _m, _m) del _m, _s I played around with this and managed to rewrite it as: from functools import partial from new import instancemethod def meth(name,self,*args): return getattr(self._sock,name)(*args) for _m in _socketmethods: p = partial(meth,_m) p.__name__ = _m p.__doc__ = getattr(_realsocket,_m).__doc__ m = instancemethod(p,None,_socketobject) setattr(_socketobject,_m,m) Have I missed something or is that a suitable replacement that gets rid of the exec nastiness? That code in socket.py is much older that functools... I don't know if the dependency matters, probably not. But anyways this is moot, the bug was only about exec in a class body *nested inside a function*. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] issue5578 - explanation
Guido van Rossum wrote: from functools import partial from new import instancemethod def meth(name,self,*args): return getattr(self._sock,name)(*args) for _m in _socketmethods: p = partial(meth,_m) p.__name__ = _m p.__doc__ = getattr(_realsocket,_m).__doc__ m = instancemethod(p,None,_socketobject) setattr(_socketobject,_m,m) Have I missed something or is that a suitable replacement that gets rid of the exec nastiness? That code in socket.py is much older that functools... I don't know if the dependency matters, probably not. But anyways this is moot, the bug was only about exec in a class body *nested inside a function*. Indeed, I just hate seeing execs and it was an interesting mental exercise to try and get rid of the above one ;-) Assuming it breaks no tests, would there be objection to me committing the above change to the Python 3 trunk? Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] issue5578 - explanation
On Thu, Apr 2, 2009 at 2:21 PM, Chris Withers ch...@simplistix.co.uk wrote: Guido van Rossum wrote: from functools import partial from new import instancemethod def meth(name,self,*args): return getattr(self._sock,name)(*args) for _m in _socketmethods: p = partial(meth,_m) p.__name__ = _m p.__doc__ = getattr(_realsocket,_m).__doc__ m = instancemethod(p,None,_socketobject) setattr(_socketobject,_m,m) Have I missed something or is that a suitable replacement that gets rid of the exec nastiness? That code in socket.py is much older that functools... I don't know if the dependency matters, probably not. But anyways this is moot, the bug was only about exec in a class body *nested inside a function*. Indeed, I just hate seeing execs and it was an interesting mental exercise to try and get rid of the above one ;-) Assuming it breaks no tests, would there be objection to me committing the above change to the Python 3 trunk? That's up to Benjamin. Personally, I live by if it ain't broke, don't fix it. :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] [issue3609] does parse_header really belong in CGI module?
http://bugs.python.org/issue3609 requests to move the function parse_header present in cgi module to email package. The reasons for this request are: 1) The MIME type header parsing methods rightly belong to email package. Confirming to RFC 2045. 2) parse_qs, parse_qsl were similarly moved from cgi to urlparse. The question here is, should the relocation happen in Python 2.7 as well as in Python 3K or only in Python 3k? If changes happen in Python 2.7, then cgi.parse_header will have DeprecationWarning just in case we go for more versions in Python 2.x series. Does anyone have any concerns with this change? -- Senthil ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Package Management - thoughts from the peanut gallery
Hey All, I have to admit to not having the willpower to plough through the 200 unread messages in the packaging thread when I got back from PyCon but just wanted to throw out a few thoughts on what my python packaging utopia would look like: - python would have a package format that included version numbers and dependencies. - this package format would play nice with os-specific ideas of how packages should be structured. - python itself would have a version number, so it could be treated as just another dependency by packages (ie: python =2.3,3) - python would ship with a package manager that would let you install and uninstall python packages, resolving dependencies in the process and complaining if it couldn't or if there were clashes - this package manager would facilitate the building of os-specific packages (.deb, .rpm) including providing dependency information, so making life *much* easier for these packagers. - the standard library packages would be no different from any other package, and could be overridden as and when new versions became available on PyPI, should an end user so desire. They would also be free to have their own release lifecycles (unittest, distutils, email, I'm looking at you!) - python would still ship batteries included with versions of these packages appropriate for the release, to keep those in corporate shackles or with no network happy. In fact, creating application-specific bundles like this would become trivial, helping those who have apps where they want to ship as single, isolated lumps which the os-specific package managers could use without having to worry about any python package dependencies. Personally I feel all of the above are perfectly possible, and can't see anyone being left unhappy by them. I'm sure I've missed something then, otherwise why not make it happen? cheers, Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] unittest package
Hello all, The unittest module is around 1500 lines of code now, and the tests are 3000 lines. It would be much easier to maintain as a package rather than a module. Shall I work on a suggested structure or are there objections in principle? Obviously all the functionality would still be available from the top-level unittest namespace (for backwards compatibility). Michael -- http://www.ironpythoninaction.com/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PyDict_SetItem hook
Wow. Can you possibly be more negative? I think it's worse to give the poor guy the run around by making him run lots of random benchmarks. In the end, someone will run a timeit or have a specific case that shows the full effect. All of the respondents so far seem to have a clear intuition that hook is right in the middle of a critical path. Their intuition matches what I learned by spending a month trying to find ways to optimize dictionaries. Am surprised that there has been no discussion of why this should be in the default build (as opposed to a compile time option). AFAICT, users have not previously requested a hook like this. Also, there has been no discussion for an overall strategy for monitoring containers in general. Lists and tuples will both defy this approach because there is so much code that accesses the arrays directly. Am not sure whether the setitem hook would work for other implementations either. It seems weird to me that Collin's group can be working so hard just to get a percent or two improvement in specific cases for pickling while python-dev is readily entertaining a patch that slows down the entire language. If my thoughts on the subject bug you, I'll happily withdraw from the thread. I don't aspire to be a source of negativity. I just happen to think this proposal isn't a good idea. Raymond - Original Message - From: Guido van Rossum gu...@python.org To: Raymond Hettinger pyt...@rcn.com Cc: Thomas Wouters tho...@python.org; John Ehresman j...@wingware.com; python-dev@python.org Sent: Thursday, April 02, 2009 2:19 PM Subject: Re: [Python-Dev] PyDict_SetItem hook Wow. Can you possibly be more negative? 2009/4/2 Raymond Hettinger pyt...@rcn.com: The measurements are just a distractor. We all already know that the hook is being added to a critical path. Everyone will pay a cost for a feature that few people will use. This is a really bad idea. It is not part of a thorough, thought-out framework of container hooks (something that would need a PEP at the very least). The case for how it helps us is somewhat thin. The case for DTrace hooks was much stronger. If something does go in, it should be #ifdef'd out by default. But then, I don't think it should go in at all. Raymond On Thu, Apr 2, 2009 at 04:16, John Ehresman j...@wingware.com wrote: Collin Winter wrote: Have you measured the impact on performance? I've tried to test using pystone, but am seeing more differences between runs than there is between python w/ the patch and w/o when there is no hook installed. The highest pystone is actually from the binary w/ the patch, which I don't really believe unless it's some low level code generation affect. The cost is one test of a global variable and then a switch to the branch that doesn't call the hooks. I'd be happy to try to come up with better numbers next week after I get home from pycon. Pystone is pretty much a useless benchmark. If it measures anything, it's the speed of the bytecode dispatcher (and it doesn't measure it particularly well.) PyBench isn't any better, in my experience. Collin has collected a set of reasonable benchmarks for Unladen Swallow, but they still leave a lot to be desired. From the discussions at the VM and Language summits before PyCon, I don't think anyone else has better benchmarks, though, so I would suggest using Unladen Swallow's: http://code.google.com/p/unladen-swallow/wiki/Benchmarks ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] unittest package
On Thu, 2009-04-02 at 16:58 -0500, Michael Foord wrote: Hello all, The unittest module is around 1500 lines of code now, and the tests are 3000 lines. It would be much easier to maintain as a package rather than a module. Shall I work on a suggested structure or are there objections in principle? Obviously all the functionality would still be available from the top-level unittest namespace (for backwards compatibility). Michael I'd like to see this; jmls' testtools package has a layout for this which is quite nice. -Rob signature.asc Description: This is a digitally signed message part ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PyDict_SetItem hook
Raymond Hettinger python at rcn.com writes: It seems weird to me that Collin's group can be working so hard just to get a percent or two improvement in specific cases for pickling while python-dev is readily entertaining a patch that slows down the entire language. I think it's really more than a percent or two: http://bugs.python.org/issue5670 Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PyDict_SetItem hook
It seems weird to me that Collin's group can be working so hard just to get a percent or two improvement in specific cases for pickling while python-dev is readily entertaining a patch that slows down the entire language. [Antoine Pitrou] I think it's really more than a percent or two: http://bugs.python.org/issue5670 For lists, it was a percent or two: http://bugs.python.org/issue5671 I expect Collin's overall efforts to payoff nicely. I was just pointing-out the contrast between module specific optimization efforts versus anti-optimizations that affect the whole language. Raymond ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PyDict_SetItem hook
On Thu, Apr 2, 2009 at 03:23, Christian Heimes li...@cheimes.de wrote: John Ehresman wrote: * To what extent should non-debugger code use the hook? At one end of the spectrum, the hook could be made readily available for non-debug use and at the other end, it could be documented as being debug only, disabled in python -O, not exposed in the stdlib to python code. To explain Collin's mail: Python's dict implementation is crucial to the performance of any Python program. Modules, types, instances all rely on the speed of Python's dict type because most of them use a dict to store their name space. Even the smallest change to the C code may lead to a severe performance penalty. This is especially true for set and get operations. A change that would have no performance impact could be to set mp-ma_lookup to another function, that calls all the hooks it wants before calling the super() method (lookdict). This ma_lookup is already an attribute of every dict, so a debugger could trace only the namespaces it monitors. The only problem here is that ma_lookup is called with the key and its hash, but not with the value, and you cannot know whether you are reading or setting the dict. It is easy to add an argument and call ma_lookup with the value (or NULL, or -1 depending on the action: set, get or del), but this may have a slight impact (benchmark needed!) even if this argument is not used by the standard function. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PyDict_SetItem hook
On Fri, Apr 3, 2009 at 00:07, Raymond Hettinger pyt...@rcn.com wrote: It seems weird to me that Collin's group can be working so hard just to get a percent or two improvement in specific cases for pickling while python-dev is readily entertaining a patch that slows down the entire language. Collin's group has unfortunately seen that you cannot know the actual impact of a change until you measure it. GCC performance, for instance, is extremely unpredictable, and I can easily see a change like this proving to have zero impact -- or even positive impact -- on most platforms because, say, it warms the cache for the common case. I doubt it will, but you can't *know* until you measure it. -- Thomas Wouters tho...@python.org Hi! I'm a .signature virus! copy me into your .signature file to help me spread! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PyDict_SetItem hook
On Thu, Apr 2, 2009 at 3:07 PM, Raymond Hettinger pyt...@rcn.com wrote: Wow. Can you possibly be more negative? I think it's worse to give the poor guy the run around Mind your words please. by making him run lots of random benchmarks. In the end, someone will run a timeit or have a specific case that shows the full effect. All of the respondents so far seem to have a clear intuition that hook is right in the middle of a critical path. Their intuition matches what I learned by spending a month trying to find ways to optimize dictionaries. Am surprised that there has been no discussion of why this should be in the default build (as opposed to a compile time option). AFAICT, users have not previously requested a hook like this. I may be partially to blame for this. John and Stephan are requesting this because it would (mostly) fulfill one of the top wishes of the users of Wingware. So the use case is certainly real. Also, there has been no discussion for an overall strategy for monitoring containers in general. Lists and tuples will both defy this approach because there is so much code that accesses the arrays directly. Am not sure whether the setitem hook would work for other implementations either. The primary use case is some kind of trap on assignment. While this cannot cover all cases, most non-local variables are stored in dicts. List mutations are not in the same league, as use case. It seems weird to me that Collin's group can be working so hard just to get a percent or two improvement in specific cases for pickling while python-dev is readily entertaining a patch that slows down the entire language. I don't actually believe that you can know whether this affects performance at all without serious benchmarking. The patch amounts to a single global flag check as long as the feature is disabled, and that flag could be read from the L1 cache. If my thoughts on the subject bug you, I'll happily withdraw from the thread. I don't aspire to be a source of negativity. I just happen to think this proposal isn't a good idea. I think we need more proof either way. Raymond - Original Message - From: Guido van Rossum gu...@python.org To: Raymond Hettinger pyt...@rcn.com Cc: Thomas Wouters tho...@python.org; John Ehresman j...@wingware.com; python-dev@python.org Sent: Thursday, April 02, 2009 2:19 PM Subject: Re: [Python-Dev] PyDict_SetItem hook Wow. Can you possibly be more negative? 2009/4/2 Raymond Hettinger pyt...@rcn.com: The measurements are just a distractor. We all already know that the hook is being added to a critical path. Everyone will pay a cost for a feature that few people will use. This is a really bad idea. It is not part of a thorough, thought-out framework of container hooks (something that would need a PEP at the very least). The case for how it helps us is somewhat thin. The case for DTrace hooks was much stronger. If something does go in, it should be #ifdef'd out by default. But then, I don't think it should go in at all. Raymond On Thu, Apr 2, 2009 at 04:16, John Ehresman j...@wingware.com wrote: Collin Winter wrote: Have you measured the impact on performance? I've tried to test using pystone, but am seeing more differences between runs than there is between python w/ the patch and w/o when there is no hook installed. The highest pystone is actually from the binary w/ the patch, which I don't really believe unless it's some low level code generation affect. The cost is one test of a global variable and then a switch to the branch that doesn't call the hooks. I'd be happy to try to come up with better numbers next week after I get home from pycon. Pystone is pretty much a useless benchmark. If it measures anything, it's the speed of the bytecode dispatcher (and it doesn't measure it particularly well.) PyBench isn't any better, in my experience. Collin has collected a set of reasonable benchmarks for Unladen Swallow, but they still leave a lot to be desired. From the discussions at the VM and Language summits before PyCon, I don't think anyone else has better benchmarks, though, so I would suggest using Unladen Swallow's: http://code.google.com/p/unladen-swallow/wiki/Benchmarks ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (home page: http://www.python.org/~guido/) -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] unittest package
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Apr 2, 2009, at 4:58 PM, Michael Foord wrote: The unittest module is around 1500 lines of code now, and the tests are 3000 lines. It would be much easier to maintain as a package rather than a module. Shall I work on a suggested structure or are there objections in principle? +1/jfdi :) Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (Darwin) iQCVAwUBSdVFMXEjvBPtnXfVAQJeeQQAl5yYTLCUT4M4jQBY0yb39uNexREytnmp Oo+8gaehi2at62WbeIXa3GRojfWcpAJfGEWIWxsIEe8vRBMfNJphfsiN62rD1CIt Awn9SPPka9Xxfd3fsdvfKxDpJysK1pcqNFi5e49lXgbmt8XJ/09RbviMUHFmhlxb eVYkHYmelFQ= =eNHE -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] OSError.errno = exception hierarchy?
(cross-posting back to python-dev to finalize discussions) 2009/4/2 Guido van Rossum gu...@python.org [...] The problem you report: try: ... except OSWinError: ... except OSLinError: ... Would be solved if both OSWinError and OSLinError were always defined in both Linux and Windows Python. Programs could be written to catch both OSWinError and OSLinError, except that on Linux OSWinError would never actually be raised, and on Windows OSLinError would never occur. Problem solved. Yeah, but now you'd have to generate the list of exceptions (which would be enormously long) based on the union of all errno codes in the universe. Unless you only want to do it for some errno codes and not for others, which sounds like asking for trouble. Also you need a naming scheme that works for all errnos and doesn't require manual work. Frankly, the only scheme that I can think of that could be automated would be something like OSError_ENAME. And, while OSError is built-in, I think these exceptions (because there are so many) should not be built-in, and probably not even live in the 'os' namespace -- the best place for them would be the errno module, so errno.OSError_ENAME. The downsides of this? I can only see memory, at the moment, but I might be missing something. It's an enormous amount of work to make it happen across all platforms. And it doesn't really solve an important problem. I partially agree. It will be a lot of work. I think the problem is valid, although not very important, I agree. Now just one final word why I think this matters. The currently correct way to remove a directory tree and only ignore the error it does not exist is: try: shutil.rmtree(dirname) except OSError, e: if errno.errorcode[e.errno] != 'ENOENT': raise However, only very experienced programmers will know to write that correct code (apparently I am not experienced enought!). That doesn't strike me as correct at all, since it doesn't distinguish between ENOENT being raised for some file deep down in the tree vs. the root not existing. (This could happen if after you did os.listdir() some other process deleted some file.) OK. Maybe in a generic case this could happen, although I'm sure this won't happen in my particular scenario. This is about a build system, and I am assuming there are no two concurrent builds (or else a lot of other things would fail anyway). A better way might be try: shutil.rmtree(dir) except OSError: if os.path.exists(dir): raise Sure, this works, but at the cost of an extra system call. I think it's more elegant to check the errno (assuming the corner case you pointed out above is not an issue). Though I don't know what you wish to happen of dir were a dangling symlink. What I am proposing is that the simpler correct code would be something like: try: shutil.rmtree(dirname) except OSNoEntryError: pass Much simpler, no? And wrong. Right now, developers are tempted to write code like: shutil.rmtree(dirname, ignore_errors=True) Or: try: shutil.rmtree(dirname) except OSError: pass Both of which follow the error hiding anti-pattern [1]. [1] http://en.wikipedia.org/wiki/Error_hiding Thanks for reading this far. Thanks for not wasting any more of my time. OK, I won't waste more time. If this were an obvious improvement beyond doubt to most people, I would pursue it, but since it's not, I can live with it. Thanks anyway, -- Gustavo J. A. M. Carneiro INESC Porto, Telecommunications and Multimedia Unit The universe is always one step beyond logic. -- Frank Herbert ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] __length_hint__
Iterators can implement a method called __length_hint__ that provides a hint to certain internal routines (such as list.extend) so they can operate more efficiently. As far as I can tell, __length_hint__ is currently undocumented. Should it be? If so, are there any constraints on what an iterator should return? I can think of 3 possible rules, each with advantages and disadvantages: 1. return your best guess 2. return your best guess that you are certain is not higher than the true value 3. return your best guess that you are certain is not lower than the true value Also, I've noticed that if a VERY large hint is returned by the iterator, list.extend will sometimes disregard the hint and try to allocate memory incrementally (correct for rule #1 or #2). However, in another code path it will throw a MemoryError immediately based on the hint (correct for rule #3). -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC http://stutzbachenterprises.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __length_hint__
Iterators can implement a method called __length_hint__ that provides a hint to certain internal routines (such as list.extend) so they can operate more efficiently. As far as I can tell, __length_hint__ is currently undocumented. Should it be? This has been discussed, and no, it is a implementation detail mostly for the optimization of builtin iterators. Right. That matches my vague recollection on the subject. If so, are there any constraints on what an iterator should return? I can think of 3 possible rules, each with advantages and disadvantages: 1. return your best guess Yes. BTW, the same rule also applies to __len__. IIRC, Tim proposed to add that to the docs somewhere. Perhaps Raymond can shed some light on these. Can't guess the future of __length_hint__(). Since it doesn't have a slot, the attribute lookup can actually slow down cases with a small number of iterands. The original idea was based on some research on map/fold operations, noting that iterators can sometimes be processed more efficiently if accompanied by some metadata (i.e. the iterator has a known length, consists of unique items, is sorted, is all of a certain type, is re-iterable, etc.). Raymond ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] UnicodeDecodeError bug in distutils
Phillip J. Eby p...@telecommunity.com writes: However, there's currently no standard, as far as I know, for what encoding the PKG-INFO file should use. Who would define such a standard? My vote goes for “default is UTF-8”. Meanwhile, the 'register' command accepts Unicode, but is broken in handling it. […] Unfortunately, this isn't fixable until there's a new 2.5.x release. For previous Python versions, both register and write_pkg_info() accepted 8-bit strings and passed them on as-is, so the only workaround for this issue at the moment is to revert to Python 2.4 or less. What is the prognosis on this issue? It's still hitting me in Python 2.5.4. -- \ “Everything you read in newspapers is absolutely true, except | `\for that rare story of which you happen to have first-hand | _o__) knowledge.” —Erwin Knoll | Ben Finney ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
At 10:33 PM 4/2/2009 +0200, M.-A. Lemburg wrote: That's going to slow down Python package detection a lot - you'd replace an O(1) test with an O(n) scan. I thought about this too, but it's pretty trivial considering that the only time it takes effect is when you have a directory name that matches the name you're importing, and that it will only happen once for that directory, unless there is no package on sys.path with that name, and the program tries to import the package multiple times. In other words, the overhead isn't likely to be much, compared to the time needed to say, open and marshal even a trivial __init__.py file. Alternative Approach: - Wouldn't it be better to stick with a simpler approach and look for __pkg__.py files to detect namespace packages using that O(1) check ? I thought the same thing (or more precisely, a single .pkg file), but when I got lower in the PEP I saw the reason was to support system packages not having overlapping filenames. The PEP could probably be a little clearer about the connection between needing *.pkg and the system-package use case. One of the namespace packages, the defining namespace package, will have to include a __init__.py file. Note that there is no such thing as a defining namespace package -- namespace package contents are symmetrical peers. The above mechanism allows the same kind of flexibility we already have with the existing normal __init__.py mechanism. * It doesn't add yet another .pth-style sys.path extension (which are difficult to manage in installations). * It always uses the same naive sys.path search strategy. The strategy is not determined by some file contents. The above are also true for using only a '*' in .pkg files -- in that event there are no sys.path changes. (Frankly, I'm doubtful that anybody is using extend_path and .pkg files to begin with, so I'd be fine with a proposal that instead used something like '.nsp' files that didn't even need to be opened and read -- which would let the directory scan stop at the first .nsp file found. * The search is only done once - on the first import of the package. I believe the PEP does this as well, IIUC. * It's possible to have a defining package dir and add-one package dirs. Also possible in the PEP, although the __init__.py must be in the first such directory on sys.path. (However, such defining packages are not that common now, due to tool limitations.) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's update CObject API so it is safe and regular!
Hrvoje Niksic wrote: I thought the entire *point* of C object was that it's an opaque box without any info whatsoever, except that which is known and shared by its creator and its consumer. But there's no way of telling who created a given CObject, so *nobody* knows anything about it for certain. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's update CObject API so it is safe and regular!
Jim Fulton wrote: The original use case for CObjects was to export an API from a module, in which case, you'd be importing the API from the module. The presence in the module indicates the type. Sure, but it can't hurt to have an additional sanity check. Also, there are wider uses for CObjects than this. I see it as a quick way of creating a wrapper when you don't want to go to the trouble of a full-blown extension type. A small amount of metadata would make CObjects much more useful. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
Martin v. Löwis schrieb: I propose the following PEP for inclusion to Python 3.1. Please comment. Regards, Martin Abstract Namespace packages are a mechanism for splitting a single Python package across multiple directories on disk. In current Python versions, an algorithm to compute the packages __path__ must be formulated. With the enhancement proposed here, the import machinery itself will construct the list of directories that make up the package. +1 speaking as a downstream packaging python for Debian/Ubuntu I welcome this approach. The current practice of shipping the very same file (__init__.py) in different packages leads to conflicts for the installation of these packages (this is not specific to dpkg, but is true for rpm packaging as well). Current practice of packaging (for downstreams) so called name space packages is: - either to split out the namespace __init__.py into a separate (linux distribution) package (needing manual packaging effort for each name space package) - using downstream specific packaging techniques to handle conflicting files (diversions) - replicating the current behaviour of setuptools simply overwriting the file conflicts. Following this proposal (downstream) packaging of namespace packages is made possible independent of any manual downstream packaging decisions or any downstream specific packaging decisions. Matthias ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
At 03:21 AM 4/3/2009 +0200, Matthias Klose wrote: +1 speaking as a downstream packaging python for Debian/Ubuntu I welcome this approach. The current practice of shipping the very same file (__init__.py) in different packages leads to conflicts for the installation of these packages (this is not specific to dpkg, but is true for rpm packaging as well). Current practice of packaging (for downstreams) so called name space packages is: - either to split out the namespace __init__.py into a separate(linux distribution) package (needing manual packaging effort for eachname space package) - using downstream specific packaging techniques to handle conflicting files(diversions) - replicating the current behaviour of setuptools simply overwriting thefile conflicts. Following this proposal (downstream) packaging of namespace packages is made possible independent of any manual downstream packaging decisions or any downstream specific packaging decisions A clarification: setuptools does not currently install the __init__.py file when installing in --single-version-externally-managed or --root mode. Instead, it uses a project-version-nspkg.pth file that essentially simulates a variation of Martin's .pkg proposal, by abusing .pth file support. If this PEP is adopted, setuptools would replace its nspkg.pth file with a .pkg file on Python versions that provide native support for .pkg imports, keeping the .pth file only for older Pythons. (.egg files and directories will not be affected by the change, unless the zipimport module will also supports .pkg files... and again, only for Python versions that support the new approach.) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Should the io-c modules be put in their own directory?
Hello, I just noticed that the new io-c modules were merged in the py3k branch (I know, I am kind late on the news—blame school work). Anyway, I am just wondering if it would be a good idea to put the io-c modules in a sub-directory (like sqlite), instead of scattering them around in the Modules/ directory. Cheers, -- Alexandre ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Package Management - thoughts from the peanut gallery
Chris Withers writes: Personally I feel all of the above are perfectly possible, and can't see anyone being left unhappy by them. I'm sure I've missed something then, otherwise why not make it happen? Labor shortage. We will need a PEP, the PEP will need a sample implementation, and a proponent. Who's gonna bell the cat? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com