Re: [Python-Dev] Created branch for PEP 302 phase 2 work (in C)
On 10/3/06, Neal Norwitz [EMAIL PROTECTED] wrote: On 10/2/06, Brett Cannon [EMAIL PROTECTED] wrote: This is why I asked for input from people on which would take less time. Almost all the answers I got was that the the C code was delicate but that it was workable.Several people said they wished for a Python implementation, but hardly anyone said flat-out, don't waste your time, the Python version will be faster to do.I didn't respond mostly because I pushed this direction to begin with. That and I'm lazy. :-)But couldn't you be lazy in a timely fashion? There is a lot of string manipulation and some list manipulation thatis a royal pain in C and trivial in python.Caching will be mucheasier to experiement with in Python too.The Python version will bemuch smaller.It will take far less time to code it in Python and recode in C, than to try to get it right in C the first time.If thecode is fast enough, there's no reason to rewrite in C.It willprobably be easier to subclass a Python based version that a C basedversion. As for the bootstrapping, I am sure it is resolvable as well.There are several ways to go about it that are all tractable.Right, I had bootstrapping with implementing xrange in Python, but it was pretty easy to resolve in the end.You might even want to usepart of that patch (from pythonrun.c?).There was some re-org to makebootstrapping easier/possible (I don't remember exactly right now). OK, OK, I get the hint. I will rewrite import in Python and just make it my research work and personal project. Probably will do the initial pure Python stuff in the sandbox to really isolate it and then move it over to the pep302_phase2 branch when C code has to be changed. -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Created branch for PEP 302 phase 2 work (in C)
On 10/2/06, Brett Cannon [EMAIL PROTECTED] wrote: This is why I asked for input from people on which would take less time. Almost all the answers I got was that the the C code was delicate but that it was workable. Several people said they wished for a Python implementation, but hardly anyone said flat-out, don't waste your time, the Python version will be faster to do. I didn't respond mostly because I pushed this direction to begin with. That and I'm lazy. :-) There is a lot of string manipulation and some list manipulation that is a royal pain in C and trivial in python. Caching will be much easier to experiement with in Python too. The Python version will be much smaller. It will take far less time to code it in Python and recode in C, than to try to get it right in C the first time. If the code is fast enough, there's no reason to rewrite in C. It will probably be easier to subclass a Python based version that a C based version. As for the bootstrapping, I am sure it is resolvable as well. There are several ways to go about it that are all tractable. Right, I had bootstrapping with implementing xrange in Python, but it was pretty easy to resolve in the end. You might even want to use part of that patch (from pythonrun.c?). There was some re-org to make bootstrapping easier/possible (I don't remember exactly right now). n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Created branch for PEP 302 phase 2 work (in C)
At 01:01 PM 10/2/2006 -0700, Brett Cannon wrote: In the interest of time I have decided to go ahead and do the PEP 302 phase 2 work in C. Just FYI, it's not possible (so far as I know) to implement phase 2 while maintaining backward compatibility with existing 2.x code. So this work shouldn't go back to the 2.x trunk without discussion of those issues. Essentially, I abandoned trying to do the phase 2 work for Python 2.5 because there's too much code in the field that depends on the current order of when special/built-in imports are processed vs. when PEP 302 imports are processed. Thus, instead of adding new PEP 302 APIs (like get_loader) to 'imp', I added them to 'pkgutil'. There are, I believe, some notes in that module's source regarding what the ordering issues are w/meta_path vs. the way import works now. That having been said, we could possibly have a transition for 2.6, but everybody who's written any PEP 302 emulation code (outside of pkgutil itself) would have to adapt their code somewhat. I'm surprised, however, that you think working on this in C is going to be *less* time than it would take to simply replace __import__ with a Python function that reimplements PEP 302... especially since pkgutil contains a whole lot of the code you'd need, e.g.: def __import__(...): ... loader = pkgutil.find_loader(fullname) if loader is not None: module = loader.load_module(fullname) ... And much of the rest of the above can probably be filled out by swiping code from ihooks, imputil, or other Python __import__ implementations. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Created branch for PEP 302 phase 2 work (in C)
On 10/2/06, Phillip J. Eby [EMAIL PROTECTED] wrote: Just FYI, it's not possible (so far as I know) to implement phase 2 while maintaining backward compatibility with existing 2.x code. So this work shouldn't go back to the 2.x trunk without discussion of those issues. While that's a fair point, we need to be clear what compatibility issues there are. The built in import mechanisms aren't well documented, so it's not a black-and-white situation. An unqualified statement there are issues isn't much help on its own... Essentially, I abandoned trying to do the phase 2 work for Python 2.5 because there's too much code in the field that depends on the current order of when special/built-in imports are processed vs. when PEP 302 imports are processed. Can you say what that code is, and who we should be talking to to understand their issues? If not, how do we find such code? Presumably, you've got a lot of feedback through your work on setuptools/eggs - do you have a record of who might participate in a discussion? Thus, instead of adding new PEP 302 APIs (like get_loader) to 'imp', I added them to 'pkgutil'. How does that help? Where the code goes doesn't seem likely to make much difference... There are, I believe, some notes in that module's source regarding what the ordering issues are w/meta_path vs. the way import works now. The only notes I could see in pkgutil.py refer to special locations like the Windows registry, and refer to the fact that they will be searched after path entries, not before (for reasons I couldn't quite follow, but that's likely because I only read the comments fairly quickly). But if the whole mechanism is moved to sys.meta_path (which is what Phase 2 is about) surely it's possible to choose the ordering just by the order the importers go on sys.meta_path? That having been said, we could possibly have a transition for 2.6, but everybody who's written any PEP 302 emulation code (outside of pkgutil itself) would have to adapt their code somewhat. I don't really see how we're going to address that other than by implementing it, and waiting for people with issues to speak up. Highlighting the changes early is good, as it avoids a mid-beta rush of people suddenly finding issues, but I doubt we'll do much better than that. I'm surprised, however, that you think working on this in C is going to be *less* time than it would take to simply replace __import__ with a Python function that reimplements PEP 302... That I do agree with. There's a bootstrapping issue (you can't import the Python module that does all this without using a C-coded import mechanism) but that should be resolvable. especially since pkgutil contains a whole lot of the code you'd need, e.g.: Yes, I'm quite surprised at how much has appeared in pkgutil. The what's new entry is very terse, and the module documentation itself hasn't been updated to mention the new stuff. That's a shame, as it looks very useful (and as you say, could form a substantial part of this change if we were coding it in Python). Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Created branch for PEP 302 phase 2 work (in C)
On 10/2/06, Paul Moore [EMAIL PROTECTED] wrote: On 10/2/06, Phillip J. Eby [EMAIL PROTECTED] wrote:[SNIP] I'm surprised, however, that you think working on this in C is going to be *less* time than it would take to simply replace __import__ with a Python function that reimplements PEP 302...That I do agree with. There's a bootstrapping issue (you can't importthe Python module that does all this without using a C-coded importmechanism) but that should be resolvable. This is why I asked for input from people on which would take less time. Almost all the answers I got was that the the C code was delicate but that it was workable. Several people said they wished for a Python implementation, but hardly anyone said flat-out, don't waste your time, the Python version will be faster to do. As for the bootstrapping, I am sure it is resolvable as well. There are several ways to go about it that are all tractable.-Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Created branch for PEP 302 phase 2 work (in C)
At 03:48 PM 10/2/2006 -0700, Brett Cannon wrote: On 10/2/06, Paul Moore mailto:[EMAIL PROTECTED][EMAIL PROTECTED] wrote: On 10/2/06, Phillip J. Eby mailto:[EMAIL PROTECTED][EMAIL PROTECTED] wrote: [SNIP] I'm surprised, however, that you think working on this in C is going to be *less* time than it would take to simply replace __import__ with a Python function that reimplements PEP 302... That I do agree with. There's a bootstrapping issue (you can't import the Python module that does all this without using a C-coded import mechanism) but that should be resolvable. This is why I asked for input from people on which would take less time. Almost all the answers I got was that the the C code was delicate but that it was workable. Several people said they wished for a Python implementation, but hardly anyone said flat-out, don't waste your time, the Python version will be faster to do. As for the bootstrapping, I am sure it is resolvable as well. There are several ways to go about it that are all tractable. When I implemented the PEP 302 fix for the import speedups, I basically prototyped it using Python code that got loaded prior to 'site.py'. Once I had the Python version solid, I converted it to a C type via straightforward code transcription. That's pretty much the route I would follow for this too, although of course freezing the Python version into C code is also an option, since there's not much performance benefit to be had from a C translation, except for two parts of __import__: the part that checks sys.modules to shortcut the process, and the part that runs after the target module has been loaded or found. Aside from this fast path part of __import__, any additional interpretation overhead will probably be dwarfed by I/O considerations. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Created branch for PEP 302 phase 2 work (in C)
On Mon, Oct 02, 2006 at 11:27:07PM +0100, Paul Moore wrote: Yes, I'm quite surprised at how much has appeared in pkgutil. The what's new entry is very terse, and the module documentation itself hasn't been updated to mention the new stuff. These two things are related, of course; I couldn't figure out which bits of pkgutil.py are intended to be publicly used and which weren't. There's an __all__ in the module, but some things such as read_code() don't look like they're intended for external use. --amk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Created branch for PEP 302 phase 2 work (in C)
At 08:21 PM 10/2/2006 -0400, A.M. Kuchling wrote: On Mon, Oct 02, 2006 at 11:27:07PM +0100, Paul Moore wrote: Yes, I'm quite surprised at how much has appeared in pkgutil. The what's new entry is very terse, and the module documentation itself hasn't been updated to mention the new stuff. These two things are related, of course; I couldn't figure out which bits of pkgutil.py are intended to be publicly used and which weren't. There's an __all__ in the module, but some things such as read_code() don't look like they're intended for external use. The __all__ listing is correct; I intended to expose read_code() for the benefit of other importer implementations and Python utilities. Over the years, I've found myself writing the equivalent of read_code() several times, so it seemed to me to make sense to expose it as a utility function, since it already needed to be there for the ImpLoader class to work. In general, the idea behind the additions to pkgutil was to make life easier for people doing import-related operations, by being a Python reference implementation of commonly-reinvented parts of the import process. The '-m' machinery in 2.5 had a bunch of this stuff in it, and so did setuptools, so I yanked the code from both and refactored it to allow reuse by both, then fleshed it out to support all the optional PEP 302 loader protocols, and additional protocols needed to support tools like pydoc being able to run against arbitrary importers (esp. zip files). ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
