[issue15867] It's hard to decypher how to build off of the provided objects from the importlib docs

[Nick Coghlan]

Nick Coghlan added the comment:

One specific change we should make is that the "see also" at the start of the 
3.3 importlib docs should link to the new section of the language reference, 
rather than Guido's packaging essay. We can probably also cull that long list 
of PEPs, moving it to the end of the language reference section.

Other than that, yeah, we've been working away at this for years, trying to 
bring it down to a more manageable level of complexity. Brett's efforts in 
finally purging the last remnants of the old pre-PEP 302 import systems are 
what made it possible for Barry to finally create a coherent spec for the 
import system (previously, none of us really wanted to write such a thing, as 
there would have been *way* too many caveats needed in attempting to 
grandfather in the legacy import mechanics).

Now that that is done, 3.4 will likely include a number of improvements to 
importlib to make it easier to use as a basis for custom import hooks (as Brett 
and Eric noted, better support for customisation of the source -> bytecode 
compilation step will definitely be one of them.

As far as books go, I think the evolution of Python's import system might make 
an interesting entry if they do a third edition of The Architecture of Open 
Source Applications [1] :)

[1] http://www.aosabook.org/en/index.html

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15867] It's hard to decypher how to build off of the provided objects from the importlib docs

[Chris Jerdonek]

Changes by Chris Jerdonek :


--
nosy: +cjerdonek

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15867] It's hard to decypher how to build off of the provided objects from the importlib docs

[Brett Cannon]

Brett Cannon added the comment:

First off, what you want to do isn't easy to begin with. =) You are right that 
you want get_code() and that SourceLoader is what you want. The problem is that 
importlib inherited PEP 302s APIs, and there are so many that the docs don't 
repeat themselves in terms of what methods each ABC implements in order to keep 
the docs readable. That makes it a little difficult to realize what ABCs 
implement what without reading the class description and/or looking at the 
class hierarchy layout to realize that SourceLoader implements ResourceLoader 
which specifies get_code().

Second, import is just plain hard. It took me over 5 years to write importlib 
and get it to where it is now, and most of that work was just trying to keep it 
all straight in my head. This also makes writing an example or two difficult as 
it becomes a massive undertaking very quickly. And there is the simple issue 
that everyone wants something different, e.g. you want to transform source 
while others want an alternative back-end storage solution. That means coming 
up with the right examples is hard in and of itself.

Third, in Python 3.4 your desire to transform source will be much easier to 
achieve thanks to http://bugs.python.org/issue15627 .

IOW, I understand your pain but solving the problem is hard without writing a 
book on the subject (which who knows, maybe I'll do someday as a $1 ebook or 
something).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15867] It's hard to decypher how to build off of the provided objects from the importlib docs

[Eric Snow]

Eric Snow added the comment:

As far as the import system goes, Barry Warsaw added a really nice page to the 
language reference[1].  However, it sounds like your concern is with taking 
advantage of the tools that importlib provides.

First of all, a good thing to recognize is that importlib Python 3.3 exposes a 
much richer public API, including exposing the "default" import machinery.  
While importlib.abc is meaningful, importlib.machinery is more useful in your 
case.  In versions prior to 3.3, things aren't as easy.

In other Python 3 versions one approach you could take is to **carefully** use 
the private importlib APIs to get what you want.  If you do that, I'd recommend 
that your code target the public 3.3 APIs and then write a wrapper around the 
earlier private APIs to get them to be compatible.  There really shouldn't be a 
lot of difference.  The key is to target Python 3.3's importlib.

For Python 2, I'd still recommend targeting 3.3's importlib API and writing 
wrappers to make that work.  This would likely involve more effort to backport 
whole chunks of the 3.3 importlib implementation.  Better to simply port your 
code to Python 3.  :)

Secondly, the import system is a complex piece of machinery.  The new reference 
that Barry did helps pull it all together, but there are simply a lot of moving 
parts in various layers.  Adding examples for the importlib API may help for 
working with that API, but any activities in hooking into the import system 
should be done with a firm understanding of how it works since it's easy to 
break things.  Currently there isn't any easy way around that and I doubt any 
of that will change for a long time (due to the effort required).

Lastly, take a look at pylt[2].  It's a project I'm working on for 
source-to-source translation that hooks into the import system.  Though it 
isn't finished yet, the import side of things is mostly done.  Hopefully I'll 
have all the tests done in the next few days.

For pylt I've made use of the 3.3 importlib API along with a couple of things 
that we should see in 3.4 that I've "backported" [3].  The code there should 
give you an idea of how I've done essentially what you are looking to do.

Ultimately, any recommendations you can give on making the import system more 
approachable would be awesome.  Though it will take a lot of thought, 
discussion, and effort to make the import system "easy", there is still a lot 
of room for improvement in making it understandable.  Your perspective would be 
meaningful in working toward a sensible improvement.


[1] http://docs.python.org/dev/reference/import.html
[2] https://bitbucket.org/ericsnowcurrently/pylt/
[3] http://bugs.python.org/issue15627

--
nosy: +eric.snow

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15867] It's hard to decypher how to build off of the provided objects from the importlib docs

[Antoine Pitrou]

Changes by Antoine Pitrou :


--
nosy: +brett.cannon, ncoghlan

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15867] It's hard to decypher how to build off of the provided objects from the importlib docs

[Julian Berman]

Changes by Julian Berman :


--
type:  -> enhancement

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15867] It's hard to decypher how to build off of the provided objects from the importlib docs

[Julian Berman]

New submission from Julian Berman:

I find that the importlib documentation is a bit too low level. Import hooks 
are not that common, so requiring a bit of reading is OK with me, but I 
somewhat *understand* PEP 302, so I have a general idea of *what* I want to do 
and what kind of objects can accomplish that, and still find it hard to figure 
out how to take the pieces provided by importlib and use them to do what I want.

If I provide a specific example, I want to create a path hook that "just" does 
the usual import behavior, but before executing the module, does some 
transformation on the source code, say. In trying to figure out the best way to 
do that I had a hard time using the docs to figure out which pieces I should 
assemble to do that. I'm going to just describe the rough series of steps I 
went through, and hopefully that will help give a picture of where in the docs 
I had trouble.

`importlib.abc` has a few things that would appear to help. None of the things 
seem like an exact match, so immediately I'm confused -- after reading PEP 302 
I'd have expected to need to find an object to implement one or both of 
`get_code` or `get_source` on, or one that has that implemented that I can 
subclass and extend. The closest thing there that I find is PyPycLoader, which 
seems to be saying it implements the standard import behavior, but the docs say 
its deprecated and to use SourceLoader. At this point, after checking out 
`SourceLoader` and seeing that it has me implementing two methods whose purpose 
I don't quite understand, even after reading the short descriptions of them, at 
least not in the context of what I want to do, I begin to suspect that what I 
really want is to combine SourceLoader with some things from the `imp` module, 
or maybe `importlib.__import__`, but am left wondering again how much I need to 
implement before I just can use that. I then notice `importlib.
 util.module_for_loader`, and add that to the simple loader I've written which 
I'm still waiting to plug `imp` into, before realizing that that decorator eats 
the `fullname` attribute and *only* passes along the module, which confuses me, 
since now I don't know how to retrieve the source for the module object that 
I'm being passed -- so I save the path name in `__init__` for the class, and 
assume that's what I should be doing, despite not seeing an example doing that. 
Assuming that's even correct as-is, it took me quite a bit to put those pieces 
together.

So I apologize for rambling -- I think essentially what'd improve things is 
providing more examples, or perhaps a HOWTO entry, that targeted assembling the 
pieces provided in the module into a few clear, complete examples of finders, 
loaders and importers.

--
assignee: docs@python
components: Documentation
messages: 169901
nosy: Julian, docs@python
priority: normal
severity: normal
status: open
title: It's hard to decypher how to build off of the provided objects from the 
importlib docs
versions: Python 3.3, Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com