[issue28816] [doc] Clarify that zipimport does not invoke import hooks to load custom files from zip.

[Irit Katriel]

Change by Irit Katriel :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28816] [doc] Clarify that zipimport does not invoke import hooks to load custom files from zip.

[Irit Katriel]

Irit Katriel  added the comment:


New changeset 0194bbbee6a12264e93d3217c774e226f0a1737d by Miss Islington (bot) 
in branch '3.10':
bpo-28816: [doc] clarify that zipimport invokes importers only for python files 
(GH-30060) (GH-30133)
https://github.com/python/cpython/commit/0194bbbee6a12264e93d3217c774e226f0a1737d


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28816] [doc] Clarify that zipimport does not invoke import hooks to load custom files from zip.

[Irit Katriel]

Irit Katriel  added the comment:


New changeset 4c1effaaee472cc67f3186411a3df4f39af3d71a by Miss Islington (bot) 
in branch '3.9':
bpo-28816: [doc] clarify that zipimport invokes importers only for python files 
(GH-30060) (GH-30134)
https://github.com/python/cpython/commit/4c1effaaee472cc67f3186411a3df4f39af3d71a


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28816] [doc] Clarify that zipimport does not invoke import hooks to load custom files from zip.

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 7.0 -> 8.0
pull_requests: +28353
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/30133

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28816] [doc] Clarify that zipimport does not invoke import hooks to load custom files from zip.

[miss-islington]

Change by miss-islington :


--
pull_requests: +28354
pull_request: https://github.com/python/cpython/pull/30134

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28816] [doc] Clarify that zipimport does not invoke import hooks to load custom files from zip.

[Irit Katriel]

Irit Katriel  added the comment:


New changeset a951c95a13c3555ac8fb1c8ee615ba3930ccc6f7 by Irit Katriel in 
branch 'main':
bpo-28816: [doc] clarify that zipimport invokes importers only for python files 
(GH-30060)
https://github.com/python/cpython/commit/a951c95a13c3555ac8fb1c8ee615ba3930ccc6f7


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28816] [doc] Clarify that zipimport does not invoke import hooks to load custom files from zip.

[Irit Katriel]

Change by Irit Katriel :


--
pull_requests: +28280
status: pending -> open
pull_request: https://github.com/python/cpython/pull/30060

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28816] [doc] Clarify that zipimport does not invoke import hooks to load custom files from zip.

[Irit Katriel]

Irit Katriel  added the comment:

I think that
(1) the patch is not worded very clearly
(2) the docs should not reference py2exe for instructions how to do something

Perhaps it would suffice to change the sentence:

Any files may be present in the ZIP archive, but only files .py and .pyc are 
available for import

to:

Any files may be present in the ZIP archive, but the importer is only invoked 
for .py and .pyc files

?

--
keywords: +easy
nosy: +iritkatriel
status: open -> pending
title: Document if zipimport can respect import hooks to load custom files from 
zip. -> [doc] Clarify that zipimport does not invoke import hooks to load 
custom files from zip.
type:  -> enhancement
versions: +Python 3.10, Python 3.11, Python 3.9 -Python 3.4, Python 3.5, Python 
3.6, Python 3.7

___
Python tracker 
<https://bugs.python.org/issue28816>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[ANN] imphook - Simple and clear import hooks for Python - import anything as if it were a Python module

[Paul Sokolovsky]

Hello,

imphook is a simple, small single-file module and a tool which allows
to easily and selectively override Python import machinery for
particular file types (including Python source files).

As an example of usage, recently on the python-ideas mailing list,
there was a discussion about "arrow functions" for Python (as a
syntactic alternative to lambda). A prototype implementation using
"imphook" was posted:
https://mail.python.org/archives/list/python-id...@python.org/thread/VOVHZUNJKCVCIPPMA4EBUJ2CQ5FJJR2F/

The project is available from:

PyPI: https://pypi.org/project/imphook/ (package includes only the
module itself).

Github: https://github.com/pfalcon/python-imphook (assorted usage
examples are included in the git repository)



Below is a documentation snapshot:


imphook - Simple and clear import hooks for Python
==

The ``imphook`` module allows to easily define per file type import
hooks, i.e. overload or extend import processing for particular file
types, without affecting processing of other file types, and at the
same time, while ensuring that new processing integrates as seamlessly
as possible with normal Python import rules.

Besides the Python-level API to install import hooks, the module also
provides command-line interface to run an existing Python script or
module with one or more import hooks preloaded (i.e. without modifying
existing script source code).

Some but not all things you can easily do using ``imphook`` (most
of these require additional modules to do the heavy lifting,
``imphook`` just allows to plug it seamlessly into the Python import
system):

* Override importing of (all or some) .py files, to support new
  syntax or semantics in them.
* Import files written using a DSL (domain-specific language)
  as if they were Python modules. E.g., config or data files.
* Import modules written in other language(s), assuming you have
  an interpreter(s) for them.
* Import binary files, e.g. Java or LLVM bytecode.

``imphook`` works both with new, lightweight legacy-free Python
API, as promoted by the `Pycopy <https://github.com/pfalcon/pycopy>`_
Python dialect (the original source of the "easy import hooks" idea),
and CPython (the older, reference Python implementation), and with
other Python implementations which are CPython-compatible.

Quick Start
---

Make sure that you already installed ``imphook`` using::

pip3 install -U imphook

Below is a complete example of an import hook module to load
``key = value`` style config files::

import imphook

def hook(modname, filename):
with open(filename) as f:
# Create a module object which will be the result of import.
mod = type(imphook)(modname)
for l in f:
k, v = [x.strip() for x in l.split("=", 1)]
setattr(mod, k, v)
return mod

imphook.add_import_hook(hook, (".conf",))

Save this as the ``mod_conf.py`` file, and add the two following
files to test it:

``example_settings.conf``::

var1 = 123
var2 = hello

``example_conf.py``::

import example_settings as settings

print(settings.var1)
print(settings.var2)

Now run::

python3 -m imphook -i mod_conf example_conf.py

As you can see, the ``example_conf.py`` is able to import
``example_settings.conf`` as if it were a normal Python module.

Besides copy-pasting the above and other examples, you can also
clone the Git repository of ``imphook``, which contains various
ready-to-use examples::

git clone https://github.com/pfalcon/python-imphook


API to install hooks and hook structure
---

The API of the module consists of one function:
`imphook.add_import_hook(hook, ext_tuple)`. *hook* is a name of
hook function. *ext_tuple* is a tuple of file extensions
the hook function should handle (the leading dot should be included).
More often than not, you will want to handle just one extension,
so don't forget to use the usual Python syntax with a trailing
comma for 1-element tuple, e.g.: ``(".ext",)``. Python modules may
not contain a dot (``"."``) in their names (they are used to separate
subpackages), so the extension you register may contain multiple
dots, e.g. ``".foo.bar"``, with filename ``my_module.foo.bar``
matching it.

It is possible to call `imphook.add_import_hook(hook, ext_tuple)`
multiple times to install multiple hooks. The hooks are installed
in the stack-like fashion, the last installed will be called
first. It is possible to install multiple hooks for the same file
extension, and earlier installed hooks may still be called in this
case, because a hook function may skip processing a particular
file, and let other hooks to take a chance, with default processing
happening if no hook handled the import.

The signature and template of the actual hook function is::

def m

[issue38721] modulefinder should use import hooks properly

[Dominic Littlewood]

Dominic Littlewood <11dlittlew...@gmail.com> added the comment:

In regards to the "unnecessary refactoring", here's a full list of what I've 
changed.

1. Previously, the module revolved around passing files and file descriptors 
around. Now, it uses specs. This means all the functions that used file 
descriptors have to be updated.

2. Due to the previous issue, several functions had to change their call 
signatures. The functions that called them had to change to react.

3. Packages work differently now. They do not have to have an __init__ 
submodule any more, so there's no need for a load_package function - it's 
integrated with load_module.

4. determine_parent was having trouble finding where the relative imports were 
relative to. I changed it to use _calc___package__, same as the import system. 
Half of determine_parent was deleted, because I couldn't understand how it 
worked and eventually worked out that there was literally no way it could ever 
run, so it didn't matter.

5. Modules with no __path__ attribute are not packages. Modules with a __path__ 
attribute of None *are* packages. Code had to be changed to reflect that. 
Looking back, this is possibly a pedantic enough point that these changes could 
be removed and hopefully nothing will go wrong.

6. fix_sys was used to update sys.path (although I have since changed it so it 
does not) and sys.modules. Code had to be changed to make use of it.

7. The only other thing I can think of is that I changed which modules were 
imported in the first few lines of modulefinder, since some had become 
unnecessary. (In fact types and warnings were already unnecessary, but never 
mind about that.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38721] modulefinder should use import hooks properly

[Dominic Littlewood]

Dominic Littlewood <11dlittlew...@gmail.com> added the comment:

You are correct about modulefinder being static. I read "run_script" and 
thought it was running a script, but it turns out I was being silly.

I shall correct the sys.path issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38721] modulefinder should use import hooks properly

[Brandt Bucher]

Brandt Bucher  added the comment:

See prior discussion on this here:

https://github.com/python/cpython/pull/11787#discussion_r256442282

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38721] modulefinder should use import hooks properly

[Brandt Bucher]

Brandt Bucher  added the comment:

I'm not sure what you mean when you say "modulefinder currently will detect 
modules imported by a script dynamically by running the script"... modulefinder 
is completely static, no?

I also think that changing sys.path like this is a bad idea (I tried this in an 
earlier PR and was told to remove it and rework my solution). We lose 
thread-safety, and it has effects far outside of the scope of modulefinder 
(i.e. the hooks we call could cache it or something, and break later imports). 
If changing sys.path is the only way to accomplish this (still not exactly sure 
what), I doubt it will be accepted. It seems like this PR does a lot of 
unnecessary refactoring too, on first glance.

With that said, adding support for hooks should be fairly straightforward by 
just manually walking down sys.path/meta_path/path_hooks in _find_module... 
maybe.

--
nosy: +brandtbucher

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38721] modulefinder should use import hooks properly

[Dominic Littlewood]

Change by Dominic Littlewood <11dlittlew...@gmail.com>:


--
keywords: +patch
pull_requests: +16813
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/17326

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38721] modulefinder should use import hooks properly

[Dominic Littlewood]

Dominic Littlewood <11dlittlew...@gmail.com> added the comment:

I now have a PR which - dare I say it - appears to be working as it should. 
I'll just write some tests for it and then I'll send it off for review.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38721] modulefinder should use import hooks properly

[Dominic Littlewood]

Dominic Littlewood <11dlittlew...@gmail.com> added the comment:

Okay, I've encountered a snag. To use meta path finders, if the path is None, 
it's sometimes okay to substitute self.path, and sometimes not. The only way to 
make absolutely sure the finder works correctly is to change sys.path. To use 
namespace packages (and possibly some other types of modules finders might 
produce), it's also necessary to adjust sys.modules. However, both of these 
things may cause problems if a hook wants to import something in the standard 
way so it can run properly. This will require some thought.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38721] modulefinder should use import hooks properly

[Dominic Littlewood]

Dominic Littlewood <11dlittlew...@gmail.com> added the comment:

Okay, I've started putting together a proper PR, and I've had some thoughts.


There's a useful script at the bottom of the importlib documentation that 
readers should consult. This can be used to correctly find the spec for a 
module, and therefore the loader. (AddPackagePath and ReplacePackage are not 
needed, and should be deprecated.) For modules already on sys.path, the loader 
can be identified from module.__loader__.

If the loader is an InspectLoader, the code can be retrieved and examined to 
see what is imported. (Remember to check whether None is returned.) If not, we 
will assume nothing is imported - which is what modulefinder currently does 
with extension modules.

Since run_script takes a file path rather than a module name, it will have to 
find the appropriate loader for the task. This will be a simple choice between 
SourceFileLoader and SourcelessFileLoader, depending on what kind of file is 
being used.


It is also possible to use functions in importlib._bootstrap instead.
This enhancement would indirectly cause open_code to be used properly, so the 
bug I originally posted is redundant.

--
title: modulefinder should use io.open_code() instead of open() -> modulefinder 
should use import hooks properly
type: security -> enhancement

___
Python tracker 
<https://bugs.python.org/issue38721>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35486] subprocess module import hooks breaks back compatibility

[Nick Coghlan]

Nick Coghlan  added the comment:

Aye, we can :)

--
assignee:  -> docs@python
components: +Documentation -Library (Lib)
nosy: +docs@python
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35486] subprocess module import hooks breaks back compatibility

[Ned Deily]

Ned Deily  added the comment:

I've merged the doc changes for 3.6, thanks.  Can we close this now?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35486] subprocess module import hooks breaks back compatibility

[Ned Deily]

Ned Deily  added the comment:


New changeset 1edb3dc6ff70db88a7e89586578e58a86ee0e75e by Ned Deily (Miss 
Islington (bot)) in branch '3.6':
bpo-35486: Note Py3.6 import system API requirement change (GH-11540) (GH-11588)
https://github.com/python/cpython/commit/1edb3dc6ff70db88a7e89586578e58a86ee0e75e


--
nosy: +ned.deily

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35486] subprocess module import hooks breaks back compatibility

[miss-islington]

miss-islington  added the comment:


New changeset 422db3777874f4f31fc8f4e718f440a2abc59347 by Miss Islington (bot) 
in branch '3.7':
bpo-35486: Note Py3.6 import system API requirement change (GH-11540)
https://github.com/python/cpython/commit/422db3777874f4f31fc8f4e718f440a2abc59347


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35486] subprocess module import hooks breaks back compatibility

[miss-islington]

Change by miss-islington :


--
pull_requests: +11279, 11280, 11281, 11282, 11283

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35486] subprocess module import hooks breaks back compatibility

[miss-islington]

Change by miss-islington :


--
pull_requests: +11278, 11279, 11280

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35486] subprocess module import hooks breaks back compatibility

[miss-islington]

Change by miss-islington :


--
pull_requests: +11279, 11280, 11281, 11283

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35486] subprocess module import hooks breaks back compatibility

[miss-islington]

Change by miss-islington :


--
pull_requests: +11279, 11280, 11283

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35486] subprocess module import hooks breaks back compatibility

[miss-islington]

miss-islington  added the comment:


New changeset cee29b46a19116261b083dc803217aa754c7df40 by Miss Islington (bot) 
(Nick Coghlan) in branch 'master':
bpo-35486: Note Py3.6 import system API requirement change (GH-11540)
https://github.com/python/cpython/commit/cee29b46a19116261b083dc803217aa754c7df40


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35486] subprocess module import hooks breaks back compatibility

[miss-islington]

Change by miss-islington :


--
pull_requests: +11278, 11279

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35486] subprocess module import hooks breaks back compatibility

[miss-islington]

Change by miss-islington :


--
pull_requests: +11278

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35486] subprocess module import hooks breaks back compatibility

[Nick Coghlan]

Nick Coghlan  added the comment:

dw: we routinely impose new requirements on folks modifying runtime internals 
in new feature releases, so the only aspect we missed for this changing is to 
explicitly call it out in the Porting section of the Python 3.6 What's New 
document as a potential forward compatibility risk for import system 
replacement authors that don't update their plugins accordingly.

For folks implementing PEP 302 finders and loaders, no change is required, as 
they indicate failure to find a module by returning None, not by raising 
ImportError.

It's only folks emulating the full import system by replacing `__import__` that 
should need to worry about this.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35486] subprocess module import hooks breaks back compatibility

[Nick Coghlan]

Change by Nick Coghlan :


--
keywords: +patch, patch
pull_requests: +11147, 11148
stage: test needed -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35486] subprocess module import hooks breaks back compatibility

[Nick Coghlan]

Change by Nick Coghlan :


--
keywords: +patch, patch, patch
pull_requests: +11147, 11148, 11149
stage: test needed -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35486] subprocess module import hooks breaks back compatibility

[Nick Coghlan]

Change by Nick Coghlan :


--
keywords: +patch
pull_requests: +11147
stage: test needed -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35486] subprocess module import hooks breaks back compatibility

[David Wilson]

David Wilson  added the comment:

Hi Nick,

The purpose of ModuleNotFoundError is clear and unrelated to the problem 
documented here. The problem is that due to the introduction of 
ModuleNotFoundError, ImportError's semantics have been changed within a minor 
release in a breaking, backwards-incompatible manner.

As of Python 3.5, ImportError meant both "ImportError" and 
"ModuleNotFoundError". As of 3.6, the senses are distinct, and thus code 
written against ImportError as it existed in 3.5 no longer works correctly as 
of 3.6.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35486] subprocess module import hooks breaks back compatibility

[Nick Coghlan]

Nick Coghlan  added the comment:

Note that the above distinction is also the rationale for introducing the new 
subtype: so that it's easy to tell the difference between "that module was not 
found at all" (ModuleNotFoundError) and "the module was found, but attempting 
to actually load it failed" (other cases of ImportError)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35486] subprocess module import hooks breaks back compatibility

[Nick Coghlan]

Nick Coghlan  added the comment:

The two errors mean different things: ModuleNotFoundError means literally that 
the module could not be found at all (i.e. no import hook offered to try to 
load it)

A plain ImportError then means that the module was located, but attempting to 
actually load it failed.

find_spec()/find_loader()/find_module() implementations on import plugins 
shouldn't be raising exceptions for modules they don't offer, and hence 
shouldn't be needing to raise ModuleNotFoundError directly.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35486] subprocess module import hooks breaks back compatibility

[Brett Cannon]

Brett Cannon  added the comment:

RE: "PEP-302 and PEP-451 are the definitive specifications for how 
sys.meta_path is supposed to work"

That's actually not true. In the case of import the language reference is 
considered the reference for import: 
https://docs.python.org/3/reference/import.html (and true for PEPs in general 
once they are implemented).

--
nosy: +davin

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35486] subprocess module import hooks breaks back compatibility

[Terry J. Reedy]

Change by Terry J. Reedy :


--
nosy: +brett.cannon, eric.snow, ncoghlan
stage:  -> test needed
title: subprocess module breaks backwards compatibility with import hooks -> 
subprocess module import hooks breaks back compatibility
type:  -> behavior

___
Python tracker 
<https://bugs.python.org/issue35486>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35486] subprocess module breaks backwards compatibility with import hooks

[David Wilson]

David Wilson  added the comment:

Having considered this for a few hours, it seems the following is clear:

- 3.6 introduces ModuleImportError
- 3.7 begins using it within importlib
- 3.8 first instance of stdlib code catching it

- PEP-302 and PEP-451 are the definitive specifications for how sys.meta_path 
is supposed to work, and neither makes mention of ModuleNotFoundError. It seems 
at least this should have been flagged during review of the original change, 
but apparently the name of the exception was more important.

- The recent work done to move most of the import machinery on to sys.meta_path 
has exposed a set of import hooks that don't quite comply to the documented 
import hook interface

- The newly advertised ModuleNotFoundError appearing in stack traces for 
everyone means that more people will continue to write new cases of "except 
ModuleNotFoundError:", which while following best practice (catch most specific 
relevant exception), at present it amounts to relying on an implementation 
detail of the default importer. GitHub search reveals this to be an accurate 
reading: https://github.com/search?q=%22except+ModuleNotFoundError%22=Code

Therefore we are in a position where:

- Natural developer habit will cause much more protocol-violating code to exist 
over time, there is no option to stop this process
- New import hooks written against the existing documentation will break in the 
face of developer habit
- Existing import hooks were broken in Python 3.6 and this is not documented 
anywhere
- Python's own import machinery contravenes its specification

Options:

- Updating PEP-302 to mention introduction of the new exception type
- Updating ModuleNotFoundError/ImportError documentation to somehow capture the 
compatibility issue

--

___
Python tracker 
<https://bugs.python.org/issue35486>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35486] subprocess module breaks backwards compatibility with import hooks

[David Wilson]

Change by David Wilson :


--
title: subprocess module breaks backwards compatibility with older import hooks 
-> subprocess module breaks backwards compatibility with import hooks

___
Python tracker 
<https://bugs.python.org/issue35486>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35486] subprocess module breaks backwards compatibility with older import hooks

[David Wilson]

David Wilson  added the comment:

Corrected GitHub link for the commit: 
https://github.com/python/cpython/commit/880d42a3b24

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35486] subprocess module breaks backwards compatibility with older import hooks

[David Wilson]

New submission from David Wilson :

The subprocess package since 880d42a3b24 / September 2018 has begun using this 
idiom:

try:
import _foo
except ModuleNotFoundError:
bar

However, ModuleNotFoundError may not be thrown by older import hook 
implementations, since that subclass was only introduced in Python 3.6 -- and 
so the test above will fail. PEP-302 continues to document ImportError as the 
appropriate exception that should be raised.

https://mitogen.readthedocs.io/en/stable/ is one such import hook that lazily 
loads packages over the network when they aren't available locally. Current 
Python subprocess master breaks with Mitogen because when it discovers the 
master cannot satisfy the import, it throws ImportError.

The new exception subtype was introduced in https://bugs.python.org/issue15767 
, however very little in the way of rationale was included, and so it's unclear 
to me what need the new subtype is addressing, whether this is a problem with 
the subprocess module or the subtype as a whole, or indeed whether any of this 
should be considered a bug.

It seems clear that some kind of regression is in the process of occurring 
during a minor release, and it also seems clear the new subtype will 
potentially spawn a whole variety of similar new regressions.

I will be updating Mitogen to throw the new subtype if it is available, but I 
felt it was important to report the regression to see what others think.

--
components: Library (Lib)
messages: 331774
nosy: dw
priority: normal
severity: normal
status: open
title: subprocess module breaks backwards compatibility with older import hooks
versions: Python 3.8

___
Python tracker 
<https://bugs.python.org/issue35486>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28816] Document if zipimport can respect import hooks to load custom files from zip.

[Decorater]

Decorater added the comment:

OK, Well I just tested and it sadly don't support import hooks that adds 
support for importing custom file types or file types python does know about 
with a uncommon extension inside of zip files which is somewhat sad. However if 
someone was to do something similar to py2exe's import hook allowing memory 
loading pyd and so files they could allow it to work for their import hook's 
needs.

--
keywords: +patch
Added file: http://bugs.python.org/file46090/zipimport.rst.patch

___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue28816>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28816] Document if zipimport can respect import hooks to load custom files from zip.

[Brett Cannon]

Brett Cannon added the comment:

The cutoff for 3.6.0 has passed, but documentation patches can still go in for 
3.6.1 (and 3.5.3).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28816] Document if zipimport can respect import hooks to load custom files from zip.

[Decorater]

Decorater added the comment:

Well, Will there be any documentations for this before 3.6 final release or no 
because this would be nice to know for other people who would like to do 
something similar.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28816] Document if zipimport can respect import hooks to load custom files from zip.

[Decorater]

Changes by Decorater :


--
nosy: +brett.cannon, eric.snow, ncoghlan

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28816] Document if zipimport can respect import hooks to load custom files from zip.

[Decorater]

Changes by Decorater :


--
nosy: +twouters

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28816] Document if zipimport can respect import hooks to load custom files from zip.

[Decorater]

Changes by Decorater :


--
nosy:  -brett.cannon, eric.snow, ncoghlan, paul.moore, tim.golden, twouters, 
zach.ware

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28816] Document if zipimport can respect import hooks to load custom files from zip.

[Steve Dower]

Changes by Steve Dower :


--
nosy: +brett.cannon, eric.snow, ncoghlan, twouters -steve.dower

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28816] Document if zipimport can respect import hooks to load custom files from zip.

[Decorater]

Changes by Decorater :


--
components:  -Windows

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28816] Document if zipimport can respect import hooks to load custom files from zip.

[Decorater]

Changes by Decorater :


--
assignee:  -> docs@python
components: +Documentation -Interpreter Core
nosy: +docs@python

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28816] Document if zipimport can respect import hooks to load custom files from zip.

[Decorater]

New submission from Decorater:

I am wondering so lets say for example if I was to make a json import hook 
(code below):

from importlib.machinery import FileFinder
import json
import sys


class ExtensionImporter(object):
"""Base Importer Class."""
def __init__(self, extension_list):
self.extensions = extension_list

def find_loader(self, name, path):
"""Filds some loaders for importing the module."""
self.path = path
return self

def load_module(self, fullname):
if fullname in sys.modules:
return sys.modules[fullname]
return None


class JsonImporter(ExtensionImporter):
"""JSON importer Class."""
def __init__(self):
super(JsonImporter, self).__init__(['.json'])

def load_module(self, fullname):
"""Loads modules found with the extension"""
premodule = super(JsonImporter, self).load_module(fullname)
if premodule is None:
 with open(self.path, 'r') as f:
module = json.load(f)
sys.modules[fullname] = module
return module
raise ImportError('Couldn't open path')

extension_importers = [JsonImporter()]
hook_list = []
for importer in extension_importers:
hook_list.append((importer.find_loader, importer.extensions))

sys.path_hooks.insert(0, FileFinder.path_hook(*hook_list))
sys.path_importer_cache.clear()


What if I had those json files in a zip file and used ZipImport on that zip 
file would it use that import hook that I shared above if all other import 
methods fail to import those json files (obvously they would fail though)?

There should be documentation to explain if it supports user created import 
hooks that they create using importlib.

This is because I would be very happy if zipimport supports my own import hook 
if any other method of importing files from that zip file fails but mine ends 
up working. It also allows people to load up many other formats (provided they 
know the format) as if it was a python script. (of if they compressed it in a 
custom file like for example a RAR file that WinRAR can create. And yes I would 
support a RARImport for rar files.

--
components: Interpreter Core, Windows
messages: 281850
nosy: Decorater, paul.moore, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: Document if zipimport can respect import hooks to load custom files from 
zip.
versions: Python 3.4, Python 3.5, Python 3.6, Python 3.7

___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue28816>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23773] importlib does not properly remove frames when invoking external import hooks

[Brett Cannon]

Changes by Brett Cannon :


--
assignee: brett.cannon -> 

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Python Import Hooks and __main__

[Sven R. Kunze]

Hi,

following up on this thread on StackOverflow 
http://stackoverflow.com/questions/16515347/python-import-hooks-and-main 
does somebody has a great idea how to manage this?


The issue at hand is, that I would like to apply a specific import hook 
right from the beginning of the interpreter run (here, a simple test 
case), i.e. also affecting the import/exec of the module __main__.


Hook: https://github.com/srkunze/fork/blob/2e7ecd4b0a/fork.py#L429
Dirty Magic to get things running: 
https://github.com/srkunze/fork/blob/2e7ecd4b0a/fork.py#L425


Best,
Sven
--
https://mail.python.org/mailman/listinfo/python-list

[issue23773] importlib does not properly remove frames when invoking external import hooks

[Brett Cannon]

Brett Cannon added the comment:

Did you happen to run the test suite with this change, Shiz? There's actually a 
test to make sure that the part of the traceback you're avoiding isn't lost in 
case there's a bug in importlib itself.

Can you show that your change won't actually affect finding errors in importlib 
itself?

--
nosy: +eric.snow, ncoghlan

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23773
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23773] importlib does not properly remove frames when invoking external import hooks

[Raúl Cumplido]

Changes by Raúl Cumplido raulcumpl...@gmail.com:


--
nosy: +raulcd

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23773
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23773] importlib does not properly remove frames when invoking external import hooks

[Shiz]

Shiz added the comment:

Ah, that is a very good point. I'll run the suite as soon as I find the time 
and will report back.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23773
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23773] importlib does not properly remove frames when invoking external import hooks

[Ned Deily]

Changes by Ned Deily n...@acm.org:


--
nosy: +brett.cannon
stage:  - patch review

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23773
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23773] importlib does not properly remove frames when invoking external import hooks

[Brett Cannon]

Changes by Brett Cannon br...@python.org:


--
assignee:  - brett.cannon

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23773
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23773] importlib does not properly remove frames when invoking external import hooks

[Shiz]

New submission from Shiz:

When adding a custom module loader to sys.meta_path, importlib does not 
properly remove its frames before invoking it. This results in a weird 
traceback with importlib._bootstrap frames in if an error occurs during 
load_module(), like such:

Traceback (most recent call last):
  File /Users/mark/Development/Projects/rave/rave/bootstrap/__init__.py, line 
102, in bootstrap_game
__import__(MODULE_PACKAGE + '.' + mod.replace('.py', ''))
  File frozen importlib._bootstrap, line 2237, in _find_and_load
  File frozen importlib._bootstrap, line 2226, in _find_and_load_unlocked
  File frozen importlib._bootstrap, line 1200, in _load_unlocked
  File frozen importlib._bootstrap, line 1129, in _exec
  File /Users/mark/Development/Projects/rave/rave/modularity.py, line 23, in 
exec_module
super().exec_module(module)
  File /.modules/opengl/__init__.py, line 1, in module
from . import common, core3
  File frozen importlib._bootstrap, line 2237, in _find_and_load
  File frozen importlib._bootstrap, line 2226, in _find_and_load_unlocked
  File frozen importlib._bootstrap, line 1200, in _load_unlocked
  File frozen importlib._bootstrap, line 1129, in _exec
  File /Users/mark/Development/Projects/rave/rave/modularity.py, line 23, in 
exec_module
super().exec_module(module)
  File /.modules/opengl/core3/__init__.py, line 11, in module
from .texture import Texture, Image
  File frozen importlib._bootstrap, line 2237, in _find_and_load
  File frozen importlib._bootstrap, line 2226, in _find_and_load_unlocked
  File frozen importlib._bootstrap, line 1200, in _load_unlocked
  File frozen importlib._bootstrap, line 1129, in _exec
  File /Users/mark/Development/Projects/rave/rave/modularity.py, line 23, in 
exec_module
super().exec_module(module)
  File /.modules/opengl/core3/texture.py, line 8, in module
raise ValueError

Attached is a patch against the current hg head that makes sure module loaders 
are invoked through _call_with_frames_removed, so that the output becomes a lot 
more readable:

Traceback (most recent call last):
  File /Users/mark/Development/Projects/rave/rave/bootstrap/__init__.py, line 
102, in bootstrap_game
__import__(MODULE_PACKAGE + '.' + mod.replace('.py', ''))
  File /Users/mark/Development/Projects/rave/rave/modularity.py, line 23, in 
exec_module
super().exec_module(module)
  File /.modules/opengl/__init__.py, line 1, in module
from . import common, core3
  File /Users/mark/Development/Projects/rave/rave/modularity.py, line 23, in 
exec_module
super().exec_module(module)
  File /.modules/opengl/core3/__init__.py, line 11, in module
from .texture import Texture, Image
  File /Users/mark/Development/Projects/rave/rave/modularity.py, line 23, in 
exec_module
super().exec_module(module)
  File /.modules/opengl/core3/texture.py, line 8, in module
raise ValueError
ValueError

Ideally it would remove the calls /within/ the module loader itself too, but I 
do not see an easy way to do this without changing the entire 
_call_with_frames_removed semantics. This fixes most of the issues, though.

--
components: Library (Lib)
files: cpython-3ac58de829ef-fix-external-module-loader-call-traceback.patch
keywords: patch
messages: 239219
nosy: shiz
priority: normal
severity: normal
status: open
title: importlib does not properly remove frames when invoking external import 
hooks
type: behavior
versions: Python 3.4, Python 3.5
Added file: 
http://bugs.python.org/file38681/cpython-3ac58de829ef-fix-external-module-loader-call-traceback.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23773
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Question about import hooks

[Ed Schofield]

Hi all, 

I am the author of the ``future`` package for Python 2/3 compatibility 
(http://python-future.org). A bug report has recently been posted about its use 
of import hooks that I don't yet have an answer for, and I am looking for some 
guidance on how to customize the import mechanism in a safer way.

The current interface is as follows:

 from future import standard_library

Any subsequent import statements using Python 3-style module names are mapped 
onto the relevant Python 2-style names (or, where needed, backported modules 
provided by ``future``). For example, these then work in the same way on both 
Python 2 and Python 3:

 from http.client import HttpConnection
 import html.parser
 import queue

 import configparser

Although this is a nice interface, reminiscent of ``from __future__ import 
...``, the problem is that the current implementation, which appends finder 
objects to the ``sys.meta_path`` list 
(http://docs.python.org/2/library/sys.html#sys.meta_path) renders the import 
hooks globally, even for modules imported by other modules. What I want instead 
is for the import hooks to apply only to a particular module, so that a script 
containing:

from future import standard_library
import requests

would not apply the import hooks to modules imported within the ``requests`` 
module, merely to import statements in the script itself.

There is a note in the Python 3.3 documentation (and the current Python 3.4 
draft) that I had hoped would provide the answer for how to implement this:

When calling __import__() 
(http://docs.python.org/3/library/functions.html#__import__)
as part of an import statement, the import system first checks the module 
global namespace for a function by that name. If it is not found, then the 
standard builtin __import__() 
(http://docs.python.org/3/library/functions.html#__import__)
is called.


If this were true, it would be possible to change the interface to something 
like this:

 from future.standard_library import __import__

which would then override the ``__import__`` function in ``builtins`` or 
``__builtin__`` affecting subsequent ``import`` statements only in that module. 
The interface wouldn't be quite as nice, but it wouldn't cause the import hooks 
to bleed into other modules that don't need them. However, the docs seem to be 
wrong; defining __import__ as a global seems to have no effect on imports in 
Py3.3, and ``future`` needs to implement this for Python 2 anyway.

Can you think of a way to implement import hooks safely, for one particular 
module, while providing a nice clean interface? Preferably this would remain 
accessible through a one-line import like ``from future import 
standard_library``.

Thanks in advance for any ideas!

Best wishes,
Ed


-- 
Dr. Edward Schofield
(M) +61 (0)405 676 229
Python Charmers
http://pythoncharmers.com


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Question about import hooks

[Mark Lawrence]

On 23/11/2013 12:23, Ed Schofield wrote:

Hi all,

I am the author of the ``future`` package for Python 2/3 compatibility 
(http://python-future.org). A bug report has recently been posted about its use 
of import hooks that I don't yet have an answer for, and I am looking for some 
guidance on how to customize the import mechanism in a safer way.

The current interface is as follows:


from future import standard_library


Any subsequent import statements using Python 3-style module names are mapped 
onto the relevant Python 2-style names (or, where needed, backported modules 
provided by ``future``). For example, these then work in the same way on both 
Python 2 and Python 3:


from http.client import HttpConnection
import html.parser
import queue



import configparser


Although this is a nice interface, reminiscent of ``from __future__ import 
...``, the problem is that the current implementation, which appends finder 
objects to the ``sys.meta_path`` list 
(http://docs.python.org/2/library/sys.html#sys.meta_path) renders the import 
hooks globally, even for modules imported by other modules. What I want instead 
is for the import hooks to apply only to a particular module, so that a script 
containing:

 from future import standard_library
 import requests

would not apply the import hooks to modules imported within the ``requests`` 
module, merely to import statements in the script itself.

There is a note in the Python 3.3 documentation (and the current Python 3.4 
draft) that I had hoped would provide the answer for how to implement this:

When calling __import__() 
(http://docs.python.org/3/library/functions.html#__import__)
as part of an import statement, the import system first checks the module 
global namespace for a function by that name. If it is not found, then the 
standard builtin __import__() 
(http://docs.python.org/3/library/functions.html#__import__)
is called.


If this were true, it would be possible to change the interface to something 
like this:


from future.standard_library import __import__


which would then override the ``__import__`` function in ``builtins`` or 
``__builtin__`` affecting subsequent ``import`` statements only in that module. 
The interface wouldn't be quite as nice, but it wouldn't cause the import hooks 
to bleed into other modules that don't need them. However, the docs seem to be 
wrong; defining __import__ as a global seems to have no effect on imports in 
Py3.3, and ``future`` needs to implement this for Python 2 anyway.

Can you think of a way to implement import hooks safely, for one particular 
module, while providing a nice clean interface? Preferably this would remain 
accessible through a one-line import like ``from future import 
standard_library``.

Thanks in advance for any ideas!

Best wishes,
 Ed




I've no idea if this http://www.python.org/dev/peps/pep-0451/ is 
relevent to your needs but thought I'd point it out anyway.  It has been 
implemented in Python 3.4 beta.


--
Python is the second best programming language in the world.
But the best has yet to be invented.  Christian Tismer

Mark Lawrence

--
https://mail.python.org/mailman/listinfo/python-list

Dealing with ImportLock deadlock in Import Hooks

[Arnaud Fontaine]

Hello,

I'm currently working on implementing Import Hooks (PEP302) with Python
2.7 to be able to import modules whose code is in ZODB. However, I have
stumbled upon a widely known issue about import deadlock[0][1] (note
that this issue is not directly related to ZODB, but a more general
question about dealing with ImportLock deadlock for Import Hooks),
basically:

  Thread 1 is trying to import a module 'foo.bar' (where 'foo' is a
  package containing dynamic modules) handled by Import Hooks I
  implemented, so ImportLock is acquired before even running the hooks
  (Python/import.c:PyImport_ImportModuleLevel()). Then, these import
  hooks try to load objects from ZODB and a request is sent and handled
  by another thread (Thread 2) which itself tries to import another
  module. Of course, this causes a deadlock because the first thread
  still holds ImportLock.

I have thought about the following solutions:

* Backport the patch applied in python 3.3 from issue 9260[0]. This
  would be the best option because it would mean that even when trying
  to import any module from package 'foo', other modules and packages
  can be imported, which would solve my issue. However, I'm not sure it
  could be released into python 2.7?

* Within the hooks, protect the Import Hooks with a separate lock for
  the loader method. This would prevent any other thread to import any
  modules from 'foo' package but still allows to call the finder method
  (ignoring module fullname not starting with 'foo') along with other
  finder methods, so that other ZODB modules can be imported.

  Then, in the loader method, until the module is actually inserted into
  sys.modules and then other load_module() PEP302 responsabilities being
  taken care of (such as exec the code), release the ImportLock so that
  Thread 2 can process requests and send objects back to Thread 1.

  However, even after trying to understand import.c, I'm not sure at all
  that this is enough and that releasing ImportLock would not have nasty
  side-effects, any thoughts about that?

* Fix the code in ZODB to not avoid import but to me this seems like a
  dirty hack because it could happen again and I would prefer to fix
  this issue once and for all.

Any thoughts or suggestion welcome, thanks!

Regards,
-- 
Arnaud Fontaine

[0] http://bugs.python.org/issue9260
[1] http://mail.python.org/pipermail/python-dev/2003-February/033436.html
-- 
http://mail.python.org/mailman/listinfo/python-list

Dealing with ImportLock deadlock in Import Hooks

[Arnaud Fontaine]

Hello,

I'm currently working on implementing Import Hooks (PEP302) with Python
2.7 to be able to import modules whose code is in ZODB. However, I have
stumbled upon a widely known issue about import deadlock[0][1] (note
that this issue is not directly related to ZODB, but a more general
question about dealing with ImportLock deadlock for Import Hooks),
basically:

  Thread 1 is trying to import a module 'foo.bar' (where 'foo' is a
  package containing dynamic modules) handled by Import Hooks I
  implemented, so ImportLock is acquired before even running the hooks
  (Python/import.c:PyImport_ImportModuleLevel()). Then, these import
  hooks try to load objects from ZODB and a request is sent and handled
  by another thread (Thread 2) which itself tries to import another
  module. Of course, this causes a deadlock because the first thread
  still holds ImportLock.

I have thought about the following solutions:

* Backport the patch applied in python 3.3 from issue 9260[0]. This
  would be the best option because it would mean that even when trying
  to import any module from package 'foo', other modules and packages
  can be imported, which would solve my issue. However, I'm not sure it
  could be released into python 2.7?

* Within the hooks, protect the Import Hooks with a separate lock for
  the loader method. This would prevent any other thread to import any
  modules from 'foo' package but still allows to call the finder method
  (ignoring module fullname not starting with 'foo') along with other
  finder methods, so that other ZODB modules can be imported.

  Then, in the loader method, until the module is actually inserted into
  sys.modules and then other load_module() PEP302 responsabilities being
  taken care of (such as exec the code), release the ImportLock so that
  Thread 2 can process requests and send objects back to Thread 1.

  However, even after trying to understand import.c, I'm not sure at all
  that this is enough and that releasing ImportLock would not have nasty
  side-effects, any thoughts about that?

* Fix the code in ZODB to not avoid import but to me this seems like a
  dirty hack because it could happen again and I would prefer to fix
  this issue once and for all.

Any thoughts or suggestion welcome, thanks!

Regards,
-- 
Arnaud Fontaine

[0] http://bugs.python.org/issue9260
[1] http://mail.python.org/pipermail/python-dev/2003-February/033436.html
-- 
http://mail.python.org/mailman/listinfo/python-list

import hooks (PEP 302) broken in Python =2.5?

[Josh Haberman]

When reading about import hooks, I came across a blog entry comment
that says:

  One additional thing to note about ihooks is that it's
  somewhat seriously broken on Python 2.5 and newer and there
  seems to be little or no interest in fixing it. It's
  probably worth *always* avoiding when writing new code.

--http://orestis.gr/blog/2008/12/20/python-import-hooks/#c264

Does anyone know what this is referring to?  Should I be wary of
relying on import hooks?

Josh
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: import hooks (PEP 302) broken in Python =2.5?

[Steven D'Aprano]

Josh Haberman wrote:

 When reading about import hooks, I came across a blog entry comment
 that says:
 
   One additional thing to note about ihooks is that it's
   somewhat seriously broken on Python 2.5 and newer and there
   seems to be little or no interest in fixing it. It's
   probably worth *always* avoiding when writing new code.
 
 --http://orestis.gr/blog/2008/12/20/python-import-hooks/#c264
 
 Does anyone know what this is referring to?  Should I be wary of
 relying on import hooks?

Are you using import hooks? What version(s) of Python are you using?

Depending on whether you are asking out of curiosity, or because you have
code that uses import hooks and you're about to migrate from Python 2.3 to
3.1, the answers may be different.



-- 
Steven

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: import hooks (PEP 302) broken in Python =2.5?

[Ryan Kelly]

On Sun, 2011-08-07 at 11:11 -0700, Josh Haberman wrote:
 When reading about import hooks, I came across a blog entry comment
 that says:
 
   One additional thing to note about ihooks is that it's
   somewhat seriously broken on Python 2.5 and newer and there
   seems to be little or no interest in fixing it. It's
   probably worth *always* avoiding when writing new code.
 
 --http://orestis.gr/blog/2008/12/20/python-import-hooks/#c264
 
 Does anyone know what this is referring to?  Should I be wary of
 relying on import hooks?

I believe that comment is referring specifically to ihooks the stdlib
module, not import hooks the general concept as defined in PEP302.
The former pre-dates the later.

I use custom PEP302 loaders all the time and they work fine in at least
2.6, 2.7 and 3.2.


  Ryan

-- 
Ryan Kelly
http://www.rfk.id.au  |  This message is digitally signed. Please visit
r...@rfk.id.au|  http://www.rfk.id.au/ramblings/gpg/ for details



signature.asc
Description: This is a digitally signed message part
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Creating Import Hooks

[Jonathan Gardner]

On Feb 17, 10:48 pm, Sreejith K sreejith...@gmail.com wrote:
 Hi everyone,

 I need to implement custom import hooks for an application 
 (http://www.python.org/dev/peps/pep-0302/). I want to restrict an application
 to import certain modules (say socket module). Google app engine is
 using a module hook to do this (HardenedModulesHook in google/
 appengine/tools/dev_appserver.py). But I want to allow that
 application to use an sdk module (custom) which imports and uses
 socket module. But the module hook restricts the access by sdk.
 Finding out, which file is importing a module give a solution?? ie. If
 the application is importing socket module, I want to restrict it. But
 if the sdk module is importing socket I want to allow it. Is there any
 way I can do this ?

 Application
 
 import sdk
 import socket               # I dont want to allow this (need to raise
 ImportError)

 SDK
 
 import socket               # need to allow this


SDK
===
import socket

App
===
import SDK
import sys
socket = sys.modules['socket']
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Creating Import Hooks

[Steven D'Aprano]

On Thu, 18 Feb 2010 00:03:51 -0800, Jonathan Gardner wrote:

 On Feb 17, 10:48 pm, Sreejith K sreejith...@gmail.com wrote:
 Hi everyone,

 I need to implement custom import hooks for an application
 (http://www.python.org/dev/peps/pep-0302/). I want to restrict an
 application to import certain modules (say socket module). Google app
 engine is using a module hook to do this (HardenedModulesHook in
 google/ appengine/tools/dev_appserver.py). But I want to allow that
 application to use an sdk module (custom) which imports and uses socket
 module. But the module hook restricts the access by sdk. Finding out,
 which file is importing a module give a solution?? ie. If the
 application is importing socket module, I want to restrict it. But if
 the sdk module is importing socket I want to allow it. Is there any way
 I can do this ?

 Application
 
 import sdk
 import socket               # I dont want to allow this (need to raise
 ImportError)

 SDK
 
 import socket               # need to allow this
 
 
 SDK
 ===
 import socket
 
 App
 ===
 import SDK
 import sys
 socket = sys.modules['socket']


I'm not sure, but I think Sreejith wants to prohibit imports from the App 
layer while allowing them from the SDK layer, not work around a 
prohibition in the SDK layer.

In other words, he wants the import hook to do something like this:

if module is socket and the caller is not SKD:
prohibit
else
allow



I could be wrong of course.



-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Creating Import Hooks

[Sreejith K]

On Feb 18, 1:57 pm, Steven D'Aprano
ste...@remove.this.cybersource.com.au wrote:
 On Thu, 18 Feb 2010 00:03:51 -0800, Jonathan Gardner wrote:
  On Feb 17, 10:48 pm, Sreejith K sreejith...@gmail.com wrote:
  Hi everyone,

  I need to implement custom import hooks for an application
  (http://www.python.org/dev/peps/pep-0302/). I want to restrict an
  application to import certain modules (say socket module). Google app
  engine is using a module hook to do this (HardenedModulesHook in
  google/ appengine/tools/dev_appserver.py). But I want to allow that
  application to use an sdk module (custom) which imports and uses socket
  module. But the module hook restricts the access by sdk. Finding out,
  which file is importing a module give a solution?? ie. If the
  application is importing socket module, I want to restrict it. But if
  the sdk module is importing socket I want to allow it. Is there any way
  I can do this ?

  Application
  
  import sdk
  import socket               # I dont want to allow this (need to raise
  ImportError)

  SDK
  
  import socket               # need to allow this

  SDK
  ===
  import socket

  App
  ===
  import SDK
  import sys
  socket = sys.modules['socket']

 I'm not sure, but I think Sreejith wants to prohibit imports from the App
 layer while allowing them from the SDK layer, not work around a
 prohibition in the SDK layer.

 In other words, he wants the import hook to do something like this:

 if module is socket and the caller is not SKD:
     prohibit
 else
     allow

 I could be wrong of course.

 --
 Steven

@Steven, Thats exactly what I want.. Anyway to do that ??
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Creating Import Hooks

[Jean-Michel Pichavant]

Sreejith K wrote:

On Feb 18, 1:57 pm, Steven D'Aprano
ste...@remove.this.cybersource.com.au wrote:
  

On Thu, 18 Feb 2010 00:03:51 -0800, Jonathan Gardner wrote:


On Feb 17, 10:48 pm, Sreejith K sreejith...@gmail.com wrote:
  

Hi everyone,

I need to implement custom import hooks for an application

(http://www.python.org/dev/peps/pep-0302/). I want to restrict an
application to import certain modules (say socket module). Google app
engine is using a module hook to do this (HardenedModulesHook in
google/ appengine/tools/dev_appserver.py). But I want to allow that
application to use an sdk module (custom) which imports and uses socket
module. But the module hook restricts the access by sdk. Finding out,
which file is importing a module give a solution?? ie. If the
application is importing socket module, I want to restrict it. But if
the sdk module is importing socket I want to allow it. Is there any way
I can do this ?

Application


import sdk
import socket   # I dont want to allow this (need to raise
ImportError)

SDK


import socket   # need to allow this


SDK
===
import socket
  
App

===
import SDK
import sys
socket = sys.modules['socket']
  

I'm not sure, but I think Sreejith wants to prohibit imports from the App
layer while allowing them from the SDK layer, not work around a
prohibition in the SDK layer.

In other words, he wants the import hook to do something like this:

if module is socket and the caller is not SKD:
prohibit
else
allow

I could be wrong of course.

--
Steven



@Steven, Thats exactly what I want.. Anyway to do that ??
  


import sys
sys.modules['socket'] = None


import socket
---
ImportError   Traceback (most recent call last)

ImportError: No module named socket


JM


--
http://mail.python.org/mailman/listinfo/python-list

Re: Creating Import Hooks

[Sreejith K]

On Feb 18, 3:49 pm, Jean-Michel Pichavant jeanmic...@sequans.com
wrote:
 Sreejith K wrote:
  On Feb 18, 1:57 pm, Steven D'Aprano
  ste...@remove.this.cybersource.com.au wrote:

  On Thu, 18 Feb 2010 00:03:51 -0800, Jonathan Gardner wrote:

  On Feb 17, 10:48 pm, Sreejith K sreejith...@gmail.com wrote:

  Hi everyone,

  I need to implement custom import hooks for an application
  (http://www.python.org/dev/peps/pep-0302/). I want to restrict an
  application to import certain modules (say socket module). Google app
  engine is using a module hook to do this (HardenedModulesHook in
  google/ appengine/tools/dev_appserver.py). But I want to allow that
  application to use an sdk module (custom) which imports and uses socket
  module. But the module hook restricts the access by sdk. Finding out,
  which file is importing a module give a solution?? ie. If the
  application is importing socket module, I want to restrict it. But if
  the sdk module is importing socket I want to allow it. Is there any way
  I can do this ?

  Application
  
  import sdk
  import socket               # I dont want to allow this (need to raise
  ImportError)

  SDK
  
  import socket               # need to allow this

  SDK
  ===
  import socket

  App
  ===
  import SDK
  import sys
  socket = sys.modules['socket']

  I'm not sure, but I think Sreejith wants to prohibit imports from the App
  layer while allowing them from the SDK layer, not work around a
  prohibition in the SDK layer.

  In other words, he wants the import hook to do something like this:

  if module is socket and the caller is not SKD:
      prohibit
  else
      allow

  I could be wrong of course.

  --
  Steven

  @Steven, Thats exactly what I want.. Anyway to do that ??

 import sys
 sys.modules['socket'] = None

 import socket
 ---
 ImportError                               Traceback (most recent call last)

 ImportError: No module named socket

 JM

@Jean. Thanks for the reply. But this is not what I wanted. The import
hook already restricts socket imports in applications. But I want them
in sdk package (alone) which is being imported in the application. I
don't want applications to directly use the socket module. That means
I want to make some exceptions for sdk in import hooks.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Creating Import Hooks

[Jean-Michel Pichavant]

Sreejith K wrote:

On Feb 18, 3:49 pm, Jean-Michel Pichavant jeanmic...@sequans.com
wrote:
  

Sreejith K wrote:


On Feb 18, 1:57 pm, Steven D'Aprano
ste...@remove.this.cybersource.com.au wrote:
  

On Thu, 18 Feb 2010 00:03:51 -0800, Jonathan Gardner wrote:


On Feb 17, 10:48 pm, Sreejith K sreejith...@gmail.com wrote:
  

Hi everyone,

I need to implement custom import hooks for an application

(http://www.python.org/dev/peps/pep-0302/). I want to restrict an
application to import certain modules (say socket module). Google app
engine is using a module hook to do this (HardenedModulesHook in
google/ appengine/tools/dev_appserver.py). But I want to allow that
application to use an sdk module (custom) which imports and uses socket
module. But the module hook restricts the access by sdk. Finding out,
which file is importing a module give a solution?? ie. If the
application is importing socket module, I want to restrict it. But if
the sdk module is importing socket I want to allow it. Is there any way
I can do this ?

Application


import sdk
import socket   # I dont want to allow this (need to raise
ImportError)

SDK


import socket   # need to allow this


SDK
===
import socket
  
App

===
import SDK
import sys
socket = sys.modules['socket']
  

I'm not sure, but I think Sreejith wants to prohibit imports from the App
layer while allowing them from the SDK layer, not work around a
prohibition in the SDK layer.

In other words, he wants the import hook to do something like this:

if module is socket and the caller is not SKD:

prohibit
else
allow

I could be wrong of course.

--

Steven


@Steven, Thats exactly what I want.. Anyway to do that ??
  

import sys
sys.modules['socket'] = None

import socket
---
ImportError   Traceback (most recent call last)

ImportError: No module named socket

JM



@Jean. Thanks for the reply. But this is not what I wanted. The import
hook already restricts socket imports in applications. But I want them
in sdk package (alone) which is being imported in the application. I
don't want applications to directly use the socket module. That means
I want to make some exceptions for sdk in import hooks.
  

give us your code (the hook import)

in your entry file:

import socket
import sys
sys.modules['sdkSocket'] = sys.modules['socket'] # allow to import 
socket ad sdkSocket

sys.modules['socket'] = None # forbid to import socket
del socket

within your SDK:
import sdkSocket # actually the socket module

print sdkSocket.__file__
'/usr/lib/python2.5/socket.pyc'

JM
--
http://mail.python.org/mailman/listinfo/python-list

Re: Creating Import Hooks

[Jonathan Gardner]

On Feb 18, 1:28 am, Sreejith K sreejith...@gmail.com wrote:
 On Feb 18, 1:57 pm, Steven D'Aprano



 ste...@remove.this.cybersource.com.au wrote:
  On Thu, 18 Feb 2010 00:03:51 -0800, Jonathan Gardner wrote:
   On Feb 17, 10:48 pm, Sreejith K sreejith...@gmail.com wrote:
   Hi everyone,

   I need to implement custom import hooks for an application
   (http://www.python.org/dev/peps/pep-0302/). I want to restrict an
   application to import certain modules (say socket module). Google app
   engine is using a module hook to do this (HardenedModulesHook in
   google/ appengine/tools/dev_appserver.py). But I want to allow that
   application to use an sdk module (custom) which imports and uses socket
   module. But the module hook restricts the access by sdk. Finding out,
   which file is importing a module give a solution?? ie. If the
   application is importing socket module, I want to restrict it. But if
   the sdk module is importing socket I want to allow it. Is there any way
   I can do this ?

   Application
   
   import sdk
   import socket               # I dont want to allow this (need to raise
   ImportError)

   SDK
   
   import socket               # need to allow this

   SDK
   ===
   import socket

   App
   ===
   import SDK
   import sys
   socket = sys.modules['socket']

  I'm not sure, but I think Sreejith wants to prohibit imports from the App
  layer while allowing them from the SDK layer, not work around a
  prohibition in the SDK layer.

  In other words, he wants the import hook to do something like this:

  if module is socket and the caller is not SKD:
      prohibit
  else
      allow

  I could be wrong of course.


 @Steven, Thats exactly what I want.. Anyway to do that ??


My point was that it's really pointless to try to enforce any such
thing on the program or programmer. There are ways around it. If you
don't want them to play with socket, write in the documentation:
Don't play with the 'socket' module.

If you want to prevent them from touching sockets at all, it's time to
rethink your design. You may want to have a talk with Bruce Schneier,
or at least read what he's written if you still think you need to
somehow shut down a part of the system to its users.

Oftentimes, programmers think they need to have control over what
other people write, forgetting that they are able to do what they do
due to the freedoms afforded them. They also forget that they are not
in control of what other programmers do, anymore than a grocery store
who refuses to stock a certain product can prevent people from getting
that product.

Write your code to expand freedoms, not limit them. If your design
depends on limiting the choices of your users, you have done something
wrong.
-- 
http://mail.python.org/mailman/listinfo/python-list

Problems with import hooks and encoding

[Vinay Sajip]

The simple program

#--
 def main():
 print repr(u'\u2029'.encode('utf-8'))

 if __name__ == __main__:
 main()
#--

works as expected when run from the command-line, but fails when
converted to an executable using PyInstaller. (For example, if the
executable is created on a Ubuntu Jaunty system but then run on a
Ubuntu Hardy system.) Digging into why this happens leads to the
observation that Py_Initialize calls _PyUnicode_Init, which in turn
causes encodings to be imported via the _codecs module (codecs.c).
However, this appears to happen before the import hook machinery is
set up. Is there a way of intercepting the importing of encodings so
that a freeze processor like PyInstaller's launcher can redirect the
import to the package bundled with the executable, rather than any
package provided with a host-installed version of Python?

Regards,

Vinay Sajip
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: import hooks

[gagsl-py2]

--- Patrick Stinson [EMAIL PROTECTED]
escribió:

 Right on, that seemed to work, thanks.
 This is different than sys.path_hooks though, which
 requires a callable or
 string subclass?

Yes, it's different, meta_path is a generic mechanism
that doesn't depend on sys.path and is tried before
sys.path is traversed; the other is triggered by a
special sys.path entry (like a .zip file).

 After some experimentation it looks like you can
 disallow an import by
 raising an import error from your meta_path hook. It
 seems a little weird
 that python will then raise a new ImportError from
 import.c:find_module(),
 but I guess the behavior is desirable..

I think you can't make an import fail completely; if
your meta_path doesn't work, the next one is tried,
and then the standard places (where the error is
finally raised).
Looks like the global name foo not defined error:
sometimes it's not a global name at all, but that's
where the search finally failed.

-- 
Gabriel Genellina

Gabriel Genellina
Softlab SRL


  Tarjeta de crédito Yahoo! de Banco Supervielle.
Solicitá tu nueva Tarjeta de crédito. De tu PC directo a tu casa. 
www.tuprimeratarjeta.com.ar 
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: import hooks

[Patrick Stinson]

Right on, that seemed to work, thanks.
This is different than sys.path_hooks though, which requires a callable or
string subclass?

After some experimentation it looks like you can disallow an import by
raising an import error from your meta_path hook. It seems a little weird
that python will then raise a new ImportError from import.c:find_module(),
but I guess the behavior is desirable..

On Wed, Apr 16, 2008 at 5:17 PM, Gabriel Genellina [EMAIL PROTECTED]
wrote:

 En Wed, 16 Apr 2008 09:04:36 -0300, Patrick Stinson
 [EMAIL PROTECTED] escribió:

  I am defining a simple finder/loader object and adding it to
  sys.meta_path
  like this:
 
  PyRun_SimpleString(import sys; import ousiainternal; sys.meta_path =
  [ousiainternal.OusiaImporter]);

 You should append to the existing meta_path, not replace it, erasing any
 previous content.
 And it must be an *instance* of your importer, not the type itself.
 Note that you're polluting the __main__ module namespace by using
 PyRun_SimpleString; I'd use API calls like PySys_GetObject(meta_path)
 and PyList_Append (PEP 302 guarantees it is a list).

  sys.meta_path.append(Importer)\n;

 Here you append to sys.meta_path, but fail to create the instance first.

  PyRun_SimpleString(importer_source);

 You should check the return value; I bet you got a -1 (failure).

 --
 Gabriel Genellina

 --
 http://mail.python.org/mailman/listinfo/python-list




-- 
Patrick Kidd Stinson
http://www.patrickkidd.com/
http://pkaudio.sourceforge.net/
http://pksampler.sourceforge.net/
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: import hooks

[Gabriel Genellina]

En Tue, 15 Apr 2008 22:14:18 -0300, Patrick Stinson  
[EMAIL PROTECTED] escribió:

 What's the current way to install an import hook? I've got an embedded  
 app
 that has a few scripts that I want to import each other, but that are  
 not in
 sys.modules. I intentionally keep them out of sys.modules because their
 names will not be unique across the app. They will, however, be unique
 between scripts that I (do* want to see each other).
 Basically, I want to return a certain module from a name-based filter.  
 I've
 already written a type in C with find_module and load_module, but it  
 doesn't
 seem to work when I add the type to sys.path_hooks. I wrote a simple one
 that worked just fine from a pure script file run through python.exe.

 From that description alone I can't say what's happening; you should post  
some code.
Also, if your importer isn't disk-based, perhaps using sys.meta_path is  
better.

-- 
Gabriel Genellina

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: import hooks

[Patrick Stinson]

I am defining a simple finder/loader object and adding it to sys.meta_path
like this:

PyRun_SimpleString(import sys; import ousiainternal; sys.meta_path =
[ousiainternal.OusiaImporter]);


The following C code defines the loader object:


static void

MyImporter_dealloc(PyObject *self)

{

self-ob_type-tp_free(self);

}



static int

MyImporter_init(MyImporter *self, PyObject *args, PyObject *kwds)

{

  return 0;

}


static PyObject *

MyImporter_new(PyTypeObject *type, PyObject *args, PyObject *kwds)

{

  MyOutput *self;



  self = (MyOutput *)type-tp_alloc(type, 0);



  return (PyObject *)self;

}




/* Check whether we can satisfy the import of the module named by

'fullname'. Return self if we can, None if we can't.


FYI: MyImporter imports modules from the local instrument.

*/

static PyObject *

MyImporter_find_module(PyObject *obj, PyObject *args)

{

MyImporter *self = (MyImporter *)obj;

PyObject *path = NULL;

char *fullname;



if (!PyArg_ParseTuple(args, s|O,

fullname, path))

return NULL;



  mod = lookup_module_in_my_app(name); //borrowed ref



  if(mod == NULL);

  {

Py_INCREF(Py_None);

return Py_None;

}



Py_INCREF(self);

return (PyObject *)self;

}



/* Load and return the module named by 'fullname'. */

static PyObject *

MyImporter_load_module(PyObject *obj, PyObject *args)

{

MyImporter *self = (MyImporter *)obj;

PyObject *mod = NULL;

char *fullname;



if (!PyArg_ParseTuple(args, s, fullname))

return NULL;




  mod = lookup_module_in_my_app(name); // borrowed ref



  if(mod)

Py_INCREF(mod);



return mod;

}



static PyMethodDef MyImporter_methods[] = {

{find_module, MyImporter_find_module, METH_VARARGS,

find a module},

{load_module, MyImporter_load_module, METH_VARARGS,

load a module},

{NULL, NULL} /* sentinel */

};



static PyTypeObject MyImporterType = {

  PyObject_HEAD_INIT(NULL)

  0, /*ob_size*/

  Myinternal.MyImporter, /*tp_name*/

  sizeof( MyImporter), /*tp_basicsize*/

  0, /*tp_itemsize*/

  (destructor)  MyImporter_dealloc,/*tp_dealloc*/

  0, /*tp_print*/

  0, /*tp_getattr*/

  0, /*tp_setattr*/

  0, /*tp_compare*/

  0, /*tp_repr*/

  0, /*tp_as_number*/

  0, /*tp_as_sequence*/

  0, /*tp_as_mapping*/

  0, /*tp_hash */

  0, /*tp_call*/

  0, /*tp_str*/

  0, /*tp_getattro*/

  0, /*tp_setattro*/

  0, /*tp_as_buffer*/

  Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /*tp_flags*/

  import hook for loading embedded modules,   /* tp_doc */

  0,   /* tp_traverse */

  0,   /* tp_clear */

  0,   /* tp_richcompare */

  0,   /* tp_weaklistoffset */

  0,   /* tp_iter */

  0,   /* tp_iternext */

  MyImporter_methods,   /* tp_methods */

  0, //MyImporter_members,   /* tp_members */

  0, /* tp_getset */

  0, /* tp_base */

  0, /* tp_dict */

  0, /* tp_descr_get */

  0, /* tp_descr_set */

  0, /* tp_dictoffset */

  (initproc)  MyImporter_init,  /* tp_init */

  0, /* tp_alloc */

  MyImporter_new, /* tp_new */

};

A simpler example that yields the same results would be this:

static const char *importer_source =

import sys\n

class Importer:\n

def __init__(self, path):\n

print \'Import.__init__\', path\n

def find_module(self, fullname, path=None):\n

print self\n

if fullname == \'bleh\':\n

return self\n

def load_module(self, fullname):\n

print self\n

if fullname == \'bleh\':\n

return sys\n

sys.meta_path.append(Importer)\n;


PyRun_SimpleString(importer_source);



For both examples none of the methods are called (I set breakpoints for the
C functions) but a statement like import os or
PyImport_ImportModule(traceback) don't work.


Thanks for your help


On Wed, Apr 16, 2008 at 12:02 AM, Gabriel Genellina [EMAIL PROTECTED]
wrote:

 En Tue, 15 Apr 2008 22:14:18 -0300, Patrick Stinson
 [EMAIL PROTECTED] escribió:

  What's the current way to install an import hook? I've got an embedded
  app
  that has a few scripts that I want to import each other, but that are
  not in
  sys.modules. I intentionally keep them out of sys.modules because their
  names will not be unique across the app. They will, however, be unique
  between scripts that I (do* want to see each other).
  Basically, I want to return a certain module from a name-based 

Re: import hooks

[Gabriel Genellina]

En Wed, 16 Apr 2008 09:04:36 -0300, Patrick Stinson  
[EMAIL PROTECTED] escribió:

 I am defining a simple finder/loader object and adding it to  
 sys.meta_path
 like this:

 PyRun_SimpleString(import sys; import ousiainternal; sys.meta_path =
 [ousiainternal.OusiaImporter]);

You should append to the existing meta_path, not replace it, erasing any  
previous content.
And it must be an *instance* of your importer, not the type itself.
Note that you're polluting the __main__ module namespace by using  
PyRun_SimpleString; I'd use API calls like PySys_GetObject(meta_path)  
and PyList_Append (PEP 302 guarantees it is a list).

 sys.meta_path.append(Importer)\n;

Here you append to sys.meta_path, but fail to create the instance first.

 PyRun_SimpleString(importer_source);

You should check the return value; I bet you got a -1 (failure).

-- 
Gabriel Genellina

-- 
http://mail.python.org/mailman/listinfo/python-list

import hooks

[Patrick Stinson]

What's the current way to install an import hook? I've got an embedded app
that has a few scripts that I want to import each other, but that are not in
sys.modules. I intentionally keep them out of sys.modules because their
names will not be unique across the app. They will, however, be unique
between scripts that I (do* want to see each other).
Basically, I want to return a certain module from a name-based filter. I've
already written a type in C with find_module and load_module, but it doesn't
seem to work when I add the type to sys.path_hooks. I wrote a simple one
that worked just fine from a pure script file run through python.exe.

Thanks!

-- 
Patrick Kidd Stinson
http://www.patrickkidd.com/
http://pkaudio.sourceforge.net/
http://pksampler.sourceforge.net/
-- 
http://mail.python.org/mailman/listinfo/python-list