[issue16392] [doc] import crashes on circular imports in ext modules

[Stefan Behnel]

Stefan Behnel  added the comment:

Given that PEP-489 has landed in Py3.5, which is already retired and has been 
for more than a year, I think we can just close this issue as outdated.

--
resolution:  -> out of date
stage: needs patch -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] [doc] import crashes on circular imports in ext modules

[Irit Katriel]

Change by Irit Katriel :


--
keywords: +easy
title: import crashes on circular imports in ext modules -> [doc] import 
crashes on circular imports in ext modules
versions: +Python 3.10, Python 3.11, Python 3.9 -Python 3.4, Python 3.5

___
Python tracker 
<https://bugs.python.org/issue16392>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44717] Improve AttributeError on circular imports of submodules

[Filipe Laíns]

Change by Filipe Laíns :


--
pull_requests: +25886
pull_request: https://github.com/python/cpython/pull/27344

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44717] Improve AttributeError on circular imports of submodules

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44717] Improve AttributeError on circular imports of submodules

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:


New changeset 0a8ae8a50a0fea3d39ec49b220a5c7a5b70e36f8 by Filipe Laíns in 
branch 'main':
bpo-44717: improve AttributeError on circular imports of submodules (GH-27338)
https://github.com/python/cpython/commit/0a8ae8a50a0fea3d39ec49b220a5c7a5b70e36f8


--

___
Python tracker 
<https://bugs.python.org/issue44717>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44717] Improve AttributeError on circular imports of submodules

[Filipe Laíns]

Change by Filipe Laíns :


--
pull_requests: +25881
pull_request: https://github.com/python/cpython/pull/27338

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44717] Improve AttributeError on circular imports of submodules

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:


New changeset 3eae8f20d7b6f88d3618b0afc94893165a914022 by Pablo Galindo Salgado 
in branch 'main':
Revert "bpo-44717: improve AttributeError on circular imports of submodules 
(GH-27299)" (GH-27331)
https://github.com/python/cpython/commit/3eae8f20d7b6f88d3618b0afc94893165a914022


--

___
Python tracker 
<https://bugs.python.org/issue44717>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44717] Improve AttributeError on circular imports of submodules

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
pull_requests: +25874
stage: resolved -> patch review
pull_request: https://github.com/python/cpython/pull/27331

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44717] Improve AttributeError on circular imports of submodules

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

There is a buildbot failure after merging this PR:

https://buildbot.python.org/all/#/builders/464/builds/574

If is not fixed in 24 we will need to revert per our buildbot policy

--
resolution: fixed -> 
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44717] Improve AttributeError on circular imports of submodules

[Łukasz Langa]

Łukasz Langa  added the comment:

Thanks a lot for your report and patch, Filipe! ✨  ✨

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44717] Improve AttributeError on circular imports of submodules

[Łukasz Langa]

Łukasz Langa  added the comment:


New changeset 8072a1181dd64135f700b44372fbf7bf91e68072 by Filipe Laíns in 
branch 'main':
bpo-44717: improve AttributeError on circular imports of submodules (GH-27299)
https://github.com/python/cpython/commit/8072a1181dd64135f700b44372fbf7bf91e68072


--
nosy: +lukasz.langa

___
Python tracker 
<https://bugs.python.org/issue44717>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44717] Improve AttributeError on circular imports of submodules

[Filipe Laíns]

Change by Filipe Laíns :


--
keywords: +patch
pull_requests: +25842
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/27299

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue44717] Improve AttributeError on circular imports of submodules

[Filipe Laíns]

New submission from Filipe Laíns :

Consider the following


$ tree a
a
├── b
│   ├── c.py
│   └── __init__.py
└── __init__.py

1 directory, 3 files
$ cat a/b/__init__.py
import a.b.c
$ cat a/b/c.py
import a.b

a.b
$ python
Python 3.9.6 (default, Jun 30 2021, 10:22:16)
[GCC 11.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import a.b
Traceback (most recent call last):
  File "", line 1, in 
  File "/home/anubis/test/a/b/__init__.py", line 1, in 
import a.b.c
  File "/home/anubis/test/a/b/c.py", line 3, in 
a.b
AttributeError: module 'a' has no attribute 'b'


This happens because even though the module `a` is initialized, the `b` 
attribute is not set due to the circular import. When a module is partially 
initialized, we get a more helpful error description hinting that we cannot 
access the attribute because the module is partially initialized, we could have 
something similar here.

--
components: Interpreter Core
messages: 398021
nosy: FFY00, pablogsal
priority: normal
severity: normal
status: open
title: Improve AttributeError on circular imports of submodules
type: enhancement

___
Python tracker 
<https://bugs.python.org/issue44717>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] import crashes on circular imports in ext modules

[Brett Cannon]

Change by Brett Cannon :


--
nosy:  -brett.cannon

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] import crashes on circular imports in ext modules

[Brett Cannon]

Brett Cannon added the comment:

https://www.python.org/dev/peps/pep-0489/ is trying to come up with a redesign 
of extension module loading and no one has submitted a patch for the 
documentation (although Stefan has inlined proposed wording in a comment).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] import crashes on circular imports in ext modules

[R. David Murray]

R. David Murray added the comment:

Yes, the resolution of this issue will be to add documentation.  Someone should 
turn Stefan's comment into a patch.

--
assignee:  - docs@python
components: +Documentation
nosy: +docs@python, r.david.murray
stage:  - needs patch
versions: +Python 3.5 -Python 3.2, Python 3.3

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] import crashes on circular imports in ext modules

[Omer Katz]

Omer Katz added the comment:

Is this issue resolved in any way?
Has there been a decision made on how to resolve it?
What's the status here?

--
nosy: +Omer.Katz

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] import crashes on circular imports in ext modules

[Arfrever Frehtes Taifersar Arahesis]

Changes by Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com:


--
nosy: +Arfrever

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] import crashes on circular imports in ext modules

[Andrew Svetlov]

Changes by Andrew Svetlov andrew.svet...@gmail.com:


--
nosy: +asvetlov

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] import crashes on circular imports in ext modules

[Eric Snow]

Changes by Eric Snow ericsnowcurren...@gmail.com:


--
nosy: +eric.snow

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] import crashes on circular imports in ext modules

[Brett Cannon]

Brett Cannon added the comment:

The fully qualified name requirement is definitely a design flaw where init 
functions should just be given the module object with everything already set, 
just like what @importlib.util.module_for_loader does. Hopefully we can come up 
with a solution through your current discussion on python-dev.

As for the comment, yes at the top of the page makes sense. Whether example 
code should go somewhere else I don't know.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] import crashes on circular imports in ext modules

[Stefan Behnel]

Stefan Behnel added the comment:

Agreed. Since it doesn't really fit into any specific function documentation, I 
would place it right at the top. Something like this:


The following functions can be used by C code to call into Python's import 
machinery.

Note that Python 3 does not automatically register an extension module in 
sys.modules on creation (see module.html#initializing-c-modules). It is only 
added after running through the whole module init function. This means that a 
request to import the current module while its init function is still running 
(either directly or transitively by other modules) will try to reimport the 
module. If you cannot be sure that this will not happen, you have to register 
the newly created module yourself as follows, using the fully qualified module 
name::

PyObject *the_module = PyModule_Create(py_module_def);
if (the_module == NULL) { /* failure ! */ };

PyObject *sys_modules = PyImport_GetModuleDict();
if (sys_modules == NULL) { /* failure ! */ };

if (PyDict_SetItemString(modules, the.package.and.module, the_module)  
0) { /* failure ! */ };


Maybe it should add another comment that this is a major quirk in the runtime 
and say sorry for being stupid here - I hope you can do better. Requiring the 
user to know the FQMN at build time because the import machinery fails to set 
it automatically is just embarrissing.

Then, after the first sentence in the module.html#initializing-c-modules 
section, I'd add this:


Note that, starting with Python 3.0, the module creation functions no longer 
register the module in sys.modules. The import machinery will only do this 
after the module init function has run. If you need to run imports as part of 
your module init function and happen to know the fully qualified module name in 
your code, it is best to register the module yourself after creating it.


I wonder if the code example shouldn't go on the module page.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] import crashes on circular imports in ext modules

[Brett Cannon]

Brett Cannon added the comment:

I was thinking somewhere in http://docs.python.org/3/c-api/import.html since 
this only comes up when you try to execute an import during extension module 
initialization that involves a circular import.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] import crashes on circular imports in ext modules

[Antoine Pitrou]

Antoine Pitrou added the comment:

 Maybe we should add a warning in some documentation somewhere about
 this and then close the issue?

I don't really know where to add it, in the C API docs perhaps,

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] import crashes on circular imports in ext modules

[Stefan Behnel]

Stefan Behnel added the comment:

The problem is a) that the module does not necessarily know to which place it 
eventually gets installed (Cython relies on the distutils Extension not lying 
to it, for example, which people do from time to time), and b) that the call to 
Py_InitModule() only receives the plain module name, not the package path. And 
yes, as mentioned in the other issue, passing a pointer to a context 
description struct into the module init function would have been the right 
thing to change for Py3 and still is the right thing to change for Py4.

BTW, I can confirm that registering the module in sys.modules explicitly right 
after creation works around this issue. Given that Cython needs to know the 
FQMN at compile time anyway, this works for us. It still leaves the problem 
open for other extension code.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] import crashes on circular imports in ext modules

[Brett Cannon]

Brett Cannon added the comment:

It sounds like Cython has its fix and CPython knows what should eventually be 
changed in Python 4 to bring extension module initialization closer to how 
Python module code is initialized.

Maybe we should add a warning in some documentation somewhere about this and 
then close the issue?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] frozen importlib crashes on circular imports in ext modules

[Stefan Behnel]

New submission from Stefan Behnel:

After compiling the stdlib with Cython with the attached script, modules that 
use circular imports fail to initialise. That includes os and posixpath, as 
well as shutil and tarfile. Example:

$ ./python -c 'import shutil'
Traceback (most recent call last):
  File string, line 1, in module
  File tarfile.py, line 44, in init tarfile (tarfile.c:44135)
import shutil
  File shutil.py, line 14, in init shutil (shutil.c:22497)
import tarfile
  File frozen importlib._bootstrap, line 1556, in _find_and_load
RuntimeError: maximum recursion depth exceeded

I've tried this with the latest CPython 3.4 hg version, but I'm pretty sure it 
fails in Py3.3 as well.

--
components: Interpreter Core
files: cystdlib.py
messages: 174612
nosy: scoder
priority: normal
severity: normal
status: open
title: frozen importlib crashes on circular imports in ext modules
type: behavior
versions: Python 3.4
Added file: http://bugs.python.org/file27851/cystdlib.py

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] frozen importlib crashes on circular imports in ext modules

[Brett Cannon]

Brett Cannon added the comment:

I don't mean for this to sound rude, but this seems like a Cython issue and not 
one for the stdlib. Can you reproduce the problem without using Cython? The new 
per-module locking mechanism Antoine prevents this from being a problem 
normally, so it makes me think Cython is at fault here.

--
nosy: +brett.cannon, pitrou

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] frozen importlib crashes on circular imports in ext modules

[Antoine Pitrou]

Antoine Pitrou added the comment:

Well, it might be a legitimate issue, but due to the setup needed to reproduce, 
I would hope a Cython developer could do the diagnosis and possibly submit a 
patch.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] frozen importlib crashes on circular imports in ext modules

[Stefan Behnel]

Stefan Behnel added the comment:

Well, it's not like the setup was all that difficult. 1) Install the latest 
github master of Cython (they provide one-click archives that pip can install 
for you), 2) change into the CPython stdlib directory and run the script I 
attached, 3) execute import os in Python. You need to install Cython rather 
than just unpacking it because it uses 2to3 for installation in Py3.

Anyway, I ran gdb on it and it turns out that the exception is correct, there 
is an infinite recursion taking place. According to the (otherwise not very 
interesting) stack trace at the point where it raises the RuntimeError, the 
module init function of the first Cython module (say, os) calls the builtin 
__import__() to import posixpath. That triggers the load of that shared 
library and the execution of its module init function. Fine so far. However, 
that module init function then executes an import of os through 
__import__(), which then runs the module init function of the os module 
again. Bug right here. It shouldn't try to reimport a module that it is already 
importing.

I could reduce the test case down to one line:

  # reimport.py
  import reimport

Compiling that with Cython gives the C code I attached. Build it, import it, 
see it fail. However, remember that fixing only this isn't enough, the import 
cycle might be nested arbitrarily deep.

--
versions: +Python 3.3
Added file: http://bugs.python.org/file27864/reimport.c

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] frozen importlib crashes on circular imports in ext modules

[Stefan Behnel]

Changes by Stefan Behnel sco...@users.sourceforge.net:


Added file: http://bugs.python.org/file27865/cystdlibbug.py

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] import crashes on circular imports in ext modules

[Stefan Behnel]

Stefan Behnel added the comment:

Since it's quite possible that this has nothing to do with the frozen part of 
the importlib specifically, I'm removing that bit from the ticket title.

--
title: frozen importlib crashes on circular imports in ext modules - import 
crashes on circular imports in ext modules

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] import crashes on circular imports in ext modules

[Antoine Pitrou]

Antoine Pitrou added the comment:

Are you sure this worked before 3.3?

Regardless, it seems you could fix this issue by adding a single line just 
after the PyModule_Create() call:

  __pyx_m = PyModule_Create(__pyx_moduledef);
  #endif
  if (PyDict_SetItemString(PyImport_GetModuleDict(),
   __Pyx_NAMESTR(reimport), __pyx_m))
goto __pyx_L1_error;


The fundamental issue is that an extension is inserted into sys.modules 
only after its initialization function returns, so importing it recursively 
won't detect that it already exists.

(by contrast, a Python module is inserted into sys.modules before its code is 
executed inside the module's global namespace)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] import crashes on circular imports in ext modules

[Antoine Pitrou]

Antoine Pitrou added the comment:

A simple test seems to confirm the problem already existed in 3.2.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] import crashes on circular imports in ext modules

[Antoine Pitrou]

Antoine Pitrou added the comment:

Note that Py_InitModule4 in Python 2 did add the module object to sys.modules 
(by calling PyImport_AddModule) before returning, but PyModule_Create in Python 
3 doesn't.

(PEP 3121 doesn't seem to mention PyModule_Create at all, strangely; it seems 
the PEP doesn't follow the implementation)

--
nosy: +loewis
versions: +Python 3.2

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] import crashes on circular imports in ext modules

[Stefan Behnel]

Stefan Behnel added the comment:

Hmm, we already do that for packages (i.e. when compiling __init__.py). Looks 
like this just needs to be done for all modules in Py3. And unless there is a 
compelling reason for Py_InitModule4() not to do it, I think it should be 
reverted to its Py2 behaviour.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] import crashes on circular imports in ext modules

[Antoine Pitrou]

Antoine Pitrou added the comment:

I have detected a compatibility issue when reverting to the 2.x behaviour: 
extension modules which lie about their name in their PyModuleDef are broken. 
There are two such modules in 3.3: _io and _decimal.

Patch attached, anyway.

--
keywords: +patch
Added file: http://bugs.python.org/file27872/modulecreate.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] import crashes on circular imports in ext modules

[Stefan Behnel]

Stefan Behnel added the comment:

Now that you mention it - wouldn't that suffer from not actually knowing the 
FQMN? There's that hack in the dynlib loader that stores the package context in 
a global variable so that the module creation function can figure it out. That 
might be a reason not to register the module automatically. Only the loader 
would know the correct package, not the module creation code.

Also see issue 13429.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16392] import crashes on circular imports in ext modules

[Antoine Pitrou]

Antoine Pitrou added the comment:

 Now that you mention it - wouldn't that suffer from not actually
 knowing the FQMN? There's that hack in the dynlib loader that stores
 the package context in a global variable so that the module creation
 function can figure it out. That might be a reason not to register the 
 module automatically.

That's possible (although I would expect a module to know in which package it's 
supposed to live).
Another solution would have been to pass the module object to the init 
function, instead of letting the init function create it, but it's a bit too 
late (or too early :-)) to change the extension module init signature again.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue16392
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: Question about circular imports

[Ethan Furman]

OKB (not okblacke) wrote:
	Anyway, testing this just reinforced my distaste for circular 
imports.  Just trying to think about how it ought to work with a 
importing c but then c and d importing each other makes my brain hurt.  
Refactoring the files so that common code is in a separate library 
imported by both is easier to understand, and has the nice side bonus 
that it works.




+1 QOTW
--
http://mail.python.org/mailman/listinfo/python-list

Re: Question about circular imports

[Jean-Michel Pichavant]

Frank Millman wrote:

Hi all

I seem to have a recurring battle with circular imports, and I am trying to 
nail it once and for all.


Let me say at the outset that I don't think I can get rid of circular 
imports altogether. It is not uncommon for me to find that a method in 
Module A needs to access something in Module B, and a method in Module B 
needs to access something in Module A. I know that the standard advice is to 
reorganise the code to avoid this, and I try to do this where possible, but 
for now I would like to address the question of how to handle the situation 
if this is otherwise unavoidable.
  

Hi Frank,

If you consider it unavoidable you've already lost. There is no reliable 
solution to circular import except refactoring the code to place all 
common code elsewhere:


wavbase.py
wavread.py (import wavbase)
wavwrite.py (import wavbase)


Any code required by both waread and wavwrite would end up in wavbase.
Is there anything that prevent you from doing this ?

JM
--
http://mail.python.org/mailman/listinfo/python-list

Re: Question about circular imports

[Ian Kelly]

On Sun, Feb 26, 2012 at 3:42 AM, Frank Millman fr...@chagford.com wrote:
 Hi all

 I seem to have a recurring battle with circular imports, and I am trying to
 nail it once and for all.

 Let me say at the outset that I don't think I can get rid of circular
 imports altogether. It is not uncommon for me to find that a method in
 Module A needs to access something in Module B, and a method in Module B
 needs to access something in Module A. I know that the standard advice is to
 reorganise the code to avoid this, and I try to do this where possible, but
 for now I would like to address the question of how to handle the situation
 if this is otherwise unavoidable.

 The problem is clearly explained in the Python Programming FAQ -

 Circular imports are fine where both modules use the import module form
 of import. They fail when the 2nd module wants to grab a name out of the
 first (from module import name) and the import is at the top level. That's
 because names in the 1st are not yet available, because the first module is
 busy importing the 2nd.

 [SNIP]

 I can think of two solutions - one is cumbersome, the other may not be good
 practice.

Solution 3: don't do the circular imports at the top level.  It's
perfectly fine to do the imports locally inside the functions that
need them, which resolves the circularity since normally the functions
won't be called until the module is fully imported.  It does add some
overhead to the functions, basically the cost of a dict lookup.

Cheers,
Ian
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Question about circular imports

[Terry Reedy]

On 2/27/2012 1:16 AM, Frank Millman wrote:


To avoid the tedious reference, follow this with
read = sound.formats.wavread # choose the identifier you prefer


I tested something like this with stdlib, but there must be some 
important difference I did not notice. It make be in the contents of 
__init__.py.



@Terry and OKB

I tried that, but it does not work.

a.py
/b
 __init__.py
 c.py
d.py

a.py -
 from b import c
c.py -
 import b.d
d.py -
 import b.c


How about import b.d as d, etc?


If I run a.py, it returns with no error.

c.py -
 import b.d
 d = b.d
d.py -
 import b.c
 c = b.c

If I run a.py, I get

Traceback (most recent call last):
   File F:\tests\a.py, line 1, inmodule
 from b import c
   File F:\tests\b\c.py, line 1, inmodule
 import b.d
   File F:\tests\b\d.py, line 2, inmodule
 c = b.c
AttributeError: 'module' object has no attribute 'c'

I get the same if I try 'import b.c as c'.


Try import b; c = b.c


--
Terry Jan Reedy

--
http://mail.python.org/mailman/listinfo/python-list

Re: Question about circular imports

[Ian Kelly]

On Mon, Feb 27, 2012 at 6:01 PM, Terry Reedy tjre...@udel.edu wrote:
 On 2/27/2012 1:16 AM, Frank Millman wrote:


 To avoid the tedious reference, follow this with
 read = sound.formats.wavread # choose the identifier you prefer


 I tested something like this with stdlib, but there must be some important
 difference I did not notice. It make be in the contents of __init__.py.

I don't know what you tried, but I can easily replicate Frank's
results with an empty __init__.py.

 How about import b.d as d, etc?

 Try import b; c = b.c

Why would any of these make a difference?  The AttributeError
indicates that at the time d tries to grab the module in b's c
attribute, the attribute does not exist (because it has not finished
importing).  Alternate syntaxes for getting the c attribute from b
are not going to change that.

Python does some magic so that you can do import b.c inside d while
b.c is still importing without resulting in infinite recursion, but
the success of the import does not signify that b.c is actually
available yet.

Cheers,
Ian
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Question about circular imports

[OKB (not okblacke)]

Frank Millman wrote:


 To avoid the tedious reference, follow this with
 read = sound.formats.wavread # choose the identifier you prefer

 
 @Terry and OKB
 
 I tried that, but it does not work.
 
 a.py
 /b
 __init__.py
 c.py
d.py
 
 a.py -
 from b import c
 c.py -
 import b.d
 d.py -
 import b.c
 
 If I run a.py, it returns with no error.
 
 c.py -
 import b.d
 d = b.d
 d.py -
 import b.c
 c = b.c
 
 If I run a.py, I get
 
 Traceback (most recent call last):
   File F:\tests\a.py, line 1, in module
 from b import c
   File F:\tests\b\c.py, line 1, in module
 import b.d
   File F:\tests\b\d.py, line 2, in module
 c = b.c
 AttributeError: 'module' object has no attribute 'c'
 
 I get the same if I try 'import b.c as c'.

Interesting, you're right.  Note that it will work in c.py but not 
in d.py

Anyway, testing this just reinforced my distaste for circular 
imports.  Just trying to think about how it ought to work with a 
importing c but then c and d importing each other makes my brain hurt.  
Refactoring the files so that common code is in a separate library 
imported by both is easier to understand, and has the nice side bonus 
that it works.

-- 
--OKB (not okblacke)
Brendan Barnwell
Do not follow where the path may lead.  Go, instead, where there is
no path, and leave a trail.
--author unknown
-- 
http://mail.python.org/mailman/listinfo/python-list

Question about circular imports

[Frank Millman]

Hi all

I seem to have a recurring battle with circular imports, and I am trying to 
nail it once and for all.

Let me say at the outset that I don't think I can get rid of circular 
imports altogether. It is not uncommon for me to find that a method in 
Module A needs to access something in Module B, and a method in Module B 
needs to access something in Module A. I know that the standard advice is to 
reorganise the code to avoid this, and I try to do this where possible, but 
for now I would like to address the question of how to handle the situation 
if this is otherwise unavoidable.

The problem is clearly explained in the Python Programming FAQ -

Circular imports are fine where both modules use the import module form 
of import. They fail when the 2nd module wants to grab a name out of the 
first (from module import name) and the import is at the top level. That's 
because names in the 1st are not yet available, because the first module is 
busy importing the 2nd.

Having recently reorganised my code into packages, I find that the same 
problem arises with packages. Assume the following structure, copied from 
the Tutorial -

sound/
__init__.py
formats/
__init__.py
wavread.py
wavwrite.py

The following fails -

in wavread.py -
from formats import wavwrite [this works]

in wavwrite.py -
from formats import wavread [this fails with ImportError]

I can think of two solutions - one is cumbersome, the other may not be good 
practice.

The first solution is -

in wavread.py -
import formats.wavwrite

in wavwrite.py -
import formats.wavread

I then have to use the full path to reference any attribute inside the 
imported module, which I find cumbersome.

The second solution is -

in formats/__init__.py
import sys
sys.path.insert(0, __path__[0])

in wavread.py -
import wavwrite

in wavwrite.py -
import wavread

This works, but I don't know if it is a good idea to add all the sub-package 
paths to sys.path. I realise that it is up to me to avoid any name clashes. 
Are there any other downsides?

So I guess my question is -

- is there a better solution to my problem?
- if not, is my second solution acceptable?

If not, I seem to be stuck with using full path names to reference any 
attributes in imported modules.

I am using Python3 exclusively now, if that makes any difference.

Any advice will be appreciated.

Frank Millman



-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Question about circular imports

[Peter Otten]

Frank Millman wrote:

 I seem to have a recurring battle with circular imports, and I am trying
 to nail it once and for all.
 
 Let me say at the outset that I don't think I can get rid of circular
 imports altogether. It is not uncommon for me to find that a method in
 Module A needs to access something in Module B, and a method in Module B
 needs to access something in Module A. I know that the standard advice is
 to reorganise the code to avoid this, and I try to do this where possible,
 but for now I would like to address the question of how to handle the
 situation if this is otherwise unavoidable.

To cut a long story short, why should circular imports be unavoidable?

 The problem is clearly explained in the Python Programming FAQ -
 
 Circular imports are fine where both modules use the import module
 form of import. They fail when the 2nd module wants to grab a name out of
 the first (from module import name) and the import is at the top level.
 That's because names in the 1st are not yet available, because the first
 module is busy importing the 2nd.
 
 Having recently reorganised my code into packages, I find that the same
 problem arises with packages. Assume the following structure, copied from
 the Tutorial -
 
 sound/
 __init__.py
 formats/
 __init__.py
 wavread.py
 wavwrite.py
 
 The following fails -
 
 in wavread.py -
 from formats import wavwrite [this works]
 
 in wavwrite.py -
 from formats import wavread [this fails with ImportError]
 
 I can think of two solutions - one is cumbersome, the other may not be
 good practice.
 
 The first solution is -
 
 in wavread.py -
 import formats.wavwrite
 
 in wavwrite.py -
 import formats.wavread

This should be

import sound.formats.wavread

 I then have to use the full path to reference any attribute inside the
 imported module, which I find cumbersome.
 
 The second solution is -
 
 in formats/__init__.py
 import sys
 sys.path.insert(0, __path__[0])
 
 in wavread.py -
 import wavwrite
 
 in wavwrite.py -
 import wavread
 
 This works, but I don't know if it is a good idea to add all the
 sub-package paths to sys.path. I realise that it is up to me to avoid any
 name clashes. Are there any other downsides?
 
 So I guess my question is -
 
 - is there a better solution to my problem?
 - if not, is my second solution acceptable?

Paths into packages are recipe for desaster. You may end up with multiple 
instances of the same module and your programs will break in interesting 
(hard to debug) ways.
 
 If not, I seem to be stuck with using full path names to reference any
 attributes in imported modules.
 
 I am using Python3 exclusively now, if that makes any difference.
 
 Any advice will be appreciated.
 
 Frank Millman


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Question about circular imports

[Frank Millman]

Frank Millman fr...@chagford.com wrote in message 
news:jid2a9$n21$1...@dough.gmane.org...
 Hi all

 I seem to have a recurring battle with circular imports, and I am trying 
 to nail it once and for all.

[...]

 The second solution is -

 in formats/__init__.py
import sys
sys.path.insert(0, __path__[0])

 in wavread.py -
import wavwrite

 in wavwrite.py -
import wavread

 This works, but I don't know if it is a good idea to add all the 
 sub-package paths to sys.path. I realise that it is up to me to avoid any 
 name clashes. Are there any other downsides?


Answering my own question, I have just found a downside that is a 
showstopper.

If a module in a different sub-package needs to import one of these modules, 
it must use a full path. This results in a new entry in sys.modules, and 
therefore any attributes referenced by the intra-package module have 
different identities from those referenced from outside. If they are static, 
there is no problem, but if not, disaster strikes!

Frank



-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Question about circular imports

[Frank Millman]

Peter Otten __pete...@web.de wrote in message 
news:jid424$vfp$1...@dough.gmane.org...
 Frank Millman wrote:


 To cut a long story short, why should circular imports be unavoidable?

 Paths into packages are recipe for desaster. You may end up with multiple
 instances of the same module and your programs will break in interesting
 (hard to debug) ways.


Thanks, Peter. I have just figured this out for myself, but you beat me to 
it.

Full paths it is, then.

Frank



-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Question about circular imports

[OKB (not okblacke)]

Frank Millman wrote:

 The first solution is -
 
 in wavread.py -
 import formats.wavwrite
 
 in wavwrite.py -
 import formats.wavread
 
 I then have to use the full path to reference any attribute inside
 the imported module, which I find cumbersome.

This isn't really true.  If this is the only thing you're worried 
about, you can always do the manual version of a from X import Y:

import formats.wavread
wavread = formats.wavread

wavread.someFunc() # refers to functions in module formats.wavread.

Also, I'm too lazy to check right now, but I wouldn't be suprised 
if mport formats.wavread as wavread also works even in the circular 
case.

-- 
--OKB (not okblacke)
Brendan Barnwell
Do not follow where the path may lead.  Go, instead, where there is
no path, and leave a trail.
--author unknown
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Question about circular imports

[Terry Reedy]

On 2/26/2012 6:12 AM, Peter Otten wrote:

Frank Millman wrote:


I seem to have a recurring battle with circular imports, and I am trying
to nail it once and for all.

...

This should be

import sound.formats.wavread


To avoid the tedious reference, follow this with
read = sound.formats.wavread # choose the identifier you prefer

--
Terry Jan Reedy

--
http://mail.python.org/mailman/listinfo/python-list

Re: Question about circular imports

[Frank Millman]

 To avoid the tedious reference, follow this with
 read = sound.formats.wavread # choose the identifier you prefer


@Terry and OKB

I tried that, but it does not work.

a.py
/b
__init__.py
c.py
   d.py

a.py -
from b import c
c.py -
import b.d
d.py -
import b.c

If I run a.py, it returns with no error.

c.py -
import b.d
d = b.d
d.py -
import b.c
c = b.c

If I run a.py, I get

Traceback (most recent call last):
  File F:\tests\a.py, line 1, in module
from b import c
  File F:\tests\b\c.py, line 1, in module
import b.d
  File F:\tests\b\d.py, line 2, in module
c = b.c
AttributeError: 'module' object has no attribute 'c'

I get the same if I try 'import b.c as c'.

Frank



-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Problem with python 3.2 and circular imports

[Frank Millman]

Rafael Durán Castañeda rafadurancastan...@gmail.com wrote...


Thank you for your answer Frank, I think I've found the problem. I was
calling modules from inside subpackages, and I need to use them from
outside, so I have package in PYTHONPATH. is that correct? But now I have
another question: Can I execute an script inside subpackage1 importig
modules from subpackage2?


Hi Rafael

I am no expert, so I cannot answer you directly.

In my case, my program started as a single module.

As it grew, I started to split some parts off and store them in separate 
modules in the same directory. I placed 'import' statements in the main 
module, and it worked.


Then I found the need for some modules to refer to objects in other modules, 
so I needed 'import' statements within the modules. I found myself hitting 
problems with circular imports from time to time, but with some help from 
this group and re-reading the docs I got over this hurdle.


Only recently has my project got big enough to start thinking about 
packages. It adds complexity, but by reading the docs again, and thinking 
carefully about the structure, I have so far managed to handle the problems 
that occur.


I found the 'modules' chapter in the tutorial a good place to start. Once 
you have fully grasped the contents, PEP 328 is a good resource for 
understanding what has changed in python 3.x.


BTW, this group prefers 'bottom posting'. You will see that I have placed my 
response below yours. If you want to reply to this, please follow the same 
practice.


Hope this helps to get you started.

Frank


--
http://mail.python.org/mailman/listinfo/python-list

Re: Problem with python 3.2 and circular imports

[Rafael Durán Castañeda]

Thank you for your answer Frank, I think I've found the problem. I was
calling modules from inside subpackages, and I need to use them from
outside, so I have package in PYTHONPATH. is that correct? But now I have
another question: Can I execute an script inside subpackage1 importig
modules from subpackage2?

2011/3/4 Frank Millman fr...@chagford.com


 On February 28 2011 Rafael Durán Castañeda wrote

  I'm stil totally stuck with relative imports, i' ve tried the example tree
 from PEP 328 without any result:

 package/
   __init__.py
   subpackage1/
   __init__.py
   moduleX.py
   moduleY.py
   subpackage2/
   __init__.py
   moduleZ.py
   moduleA.py

 Assuming that the current file is either moduleX.py or
 subpackage1/__init__.py, following are correct usages of the new
 syntax:

 from .moduleY import spam
 from .moduleY import spam as ham
 from . import moduleY
 from ..subpackage1 import moduleY
 from ..subpackage2.moduleZ import eggs
 from ..moduleA import foo
 from ...package import bar
 from ...sys import path

 I always get:

 Traceback (most recent call last):
  File moduleY.py, line 1, in module
   from ..moduleA import a
 ValueError: Attempted relative import in non-package



 Hi Rafael

 I only just noticed your message now. For some reason it did not appear on
 google.groups.

 I just tried it and it worked for me. This is what I did.

 I set up the above structure. I added a 'def spam()' to moduleY, 'def
 eggs()' to moduleZ, and 'def foo()' to moduleA.

 I pasted all of the 'from ... import ...' lines above into moduleX.

 In the same directory that contains the subdirectory 'package' I create a
 test script containing the following line -
   import package.subpackage1.moduleX

 To begin with it failed on the last two lines starting with 'from ...' [how
 do you indicate three dots followed by etc etc, which would normally be
 indicated by three dots!]. The traceback said 'Attempted relative import
 beyond toplevel package'.

 I did not want to investigate too deeply so I just commented those lines
 out, and now it runs with no errors. I even put a couple of print statements
 (or must I call them print functions now) into the modules being imported,
 and the messages do appear, so the modules are being imported.

 HTH

 Frank Millman


 --
 http://mail.python.org/mailman/listinfo/python-list

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Problem with python 3.2 and circular imports

[Frank Millman]

On February 28 2011 Rafael Durán Castañeda wrote


I'm stil totally stuck with relative imports, i' ve tried the example tree
from PEP 328 without any result:

package/
   __init__.py
   subpackage1/
   __init__.py
   moduleX.py
   moduleY.py
   subpackage2/
   __init__.py
   moduleZ.py
   moduleA.py

Assuming that the current file is either moduleX.py or
subpackage1/__init__.py, following are correct usages of the new
syntax:

from .moduleY import spam
from .moduleY import spam as ham
from . import moduleY
from ..subpackage1 import moduleY
from ..subpackage2.moduleZ import eggs
from ..moduleA import foo
from ...package import bar
from ...sys import path

I always get:

Traceback (most recent call last):
 File moduleY.py, line 1, in module
   from ..moduleA import a
ValueError: Attempted relative import in non-package




Hi Rafael

I only just noticed your message now. For some reason it did not appear on 
google.groups.


I just tried it and it worked for me. This is what I did.

I set up the above structure. I added a 'def spam()' to moduleY, 'def 
eggs()' to moduleZ, and 'def foo()' to moduleA.


I pasted all of the 'from ... import ...' lines above into moduleX.

In the same directory that contains the subdirectory 'package' I create a 
test script containing the following line -

   import package.subpackage1.moduleX

To begin with it failed on the last two lines starting with 'from ...' [how 
do you indicate three dots followed by etc etc, which would normally be 
indicated by three dots!]. The traceback said 'Attempted relative import 
beyond toplevel package'.


I did not want to investigate too deeply so I just commented those lines 
out, and now it runs with no errors. I even put a couple of print statements 
(or must I call them print functions now) into the modules being imported, 
and the messages do appear, so the modules are being imported.


HTH

Frank Millman


--
http://mail.python.org/mailman/listinfo/python-list

Re: Problem with python 3.2 and circular imports

[Rafael Durán Castañeda]

I'm stil totally stuck with relative imports, i' ve tried the example tree
from PEP 328 without any result:

package/
__init__.py
subpackage1/
__init__.py
moduleX.py
moduleY.py
subpackage2/
__init__.py
moduleZ.py
moduleA.py

Assuming that the current file is either moduleX.py or
subpackage1/__init__.py, following are correct usages of the new
syntax:

from .moduleY import spam
from .moduleY import spam as ham
from . import moduleY
from ..subpackage1 import moduleY
from ..subpackage2.moduleZ import eggs
from ..moduleA import foo
from ...package import bar
from ...sys import path

I always get:

Traceback (most recent call last):
  File moduleY.py, line 1, in module
from ..moduleA import a
ValueError: Attempted relative import in non-package


2011/2/27 Frank Millman fr...@chagford.com


 Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote in message
 news:4d6a56aa$0$29972$c3e8da3$54964...@news.astraweb.com...

  On Sun, 27 Feb 2011 12:08:12 +0200, Frank Millman wrote:

  Assume the following structure -

 main.py
 /pkg
__init__.py
mod1.py
mod2.py

 main.py
from pkg import mod1

 mod1.py
import mod2

 mod2.py
  import mod1



 If you change the import mod* lines to import pkg.mod* it works for
 me in Python 3.1 and 3.2.

 According to my understand of PEP 328, from . import mod* should work,
 but I agree with you that it doesn't.

 If you get rid of the circular import, it does work. So I suspect a bug.



 Thanks, Steven.

 I confirm that 'import pkg.mod* works. Unfortunately I am using
 sub-packages as well, which means that to refer to an object in the
 sub-package I need to use w.x.y.z every time, which gets to be a lot of
 typing! I will stick to my hack of putting the package name in sys.path for
 now, unless someone comes up with a better idea.

 Frank



 --
 http://mail.python.org/mailman/listinfo/python-list

-- 
http://mail.python.org/mailman/listinfo/python-list

Problem with python 3.2 and circular imports

[Frank Millman]

Hi all

I thought I was getting the hang of circular imports, but after upgrading to 
python 3.2 I am stumped again. I know some people think that circular 
imports are always bad, but others suggest that, provided you understand the 
potential problems, they can be acceptable.


Assume the following structure -

main.py
/pkg
   __init__.py
   mod1.py
   mod2.py

main.py
   from pkg import mod1

mod1.py
   import mod2

mod2.py
 import mod1

Using python 2.6, running main.py works.

After running 2to3.py on the above directory, mod1.py was changed to 'from . 
import mod2' and mod2.py was changed to 'from . import mod1'.


With python 3.2, it now fails with the following traceback -

Traceback (most recent call last):
 File main.py, line 1, in module
   from pkg import mod1
 File pkg\mod1.py, line 1, in module
   from . import mod2
 File pkg\mod2.py, line 1, in module
   from . import mod1
ImportError: cannot import name mod1

I have read the relevant peps and various other material, understood them to 
a certain extent, tried several alternatives, but could not find a solution.


I have found a hack that works, but I don't like it very much. I added the 
following to '__init__.py' -

 import os
 import sys
 sys.path.insert(0, os.path.dirname(__file__))

This adds the package name into the search path.

Then I changed mod1.py and mod2.py back to the original 'import mod2' and 
'import mod1'.


It works, but it seems to be defeating the purpose of PEP 328, which I 
thought was an improvement.


Any comments or suggestions will be appreciated.

Frank Millman


--
http://mail.python.org/mailman/listinfo/python-list

Re: Problem with python 3.2 and circular imports

[Ben Finney]

Frank Millman fr...@chagford.com writes:

 Assume the following structure -

 main.py
 /pkg
__init__.py
mod1.py
mod2.py

 main.py
from pkg import mod1

 mod1.py
import mod2

 mod2.py
  import mod1

What are you expecting the result to be?

If it's about sharing objects between the modules, why not break the
circular dependency: factor out the common code to a module that both
the others import?

 Any comments or suggestions will be appreciated.

Special cases aren't special enough to break the rules. If you think you
have a practical reason to do so, it would be best to make it explicit
when asking for help about this.

-- 
 \   “The long-term solution to mountains of waste is not more |
  `\  landfill sites but fewer shopping centres.” —Clive Hamilton, |
_o__)_Affluenza_, 2005 |
Ben Finney
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Problem with python 3.2 and circular imports

[Frank Millman]

Ben Finney ben+pyt...@benfinney.id.au wrote in message 
news:87aahh6401@benfinney.id.au...

Frank Millman fr...@chagford.com writes:


Assume the following structure -

main.py
/pkg
   __init__.py
   mod1.py
   mod2.py

main.py
   from pkg import mod1

mod1.py
   import mod2

mod2.py
 import mod1


What are you expecting the result to be?



Simply that mod1 can refer to objects in mod2, and mod2 can refer to objects 
in mod1.



If it's about sharing objects between the modules, why not break the
circular dependency: factor out the common code to a module that both
the others import?


Any comments or suggestions will be appreciated.


Special cases aren't special enough to break the rules. If you think you
have a practical reason to do so, it would be best to make it explicit
when asking for help about this.



I am trying to understand what 'the rule' is. Your advice above suggests 
that you are one of those who recommend that circular imports are best 
avoided altogether. In an ideal world I would agree. However, the fact is 
that, no doubt due to a mental block I have, I do find myself in this 
situation from time to time, and I have not seen anything in the 
documentation or other literature that says it is absolutely wrong. 
Therefore, while I do try to avoid circular imports where possible, I would 
also like to know how to manage it in situations where I don't see a simple 
alternative.


From everything I have read about how the import mechanism works, I don't 
understand *why* the above construct fails. That is actually what I am 
asking for help with.


Thanks

Frank


--
http://mail.python.org/mailman/listinfo/python-list

Re: Problem with python 3.2 and circular imports

[Steven D'Aprano]

On Sun, 27 Feb 2011 12:08:12 +0200, Frank Millman wrote:

 Assume the following structure -
 
 main.py
 /pkg
 __init__.py
 mod1.py
 mod2.py
 
 main.py
 from pkg import mod1
 
 mod1.py
 import mod2
 
 mod2.py
   import mod1


If you change the import mod* lines to import pkg.mod* it works for 
me in Python 3.1 and 3.2.

According to my understand of PEP 328, from . import mod* should work, 
but I agree with you that it doesn't.

If you get rid of the circular import, it does work. So I suspect a bug.



-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Problem with python 3.2 and circular imports

[Frank Millman]

Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote in message 
news:4d6a56aa$0$29972$c3e8da3$54964...@news.astraweb.com...

On Sun, 27 Feb 2011 12:08:12 +0200, Frank Millman wrote:


Assume the following structure -

main.py
/pkg
__init__.py
mod1.py
mod2.py

main.py
from pkg import mod1

mod1.py
import mod2

mod2.py
  import mod1



If you change the import mod* lines to import pkg.mod* it works for
me in Python 3.1 and 3.2.

According to my understand of PEP 328, from . import mod* should work,
but I agree with you that it doesn't.

If you get rid of the circular import, it does work. So I suspect a bug.




Thanks, Steven.

I confirm that 'import pkg.mod* works. Unfortunately I am using sub-packages 
as well, which means that to refer to an object in the sub-package I need to 
use w.x.y.z every time, which gets to be a lot of typing! I will stick to my 
hack of putting the package name in sys.path for now, unless someone comes 
up with a better idea.


Frank


--
http://mail.python.org/mailman/listinfo/python-list

Re: Class changes in circular imports when __name__ == '__main__'

[Spencer Pearson]

All right, thank you for helping! I'd had a little voice in the back
of my mind nagging me that it might not be logical to include a bunch
of classes and function definitions in my startup file, but I never
got around to splitting it up. The module/script distinction makes
sense, and it seems more elegant, too. Also, my program works now that
I've rearranged things, which is a plus. Thanks!
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Class changes in circular imports when __name__ == '__main__'

[Dave Angel]

On 2:59 PM, Carl Banks wrote:

On Sep 5, 5:07 pm, Dave Angelda...@ieee.org  wrote:

On 2:59 PM, Carl Banks wrote:

All of this gets a lot more complicated when packages are involved.

Perhaps a better answer would be to import __main__ from the second module.

Then what if the module is imported from a different script?  It'll
try to import __main__ but get a different script than expected.


Then the module needs to adjust its expectations.  The point is it 
should never try to import the script by name.


DaveA

--
http://mail.python.org/mailman/listinfo/python-list

Re: Class changes in circular imports when __name__ == '__main__'

[Carl Banks]

On Sep 6, 4:44 pm, Dave Angel da...@ieee.org wrote:
 On 2:59 PM, Carl Banks wrote: On Sep 5, 5:07 pm, Dave Angelda...@ieee.org  
 wrote:
  On 2:59 PM, Carl Banks wrote:
  All of this gets a lot more complicated when packages are involved.
  Perhaps a better answer would be to import __main__ from the second module.
  Then what if the module is imported from a different script?  It'll
  try to import __main__ but get a different script than expected.

 Then the module needs to adjust its expectations.

No, it shouldn't.  It shouldn't have any expectations at all, because
importing __main__ and expecting to get a particular module is a
foolish thing to do.  There are a bunch of reasons why __main__ might
not be the original script.  Example: running the profiler on it.


 The point is it
 should never try to import the script by name.

Importing __main__ directly is worse than the problem it's trying to
solve.

And never is too strong a word.  I already in this thread gave a
solution whereby the script can be imported by name safely, by
renaming itself and assigning itself an item in sys.modules.  When you
do that, you can import the main script by name.


Carl Banks
-- 
http://mail.python.org/mailman/listinfo/python-list

Class changes in circular imports when __name__ == '__main__'

[Spencer Pearson]

Hi! I'm writing a package with several files in it, and I've found
that isinstance doesn't work the way I expect under certain
circumstances.

Short example: here are two files.
# fileone.py
import filetwo

class AClass( object ):
  pass

if __name__ == '__main__':
  a = AClass()
  filetwo.is_aclass( a )

# filetwo.py

import fileone

def is_aclass( a ):
  print The argument is, ( if isinstance(a, fileone.AClass) else
not), an instance of fileone.AClass


If you run fileone.py, it will tell you that The argument is not an
instance of fileone.AClass, which seems strange to me, given that the
fileone module is the one that CREATES the object with its own AClass
class. And if you replace if __name__ == '__main__' with def
main(), start Python, import fileone, and call fileone.main(), it
tells you that the argument IS an instance of AClass.

So, the module's name change to __main__ when you run it on its own...
well, it looks like it puts all of the things defined in fileone in
the __main__ namespace INSTEAD of in the fileone module's namespace,
and then when filetwo imports fileone, the class is created again,
this time as fileone.AClass, and though it's identical in function to
__main__.AClass, one is not the other.

Is this kind of doubled-back 'isinstance' inherently sinful? I mean, I
could solve this problem by giving all of my classes classname
attributes or something, but maybe it's just a sign that I shouldn't
have to do this in the first place.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Class changes in circular imports when __name__ == '__main__'

[Arnaud Delobelle]

Spencer Pearson speeze.pear...@gmail.com writes:

 Hi! I'm writing a package with several files in it, and I've found
 that isinstance doesn't work the way I expect under certain
 circumstances.

 Short example: here are two files.
 # fileone.py
 import filetwo

 class AClass( object ):
   pass

 if __name__ == '__main__':
   a = AClass()
   filetwo.is_aclass( a )

 # filetwo.py

 import fileone

 def is_aclass( a ):
   print The argument is, ( if isinstance(a, fileone.AClass) else
 not), an instance of fileone.AClass


 If you run fileone.py, it will tell you that The argument is not an
 instance of fileone.AClass, which seems strange to me, given that the
 fileone module is the one that CREATES the object with its own AClass
 class. And if you replace if __name__ == '__main__' with def
 main(), start Python, import fileone, and call fileone.main(), it
 tells you that the argument IS an instance of AClass.

 So, the module's name change to __main__ when you run it on its own...
 well, it looks like it puts all of the things defined in fileone in
 the __main__ namespace INSTEAD of in the fileone module's namespace,
 and then when filetwo imports fileone, the class is created again,
 this time as fileone.AClass, and though it's identical in function to
 __main__.AClass, one is not the other.

 Is this kind of doubled-back 'isinstance' inherently sinful? I mean, I
 could solve this problem by giving all of my classes classname
 attributes or something, but maybe it's just a sign that I shouldn't
 have to do this in the first place.

The behaviour is normal.  I suppose you could do something like this
(untested):

# fileone.py

if __name__ == '__main__':
  from fileone import *
  
  a = AClass()
  filetwo.is_aclass( a )
  
  import sys; sys.exit()

import filetwo

class AClass( object ):
  pass

-- 
Arnaud
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Class changes in circular imports when __name__ == '__main__'

[Carl Banks]

On Sep 5, 1:19 pm, Spencer Pearson speeze.pear...@gmail.com wrote:
 Hi! I'm writing a package with several files in it, and I've found
 that isinstance doesn't work the way I expect under certain
 circumstances.

 Short example: here are two files.
 # fileone.py
 import filetwo

 class AClass( object ):
   pass

 if __name__ == '__main__':
   a = AClass()
   filetwo.is_aclass( a )

 # filetwo.py

 import fileone

 def is_aclass( a ):
   print The argument is, ( if isinstance(a, fileone.AClass) else
 not), an instance of fileone.AClass

 If you run fileone.py, it will tell you that The argument is not an
 instance of fileone.AClass, which seems strange to me, given that the
 fileone module is the one that CREATES the object with its own AClass
 class. And if you replace if __name__ == '__main__' with def
 main(), start Python, import fileone, and call fileone.main(), it
 tells you that the argument IS an instance of AClass.

 So, the module's name change to __main__ when you run it on its own...
 well, it looks like it puts all of the things defined in fileone in
 the __main__ namespace INSTEAD of in the fileone module's namespace,
 and then when filetwo imports fileone, the class is created again,
 this time as fileone.AClass, and though it's identical in function to
 __main__.AClass, one is not the other.

Correct.  Python always treats the main script as a module called
__main__.  If you then try to import the main script file from another
module, Python will actually import it again with whatever its usual
name is.

This is easily one of the most confusing and unfortunate aspects of
Python.


 Is this kind of doubled-back 'isinstance' inherently sinful? I mean, I
 could solve this problem by giving all of my classes classname
 attributes or something, but maybe it's just a sign that I shouldn't
 have to do this in the first place.

Even if there are better ways than isinstance, the weird behavior of
__main__ shouldn't be the reason not to use it.

My recommendation for most programmers is to treat Python files either
as scripts (which you start Python interpreter with) or modules (which
you import from within Python); never both.  Store most functionality
in modules and keep startup scripts small.  If you do this, the weird
semantics of __main__ is a moot point.

If you want to be able to run a module as a script while avoiding side
effects due to it being named __main__, the easiest thing to do is to
put something like the following boilerplate at the top of the module
(this causes the module to rename itself).

import sys
if __name__ == '__main__':
is_main = True # since you're overwriting __name__ you'll need
this later
__name__ = 'foo'
sys.modules['foo'] = sys.modules['__main__']
else:
is_main = False


All of this gets a lot more complicated when packages are involved.


Carl Banks
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Class changes in circular imports when __name__ == '__main__'

[Dave Angel]

On 2:59 PM, Carl Banks wrote:

On Sep 5, 1:19 pm, Spencer Pearsonspeeze.pear...@gmail.com  wrote:

Hi! I'm writing a package with several files in it, and I've found
that isinstance doesn't work the way I expect under certain
circumstances.

Short example: here are two files.
# fileone.py
import filetwo

class AClass( object ):
   pass

if __name__ ='__main__':
   a =Class()
   filetwo.is_aclass( a )

# filetwo.py

import fileone

def is_aclass( a ):
   print The argument is, ( if isinstance(a, fileone.AClass) else
not), an instance of fileone.AClass

If you run fileone.py, it will tell you that The argument is not an
instance of fileone.AClass, which seems strange to me, given that the
fileone module is the one that CREATES the object with its own AClass
class. And if you replace if __name__ ='__main__' with def
main(), start Python, import fileone, and call fileone.main(), it
tells you that the argument IS an instance of AClass.

So, the module's name change to __main__ when you run it on its own...
well, it looks like it puts all of the things defined in fileone in
the __main__ namespace INSTEAD of in the fileone module's namespace,
and then when filetwo imports fileone, the class is created again,
this time as fileone.AClass, and though it's identical in function to
__main__.AClass, one is not the other.

Correct.  Python always treats the main script as a module called
__main__.  If you then try to import the main script file from another
module, Python will actually import it again with whatever its usual
name is.

This is easily one of the most confusing and unfortunate aspects of
Python.



Is this kind of doubled-back 'isinstance' inherently sinful? I mean, I
could solve this problem by giving all of my classes classname
attributes or something, but maybe it's just a sign that I shouldn't
have to do this in the first place.

Even if there are better ways than isinstance, the weird behavior of
__main__ shouldn't be the reason not to use it.

My recommendation for most programmers is to treat Python files either
as scripts (which you start Python interpreter with) or modules (which
you import from within Python); never both.  Store most functionality
in modules and keep startup scripts small.  If you do this, the weird
semantics of __main__ is a moot point.

If you want to be able to run a module as a script while avoiding side
effects due to it being named __main__, the easiest thing to do is to
put something like the following boilerplate at the top of the module
(this causes the module to rename itself).

import sys
if __name__ ='__main__':
 is_main =rue # since you're overwriting __name__ you'll need
this later
 __name__ =foo'
 sys.modules['foo'] =ys.modules['__main__']
else:
 is_main =alse


All of this gets a lot more complicated when packages are involved.


Carl Banks


Perhaps a better answer would be to import __main__ from the second module.

But to my way of thinking, the answer should be to avoid ever having 
circular imports.  This is just the most blatant of the problems that 
circular imports can cause.


I don't know of any cases where circular dependencies are really 
necessary, but if one decides to use them, then two things should be done:


1) do almost nothing in top-level code in any module involved in such 
circular dependency.  Top-level should have all of the imports, and none 
of the executable code.
2) do not ever involve the startup script in the loop.  If necessary, 
make it two lines, importing,then calling the real mainline.


DaveA

--
http://mail.python.org/mailman/listinfo/python-list

Re: Class changes in circular imports when __name__ == '__main__'

[Carl Banks]

On Sep 5, 5:07 pm, Dave Angel da...@ieee.org wrote:
 On 2:59 PM, Carl Banks wrote:
  All of this gets a lot more complicated when packages are involved.

 Perhaps a better answer would be to import __main__ from the second module.

Then what if the module is imported from a different script?  It'll
try to import __main__ but get a different script than expected.


 But to my way of thinking, the answer should be to avoid ever having
 circular imports.  This is just the most blatant of the problems that
 circular imports can cause.

 I don't know of any cases where circular dependencies are really
 necessary, but if one decides to use them, then two things should be done:

I don't think they're ever necessary but sometimes it's convenient.
This could be one of those cases.  One of the less misguided reasons
to invoke a module as a script is to run tests on the module.  When
you do that you might need to call an outside module to set up a test
environment, and that module might happen to import the calling
module.  You could refactor the test to avoid the circular import, but
that kind of defeats the convenience of stowing the test in the
module.


 1) do almost nothing in top-level code in any module involved in such
 circular dependency.  Top-level should have all of the imports, and none
 of the executable code.
 2) do not ever involve the startup script in the loop.  If necessary,
 make it two lines, importing,then calling the real mainline.

All good advice for that situation. I would add that if you define a
base class in one module and subclass in another, you want to keep
those modules out of cycles.  The problem with circular imports is
that you don't usually know what order the modules will be imported,
but you need to be sure the base class is defined when you subclass.
(I learned that lesson the hard way, and I had to hack up an import
hook to enforce that imports occurred in the correct order.)


Carl Banks
-- 
http://mail.python.org/mailman/listinfo/python-list

[issue9657] Add circular imports test

[Nick Coghlan]

Nick Coghlan ncogh...@gmail.com added the comment:

It would be good if the test timed out rather than deadlocking in the face of a 
broken import lock.

I suggest:
1. Make the two test threads daemon threads
2. Specify a timeout to the join() calls
3. Use self.assertFalse(t1.is_alive()) and self.assertFalse(t2.is_alive()) to 
check the calls actually finished

For your #9260 testing, this looks like it may be a little probabilistic. I 
think you can make the deadlock a near certainty by defining your test modules 
and threads as follows:

# Create a circular import structure: A - C - B - D - A
circular_imports_modules = {
'A': if 1:
import ev
ev.evA.wait()
import time
time.sleep(%(delay)s)
x = 'a'
import C
,
'B': if 1:
import ev
ev.evB.wait()
import time
time.sleep(%(delay)s)
x = 'b'
import D
,
'C': import B,
'D': import A,
'ev': if 1:
 import threading;
 evA = threading.Event()
 evB = threading.Event()
 ,
}

def import_ab():
import ev
ev.evB.set()
import A
results.append(getattr(A, 'x', None))
def import_ba():
import ev
ev.evA.set()
import B
results.append(getattr(B, 'x', None))


(I've done a trick along these lines before, and the above doesn't look quite 
right. Maybe the details will come back to me if I sit on it for a while)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9657
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9657] Add circular imports test

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

 It would be good if the test timed out rather than deadlocking in the
 face of a broken import lock.

I could do that indeed. I'm not sure it would be much better, though, because 
the lock will still be held and posterior tests may freeze mysteriously.

 For your #9260 testing, this looks like it may be a little probabilistic

It always freezes here (with the per-module lock). I guess on modern machines, 
0.5s is more than enough to launch a thread and parse a nearly empty Python 
module.

I should add a comment explaining a purpose of the test, though.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9657
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9657] Add circular imports test

[Nick Coghlan]

Nick Coghlan ncogh...@gmail.com added the comment:

Both good points - don't worry about it then. +1 to add to the test suite as-is.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9657
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9657] Add circular imports test

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

Thank you Nick! I've committed the patch in r84268 (py3k) and r84269 (3.1).

--
resolution:  - fixed
stage: patch review - committed/rejected
status: open - closed
versions: +Python 3.1

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9657
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9657] Add circular imports test

[Antoine Pitrou]

New submission from Antoine Pitrou pit...@free.fr:

Here is a test for multi-threaded circular imports (in order to exercise the 
import lock), following a comment by Graham Dumpleton on #9260.

--
components: Tests
files: circulartest.patch
keywords: patch
messages: 114541
nosy: brett.cannon, ncoghlan, pitrou
priority: normal
severity: normal
stage: patch review
status: open
title: Add circular imports test
type: behavior
versions: Python 3.2
Added file: http://bugs.python.org/file18600/circulartest.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9657
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: Circular imports (again)

[Frank Millman]

Frank Millman fr...@chagford.com wrote in message 
news:i3ov9e$du...@dough.gmane.org...

Hi all

I know the problems related to circular imports, and I know some of the 
techniques to get around them. However, I find that I bump my head into 
them from time to time, which means, I guess, that I have not fully 
understood how to organise my code so that I avoid them in the first 
place.



[...]


So I think my main question is, is this a symptom of a flaw in my 
approach, or is this something that all programmers bump into from time to 
time?




Thanks for the replies. All good info, but it was Ethan that put me onto the 
right track.


I omitted to mention one thing originally, as I did not think it important, 
but it turns out to be crucial. My code is organised into three 'packages', 
not 'modules'.


To reproduce my situation, I did some tests with the following hierarchy -

top/
   a.py
   /bb
   __init__.py
   b.py
   /cc
   __init__.py
   c.py

a.py

import bb.b
import cc.c
bb.b.foo()
cc.c.foo()

b.py

import cc.c
def foo():
   print 'in b.foo, call c.bar'
   cc.c.bar()
def bar():
   print '  bar in b'

c.py

import bb.b
def foo():
   print 'in c.foo, call b.bar'
   bb.b.bar()
def bar():
   print '  bar in c'

If I run 'a.py', I get the correct result -

in b.foo, call c.bar
 bar in c
in c.foo, call b.bar
 bar in b

I changed 'a.py' -

a.py

from bb import b
from cc import c
b.foo()
c.foo()

It still worked.

Next I changed 'b.py' -

b.py

from cc import c
def foo():
   print 'in b.foo, call c.bar'
   c.bar()
def bar():
   print '  bar in b'

It still worked.

Then I changed 'c.py' -

c.py

from bb import b
def foo():
   print 'in b.foo, call c.bar'
   b.bar()
def bar():
   print '  bar in b'

Now I get the following traceback -
Traceback (most recent call last):
 File F:\dd\a.py, line 1, in module
   from bb import b
 File F:\dd\bb\b.py, line 1, in module
   from cc import c
 File F:\dd\cc\c.py, line 1, in module
   from bb import b
ImportError: cannot import name b

Now that I understand this, I can work around my problem by using 
fully-qualified module names.


But it would be interesting to know the underlying reason for this 
behaviour.


I am using python 2.6.2.

Thanks for any insights.

Frank


--
http://mail.python.org/mailman/listinfo/python-list

Circular imports (again)

[Frank Millman]

Hi all

I know the problems related to circular imports, and I know some of the 
techniques to get around them. However, I find that I bump my head into them 
from time to time, which means, I guess, that I have not fully understood 
how to organise my code so that I avoid them in the first place.


It has just happened again. I have organised my code into three modules, 
each representing a fairly cohesive functional area of the overall 
application. However, there really are times when Module A wants to invoke 
something from Module B, ditto for B and C, and ditto for C and A.


I can think of two workarounds. One is to place the import statement inside 
the function that actually requires it. It is therefore not executed when 
the module itself is imported, thereby avoiding the problem. It works, but 
breaks the convention that all imports should be declared at the top of the 
program.


A second solution is to avoid invoking the other modules directly, but 
rather use global Queue.Queues to pass requests from one module to another. 
Again, it works, but adds complication, especially if the 'invoker' needs to 
get a return value.


So I think my main question is, is this a symptom of a flaw in my approach, 
or is this something that all programmers bump into from time to time?


If the former, I can expand on my current requirement if anyone would like 
to suggest a better approach. If the latter, is either of the above 
solutions preferred, or are there other techniques to get around the 
problem.


Any suggestions will be appreciated.

Thanks

Frank Millman


--
http://mail.python.org/mailman/listinfo/python-list

Re: Circular imports (again)

[Ethan Furman]

Frank Millman wrote:

Hi all

I know the problems related to circular imports...


It has just happened again. I have organised my code into three modules, 
each representing a fairly cohesive functional area of the overall 
application. However, there really are times when Module A wants to 
invoke something from Module B, ditto for B and C, and ditto for C and A.


I believe the issue arises when you have top-level code (module global 
code, or unindented code for the visual) that is calling the other 
module.  If you keep your references to the other module in functions, 
you should be able to have your imports at module level.


The below works fine.

a.py
-
import b
def spam():
b.eggs()
def foo():
print my circular-import-fu is strong! # to amuse myself only!)
-

b.py
-
import a
def eggs():
print 'sunnyside?'
def ham():
a.foo()
-

If you put an a.spam() in b.py, then you get:

Traceback (most recent call last):
  File stdin, line 1, in module
  File a.py, line 1, in module
import b
  File b.py, line 3, in module
a.spam()
AttributeError: 'module' object has no attribute 'spam'


Hope this helps.

~Ethan~
--
http://mail.python.org/mailman/listinfo/python-list

Re: Circular imports (again)

[Carl Banks]

On Aug 9, 6:19 am, Frank Millman fr...@chagford.com wrote:
 It has just happened again. I have organised my code into three modules,
 each representing a fairly cohesive functional area of the overall
 application. However, there really are times when Module A wants to invoke
 something from Module B, ditto for B and C, and ditto for C and A.

There's a dozen reasons why circular imports can go wrong.  Can you
describe the problem you're having getting them to work?  If there's a
traceback when you try to do it, cut-and-paste the traceback and
relevant code here.  If it's running, but you are not getting the
behavior you expect, tell us what you expected to happen and what
actually did.


[snip]
 So I think my main question is, is this a symptom of a flaw in my approach,
 or is this something that all programmers bump into from time to time?

I consider the need to resort to circular imports a red flag, not a
manifest error.  You say you are organizing the modules
functionally.  I find that when I do that, I will occasionally get
circular references, so I can believe you.  I also find that in most
such cases, reorganizing the modules according to dependencies, so
as to remove the circular import, makes it more confusing.  Sometimes
organizing by function makes more sense than organizing by dependency,
and it's better to live with circular imports than to use a less-
sensical organization.


Carl Banks
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Circular imports (again)

[Michael Torrie]

On Aug 9, 6:19 am, Frank Millman fr...@chagford.com wrote:
 It has just happened again. I have organised my code into three modules,
 each representing a fairly cohesive functional area of the overall
 application. However, there really are times when Module A wants to invoke
 something from Module B, ditto for B and C, and ditto for C and A.

There are a number of ways to avoid circular imports, in order of my own
preference:

1. Make common stuff a new module.  So if A needs something from B, and
vice versa, you must factor out the stuff and stick it in its own
module.  The fact that you have circular dependencies means that
although things are cohesive, they are way too closely coupled.

2. Instead of having A refer directly to something in B (which would
cause a circular dependency, have the caller pass in as a parameter to
the function in A, whatever is needed from B.  This could be just a
variable, complex object, or even a function or method.  Take advantage
of the fact that everything in Python is a first-class object.

3. Take advantage of Python's dynamicism.  Write an initializer function
in A that allows you to tell it about B and C.  In other words, you can
pass B and C to some method in A and have it bind B and C to local
attributes in A.   Then you can call B and C's methods just fine from A
since everything is looked up as it is called.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Circular imports

[jim-on-linux]

On Tuesday 29 May 2007 00:08, Carsten Haese wrote:
 On Mon, 28 May 2007 23:46:00 -0400, Ron Provost
 wrote

  [...] python is not happy about my circular
  imports [...]

 A circular import is not a problem in itself.
 I'm guessing you're running into a situation
 like this:

 Module A imports module B and then defines an
 important class. Module B imports A (which does
 nothing because A is already partially
 imported) and then defines a class that depends
 on the class that is defined in module A. That
 causes a NameError.

 The root of the problem is that all statements
 are executed in the order in which they appear,
 and B is imported before A had a chance to
 define the class that B depends on.

 Note that import statements don't have to be at
 the top of the file. 

I agree, waite until python complains.

You might try to remove all of the import 
statements then add then as they are requested by 
the program by a traceback error.


jim-on-linux


 Try moving each import 
 statement to the latest possible point in the
 code, i.e. right before the first occurence of
 whatever names you're importing from the
 respective modules. That way, each module gets
 to define as much as it possibly can before it
 gets side-tracked by importing other modules.

 If my guess doesn't help, you're going to have
 to post at least the exception/traceback you're
 getting, and you're probably going to have to
 post some code, too.

 Good luck,

 --
 Carsten Haese
 http://informixdb.sourceforge.net
-- 
http://mail.python.org/mailman/listinfo/python-list

Circular imports

[Ron Provost]

Hello,

I'm in the process of porting a project from Java to python.  The original 
project is very poorly organized.  Nearly each class is depentant upon nearly 
every other class.  Before I go playing around with the logic by moving things 
around and cleaning up the code, I want to get it working.  However, because of 
all these dependancies, python is not happy about my circular imports.  Is 
there any way to force python to accept these imports for now?  the only 
solution I've come up with myself would be to combine all the files into one.  
This isn't really possible anyway because three of the files are actually 
automatically generated.

Thanks,
Ron-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Circular imports

[Carsten Haese]

On Mon, 28 May 2007 23:46:00 -0400, Ron Provost wrote 
 [...] python is not happy about my circular imports [...]

A circular import is not a problem in itself. I'm guessing you're running into
a situation like this:

Module A imports module B and then defines an important class. Module B
imports A (which does nothing because A is already partially imported) and
then defines a class that depends on the class that is defined in module A.
That causes a NameError.

The root of the problem is that all statements are executed in the order in
which they appear, and B is imported before A had a chance to define the class
that B depends on.

Note that import statements don't have to be at the top of the file. Try
moving each import statement to the latest possible point in the code, i.e.
right before the first occurence of whatever names you're importing from the
respective modules. That way, each module gets to define as much as it
possibly can before it gets side-tracked by importing other modules.

If my guess doesn't help, you're going to have to post at least the
exception/traceback you're getting, and you're probably going to have to post
some code, too.

Good luck,

--
Carsten Haese
http://informixdb.sourceforge.net

-- 
http://mail.python.org/mailman/listinfo/python-list

circular imports

[qhfgva]

I'm working with a large code base that I'm slowly trying to fix
unpythonic features of.

One feature I'm trying to fix is the use of:

# how things are now
sys.path.append('/general/path/aaa/bbb') # lots of lines like this to
specific dirs
import foo

Insead I'd rather have PYTHONPATH already include '/general/path/'
and then just use:

# how I'd like them to be
from aaa.bbb import foo

So I thought I'd just add the necessary __init__.py files and then
things would just work.  Unfortunately trying this exposed a large
number of circular imports which now cause the files to fail to load.

Any ideas why the sys.path.append method has no problem with circular
imports whereas doing thing the right way chokes.

thanks,

dustin

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: circular imports

[Martin v. Löwis]

[EMAIL PROTECTED] wrote:
 So I thought I'd just add the necessary __init__.py files and then
 things would just work.  Unfortunately trying this exposed a large
 number of circular imports which now cause the files to fail to load.

You didn't describe you you created the necessary __init__.py files.
If they are not empty, there is a good chance that you made some
error in creating these files.

What is the error that you get with the circular imports?

 Any ideas why the sys.path.append method has no problem with circular
 imports whereas doing thing the right way chokes.

One reason is that there could be duplicate module names, e.g. overlaps
with the standard library. When you fully-qualified the modules, you
actually changed the program, which would now import the user-provided
modules instead of the predefined ones.

Regards,
Martin
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: circular imports

[qhfgva]

All of the __init__.py files are empty and I don't know of any
overlapping of names.  Like I said this is code that works fine, I'm
just trying to clean up some things as I go.  Here are my working
examples:

x1.py
==
# how things work in our code now:
# called with home/dlee/test/module python aaa/x1.py
import sys
sys.path.append('/home/dlee/test/module')

import x2

def goo():
print 'hi from goo'

if __name__ == '__main__':
x2.foo()

x2.py
==
import sys
sys.path.append('/home/dlee/test/module')
import x1

def foo():
print 'entered foo'
x1.goo()


y1.py
==
# this works but is not quite what I want
# called with PYTHONPATH=$PYTHONPATH:/home/dlee/test/module python
aaa/y1.py

import aaa.y2

def goo():
print 'hi from goo'

if __name__ == '__main__':
aaa.y2.foo()


y2.py
==
import aaa.y1

def foo():
print 'entered foo'
aaa.y1.goo()


z1.py
==
# how I'd like things to work, but is failing for me
# called with PYTHONPATH=$PYTHONPATH:/home/dlee/test/module python
aaa/z1.py

from aaa import z2

def goo():
print 'hi from goo'

if __name__ == '__main__':
z2.foo()


z2.py
==
om aaa import z1

def foo():
print 'entered foo'
z1.goo()




w1.py
==
# this would also be acceptible
# called with PYTHONPATH=$PYTHONPATH:/home/dlee/test/module python
aaa/w1.py

import aaa.w2 as w2

def goo():
print 'hi from goo'

if __name__ == '__main__':
w2.foo()


w2.py
==
import aaa.w1 as w1

def foo():
print 'entered foo'
w1.goo()

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: circular imports

[Peter Hansen]

[EMAIL PROTECTED] wrote:
 I'm working with a large code base that I'm slowly trying to fix
 unpythonic features of.
[...]
 Insead I'd rather have PYTHONPATH already include '/general/path/'
 and then just use:

One option you might not have considered, which I find more pythonic 
than environment variables, is to use .pth files as documented at the 
top of the site.py module in the standard library (or in the online docs 
for site).

You can also sometimes remove even more problems by using a 
sitecustomize.py file in addition to the .pth files.  (Also in the docs, 
I believe, under site.)

-Peter
-- 
http://mail.python.org/mailman/listinfo/python-list