[issue30814] Import dotted name as alias breaks with concurrency

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
resolution:  -> fixed
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30814] Import dotted name as alias breaks with concurrency

[Serhiy Storchaka]

Serhiy Storchaka added the comment:


New changeset 95b16a9705d6b4d31c016c014e59744fc33d53ea by Serhiy Storchaka in 
branch '3.6':
[3.6] bpo-30814, bpo-30876: Add new import test files to projects. (GH-2851). 
(#2912)
https://github.com/python/cpython/commit/95b16a9705d6b4d31c016c014e59744fc33d53ea


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30814] Import dotted name as alias breaks with concurrency

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
pull_requests: +2963

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30814] Import dotted name as alias breaks with concurrency

[Serhiy Storchaka]

Serhiy Storchaka added the comment:


New changeset d5ed47dea25e04a3a144eddf99a4ac4a29242dbc by Serhiy Storchaka in 
branch 'master':
bpo-30814, bpo-30876: Add new import test files to projects. (#2851)
https://github.com/python/cpython/commit/d5ed47dea25e04a3a144eddf99a4ac4a29242dbc


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30814] Import dotted name as alias breaks with concurrency

[STINNER Victor]

STINNER Victor added the comment:

I also opened https://github.com/python/devguide/issues/241 : "Document how to 
add a new file or a new directory". It's not the first time that we make such 
mistake.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30814] Import dotted name as alias breaks with concurrency

[STINNER Victor]

STINNER Victor added the comment:

Serhiy on PR 2851:
> I forget to include directories with new test files in the list of the 
> library directories. This caused that these test files were not installed.

Oh, too bad that Zach's "x86 Gentoo Installed with X 3.6" buildbot is offline.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30814] Import dotted name as alias breaks with concurrency

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
pull_requests: +2900

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30814] Import dotted name as alias breaks with concurrency

[Ned Deily]

Ned Deily added the comment:

> Which Git revisions are you testing? On master, do you have my latest commit 
> e72b1359f81d1dd42bd8a5c5cc2b3928b74f8023 ?

Yes, this is with current top of trunk of both branches.  Perhaps I wasn't 
clear about "installed location".  The test does not fail if you run it 
directly from the build directory like with "make test".  It fails when you do 
a "make install" and then run the test using the newly installed Python.  Tests 
need to work both ways.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30814] Import dotted name as alias breaks with concurrency

[STINNER Victor]

STINNER Victor added the comment:

> When running tests from installed location, test_import now fails on master 
> and 3.6 with:

Which Git revisions are you testing? On master, do you have my latest commit 
e72b1359f81d1dd42bd8a5c5cc2b3928b74f8023 ?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30814] Import dotted name as alias breaks with concurrency

[Ned Deily]

Ned Deily added the comment:

When running tests from installed location, test_import now fails on master and 
3.6 with:

==
ERROR: test_concurrency (test.test_import.ImportTests)
--
Traceback (most recent call last):
  File 
"/Users/nad/Projects/PyDev/active/dev/3x/root/uxd/lib/python3.7/test/test_import/__init__.py",
 line 408, in test_concurrency
raise exc
  File 
"/Users/nad/Projects/PyDev/active/dev/3x/root/uxd/lib/python3.7/test/test_import/__init__.py",
 line 393, in run
import package
ModuleNotFoundError: No module named 'package'

--
nosy: +ned.deily

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30814] Import dotted name as alias breaks with concurrency

[STINNER Victor]

Changes by STINNER Victor :


--
pull_requests: +2851

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30814] Import dotted name as alias breaks with concurrency

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

It is worth to backport the test to older versions. Just to be sure that this 
is a 3.6 regression.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30814] Import dotted name as alias breaks with concurrency

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
pull_requests: +2710

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30814] Import dotted name as alias breaks with concurrency

[STINNER Victor]

STINNER Victor added the comment:

test_concurrency() of test_import fails randomly on Windows: see bpo-30891.

--
nosy: +haypo
resolution: fixed -> 
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30814] Import dotted name as alias breaks with concurrency

[Brett Cannon]

Brett Cannon added the comment:

I just wanted to say thanks to everyone who helped to fix this bug. The locking 
situation in import is probably the trickiest part of it and has also been 
tweaked the most as of late and so solving these kinds of issues is tricky.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30814] Import dotted name as alias breaks with concurrency

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30814] Import dotted name as alias breaks with concurrency

[Serhiy Storchaka]

Serhiy Storchaka added the comment:


New changeset 03b0e8374b2ea93adf8fb6a48db2916f3b0388cc by Serhiy Storchaka in 
branch '3.6':
[3.6] bpo-30814: Fixed a race condition when import a submodule from a package. 
(GH-2580). (#2598)
https://github.com/python/cpython/commit/03b0e8374b2ea93adf8fb6a48db2916f3b0388cc


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30814] Import dotted name as alias breaks with concurrency

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
pull_requests: +2666

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30814] Import dotted name as alias breaks with concurrency

[Serhiy Storchaka]

Serhiy Storchaka added the comment:


New changeset b4baacee1adc06edbe30ac7574d17a8cd168e2e0 by Serhiy Storchaka in 
branch 'master':
bpo-30814: Fixed a race condition when import a submodule from a package. 
(#2580)
https://github.com/python/cpython/commit/b4baacee1adc06edbe30ac7574d17a8cd168e2e0


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30814] Import dotted name as alias breaks with concurrency

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

I have came to the same conclusion.

PR 2580 adds a double check in _bootstrap._find_and_load() (and also moves the 
locking into it). It also simplifies the C code by removing the fast path for 
the case sys.module[name] is None.

--
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30814] Import dotted name as alias breaks with concurrency

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
pull_requests: +2650

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30814] Import dotted name as alias breaks with concurrency

[Nick Coghlan]

Nick Coghlan added the comment:

Documenting explicitly what I believe the expected order of module lock 
acquisition would be in this case:

Thread A acquires package.subpackage lock
Thread B blocks on package.subpackage lock
Thread A acquires package lock
Thread A releases package lock
Thread A releases package.subpackage lock
Thread B acquires & immediately releases package.subpackage lock

Looking at https://hg.python.org/cpython/rev/64f195790a3a#l4.367 (the commit 
for issue 22557), there's a potentially suspect change in the scope of a 
"_PyImport_AcquireLock/_PyImport_ReleaseLock" pair inside 
PyImport_ImportModuleLevelObject

Specifically, I think we may now have a race condition at 
https://github.com/python/cpython/blob/master/Python/import.c#L1534, where two 
threads can *both* end up trying to initialize the same module, since we're no 
longer holding the global import lock around that "mod = 
PyDict_GetItem(interp->modules, abs_name);" call and the associated state 
updates where the first thread indicates that it is already initializing that 
module so the second thread should just wait for it to finish doing so.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30814] Import dotted name as alias breaks with concurrency

[Antoine Pitrou]

Changes by Antoine Pitrou :


--
priority: normal -> high

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30814] Import dotted name as alias breaks with concurrency

[Brett Cannon]

Brett Cannon added the comment:

My guess is the import of package has completed but package.submodule hasn't (I 
don't know what state it would be in, but obviously before the module is 
assigned as an attribute on the package), a context switch occurs, and then the 
assignment fails due to package.submodule not being set on package yet.

I wonder if package.submodule is in sys.modules when the AttributeError is 
triggered? I don't remember when the assignment of a submodule to an attriute 
on a package occurs, but this suggests it's rather late and that's why this is 
failing. Maybe the assignment is outside of the per-module lock being released 
and that is what's causing this?

Hey, at least importing in a thread doesn't deadlock anymore like it used to. ;)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30814] Import dotted name as alias breaks with concurrency

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
assignee:  -> serhiy.storchaka
keywords: +3.6regression
versions: +Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30814] Import dotted name as alias breaks with concurrency

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

On other side, issue23203 fixed just a symptom. The issue is reproduced in 
master when replace "import foobar.submodule as foo" with "import 
foobar.submodule; foo = foobar.submodule".

The regression may be related to issue22557. Either the optimization introduced 
a bug, or it exposed an existing bug by changing timings.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30814] Import dotted name as alias breaks with concurrency

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

See also issue23203.

This was fixed in master in issue30024. Maybe it is worth to backport that 
change to maintained versions?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30814] Import dotted name as alias breaks with concurrency

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
components: +Interpreter Core
nosy: +brett.cannon, eric.snow, ncoghlan, serhiy.storchaka
type:  -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30814] Import dotted name as alias breaks with concurrency

[Frazer McLean]

Changes by Frazer McLean :


--
nosy: +RazerM

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30814] Import dotted name as alias breaks with concurrency

[Samuele Santi]

New submission from Samuele Santi:

I noticed this weird behavior in Python 3.6:

Apparently ``import package.module as _alias`` fails in a threaded environment, 
sometimes raising an ``ImportError``, seemingly at random.

Everything seems to be working perfectly fine:

- On 3.5 and lower
- When *not* defining an alias, eg. ``import package.module``
- When using the ``from package import module as alias`` style


See this repo with code for reproducing the issue:

https://github.com/rshk/python36-import-alias-concurrency-bug


I tested this on the system Python 3.6.1 shipped with Archlinux (both inside 
and outside of a virtualenv), and in the official docker image for 3.6.

This all started here https://github.com/psycopg/psycopg2/issues/550 as others 
were experiencing this issue too.

--
messages: 297380
nosy: Samuele Santi2
priority: normal
severity: normal
status: open
title: Import dotted name as alias breaks with concurrency
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com