New submission from Anentropic <e...@anentropic.com>:
We have a Django 2.2.19 project on Python 3.9.2 on Debian (slim-buster) in
Docker.
A bizarre problem started happening to us this week.
First I'll show the symptom, we started getting the following error:
...
File
"/root/.pyenv/versions/3.9.2/lib/python3.9/site-packages/django/db/migrations/autodetector.py",
line 10, in <module>
from django.db.migrations.optimizer import MigrationOptimizer
File "<frozen importlib._bootstrap>", line 1004, in _find_and_load
File "<frozen importlib._bootstrap>", line 158, in __enter__
File "<frozen importlib._bootstrap>", line 110, in acquire
KeyError: 140426340123264
If I look at the source for _bootstrap.py, that error should be impossible:
https://github.com/python/cpython/blob/v3.9.2/Lib/importlib/_bootstrap.py#L110
At the top of the acquire method it does:
tid = _thread.get_ident()
_blocking_on[tid] = self
and then on line 110 where we get the KeyError:
del _blocking_on[tid]
both `tid` and `_blocking_on` are local vars and none of the other lines in the
method touch them
So how do we get a KeyError?
I can only think that something mutates the underlying value of `tid`, but it's
supposed to be an int so that's very weird.
I started with the symptom because our context for this is complicated to
explain. I did find a fix that prevents the error but I do not understand the
link between cause and effect.
Our context:
- we have a large unit test suite for the project which we run in Jenkins
- we split the tests across several Jenkins nodes to run in parallel in
isolated docker environments
- we use some bash to like this to split the test cases:
find project/ -iname "test*.py" -print0 | \
xargs --null grep -E '(def test)|(def step_)' -l | \
split -n "r/$NODE_ID/$NODES" | \
xargs ci/bin/run-tests
- ci/bin/run-tests is just a wrapper which calls Django's manage.py test command
so it receives a list of filenames like
"project/metrics/tests/test_client.py" as args
- using "nose" test runner via django-nose FWIW
We currently split tests across 3 nodes, and it was always node 2 which would
fail.
I found that commenting out a test case in any of the files being passed to
node 2 would prevent the error from occurring.
Note that in this case we are still passing *exactly the same filenames* as cli
args to the test runner.
Splitting the tests across 4 nodes instead of 3 also seems to prevent the error.
So it seems like, in some way I don't understand, we just have too many test
cases.
Perhaps nose is doing something wrong or inefficient when given lots of
filenames.
But I'm reporting here because the error we get from importlib._bootstrap looks
like it should be impossible.
----------
messages: 389034
nosy: anentropic
priority: normal
severity: normal
status: open
title: "Impossible" KeyError from importlib._bootstrap acquire line 110
type: crash
versions: Python 3.9
_______________________________________
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue43546>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
