Bug#1004107: meson: flaky autopkgtest on armhf: dictionary changed size during iteration -> timeout

[Paul Gevers]

Hi Jussi,

On 21-01-2022 00:05, Jussi Pakkanen wrote:

On Thu, 20 Jan 2022 at 23:33, Paul Gevers <elb...@debian.org> wrote:

I looked at the results of the autopkgtest of you package on armhf
because it was showing up as a regression for the upload of
python-defaults and setuptools. I noticed that the test regularly fails,
what's worse, it also seems to hang as the test is killed because it
hits an autopkgtest timeout.

If we look at the backtrace:

Running tests with 160 workers
Exception in thread Thread-1:
Traceback (most recent call last):
    File "/usr/lib/python3.9/threading.py", line 973, in _bootstrap_inner
      self.run()
    File "/usr/lib/python3.9/concurrent/futures/process.py", line 317, in run
      result_item, is_broken, cause = self.wait_result_broken_or_wakeup()
    File "/usr/lib/python3.9/concurrent/futures/process.py", line 376, in
wait_result_broken_or_wakeup
      worker_sentinels = [p.sentinel for p in self.processes.values()]
    File "/usr/lib/python3.9/concurrent/futures/process.py", line 376, in
<listcomp>
      worker_sentinels = [p.sentinel for p in self.processes.values()]
RuntimeError: dictionary changed size during iteration

This is all Python's internal code. Further, Meson does not do any
fancy threading stuff itself, it uses Python's thread and process
executors to queue up a bunch of work and then wait for it to be done.
According to Python's documentation you don't need to do any locking
or similar to submit new work, you can call the submit method
directly. All of this would seem to indicate that the issue might lie
somewhere in Python's multithreading code. At the very least I have no
idea how I should go about debugging that issue.

Of course, it's still true that it's difficult to troubleshoot.

Paul

Reading the log, it's not clear to me if that backtrace is even related to the hang. I mean, in my (non-Python related) experience if something goes wrong in a parallel process, you can get logs from parallel pieces that fail, while not really causes problems with the actual processing.

OpenPGP_signature
Description: OpenPGP digital signature