New submission from Asheesh Laroia <ashe...@asheesh.org>:
Some parts of test_concurrent_futures make sense to run even on Python builds where multiprocessing.synchronize is absent. At the moment, test_concurrent_futures.py skips all tests in the file if multiprocessing.synchronize is missing from the Python build. I have a patch to enable more tests, which I'll submit as a GitHub pull request. The reason I want this patch is while working on CPython on Android, I saw that test_concurrent_futures refused to run its test suite, and I became very afraid that this meant that asyncio on CPython on Android was doomed. In fact, I was afraid for no reason. I want future people porting CPython to be less scared than I was. :) I discovered that the only part of concurrent.futures that requires multiprocessing.synchronize is concurrent.futures.ProcessPoolExecutor. concurrent.futures.ProcessPoolExecutor already has a way to check at runtime if it can work on the host system, so this patch leans on that. This patch fixes a few other tests to skip themselves in the case that ProcessPoolExecutor cannot function. I tested this by building CPython (with my patch) with adding a configure flag on macOS: $ ./configure --with-pydebug --with-openssl=$(brew --prefix openssl) ac_cv_posix_semaphores_enabled=no Why multiprocessing.synchronize is missing ------------------------------------------ multiprocessing.synchronize imports _multiprocessing.SemLock, which is based on a libc function called sem_open(), which implements named semaphores, which allows the creation of lock-esque counters that can be used from multiple processes. multiprocessing.c currently only creates the SemLock property if sem_open() is considered "working" through a check in ./configure; I've short-circuited that check above to be a "no". On Android, sem_open() exists but simply returns ENOSYS; see its implementation here: https://android.googlesource.com/platform/external/pthreads/+/master/sem_open.c Android isn't the only platform missing sem_open() Past work --------- issue3770 is an older conversation about a related issue. issue3770 has also attracted comments from someone who wants CPython's test suite to run to completion on their platform lacking sem_open() and therefore lacking multiprocessing.synchronize. This patch should solve their problem, I hope. Future work ----------- Automated testing: In order to prevent regressions, I'm interested in hosting a buildbot for at least one year, hopefully in perpetuity, on which we call ./configure with ac_cv_posix_semaphores_enabled=no. I'd like to improve the error message when the user imports multiprocessing.synchronize so it points at some new documentation text that I (or someone) adds to CPython, rather than the issue number. I'm OK taking on the work of writing those docs in a follow-up issue if we merge this. Thanks ------ Thanks to Zachary Ware for discussing this with me in an CPython contributor office hours time slot before I filed the issue. Thanks to Dr. Russell Keith-Magee, Glyph Lefkowitz, Nathaniel Smith, and Geoffrey Thomas for informal conversations and technical research. ---------- components: Tests, asyncio messages: 369393 nosy: asheesh, asvetlov, yselivanov, zach.ware priority: normal severity: normal status: open title: Adjust test_concurrent_futures to run more of its tests if multiprocessing.synchronize is missing type: enhancement versions: Python 3.9 _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue40692> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
