New submission from Łukasz Langa <luk...@langa.pl>: When the CPython test suite re-runs a flaky failed test, it doesn’t actually re-run just that failed test but the entire test file. The most common case for flaky tests is when networking, threading, or multiprocessing is involved. Frustratingly those are some of the largest test files we have. Re-running them takes a lot of time.
Instead of re-running the entire file, regrtest should only re-run what actually failed. NOTE: I added 3.10 and 3.9 to this issue even though it's not a pure bugfix. The reason is that this can visibly speed up CI for open pull requests. (Of course, it would be best to avoid flaky tests altogether and we’re working on that, but re-running what failed is a pragmatic stopgap that is necessary in times of distributed CI that we don’t fully control.) ---------- assignee: lukasz.langa components: Tests messages: 397985 nosy: lukasz.langa, pablogsal priority: normal severity: normal stage: patch review status: open title: Regression tests with -w should only re-run affected test methods, not the entire file type: resource usage versions: Python 3.10, Python 3.11, Python 3.9 _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue44708> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
