[issue31324] support._match_test() used by test.bisect is very inefficient

STINNER Victor added the comment: I tested set_match_tests() on python2.7 with: python2.7 -m test --list-cases > cases python2.7 -m test.bisect -i cases Good news: the set().__contains__ matcher can be used! -- ___ Python tracker

[issue31324] support._match_test() used by test.bisect is very inefficient

STINNER Victor added the comment: New changeset 35d99830f1878867e3964577741d9a2d5a7fc8f7 by Victor Stinner in branch '2.7': bpo-31324: Optimize support._match_test() (#4523) (#4524) https://github.com/python/cpython/commit/35d99830f1878867e3964577741d9a2d5a7fc8f7 --

[issue31324] support._match_test() used by test.bisect is very inefficient

Change by STINNER Victor : -- pull_requests: +4461 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mai

[issue31324] support._match_test() used by test.bisect is very inefficient

STINNER Victor added the comment: New changeset 70b2f8797146a56a6880743424f0bedf4fc30c62 by Victor Stinner in branch '3.6': [3.6] bpo-31324: Optimize support._match_test() (#4523) https://github.com/python/cpython/commit/70b2f8797146a56a6880743424f0bedf4fc30c62 -- __

[issue31324] support._match_test() used by test.bisect is very inefficient

Change by STINNER Victor : -- pull_requests: +4460 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mai

[issue31324] support._match_test() used by test.bisect is very inefficient

STINNER Victor added the comment: New changeset bb11c3c967afaf263e00844d4ab461b7fafd6d36 by Victor Stinner in branch 'master': bpo-31324: Fix test.support.set_match_tests(None) (#4505) https://github.com/python/cpython/commit/bb11c3c967afaf263e00844d4ab461b7fafd6d36 -- _

[issue31324] support._match_test() used by test.bisect is very inefficient

STINNER Victor added the comment: Oops, the commit 803ddd8ce22f0de3ab42fb98a225a704c000ef06 broke the GCC job on Travis CI: --- Run tests sequentially 0:00:00 load avg: 48.91 [ 1/404] test_grammar 0:00:00 load avg: 48.91 [ 2/404] test_opcodes 0:00:00 load avg: 48.91 [ 3/404] test_dict 0:

[issue31324] support._match_test() used by test.bisect is very inefficient

Change by STINNER Victor : -- pull_requests: +4443 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mai

[issue31324] support._match_test() used by test.bisect is very inefficient

STINNER Victor added the comment: I merged my PR 4421 which is based on Serhiy's PR 4420. Thank you Serhiy for your reviews! -- Serhiy Storchaka: "We could split patterns on two parts and create both matchers (...) I don't know whether it is worth to do." My use bisect is to find a regressi

[issue31324] support._match_test() used by test.bisect is very inefficient

STINNER Victor added the comment: New changeset 803ddd8ce22f0de3ab42fb98a225a704c000ef06 by Victor Stinner in branch 'master': bpo-31324: Optimize support._match_test() (#4421) https://github.com/python/cpython/commit/803ddd8ce22f0de3ab42fb98a225a704c000ef06 -- _

[issue31324] support._match_test() used by test.bisect is very inefficient

Serhiy Storchaka added the comment: We could split patterns on two parts and create both matchers. Then the final matching function could look like: return _match_test_patterns is None or test_id in id_set or regex_match(test_id) or any(map(regex_match, test_id.split("."))) I don't know whe

[issue31324] support._match_test() used by test.bisect is very inefficient

STINNER Victor added the comment: I created a list of all test cases using: ./python -m test --list-cases > all_cases The list contains 29,569 test cases. Sadly, the set().__contains__ matcher of my PR 4421 cannot be taken because test_json produces two test cases called "json". IMHO it's a

[issue31324] support._match_test() used by test.bisect is very inefficient

Serhiy Storchaka added the comment: I know :-) Unfortunately it is not obviously that these optimization actually optimize regular expressions. Due to the implementation particularities sometimes "unoptimized" code is faster than "optimized". I'm going to implement these optimizations in the

[issue31324] support._match_test() used by test.bisect is very inefficient

STINNER Victor added the comment: > There is good opportunity for optimizing the regular expression. All cases > have the literal 'test.test_asyncio.' prefix, and even longer literal > prefixes are common for tens cases. I implemented such code long time ago :-) https://github.com/haypo/hach

[issue31324] support._match_test() used by test.bisect is very inefficient

Change by STINNER Victor : -- pull_requests: +4369 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mai

[issue31324] support._match_test() used by test.bisect is very inefficient

Serhiy Storchaka added the comment: $ time ./python -m test.bisect --fail-env-changed -o bisect test_asyncio -N1 ... Bisection failed after 2 iterations and 0:00:11 real0m10,441s user0m8,726s sys 0m0,292s There is good opportunity for optimizing the regular expression. All cases ha

[issue31324] support._match_test() used by test.bisect is very inefficient

STINNER Victor added the comment: I tested manually Serhiy's PR 4420: it works as expected! haypo@selma$ ./python -m test test_os --list-cases -m test.test_os.FileTests.test_access test.test_os.FileTests.test_access haypo@selma$ ./python -m test test_os --list-cases -m test_access test.test_o

[issue31324] support._match_test() used by test.bisect is very inefficient

Change by Serhiy Storchaka : -- keywords: +patch pull_requests: +4368 stage: -> patch review ___ Python tracker ___ ___ Python-bugs-

[issue31324] support._match_test() used by test.bisect is very inefficient

Serhiy Storchaka added the comment: _match_test1 = re.compile('|'.join(map(fnmatch.translate, match_tests)), re.I).match def _match_test(test): test_id = test.id() return bool(_match_test1(test_id) or any(map(_match_test1, test_id.split("."))) -- _

[issue31324] support._match_test() used by test.bisect is very inefficient

STINNER Victor added the comment: Oops. Ignore my previous comment, I was using my temporary workaround! (msg301126) -- ___ Python tracker ___ __

[issue31324] support._match_test() used by test.bisect is very inefficient

STINNER Victor added the comment: Oh, by the way, -m test -m TestCaseName doesn't work fully on test_asyncio. It only works if I pass the full test identifier. With TestCaseName, no test is run: haypo@selma$ ./python -m test -u all test_asyncio --fail-env-changed -m PollEventLoopTests Run te

[issue31324] support._match_test() used by test.bisect is very inefficient

STINNER Victor added the comment: Workaround to run test.bisect on test_asyncio: diff --git a/Lib/test/support/__init__.py b/Lib/test/support/__init__.py index 522804be60..201d0665b2 100644 --- a/Lib/test/support/__init__.py +++ b/Lib/test/support/__init__.py @@ -1911,15 +1911,7 @@ def _match_te

[issue31324] support._match_test() used by test.bisect is very inefficient

STINNER Victor added the comment: Example of the performance issue: $ ./python -m test.bisect --fail-env-changed -o bisect test_asyncio [+] Iteration 1: run 756 tests/1512 (...) The first iteration takes forever because it runs tests with support.match_tests which contains 756 patterns.

[issue31324] support._match_test() used by test.bisect is very inefficient

Changes by STINNER Victor : -- nosy: +louielu ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.pyth

[issue31324] support._match_test() used by test.bisect is very inefficient

New submission from STINNER Victor: support._match_test() uses a nested loop calling fnmatch.fnmatchcase(). This function creates a temporary regular expression object. The cache of the re module works around the performance... if the length of support.match_tests fits into the cache. But when