[issue31671] IntFlag makes re.compile slower

STINNER Victor added the comment: Serhiy: "PR 3867 looks unpythonic to me. We usually don't check the type of arguments. This complicate and slow down a code. Do you have a realistic example when the current behavior harms?" No, I don't have any realistic example. I

[issue31671] IntFlag makes re.compile slower

STINNER Victor added the comment: Hum, I ran again my microbenchmark on re.compile() cache: haypo@selma$ ./python -m perf timeit -s 'import re; re_compile=re.compile' 're_compile("a", flags=2)' --duplicate=1024 -o ref.json --inherit=PYTHONPATH -v Sadly, the commit

[issue31671] IntFlag makes re.compile slower

Serhiy Storchaka added the comment: PR 3867 looks unpythonic to me. We usually don't check the type of arguments. This complicate and slow down a code. Do you have a realistic example when the current behavior harms? --

[issue31671] IntFlag makes re.compile slower

INADA Naoki added the comment: Instead caching type(flags), how about this? if not isinstance(flags, int): raise TypeError(f"flags must be int or RegexFlag, got {flags!r}") flags = int(flags) -- ___ Python tracker

[issue31671] IntFlag makes re.compile slower

STINNER Victor added the comment: My PR 3867 fixes a corner case when the re.compile() is misused ("on purpose"?). I'm going to reject (abandon) my own PR since it makes re.compile() slower: Mean +- std dev: [ref] 364 ns +- 6 ns -> [patch] 459 ns +- 10 ns: 1.26x

[issue31671] IntFlag makes re.compile slower

Change by Serhiy Storchaka : -- type: -> performance ___ Python tracker ___

[issue31671] IntFlag makes re.compile slower

Serhiy Storchaka added the comment: Thanks Naoki! -- ___ Python tracker ___

[issue31671] IntFlag makes re.compile slower

Change by INADA Naoki : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___

[issue31671] IntFlag makes re.compile slower

INADA Naoki added the comment: New changeset c1c47c166b1012d34f2c6e111ee9ccb5c4d12de7 by INADA Naoki in branch 'master': bpo-31671: re: Convert RegexFlag to int before compile (GH-3862) https://github.com/python/cpython/commit/c1c47c166b1012d34f2c6e111ee9ccb5c4d12de7

[issue31671] IntFlag makes re.compile slower

Ethan Furman added the comment: INADA Naoki said: > But while new instance is not created each time, 4 Python method > calls (e,g. IntFlag.__and__() -> IntFlag.__new__() > -> IntFlag._missing_() -> IntFlag._create_pseudo_member_()) > are much slower than int & int. Only

[issue31671] IntFlag makes re.compile slower

STINNER Victor added the comment: The perf module always starts with a "warmup" run to fill caches. If enum has a cache, it should be filled automatically. -- ___ Python tracker

[issue31671] IntFlag makes re.compile slower

INADA Naoki added the comment: > IntFlag.__and__ does not create a new instance every time -- all new > instances are cached in the IntFlag machinery (so RegexFlag(7) is only > created once). I'm sorry, I misunderstood. But while new instance is not created each time,

[issue31671] IntFlag makes re.compile slower

Ethan Furman added the comment: IntFlag.__and__ does not create a new instance every time -- all new instances are cached in the IntFlag machinery (so RegexFlag(7) is only created once). If all the RegexFlag combinations are created before the regex compile benchmark do

[issue31671] IntFlag makes re.compile slower

STINNER Victor added the comment: I don't really care if Python 3.7 is still slower than 3.6 with PR 3862. The speedup is significant, the change is short and safe, so the PR LGTM :-) We can implement further optimizations later ;-) By the way, speed.python.org can

[issue31671] IntFlag makes re.compile slower

Serhiy Storchaka added the comment: It makes sense to report the performance gain only in comparison with the previous released version. I expect that re compiling is slower in 3.7 due to new features and optimizations. Generating more optimal code takes a time.

[issue31671] IntFlag makes re.compile slower

STINNER Victor added the comment: > Did you compare the benchmarking results against 3.7 or 3.6? My lasted benchmark is on the master branch: original code ("ref") vs code patched with PR 3862 ("patch"). -- ___ Python

[issue31671] IntFlag makes re.compile slower

Serhiy Storchaka added the comment: Did you compare the benchmarking results against 3.7 or 3.6? -- ___ Python tracker ___

[issue31671] IntFlag makes re.compile slower

STINNER Victor added the comment: Naoki: "I've added news entry about it." Nice. You might also mention the "optimization" in What's New in Python 3.7 doc, in the Optimisations section. The issue here is that it's only faster than Python 3.6, but Python 3.6 was

[issue31671] IntFlag makes re.compile slower

INADA Naoki added the comment: Thank you for benchmarking. I've added news entry about it. -- ___ Python tracker ___

[issue31671] IntFlag makes re.compile slower

STINNER Victor added the comment: Nice speedup! I ran a benchmark on PR 3862: 1002 ./python ~/prog/python/performance/performance/benchmarks/bm_regex_compile.py --inherit=PYTHONPATH -v -o patch.json 1003 git co master 1004 make 1005 ./python

[issue31671] IntFlag makes re.compile slower

INADA Naoki added the comment: > When proposing to undo recent decisions, please add the people to the nosy > list who were involved in making that decision in the first place. I don't propose reverting IntFlag. I just propose convert IntFlag to int in re module,

[issue31671] IntFlag makes re.compile slower

Raymond Hettinger added the comment: When proposing to undo recent decisions, please add the people to the nosy list who were involved in making that decision in the first place. -- nosy: +ethan.furman, gvanrossum, rhettinger

[issue31671] IntFlag makes re.compile slower

Serhiy Storchaka added the comment: I don't think there is a problem which is worth to be solved by PR 3867. It is very unlikely that anyone uses re functions with a pattern and float flags which accidentally matches already cached valid pattern and integer flag.

[issue31671] IntFlag makes re.compile slower

Change by Barry A. Warsaw : -- nosy: +barry ___ Python tracker ___ ___ Python-bugs-list

[issue31671] IntFlag makes re.compile slower

STINNER Victor added the comment: Serhiy: "Victor, how large is performance regression of your patch?" I tested bm_regex_compile.py of the Python performance benchmark suite: https://pyperformance.readthedocs.io/benchmarks.html#regex-compile My patch has no impact on

[issue31671] IntFlag makes re.compile slower

Serhiy Storchaka added the comment: Victor, how large is performance regression of your patch? -- ___ Python tracker ___

[issue31671] IntFlag makes re.compile slower

STINNER Victor added the comment: > :) This is due to caching. 2.0 == 2. I proposed PR 3867 to fix the re.compile() cache: check flags type. -- ___ Python tracker

[issue31671] IntFlag makes re.compile slower

Change by STINNER Victor : -- pull_requests: +3847 ___ Python tracker ___ ___

[issue31671] IntFlag makes re.compile slower

STINNER Victor added the comment: Oh, right. Strange. >>> re.compile("e", flags=2.0) Traceback (most recent call last): File "", line 1, in File "/home/haypo/prog/python/master/Lib/re.py", line 240, in compile return _compile(pattern, flags) File

[issue31671] IntFlag makes re.compile slower

Serhiy Storchaka added the comment: :) This is due to caching. 2.0 == 2. -- ___ Python tracker ___

[issue31671] IntFlag makes re.compile slower

STINNER Victor added the comment: Oh, Python already accepts floating point numbers: haypo@selma$ ./python Python 3.7.0a1+ (heads/master-dirty:efb560eee2, Oct 3 2017, 12:15:58) [GCC 7.2.1 20170915 (Red Hat 7.2.1-2)] on linux Type "help", "copyright", "credits" or

[issue31671] IntFlag makes re.compile slower

Change by STINNER Victor : -- nosy: +haypo ___ Python tracker ___ ___

[issue31671] IntFlag makes re.compile slower

Serhiy Storchaka added the comment: See also issue28637. Using IntFlag in the re module impacted the Python startup time. This was "fixed" by getting rid of re in site.py. -- nosy: +serhiy.storchaka ___ Python tracker

[issue31671] IntFlag makes re.compile slower

Change by INADA Naoki : -- keywords: +patch pull_requests: +3842 stage: -> patch review ___ Python tracker ___

[issue31671] IntFlag makes re.compile slower

New submission from INADA Naoki : flags exposed by re module are IntFlag from Python 3.4. Since it is passed to underlying sre_parse and sre_compile, tests in loop (e.g. `flags & SRE_FLAG_IGNORECASE`) calls IntFlag.__and__ and creates new instance everytime. It makes