[issue23689] Memory leak in Modules/sre_lib.h

Serhiy Storchaka added the comment: Thank you Ma Lin for all your work. The fix changes interfaces of some internal functions which can be used in third-party code, and the bug occurs only in special circumstances, so it is not practical to backport it. -- resolution: -> fixed stag

[issue23689] Memory leak in Modules/sre_lib.h

Serhiy Storchaka added the comment: New changeset 6e3eee5c11b539e9aab39cff783acf57838c355a by Ma Lin in branch 'main': bpo-23689: re module, fix memory leak when a match is terminated by a signal or memory allocation failure (GH-32283) https://github.com/python/cpython/commit/6e3eee5c11b539e

[issue23689] Memory leak in Modules/sre_lib.h

Change by Ma Lin : -- pull_requests: +30344 pull_request: https://github.com/python/cpython/pull/32283 ___ Python tracker ___ ___ Py

[issue23689] Memory leak in Modules/sre_lib.h

Change by Ma Lin : -- pull_requests: +30298 pull_request: https://github.com/python/cpython/pull/32223 ___ Python tracker ___ ___ Py

[issue23689] Memory leak in Modules/sre_lib.h

Change by Ma Lin : -- pull_requests: +30265 pull_request: https://github.com/python/cpython/pull/32188 ___ Python tracker ___ ___ Py

[issue23689] Memory leak in Modules/sre_lib.h

Serhiy Storchaka added the comment: This looks promising. Please, go ahead! You are free to add any fields to any opcodes. It may break some third-party code which generates compiled patterns from a sequence of opcodes, it the stability of this interface was not promised. And they will be br

[issue23689] Memory leak in Modules/sre_lib.h

Ma Lin added the comment: My PR methods are suboptimal, so I closed them. The number of REPEAT can be counted when compiling a pattern, and allocate a `SRE_REPEAT` array in `SRE_STATE` (with that number items). It seem at any time, a REPEAT will only have one in active, so a `SRE_REPEAT` ar

[issue23689] Memory leak in Modules/sre_lib.h

Ma Lin added the comment: PR11926 (closed) tried to allocate SRE_REPEAT on state's stack. It's feasible, but messes up the code in sre_lib.h, and reduces performance a little (roughly 6% slower), so I gave up this solution. PR12160 uses a memory pool, this solution doesn't mess up the code. 🔸

[issue23689] Memory leak in Modules/sre_lib.h

Change by Ma Lin : -- pull_requests: +12158 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.pyth

[issue23689] Memory leak in Modules/sre_lib.h

Ma Lin added the comment: Try to allocate SRE_REPEAT on state's stack, the performance has not changed significantly. It passes the other tests, except this one (test_stack_overflow): https://github.com/python/cpython/blob/v3.8.0a1/Lib/test/test_re.py#L1225-L1230 I'll try to fix issue35859,

[issue23689] Memory leak in Modules/sre_lib.h

Change by Ma Lin : -- pull_requests: +11951 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.pyth

[issue23689] Memory leak in Modules/sre_lib.h

Change by Serhiy Storchaka : -- nosy: +Ma Lin ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.py

[issue23689] Memory leak in Modules/sre_lib.h

Change by Serhiy Storchaka : -- versions: +Python 3.6, Python 3.7 -Python 3.4, Python 3.5 ___ Python tracker ___ ___ Python-bugs-list

[issue23689] Memory leak in Modules/sre_lib.h

Changes by Serhiy Storchaka : -- assignee: -> serhiy.storchaka ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue23689] Memory leak in Modules/sre_lib.h

Changes by Alexei Romanov : -- nosy: +alexei.romanov ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://ma

[issue23689] Memory leak in Modules/sre_lib.h

Evgeny Kapun added the comment: This patch doesn't fix the issue. The problem is that the list starting with state->repeat doesn't necessarily contains all repeat contexts that are allocated. Indeed, here [1] and here [2] repeat contexts are temporarily removed from the list. If the match proc

[issue23689] Memory leak in Modules/sre_lib.h

STINNER Victor added the comment: Oh cool, you wrote a script to reproduce the issue! And Serhiy wrote a patch, great! Great job guys. sre_clean_repeat_data.patch looks good to me. @Serhiy: Can you try the example to ensure that it fixes the issue? If yes, go ahead! -- _

[issue23689] Memory leak in Modules/sre_lib.h

Serhiy Storchaka added the comment: May be this patch helps. -- keywords: +patch stage: -> patch review versions: +Python 2.7, Python 3.5 Added file: http://bugs.python.org/file38529/sre_clean_repeat_data.patch ___ Python tracker

[issue23689] Memory leak in Modules/sre_lib.h

Changes by Serhiy Storchaka : -- nosy: +serhiy.storchaka ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https:

[issue23689] Memory leak in Modules/sre_lib.h

Evgeny Kapun added the comment: Tracemalloc code: import re import signal import tracemalloc class AlarmError(Exception): pass def handle_alarm(signal, frame): raise AlarmError signal.signal(signal.SIGALRM, handle_alarm) s1 = tracemalloc.take_snapsho

[issue23689] Memory leak in Modules/sre_lib.h

Evgeny Kapun added the comment: Memory leak only happens if match operation terminates abruptly, e.g. because of SIGINT. In this case, DO_JUMP doesn't come back. -- ___ Python tracker _

[issue23689] Memory leak in Modules/sre_lib.h

STINNER Victor added the comment: There is maybe a bug. Can you show an example of regex and a text where the memory leak occurs? You can use the tracemalloc module to check if there is a memory leak. Or use sys.getcounts() if you compiled Python in debug mode. sre_lib.h is very complex, it us

[issue23689] Memory leak in Modules/sre_lib.h

New submission from Evgeny Kapun: In Modules/sre_lib.h on line 882 [1], a block of memory is allocated. If SRE(match) function later terminates abruptly, either because of a signal or because subsequent memory allocation fails, this block is never released. [1] https://hg.python.org/cpython/fi