[issue37146] opcode cache for LOAD_GLOBAL emits false alarm in memory leak hunting

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 9f672a52f55ce198d5689b5771948350028b4a3b by Victor Stinner in 
branch 'master':
bpo-37146: Move _PyEval_DeactivateOpCache() to the internal C API (GH-24786)
https://github.com/python/cpython/commit/9f672a52f55ce198d5689b5771948350028b4a3b


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37146] opcode cache for LOAD_GLOBAL emits false alarm in memory leak hunting

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +23552
pull_request: https://github.com/python/cpython/pull/24786

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37146] opcode cache for LOAD_GLOBAL emits false alarm in memory leak hunting

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37146] opcode cache for LOAD_GLOBAL emits false alarm in memory leak hunting

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:


New changeset af5fa13ef6f648fc7a7a33a7556db13887e7d643 by Pablo Galindo in 
branch 'master':
bpo-37146: Deactivate opcode cache only when using huntrleaks in the test suite 
(GH-24643)
https://github.com/python/cpython/commit/af5fa13ef6f648fc7a7a33a7556db13887e7d643


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37146] opcode cache for LOAD_GLOBAL emits false alarm in memory leak hunting

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
pull_requests: +23428
stage: resolved -> patch review
pull_request: https://github.com/python/cpython/pull/24643

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37146] opcode cache for LOAD_GLOBAL emits false alarm in memory leak hunting

[Christian Heimes]

Change by Christian Heimes :


--
components: +Interpreter Core -SSL
nosy:  -christian.heimes
title: SSl Securité version 3.9.2 -> opcode cache for LOAD_GLOBAL emits false 
alarm in memory leak hunting
type: security -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37146] opcode cache for LOAD_GLOBAL emits false alarm in memory leak hunting

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

> Or maybe need a generic -X flag to tell Python that libregrtest is tracking 
> leaks, since we might need to change other parameters than only the opcode 
> cache size.

I think we should go this way, maybe even less "public". If we add a -X, then 
it will be too visible. Maybe some env variable?

--
nosy: +pablogsal

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37146] opcode cache for LOAD_GLOBAL emits false alarm in memory leak hunting

[Yury Selivanov]

Yury Selivanov  added the comment:

> Giving the ability to control the cache size, at least at Python startup, is 
> one option.

I'd really prefer not to allow users to control cache sizes. There's basically 
no point in that; the only practically useful thing is to enable or disable it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37146] opcode cache for LOAD_GLOBAL emits false alarm in memory leak hunting

[STINNER Victor]

STINNER Victor  added the comment:

> (Or perhaps provide an -X flag to disable it without the need to recompile.)

Giving the ability to control the cache size, at least at Python startup, is 
one option.

Or maybe need a generic -X flag to tell Python that libregrtest is tracking 
leaks, since we might need to change other parameters than only the opcode 
cache size.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37146] opcode cache for LOAD_GLOBAL emits false alarm in memory leak hunting

[STINNER Victor]

STINNER Victor  added the comment:

I reopen the issue (but I'm not interested by trying to fix it).

--
resolution: fixed -> 
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37146] opcode cache for LOAD_GLOBAL emits false alarm in memory leak hunting

[Guido van Rossum]

Guido van Rossum  added the comment:

Please revert this and use a separate build flag (e.g. DISABLE_INLINE_CACHES) 
for the refleaks run. (Or perhaps provide an -X flag to disable it without the 
need to recompile.)

I am developing new inline cache ideas and of course I need to run in debug 
mode, so I regularly end up wasting some time until I remember to add a patch 
that changes OPCACHE_MIN_RUNS back.

--
nosy: +gvanrossum

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37146] opcode cache for LOAD_GLOBAL emits false alarm in memory leak hunting

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +23365
pull_request: https://github.com/python/cpython/pull/24582

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37146] opcode cache for LOAD_GLOBAL emits false alarm in memory leak hunting

[Yury Selivanov]

Yury Selivanov  added the comment:

> It seems like nobody came up with a solution for the debug mode since June. I 
> close the issue. Reopen it if you would like to propose a solution.

I think the only solution is to have a flag to disable optimizations, 
inlcluding this one.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37146] opcode cache for LOAD_GLOBAL emits false alarm in memory leak hunting

[STINNER Victor]

STINNER Victor  added the comment:

It seems like nobody came up with a solution for the debug mode since June. I 
close the issue. Reopen it if you would like to propose a solution.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37146] opcode cache for LOAD_GLOBAL emits false alarm in memory leak hunting

[STINNER Victor]

STINNER Victor  added the comment:

INADA-san wrote PR 13787 to disable opcache when Python is compiled in debug 
mode. Pablo and me approved the change, so I merged it.

Pablo wrote a more robust solution, PR 13789, to disable opcache only in 
regrtest, to look for memory leaks. But INADA-san had a good argument against 
this approach:
https://github.com/python/cpython/pull/13789#issuecomment-498449735

"The code object will be optimized only when ++co->co_opcache_flag == 
opcacheminruns. So decreasing min_runs by _setopcacheminruns()will cause some 
hot codes will be not optimized forever. I don't want to expose such switch."

I would prefer to keep this issue until a long term approach is designed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37146] opcode cache for LOAD_GLOBAL emits false alarm in memory leak hunting

[STINNER Victor]

STINNER Victor  added the comment:


New changeset eddef861b49f1635222a9e1771231c34a807debf by Victor Stinner (Inada 
Naoki) in branch 'master':
bpo-37146: disable opcache when Py_DEBUG is defined (GH-13787)
https://github.com/python/cpython/commit/eddef861b49f1635222a9e1771231c34a807debf


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37146] opcode cache for LOAD_GLOBAL emits false alarm in memory leak hunting

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
pull_requests: +13673
pull_request: https://github.com/python/cpython/pull/13789

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37146] opcode cache for LOAD_GLOBAL emits false alarm in memory leak hunting

[Inada Naoki]

Change by Inada Naoki :


--
keywords: +patch
pull_requests: +13671
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/13787

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37146] opcode cache for LOAD_GLOBAL emits false alarm in memory leak hunting

[STINNER Victor]

New submission from STINNER Victor :

opcode cache for LOAD_GLOBAL introduced false alarm in memory leak hunting 
(python3 -m test -R 3:3 ...).

=> opcache: bpo-26219.


Before the change:

$ git checkout 91234a16367b56ca03ee289f7c03a34d4cfec4c8^
$ make && ./python -m test -R 3:3 test_pprint 
...
Tests result: SUCCESS

After the change:

$ git checkout 91234a16367b56ca03ee289f7c03a34d4cfec4c8
$ make && ./python -m test -R 3:3 test_pprint 
...
test_pprint leaked [4, 2, 4] memory blocks, sum=10
...

The problem is that at each iteration of regrtest -R 3:3 (6 iterations), a few 
more code objects get this opcache allocated.

There are different solutions to fix regrtest -R 3:3.


(*) Always optimize:

diff --git a/Python/ceval.c b/Python/ceval.c
index 411ba3d73c..6cd148efba 100644
--- a/Python/ceval.c
+++ b/Python/ceval.c
@@ -103,7 +103,7 @@ static long dxp[256];
 #endif
 
 /* per opcode cache */
-#define OPCACHE_MIN_RUNS 1024  /* create opcache when code executed this time 
*/
+#define OPCACHE_MIN_RUNS 1  /* create opcache when code executed this time */
 #define OPCACHE_STATS 0  /* Enable stats */
 
 #if OPCACHE_STATS

$ make && ./python -m test -R 3:3 test_pprint 
...
Tests result: SUCCESS


(*) Never optimmize: disable opcache until a better fix can be found

diff --git a/Python/ceval.c b/Python/ceval.c
index 411ba3d73c..3c85df6fea 100644
--- a/Python/ceval.c
+++ b/Python/ceval.c
@@ -1230,6 +1230,7 @@ _PyEval_EvalFrameDefault(PyFrameObject *f, int throwflag)
 f->f_stacktop = NULL;   /* remains NULL unless yield suspends frame */
 f->f_executing = 1;
 
+#if 0
 if (co->co_opcache_flag < OPCACHE_MIN_RUNS) {
 co->co_opcache_flag++;
 if (co->co_opcache_flag == OPCACHE_MIN_RUNS) {
@@ -1244,6 +1245,7 @@ _PyEval_EvalFrameDefault(PyFrameObject *f, int throwflag)
 #endif
 }
 }
+#endif
 
 #ifdef LLTRACE
 lltrace = _PyDict_GetItemId(f->f_globals, ___ltrace__) != NULL;

$ make && ./python -m test -R 3:3 test_pprint 
...
Tests result: SUCCESS


(*) Find a way to explicitly deoptimize all code objects

Modules/gcmodule.c has a clear_freelists() function called by collect() if 
generation == NUM_GENERATIONS-1: on when gc.collect() is collected explicitly 
for example.

Lib/test/libregrtest/refleak.py also has a dash_R_cleanup() function which 
clears many caches.

Problem: currently, code objects are not explicitly tracked (for example, they 
are not tracked in a double linked list).


(*) Add way more warmup iterations to regrtest in buildbots.

I dislike this option. A build on a refleak buildbot worker already takes 2 to 
3 hours. Adding more warmup would make a build even way more slower.

--
components: Interpreter Core
messages: 344469
nosy: inada.naoki, vstinner, yselivanov
priority: normal
severity: normal
status: open
title: opcode cache for LOAD_GLOBAL emits false alarm in memory leak hunting
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com