[issue45953] Statically allocate interpreter states as much as possible.

[Julien Palard]

Julien Palard  added the comment:

> Since you seem to be challenging the value of 9, my question to you 
> is, why do you care what the refcount of 1 is?

Yesterday I was teaching Python, and we were speaking of integer immutability, 
names being "labels to objects" and so on, and I was showing the memory layout 
of all of this by hand on a whiteboard while "prooving" my drawings using an 
interpreter.

While doing so came a question like "So, many modules can use the object 
int(1)?" So I answered yes, told that I expected many reuse of 1, and went 
importing sys.getrefcount to show them.

And boom, it printed 100209 so I bugged for a few seconds, the value was 
obviously not the real refcount, and was also obviously bumped by a constant 
like 1, so I went inspecting why and found this commit.

I have nothing against keeping 9, but in the other hand it could 
surprise other people, maybe we should at least document it near 
sys.getrefcount.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45953] Statically allocate interpreter states as much as possible.

[Guido van Rossum]

Guido van Rossum  added the comment:

I used 9 in deepfreeze.py to signify "immortal object". It has been 
copied by others (small integers are essentially immortal too). I wasn't too 
sure that the refcount wouldn't go below zero if the interpreter is repeatedly 
finalized and reinitialized. Once we have official immortal objects (see PEP 
683) we should switch to that.

Since you seem to be challenging the value of 9, my question to you is, 
why do you care what the refcount of 1 is?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45953] Statically allocate interpreter states as much as possible.

[Julien Palard]

Julien Palard  added the comment:

Hum, and why 9? I am probably missing something obvious but 1 should be 
enough to ensure the value never hits 0. Except for refcount bugs obviously, 
but I don't think this is the right reason?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45953] Statically allocate interpreter states as much as possible.

[Guido van Rossum]

Guido van Rossum  added the comment:

Please don’t try to “fix” anything. The value is only useful if you
understand the implementation. It should map straightforwardly to what’s in
memory.

On Mon, Mar 28, 2022 at 05:16 STINNER Victor  wrote:

>
> STINNER Victor  added the comment:
>
> > Should sys.getrefcount try to "fix" the value (...)
>
> https://peps.python.org/pep-0683/ would make it possible. Right now, I
> don't think that it's possible.
>
> Right now, a refcount of 100210 can be a real value, or it can be an
> immortal object.
>
> --
>
> ___
> Python tracker 
> 
> ___
>
-- 
--Guido (mobile)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45953] Statically allocate interpreter states as much as possible.

[STINNER Victor]

STINNER Victor  added the comment:

> Should sys.getrefcount try to "fix" the value (...)

https://peps.python.org/pep-0683/ would make it possible. Right now, I don't 
think that it's possible.

Right now, a refcount of 100210 can be a real value, or it can be an 
immortal object.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45953] Statically allocate interpreter states as much as possible.

[Julien Palard]

Julien Palard  added the comment:

Since 
https://github.com/python/cpython/commit/121f1f893a39d0b58d3d2b5597505c154ecaac2a,
 `sys.getrefcount(1)` is surprising:

>>> __import__("sys").getrefcount(1)
100210

Should sys.getrefcount try to "fix" the value like by returning 
`PyREFCNT(object) % 9`? (But using a define to avoid the "magic, 
copy/pasted value").

--
nosy: +mdk

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45953] Statically allocate interpreter states as much as possible.

[miss-islington]

miss-islington  added the comment:


New changeset f78be59c83c151d94902daef56218530c52e29e7 by Eric Snow in branch 
'main':
bpo-45953: Preserve backward compatibility on some PyThreadState field names. 
(GH-31038)
https://github.com/python/cpython/commit/f78be59c83c151d94902daef56218530c52e29e7


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45953] Statically allocate interpreter states as much as possible.

[Eric Snow]

Change by Eric Snow :


--
pull_requests: +29221
pull_request: https://github.com/python/cpython/pull/31038

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45953] Statically allocate interpreter states as much as possible.

[Eric Snow]

Eric Snow  added the comment:

> Any chance we could revert the recent renaming of tstate.exc_state and 
> tstate.root_cframe

Yeah, I'll sort this out.  Sorry for that.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45953] Statically allocate interpreter states as much as possible.

[Brandt Bucher]

Brandt Bucher  added the comment:

Any chance we could revert the recent renaming of tstate.exc_state and 
tstate.root_cframe in https://github.com/python/cpython/pull/30590? It broke 
Greenlet again:

https://github.com/python-greenlet/greenlet/issues/288

If it's only a name change (and the members themselves are the same), I think 
reverting it is preferable to burying Greenlet in more compatibility macros and 
bugging them to put out another new release.

--
nosy: +brandtbucher

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45953] Statically allocate interpreter states as much as possible.

[STINNER Victor]

Change by STINNER Victor :


--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45953] Statically allocate interpreter states as much as possible.

[Eric Snow]

Eric Snow  added the comment:


New changeset 322f962f3ee31d0dbde99e36379de8488ccc6804 by Eric Snow in branch 
'main':
bpo-45953: Statically initialize all the non-object PyInterpreterState fields 
we can. (gh-30589)
https://github.com/python/cpython/commit/322f962f3ee31d0dbde99e36379de8488ccc6804


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45953] Statically allocate interpreter states as much as possible.

[Eric Snow]

Eric Snow  added the comment:


New changeset 324908ba936d5d262026deebb81f050803848c41 by Eric Snow in branch 
'main':
bpo-45953: Statically initialize all the PyThreadState fields we can. (gh-30590)
https://github.com/python/cpython/commit/324908ba936d5d262026deebb81f050803848c41


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45953] Statically allocate interpreter states as much as possible.

[Eric Snow]

Change by Eric Snow :


--
pull_requests: +28788
pull_request: https://github.com/python/cpython/pull/30590

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45953] Statically allocate interpreter states as much as possible.

[Eric Snow]

Change by Eric Snow :


--
pull_requests: +28787
pull_request: https://github.com/python/cpython/pull/30589

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45953] Statically allocate interpreter states as much as possible.

[Eric Snow]

Change by Eric Snow :


--
pull_requests: +28786
pull_request: https://github.com/python/cpython/pull/30588

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45953] Statically allocate interpreter states as much as possible.

[Eric Snow]

Eric Snow  added the comment:


New changeset ed57b36c32e521162dbb97199e64a340d3bff827 by Eric Snow in branch 
'main':
bpo-45953: Statically allocate the main interpreter (and initial thread state). 
(gh-29883)
https://github.com/python/cpython/commit/ed57b36c32e521162dbb97199e64a340d3bff827


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45953] Statically allocate interpreter states as much as possible.

[Eric Snow]

Eric Snow  added the comment:


New changeset cf496d657a1a82eaf9ebfb47d721676fef6effa5 by Eric Snow in branch 
'main':
bpo-45953: Statically allocate and initialize global bytes objects. (gh-30096)
https://github.com/python/cpython/commit/cf496d657a1a82eaf9ebfb47d721676fef6effa5


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45953] Statically allocate interpreter states as much as possible.

[Eric Snow]

Change by Eric Snow :


--
pull_requests: +28319
pull_request: https://github.com/python/cpython/pull/30096

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45953] Statically allocate interpreter states as much as possible.

[Eric Snow]

Eric Snow  added the comment:


New changeset 121f1f893a39d0b58d3d2b5597505c154ecaac2a by Eric Snow in branch 
'main':
bpo-45953: Statically initialize the small ints. (gh-30092)
https://github.com/python/cpython/commit/121f1f893a39d0b58d3d2b5597505c154ecaac2a


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45953] Statically allocate interpreter states as much as possible.

[Eric Snow]

Change by Eric Snow :


--
pull_requests: +28315
pull_request: https://github.com/python/cpython/pull/30092

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45953] Statically allocate interpreter states as much as possible.

[Guido van Rossum]

Change by Guido van Rossum :


--
nosy: +gvanrossum

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45953] Statically allocate interpreter states as much as possible.

[Eric Snow]

Change by Eric Snow :


--
keywords: +patch
pull_requests: +28109
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/29883

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45953] Statically allocate interpreter states as much as possible.

[Eric Snow]

New submission from Eric Snow :

Currently we allocate each new PyInterpreterState in PyInterpreterState_New().  
Furthermore, PyInterpreterState is full of pointers which are almost all 
allocated on the heap during runtime init.  We can statically allocate (and 
initialize?) most of what goes into PyInterpreterState, as well as the main 
interpreter itself (as part of _PyRuntimeState).  This includes each 
interpreter's initial PyThreadState.

TODO:

* [ ] add `PyInterpreterState main;` to _PyRuntimeState.interpreters and use it
* [ ] change references from the pointer to the value
* [ ] add `PyThreadState _main;` to PyInterpreterState.threads and use it
* [ ] change references from the pointer to the value
* [ ] change PyInterpreterState pointer fields to values (as much as possible)
* [ ] change PyThreadState pointer fields to values (as much as possible)


benefits:

* fewer possible failures (no memory) during runtime/interpreter/thread init
* improved memory locality for pointers looked up relative to 
interpreter/thread state


There is one non-trivial bit: embedding the various PyObject values in 
PyInterpreterState and PyThreadState means hard-coding the various pieces of 
the object there (e.g. for dict, its keys/values; for list, its array), as well 
as adding necessary init code to PyInterpreterState_New() and 
PyThreadState_New().  The resulting added complexity can be mitigated somewhat 
with macros or even code generation.  (In fact, there is probably significant 
overlap with Guido's deepfreeze tool.)  Regardless, we'll probably need to 
factor out init funcs for a number of object types, where currently there are 
only "Py*_New()" funcs that combine allocation and init.

--
assignee: eric.snow
components: Interpreter Core
messages: 407476
nosy: Mark.Shannon, eric.snow
priority: normal
severity: normal
stage: needs patch
status: open
title: Statically allocate interpreter states as much as possible.
versions: Python 3.11

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com