[issue30860] Consolidate stateful C globals under a single struct.

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 9e87e7776f7ace66baaf7247233afdabd00c2b44 by Victor Stinner in 
branch 'master':
bpo-32096: Remove obj and mem from _PyRuntime (#4532)
https://github.com/python/cpython/commit/9e87e7776f7ace66baaf7247233afdabd00c2b44


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30860] Consolidate stateful C globals under a single struct.

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +4468

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30860] Consolidate stateful C globals under a single struct.

[Eric Snow]

Changes by Eric Snow :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30860] Consolidate stateful C globals under a single struct.

[STINNER Victor]

STINNER Victor added the comment:


New changeset ccb3c7654cc3d031fb93bf443a6ef9cfb11f6b43 by Victor Stinner in 
branch 'master':
bpo-30860: Fix deadcode in obmalloc.c (#3499)
https://github.com/python/cpython/commit/ccb3c7654cc3d031fb93bf443a6ef9cfb11f6b43


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30860] Consolidate stateful C globals under a single struct.

[Eric Snow]

Eric Snow added the comment:


New changeset dae0276bb6bc7281d59fb0b8f1aab31634ee80dc by Eric Snow in branch 
'master':
bpo-30860: Fix a refleak. (#3567)
https://github.com/python/cpython/commit/dae0276bb6bc7281d59fb0b8f1aab31634ee80dc


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30860] Consolidate stateful C globals under a single struct.

[Eric Snow]

Changes by Eric Snow :


--
pull_requests: +3556

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30860] Consolidate stateful C globals under a single struct.

[Eric Snow]

Eric Snow added the comment:


New changeset fc1bf872e9d31f3e837f686210f94e57ad3d6582 by Eric Snow in branch 
'master':
bpo-30860: Move windows.h include out of internal/*.h. (#3458)
https://github.com/python/cpython/commit/fc1bf872e9d31f3e837f686210f94e57ad3d6582


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30860] Consolidate stateful C globals under a single struct.

[STINNER Victor]

STINNER Victor added the comment:


New changeset 8728018624f257c7cfe44014742ae46134047f49 by Victor Stinner (Eric 
Snow) in branch 'master':
bpo-30860: Fix a refleak. (#3506)
https://github.com/python/cpython/commit/8728018624f257c7cfe44014742ae46134047f49


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30860] Consolidate stateful C globals under a single struct.

[Eric Snow]

Eric Snow added the comment:


New changeset ba6d5d1defd7a281c8c8804e4b4cfd7370886236 by Eric Snow in branch 
'master':
bpo-30860: Always provide serialno. (#3507)
https://github.com/python/cpython/commit/ba6d5d1defd7a281c8c8804e4b4cfd7370886236


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30860] Consolidate stateful C globals under a single struct.

[Eric Snow]

Changes by Eric Snow :


--
pull_requests: +3501

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30860] Consolidate stateful C globals under a single struct.

[Eric Snow]

Changes by Eric Snow :


--
pull_requests: +3499

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30860] Consolidate stateful C globals under a single struct.

[STINNER Victor]

STINNER Victor added the comment:


New changeset 4866957e86594935ec2e6434b6e470ebeb0c79b4 by Victor Stinner in 
branch 'master':
bpo-30860: Add Include/internal/ in "make tags" (#3498)
https://github.com/python/cpython/commit/4866957e86594935ec2e6434b6e470ebeb0c79b4


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30860] Consolidate stateful C globals under a single struct.

[STINNER Victor]

STINNER Victor added the comment:

The commit 2ebc5ce42a8a9e047e790aefbf9a94811569b2b6 introduced reference leaks: 
see bpo-31420 which tracks them.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30860] Consolidate stateful C globals under a single struct.

[STINNER Victor]

Changes by STINNER Victor :


--
pull_requests: +3493

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30860] Consolidate stateful C globals under a single struct.

[STINNER Victor]

Changes by STINNER Victor :


--
pull_requests: +3492

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30860] Consolidate stateful C globals under a single struct.

[Eric Snow]

Eric Snow added the comment:

FYI, the merged code introduced a bunch of new warnings on Windows.  I'm 
looking into it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30860] Consolidate stateful C globals under a single struct.

[Eric Snow]

Changes by Eric Snow :


--
keywords: +patch
pull_requests: +3452

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30860] Consolidate stateful C globals under a single struct.

[Eric Snow]

Eric Snow added the comment:


New changeset 2ebc5ce42a8a9e047e790aefbf9a94811569b2b6 by Eric Snow in branch 
'master':
bpo-30860: Consolidate stateful runtime globals. (#3397)
https://github.com/python/cpython/commit/2ebc5ce42a8a9e047e790aefbf9a94811569b2b6


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30860] Consolidate stateful C globals under a single struct.

[Nick Coghlan]

Nick Coghlan added the comment:

Regarding the accidental exposure of _Py_CheckRecursionLimit, the main active 
usage of the stable ABI that we're aware of is Riverbank's C/C++ binding 
generator for PyQt: 
http://pyqt.sourceforge.net/Docs/sip4/directives.html#directive-%Module (see 
the use_limited_api option for the linked directive)

I checked with Phil Thomson (sip's maintainer), and the bindings sip generates 
don't use either of the public macros that access this nominally private value.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30860] Consolidate stateful C globals under a single struct.

[Eric Snow]

Changes by Eric Snow :


--
pull_requests: +3403

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30860] Consolidate stateful C globals under a single struct.

[Eric Snow]

Eric Snow added the comment:


New changeset 05351c1bd8b70d1878527762174cdaaba3572395 by Eric Snow in branch 
'master':
Revert "bpo-30860: Consolidate stateful runtime globals." (#3379)
https://github.com/python/cpython/commit/05351c1bd8b70d1878527762174cdaaba3572395


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30860] Consolidate stateful C globals under a single struct.

[Eric Snow]

Changes by Eric Snow :


--
pull_requests:  -3387

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30860] Consolidate stateful C globals under a single struct.

[Eric Snow]

Changes by Eric Snow :


--
pull_requests: +3388

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30860] Consolidate stateful C globals under a single struct.

[Eric Snow]

Changes by Eric Snow :


--
pull_requests: +3387

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30860] Consolidate stateful C globals under a single struct.

[Benjamin Peterson]

Benjamin Peterson added the comment:

I'm also a bit skeptical about the header design in this PR. We should not to 
have a monolithic _Python.h header, and it should not be sometimes included in 
Python.h. Rather, code that needs the internal headers should include them 
directly. In fact, this PR regresses that by moving all the internal headers 
into Include/...

--
nosy: +benjamin.peterson

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30860] Consolidate stateful C globals under a single struct.

[STINNER Victor]

STINNER Victor added the comment:

Sadly, Windows doesn't like much your big change.

Just one example:

 2>C:\buildbot.python.org\3.x.kloth-win64\build\Include\Python-ast.h(563): 
warning C4005: 'Yield': macro redefinition 
[C:\buildbot.python.org\3.x.kloth-win64\build\PCbuild\pythoncore.vcxproj]
 C:\Program Files (x86)\Windows Kits\8.1\Include\um\winbase.h(97): 
note: see previous definition of 'Yield'
 bltinmodule.c

http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.x/builds/954/steps/compile/logs/stdio

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30860] Consolidate stateful C globals under a single struct.

[Eric Snow]

Eric Snow added the comment:


New changeset 76d5abc8684bac4f2fc7cccfe2cd940923357351 by Eric Snow in branch 
'master':
bpo-30860: Consolidate stateful runtime globals. (#2594)
https://github.com/python/cpython/commit/76d5abc8684bac4f2fc7cccfe2cd940923357351


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30860] Consolidate stateful C globals under a single struct.

[Nick Coghlan]

Nick Coghlan added the comment:

The core motivation driving the original refactoring was to better understand 
and consolidate our runtime state in order to clarify what the GIL is actually 
protecting (aside from the reference counts). That then turned out to have 
surprising performance benefits (presumably by way of loading all of this state 
permanently into the CPU cache and then keeping it there).

The changes to the public include files arise more from the fact we don't 
currently have a common place to put internal CPython headers for declarations 
we want to share across compilation modules.

I agree the latter is confusing, and it could be worth having a separate 
"Include/internal/CPython.h" header that outright failed the build if you 
attempted to include it without Py_BUILD_CORE set.

(This also ties into my comments on the PR, where I believe the current 
approach of mixing public and private state in the same structs will cause 
problems with compilers generating incorrect field offsets in extension modules 
and embedding applications)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30860] Consolidate stateful C globals under a single struct.

[Antoine Pitrou]

Antoine Pitrou added the comment:

After looking at the PR, I'm a bit skeptical about this.  Suddenly a lot of 
things which are implementation details get moved to the public include files, 
which makes things more confusing from my POV.  I also don't know why all 
globals should be consolidated in a single place (it is not a widespread policy 
in other projects AFAIK, but I may be mistaken).

Perhaps it would be more reasonable to consolidate the globals in each .c file 
independently, without trying to join all of them together in one place nor 
expose them in the public include files.

--
nosy: +pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30860] Consolidate stateful C globals under a single struct.

[Jeremy Kloth]

Changes by Jeremy Kloth :


--
nosy: +jkloth

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30860] Consolidate stateful C globals under a single struct.

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

See also issue29881.

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30860] Consolidate stateful C globals under a single struct.

[Eric Snow]

Changes by Eric Snow :


--
pull_requests: +2665

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30860] Consolidate stateful C globals under a single struct.

[Eric Snow]

Eric Snow added the comment:

One thing I'd like to also try is to use a freelist embedded in _PyRuntimeState 
for the PyInterpreterState and PyThreadState linked lists.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30860] Consolidate stateful C globals under a single struct.

[Eric Snow]

New submission from Eric Snow:

CPython's C code makes extensive use of global variables.  The globals fall 
into one of several categories:

* (effectively) constants (incl. static types)
* freelists, caches, and counters
* used exclusively in main or in REPL
* process-global state
* module state
* Python runtime state

Those in the last category are not explicitly organized nor easily 
discoverable.  Among other things, this has an impact on efforts that change 
the runtime (e.g. my multi-core Python project).  To improve the situation I'd 
like to do the following:

1. group the (stateful) runtime globals into various topical structs
2. consolidate the topical structs under a single top-level _PyRuntimeState 
struct
3. add a check-c-globals.py script that helps identify runtime globals

One side effect of consolidating these globals is a significant performance 
improvement of CPython.  Presumably this is due to the caching behavior of a 
single struct vs. that of dozens of separate globals.

I have a patch and will put up a PR momentarily.

--
assignee: eric.snow
messages: 297780
nosy: eric.snow, haypo, ncoghlan, yselivanov
priority: normal
severity: normal
stage: patch review
status: open
title: Consolidate stateful C globals under a single struct.
type: enhancement
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com