[issue32280] Expose `_PyRuntime` through a section name

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:


New changeset ebba286709bdb9401df8e5eb196b50008d3b09d2 by Miss Islington (bot) 
in branch '3.10':
bpo-32280: Store _PyRuntime in a named section (GH-4802) (#27448)
https://github.com/python/cpython/commit/ebba286709bdb9401df8e5eb196b50008d3b09d2


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32280] Expose `_PyRuntime` through a section name

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 12.0 -> 13.0
pull_requests: +25977
pull_request: https://github.com/python/cpython/pull/27448

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32280] Expose `_PyRuntime` through a section name

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:


New changeset 35002aa8f62dda1f79035e9904abdf476683e9be by Max Bélanger in 
branch 'main':
bpo-32280: Store _PyRuntime in a named section (GH-4802)
https://github.com/python/cpython/commit/35002aa8f62dda1f79035e9904abdf476683e9be


--
nosy: +pablogsal

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32280] Expose `_PyRuntime` through a section name

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32280] Expose `_PyRuntime` through a section name

[Steve Dower]

Steve Dower  added the comment:

It looks fine to me, but we probably want to clearly document it as 
unsupported/likely to change between releases/etc., especially since it's 
relying on an assumption that we want to change (one static runtime per 
process).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32280] Expose `_PyRuntime` through a section name

[Cheryl Sabella]

Cheryl Sabella  added the comment:

@steve.dower, please take a look at this PR when you have a chance.  Thanks!

--
nosy: +cheryl.sabella

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32280] Expose `_PyRuntime` through a section name

[Maxime Belanger]

Maxime Belanger  added the comment:

Thanks for taking a look! To answer your question, the need for the named 
section comes not from the API being being "internal", but because we need to 
access it at runtime from a tool running in a separate process.

We have augmented Crashpad (Google's crash reporter) tool, which inspects the 
memory of a crashing process to create/upload a report, to access each native 
thread's stack and reconstruct its Python stack frames. This allows us to 
quickly make sense of native crashes involving Python code. 

To do this, the tool needs to know where the Python runtime stores state within 
thread-local storage. This is actually the only reason we need to access 
`_PyRuntime`: it allows us to retrieve `autoTSSKey` for the process, which we 
can use to look up the `PyThreadState` for each underlying/native thread.

At the time, `_PyRuntime` seemed like a natural structure to expose and a named 
section a simple-enough way of doing so. We're certainly open to alternatives 
if you think there's a better way.

Given your plans for 3.9: I'm assuming `autoTSSKey` will remain the same 
per-process?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32280] Expose `_PyRuntime` through a section name

[Eric Snow]

Eric Snow  added the comment:

(Sorry for the delay!)

Is the need for a named section due to the fact that _PyRuntime is part of the 
"internal" C-API (hidden behind the Py_BUILD_CORE macro)?  I.E. would it help 
to build with Py_BUILD_CORE?


Regarding the GIL:

The plan for 3.9 is to have one GIL per interpreter, so the relevant struct 
would be PyInterpreterState (a part of the public C-API).  That would be 
accessible at runtime via the fields of _PyRuntime.interpreters.


Regarding what I said before about multiple runtimes per process:

Honestly, I've yet to see any use case that would justify providing 
multi-runtime support.  However, there are code-health benefits to keeping 
*all* runtime state out of static globals.  So eliminating the _PyRuntime 
static variable is still a realistic possibility (for 3.9 or maybe later).

If that happens then there would have to be a new embedding C-API oriented 
around (opaque) PyRuntimestate pointers.  See PEP 432 and 587 for ongoing work 
to improve the embedding experience (for runtime initialization), including 
what the successors to Py_Initialize() look like.

--
versions: +Python 3.9 -Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32280] Expose `_PyRuntime` through a section name

[Eric Snow]

Change by Eric Snow :


--
versions: +Python 3.8 -Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32280] Expose `_PyRuntime` through a section name

[STINNER Victor]

Change by STINNER Victor :


--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32280] Expose `_PyRuntime` through a section name

[Maxime Belanger]

Maxime Belanger  added the comment:

Interesting, would this imply potentially multiple GILs? The major thing we 
need out of the structure is the (`Py_tss_t`) `autoTSSKey` in order to 
associate a native thread with its `PyThreadState`.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32280] Expose `_PyRuntime` through a section name

[Eric Snow]

Eric Snow  added the comment:

Note that in the long term we are considering support for embedding multiple 
runtimes in a single process.  So anything that assumes there is only a single 
runtime in each process is problematic.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32280] Expose `_PyRuntime` through a section name

[Max Bélanger]

Change by Max Bélanger :


--
keywords: +patch
pull_requests: +4700
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32280] Expose `_PyRuntime` through a section name

[Steve Dower]

Change by Steve Dower :


--
nosy: +eric.snow

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32280] Expose `_PyRuntime` through a section name

[Steve Dower]

Change by Steve Dower :


--
nosy: +ncoghlan

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32280] Expose `_PyRuntime` through a section name

[Maxime Belanger]

New submission from Maxime Belanger :

We've recently been toying with more sophisticated crash reporting machinery 
for our Desktop Python application (which we deploy on macOS, Windows and 
Linux).

To assist in debugging, we are storing `_PyRuntime` in a predictable location 
our crash reporter can later look up without needing the symbol table (this can 
grow complicated across multiple platforms).

Taking a page from `crashpad`'s book 
(https://chromium.googlesource.com/crashpad/crashpad/+/master/client/crashpad_info.cc),
 we've patched `pylifecycle.c` to store the `_PyRuntime` struct in a section of 
the same name. Upon a crash,  this section is then used by the tool to annotate 
each report with enough information to reconstruct the Python stack frames in 
each thread (as applicable).

We're contributing our patch here in the hopes this can be helpful to others.

--
components: Interpreter Core, Windows, macOS
messages: 308079
nosy: Maxime Belanger, ned.deily, paul.moore, ronaldoussoren, steve.dower, 
tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: Expose `_PyRuntime` through a section name
type: enhancement
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com