[issue36142] Add a new _PyPreConfig step to Python initialization to setup memory allocator and encodings

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 25d13f37aa6743282d0b8b4df687ff8964b2 by Victor Stinner in 
branch 'master':
bpo-36142: PYTHONMALLOC overrides PYTHONDEV (GH-12191)
https://github.com/python/cpython/commit/25d13f37aa6743282d0b8b4df687ff8964b2


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36142] Add a new _PyPreConfig step to Python initialization to setup memory allocator and encodings

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +12186

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36142] Add a new _PyPreConfig step to Python initialization to setup memory allocator and encodings

[STINNER Victor]

STINNER Victor  added the comment:

Ok, the _PyPreConfig structure is now available and used internally. The API 
can likely be enhanced, but I prefer to open new follow-up issues like 
bpo-36202, since this issue has already a long list of changes :-) I close this 
issue.

Nick: thanks for updating the PEP 432 ;-

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36142] Add a new _PyPreConfig step to Python initialization to setup memory allocator and encodings

[STINNER Victor]

STINNER Victor  added the comment:

I created bpo-36202: "Calling Py_DecodeLocale() before _PyPreConfig_Write() can 
produce mojibake".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36142] Add a new _PyPreConfig step to Python initialization to setup memory allocator and encodings

[STINNER Victor]

STINNER Victor  added the comment:

Description of this long serie of changes.

I modified Py_Main() and _Py_InitializeCore() to clearly separate 
"pre-configuration" from "configuration" steps. The pre-configuration now 
decodes temporarily the command line arguments and uses its own command line 
parser to get -E, -I and -X options (-X is needed for -X utf8). The 
pre-configuration is designed to be as small as possible, it configures:

* memory allocators
* LC_CTYPE locale and set the UTF-8 mode

The _PyPreConfig structure has 8 fields:

* allocator
* coerce_c_locale
* coerce_c_locale_warn
* dev_mode
* isolated
* (Windows only) legacy_windows_fs_encoding
* use_environment
* utf8_mode

I had to include fields which have an impact on other fields. Examples:

* dev_mode=1 sets allocator to "default";
* isolated=1 sets use_environment to 0;
* legacy_windows_fs_encoding=& sets utf8_mode to 0.

_PyCoreConfig_Read() is now only called after the memory allocator and the 
locale (LC_CTYPE locale and UTF-8 mode) are properly configured.

I removed the last side effects of _PyCoreConfig_Read(): it no longer modify 
the locale. Same for the new _PyPreConfig_Read(): zero size effect.

The new _PyPreConfig_Write() and _PyCoreConfig_Write() are now responsible to 
write the new configurations.

There are functions to read the configuration from command line arguments:

* _PyPreConfig_ReadFromArgv()
* _PyCoreConfig_ReadFromArgv()

These functions expect a _PyArgv structure which accepts bytes (wchar*) or 
Unicode (wchar_t*).

I moved coreconfig.h from Include/ to Include/cpython/ to be more explicit that 
it's excluded from the stable API and that it's CPython specific.

I moved all config functions to a new Include/internal/pycore_coreconfig.h. 
Functions are internal to allow us to modify us anytime until a proper clean 
public API is designed on top of it.

If _PyPreConfig.allocator is set, _PyPreConfig_Write() re-allocate the 
configuration with the new memory allocator. This tiny thing avoids the new to 
force a specific memory allocator in many functions. I was able to remove the 
following code:

PyMemAllocatorEx old_alloc;
_PyMem_SetDefaultAllocator(PYMEM_DOMAIN_RAW, &old_alloc);
...
PyMem_SetAllocator(PYMEM_DOMAIN_RAW, &old_alloc);

Calling Py_Main() after Py_Initialize() is still supported. In this case, it no 
longer checks the memory allocators name because _PyMem_GetAllocatorsName() 
returns "pymalloc_debug" (or "malloc_debug" if pymalloc is disabled) after 
_PyMem_SetupAllocators("debug") is called: names are diffrent.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36142] Add a new _PyPreConfig step to Python initialization to setup memory allocator and encodings

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 4fffd380a4070aff39b7fd443d90e60746c1b623 by Victor Stinner in 
branch 'master':
bpo-36142: _PyPreConfig_Read() sets LC_CTYPE (GH-12188)
https://github.com/python/cpython/commit/4fffd380a4070aff39b7fd443d90e60746c1b623


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36142] Add a new _PyPreConfig step to Python initialization to setup memory allocator and encodings

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +12183

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36142] Add a new _PyPreConfig step to Python initialization to setup memory allocator and encodings

[STINNER Victor]

STINNER Victor  added the comment:


New changeset c656e25667c9acc0d13e5bb16d3df2938d0f614b by Victor Stinner in 
branch 'master':
bpo-36142: Add _PyPreConfig_SetAllocator() (GH-12187)
https://github.com/python/cpython/commit/c656e25667c9acc0d13e5bb16d3df2938d0f614b


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36142] Add a new _PyPreConfig step to Python initialization to setup memory allocator and encodings

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +12182

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36142] Add a new _PyPreConfig step to Python initialization to setup memory allocator and encodings

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 7d2ef3ef5042356aaeaf832ad4204b7dad2e1b8c by Victor Stinner in 
branch 'master':
bpo-36142: _PyPreConfig_Write() sets the allocator (GH-12186)
https://github.com/python/cpython/commit/7d2ef3ef5042356aaeaf832ad4204b7dad2e1b8c


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36142] Add a new _PyPreConfig step to Python initialization to setup memory allocator and encodings

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +12181

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36142] Add a new _PyPreConfig step to Python initialization to setup memory allocator and encodings

[STINNER Victor]

STINNER Victor  added the comment:


New changeset a9df651eb4c18a07ec309df190419613e95cba7b by Victor Stinner in 
branch 'master':
bpo-36142: Add _PyMem_GetDebugAllocatorsName() (GH-12185)
https://github.com/python/cpython/commit/a9df651eb4c18a07ec309df190419613e95cba7b


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36142] Add a new _PyPreConfig step to Python initialization to setup memory allocator and encodings

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +12180

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36142] Add a new _PyPreConfig step to Python initialization to setup memory allocator and encodings

[STINNER Victor]

STINNER Victor  added the comment:


New changeset b35be4b3334fbc471a39abbeb68110867b72e3e5 by Victor Stinner in 
branch 'master':
bpo-36142: Add _PyPreConfig.allocator (GH-12181)
https://github.com/python/cpython/commit/b35be4b3334fbc471a39abbeb68110867b72e3e5


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36142] Add a new _PyPreConfig step to Python initialization to setup memory allocator and encodings

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +12177

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36142] Add a new _PyPreConfig step to Python initialization to setup memory allocator and encodings

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 5a02e0d1c8a526fc4e80a2fb8b4a9d5bc64c7d82 by Victor Stinner in 
branch 'master':
bpo-36142: Add _PyPreConfig.utf8_mode (GH-12174)
https://github.com/python/cpython/commit/5a02e0d1c8a526fc4e80a2fb8b4a9d5bc64c7d82


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36142] Add a new _PyPreConfig step to Python initialization to setup memory allocator and encodings

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +12171

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36142] Add a new _PyPreConfig step to Python initialization to setup memory allocator and encodings

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 6dcb54228e7520abd058897440c26e323f62afcd by Victor Stinner in 
branch 'master':
bpo-36142: Add _PyPreConfig_ReadFromArgv() (GH-12173)
https://github.com/python/cpython/commit/6dcb54228e7520abd058897440c26e323f62afcd


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36142] Add a new _PyPreConfig step to Python initialization to setup memory allocator and encodings

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +12170

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36142] Add a new _PyPreConfig step to Python initialization to setup memory allocator and encodings

[STINNER Victor]

STINNER Victor  added the comment:


New changeset cad1f747da47849ab5d8b0b881f7a0b94564d290 by Victor Stinner in 
branch 'master':
bpo-36142: Add _PyPreConfig structure (GH-12172)
https://github.com/python/cpython/commit/cad1f747da47849ab5d8b0b881f7a0b94564d290


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36142] Add a new _PyPreConfig step to Python initialization to setup memory allocator and encodings

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +12169

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36142] Add a new _PyPreConfig step to Python initialization to setup memory allocator and encodings

[Nick Coghlan]

Nick Coghlan  added the comment:

PEP 432 tweaked: https://github.com/python/peps/pull/904/files

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36142] Add a new _PyPreConfig step to Python initialization to setup memory allocator and encodings

[Nick Coghlan]

Nick Coghlan  added the comment:

Agreed - I think the biggest thing we learned from the pre-implementation in 
Python 3.7 is that the "Let's move as much config as we can to Python C API 
data types" fell down in a couple of areas:

1. The embedding application is likely to speak char* and/or wchar_* natively, 
not PyObject*, and this applies even for CPython's own current `Py_Main` 
implementation.

2. There's some core system libc interaction scaffolding that we need in place 
first, giving 3 phases, not two:

- initialise anything needed to read configuration settings from the 
environment and command line (i.e. memory allocators, interface encodings)
- initialise the things needed to execute builtin and frozen Python modules 
(core data types, random hash seed, etc)
- initialise the things needed to execute external Python modules (sys.path, 
etc)

I'll update PEP 432 so it at least mentions some of the lessons learned, and 
points to the current internal configuration API definitions in the CPython 
source tree.

--
nosy: +ncoghlan

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36142] Add a new _PyPreConfig step to Python initialization to setup memory allocator and encodings

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 62be763348d16ba90f96667aa0240503261393f0 by Victor Stinner in 
branch 'master':
bpo-36142: Remove _PyMain structure (GH-12120)
https://github.com/python/cpython/commit/62be763348d16ba90f96667aa0240503261393f0


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36142] Add a new _PyPreConfig step to Python initialization to setup memory allocator and encodings

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 91b9ecf82c3287b45f39158c5134a87414ff26bc by Victor Stinner in 
branch 'master':
bpo-36142: Add preconfig.c (GH-12128)
https://github.com/python/cpython/commit/91b9ecf82c3287b45f39158c5134a87414ff26bc


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36142] Add a new _PyPreConfig step to Python initialization to setup memory allocator and encodings

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +12131

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36142] Add a new _PyPreConfig step to Python initialization to setup memory allocator and encodings

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 95e2cbf32f8156c239b27dae558ba058d0f2d496 by Victor Stinner in 
branch 'master':
bpo-36142: Move command line parsing to coreconfig.c (GH-12123)
https://github.com/python/cpython/commit/95e2cbf32f8156c239b27dae558ba058d0f2d496


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36142] Add a new _PyPreConfig step to Python initialization to setup memory allocator and encodings

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +12127

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36142] Add a new _PyPreConfig step to Python initialization to setup memory allocator and encodings

[STINNER Victor]

STINNER Victor  added the comment:


New changeset dfe884759d1f4441c889695f8985bc9feb9f37eb by Victor Stinner in 
branch 'master':
bpo-36142: Rework error reporting in pymain_main() (GH-12113)
https://github.com/python/cpython/commit/dfe884759d1f4441c889695f8985bc9feb9f37eb


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36142] Add a new _PyPreConfig step to Python initialization to setup memory allocator and encodings

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +12125

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36142] Add a new _PyPreConfig step to Python initialization to setup memory allocator and encodings

[STINNER Victor]

STINNER Victor  added the comment:

TODO: check if _Py_ClearFileSystemEncoding() uses the right memory allocator. 
_Py_SetFileSystemEncoding() doesn't change temporarily the memory allocator.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36142] Add a new _PyPreConfig step to Python initialization to setup memory allocator and encodings

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +12119

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36142] Add a new _PyPreConfig step to Python initialization to setup memory allocator and encodings

[STINNER Victor]

STINNER Victor  added the comment:


New changeset f684d83d86e1990784816d4b243d724e6ab8304f by Victor Stinner in 
branch 'master':
bpo-36142: Exclude coreconfig.h from Py_LIMITED_API (GH-12111)
https://github.com/python/cpython/commit/f684d83d86e1990784816d4b243d724e6ab8304f


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36142] Add a new _PyPreConfig step to Python initialization to setup memory allocator and encodings

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +12118

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36142] Add a new _PyPreConfig step to Python initialization to setup memory allocator and encodings

[STINNER Victor]

STINNER Victor  added the comment:

PR 12087 is a WIP change which implements everything as a single commit.

I'm not 100% sure yet that it's best approach for Python initialization, but 
I'm sure that it solves real interdependencies issues between _PyCoreConfig 
parameters. IHMO have a "pre-initialization" step to setup the memory allocator 
and the encodings is a better design.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36142] Add a new _PyPreConfig step to Python initialization to setup memory allocator and encodings

[STINNER Victor]

Change by STINNER Victor :


--
keywords: +patch
pull_requests: +12097
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36142] Add a new _PyPreConfig step to Python initialization to setup memory allocator and encodings

[STINNER Victor]

New submission from STINNER Victor :

I added a _PyCoreConfig structure to Python 3.7 which contains almost all 
parameters used to configure Python. Problems: _PyCoreConfig uses bytes and 
Unicode strings (char* and wchar_t*) whereas it is also used to setup the 
memory allocator and (filesystem, locale and stdio) encodings.

I propose to add a new _PyPreConfig which is the "strict minimum" configuration 
to setup encodings and the memory allocator. In practice, it also contains 
parameters which directly or indirectly impacts the allocator and encodings. 
For example, isolated impacts use_environment which impacts the allocator 
(PYTHONMALLOC environment variable). Another example: dev_mode=1 sets the 
allocator to "debug".

The command line arguments are now parsed twice. _PyPreConfig only parses a few 
parameters like -E, -I and -X. A temporary _PyPreCmdline is used to store 
command line arguments like -X options.

I moved structures closer to where they are used. "Global" _PyMain structure 
has been removed. _PyCmdline now lives way shorter than previously and is moved 
from main.c to coreconfig.c. The idea is to better control when and how memory 
is allocated.

In term of API, we get something like:

_PyCoreConfig config = _PyCoreConfig_INIT;
config.preconfig.stdio_encoding = "iso8859-1";
config.preconfig.stdio_errors = "replace";
config.user_site_directory = 0;
...

_PyInitError err = _Py_InitializeFromConfig(&config);
if (_Py_INIT_FAILED(err)) {
_Py_ExitInitError(err);
}
...
Py_Finalize();
return 0;

"config.preconfig.stdio_errors" syntax isn't great, but it's simpler to 
implement than duplicating all _PyPreConfig fields into _PyCoreConfig.

--
components: Interpreter Core
messages: 336791
nosy: vstinner
priority: normal
severity: normal
status: open
title: Add a new _PyPreConfig step to Python initialization to setup memory 
allocator and encodings
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com