[issue33042] New 3.7 startup sequence crashes PyInstaller

Nick Coghlan added the comment: Just noting that while I've closed this issue as fixed, I think we still have quite a bit of work to do to get our handling of these pre-init config settings to a place we're genuinely happy with (or as happy as backwards compatibility

[issue33042] New 3.7 startup sequence crashes PyInstaller

Change by Nick Coghlan : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___

[issue33042] New 3.7 startup sequence crashes PyInstaller

Nick Coghlan added the comment: Harmut, thanks for the status update! Victor, the problem is that both sys.warnoptions and sys._xoptions get read by Py_Initialize(), so setting them afterwards is essentially pointless - while you do still change the contents of the sys

[issue33042] New 3.7 startup sequence crashes PyInstaller

Hartmut Goebel added the comment: I can confirm that c6d94c37f4fd863c73fbfbcc918fd23b458b5301 makes the PyInstaller test-case, which war the origin of this bug-report, pass. Thanks for fixing! -- ___ Python tracker

[issue33042] New 3.7 startup sequence crashes PyInstaller

STINNER Victor added the comment: Would it be possible to not officially allow calling PySys_AddWarnOption() before Py_Initialize()? Fix the bug, but don't promote doing that. I just suggest to revert: Doc/c-api/init.rst: + * :c:func:`PySys_AddWarnOption` + *

[issue33042] New 3.7 startup sequence crashes PyInstaller

Nick Coghlan added the comment: New changeset bc77eff8b96be4f035e665ab35c1d06e22f46491 by Nick Coghlan in branch 'master': bpo-33042: Fix pre-initialization sys module configuration (GH-6157) https://github.com/python/cpython/commit/bc77eff8b96be4f035e665ab35c1d06e22f46491

[issue33042] New 3.7 startup sequence crashes PyInstaller

Nick Coghlan added the comment: While I've merged an initial fix for this (so it should be working again in 3.7.0b3), I'm not going to close the issue until folks have had a chance to review and comment on the linked list based approach I've taken. -- versions:

[issue33042] New 3.7 startup sequence crashes PyInstaller

miss-islington added the comment: New changeset c6d94c37f4fd863c73fbfbcc918fd23b458b5301 by Miss Islington (bot) in branch '3.7': bpo-33042: Fix pre-initialization sys module configuration (GH-6157)

[issue33042] New 3.7 startup sequence crashes PyInstaller

Change by miss-islington : -- pull_requests: +5971 ___ Python tracker ___

[issue33042] New 3.7 startup sequence crashes PyInstaller

Nick Coghlan added the comment: I added the new test case, and found to my surprise that it didn't fail, even in debug mode (where there aren't any default filters). The point I had missed was that even though warnoptions can be NULL in CoreConfig,

[issue33042] New 3.7 startup sequence crashes PyInstaller

Nick Coghlan added the comment: PR has been updated to be mostly complete (just pending docs changes now), but I think I've found a potential issue with the interaction between the way I've currently implemented it and the way _Py_InitializeCore and

[issue33042] New 3.7 startup sequence crashes PyInstaller

Nick Coghlan added the comment: Thinking about PySys_AddWarnOptionUnicode a little further, I'm not sure we actually need to change anything in the implementation there - I think we can just make it clearer in the docs that *only* PySys_AddWarnOption is supported prior to

[issue33042] New 3.7 startup sequence crashes PyInstaller

Nick Coghlan added the comment: The linked PR has the draft test case for this - it goes beyond the ones I used to find the cause of the problem by actually checking that sys.warnoptions and sys._xoptions have been populated as expected. --

[issue33042] New 3.7 startup sequence crashes PyInstaller

Change by Nick Coghlan : -- pull_requests: +5914 stage: -> patch review ___ Python tracker ___

[issue33042] New 3.7 startup sequence crashes PyInstaller

Nick Coghlan added the comment: This shouldn't affect the public ABI, so I don't think it's a blocker for the ABI freeze, but it's a regression that effectively makes PySys_AddWarnOption (and related APIs) unusable, so we should definitely fix it before shipping 3.7. For

[issue33042] New 3.7 startup sequence crashes PyInstaller

Change by pmpp : -- nosy: +pmpp ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue33042] New 3.7 startup sequence crashes PyInstaller

Ned Deily added the comment: Nick, what's your assessment of this issue's priority? Is it a release blocker for 3.7.0b3 (proposed ABI freeze)? For 3.7.0rc1 (code freeze)? None of the above? -- nosy: +ned.deily ___ Python

[issue33042] New 3.7 startup sequence crashes PyInstaller

Change by Jonathan Springer : -- nosy: +springermac ___ Python tracker ___ ___

[issue33042] New 3.7 startup sequence crashes PyInstaller

Nick Coghlan added the comment: Patched test_capi results for 2ebc5ce42a8a9e047e790aefbf9a94811569b2b6 (the global state consolidation commit): both pre-initialization tests fail Patched test_capi results for bab21faded31c70b142776b9a6075a4cda055d7f (the immediately

[issue33042] New 3.7 startup sequence crashes PyInstaller

Nick Coghlan added the comment: While I haven't definitely narrowed it down yet, I suspect it isn't your changes that are the problem: since the reported crash relates to attempting to access a not-yet-created thread state, `warnoptions` and `_xoptions` likely need to

[issue33042] New 3.7 startup sequence crashes PyInstaller

STINNER Victor added the comment: I suggest to revert my changes on PySys_AddWarnOption*() and "just" re-use the code from Python 3.6. Previously, I made the code more complex since I had to expose a private function to add manually warning options. But later, the

[issue33042] New 3.7 startup sequence crashes PyInstaller

Nick Coghlan added the comment: After reworking the patch to backport the pre-initialization embedding tests to 3.7.0a1, they *both* failed, as that was before Victor fixed Py_DecodeLocale to work prior to initialization again. As a result, I tried

[issue33042] New 3.7 startup sequence crashes PyInstaller

Nick Coghlan added the comment: https://github.com/python/cpython/commit/b4d1e1f7c1af6ae33f0e371576c8bcafedb099db (the first attempted fix for bpo-20891) is the last commit where the draft test case patch applies cleanly. That still segfaults, so I'll jump all the way

[issue33042] New 3.7 startup sequence crashes PyInstaller

Nick Coghlan added the comment: Adding a bit of context from my prior email discussion with Hartmut: CPython actually reads sys.warnoptions at the end of Py_Initialize (its the last thing we do before the function returns). It also reads sys._xoptions during startup,

[issue33042] New 3.7 startup sequence crashes PyInstaller

STINNER Victor added the comment: Python documentation was enhanced in Python 3.7 to explicitly list all functions safe to call *before* Py_Initialize(): https://docs.python.org/dev/c-api/init.html#before-python-initialization PySys_AddWarnOption() is not part of

[issue33042] New 3.7 startup sequence crashes PyInstaller

Change by Nick Coghlan : -- assignee: -> ncoghlan ___ Python tracker ___ ___

[issue33042] New 3.7 startup sequence crashes PyInstaller

Nick Coghlan added the comment: I don't think we made any start-up changes that were specific to PySys_AddWarnOption, so my suspicion is that the crash is going to be related to a change in the constraints on either the unicode object creation or the list append

[issue33042] New 3.7 startup sequence crashes PyInstaller

Change by Ned Deily : -- nosy: +eric.snow, vstinner ___ Python tracker ___ ___

[issue33042] New 3.7 startup sequence crashes PyInstaller

New submission from Hartmut Goebel : PyInstaller is a tool for freezing Python applications into stand-alone packages, much like py2exe. py2app, and bbfreeze. PyInstaller is providing *one* bootloader for all versions of Python supported (2.7, 3.4-3.6). In