[issue18309] Make python slightly more relocatable
Shakeeb Alireza added the comment: Thanks, Mathias. This is all about improving python's 'relocatability'. Just to expand on my prior point: the scenario we are talking about is where one embeds python in a host application's plugin system rather than in the host application itself. In this case, sys.executable is the host application and a relocatable plugin embeds a 'python client'. If a full python distribution is not bundled within this client[*], it needs to (1) link to libpythonX.Y.dylib and (2) get the location of the standard library. There are standard cross-platform methods for (1) to be achieved by way of symmetrical @rpath lookups on the client and libpythonX.Y.dylib sides. So this resolvable even in the case when python is compiled with --enabled-shared. However, even if (1) is achieved, the client cannot get, programmatically via the python c-api, the location of libpythonX.Y.dylib (even if it is properly dynamically linking to it), because it cannot rely on sys.executable. I think this is crux of Mathias' argument. Of course there are workarounds, but they are (at least to me) all platform specific. The first and easiest is to just build using the Framework structure and don't ever use --enable-shared, provided you find Greg Neagle's solution (https://bugs.python.org/issue42514) Another workaround which is specific to my context (which I have attached), is to use Apple's CoreFoundation library to get the path to the plugin bundle and from there find our way to the python distribution in the containing folder structure (package). [*] It is possible to insert a full python distribution into a bundle (in the osx meaning), but then it becomes necessarily frozen or 'sealed' due to Apple's codesigning and notarization requirements, and it basically means that the user cannot extend it with further installations of python packages which contain c-extensions unless they jump through some additional codesigning and notarization hoops. -- Added file: https://bugs.python.org/file50677/workaround.c ___ Python tracker <https://bugs.python.org/issue18309> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18309] Make python slightly more relocatable
Shakeeb Alireza added the comment: Thanks, Victor. I can imagine getpath.py will be more hackable (even if it is frozen). Still, it replicates the old algorithm: # Before any searches are done, the location of the executable is # determined. If Py_SetPath() was called, or if we are running on # Windows, the 'real_executable' path is used (if known). Otherwise, # we use the config-specified program name or default to argv[0]. In my case (and I think for Mathias), the executable is a non python application and what is actually dynamically linking to libpythonX.Y.dylib (built via --enable-shared) is a c-based plugin (which calls PyInitialize()), and these two are only aware of their relative locations via the @rpath, @loader_path mechanism. Currently, in this scenario, libpythonX.Y.dylib doesn't know here pythonhome is unless explicitly told via PySetPath() or it defaults to the hardcoded sys.prefix . If this is to be relocatable, then PySetPath() should be able to handle relative paths. -- ___ Python tracker <https://bugs.python.org/issue18309> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18309] Make python slightly more relocatable
Shakeeb Alireza added the comment: I have exactly the same need and use-case as Mathias in my project which includes a requirement to embed python3 in a relocatable folder structure w which serves as an application package (https://github.com/shakfu/py-js). This can be done using the Framework structure, thanks to Greg Neagle's solution referenced in (https://bugs.python.org/issue42514), but not for any python builds with --enabled-shared. In any case, providing options (at the c-level or otherwise, for embedded applications as described by Mathias would be ideal: "I can imagine to provide several functions to build up the pythonpath starting from something. So say, have a 'get python path from argv[0]', a 'get python path from shared python library' and a 'get python path from prefix' function (I may miss a variant)." -- nosy: +shakfu ___ Python tracker <https://bugs.python.org/issue18309> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42514] Relocatable framework for macOS
Shakeeb Alireza added the comment: I have struggled with this exact issue in my py-js project (https://github.com/shakfu/py-js) which embeds a python3 interpreter in a max/msp plugin or in a relocatable folder (package) structure. For the latter case, Greg's solution, which is based on standard methods, has been absolutely crucial. Thank you! I have found that trying to do the same with a python distro built with --enable-shared is basically impossible because in this case sys.prefix is hardcoded and it dos not respond (like the framework structure) to the same @rpath methods used by Greg. It would be great for those who embed python for a fun or profit to have these issues addressed across all of the standard build structures. -- nosy: +shakfu ___ Python tracker <https://bugs.python.org/issue42514> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34309] Trouble when reloading extension modules.
Shakeeb Alireza added the comment: In my project (https://github.com/shakfu/py-js), which provides an embedded python3 interpreter to Max/MSP in the form of an 'external' plugin, I have faced similar issues of being unable to reload extension modules, namely numpy, without reliably crashing the host application, in this case Max. Being able to reload extension modules cleanly is absolutely critical especially in case when python is embedded. Since Numpy is one of the key reasons why people would want to use Python, such a constraint, in this embedded context, becomes a sufficient reason not to use Python at all. For example, I have recently taken note of similar frustration with this exact same issue from the VCV project (https://community.vcvrack.com/t/blowing-the-dust-off-python-in-prototype/12909). I quote: "I should add that CPython and more notably numpy do not support nor advise a complete restart of the interpreter in embedded scenarios without restarting the host process which kind of defeats our purpose in Prototype. At that point I think I can safely take a step back and turn to the dev community looking for suggestions. Should we throw away numpy, or even python, altogether?" -- nosy: +shakfu versions: +Python 3.9 -Python 3.8 ___ Python tracker <https://bugs.python.org/issue34309> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
