[issue32516] Add a shared library mechanism for win32

Marc-Andre Lemburg added the comment: On 17.01.2018 02:52, xoviat wrote: > > xoviat added the comment: > > For the record, moving the DLL path manipulation code into the interpreter > would address the concern that importing a module would not manipulate

[issue32516] Add a shared library mechanism for win32

Nathaniel Smith added the comment: > That looks like recompilation (or at least how recompilation would look if > you'd been granted permission). Assuming you can recompile the binary, you > could rename the dependency and regenerate the import library so that you can > link

[issue32516] Add a shared library mechanism for win32

xoviat added the comment: Just to be clear, I'm not considering doing this with respect to the C/C++ runtimes, but all of the other shared libraries. And in case you weren't aware, this is exactly what auditwheel does (except that I believe it uses patchelf, whereas I will

[issue32516] Add a shared library mechanism for win32

Steve Dower added the comment: That looks like recompilation (or at least how recompilation would look if you'd been granted permission). Assuming you can recompile the binary, you could rename the dependency and regenerate the import library so that you can link

[issue32516] Add a shared library mechanism for win32

Nathaniel Smith added the comment: > it still won't help if another import has loaded a different DLL by the same > name. This is where you can't avoid recompilation or activation contexts. Ah, there's an important bit of context you're missing: there is actually a third

[issue32516] Add a shared library mechanism for win32

Steve Dower added the comment: The import hook could theoretically modify the search path before and after loading the module, though that could make debugging a real pain. I was also thinking of just having an explicit list of DLLs to load, rather than inspecting the

[issue32516] Add a shared library mechanism for win32

xoviat added the comment: For the record, moving the DLL path manipulation code into the interpreter would address the concern that importing a module would not manipulate the search path because the behavior would move into Python itself. --

[issue32516] Add a shared library mechanism for win32

Nathaniel Smith added the comment: Steve said: > The import hook was my proposal to let you only preload DLLs when the > extension module is being loaded, rather than having to load all the DLLs on > the first "import scipy" just in case one of its submodules gets imported >

[issue32516] Add a shared library mechanism for win32

Steve Dower added the comment: I am very okay with it not being Python's responsibility to do this. -- resolution: -> rejected stage: -> resolved status: open -> closed ___ Python tracker

[issue32516] Add a shared library mechanism for win32

xoviat added the comment: This is what ray said: "Please do not do this. Importing a specific module should not modify the way that process loads subsequent DLLs." (https://github.com/numpy/numpy/pull/10229#issuecomment-354846459) What I'm proposing to do is write a tool,

[issue32516] Add a shared library mechanism for win32

Steve Dower added the comment: I understood the proposal just fine, and I understand the problems involved and appreciate why the ideal isn't sufficient here. The import hook was my proposal to let you only preload DLLs when the extension module is being loaded,

[issue32516] Add a shared library mechanism for win32

Nathaniel Smith added the comment: > However, the principal reason that I'm opening this issue is that Ray > complained that packages shouldn't be altering the DLL search path If that's crucial context, can you link to it? -- ___

[issue32516] Add a shared library mechanism for win32

xoviat added the comment: As Nathaniel noted, the "solution" of placing DLLs in the same directory as extension modules is not a solution. I also think that some people here misunderstand my proposal: I'm not necessarily proposing that these directories are added using an

[issue32516] Add a shared library mechanism for win32

Nathaniel Smith added the comment: Conda is cool but we're not currently planning to abandon wheels. -- ___ Python tracker ___

[issue32516] Add a shared library mechanism for win32

Marc-Andre Lemburg added the comment: Probably better overall to go with a conda package which puts the DLLs in a central location and manages the dependencies. You can then load the DLL in the package before loading the PYD and you're all set. Whether in an __init__.py or

[issue32516] Add a shared library mechanism for win32

Nathaniel Smith added the comment: Putting .dll's next to their .pyd's isn't a general solution, because sometimes you want to be able to use the same .dll's from .pyd's that are in different directories. For example, scipy.spatial.qhull, scipy.optimize._lbfsgb, and

[issue32516] Add a shared library mechanism for win32

Brett Cannon added the comment: To answer Steve's question, what you would want is a finder which recognized the directory of the package so as to return a special loader just for that package (basically __path__ is sent through the normal import mechanism and so you would

[issue32516] Add a shared library mechanism for win32

Marc-Andre Lemburg added the comment: >From experience with doing something similar in egenix-pyopenssl, I recommend >putting the DLLs into the same directory as the PYD file on Windows. If you >want to be extra safe, you can explicitly load the DLL, but normally this is

[issue32516] Add a shared library mechanism for win32

Steve Dower added the comment: The more details you reveal, the less I want it to be an officially supported pattern :) Perhaps you actually want an import hook? Brett can confirm, but I believe you can overwrite __loader__ in a package and change how submodules are

[issue32516] Add a shared library mechanism for win32

xoviat added the comment: My current plan is to patch the __init__ package files to add the '.libs' folder to the search path. However, I think it would be better for Python to do this so that there is a consistent mechanism for loading shared libraries. --

[issue32516] Add a shared library mechanism for win32

xoviat added the comment: So the idea here is actually to write a replacement for auditwheel that works on windows. In order to do that, I need to recursively analyze DLL dependencies, randomize the DLL filenames, rewrite the pe header table, and then copy them into the

[issue32516] Add a shared library mechanism for win32

Eryk Sun added the comment: > on Windows, the rpath mechanism doesn't exist It seems we can locate a dependent assembly up to two directories up from a DLL using a relative path. According to MSDN [1], this is supported in Windows 7+. 3.7 no longer supports Vista, so this

[issue32516] Add a shared library mechanism for win32

Steve Dower added the comment: How is this an improvement over loading the DLL explicitly? Even if subpackages require the file, your top level __init__.py will run before they are loaded and it can import a pyd that's known to be next to the file or it can

[issue32516] Add a shared library mechanism for win32

Change by Antoine Pitrou : -- nosy: +paul.moore, steve.dower, tim.golden, zach.ware ___ Python tracker ___

[issue32516] Add a shared library mechanism for win32

Change by Antoine Pitrou : -- nosy: +Ray Donnelly ___ Python tracker ___ ___

[issue32516] Add a shared library mechanism for win32

Change by Barry A. Warsaw : -- nosy: +barry ___ Python tracker ___ ___ Python-bugs-list

[issue32516] Add a shared library mechanism for win32

xoviat added the comment: Sorry, that should have read: 2. A folder two levels under site-packages that is named '.libs' Please consult the auditwheel source to determine the specific pattern if there is doubt. -- ___ Python

[issue32516] Add a shared library mechanism for win32

New submission from xoviat : On linux and macOS, a mechanism exists to relocate shared libraries inside of a wheel. Auditwheel creates a .libs folder and places the shared libraries inside of it. The problem is that on Windows, the rpath mechanism doesn't exist. We've