Re: Unable to compile my C Extension on Windows: unresolved external link errors
On 11/14/21, Marco Sulla wrote: > On Sun, 14 Nov 2021 at 16:42, Barry Scott wrote: > >> On macOS .dynlib and Unix .so its being extern that does this. > > And extern is the default. I understand now. Per Include/exports.h and Include/pyport.h, Python should be built in Unix with "default" visibility (per global/local binding) for the API PyAPI_FUNC(RTYPE) and PyAPI_DATA(RTYPE) symbols. Everything else with global binding will be hidden via the "-fvisibility=hidden" compiler option that's configured in the makefile. For example: $ readelf -s Objects/dictobject.o | grep HIDDEN | cut -b 40- GLOBAL HIDDEN 4 _pydict_global_version GLOBAL HIDDEN 1 _PyDict_ClearFreeList GLOBAL HIDDEN 1 _PyDict_Fini GLOBAL HIDDEN 1 _PyDictKeys_StringLookup GLOBAL HIDDEN 9 _Py_dict_lookup GLOBAL HIDDEN 1 _PyDict_GetItemHint GLOBAL HIDDEN 1 _PyDict_LoadGlobal GLOBAL HIDDEN 1 _PyDict_Pop_KnownHash GLOBAL HIDDEN 1 _PyDict_FromKeys GLOBAL HIDDEN 1 _PyDict_KeysSize GLOBAL HIDDEN 1 _PyDict_NewKeysForClass GLOBAL HIDDEN 1 _PyObject_InitializeDict GLOBAL HIDDEN 1 _PyObject_MakeDictFromIns GLOBAL HIDDEN 1 _PyObject_StoreInstanceAt GLOBAL HIDDEN 1 _PyObject_GetInstanceAttr GLOBAL HIDDEN 1 _PyObject_IsInstanceDictE GLOBAL HIDDEN 1 _PyObject_VisitInstanceAt GLOBAL HIDDEN 1 _PyObject_ClearInstanceAt GLOBAL HIDDEN 1 _PyObject_FreeInstanceAtt GLOBAL HIDDEN 1 _PyObjectDict_SetItem GLOBAL HIDDEN 1 _PyDictKeys_DecRef GLOBAL HIDDEN 1 _PyDictKeys_GetVersionFor These hidden symbols get linked in the executable or shared-object with local binding. For example: $ readelf -s python | grep _PyDict_FromKeys | cut -b 40- LOCAL DEFAULT 16 _PyDict_FromKeys I suggest testing your project with Python built as a shared library, i.e. --enable-shared. The local binding on internal symbols may be a problem in this case. -- https://mail.python.org/mailman/listinfo/python-list
Re: Unable to compile my C Extension on Windows: unresolved external link errors
On Sun, 14 Nov 2021 at 16:42, Barry Scott wrote: > > Sorry iPad sent the message before it was complete... > > > On 14 Nov 2021, at 10:38, Marco Sulla wrote: > > > > Okay, now the problem seems to be another: I get the same "unresolved > > external link" errors, but only for internal functions. > > > > This seems quite normal. The public .lib does not expose the internals > > of Python. > > The strange fact is: why can I compile it on Linux and MacOS? Their > > external libraries expose the internal functions? > > Windows is not Linux is not macOS, > The toolchain on each OS has its own strengths, weaknesses and quirks. > > On Windows DLLs only allow access to the symbols that are explicitly listed > to be access. Where are those symbols listed? > On macOS .dynlib and Unix .so its being extern that does this. And extern is the default. I understand now. > Maybe you could copy the code that you want and add it to your code? > Change any conflicting symbols of course. It's quite hard. I have to compile dictobject.c, which needs a lot of internal functions. And I suppose that every internal function may require 1 or more other internal functions. I have other two other solutions: * compile a whole python DLL with the symbols I need and link against it. I have to put this DLL in my code, which is ugly. * drop the support of the C Extension for Windows users and make for them the slow, pure py version only. Since my interest in Windows now is near to zero, I think I'll opt for the third for now. -- https://mail.python.org/mailman/listinfo/python-list
Re: Unable to compile my C Extension on Windows: unresolved external link errors
Sorry iPad sent the message before it was complete... > On 14 Nov 2021, at 10:38, Marco Sulla wrote: > > Okay, now the problem seems to be another: I get the same "unresolved > external link" errors, but only for internal functions. > > This seems quite normal. The public .lib does not expose the internals > of Python. > The strange fact is: why can I compile it on Linux and MacOS? Their > external libraries expose the internal functions? Windows is not Linux is not macOS, The toolchain on each OS has its own strengths, weaknesses and quirks. On Windows DLLs only allow access to the symbols that are explicitly listed to be access. On macOS .dynlib and Unix .so its being extern that does this. > > Anyway, is there a way to compile Python on Windows in such a way that > I get a shared library that exposes all the functions? Yes you can do your own build of python that exposes the symbols you want. But that build will be private to you and will not allow others to use your work (on the assumption that they will not use your private build of python). Maybe you could copy the code that you want and add it to your code? Change any conflicting symbols of course. Barry > > On Sat, 13 Nov 2021 at 12:17, Marco Sulla > wrote: >> >> . Sorry, the problem is I downloaded the 32 bit version of VS >> compiler and 64 bit version of Python.. >> >> On Sat, 13 Nov 2021 at 11:10, Barry Scott wrote: >>> >>> >>> On 13 Nov 2021, at 09:00, Barry wrote: > On 12 Nov 2021, at 22:53, Marco Sulla > wrote: > > It seems that on Windows it doesn't find python3.lib, > even if I put it in the path. So I get the `unresolved external link` > errors. I think you need the python310.lib (not sure of file name) to get to the internal symbols. >>> >>> Another thing that you will need to check is that the symbols you are after >>> have been >>> exposed in the DLL at all. Being external in the source is not enough they >>> also have to >>> listed in the .DLL's def file ( is that the right term?) as well. >>> >>> If its not clear yet, you are going to have to read a lot or source code >>> and understand >>> the tool chain used on Windows to solve this. >>> >>> You can use the objdump(?) utility to check that the symbols are in the lib. Barry >>> >>> Barry >>> > -- https://mail.python.org/mailman/listinfo/python-list
Re: Unable to compile my C Extension on Windows: unresolved external link errors
Okay, now the problem seems to be another: I get the same "unresolved external link" errors, but only for internal functions. This seems quite normal. The public .lib does not expose the internals of Python. The strange fact is: why can I compile it on Linux and MacOS? Their external libraries expose the internal functions? Anyway, is there a way to compile Python on Windows in such a way that I get a shared library that exposes all the functions? On Sat, 13 Nov 2021 at 12:17, Marco Sulla wrote: > > . Sorry, the problem is I downloaded the 32 bit version of VS > compiler and 64 bit version of Python.. > > On Sat, 13 Nov 2021 at 11:10, Barry Scott wrote: > > > > > > > > > On 13 Nov 2021, at 09:00, Barry wrote: > > > > > > > > > > > >> On 12 Nov 2021, at 22:53, Marco Sulla > > >> wrote: > > >> > > >> It seems that on Windows it doesn't find python3.lib, > > >> even if I put it in the path. So I get the `unresolved external link` > > >> errors. > > > > > > I think you need the python310.lib (not sure of file name) to get to the > > > internal symbols. > > > > Another thing that you will need to check is that the symbols you are after > > have been > > exposed in the DLL at all. Being external in the source is not enough they > > also have to > > listed in the .DLL's def file ( is that the right term?) as well. > > > > If its not clear yet, you are going to have to read a lot or source code > > and understand > > the tool chain used on Windows to solve this. > > > > > > > > > > You can use the objdump(?) utility to check that the symbols are in the > > > lib. > > > > > > Barry > > > > Barry > > -- https://mail.python.org/mailman/listinfo/python-list
Re: Unable to compile my C Extension on Windows: unresolved external link errors
. Sorry, the problem is I downloaded the 32 bit version of VS compiler and 64 bit version of Python.. On Sat, 13 Nov 2021 at 11:10, Barry Scott wrote: > > > > > On 13 Nov 2021, at 09:00, Barry wrote: > > > > > > > >> On 12 Nov 2021, at 22:53, Marco Sulla wrote: > >> > >> It seems that on Windows it doesn't find python3.lib, > >> even if I put it in the path. So I get the `unresolved external link` > >> errors. > > > > I think you need the python310.lib (not sure of file name) to get to the > > internal symbols. > > Another thing that you will need to check is that the symbols you are after > have been > exposed in the DLL at all. Being external in the source is not enough they > also have to > listed in the .DLL's def file ( is that the right term?) as well. > > If its not clear yet, you are going to have to read a lot or source code and > understand > the tool chain used on Windows to solve this. > > > > > > You can use the objdump(?) utility to check that the symbols are in the lib. > > > > Barry > > Barry > -- https://mail.python.org/mailman/listinfo/python-list
Re: Unable to compile my C Extension on Windows: unresolved external link errors
> On 13 Nov 2021, at 09:00, Barry wrote: > > > >> On 12 Nov 2021, at 22:53, Marco Sulla wrote: >> >> It seems that on Windows it doesn't find python3.lib, >> even if I put it in the path. So I get the `unresolved external link` >> errors. > > I think you need the python310.lib (not sure of file name) to get to the > internal symbols. Another thing that you will need to check is that the symbols you are after have been exposed in the DLL at all. Being external in the source is not enough they also have to listed in the .DLL's def file ( is that the right term?) as well. If its not clear yet, you are going to have to read a lot or source code and understand the tool chain used on Windows to solve this. > > You can use the objdump(?) utility to check that the symbols are in the lib. > > Barry Barry -- https://mail.python.org/mailman/listinfo/python-list
Re: Unable to compile my C Extension on Windows: unresolved external link errors
> On 12 Nov 2021, at 22:53, Marco Sulla wrote: > > It seems that on Windows it doesn't find python3.lib, > even if I put it in the path. So I get the `unresolved external link` > errors. I think you need the python310.lib (not sure of file name) to get to the internal symbols. You can use the objdump(?) utility to check that the symbols are in the lib. Barry -- https://mail.python.org/mailman/listinfo/python-list
Re: Unable to compile my C Extension on Windows: unresolved external link errors
On Fri, 12 Nov 2021 at 21:09, Chris Angelico wrote: > > On Sat, Nov 13, 2021 at 7:01 AM Marco Sulla > wrote: > > On Fri, 12 Nov 2021 at 17:38, Chris Angelico wrote: > > > Are you sure that you really need Py_BUILD_CORE? > > > > Yes, because I need the internal functions of `dict`. So I need to > > compile also dictobject.c and include it. So I need that flag. > > > > This is the code: > > > > https://github.com/Marco-Sulla/python-frozendict.git > > > > Ah, gotcha. > > Unfortunately that does mean you're delving deep into internals, and a > lot of stuff that isn't designed for extensions to use. So my best > recommendation is: dig even deeper into internals, and duplicate how > the core is doing things (maybe including another header or > something). It may be that, by declaring Py_BUILD_CORE, you're getting > a macro version of that instead of the normal exported function. I've not understood what I have to do in practice but anyway, as I said, Py_BUILD_CORE works on Linux and MacOS. And it works also on Windows. Indeed dictobject.c is compiled. The only problem is in the linking phase, when the two objects should be linked in one library, _the_ library. It seems that on Windows it doesn't find python3.lib, even if I put it in the path. So I get the `unresolved external link` errors. -- https://mail.python.org/mailman/listinfo/python-list
Re: Unable to compile my C Extension on Windows: unresolved external link errors
On Sat, Nov 13, 2021 at 7:01 AM Marco Sulla wrote: > On Fri, 12 Nov 2021 at 17:38, Chris Angelico wrote: > > Are you sure that you really need Py_BUILD_CORE? > > Yes, because I need the internal functions of `dict`. So I need to > compile also dictobject.c and include it. So I need that flag. > > This is the code: > > https://github.com/Marco-Sulla/python-frozendict.git > Ah, gotcha. Unfortunately that does mean you're delving deep into internals, and a lot of stuff that isn't designed for extensions to use. So my best recommendation is: dig even deeper into internals, and duplicate how the core is doing things (maybe including another header or something). It may be that, by declaring Py_BUILD_CORE, you're getting a macro version of that instead of the normal exported function. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Unable to compile my C Extension on Windows: unresolved external link errors
Chris? Maybe I'm dreaming X-D On Fri, 12 Nov 2021 at 17:38, Chris Angelico wrote: > Are you sure that you really need Py_BUILD_CORE? Yes, because I need the internal functions of `dict`. So I need to compile also dictobject.c and include it. So I need that flag. This is the code: https://github.com/Marco-Sulla/python-frozendict.git On Linux and MacOS it works like a charme. On Windows, it seems it does not find python3.lib. I also added its path to the PATH variable -- https://mail.python.org/mailman/listinfo/python-list
Re: Unable to compile my C Extension on Windows: unresolved external link errors
On Sat, Nov 13, 2021 at 3:28 AM Marco Sulla wrote: > > On Fri, 12 Nov 2021 at 15:55, Gisle Vanem wrote: > > Marco Sulla wrote: > > > Error LNK2001: unresolved external symbol PyErr_SetObject > > > > > > and so on. > > > > > > I post the part of my setup.py about the C Extension: > > > > > > extra_compile_args = ["-DPY_SSIZE_T_CLEAN", "-DPy_BUILD_CORE"] > > > > Shouldn't this be "-DPy_BUILD_CORE_MODULE"? > > I tried it, but now I get three > > error C2099: initializer is not a constant > > when I try to compile dictobject.c. Yes, my extension needs dictobject. Are you sure that you really need Py_BUILD_CORE? Here's what the code has to say about it: https://github.com/python/cpython/blob/main/Include/pyport.h#L17 Py_BUILD_CORE: Build Python core. Give access to Python internals, but should not be used by third-party modules. If your extension truly needs to reach into the internals of CPython, then you're going to have to figure a lot of things out for yourself, including how to make sure it works on every platform. A more productive approach would be to stick to the actual API: https://docs.python.org/3/c-api/stable.html What is it that doesn't work without Py_BUILD_CORE? ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Unable to compile my C Extension on Windows: unresolved external link errors
On Fri, 12 Nov 2021 at 15:55, Gisle Vanem wrote: > Marco Sulla wrote: > > Error LNK2001: unresolved external symbol PyErr_SetObject > > > > and so on. > > > > I post the part of my setup.py about the C Extension: > > > > extra_compile_args = ["-DPY_SSIZE_T_CLEAN", "-DPy_BUILD_CORE"] > > Shouldn't this be "-DPy_BUILD_CORE_MODULE"? I tried it, but now I get three error C2099: initializer is not a constant when I try to compile dictobject.c. Yes, my extension needs dictobject. -- https://mail.python.org/mailman/listinfo/python-list
Re: Unable to compile my C Extension on Windows: unresolved external link errors
Marco Sulla wrote: Error LNK2001: unresolved external symbol PyErr_SetObject and so on. I post the part of my setup.py about the C Extension: extra_compile_args = ["-DPY_SSIZE_T_CLEAN", "-DPy_BUILD_CORE"] Shouldn't this be "-DPy_BUILD_CORE_MODULE"? -- --gv -- https://mail.python.org/mailman/listinfo/python-list
Unable to compile my C Extension on Windows: unresolved external link errors
I have no problem compiling my extension under Linux and MacOS. Under Windows, I get a lot of Error LNK2001: unresolved external symbol PyErr_SetObject and so on. I post the part of my setup.py about the C Extension: extra_compile_args = ["-DPY_SSIZE_T_CLEAN", "-DPy_BUILD_CORE"] undef_macros = [] setuptools.Extension( ext1_fullname, sources = cpython_sources, include_dirs = cpython_include_dirs, extra_compile_args = extra_compile_args, undef_macros = undef_macros, ) Here is the full code: https://github.com/Marco-Sulla/python-frozendict/blob/master/setup.py Steps to reproduce: I installed python3.10 and VS compiler on my Windows 10 machine, then I created a venv, activated it, run pip install -U pip setuptools wheel and then python setup.py bdist_wheel -- https://mail.python.org/mailman/listinfo/python-list