Re: [R-pkg-devel] Package Load fails to find 3rd Party DLL
Not sure if I reported the success here or not. Copying the 3rd party DLL, but not the .lib file, to the src directory does work around the bug in the linker. The complete working solution can be seen at https://github.com/ralmond/RNetica. Thanks for the help. --Russell On 7/17/23 5:30 AM, Ivan Krylov wrote: В Fri, 14 Jul 2023 22:25:51 +0300 Ivan Krylov пишет: Maybe it's a red herring. Maybe the message from nm about missing file has always been harmless, and what we're seeing here is a bug in the toolchain; perhaps ld.exe doesn't like something about Netica.lib so much that it crashes. I think that's less likely. If you run the commands manually but without mentioning NeticaDLL, do you get a DLL in the end? Judging by your build logs, this could be a toolchain bug. If you set the *.lib file aside and only give the *.dll to the linker (using -l:Netica.dll if necessary), does it still fail? I know that GCC can link directly to *.dll files, without relying on import libraries. __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] Package Load fails to find 3rd Party DLL
Okay, I've changed my Makevars.win so that it has:
PKG_LIBS = -L. -L${NETICA_LIB} -lNetica
and
all: NeticaDLL
where ${NETICA_LIB} is set to the appropriate lib subdirectory of the
unpacked sources. I'm no longer getting the nm.exe error, so it may be
a bug in ld.exe.
| gcc -shared -s -static-libgcc -o RNetica.dll tmp.def Cases.o
Continuous.o Edges.o Experience.o Inference.o Networks.o Node.o Random.o
Registration.o Session.o -L.
-L/c/Users/ralmond/Projects/RNetica/src/Netica/Netica_API_510/lib64
-lNetica -LC:/rtools43/x86_64-w64-mingw32.static.posix/lib/x64
-LC:/rtools43/x86_64-w64-mingw32.static.posix/lib
-LC:/PROGRA~1/R/R-43~1.1/bin/x64 -lR
| C:\rtools43\x86_64-w64-mingw32.static.posix\bin/ld.exe: internal
error: aborting at ../../binutils-2.40/ld/ldlang.c:527 in compare_section
| C:\rtools43\x86_64-w64-mingw32.static.posix\bin/ld.exe: please report
this bug
| collect2.exe: error: ld returned 1 exit status
However, when I manually moved Netica.dll to the source directory, and
removed NETICA_LIB from the path, it compiled correctly.
So, I'm guessing this is a bug in ld.exe, and in fact it is here:
https://github.com/msys2/MINGW-packages/issues/15469
So I'll see if I can try to fix my script to only copy the .dll and not
the .lib.
--Russell
On 7/17/23 5:30 AM, Ivan Krylov wrote:
В Fri, 14 Jul 2023 22:25:51 +0300
Ivan Krylov пишет:
Maybe it's a red herring. Maybe the message from nm about missing file
has always been harmless, and what we're seeing here is a bug in the
toolchain; perhaps ld.exe doesn't like something about Netica.lib so
much that it crashes. I think that's less likely. If you run the
commands manually but without mentioning NeticaDLL, do you get a DLL
in the end?
Judging by your build logs, this could be a toolchain bug. If you set
the *.lib file aside and only give the *.dll to the linker (using
-l:Netica.dll if necessary), does it still fail? I know that GCC can
link directly to *.dll files, without relying on import libraries.
__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] Package Load fails to find 3rd Party DLL
Thanks for the suggestion. In previous versions of the build tools, I know I needed the .lib files. I'm also not sure I'm copying the DLL files into the right directory, so that maybe the linker isn't seeing it. This always confuses me as the location used to build and compile is (potentially) different from the test location and the final build location, and I'm not sure of the best way to refer to these directories in my script. I'm thinking maybe I need to copy the DLL into the src directory. Unfortunately, I don't have a windows box on which to easily test. I'm trying to get a virtualBox setup working so I can test more quickly. --Russell On 7/17/23 5:30 AM, Ivan Krylov wrote: В Fri, 14 Jul 2023 22:25:51 +0300 Ivan Krylov пишет: Maybe it's a red herring. Maybe the message from nm about missing file has always been harmless, and what we're seeing here is a bug in the toolchain; perhaps ld.exe doesn't like something about Netica.lib so much that it crashes. I think that's less likely. If you run the commands manually but without mentioning NeticaDLL, do you get a DLL in the end? Judging by your build logs, this could be a toolchain bug. If you set the *.lib file aside and only give the *.dll to the linker (using -l:Netica.dll if necessary), does it still fail? I know that GCC can link directly to *.dll files, without relying on import libraries. __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] Package Load fails to find 3rd Party DLL
В Fri, 14 Jul 2023 22:25:51 +0300 Ivan Krylov пишет: > Maybe it's a red herring. Maybe the message from nm about missing file > has always been harmless, and what we're seeing here is a bug in the > toolchain; perhaps ld.exe doesn't like something about Netica.lib so > much that it crashes. I think that's less likely. If you run the > commands manually but without mentioning NeticaDLL, do you get a DLL > in the end? Judging by your build logs, this could be a toolchain bug. If you set the *.lib file aside and only give the *.dll to the linker (using -l:Netica.dll if necessary), does it still fail? I know that GCC can link directly to *.dll files, without relying on import libraries. -- Best regards, Ivan __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] Package Load fails to find 3rd Party DLL
On Fri, 14 Jul 2023 13:29:32 -0400
Russell Almond wrote:
> So I'm confused. Why is the Makevars -> Makefile conversion assuming
> that all targets of $(SHLIB) (or all) are executable files, ignoring
> the PHONY declaration?
The $(SHLIB) target is defined in ${R_HOME}/share/make/winshlib.mk. In
particular, it says:
echo EXPORTS > tmp.def; \
$(NM) $^ | $(SED) -n $(SYMPAT) $(NM_FILTER) | \
$(SED) $(ADDQU) >> tmp.def; \
Unfortunately for our case, the $^ automatic variable contains the
names of all prerequisites of the $(SHLIB) target [*], including .PHONY
ones. So as long as $(SHLIB) is declared to depend on NeticaDLL, this
string will get mentioned here.
Maybe it's a red herring. Maybe the message from nm about missing file
has always been harmless, and what we're seeing here is a bug in the
toolchain; perhaps ld.exe doesn't like something about Netica.lib so
much that it crashes. I think that's less likely. If you run the
commands manually but without mentioning NeticaDLL, do you get a DLL in
the end?
> Also, does the example in the Writing R Extensions manual still work?
It should. At the very least, rgl uses the same approach without any
apparent problems:
https://github.com/dmurdoch/rgl/blob/master/src/Makevars.ucrt
(But then it also provides rgl-win.def, which sidesteps the automatic
.def generation, and then has $(OBJECTS), not $^, linked into $(SHLIB).
You may have found a bug in winshlib.mk.)
There's also a different idiom, where all: but not $(SHLIB): depends on
the non-file target:
https://github.com/eddelbuettel/rprotobuf/blob/master/src/Makevars.win
(Not sure how the dependency resolution order is supposed to work in
this case. If winshlib.mk declares that all: depends on $(SHLIB) and
later Makevars declares that all: also depends on winlibs, why doesn't
Make attempt to build $(SHLIB) first, with no regard for winlibs?)
--
Best regards,
Ivan
[*]
https://www.gnu.org/software/make/manual/html_node/Automatic-Variables.html
__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] Package Load fails to find 3rd Party DLL
Thanks. I know know the problem is in the Makevars.win; however, I'm
still confused.
My `Makevars.win` had
| .PHONY: all NeticaDLL clean
| all: $(SHLIB)
| $(SHLIB): NeticaDLL
|
| NeticaDLL:
| mkdir -p "$(INSTALL_LIB)"
| cp "${NETICA_LIB}/Netica.dll" "${INSTALL_LIB}"
| cp "${NETICA_LIB}/Netica.lib" "${INSTALL_LIB}"
[BTW, I tried change this to
| all: NeticaDLL $(SHLIB)
and got the same problem.]
This looks very much like the `mylibs` example in
https://cran.r-project.org/doc/manuals/r-release/R-exts.html#Using-Makevars
So I'm confused. Why is the Makevars -> Makefile conversion assuming
that all targets of $(SHLIB) (or all) are executable files, ignoring the
PHONY declaration?
Also, does the example in the Writing R Extensions manual still work?
--Russell
On 7/14/23 11:14 AM, Ivan Krylov wrote:
> В Wed, 12 Jul 2023 09:41:11 -0400
> Russell Almond пишет:
>
>> C:\rtools43\x86_64-w64-mingw32.static.posix\bin\nm.exe:
>> 'NeticaDLL': No such file
> This is where the problem starts. You can retrace the steps that R
> takes when building and installing the package by running sh
> configure.win manually and then running something like
> R_PACKAGE_DIR="$(pwd)" R CMD SHLIB -n *.c in the src subdirectory of
> the package. That will in turn tell you the exact command lines to be
> run while building your package, including the following:
>
> nm Cases.o Continuous.o Edges.o Experience.o Inference.o Networks.o \
>Node.o Random.o Registration.o Session.o NeticaDLL \
> | sed -n 's/^.* [BCDRT] / /p' \
> | sed -e '/[.]refptr[.]/d' -e '/[.]weak[.]/d' \
> | sed 's/[^ ][^ ]*/"&"/g' \
> >> tmp.def;
>
> That "NeticaDLL" at the end of the list of object files doesn't belong
> there. I think it gets added because it's among the dependencies of the
> $(SHLIB) Make target. It would be best to make that target a real
> object file that nm.exe can process. Otherwise, you could also write
> your own .def file and skip its automatic generation.
>
> After nm fails, you get a crash in the linker (while parsing the
> resulting incomplete .def file?), which leaves your package without a
> shared library to use:
>
>> C:\rtools43\x86_64-w64-mingw32.static.posix\bin/ld.exe: internal
>> error: aborting at ../../binutils-2.40/ld/ldlang.c:527 in
>> compare_section
>> C:\rtools43\x86_64-w64-mingw32.static.posix\bin/ld.exe: please report
>> this bug
> There must be a way to streamline this process. Maybe put all the
> library-downloading and extraction code into a portable
> tools/configure.R (to be launched manually from the configure shell
> script), leaving src/Makevars only to compile your own code, link with
> Netica using PKG_LIBS, then copy the additional Netica DLL from a
> custom install.libs.R file?
>
--
Russell Almond
https://ralmond.net/
[[alternative HTML version deleted]]
__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] Package Load fails to find 3rd Party DLL
В Wed, 12 Jul 2023 09:41:11 -0400 Russell Almond пишет: > C:\rtools43\x86_64-w64-mingw32.static.posix\bin\nm.exe: > 'NeticaDLL': No such file This is where the problem starts. You can retrace the steps that R takes when building and installing the package by running sh configure.win manually and then running something like R_PACKAGE_DIR="$(pwd)" R CMD SHLIB -n *.c in the src subdirectory of the package. That will in turn tell you the exact command lines to be run while building your package, including the following: nm Cases.o Continuous.o Edges.o Experience.o Inference.o Networks.o \ Node.o Random.o Registration.o Session.o NeticaDLL \ | sed -n 's/^.* [BCDRT] / /p' \ | sed -e '/[.]refptr[.]/d' -e '/[.]weak[.]/d' \ | sed 's/[^ ][^ ]*/"&"/g' \ >> tmp.def; That "NeticaDLL" at the end of the list of object files doesn't belong there. I think it gets added because it's among the dependencies of the $(SHLIB) Make target. It would be best to make that target a real object file that nm.exe can process. Otherwise, you could also write your own .def file and skip its automatic generation. After nm fails, you get a crash in the linker (while parsing the resulting incomplete .def file?), which leaves your package without a shared library to use: > C:\rtools43\x86_64-w64-mingw32.static.posix\bin/ld.exe: internal > error: aborting at ../../binutils-2.40/ld/ldlang.c:527 in > compare_section > C:\rtools43\x86_64-w64-mingw32.static.posix\bin/ld.exe: please report > this bug There must be a way to streamline this process. Maybe put all the library-downloading and extraction code into a portable tools/configure.R (to be launched manually from the configure shell script), leaving src/Makevars only to compile your own code, link with Netica using PKG_LIBS, then copy the additional Netica DLL from a custom install.libs.R file? -- Best regards, Ivan __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] Package Load fails to find 3rd Party DLL
Use of precompiled code is not allowed in CRAN. This looks like your package needs to be distributed elsewhere... e.g. via GitHub. On July 12, 2023 6:41:11 AM PDT, Russell Almond wrote: >I have an R package (RNetica available at >https://ralmond.r-universe.dev/RNetica and >https://github.com/ralmond/RNetica) which links to a 3rd party library >Netica.dll, so RNetica.dll (built from my C code) calls the 3rd party code. > >The config.win script downloads Netica.dll and moves it into the >libs/x64 directory, where it should get loaded when RNetica.dll is >loaded. However this is not happening: > >Here is the relevant portion of the build log (build is on R-universe, >but I think it is the same script as CRAN): > >``` > >cp >"/d/a/ralmond/ralmond/RNETIC~1.RCH/00_PKG~1/RNetica/src/Netica/Netica_API_5 >10/lib64/Netica.dll" >"D:/a/ralmond/ralmond/RNetica.Rcheck/00LOCK-RNetica/00new/R >Netica/libs/x64" > cp >"/d/a/ralmond/ralmond/RNETIC~1.RCH/00_PKG~1/RNetica/src/Netica/Netica_API_5 >10/lib64/Netica.lib" >"D:/a/ralmond/ralmond/RNetica.Rcheck/00LOCK-RNetica/00new/R >Netica/libs/x64" > C:\rtools43\x86_64-w64-mingw32.static.posix\bin\nm.exe: 'NeticaDLL': >No such f >ile > gcc -shared -s -static-libgcc -o RNetica.dll tmp.def Cases.o >Continuous.o Edge >s.o Experience.o Inference.o Networks.o Node.o Random.o Registration.o >Session.o > -L. >-LD:/a/ralmond/ralmond/RNetica.Rcheck/00LOCK-RNetica/00new/RNetica/libs/x64 > -lNetica -LC:/rtools43/x86_64-w64-mingw32.static.posix/lib/x64 >-LC:/rtools43/x8 >6_64-w64-mingw32.static.posix/lib -LC:/R/bin/x64 -lR > C:\rtools43\x86_64-w64-mingw32.static.posix\bin/ld.exe: internal >error: aborti >ng at ../../binutils-2.40/ld/ldlang.c:527 in compare_section > C:\rtools43\x86_64-w64-mingw32.static.posix\bin/ld.exe: please report >this bug > collect2.exe: error: ld returned 1 exit status >``` > >A little bit of searching on the internet, indicates that Windows >sometimes reports Dll A not found when Dll A needs Dll B and it can't >find B. > >This used to work under older versions of R and the tool chain and I >don't think I've changed anything related to the C side of the code. > >1) Have the paths changed, so I no longer should be moving the (64 bit >version of the) 3rd party DLL to `libs/x64`? > >2) Is there something that has changed with the mingw tools (nm.exe and >ld.exe) which are changing things? > >3) Is there a change on how win32 and win64 variants are handled (I have >both 32 and 64 bit copies of the 3rd party DLL, I just need to move them >to the right places). > >Thanks for any enlightenment you can offer, > > --Russell Almond > > > > -- Sent from my phone. Please excuse my brevity. __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
[R-pkg-devel] Package Load fails to find 3rd Party DLL
I have an R package (RNetica available at https://ralmond.r-universe.dev/RNetica and https://github.com/ralmond/RNetica) which links to a 3rd party library Netica.dll, so RNetica.dll (built from my C code) calls the 3rd party code. The config.win script downloads Netica.dll and moves it into the libs/x64 directory, where it should get loaded when RNetica.dll is loaded. However this is not happening: Here is the relevant portion of the build log (build is on R-universe, but I think it is the same script as CRAN): ``` cp "/d/a/ralmond/ralmond/RNETIC~1.RCH/00_PKG~1/RNetica/src/Netica/Netica_API_5 10/lib64/Netica.dll" "D:/a/ralmond/ralmond/RNetica.Rcheck/00LOCK-RNetica/00new/R Netica/libs/x64" cp "/d/a/ralmond/ralmond/RNETIC~1.RCH/00_PKG~1/RNetica/src/Netica/Netica_API_5 10/lib64/Netica.lib" "D:/a/ralmond/ralmond/RNetica.Rcheck/00LOCK-RNetica/00new/R Netica/libs/x64" C:\rtools43\x86_64-w64-mingw32.static.posix\bin\nm.exe: 'NeticaDLL': No such f ile gcc -shared -s -static-libgcc -o RNetica.dll tmp.def Cases.o Continuous.o Edge s.o Experience.o Inference.o Networks.o Node.o Random.o Registration.o Session.o -L. -LD:/a/ralmond/ralmond/RNetica.Rcheck/00LOCK-RNetica/00new/RNetica/libs/x64 -lNetica -LC:/rtools43/x86_64-w64-mingw32.static.posix/lib/x64 -LC:/rtools43/x8 6_64-w64-mingw32.static.posix/lib -LC:/R/bin/x64 -lR C:\rtools43\x86_64-w64-mingw32.static.posix\bin/ld.exe: internal error: aborti ng at ../../binutils-2.40/ld/ldlang.c:527 in compare_section C:\rtools43\x86_64-w64-mingw32.static.posix\bin/ld.exe: please report this bug collect2.exe: error: ld returned 1 exit status ``` A little bit of searching on the internet, indicates that Windows sometimes reports Dll A not found when Dll A needs Dll B and it can't find B. This used to work under older versions of R and the tool chain and I don't think I've changed anything related to the C side of the code. 1) Have the paths changed, so I no longer should be moving the (64 bit version of the) 3rd party DLL to `libs/x64`? 2) Is there something that has changed with the mingw tools (nm.exe and ld.exe) which are changing things? 3) Is there a change on how win32 and win64 variants are handled (I have both 32 and 64 bit copies of the 3rd party DLL, I just need to move them to the right places). Thanks for any enlightenment you can offer, --Russell Almond -- Russell Almond https://ralmond.net/ [[alternative HTML version deleted]] __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
