Re: [Rd] Problem with dyn.load'ed code
Andrew, Thanks! The version script worked like a charm. Specifically I now build using: g++ -shared -Wl,--version-script=ver.map to_dyn_load.cc -o to_dyn_load.so -larpack where ver.map is the file: { global: R_func_*; local:*; }; and any function I want exported to R is named R_func_*. This is going to be my new SOP. Thanks again Andrew, and also Simon. I greatly appreciate you taking your time to solve this problem for me. Matt On Mon, 2007-12-31 at 15:30 -0500, Andrew Piskorski wrote: On Sun, Dec 30, 2007 at 10:43:50PM -0500, Matt Calder wrote: Simon, Thanks for the reply. Indeed, declaring the function static fixes the example. Unfortunately, the real problem that gave rise to the example arises in a large Fortran library that is not under my control (ARPACK). The author is providing BLAS and LAPACK functionality intentionally. That may or may not be good practice, but it is a given in this case. Ok, so R is calling its one dnrm2_ function, let's call this A, while ARPACK defines a second, different dnrm2_, which we'll call B. You want to call function A from your own C code, while R keeps calling function A as before without any change or interference. And of course, A and B are two C-coded functions with different behaviors but the exact same name. You can make that work, it just requires some tricks. I still feel like the linker ought to be able to solve this problem for me. My impression was that the static keyword passed to the linker It can, you just need to tell it exactly what you want. I assume you are building your own custom C code into a shared library, which you then load into R. Thus, one solution is to statically link the ARPACK library into your own shared library, and then carefully tell the linker which symbols to export and which to keep private inside your shared library. As long as the symbol ARPACK's B dnrm2_ function is kept private inside your own shared library (not exported), R will never see it and will happily keep using dnrm2_ A as before. That's how I've solved this sort of name collision problem in the past. In your src/Makevars, you may want something like to this: PKG_LIBS = -Wl,--version-script=vis.map -Wl,-Bstatic -L/usr/local/lib/ARPACK -lARPACK -Wl,-Bdynamic You may also need a PG_PKG_LIBS with the same stuff, but I don't remember why. The '--version-script=' and related matters were also disccussed here back in February: https://stat.ethz.ch/pipermail/r-devel/2007-February/044531.html __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Problem with dyn.load'ed code
Matt, On Jan 1, 2008, at 12:08 PM, Matt Calder wrote: Andrew, Thanks! The version script worked like a charm. Specifically I now build using: g++ -shared -Wl,--version-script=ver.map to_dyn_load.cc -o to_dyn_load.so -larpack just a word of warning - this is in no way portable. It is probably ok for your private compilation, but it won't work in general. In any case, I'd strongly recommend enabling this hack only if you know that the target system supports it for the sake of portability. However, if you are concerned about the latter, you should probably have a look at libtool and --export-symbols - it allows you to control the visibility on most systems that support it. Note that there are systems that don't support it at all. Cheers, Simon where ver.map is the file: { global: R_func_*; local:*; }; and any function I want exported to R is named R_func_*. This is going to be my new SOP. Thanks again Andrew, and also Simon. I greatly appreciate you taking your time to solve this problem for me. Matt On Mon, 2007-12-31 at 15:30 -0500, Andrew Piskorski wrote: On Sun, Dec 30, 2007 at 10:43:50PM -0500, Matt Calder wrote: Simon, Thanks for the reply. Indeed, declaring the function static fixes the example. Unfortunately, the real problem that gave rise to the example arises in a large Fortran library that is not under my control (ARPACK). The author is providing BLAS and LAPACK functionality intentionally. That may or may not be good practice, but it is a given in this case. Ok, so R is calling its one dnrm2_ function, let's call this A, while ARPACK defines a second, different dnrm2_, which we'll call B. You want to call function A from your own C code, while R keeps calling function A as before without any change or interference. And of course, A and B are two C-coded functions with different behaviors but the exact same name. You can make that work, it just requires some tricks. I still feel like the linker ought to be able to solve this problem for me. My impression was that the static keyword passed to the linker It can, you just need to tell it exactly what you want. I assume you are building your own custom C code into a shared library, which you then load into R. Thus, one solution is to statically link the ARPACK library into your own shared library, and then carefully tell the linker which symbols to export and which to keep private inside your shared library. As long as the symbol ARPACK's B dnrm2_ function is kept private inside your own shared library (not exported), R will never see it and will happily keep using dnrm2_ A as before. That's how I've solved this sort of name collision problem in the past. In your src/Makevars, you may want something like to this: PKG_LIBS = -Wl,--version-script=vis.map -Wl,-Bstatic -L/usr/local/ lib/ARPACK -lARPACK -Wl,-Bdynamic You may also need a PG_PKG_LIBS with the same stuff, but I don't remember why. The '--version-script=' and related matters were also disccussed here back in February: https://stat.ethz.ch/pipermail/r-devel/2007-February/044531.html __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Problem with dyn.load'ed code
On Sun, Dec 30, 2007 at 10:43:50PM -0500, Matt Calder wrote: Simon, Thanks for the reply. Indeed, declaring the function static fixes the example. Unfortunately, the real problem that gave rise to the example arises in a large Fortran library that is not under my control (ARPACK). The author is providing BLAS and LAPACK functionality intentionally. That may or may not be good practice, but it is a given in this case. Ok, so R is calling its one dnrm2_ function, let's call this A, while ARPACK defines a second, different dnrm2_, which we'll call B. You want to call function A from your own C code, while R keeps calling function A as before without any change or interference. And of course, A and B are two C-coded functions with different behaviors but the exact same name. You can make that work, it just requires some tricks. I still feel like the linker ought to be able to solve this problem for me. My impression was that the static keyword passed to the linker It can, you just need to tell it exactly what you want. I assume you are building your own custom C code into a shared library, which you then load into R. Thus, one solution is to statically link the ARPACK library into your own shared library, and then carefully tell the linker which symbols to export and which to keep private inside your shared library. As long as the symbol ARPACK's B dnrm2_ function is kept private inside your own shared library (not exported), R will never see it and will happily keep using dnrm2_ A as before. That's how I've solved this sort of name collision problem in the past. In your src/Makevars, you may want something like to this: PKG_LIBS = -Wl,--version-script=vis.map -Wl,-Bstatic -L/usr/local/lib/ARPACK -lARPACK -Wl,-Bdynamic You may also need a PG_PKG_LIBS with the same stuff, but I don't remember why. The '--version-script=' and related matters were also disccussed here back in February: https://stat.ethz.ch/pipermail/r-devel/2007-February/044531.html -- Andrew Piskorski [EMAIL PROTECTED] http://www.piskorski.com/ __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Problem with dyn.load'ed code
Matt, On Dec 30, 2007, at 5:25 PM, Matt Calder wrote: I am still having trouble dyn.load'ing some code into R. I have isolated the problem, I wonder if someone could explain what I am seeing. I think the problem is that a symbol defined in my compiled code clashes with one already defined in R. The result of redefining a symbol is very much system-dependent. For example on Mac OS X it has no adverse effects (i.e your symbols are always private unless linked against), because it uses a two-level namespace, but on Linux (and most other unices) it does as the namespace of an executable is shared among all modules (dynamic and static). The result is that the function in my code is not called. Here is an example // lnktst.cc extern C { void func(double *out1, double *out2); void dnrm2_(double *out); void dnrm3_(double *out); } void func(double *out1, double *out2) { dnrm2_(out1); dnrm3_(out2); } void dnrm2_(double *out) { *out = 1234.5; } void dnrm3_(double *out) { *out = 6789.0; } // End of lnktst.cc When I compile: g++ -shared -static -o lnktst.so lnktst.cc and then in R I call func dyn.load(lnktst.so) .C('func', double(1), double(1)) [[1]] [1] 0 [[2]] [1] 6789 So, as you can see, the function dnrm2_ is not called whereas dnrm3_ is, even though both functions are identical in form. Now, I believe dnrm2_ is a BLAS function, and so it is likely R already has a copy floating around. Yes, indeed (it's a BLAS level 1 Fortran function, and usually to be found in libRblas.so or the external BLAS implementation). However, it surprises me that the -static option does not force the call in my code to dnrm2_ to be linked to the function defined in my code. You are confusing the purpose of -static: it only ensures that static libraries are used at link time where possible, it doesn't affect your code in any way. What you really want is to use static void dnrm2_(double *out); in your code instead. In general, it is a bad idea to use external symbols that clash with other libraries (in particular widespread ones such as BLAS), especially if your function doesn't perform the same operation. It is a good idea to declare all functions that you use internally (i.e. that should not be visible to R) as static. However, all this is true for C programming in general, not just in conjunction with R. Cheers, Simon I have been writing C code for Splus for quite a while and don't recall ever running across this issue. However, I am new to R, so I wonder, am I missing something obvious? I am running this on Ubuntu Linux, the output of uname -a is: Linux calder-linux 2.6.22-14-generic #1 SMP Sun Oct 14 23:05:12 GMT 2007 i686 GNU/Linux Thanks for any help, Matt __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Problem with dyn.load'ed code
Simon, Thanks for the reply. Indeed, declaring the function static fixes the example. Unfortunately, the real problem that gave rise to the example arises in a large Fortran library that is not under my control (ARPACK). The author is providing BLAS and LAPACK functionality intentionally. That may or may not be good practice, but it is a given in this case. So, I have a set of Fortran code in which some BLAS functionality is replicated. I am writing an interface to some of the functions in that code (not to the BLAS part, that is used internally by ARPACK). I would like it if that interface did not require alteration of the library source code or build process (though it is open-source, so if need be I can change it). I still feel like the linker ought to be able to solve this problem for me. My impression was that the static keyword passed to the linker caused it to resolve all references at link time. So something like: ld -o my_code_and_arpack.o -static my_code.o -larpack would pull all the references from the two object files (my_code.o and libarpack.a) and link them as needed, and unresolved references would cause an error. I guess that impression is wrong, but how does one accomplish the same thing? Thanks for any help, Matt On Sun, 2007-12-30 at 20:21 -0500, Simon Urbanek wrote: Matt, On Dec 30, 2007, at 5:25 PM, Matt Calder wrote: I am still having trouble dyn.load'ing some code into R. I have isolated the problem, I wonder if someone could explain what I am seeing. I think the problem is that a symbol defined in my compiled code clashes with one already defined in R. The result of redefining a symbol is very much system-dependent. For example on Mac OS X it has no adverse effects (i.e your symbols are always private unless linked against), because it uses a two-level namespace, but on Linux (and most other unices) it does as the namespace of an executable is shared among all modules (dynamic and static). The result is that the function in my code is not called. Here is an example // lnktst.cc extern C { void func(double *out1, double *out2); void dnrm2_(double *out); void dnrm3_(double *out); } void func(double *out1, double *out2) { dnrm2_(out1); dnrm3_(out2); } void dnrm2_(double *out) { *out = 1234.5; } void dnrm3_(double *out) { *out = 6789.0; } // End of lnktst.cc When I compile: g++ -shared -static -o lnktst.so lnktst.cc and then in R I call func dyn.load(lnktst.so) .C('func', double(1), double(1)) [[1]] [1] 0 [[2]] [1] 6789 So, as you can see, the function dnrm2_ is not called whereas dnrm3_ is, even though both functions are identical in form. Now, I believe dnrm2_ is a BLAS function, and so it is likely R already has a copy floating around. Yes, indeed (it's a BLAS level 1 Fortran function, and usually to be found in libRblas.so or the external BLAS implementation). However, it surprises me that the -static option does not force the call in my code to dnrm2_ to be linked to the function defined in my code. You are confusing the purpose of -static: it only ensures that static libraries are used at link time where possible, it doesn't affect your code in any way. What you really want is to use static void dnrm2_(double *out); in your code instead. In general, it is a bad idea to use external symbols that clash with other libraries (in particular widespread ones such as BLAS), especially if your function doesn't perform the same operation. It is a good idea to declare all functions that you use internally (i.e. that should not be visible to R) as static. However, all this is true for C programming in general, not just in conjunction with R. Cheers, Simon I have been writing C code for Splus for quite a while and don't recall ever running across this issue. However, I am new to R, so I wonder, am I missing something obvious? I am running this on Ubuntu Linux, the output of uname -a is: Linux calder-linux 2.6.22-14-generic #1 SMP Sun Oct 14 23:05:12 GMT 2007 i686 GNU/Linux Thanks for any help, Matt __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Problem with dyn.load'ed code
You are missing some extern C declarations? R needs C linkage style. Basically a lot of: extern C { } around your c++ code. Matt Calder wrote: Hi, I am having trouble with some code that I am dyn.loading. I am writing an interface to ARPACK. I compile my interface (dssimp.cc), and link it against the ARPACK library (libarpack_SUN4.a): g++ -shared -static -fPIC dssimp.cc -o dssimp.so -larpack_SUN4 -lg2c -lm I can dyn.load the code and it appears OK. However, when I call my function, the call to the function in the ARPACK library returns an error. I can print the arguments to my function when I call it from R, so I can see that I am dyn.load'd, and the arguments are passed correctly. Moreover, I can load and call the shared object (dssimp.so) using dlopen and dlsym from a stand alone C++ program. So I can see that the shared object is OK. In summary, I have a shared object that works correctly outside of R, but when dyn.load'd into R, does not work. Short of calling errors (which is not the case here) I wonder how that might happen, and how I might fix it? Matt __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel