Re: [Rcpp-devel] RcppParallel on Solaris
Le 13 avr. 2015 à 14:03, JJ Allaire jj.alla...@gmail.com a écrit : That's excellent I just updated the branch to reflect this change and also successfully ran the tests on the Solaris config that you provided me access to. I'll have to take a closer look at the warnings. One other issue that needs to be resolved prior to the next submission to CRAN revolves around pedantic warnings on Debian testing that prohibit long long (used by both TinyThread and TBB). The easy workaround is SystemRequirements: C++11 however this will mean that package won't compile on pre-Mavericks Macs (~30% of all Macs) nor RedHat/CentOS systems. Perhaps I can modify TinyThread and TBB to no longer use long long but I'll need to do this very carefully. FWIW tbb with C++11, in particular lambdas is awesome On Sun, Apr 12, 2015 at 10:30 PM, Gábor Csárdi csardi.ga...@gmail.com wrote: Hi JJ all, I had some time, had an idea, and made it to work. The problem was that TBB was not compiled with the same flags as R and the rest of the R package, so just had to find what is the difference that is incompatible. It is simple, you need to compile TBB with -library=stlport4 So if you change line 34 in SunOS.suncc.inc to CPLUS = CC -library=stlport4 then it compiles and installs fine. What's even better, the tests run fine, too. They spit out a lot of compiler warnings, but they all pass. Best, Gabor On Wed, Apr 8, 2015 at 11:38 AM, Gábor Csárdi csardi.ga...@gmail.com wrote: Ok, the server seems to work. JJ, I'll send you a private email. If anyone wants access, please email me in private. Remember that this is just a mac mini, so it might not be super fast. It seems fast enough for a single user, though. Gabor On Tue, Apr 7, 2015 at 7:17 PM, Gábor Csárdi csardi.ga...@gmail.com wrote: On Tue, Apr 7, 2015 at 6:50 PM, Jeroen Ooms jeroeno...@gmail.com wrote: [...] So that's why I thought they probably use GCC for packages that don't work with Solaris Studio. I see. That would indeed make sense. G. ___ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel ___ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
Re: [Rcpp-devel] RcppParallel on Solaris
One other issue to run down is detecting the current architecture on Solaris builds. Right now we force 32-bit in Makevars as follows: ifeq ($(USE_TBB), SunOS) MAKE_ARGS += arch=ia32 endif Does anyone know if there is a variable we could inspect from within Makevars that would tell us the architecture for R itself? (as opposed to the underlying system). On Mon, Apr 13, 2015 at 8:03 AM, JJ Allaire jj.alla...@gmail.com wrote: That's excellent I just updated the branch to reflect this change and also successfully ran the tests on the Solaris config that you provided me access to. I'll have to take a closer look at the warnings. One other issue that needs to be resolved prior to the next submission to CRAN revolves around pedantic warnings on Debian testing that prohibit long long (used by both TinyThread and TBB). The easy workaround is SystemRequirements: C++11 however this will mean that package won't compile on pre-Mavericks Macs (~30% of all Macs) nor RedHat/CentOS systems. Perhaps I can modify TinyThread and TBB to no longer use long long but I'll need to do this very carefully. On Sun, Apr 12, 2015 at 10:30 PM, Gábor Csárdi csardi.ga...@gmail.com wrote: Hi JJ all, I had some time, had an idea, and made it to work. The problem was that TBB was not compiled with the same flags as R and the rest of the R package, so just had to find what is the difference that is incompatible. It is simple, you need to compile TBB with -library=stlport4 So if you change line 34 in SunOS.suncc.inc to CPLUS = CC -library=stlport4 then it compiles and installs fine. What's even better, the tests run fine, too. They spit out a lot of compiler warnings, but they all pass. Best, Gabor On Wed, Apr 8, 2015 at 11:38 AM, Gábor Csárdi csardi.ga...@gmail.com wrote: Ok, the server seems to work. JJ, I'll send you a private email. If anyone wants access, please email me in private. Remember that this is just a mac mini, so it might not be super fast. It seems fast enough for a single user, though. Gabor On Tue, Apr 7, 2015 at 7:17 PM, Gábor Csárdi csardi.ga...@gmail.com wrote: On Tue, Apr 7, 2015 at 6:50 PM, Jeroen Ooms jeroeno...@gmail.com wrote: [...] So that's why I thought they probably use GCC for packages that don't work with Solaris Studio. I see. That would indeed make sense. G. ___ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
Re: [Rcpp-devel] RcppParallel on Solaris
We only see the long long warnings with clang on Debian testing with -pedantic enabled. Note that not all CRAN maintainers test under this configuration so sometimes it depends on who is reviewing the submission. My first preference would be to ensure that these warnings don't occur so we always pass muster. However as you point out it can be fragile and risky to modify upstream code so perhaps asking for an exception is the right course. On Mon, Apr 13, 2015 at 9:42 AM, Gábor Csárdi csardi.ga...@gmail.com wrote: On Mon, Apr 13, 2015 at 8:32 AM, Dirk Eddelbuettel e...@debian.org wrote: On 13 April 2015 at 08:03, JJ Allaire wrote: | I'll have to take a closer look at the warnings. One other issue that | needs to be resolved prior to the next submission to CRAN revolves | around pedantic warnings on Debian testing that prohibit long long | (used by both TinyThread and TBB). The easy workaround is | SystemRequirements: C++11 however this will mean that package won't | compile on pre-Mavericks Macs (~30% of all Macs) nor RedHat/CentOS | systems. Perhaps I can modify TinyThread and TBB to no longer use | long long but I'll need to do this very carefully. I never found another way to get 'long long' besides requiring C++11. It might be fine, actually. I have a package with long long on CRAN, and yes, they asked me to fix it, but like for you, it is in upstream code, so I am not really comfortable removing it, and CRAN seems to be OK with it now. Other packages seem to have it, too: https://github.com/search?q=user%3Acran+%22long+long%22type=Codeutf8=%E2%9C%93 I doubt that these all require C++11. Also, all compilers they use have long long, so what is the point, really? Gabor [...] ___ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
Re: [Rcpp-devel] RcppParallel on Solaris
On 13 April 2015 at 09:42, Gábor Csárdi wrote: | On Mon, Apr 13, 2015 at 8:32 AM, Dirk Eddelbuettel e...@debian.org wrote: | | | On 13 April 2015 at 08:03, JJ Allaire wrote: | | I'll have to take a closer look at the warnings. One other issue that | | needs to be resolved prior to the next submission to CRAN revolves | | around pedantic warnings on Debian testing that prohibit long long | | (used by both TinyThread and TBB). The easy workaround is | | SystemRequirements: C++11 however this will mean that package won't | | compile on pre-Mavericks Macs (~30% of all Macs) nor RedHat/CentOS | | systems. Perhaps I can modify TinyThread and TBB to no longer use | | long long but I'll need to do this very carefully. | | I never found another way to get 'long long' besides requiring C++11. | | | It might be fine, actually. I have a package with long long on CRAN, and yes, | they asked me to fix it, but like for you, it is in upstream code, so I am | not really comfortable removing it, and CRAN seems to be OK with it now. | | Other packages seem to have it, too: https://github.com/search?q= | user%3Acran+%22long+long%22type=Codeutf8=%E2%9C%93 | I doubt that these all require C++11. | | Also, all compilers they use have long long, so what is the point, really? It has been discussed on this list a million times. Someone define C99 to be ok (has long long) yet insists on C++98 (does not). End of story because C++03 does not exist in that parallel universe but switching to C++11 helps. Dirk -- http://dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org ___ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
Re: [Rcpp-devel] RcppParallel on Solaris
On Mon, Apr 13, 2015 at 8:11 AM, JJ Allaire jj.alla...@gmail.com wrote: One other issue to run down is detecting the current architecture on Solaris builds. Right now we force 32-bit in Makevars as follows: ifeq ($(USE_TBB), SunOS) MAKE_ARGS += arch=ia32 endif Does anyone know if there is a variable we could inspect from within Makevars that would tell us the architecture for R itself? (as opposed to the underlying system). I think the way to go is just to strip the arch flags from the TBB config, like here: https://github.com/gaborcsardi/RcppParallel/commit/8d9b818042390ded7d00f9ef3cc4d5f9e4096b53 and then make sure that TBB is compiled with the appropriate compilers and flags, taken from R CMD config CXX R CMD config CXXFLAGS ... etc. On Solaris R is 32 bit by default, and if it is not, the appropriate arch flags will be reported by R CMD config. This solutions also has the advantage that if CRAN's config changes, you likely don't need to update anything. If you still want to detect the arch, then AFAIK it is in .Platform$r_arch. This is empty on the 32 bit Solaris, I guess because this is the default arch on this platform. Maybe it is set something else on 64 bit Solaris, but I am not sure. To decide between 32 bit and 64 bit I think you can use .Machine$sizeof.pointer. G. On Mon, Apr 13, 2015 at 8:03 AM, JJ Allaire jj.alla...@gmail.com wrote: That's excellent I just updated the branch to reflect this change and also successfully ran the tests on the Solaris config that you provided me access to. I'll have to take a closer look at the warnings. One other issue that needs to be resolved prior to the next submission to CRAN revolves around pedantic warnings on Debian testing that prohibit long long (used by both TinyThread and TBB). The easy workaround is SystemRequirements: C++11 however this will mean that package won't compile on pre-Mavericks Macs (~30% of all Macs) nor RedHat/CentOS systems. Perhaps I can modify TinyThread and TBB to no longer use long long but I'll need to do this very carefully. On Sun, Apr 12, 2015 at 10:30 PM, Gábor Csárdi csardi.ga...@gmail.com wrote: Hi JJ all, I had some time, had an idea, and made it to work. The problem was that TBB was not compiled with the same flags as R and the rest of the R package, so just had to find what is the difference that is incompatible. It is simple, you need to compile TBB with -library=stlport4 So if you change line 34 in SunOS.suncc.inc to CPLUS = CC -library=stlport4 then it compiles and installs fine. What's even better, the tests run fine, too. They spit out a lot of compiler warnings, but they all pass. Best, Gabor On Wed, Apr 8, 2015 at 11:38 AM, Gábor Csárdi csardi.ga...@gmail.com wrote: Ok, the server seems to work. JJ, I'll send you a private email. If anyone wants access, please email me in private. Remember that this is just a mac mini, so it might not be super fast. It seems fast enough for a single user, though. Gabor On Tue, Apr 7, 2015 at 7:17 PM, Gábor Csárdi csardi.ga...@gmail.com wrote: On Tue, Apr 7, 2015 at 6:50 PM, Jeroen Ooms jeroeno...@gmail.com wrote: [...] So that's why I thought they probably use GCC for packages that don't work with Solaris Studio. I see. That would indeed make sense. G. ___ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
Re: [Rcpp-devel] RcppParallel on Solaris
On 13 April 2015 at 10:26, JJ Allaire wrote: | We only see the long long warnings with clang on Debian testing with | -pedantic enabled. Weird. I don't think I encountered such a difference before. | Note that not all CRAN maintainers test under this | configuration so sometimes it depends on who is reviewing the | submission. AFAIK it is an OR -- any one failure pushes you back to start. Dirk -- http://dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org ___ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
Re: [Rcpp-devel] RcppParallel on Solaris
I think this is just a matter of passing the right runtime library flag to the TBB configure script (but perhaps this masks a bigger issue, we'll see). Gábor is going to make his VM available to me for further testing and iteration later this week. Hope to be able to report some progress after that. On Mon, Apr 6, 2015 at 9:41 PM, Gábor Csárdi csardi.ga...@gmail.com wrote: Yeah, you need GNU make. We had some progress off-list. This is where we are now: Btw. if you remove -m64, then the 32 bit version is built and there is another linking problem: Error : .onLoad failed in loadNamespace() for 'RcppParallel', details: call: dyn.load(tbb, local = FALSE, now = TRUE) error: unable to load shared object '/export/home/csardi/R/R-3.1.3/lib/R/library/RcppParallel/lib/libtbb.so': ld.so.1: R: fatal: relocation error: file /export/home/csardi/R/R-3.1.3/lib/R/library/RcppParallel/lib/libtbb.so: symbol __1cDstdNruntime_error2T6M_v_: referenced symbol not found https://github.com/gaborcsardi/RcppParallel/commits/feature/tbb-solaris G. On Mon, Apr 6, 2015 at 9:36 PM, Jeroen Ooms jeroeno...@gmail.com wrote: On Fri, Apr 3, 2015 at 4:44 AM, JJ Allaire jj.alla...@gmail.com wrote: git clone -b feature/tbb-solaris g...@github.com:RcppCore/RcppParallel.git R CMD build RcppParallel R CMD check --as-cran RcppParallel_4.3.7.tar.gz It complains about a problem with make: root@solaris11:~/Desktop# /opt/csw/bin/R CMD INSTALL RcppParallel_4.3.7.tar.gz * installing to library '/opt/csw/lib/R/library' * installing *source* package 'RcppParallel' ... ** libs make: Fatal error in reader: Makevars, line 5: Unexpected end of line seen ERROR: compilation failed for package 'RcppParallel' * removing '/opt/csw/lib/R/library/RcppParallel' ___ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
Re: [Rcpp-devel] RcppParallel on Solaris
Thanks Jeroen! Does the ORD version of R use GNU make? (I think we need that for the RcppParallel makefile). If it doesn't, Gábor do you know if there is a straightforward way to force it to use GNU make? (or as Jeroen hypothesizes does CRAN use gcc when testing packages that have GNU make as a SystemDependency?) On Tue, Apr 7, 2015 at 4:04 PM, Jeroen Ooms jeroeno...@gmail.com wrote: I've put a copy of my vm image here in case anyone is interested: https://gist.github.com/jeroenooms/4c61c821172ad640545d It is still unclear to me what the exact CRAN setup is. For example for the openssl package I had to use the $OPENSSL_INCLUDES variable which points to /opt/csw/includes. However on the Solaris 11 vm, the openssl headers are simply in /usr/local and can be found without any specific flags. On Tue, Apr 7, 2015 at 2:47 AM, JJ Allaire jj.alla...@gmail.com wrote: I think this is just a matter of passing the right runtime library flag to the TBB configure script (but perhaps this masks a bigger issue, we'll see). Gábor is going to make his VM available to me for further testing and iteration later this week. Hope to be able to report some progress after that. On Mon, Apr 6, 2015 at 9:41 PM, Gábor Csárdi csardi.ga...@gmail.com wrote: Yeah, you need GNU make. We had some progress off-list. This is where we are now: Btw. if you remove -m64, then the 32 bit version is built and there is another linking problem: Error : .onLoad failed in loadNamespace() for 'RcppParallel', details: call: dyn.load(tbb, local = FALSE, now = TRUE) error: unable to load shared object '/export/home/csardi/R/R-3.1.3/lib/R/library/RcppParallel/lib/libtbb.so': ld.so.1: R: fatal: relocation error: file /export/home/csardi/R/R-3.1.3/lib/R/library/RcppParallel/lib/libtbb.so: symbol __1cDstdNruntime_error2T6M_v_: referenced symbol not found https://github.com/gaborcsardi/RcppParallel/commits/feature/tbb-solaris G. On Mon, Apr 6, 2015 at 9:36 PM, Jeroen Ooms jeroeno...@gmail.com wrote: On Fri, Apr 3, 2015 at 4:44 AM, JJ Allaire jj.alla...@gmail.com wrote: git clone -b feature/tbb-solaris g...@github.com:RcppCore/RcppParallel.git R CMD build RcppParallel R CMD check --as-cran RcppParallel_4.3.7.tar.gz It complains about a problem with make: root@solaris11:~/Desktop# /opt/csw/bin/R CMD INSTALL RcppParallel_4.3.7.tar.gz * installing to library '/opt/csw/lib/R/library' * installing *source* package 'RcppParallel' ... ** libs make: Fatal error in reader: Makevars, line 5: Unexpected end of line seen ERROR: compilation failed for package 'RcppParallel' * removing '/opt/csw/lib/R/library/RcppParallel' ___ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
Re: [Rcpp-devel] RcppParallel on Solaris
On Thu, Apr 2, 2015 at 3:42 PM, Gábor Csárdi csardi.ga...@gmail.com wrote: I have Solaris x86_64 on Virtualbox, and it runs fine. The problem is, it is x86, and not Sparc, so for example byte order issues do not come up. Would you be willing to try the following as a minimal test that our changes are enough to make RcppParallel work on Solaris x86_64: git clone -b feature/tbb-solaris g...@github.com:RcppCore/RcppParallel.git R CMD build RcppParallel R CMD check --as-cran RcppParallel_4.3.7.tar.gz ___ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
Re: [Rcpp-devel] RcppParallel on Solaris
I am happy to share my experience with Solaris. I have been meaning to write a step-by-step guide to setting up a Solaris testing platform for R on a VirtualBox VM, so this gives me some impetus to write it. That would be excellent! I got as far as getting R running on a VirtualBox but got hung up on compiler configuration such that was unable to build packages that included native code. ___ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
Re: [Rcpp-devel] RcppParallel on Solaris
As far as I know it is possible to run OpenSolaris. Let me check. On 2 April 2015 at 16:31, Romain Francois wrote: | Would be useful to have a solaris machine on aws or whatever with R all these toils installed. | | I once had a virtual machine set up thanks to martyn but the user experience of virtualbox + keyboard conflicts with my french mac book keyboard was not that good. | | Does someone have the skills to set up a machine on aws once and (almost for all) ? | Would someone be willing to pay for the machine ? The R Foundation (and related entities) really should own this. We have some paperwork to get out of the way first but given how we all suffer from Solaris complaints it really should be possible to submit build jobs as we do on Windows. Maybe by end-of-year. If only we had someone with Solaris skills :-/ ... but maybe proper Unix skills will do for creating / administering a build queue. Dirk | | Romain | | Le 2 avr. 2015 à 15:58, JJ Allaire jj.alla...@gmail.com a écrit : | | We've recently done some work on RcppParallel to make it work on | Windows (previously it worked on Linux and OS X only). It would be | wonderful to also get it working on Solaris, and in theory this | shouldn't be difficult as the library at the core of RcppParallel, | Intel Threading Building Blocks (TBB), has full support for both Sun | and GCC compilers on Solaris | | This branch does what I think is necessary: | | https://github.com/RcppCore/RcppParallel/tree/feature/tbb-solaris | | Unfortunately none of my attempts to setup a working Solaris | development environment have succeeded so I have no way of testing | this. I'm writing to see if anyone on the list would be willing to | help with testing/tweaking on Solaris. | | More background on RcppParallel is here: http://rcppcore.github.io/RcppParallel/ | | Thanks, | | J.J. | ___ | Rcpp-devel mailing list | Rcpp-devel@lists.r-forge.r-project.org | https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel | ___ | Rcpp-devel mailing list | Rcpp-devel@lists.r-forge.r-project.org | https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel -- http://dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org ___ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel ___ Rcpp-devel mailing list Rcpp-devel@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
