Re: [R-pkg-devel] callr and CRAN policy on the max number of cores
Two parallel processes are okay, so you can use two callr background
processes ("children"), as long as you don't do heavy computations on
the main R process ("parent"). This is commonly done in many R
packages, e.g.
library(parallel)
cl <- makeCluster(2)
y <- parLapply(cl = cl, X = 1:10, fun = slow_fcn)
stopCluster(cl)
/Henrik
On Wed, May 22, 2024 at 7:50 PM Shu Fai Cheung wrote:
>
> Hil,
>
> I am exploring the use of callr in a package. I know that for packages
> that do parallel computing, we should not use more than two cores in
> examples, tests, etc. when being checked on CRAN.
>
> I believe I can use callr in tests and examples. How should I use
> callr in compliance with CRAN policy? Because even just one call to
> callr will start a new R session, should I limit the use of callr such
> that at any one time, only one background R session created by callr
> is active?
>
> Regards,
> Shu Fai
>
> __
> R-package-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-package-devel
__
R-package-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-package-devel
[R-pkg-devel] callr and CRAN policy on the max number of cores
Hil, I am exploring the use of callr in a package. I know that for packages that do parallel computing, we should not use more than two cores in examples, tests, etc. when being checked on CRAN. I believe I can use callr in tests and examples. How should I use callr in compliance with CRAN policy? Because even just one call to callr will start a new R session, should I limit the use of callr such that at any one time, only one background R session created by callr is active? Regards, Shu Fai __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] Compile issues on r-devel-linux-x86_64-debian-clang with OpenMP
On 2024-05-22 10:18 a.m., Dirk Eddelbuettel wrote:
On 22 May 2024 at 13:54, Nixon, Michelle Pistner wrote:
| Thank you both for your responses and help! Kurt-- your message makes a lot of
| sense. I'll try to debug soon and will reach out if I have more questions.
Interesting.
Kurt, is there a recommended way to test for this (rare, I may add) case of
'OpenMP present but usage verboten by R' ? I do not see an option for 'R CMD
config' jumping out, and there is no pkg-config file either.
Testing via 'nm' as you show is possible but not exactly 'portable'. So any
suggestions as to what to condition on here? Michelle did AFAICT the Right
Thing (TM) by 'borrowing' from the fairly mature check in RcppArmadillo.
Not that I know much about writing configure tests, but won't
src/include/config.h in the R build dir have "HAVE_OPENMP" defined if
OpenMP is supported?
Duncan
Dirk
|
| Thanks,
| Michelle
|
━━━
| From: Kurt Hornik
| Sent: Wednesday, May 22, 2024 3:57 AM
| To: Dirk Eddelbuettel
| Cc: Nixon, Michelle Pistner ; r-package-devel@r-project.org
| ; Kurt Hornik
| Subject: Re: [R-pkg-devel] Compile issues on r-devel-linux-x86_64-debian-clang
| with OpenMP
|
| [You don't often get email from kurt.hor...@wu.ac.at. Learn why this is
| important at https://aka.ms/LearnAboutSenderIdentification ]
|
| > Dirk Eddelbuettel writes:
|
| Friends, the Debian pretest check system uses LLVM 18 and has the
| corresponding OpenMP headers and libraries installed, but R is
| configured not to use these. However, the configure test in fido does
|
| cat < test-omp.cpp
| #include
| int main() {
| return omp_get_num_threads();
| }
| EOF
|
| ## Execute R CMD SHLIB.
| "${R_HOME}/bin/R" CMD SHLIB test-omp.cpp >/dev/null 2>&1
| if test x"$?" = x"0"; then
| AC_MSG_RESULT([yes])
| openmp_already_works="yes"
| else
| AC_MSG_RESULT([no])
| fi
|
| which does not work as intended: R CMD SHLIB will happily link but the
| .so will end up with an unresolved symbol:
|
| $ nm test-omp.so | grep omp_
| U omp_get_num_threads
|
| Hth
| -k
|
| > Hi Michelle,
|
| > On 21 May 2024 at 13:46, Nixon, Michelle Pistner wrote:
| > | Hi all,
| > |
| > | I'm running into build issues for my package (fido: https://
| nam10.safelinks.protection.outlook.com/?url=
| https%3A%2F%2Fgithub.com%2Fjsilve24%2Ffido&data=
|
05%7C02%7Cmap5672%40psu.edu%7C7993944f52ae4a017d3d08dc7a34d1d6%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C638519614500263726%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C
| &sdata=C4BI6%2FUgTjd%2FCRPzBF8NuUkPvlQkIxb2r1%2BzIqKNPpE%3D&reserved=0) on the
| r-devel-linux-x86_64-debian-clang system on CRAN (full check log here:
https://
| nam10.safelinks.protection.outlook.com/?url=
|
https%3A%2F%2Fwin-builder.r-project.org%2Fincoming_pretest%2Ffido_1.1.0_20240515_211644%2FDebian%2F00install.out
| &data=
|
05%7C02%7Cmap5672%40psu.edu%7C7993944f52ae4a017d3d08dc7a34d1d6%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C638519614500272862%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C
| &sdata=TKpQrkImemwz%2FuK7yYqCLEECU6WCCvgYwcrAId%2Bpmb0%3D&reserved=0). fido
| relies on several of the Rcpp packages, and I think the error is due to how
| OpenMP is set up in our package. The error in question states:
| > |
| > | "Error: package or namespace load failed for �fido� in dyn.load(file,
| DLLpath = DLLpath, ...):
| > | unable to load shared object
'/home/hornik/tmp/R.check/r-devel-clang/Work/
| build/Packages/00LOCK-fido/00new/fido/libs/fido.so':
| > | /home/hornik/tmp/R.check/r-devel-clang/Work/build/Packages/00LOCK-fido/
| 00new/fido/libs/fido.so: undefined symbol: omp_get_thread_num"
| > |
| > | I've had a hard time recreating the error, as I can successfully get the
| package to build on other systems (GitHub action results here: https://
| nam10.safelinks.protection.outlook.com/?url=
| https%3A%2F%2Fgithub.com%2Fjsilve24%2Ffido%2Factions&data=
|
05%7C02%7Cmap5672%40psu.edu%7C7993944f52ae4a017d3d08dc7a34d1d6%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C638519614500275695%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C
| &sdata=zXCul23gWAevfEhCsfyJg8KewU8fjSuy1qZgZemNy7M%3D&reserved=0) including a
| system using the same version of R/clang as the failing CRAN check. Looking at
| the logs between the two, the major difference is the lack of -fopenmp in the
| compiling function on the CRAN version (which is there on the r-hub check
| version with the same specifications):
| > |
| > | (From the CRAN version) clang++-18 -std=gnu++17 -shared
-L/home/hornik/tmp/
| R-d-clang-18/lib -Wl,-O1 -o fido.so ConjugateLinearModel.o
| MaltipooCollapsed_LGH.o MaltipooCollapsed_Optim.o MatrixAlgebra.o
| PibbleCollapsed_LGH.o PibbleCollapsed_Optim.o PibbleCollapsed_Uncollapse.
Re: [R-pkg-devel] Compile issues on r-devel-linux-x86_64-debian-clang with OpenMP
On 22 May 2024 at 13:54, Nixon, Michelle Pistner wrote:
| Thank you both for your responses and help! Kurt-- your message makes a lot of
| sense. I'll try to debug soon and will reach out if I have more questions.
Interesting.
Kurt, is there a recommended way to test for this (rare, I may add) case of
'OpenMP present but usage verboten by R' ? I do not see an option for 'R CMD
config' jumping out, and there is no pkg-config file either.
Testing via 'nm' as you show is possible but not exactly 'portable'. So any
suggestions as to what to condition on here? Michelle did AFAICT the Right
Thing (TM) by 'borrowing' from the fairly mature check in RcppArmadillo.
Dirk
|
| Thanks,
| Michelle
|
━━━
| From: Kurt Hornik
| Sent: Wednesday, May 22, 2024 3:57 AM
| To: Dirk Eddelbuettel
| Cc: Nixon, Michelle Pistner ; r-package-devel@r-project.org
| ; Kurt Hornik
| Subject: Re: [R-pkg-devel] Compile issues on r-devel-linux-x86_64-debian-clang
| with OpenMP
|
| [You don't often get email from kurt.hor...@wu.ac.at. Learn why this is
| important at https://aka.ms/LearnAboutSenderIdentification ]
|
| > Dirk Eddelbuettel writes:
|
| Friends, the Debian pretest check system uses LLVM 18 and has the
| corresponding OpenMP headers and libraries installed, but R is
| configured not to use these. However, the configure test in fido does
|
| cat < test-omp.cpp
| #include
| int main() {
| return omp_get_num_threads();
| }
| EOF
|
| ## Execute R CMD SHLIB.
| "${R_HOME}/bin/R" CMD SHLIB test-omp.cpp >/dev/null 2>&1
| if test x"$?" = x"0"; then
| AC_MSG_RESULT([yes])
| openmp_already_works="yes"
| else
| AC_MSG_RESULT([no])
| fi
|
| which does not work as intended: R CMD SHLIB will happily link but the
| .so will end up with an unresolved symbol:
|
| $ nm test-omp.so | grep omp_
| U omp_get_num_threads
|
| Hth
| -k
|
| > Hi Michelle,
|
| > On 21 May 2024 at 13:46, Nixon, Michelle Pistner wrote:
| > | Hi all,
| > |
| > | I'm running into build issues for my package (fido: https://
| nam10.safelinks.protection.outlook.com/?url=
| https%3A%2F%2Fgithub.com%2Fjsilve24%2Ffido&data=
|
05%7C02%7Cmap5672%40psu.edu%7C7993944f52ae4a017d3d08dc7a34d1d6%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C638519614500263726%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C
| &sdata=C4BI6%2FUgTjd%2FCRPzBF8NuUkPvlQkIxb2r1%2BzIqKNPpE%3D&reserved=0) on the
| r-devel-linux-x86_64-debian-clang system on CRAN (full check log here:
https://
| nam10.safelinks.protection.outlook.com/?url=
|
https%3A%2F%2Fwin-builder.r-project.org%2Fincoming_pretest%2Ffido_1.1.0_20240515_211644%2FDebian%2F00install.out
| &data=
|
05%7C02%7Cmap5672%40psu.edu%7C7993944f52ae4a017d3d08dc7a34d1d6%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C638519614500272862%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C
| &sdata=TKpQrkImemwz%2FuK7yYqCLEECU6WCCvgYwcrAId%2Bpmb0%3D&reserved=0). fido
| relies on several of the Rcpp packages, and I think the error is due to how
| OpenMP is set up in our package. The error in question states:
| > |
| > | "Error: package or namespace load failed for �fido� in dyn.load(file,
| DLLpath = DLLpath, ...):
| > | unable to load shared object
'/home/hornik/tmp/R.check/r-devel-clang/Work/
| build/Packages/00LOCK-fido/00new/fido/libs/fido.so':
| > | /home/hornik/tmp/R.check/r-devel-clang/Work/build/Packages/00LOCK-fido/
| 00new/fido/libs/fido.so: undefined symbol: omp_get_thread_num"
| > |
| > | I've had a hard time recreating the error, as I can successfully get the
| package to build on other systems (GitHub action results here: https://
| nam10.safelinks.protection.outlook.com/?url=
| https%3A%2F%2Fgithub.com%2Fjsilve24%2Ffido%2Factions&data=
|
05%7C02%7Cmap5672%40psu.edu%7C7993944f52ae4a017d3d08dc7a34d1d6%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C638519614500275695%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C
| &sdata=zXCul23gWAevfEhCsfyJg8KewU8fjSuy1qZgZemNy7M%3D&reserved=0) including a
| system using the same version of R/clang as the failing CRAN check. Looking at
| the logs between the two, the major difference is the lack of -fopenmp in the
| compiling function on the CRAN version (which is there on the r-hub check
| version with the same specifications):
| > |
| > | (From the CRAN version) clang++-18 -std=gnu++17 -shared
-L/home/hornik/tmp/
| R-d-clang-18/lib -Wl,-O1 -o fido.so ConjugateLinearModel.o
| MaltipooCollapsed_LGH.o MaltipooCollapsed_Optim.o MatrixAlgebra.o
| PibbleCollapsed_LGH.o PibbleCollapsed_Optim.o PibbleCollapsed_Uncollapse.o
| PibbleCollapsed_Uncollapse_sigmaKnown.o RcppExports.o SpecialFunctions.o
| test_LaplaceApproximation.o test_MultDirichletBoot.o test_utils.o -L/home/
| hornik/tmp/R-d-clang-18/lib -lR
| > |
| > | My initi
Re: [R-pkg-devel] Compile issues on r-devel-linux-x86_64-debian-clang with OpenMP
Thank you both for your responses and help! Kurt-- your message makes a lot of
sense. I'll try to debug soon and will reach out if I have more questions.
Thanks,
Michelle
From: Kurt Hornik
Sent: Wednesday, May 22, 2024 3:57 AM
To: Dirk Eddelbuettel
Cc: Nixon, Michelle Pistner ; r-package-devel@r-project.org
; Kurt Hornik
Subject: Re: [R-pkg-devel] Compile issues on r-devel-linux-x86_64-debian-clang
with OpenMP
[You don't often get email from kurt.hor...@wu.ac.at. Learn why this is
important at https://aka.ms/LearnAboutSenderIdentification ]
> Dirk Eddelbuettel writes:
Friends, the Debian pretest check system uses LLVM 18 and has the
corresponding OpenMP headers and libraries installed, but R is
configured not to use these. However, the configure test in fido does
cat < test-omp.cpp
#include
int main() {
return omp_get_num_threads();
}
EOF
## Execute R CMD SHLIB.
"${R_HOME}/bin/R" CMD SHLIB test-omp.cpp >/dev/null 2>&1
if test x"$?" = x"0"; then
AC_MSG_RESULT([yes])
openmp_already_works="yes"
else
AC_MSG_RESULT([no])
fi
which does not work as intended: R CMD SHLIB will happily link but the
.so will end up with an unresolved symbol:
$ nm test-omp.so | grep omp_
U omp_get_num_threads
Hth
-k
> Hi Michelle,
> On 21 May 2024 at 13:46, Nixon, Michelle Pistner wrote:
> | Hi all,
> |
> | I'm running into build issues for my package (fido:
> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fjsilve24%2Ffido&data=05%7C02%7Cmap5672%40psu.edu%7C7993944f52ae4a017d3d08dc7a34d1d6%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C638519614500263726%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=C4BI6%2FUgTjd%2FCRPzBF8NuUkPvlQkIxb2r1%2BzIqKNPpE%3D&reserved=0)
> on the r-devel-linux-x86_64-debian-clang system on CRAN (full check log
> here:
> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwin-builder.r-project.org%2Fincoming_pretest%2Ffido_1.1.0_20240515_211644%2FDebian%2F00install.out&data=05%7C02%7Cmap5672%40psu.edu%7C7993944f52ae4a017d3d08dc7a34d1d6%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C638519614500272862%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=TKpQrkImemwz%2FuK7yYqCLEECU6WCCvgYwcrAId%2Bpmb0%3D&reserved=0).
> fido relies on several of the Rcpp packages, and I think the error is due to
> how OpenMP is set up in our package. The error in question states:
> |
> | "Error: package or namespace load failed for �fido� in dyn.load(file,
> DLLpath = DLLpath, ...):
> | unable to load shared object
> '/home/hornik/tmp/R.check/r-devel-clang/Work/build/Packages/00LOCK-fido/00new/fido/libs/fido.so':
> |
> /home/hornik/tmp/R.check/r-devel-clang/Work/build/Packages/00LOCK-fido/00new/fido/libs/fido.so:
> undefined symbol: omp_get_thread_num"
> |
> | I've had a hard time recreating the error, as I can successfully get the
> package to build on other systems (GitHub action results here:
> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fjsilve24%2Ffido%2Factions&data=05%7C02%7Cmap5672%40psu.edu%7C7993944f52ae4a017d3d08dc7a34d1d6%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C638519614500275695%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=zXCul23gWAevfEhCsfyJg8KewU8fjSuy1qZgZemNy7M%3D&reserved=0)
> including a system using the same version of R/clang as the failing CRAN
> check. Looking at the logs between the two, the major difference is the lack
> of -fopenmp in the compiling function on the CRAN version (which is there on
> the r-hub check version with the same specifications):
> |
> | (From the CRAN version) clang++-18 -std=gnu++17 -shared
> -L/home/hornik/tmp/R-d-clang-18/lib -Wl,-O1 -o fido.so ConjugateLinearModel.o
> MaltipooCollapsed_LGH.o MaltipooCollapsed_Optim.o MatrixAlgebra.o
> PibbleCollapsed_LGH.o PibbleCollapsed_Optim.o PibbleCollapsed_Uncollapse.o
> PibbleCollapsed_Uncollapse_sigmaKnown.o RcppExports.o SpecialFunctions.o
> test_LaplaceApproximation.o test_MultDirichletBoot.o test_utils.o
> -L/home/hornik/tmp/R-d-clang-18/lib -lR
> |
> | My initial thought was an issue in the configure scripts (which we borrowed
> heavily from RcppArmadillo but made slight changes to (which is the most
> likely cause if there is issue here)) or that there is some mismatch
> somewhere as to whether or not OpenMP is available, but there isn't an
> obvious bug to me.
> |
> | Any guidance on how to debug would be greatly appreciated!
> I seem to recall that that machine is 'known-bad' for OpenMP due to the
> reliance on clang-18 which cannot (?) build with it. Might be best to
> contact
