Re: [R] Parallel processes collapse into one

2019-06-11 Thread Nicola Lunardon
Hi, thanks for the hints. I am quite sure that at some points R stalls as I
am used to monitor the progress of simulations (with a custom function
similar to pbmclapply) and save partial results: whenever all R instances
collapse into one, then no more updates show up.
I was able to set OPENBLAS_NUM_THREADS=1 (1=the value calculated by using
the formula you provided) as suggested here
https://stat.ethz.ch/pipermail/r-sig-debian/2016-August/002588.html, but it
seemed to have no effect on the machine I am using. However, I managed to
force R to use standard BLAS/LAPACK libraries, hope this is going to solve
the problem

sessionInfo()
R version 3.6.0 (2019-04-26)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 16.04.6 LTS

Matrix products: default
BLAS:   /usr/lib/libblas/libblas.so.3.6.0
LAPACK: /usr/lib/lapack/liblapack.so.3.6.0

Il giorno gio 6 giu 2019 alle ore 13:20 Ivan Krylov 
ha scritto:

> On Mon, 3 Jun 2019 06:37:46 +0200
> Nicola Lunardon  wrote:
>
> > R seems to be running, but simulations do not progress further.
>
> Have you tried message() (or REprintf() in C code) to narrow down the
> specific part of the code where simulations stop in their progress?
> It's less convenient than a good debugger, but with parallel code it is
> sometimes the only way to reproduce the problem and get some
> information about it.
>
> > If I run simulations on 16 cores I end up having an R instance with
> > CPU usage about 1600%, never experienced such a behaviour.
>
> > BLAS:   /usr/lib/openblas-base/libblas.so.3
> > LAPACK: /usr/lib/libopenblasp-r0.2.18.so
>
> OpenBLAS can, indeed, use multiple threads on its own, inside a single
> process. Combined with mclapply, this might create a situation when
> there are far more threads competing for CPU time than CPU cores
> available. Does it help if you set an environment variable such as
> OPENBLAS_NUM_THREADS [*] to a number less than or equal to (number of
> CPU cores / mc.cores mclapply argument)?
>
> --
> Best regards,
> Ivan
>
> [*]
>
> https://github.com/xianyi/OpenBLAS#setting-the-number-of-threads-using-environment-variables
>

[[alternative HTML version deleted]]

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.


[R] Parallel processes collapse into one

2019-06-04 Thread Nicola Lunardon
I have written an R package to collect some functions to run simulations
for a research project. Main functions are written in C and make use of
BLAS routines, such as dsymm, dgemm, and ddot. I run simulations in
parallel by using mclapply and the problem is that after some point all R
instances collapse into one: R seems to be running, but simulations do not
progress further. If I run simulations on 16 cores I end up having an R
instance with CPU usage about 1600%, never experienced such a behaviour.

sessionInfo()
R version 3.6.0 (2019-04-26)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 16.04.6 LTS

Matrix products: default
BLAS:   /usr/lib/openblas-base/libblas.so.3
LAPACK: /usr/lib/libopenblasp-r0.2.18.so

[[alternative HTML version deleted]]

__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.