Re: Running synchronized JRI code

2016-02-15 14:02 GMT+01:00 Sun, Rui : > On computation, RRDD launches one R process for each partition, so there > won't be thread-safe issue > > Could you give more details on your new environment? Running on EC2, I start the executors via /usr/bin/R CMD javareconf -e

RE: Running synchronized JRI code

rui@intel.com> Cc: user <user@spark.apache.org> Subject: Re: Running synchronized JRI code 2016-02-15 4:35 GMT+01:00 Sun, Rui <rui@intel.com>: > Yes, JRI loads an R dynamic library into the executor JVM, which faces > thread-safe issue when there are multiple task threads wi

Re: Running synchronized JRI code

2016-02-15 4:35 GMT+01:00 Sun, Rui : > Yes, JRI loads an R dynamic library into the executor JVM, which faces > thread-safe issue when there are multiple task threads within the executor. > > I am thinking if the demand like yours (calling R code in RDD > transformations) is

RE: Running synchronized JRI code

For YARN mode, you can set --executor-cores 1 -Original Message- From: Sun, Rui [mailto:rui@intel.com] Sent: Monday, February 15, 2016 11:35 AM To: Simon Hafner <reactorm...@gmail.com>; user <user@spark.apache.org> Subject: RE: Running synchronized JRI code Yes, JR

RE: Running synchronized JRI code

it is currently intended for internal use by SparkR and not a public API. -Original Message- From: Simon Hafner [mailto:reactorm...@gmail.com] Sent: Monday, February 15, 2016 5:09 AM To: user <user@spark.apache.org> Subject: Running synchronized JRI code Hello I'm currently running

Running synchronized JRI code

Hello I'm currently running R code in an executor via JRI. Because R is single-threaded, any call to R needs to be wrapped in a `synchronized`. Now I can use a bit more than one core per executor, which is undesirable. Is there a way to tell spark that this specific application (or even specific