Hi to all
I am working on Geostatistics and WPS and I was using Rpy to access
Gstat and run geostatistics interpolations. This was working fine but R
only runs on one core so things where going very slowly. So I looked at
several possibilities and the best was RSOAP.
My current system makes python threads and each thread requests a new
SOAP port and sends sub-sets of data to R parallel sessions.
Now I have 100% CPU time in my dual core machine, but to be honest the
SOAP approach requires the same amount of time as Rpy because of all the
XML that has to go back and forward (I am running everything in debug
mode). But my implementation machine is a 8 core server so I am
expecting to have some gain in this machine.
I tried to use Paralel Python (PP) instead of using native threads but
the list of modules/global variables on the SOAPpy module of python is
to extensive and PP didn't worked.
Also when using several threads in Python and one of the threads running
RPy with some intensive calculation, the other Python threads will lag
behind, this is due to the 100byte code execution at the time, but I
haven't checked if changing the sys.setcheckinterval() parameter for
threads would help (maybe making the execution more atomically would
solve this problem)
Best Jorge
--
Ph.D. Jorge Samuel Mendes de Jesus
European Commission (EC)
Joint Research Centre Directorate (DG JRC)
Institute for Environment and Sustainability (IES)
TP 441, Via Fermi 1
21020 Ispra (VA)
Italy
Phone: +39 0332 78 3536
Fax: +39 0332 78 5466
http://rem.jrc.cec.eu.int
"The views expressed are purely those of the writer and may not in any
circumstances be regarded as stating an official position of the
European Commission"
Gregory Warnes wrote:
Hi Laurent,
It does seem desirable to include an component with Rpy for handling
multi-threading and R. We should think a bit about the right way to
implement this. I think it would be good to make sure and use an code
implementation that is a good match for the right conceptual model.
To this end, perhaps we should implement an "R thread object" which
has 'exclusive' access to R. All of the "other" threads communicate
with R through this separate R thread object.
..
-G
On 6/26/08 11:42AM , "Laurent Gautier" <[EMAIL PROTECTED]> wrote:
Speaking of which, it would be nice if rpy was shipped with a vanilla
locking mechanism.
R remaining probably thread-unsafe for the foreseeable future, I was
thinking of having something
like a two-locks mechanism that would lock R and release Python's GIL
whenever an access to R is made.
The expected benefits are:
- explicit error message when someone tries a multithreaded approach
(currently, I suspect that
what is happening in such a case is unpredictable)
- ability to have some level of parallelization where python-ish
things can still be done while R is busy (useful for GUIs, or for
cases where preparing a next batch of data can be done by python while
R is still computing on the current batch)
The downside I am seeing is it will make the handling of Python
callbacks (python functions passed to R functions as arguments)
complicated (if possible).
Ideas, or help, on the matter are welcome.
L.
2008/6/26 Gregory Warnes <[EMAIL PROTECTED]>:
>
> Hello Laurent,
>
> The R system itself is not thread safe, and has quite a bit of
persistent
> state. Therefore if you want to use R from multiple threads, you
will need
> to arrange to have a single thread interact with R at a time via
appropriate
> locking or delegation.
>
> -G
>
>
> On 6/26/08 11:17AM , "laurent oget" <[EMAIL PROTECTED]> wrote:
>
> fellows,
>
> did anybody experiment with running rpy in 2 threads?
>
> ________________________________
> -------------------------------------------------------------------------
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services for
> just about anything Open Source.
> http://sourceforge.net/services/buy/index.php
>
> --
> Gregory R. Warnes, Ph.D
> Program Director
> Center for Computational Arts, Sciences, and Engineering
> University of Rochester
>
> Tel: 585-273-2794
> Fax: 585-276-2097
> Email: [EMAIL PROTECTED]
>
>
> -------------------------------------------------------------------------
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services for
> just about anything Open Source.
> http://sourceforge.net/services/buy/index.php
> _______________________________________________
> rpy-list mailing list
> rpy-list@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/rpy-list
>
>
-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
rpy-list mailing list
rpy-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rpy-list
--
Gregory R. Warnes, Ph.D
Program Director
Center for Computational Arts, Sciences, and Engineering
University of Rochester
Tel: 585-273-2794
Fax: 585-276-2097
Email: [EMAIL PROTECTED]
------------------------------------------------------------------------
-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
------------------------------------------------------------------------
_______________________________________________
rpy-list mailing list
rpy-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rpy-list
--
Ph.D. Jorge Samuel Mendes de Jesus
European Commission (EC)
Joint Research Centre Directorate (DG JRC)
Institute for Environment and Sustainability (IES)
TP 441, Via Fermi 1
21020 Ispra (VA)
Italy
Phone: +39 0332 78 3536
Fax: +39 0332 78 5466
http://rem.jrc.cec.eu.int
"The views expressed are purely those of the writer and may not in any circumstances
be regarded as stating an official position of the European Commission"
-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
rpy-list mailing list
rpy-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/rpy-list
